Welcome To The Home Of The Visual FoxPro Experts  
home. signup. forum. archives. search. google. articles. downloads. faq. members. weblogs. file info. rss.
 From: Pete Sass
  Where is Pete Sass?
 Marathon, Ontario
 Canada
 Pete Sass
 To: Tony Vignone
  Where is Tony Vignone?
 St Augustine
 Florida - United States
 Tony Vignone
 Tags
Subject: RE: NEED FOR SPEED STRING PARSING
Thread ID: 416207 Message ID: 416311 # Views: 39 # Ratings: 0
Version: Visual FoxPro 9 SP2 Category: General VFP Topics
Date: Friday, December 26, 2014 4:09:19 PM         
   


> Hi,
> It looks like you have a handle on the problem. I would have analyzed the 1st record with the editor to get the locations of the parts to be discarded then discard them like this:
>
>
tau = FOPEN("tester.dat")
> leadingStuff = FREAD(tau,687)	&& Bypass leading file stuff
> DO WHILE NOT FEOF(tau)
> 	part1 = FREAD(tau,100)
> 	= FREAD(tau,10)		&& Discard
> 	part2 = FREAD(tau,70)
> 	= FREAD(tau,5)		&& Discard
> 	part3 = FREAD(tau,52)
> 	rec = part1+part2+part3
> 	INSERT INTO extract VALUES (rec)
> ENDDO

>
> But if you have solved it, then full steam ahead.
>
> Happy New Year
>
> Tony
>
> > Hi,
> >
> > The binary characters are all though then file and this 30 year old Unix file is as
> > mentioned non-standard.
> > As it turned out with a lot of fooling around and finally downloaded a Windows like
> > BVI Unix editor I found that each record has a record starting high-bit character
> > and also each record has a record ending high-bit character. Each input text string
> > has a binary tag identifier as well at the end of the input string.
> >
> > The header provides the field layout and the record string length: character string
> > storage length without the embedded binary characters.
> >
> > I stripped way the file header from the looks of thing now I can get thing to line
> > up so a field parsing routine would now work.
> > The DBF raw in Browse window now looked like this . . .
> >
> >
> > I still had remaining the secondary issue; "if I leave the binary
> > characters in the string prior to writing them into the dbf the data in the dbf field
> > becomes unreadable with VFP . . . ?
> >
> > But I think I just found the reason I think???
> > If I modify your code like so:
> >
do while not feof(tau)
> > 	rec = fread(tau,216) + CHR(13) + CHR(10)
> > 	insert into extract values (rec)
> > enddo

> > Now adding a CR + LF on the end of the string prior to adding in the DBF it appears
> > I now can talk to the DBF record and perform a substr() that returns the values out
> > of the DBF or cursor field.
> > Prior to adding in this end of record marker attempting to use the substr() well all
> > it returned was ""
> > I now can read the data out of the field with the CHR(13) + CHR(10) added prior to
> > writing into the field.
> >
> > You will notice in the browse window there appears to be 2 spaces at the beginning
> > of each record. This is in fact a hidden binary tag to denote the beginning of
> > each record.
> >
> > I will play around with this as now I can look at the parsing inside the DBF to
> > remove the unwanted non-ascii characters to evaluate what my elapse time is.
> >
> > At this point I should have I hope a pretty clean table prior to create a flat
> > ascii file for importing purposes into SQL Server.
> >
> > This one has been a real hair puller and to make matters worse dealing with ancient
> > home blown Unix data files.
> >
> > Thanks again as usually for you addition ideas in the matter.
> >
> > Merry Xmas !
> >
> > Pete "the IceMan", from the Great White North of Canada.
> > www.marathongriffincomputers.com


Hi Tony,

Yep I think this is now licked.
The low-level to load up into a cursor and then any final clean up in the FoxPro cursor
id definitely light years faster than my fist kick at the cat.

I just processed a 1.5 Gig binary file and this includes the final clean up work and the
entire process elapse time is under 10 minutes.

The only thing I have not decided upon yet is where the final data is to go . . .
DBF, CSV or SQL Server table?

Pete "the IceMan", from the Great White North of Canada.
www.marathongriffincomputers.com

ENTIRE THREAD

NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 6:53:08 PM
RE: NEED FOR SPEED STRING PARSING Posted by Ali Koumaiha @ 12/23/2014 7:04:00 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 9:18:02 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/23/2014 9:32:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:34:52 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 12:40:30 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 1:59:48 AM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 4:06:32 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 1:58:25 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:48:33 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 3:08:43 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:26:18 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:57:22 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:35:06 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 3:01:56 AM
RE: NEED FOR SPEED STRING PARSING Posted by Doug Hennig @ 12/23/2014 10:28:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:10:55 PM
RE: NEED FOR SPEED STRING PARSING Posted by Randy Bosma @ 12/26/2014 4:03:13 PM
RE: NEED FOR SPEED STRING PARSING Posted by Chuanbing Chen @ 12/24/2014 12:16:21 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:06:53 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:49:20 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/29/2014 6:05:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/29/2014 7:30:28 PM
RE: NEED FOR SPEED STRING PARSING Posted by Zia Mughal @ 12/24/2014 8:23:15 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 4:50:30 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 5:07:32 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:53:57 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 9:21:34 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 8:31:14 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 4:56:51 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 5:57:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 9:01:03 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 2:51:27 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/26/2014 4:09:19 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 4:25:35 PM