Welcome To The Home Of The Visual FoxPro Experts  
home. signup. forum. archives. search. google. articles. downloads. faq. members. weblogs. file info. rss.
 From: Tony Vignone
  Where is Tony Vignone?
 St Augustine
 Florida - United States
 Tony Vignone
 To: Pete Sass
  Where is Pete Sass?
 Marathon, Ontario
 Canada
 Pete Sass
 Tags
Subject: RE: NEED FOR SPEED STRING PARSING
Thread ID: 416207 Message ID: 416276 # Views: 38 # Ratings: 1
Version: Visual FoxPro 9 SP2 Category: General VFP Topics
Date: Thursday, December 25, 2014 5:57:14 PM         
   


Does this unreadable stuff occur at the beginning only? Or is it scattered throughout the file.
Could this file be like a DBF with binary definitions at the beginning?
If it is at the beginning only, it must be binary stuff defining the file the same way there is binary stuff at the beginning of DBFs.
So you could just bypass it. First measure how long it is, then do a separate FREAD before the loop. Something like this:

tau = FOPEN("tester.dat")
leadingStuff = FREAD(tau,687)	&& Pre-strip the leading file stuff (Say, you found it to be 687 bytes long) 
DO WHILE NOT FEOF(tau)
	rec = FREAD(tau,216)
	INSERT INTO extract VALUES (rec)
ENDDO


If there is more binary stuff, then intersperse the extra FREAD within the loop. The best way to view this file and get the lengths of these things in by using and editor like ULTRAEDIT-32 which can read any file and show the contents in hex and ascii.

From the dbf below, it look like the leading stuff is 6*216 + some more in the 7th record. Find out how much by pointing to the 7th record and in the Command Window, find out how long with ?AT("CRMG",fld) - 1.

Tony

>
> Hi Tony,
> I think is most Windows type files your low-level approach would work
> just fine.
> However, I ran it against my Unix based file filled with non-ascii characters
> and no pre-prepped CR + LF at the end of each line and I can look at the
> lines in a dbf, but I cannot work with them.
>
> In some line showing data if you click in the record raw in a browse window
> all data vanishes and cannot be seen anymore.
> VIEWING RECORD PRIOR TO CLICKING INTO THE RECORD IN RAW BROWSE WINDOW
>
>
> DATA VANISHES WHEN RECORD POINTER ON RECORD IN BROWSE WINDOW
>
>
> Also if I perform a substr(1,216) on this same record the sting returned is
> "". FoxPro refuses to read any data in the dbf and all attempts to read in the
> visible data returns a blank or "" value.
>
> So although data exists in the file, any and all attempts to work with the
> data fails. Seeming my only option is to totally pre-strip the Unix non-ascii
> characters out first and add in the required CR + LF characters to mark the
> end of a record under Windows and then stuff the data into the dbf field.
>
> You will notice as well that by leaving the binary characters in the record
> this also compromises the record string length of 216 characters as the file
> header contains this record string length and if you leave the binary characters
> in then the string length you take out with the low-level read in 216 length
> strings is the character data in the 216 length string plus any binary characters.
> This now totally throws off the correct string length and now even if you could
> read the record parsing my becomes impossible as the same field for each record
> is in a different substr() position in the record string.
>
> So with the old Unix bindary characters left in the file it is unusable to
> work with or manipulate.
>
> I tested the time on the processes and the adding into a dbf is not the time
> issue, the time is being eaten up with the stripping that has to evaluate each
> character one at a time and remove it or replace it.
>
> So in order to use this approach I would have to pre-strip the file and copy
> to cleaned txt file and then use the low-level approach to add in the records.
>
> I short test on 8,000 records did not result in any improvements due to the
> pre-prep that must be done first.
>
> Pete "the IceMan", from the Great White North of Canada.
> www.marathongriffincomputers.com

ENTIRE THREAD

NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 6:53:08 PM
RE: NEED FOR SPEED STRING PARSING Posted by Ali Koumaiha @ 12/23/2014 7:04:00 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 9:18:02 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/23/2014 9:32:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:34:52 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 12:40:30 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 1:59:48 AM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 4:06:32 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 1:58:25 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:48:33 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 3:08:43 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:26:18 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:57:22 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:35:06 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 3:01:56 AM
RE: NEED FOR SPEED STRING PARSING Posted by Doug Hennig @ 12/23/2014 10:28:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:10:55 PM
RE: NEED FOR SPEED STRING PARSING Posted by Randy Bosma @ 12/26/2014 4:03:13 PM
RE: NEED FOR SPEED STRING PARSING Posted by Chuanbing Chen @ 12/24/2014 12:16:21 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:06:53 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:49:20 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/29/2014 6:05:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/29/2014 7:30:28 PM
RE: NEED FOR SPEED STRING PARSING Posted by Zia Mughal @ 12/24/2014 8:23:15 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 4:50:30 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 5:07:32 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:53:57 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 9:21:34 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 8:31:14 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 4:56:51 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 5:57:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 9:01:03 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 2:51:27 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/26/2014 4:09:19 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 4:25:35 PM