Welcome To The Home Of The Visual FoxPro Experts  
home. signup. forum. archives. search. google. articles. downloads. faq. members. weblogs. file info. rss.
 From: Pete Sass
  Where is Pete Sass?
 Marathon, Ontario
 Canada
 Pete Sass
 To: Zia Mughal
  Where is Zia Mughal?
 Rawalpindi
 Pakistan
 Zia Mughal
 Tags
Subject: RE: NEED FOR SPEED STRING PARSING
Thread ID: 416207 Message ID: 416247 # Views: 45 # Ratings: 0
Version: Visual FoxPro 9 SP2 Category: General VFP Topics
Date: Wednesday, December 24, 2014 4:50:30 PM         
   


> > Hi Folks,
> >
> > I posted a thread a couple of days ago about conversion of a Unix file that
> > is in an old Unix system 25 - 30 years old and the original developer is
> > no longer with us. Client knows nothing about the file format and no
> > documentation exists.
> >
> > I have proven that this data files in this system were home spun and follow
> > no standard header and nothing can read the data files. Unix VI and BVI will not
> > read them . . . however I can hexdump the data files containing a you know
> > everything. The test is in the data files, but as one knows so is all the high-bit
> > characters.
> >
> > There is no Windows per-say CR or LF and these have to be added into the data
> > and I have determined that one record is a 216 character string.
> >
> > Now to my current issue . . .
> > I had to write a custom stripper program to walk through the file 1 character at
> > a time and keep it if it is a ascii character "number or letter" and remove it
> > if binary.
> >
> > OK this was not hard to do, but now comes my question. I see no other way to
> > extract the real data and due to the size of the data files 153 Meg, 180 Meg
> > and some over 1 GB in size .... Yikes. The smaller file of 153 Meg the actual
> > testing an each character level means 153 mission checks and decisions to keep
> > the character or delete it.
> > On a 150 Meg binary data file the stripper worked for 16 hours and was 12 % done.
> > So am I looking at 1 week 24X7 do accomplish just phase - I of the stripping?
> >
> > Looking a the 1 GIG plus files am I looking at ?? 6 weeks ??
> >
> > I am running on a Dell OptiPlex with i7 3rd Generation processor with no other load
> > and 16 GIG of memory.
> >
> > ANY IDEAS AT ALL ABOUT THIS FOLKS??
> >
> > The core of the stripper is a For loop where lnBitLength = 153,000,000 characters in total to test.
> > I turned off the Message and starred this out and this made little if any difference in the speed.
> >
> >
> >
FOR lnX = 1 TO lnBitLength
> > 
> >     lnPC = lnx / lnBitLength * 100
> >     
> >     * --- Hard carriage return counter.
> >     lnCRcounter = lnCRcounter + 1
> > 
> >     lcCurChar  = SUBSTR(lcDataString, lnX, 1)
> >     lcCurChar2 = SUBSTR(lcDataString, lnX - 1, 1)
> >     * --- This is going to perform a step-by-step full rewrite
> >     * --- of the entire string one character at a time.
> >     IF BETWEEN(ASC(lcCurChar), lnAscLow, lnAscHigh) && Is upper/lower alphas or numeric. 
> >           lcGoodInfo = lcGoodInfo + lcCurChar 
> >       
> >         * --- Pattern regoginzer to only add 1 space in between character information.  
> >         IF ASC(lcCurChar) = 32 .and. ASC(lcCurChar) <> 32
> >               lcGoodInfo = lcGoodInfo + " "
> >         ENDIF
> >          
> >     ENDIF 
> >     
> >     * --- Calculated string length per record at 216 characters long.
> >     * --- The only way to find the below record string length is to 
> >     * --- parse a test run of say 4,000 bits and physically count
> >     * --- the total number of character encompassing one record.
> >     IF lnCRcounter = 216
> >         lcGoodInfo = lcGoodInfo + CHR(13) + CHR(10)
> >         lnCRcounter = 0
> >     ENDIF      
> >     
> >     * --- Increment the % counter.
> >     lnPCcounter = lnPCcounter + 1
> >  
> >     * --- Show % status.
> >     lcPC = ALLTRIM(STR(lnPC,12,4))
> >     SET MESSAGE TO "Percent conversion currently at : " + lcPC
> >     * --- lcGoodInfo = lcGoodInfo + " "
> >  ENDFOR

> >
> >
> >
> > Pete "the IceMan", from the Great White North of Canada.
> > www.marathongriffincomputers.com
>
> Respected pete Sass,
>
> might this code help you for taking idea.
>
>
>
> set talk off
> close all
> clea all
> CLEAR
> *CREATE TABLE uindex1 ( data c(254))
> use uindex1 alias uindex1 in 1 order 1
> 
> mrecsize=90
> 
> STORE FOPEN('uindex1.dat') TO file_handle		&& Open the file
> STORE FSEEK(file_handle, 0, 2) TO ifp_size	&& Move pointer to EOF
> STORE FSEEK(file_handle, 0) TO ifp_top		&& Move pointer to BOF
> IF ifp_size <= 0		&& Is File empty?
> = FCLOSE(file_handle)	&& Close the file
> ELSE					&& If not empty
>  ? ifp_size
>  mtrecs= ifp_size/90  **  you can change here as record size on 216
>  mloop=0
> sele uindex1
> if eof()
> 	go bott
> endif
>  
>  do while mloop <= mtrecs
>  l_string = FREAD(file_handle, mrecsize)	&& Store to memory
> ? l_string  
> appe blank
> repl data with l_string
> mloop=mloop+1
> enddo
> 
> 
> 			= FCLOSE(file_handle)	&& Close the file
> ENDIF
> = FCLOSE(file_handle)	&& Close the file
> brow
> close data
> 			
> 

>
>
> Zia Mughal
> Pcland Software inc. Pakistan
> http://www.pcland.com.pk
> http://pclandpk.blogspot.com


Hi Zia,

This was going to be my second time test using low-level file handling
after I did a full elapse time on the straight VFP function calls and
handling routine.

I appreciate this as you save me some time in not having to do a lot
in putting together the low level approach.

This will be an interesting time study as with your code an stuffing
the binary and text string into a VFP table one ponders what will be
the speed difference when I apply the non-ascii character stripping
on a VFP table field and add in the CF + LF instead of doing everything
inside a FOR loop.

I will post and left the forum know what are the speed differences
with each approach.

Pete "the IceMan", from the Great White North of Canada.
www.marathongriffincomputers.com

ENTIRE THREAD

NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 6:53:08 PM
RE: NEED FOR SPEED STRING PARSING Posted by Ali Koumaiha @ 12/23/2014 7:04:00 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 9:18:02 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/23/2014 9:32:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:34:52 PM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 12:40:30 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 1:59:48 AM
RE: NEED FOR SPEED STRING PARSING Posted by Greg Green @ 12/24/2014 4:06:32 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 1:58:25 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:48:33 AM
RE: NEED FOR SPEED STRING PARSING Posted by Jun Tangunan @ 12/24/2014 3:08:43 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:26:18 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:57:22 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/23/2014 10:35:06 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 3:01:56 AM
RE: NEED FOR SPEED STRING PARSING Posted by Doug Hennig @ 12/23/2014 10:28:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/23/2014 11:10:55 PM
RE: NEED FOR SPEED STRING PARSING Posted by Randy Bosma @ 12/26/2014 4:03:13 PM
RE: NEED FOR SPEED STRING PARSING Posted by Chuanbing Chen @ 12/24/2014 12:16:21 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 2:06:53 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:49:20 AM
RE: NEED FOR SPEED STRING PARSING Posted by Mike Yearwood @ 12/29/2014 6:05:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/29/2014 7:30:28 PM
RE: NEED FOR SPEED STRING PARSING Posted by Zia Mughal @ 12/24/2014 8:23:15 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 4:50:30 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 5:07:32 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/24/2014 5:53:57 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/24/2014 9:21:34 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 8:31:14 AM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 4:56:51 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/25/2014 5:57:14 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/25/2014 9:01:03 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 2:51:27 PM
RE: NEED FOR SPEED STRING PARSING Posted by Pete Sass @ 12/26/2014 4:09:19 PM
RE: NEED FOR SPEED STRING PARSING Posted by Tony Vignone @ 12/26/2014 4:25:35 PM