VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
ALINES()?
--
rk -----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of mbsoftwaresolutions@mbsoftwaresolutions.com Sent: Friday, April 21, 2017 10:45 AM To: profoxtech@leafe.com Subject: Getting count of rows in a text file -- best approach?
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
On 2017-04-21 10:47, Richard Kaye wrote:
ALINES()?
Is that your answer for everything today? lol
Nah, that feels no less dirty than my approach. My bet is that would require as many (if not more) processing cycles, but of course only testing would show conclusively if at all.
Thanks though! --Mike
Only when it fits the situation. :-)
Dirty? Really? One line of code returns the number of rows in your string. Which was your original question.
Benchmark it and you'll find out which one is faster and whether or not you can live with that slightly soiled sensations... ;-)
--
rk -----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of mbsoftwaresolutions@mbsoftwaresolutions.com Sent: Friday, April 21, 2017 11:20 AM To: profoxtech@leafe.com Subject: RE: Getting count of rows in a text file -- best approach?
On 2017-04-21 10:47, Richard Kaye wrote:
ALINES()?
Is that your answer for everything today? lol
Nah, that feels no less dirty than my approach. My bet is that would require as many (if not more) processing cycles, but of course only testing would show conclusively if at all.
Thanks though! --Mike
On 2017-04-21 11:24, Richard Kaye wrote:
Only when it fits the situation. :-)
Dirty? Really? One line of code returns the number of rows in your string. Which was your original question.
My solution was 1 line too: RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) <g>
Benchmark it and you'll find out which one is faster and whether or not you can live with that slightly soiled sensations... ;-)
Testing showed these two pretty much a dead heat:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) RowCount = ALINES(aCrap,FILETOSTR(m.Filename))
Thanks. I was curious. :-)
--
rk -----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of mbsoftwaresolutions@mbsoftwaresolutions.com Sent: Friday, April 21, 2017 4:38 PM To: profoxtech@leafe.com Subject: RE: Getting count of rows in a text file -- best approach?
On 2017-04-21 11:24, Richard Kaye wrote:
Only when it fits the situation. :-)
Dirty? Really? One line of code returns the number of rows in your string. Which was your original question.
My solution was 1 line too: RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) <g>
Benchmark it and you'll find out which one is faster and whether or not you can live with that slightly soiled sensations... ;-)
Testing showed these two pretty much a dead heat:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) RowCount = ALINES(aCrap,FILETOSTR(m.Filename))
On Fri, Apr 21, 2017 at 4:37 PM, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
Testing showed these two pretty much a dead heat:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) RowCount = ALINES(aCrap,FILETOSTR(m.Filename))
But on the bright side, with Richard's one-liner, you're done reading in the file: it's sitting in the array.
On 2017-04-21 16:47, Ted Roche wrote:
On Fri, Apr 21, 2017 at 4:37 PM, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
Testing showed these two pretty much a dead heat:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) RowCount = ALINES(aCrap,FILETOSTR(m.Filename))
But on the bright side, with Richard's one-liner, you're done reading in the file: it's sitting in the array.
I don't see the difference; yes, it's in the array, but what do I want it for after that? For my processing, I was just comparing this year's file to last year's file and seeing how their record counts differed. Not really working with the file after that.
Actually, in looking at those two tests above, I would think the OCCURS would be better because in the upper, the file bulk is in whatever temporary variable created by FILETOSTR(..) but then in Richard's ALINES test, that same file is in the temporary variable like Test1 BUT it's ALSO in the aCrap array. Correct????
You just listen to Ted, Mike. He is invariably right. <g,d&r>
It's all in memory either way. As you saw the overhead either way isn't much different.
--
rk -----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of mbsoftwaresolutions@mbsoftwaresolutions.com Sent: Friday, April 21, 2017 4:50 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
On 2017-04-21 16:47, Ted Roche wrote:
On Fri, Apr 21, 2017 at 4:37 PM, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
Testing showed these two pretty much a dead heat:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename)) RowCount = ALINES(aCrap,FILETOSTR(m.Filename))
But on the bright side, with Richard's one-liner, you're done reading in the file: it's sitting in the array.
I don't see the difference; yes, it's in the array, but what do I want it for after that? For my processing, I was just comparing this year's file to last year's file and seeing how their record counts differed. Not really working with the file after that.
Actually, in looking at those two tests above, I would think the OCCURS would be better because in the upper, the file bulk is in whatever temporary variable created by FILETOSTR(..) but then in Richard's ALINES test, that same file is in the temporary variable like Test1 BUT it's ALSO in the aCrap array. Correct????
[excessive quoting removed by server]
On Fri, 21 Apr 2017 10:44:34 -0400, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
Hi Mike,
I have never used fsize this way, I always use fseek to calculate files size.
m.handle = fopen(m.Filename, 0) m.size = fseek(m.handle, 0, 2) fclose(m.handle)
To answer your question, try this:
create cursor TMP_CURSOR (DUMMY Character (1))
append from (m.Filename) sdf
m.RowCount = reccount() +1
USE in TMP_CURSOR
? m.RowCount
On Fri, 21 Apr 2017 17:02:23 +0200, Gianni Turri giannit62@gmail.com wrote:
On Fri, 21 Apr 2017 10:44:34 -0400, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
Hi Mike,
I have never used fsize this way, I always use fseek to calculate files size.
m.handle = fopen(m.Filename, 0) m.size = fseek(m.handle, 0, 2) fclose(m.handle)
To answer your question, try this:
create cursor TMP_CURSOR (DUMMY Character (1))
append from (m.Filename) sdf
m.RowCount = reccount() +1
USE in TMP_CURSOR
? m.RowCount
May be you must use only reccount() instead of reccount() +1.
On 2017-04-21 11:27, Gianni Turri wrote:
On Fri, 21 Apr 2017 17:02:23 +0200, Gianni Turri giannit62@gmail.com wrote:
On Fri, 21 Apr 2017 10:44:34 -0400, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
Hi Mike,
I have never used fsize this way, I always use fseek to calculate files size.
m.handle = fopen(m.Filename, 0) m.size = fseek(m.handle, 0, 2) fclose(m.handle)
To answer your question, try this:
create cursor TMP_CURSOR (DUMMY Character (1))
append from (m.Filename) sdf
m.RowCount = reccount() +1
USE in TMP_CURSOR
? m.RowCount
May be you must use only reccount() instead of reccount() +1.
Ok now I get your approach. Cool trick. I tried it. In testing, it actually wasn't any faster than the OCCURS approach. But thanks for the response though!
Did you benchmark with ALINES? :-)
--
rk -----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of mbsoftwaresolutions@mbsoftwaresolutions.com Sent: Friday, April 21, 2017 2:34 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok now I get your approach. Cool trick. I tried it. In testing, it actually wasn't any faster than the OCCURS approach. But thanks for the response though!
How about an old functions like MLINE()?
On Fri, Apr 21, 2017 at 10:44 PM, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Sorry, memlines() not mline()
On Fri, Apr 21, 2017 at 10:44 PM, mbsoftwaresolutions@mbsoftwaresolutions.com wrote:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
On Thu, 27 Apr 2017 19:02:39 +0800, Man-wai Chang changmw@gmail.com wrote:
Sorry, memlines() not mline()
The returned count can be wrong due to the 8192 characters max line lenght limit.
Only the method that counts the carriage return characters give the correct result.
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
Regards
El 21/4/2017 4:45 p. m., mbsoftwaresolutions@mbsoftwaresolutions.com escribió:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
Have you thought about the ALINES() function?
Laurie
On 27 April 2017 at 16:12, Fernando D. Bozzo fdbozzo@gmail.com wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
Regards
El 21/4/2017 4:45 p. m., mbsoftwaresolutions@mbsoftwaresolutions.com escribió:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
Yeah, and did you try reading a +1GiB txt file? ;-)
El 28/4/2017 12:22, "Laurie Alvey" trukker41@gmail.com escribió:
Have you thought about the ALINES() function?
Laurie
On 27 April 2017 at 16:12, Fernando D. Bozzo fdbozzo@gmail.com wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
Regards
El 21/4/2017 4:45 p. m., mbsoftwaresolutions@mbsoftwaresolutions.com escribió:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text
files
so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
"One of the great things about FoxPro," a wise man (or was it a wise guy?) said, a couple of decades ago, "is that there's three ways to do anything in FoxPro. Or no way at all."
This thread is a great illustration of this. Every three days, someone else drifts through the forum and suggests MEMLINES(), MLINE(), ALINES(), LLFFs, FileToStr(), the OS's FileSystemObject, APPEND, APPEND TO MEMO, file size limits (16Mb or 2Gb, depending on which), shelling out to PowerShell (1), using the Linux wc (wordcount) ported to Windows or using the new Linux under Windows, using Office Automation to import the file to Excel (or Word!) and querying the number of lines, or some other solution.
There's some elegant solutions in Perl or Python, too, if you happen to have them installed.
(1) https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/09/use-a-powersh...
On Fri, Apr 28, 2017 at 6:22 AM, Laurie Alvey trukker41@gmail.com wrote:
Have you thought about the ALINES() function?
Laurie
On 27 April 2017 at 16:12, Fernando D. Bozzo fdbozzo@gmail.com wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
Regards
El 21/4/2017 4:45 p. m., mbsoftwaresolutions@mbsoftwaresolutions.com escribió:
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number-of-lines...
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to- count-number-of-lines-in-a-text-file
-- Gianni
[excessive quoting removed by server]
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count -number-of-lines-in-a-text-file
-- Gianni
[excessive quoting removed by server]
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number-of-lines...
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
1. C++ FLL 0.4064 seconds 2. C# .NET 1.2779 seconds : Tip on how to do this is at http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
1. C++ FLL 3.2426 seconds 2. C# .NET 10.0600 seconds 3. FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number- of-lines-in-a-text-file
[excessive quoting removed by server]
Update to below.
Using FSO doesn't crash - I missed the parameters ... 8,.F.) but it is somewhat slower.
59.3775 seconds using the OpenTextFile(<filename>, 8, .F.) approach and 164.4686 seconds using the AtEndOfStream() | SkipLine approach.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Darren Sent: Saturday, 29 April 2017 1:15 AM To: profoxtech@leafe.com Subject: RE: Getting count of rows in a text file -- best approach?
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
1. C++ FLL 0.4064 seconds 2. C# .NET 1.2779 seconds : Tip on how to do this is at http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
1. C++ FLL 3.2426 seconds 2. C# .NET 10.0600 seconds 3. FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number- of-lines-in-a-text-file
[excessive quoting removed by server]
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the occurrences of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" foxdev@ozemail.com.au wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
1. C++ FLL 0.4064 seconds 2. C# .NET 1.2779 seconds : Tip on how to do this is at http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
1. C++ FLL 3.2426 seconds 2. C# .NET 10.0600 seconds 3. FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number-of-lines...
Gianni,
When working with large files like you are you may want to consider using Python. This need pushed me to explore Python and I've never looked back. Working with text files is a great way to dip your toes into Python without feeling you need to learn the full language.
Python has no practical limits on file size other than those imposed by the OS and the time required to process huge files.
Good luck!
Malcolm
Sure
The FLL ... code is below. (And I am in no way fluent with C++ or FLL's so may be much better ways ..)
You 'll need a few other components but if you chase up how to create an FLL then you'll have it sorted.
Once below is built into FLL it can be used directly in VFP.
Viz. SET library to ". fll" | lnLines = FllGetFileLineCount(<FilePath>)
void GetFileLineCount(ParamBlk *parm)
{
#define p0 (parm->p[0].val)
((char *)_HandToPtr(p0.ev_handle))[p0.ev_length] = '\0';
char *srcFileName = (char *)_HandToPtr(p0.ev_handle) ;
FILE *istream = fopen(srcFileName, "rb");
if (istream == 0)
{
_RetInt(-1, 10);
}
int numLines = 0;
const int len = 8192;
char buf[8192];
do
{
if( feof(istream))
{
break;
}
int nBytesRead = fread(buf, 1, len, istream);
if(nBytesRead <= 0)
{
break;
}
for(int i = 0; i < nBytesRead; i++)
{
if(buf[i] == '\n')
numLines++;
}
} while(1);
fclose(istream);
_RetInt(numLines, 10);
}
FoxInfo vfpUtilFLLInfo[] =
{
{"FllGetFileLineCount", (FPFI) GetFileLineCount, 1, "C"}
};
#ifdef __cplusplus
extern "C"
#endif
FoxTable _FoxTable = {
(FoxTable *)0, sizeof(vfpUtilFLLInfo)/sizeof(FoxInfo), vfpUtilFLLInfo
};
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Saturday, 29 April 2017 2:24 AM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the occurrences of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" < mailto:foxdev@ozemail.com.au foxdev@ozemail.com.au> wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
1. C++ FLL 0.4064 seconds
2. C# .NET 1.2779 seconds : Tip on how to do this is at http://www.tek-tips.com/faqs.cfm?fid=3836 http://www.tek-tips.com/faqs.cfm?fid=3836
3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
1. C++ FLL 3.2426 seconds
2. C# .NET 10.0600 seconds
3. FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message-----
From: ProfoxTech [ mailto:profoxtech-bounces@leafe.com mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri
Sent: Friday, 28 April 2017 11:39 PM
To: mailto:profoxtech@leafe.com profoxtech@leafe.com
Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream()
loFile.SkipLine()
enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" < mailto:fdbozzo@gmail.com fdbozzo@gmail.com> wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo < mailto:fdbozzo@gmail.com
fdbozzo@gmail.com>:
Gianni, you skipped something very important, the part that skip the
lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True
txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri < mailto:giannit62@gmail.com
giannit62@gmail.com>:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" <
mailto:fdbozzo@gmail.com fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject")
loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method.
It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source:
http://stackoverflow.com/questions/7416553/function-to-count-number-
http://stackoverflow.com/questions/7416553/function-to-count-number-
of-lines-in-a-text-file
_______________________________________________
Post Messages to: mailto:ProFox@leafe.com ProFox@leafe.com
Subscription Maintenance: http://mail.leafe.com/mailman/listinfo/profox http://mail.leafe.com/mailman/listinfo/profox
OT-free version of this list: http://mail.leafe.com/mailman/listinfo/profoxtech http://mail.leafe.com/mailman/listinfo/profoxtech
Searchable Archive: http://leafe.com/archives/search/profox http://leafe.com/archives/search/profox
This message: http://leafe.com/archives/byMID/profox/p0q6gc9cisfda2hrsaf5vsi0jmqnap1emi@4 ax.com http://leafe.com/archives/byMID/profox/p0q6gc9cisfda2hrsaf5vsi0jmqnap1emi@4a x.com
** All postings, unless explicitly stated otherwise, are the opinions of the author, and do not constitute legal or medical advice. This statement is added to the messages for those lawyers who are too stupid to see the obvious.
Report [OT] Abuse: http://leafe.com/reportAbuse/p0q6gc9cisfda2hrsaf5vsi0jmqnap1emi@4ax.com http://leafe.com/reportAbuse/p0q6gc9cisfda2hrsaf5vsi0jmqnap1emi@4ax.com
--- StripMime Report -- processed MIME parts --- multipart/alternative text/plain (text body -- kept) text/html ---
Gianni
The approach I took for .NET was ... (And this is the first time I have done this so was a bit of a learning exercise) Again .... very likely better ways to achieve this.
Create a .NET DLL and use in VFP.
Primary reference here: http://www.tek-tips.com/faqs.cfm?fid=3836 (bit out of date)
1. Using Visual Studio .NET create a C# Class Library project – named Utilities 2. Add reference to “System.EnterpriseServices” 3. Code the Class e.g. using System; using System.Text; using System.Linq; using System.Runtime.InteropServices; using System.Collections.Generic; using System.Threading.Tasks;
namespace Utilities { public class FileInfo : System.EnterpriseServices.ServicedComponent { public Int32 LineCount(String FilePath) { int count = System.IO.File.ReadLines(FilePath).Count() ; return count; } } }
4. Create key pair (To sign the assembly). In my case I used (from an Admin command window):
CD “c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>” sn -k Utilities.SNK
Returned:
Microsoft (R) .NET Framework Strong Name Utility Version 3.5.30729.1 Copyright (c) Microsoft Corporation. All rights reserved.
Key pair written to Utilities.SNK
5. Copy the SNK file to the DLL build directory.
6. Edit Assembly.cs file.
Change the entry [assembly: ComVisible(false)] to [assembly: ComVisible(true)]
Ignore the [assembly: AssemblyKeyFile("")] step in Craig’s tip. Instead add the SNK file in the “Signing” section per step 7. 7. Under properties of the project select “Signing” • Check “Sign the assembly” • Type in the “Key File name:” in this case Utilities.SNK • I de-selected “Protect my key file with a password” – too many to remember already. 8. Build the Solution in Visual Studio
9. Register the assembly. (from Admin command prompt)
CD “C:\Windows\Microsoft.NET\Framework\v4.0.30319” regasm.exe D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\Utilities.dll /tlb /nologo /codebase
Returned: Types registered successfully Assembly exported to 'D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\Utilities.tlb’, and the type library was registered successfully
C:\Windows\Microsoft.NET\Framework\v4.0.30319>
10. Then to use it in VFP ..
oUtil = CREATEOBJECT("Utilities.FileInfo") nLines = oUtil.LineCount(<FilePath>)
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Saturday, 29 April 2017 2:24 AM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the occurrences of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" foxdev@ozemail.com.au wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
1. C++ FLL 0.4064 seconds 2. C# .NET 1.2779 seconds : Tip on how to do this is at http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
1. C++ FLL 3.2426 seconds 2. C# .NET 10.0600 seconds 3. FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number- of-lines-in-a-text-file
[excessive quoting removed by server]
Hi All,
This is all very ingenious, but I am curious to know why you would want to know the number of lines.
Laurie
On 29 April 2017 at 01:20, Darren foxdev@ozemail.com.au wrote:
Gianni
The approach I took for .NET was ... (And this is the first time I have done this so was a bit of a learning exercise) Again .... very likely better ways to achieve this.
Create a .NET DLL and use in VFP.
Primary reference here: http://www.tek-tips.com/faqs.cfm?fid=3836 (bit out of date)
Using Visual Studio .NET create a C# Class Library project – namedUtilities 2. Add reference to “System.EnterpriseServices” 3. Code the Class e.g. using System; using System.Text; using System.Linq; using System.Runtime.InteropServices; using System.Collections.Generic; using System.Threading.Tasks;
namespace Utilities { public class FileInfo : System.EnterpriseServices.ServicedComponent { public Int32 LineCount(String FilePath) { int count = System.IO.File.ReadLines(FilePath).Count() ; return count; } } }
Create key pair (To sign the assembly). In my case I used (froman Admin command window):
CD “c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>” sn -k Utilities.SNK
Returned:
Microsoft (R) .NET Framework Strong Name Utility Version 3.5.30729.1 Copyright (c) Microsoft Corporation. All rights reserved.
Key pair written to Utilities.SNK
Copy the SNK file to the DLL build directory.Edit Assembly.cs file.Change the entry [assembly: ComVisible(false)] to [assembly: ComVisible(true)]
Ignore the [assembly: AssemblyKeyFile("")] step in Craig’s tip. Instead add the SNK file in the “Signing” section per step 7. 7. Under properties of the project select “Signing” • Check “Sign the assembly” • Type in the “Key File name:” in this case Utilities.SNK • I de-selected “Protect my key file with a password” – too many to remember already. 8. Build the Solution in Visual Studio
Register the assembly. (from Admin command prompt)CD “C:\Windows\Microsoft.NET\Framework\v4.0.30319” regasm.exe D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\Utilities.dll /tlb /nologo /codebase
Returned: Types registered successfully Assembly exported to 'D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\ Utilities.tlb’, and the type library was registered successfully
C:\Windows\Microsoft.NET\Framework\v4.0.30319>
Then to use it in VFP ..oUtil = CREATEOBJECT("Utilities.FileInfo") nLines = oUtil.LineCount(<FilePath>)
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Saturday, 29 April 2017 2:24 AM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the occurrences of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" foxdev@ozemail.com.au wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
- C++ FLL 0.4064 seconds
- C# .NET 1.2779 seconds : Tip on how to do this is at
http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
- C++ FLL 3.2426 seconds
- C# .NET 10.0600 seconds
- FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number- of-lines-in-a-text-file
[excessive quoting removed by server]
Laurie
For me my goto has been C++ and FLL whenever I want something more performant and I have always been a huge fan using the Windows API and all it has to offer as well as other extensions that the fox is so good at consuming.
Recently I have been getting more familiar with .NET (and I do like it to be honest) so I thought here's an opportunity to do something quickly using the resources and .NET and work out how to get that to work with VFP. So for me it wasn’t about counting lines but rather about how to get to use .NET in VFP (and I am aware of things like wwwDotNetBridge but I wanted to get a bit of a more in depth understanding of what is going on, hence the approach I took.)
I think the original post was to ascertain the best way to do it with large files. It grew from there.
Cheers
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Laurie Alvey Sent: Saturday, 29 April 2017 8:51 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Hi All,
This is all very ingenious, but I am curious to know why you would want to know the number of lines.
Laurie
On 29 April 2017 at 01:20, Darren foxdev@ozemail.com.au wrote:
Gianni
The approach I took for .NET was ... (And this is the first time I have done this so was a bit of a learning exercise) Again .... very likely better ways to achieve this.
Create a .NET DLL and use in VFP.
Primary reference here: http://www.tek-tips.com/faqs.cfm?fid=3836 (bit out of date)
Using Visual Studio .NET create a C# Class Library project – namedUtilities 2. Add reference to “System.EnterpriseServices” 3. Code the Class e.g. using System; using System.Text; using System.Linq; using System.Runtime.InteropServices; using System.Collections.Generic; using System.Threading.Tasks;
namespace Utilities { public class FileInfo : System.EnterpriseServices.ServicedComponent { public Int32 LineCount(String FilePath) { int count = System.IO.File.ReadLines(FilePath).Count() ; return count; } } }
Create key pair (To sign the assembly). In my case I used (froman Admin command window):
CD “c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>” sn -k Utilities.SNK
Returned:
Microsoft (R) .NET Framework Strong Name Utility Version 3.5.30729.1 Copyright (c) Microsoft Corporation. All rights reserved.
Key pair written to Utilities.SNK
Copy the SNK file to the DLL build directory.Edit Assembly.cs file.Change the entry [assembly: ComVisible(false)] to [assembly: ComVisible(true)]
Ignore the [assembly: AssemblyKeyFile("")] step in Craig’s tip. Instead add the SNK file in the “Signing” section per step 7. 7. Under properties of the project select “Signing” • Check “Sign the assembly” • Type in the “Key File name:” in this case Utilities.SNK • I de-selected “Protect my key file with a password” – too many to remember already. 8. Build the Solution in Visual Studio
Register the assembly. (from Admin command prompt)CD “C:\Windows\Microsoft.NET\Framework\v4.0.30319” regasm.exe D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\Utilities.dll /tlb /nologo /codebase
Returned: Types registered successfully Assembly exported to 'D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\ Utilities.tlb’, and the type library was registered successfully
C:\Windows\Microsoft.NET\Framework\v4.0.30319>
Then to use it in VFP ..oUtil = CREATEOBJECT("Utilities.FileInfo") nLines = oUtil.LineCount(<FilePath>)
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Saturday, 29 April 2017 2:24 AM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the occurrences of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" foxdev@ozemail.com.au wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same # of lines.
Timing done with high resolution timers so reasonably accurate. ...
- C++ FLL 0.4064 seconds
- C# .NET 1.2779 seconds : Tip on how to do this is at
http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
- C++ FLL 3.2426 seconds
- C# .NET 10.0600 seconds
- FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-numbe r- of-lines-in-a-text-file
[excessive quoting removed by server]
It is real easy in Powershell.
https://blogs.technet.microsoft.com/heyscriptingguy/2011/10/09/use-a-powersh...
On Sat, Apr 29, 2017 at 5:50 AM, Laurie Alvey trukker41@gmail.com wrote:
Hi All,
This is all very ingenious, but I am curious to know why you would want to know the number of lines.
Laurie
On 29 April 2017 at 01:20, Darren foxdev@ozemail.com.au wrote:
Gianni
The approach I took for .NET was ... (And this is the first time I have done this so was a bit of a learning exercise) Again .... very likely better ways to achieve this.
Create a .NET DLL and use in VFP.
Primary reference here: http://www.tek-tips.com/faqs.cfm?fid=3836 (bit out of date)
Using Visual Studio .NET create a C# Class Library project –named
Utilities 2. Add reference to “System.EnterpriseServices” 3. Code the Class e.g. using System; using System.Text; using System.Linq; using System.Runtime.InteropServices; using System.Collections.Generic; using System.Threading.Tasks;
namespace Utilities { public class FileInfo : System.EnterpriseServices.ServicedComponent
{
public Int32 LineCount(String FilePath) { int count = System.IO.File.ReadLines(FilePath).Count() ; return count; } }}
Create key pair (To sign the assembly). In my case I used (froman Admin command window):
CD “c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin>” sn -k Utilities.SNK
Returned:
Microsoft (R) .NET Framework Strong Name Utility Version 3.5.30729.1 Copyright (c) Microsoft Corporation. All rights reserved.
Key pair written to Utilities.SNK
Copy the SNK file to the DLL build directory.Edit Assembly.cs file.Change the entry [assembly: ComVisible(false)] to [assembly: ComVisible(true)]
Ignore the [assembly: AssemblyKeyFile("")] step in Craig’s tip. Instead add the SNK file in the “Signing” section per step 7. 7. Under properties of the project select “Signing” • Check “Sign the assembly” • Type in the “Key File name:” in this case Utilities.SNK • I de-selected “Protect my key file with a password” – too many to remember already. 8. Build the Solution in Visual Studio
Register the assembly. (from Admin command prompt)CD “C:\Windows\Microsoft.NET\Framework\v4.0.30319” regasm.exe D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\Utilities.dll /tlb /nologo /codebase
Returned: Types registered successfully Assembly exported to 'D:\VFP9\apps\c#\Utilities\Utilities\bin\Release\ Utilities.tlb’, and the type library was registered successfully
C:\Windows\Microsoft.NET\Framework\v4.0.30319>
Then to use it in VFP ..oUtil = CREATEOBJECT("Utilities.FileInfo") nLines = oUtil.LineCount(<FilePath>)
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Saturday, 29 April 2017 2:24 AM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Test file of 1.67 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 43 (There is not enough memory to complete this operation).
Test file of 5.31 GB correctly managed by FSO but not by VFP9 SP2 that gives Error 1103 (Invalid seek offset).
FSO method counts chr(10) / LF characters.
alines() by default counts the occurrences of chr(10) plus the
occurrences
of chr(13) minus the occurrences of chr(13) + chr(10)
Please tell us more about the FLL and the code you used!
Gianni
On Sat, 29 Apr 2017 01:15:06 +1000, "Darren" foxdev@ozemail.com.au wrote:
Many ways to do this. I've compared 3.
With a text file 350Mb | 5.3Million lines . Each method reported same #
of
lines.
Timing done with high resolution timers so reasonably accurate. ...
- C++ FLL 0.4064 seconds
- C# .NET 1.2779 seconds : Tip on how to do this is at
http://www.tek-tips.com/faqs.cfm?fid=3836 3. FSO 7.3874 seconds : (using OpenTextFile etc.)
With a text file 2.6GB file. | 42 Million lines. FSO died - reported no lines and finished in 0.0002 seconds - other two methods reported accurately.
- C++ FLL 3.2426 seconds
- C# .NET 10.0600 seconds
- FSO 0.0002 seconds : FAILED
So I'd guess if you are doing many of these and time is an issue then perhaps a FLL approach might be beneficial. Certainly if file is large enough (probably hits the 2Gb limit in VFP) then FSO is not an option.
-----Original Message----- From: ProfoxTech [mailto:profoxtech-bounces@leafe.com] On Behalf Of Gianni Turri Sent: Friday, 28 April 2017 11:39 PM To: profoxtech@leafe.com Subject: Re: Getting count of rows in a text file -- best approach?
Ok.
Anyway this is the slower method:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 1)
do while ! loFile.AtEndOfStream() loFile.SkipLine() enddo
? loFile.Line -1
Gianni
On Fri, 28 Apr 2017 15:29:48 +0200, "Fernando D. Bozzo" <
fdbozzo@gmail.com>
wrote:
Forget my comment, I've tested it and works beautifully :)
2017-04-28 15:24 GMT+02:00 Fernando D. Bozzo fdbozzo@gmail.com:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
On Thu, 27 Apr 2017 17:12:49 +0200, "Fernando D. Bozzo" < fdbozzo@gmail.com> wrote:
Hi Mike:
A very fast method is using the FileSystemObject:
loFSO = CREATEOBJECT("Scripting.FileSystemObject") loFile1 = loFSO.OpenTextFile(lcArchivo1, 1)
Look at the syntax on Microsoft web site for the read method. It does not have the limitation of VFP's fread/fgets
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number- of-lines-in-a-text-file
[excessive quoting removed by server]
On 2017-04-29 06:50, Laurie Alvey wrote:
Hi All,
This is all very ingenious, but I am curious to know why you would want to know the number of lines.
Laurie
As the original poster, I'll answer: we want to compare the # of records sent this year compared to what they sent us last year. I was looking on the raw text file side initially; I believe I've changed my focus though and am looking at the table count after importing into SQL Server instead now. But this thread was good to learn of other options!
No, in the call to OpenTextFile 8 means "append" and .f. means don't create if the file don't exist.
loFile = loFSO.OpenTextFile(m.filename, 8, .f.)
So:
? loFile.Line -1
gives the correct result.
Try it!
Gianni
On Fri, 28 Apr 2017 15:24:17 +0200, "Fernando D. Bozzo" fdbozzo@gmail.com wrote:
Gianni, you skipped something very important, the part that skip the lines so the Line property is updated:
'Skip lines one by one Do While txsInput.AtEndOfStream <> True txsInput.SkipLine ' or strTemp = txsInput.ReadLineLoop
2017-04-28 15:08 GMT+02:00 Gianni Turri giannit62@gmail.com:
loFSO = createobject("Scripting.FileSystemObject")
loFile = loFSO.OpenTextFile(m.filename, 8, .f.) ? loFile.Line -1
This method overcome VFP memory / file size limits.
Source: http://stackoverflow.com/questions/7416553/function-to-count-number-of-lines...
Create a dummy database with one field append from text.txt delimited go bottomnumberoflines = recno()
From: "mbsoftwaresolutions@mbsoftwaresolutions.com" mbsoftwaresolutions@mbsoftwaresolutions.com To: ProFox profox@leafe.com Sent: Friday, April 21, 2017 10:45 AM Subject: Getting count of rows in a text file -- best approach?
VF9SP2
Currently, I'm simply doing this for now:
RowCount = OCCURS(CHR(13),FILETOSTR(m.Filename))
Is there a better (read: FASTER) way? These are tab delimited text files so I can't really use FSEEK or FSIZE(m.Filename) (which requires SET COMPATIBLE ON) because I can't be sure of each record's length.
tia, --Mike
[excessive quoting removed by server]
-
Darren -
Fernando D. Bozzo -
Gianni Turri -
Laurie Alvey -
Malcolm Greene -
Man-wai Chang -
mbsoftwaresolutions＠mbsoftwaresolutions.com -
Michael Madigan -
Richard Kaye -
Stephen Russell -
Ted Roche