Working with large text files in Visual FoxPro

How to use the FileSystemObject to read and write files larger than 2 GB.

By Mike Lewis

You probably know about Visual FoxPro's two gigabytes (2 GB) file-size limit. But did you know that the limit doesn't only apply to database tables? If you've ever tried to access a text file that exceeds that size, you'll know it can't be done. That's true whether you are trying to import or export the file, append or copy it, access it with low-level file functions, or read it into a memory variable.

In this article, I'll show you a workaround. It makes use of the FileSystemObject, which is part of the Windows Script Host (WSH). It doesn't require any additional components to be installed, and can be used with any version of VFP. I have successfully used it to read and write files of ten gigabytes and more.

Writing a file

To illustrate, here is some skeleton code that will write a text file of arbitrary length:

loFS = CREATEOBJECT("Scripting.FileSystemObject")
loFile= loFS.CreateTextFile("c:\MyFile.txt", .T.)
DO WHILE < there is more data to be written >
  lcData = < next line of data to be written >
  loFile.WriteLine(lcData)
ENDDO 
loFile.Close()

The first line in the above code instantiates the FileSystemObject. The second line calls that object's CreateTextFile method to create a new output file, returning a File object. As you can see, the method takes the fully-qualified name of the output file as its first parameter. The second parameter (.T. in this case) says that, if the file already exists, the method should overwrite it (without warning). If you set this parameter to .F. and the file already exists, the method will fail with an error.

The next bit of code simply loops through the available data, writing one line at a time to the output file. It does that by calling the File object's WriteLine method, passing whatever data you want to be written. As the method's name suggests, the data is assumed to be line-oriented, which means that the method will append a carriage-return and line-feed to each line that it receives.

Finally, we call the File object's close method to close the output file.

Reading a file

Now lets look as some skeleton code that reads a file and processes the contents in some way:

loFS = CREATEOBJECT("Scripting.FileSystemObject")
loFile= loFS.OpenTextFile("c:\MyFile.txt", 1)
DO WHILE NOT loFile.AtEndOfStream
  lcLine = loFile.ReadLine()
  * code here to process the data in lcLine
ENDDO 	
loFile.Close()

We again start by instantiating the FileSystemObject. We then call its OpenTextFile method to open the input file. The second parameter indicates the I/O mode: 1 means open the file for reading, 2 for writing, and 8 for appending (that is, writing to the end of the file). As before, the method returns a File object.

The next bit of code runs in a loop until we reach the end of the file - as indicated by the File object's AtEndOfStream property. Within the loop, the ReadLine() method returns the next line from the file. Finally, we close the file, as before.

This technique probably won't help you import a text file larger than 2 GB into a VFP table. That's because the table itself would likely be larger than 2 GB. But you can use it to split the file into smaller chunks, and to process each chunk on its own in some way - perhaps to import it into several tables, or to send its data to a back-end database.

Going further

The FileSystemObject can do a lot more than read and write text files. It can be used to create, move, copy and delete files and folders, to get information about folders and drives, to read and write binary files, and quite a lot more. Although most of its functionality is available natively in VFP, it does have some capabilities that FoxPro doesn't provide - not least of which is its ability to work with extremely large files (the size is limited only by the file storage system and the available disk space).

For further details of the FileSystemObject, see the official MSDN documentation.

May 2013

Please note: The information given on this site has been carefully checked and is believed to be correct, but no legal liability can be accepted for its use. Do not use code, components or techniques unless you are satisfied that they will work correctly with your sites or applications.

If you found this article useful, please tell your friends: