I have a series of large, flat text files that I need to parse in order insert into a SQL database. Each record spans multiple lines and consists of about a hundred fixed-length fields. I am trying to figure out how to efficiently parse them without loading the entire file into memory.
Each record starts with a numeric "1" as the first character on a new line (though not every line that starts with "1" is a new record) and terminates many lines later with a series of 20 spaces. While each field is fixed-width, each record is variable-length because it may or may not contain several optional fields. So I've been using
"...20 spaces...\n1" as a record delimiter.
I've been trying to work with something like this to process 1kb at a time:
def read_in_chunks(file_object, chunk_size):
data = file_object.read(chunk_size)
if not data:
file = open('test.txt')
for piece in read_in_chunks(file, chunk_size=1024):
# Do stuff
However, the problem I'm running into is when a single record spans multiple chunks. Am I overlooking an obvious design pattern? This problem would seem to be somewhat common. Thanks!
def recordsFromFile(inputFile): record = '' terminator = ' ' * 20 for line in inputFile: if line.startswith('1') and record.endswith(terminator): yield record record = '' record += line yield record inputFile = open('test.txt') for record in recordsFromFile(inputFile): # Do stuff
file is a built-in function. It's bad style to change its value.