Limiting the searching of a large file.

Question

So here is my program with some new modifications:

datafile = open('C:\\text2.txt', 'r')
completedataset = open('C:\\bigfile.txt', 'r')
smallerdataset = open('C:\\smallerdataset.txt', 'w')
matchedLines = []
for line in datafile:
    splitline = line.split()
    for item in splitline:
        if not item.endswith("NOVA"):
            if item.startswith("JJJ") or item.startswith("KOS"):
                matchedLines.append( item )
counter = 1
for line in completedataset:
    print counter
    counter +=1
    for t in matchedLines:
        if t in line:
            smallerdataset.write(line)
datafile.close()
completedataset.close()
smallerdataset.close()

The problem that I have now is that I want to search through the "bigfile" but at a faster rate. I would like to limit the searching of each line in bigfile to the string that occurs before the first ','

I want to use something like index = aString.find(',') I beleive but I'm not having much luck limiting the search of the big file to the string that occurs before the first comma.

Answer 1

For your splitting issue you can just limit the number of times it splits the line:

first_item = str.split(",",maxsplit=1)[0]

Answer 2

You could change

if t in line:

to

if t in line[:line.find(',')]:

This may make the program faster if line is very very long and the comma appears near the beginning. Or it may make the program slower if , appears near the end of the line.

PS. Is every line guaranteed to have a comma in it? The above code acts a bit funky if there is no comma. For example,

In [21]: line='a line of text'

In [22]: line[:line.find(',')]
Out[22]: 'a line of tex'

If you want to ignore lines without a comma, this might be better:

In [23]: line[:line.find(',')+1]
Out[23]: ''

Answer 3

this may help speed it up...sorry about the lack of tabbing

datafile = open('C:\\text2.txt', 'r')
completedataset = open('C:\\bigfile.txt', 'r')
smallerdataset = open('C:\\smallerdataset.txt', 'w')
matchedLines = []
counter = 1
for line in datafile.readlines():
if line[-4:] == "NOVA":
if (line[:3] == "JJJ") or (line[:3] =="KOS"):
counter += 1
for compline in completedataset:
if line in compline:
smallerdataset.write(line)
datafile.close()
completedataset.close()
smallerdataset.close()