Search a file for strings from a second file

Question

I have two files. The first file contains a list of 6 character keys (SA0001, SA1001, etc.). The second file contains a list of dates and amounts where the first six positions will match the key in the first file. I want to verify that every key in the first file has at least one match in the second file. There may be more than one match which is okay and there may be records in the second file with no key in the first file which is also okay. So basically a loop within a loop. The problem arises when I want to break out of the inner loop after the first match because the second file could be quite large. It prints out the "found" message correctly and breaks, but it won't print the "not found" message if it reaches the end of the second file with out finding a match. My code so far is:

unvalues = open("file1.txt", "r")
newfunds = open("file2.txt", "r").readlines()
i = 1
for line in newfunds:
    line = line.strip()
    for line2 in iter(unvalues.readline, ""):
        try:
            if line == line2[:6]:
                print "%s: Matching %s to %s for date %s" % (i, line, line2[:6], line2[6:14])
                break
        except StopIteration: print "%s: No match for %s" % (i, line)
    i += 1
    unvalues.seek(0)

Answer 1

I don't think break; throws a StopIteration.

You generally don't want to use exceptions for flow control like that.

Answer 2

Go through each file once, adding each record to a hash with value equal to 1. Then make sure that the keys of the first hash are a subset of the keys of the second.

hashes = []
for f in ["file1.txt","file2.txt"]:
    lines = open(f,"r").readlines()
    hash = {}
    for line in lines:
        hash[line[:6] = 1
    hashes.append(hash)

set_keys1 = set(hashes[0].keys())
set_keys2 = set(hashes[1].keys())
assert(set_keys1.issubset(set_keys2))

Answer 3

Use sets instead:

set1=set(line[:6] for line in open('file1.txt'))
set2=set(line[:6] for line in open('file2.txt'))
not_found = set1 - set2
if not_found:
    print "Some keys not found: " + ', '.join(not_found)

Answer 4

I think this might be closer to what you want:

unvalues = dict((line[:6], line[6:14]) for line in open("file1.txt", "r"))
newfunds = [line for line in open("file2.txt", "r")]
for i, line in enumerate(newfunds):
    key = line.strip()
    if key in unvalues:
        v = unvalues[key]
        print "%s: Matching %s to %s for date %s" % (i+1, line, key, v)
    else:
        print "%s: No match for %s" % (i+1, line)

Answer 5

first_file=open("file1.txt","r")
#save all items from first file into a set
first_file_items=set(line.strip() for line in first_file)
second_file=open("file2.txt","r")
for line in second_file:
   if line[:6] in first_file_items:
       #if this is item from the first file, remove it from the set
       first_file_items.remove(line[:6])
       #when nothing is left in the set, we found everything
       if not first_file_items: break

if first_file_items:
   print "Elements in first file but not in second", first_file_items

Answer 6

You cannot (and need not) catch the StopIteration exception that occurs when the iterator is finished, because it gets caught by the for loop automatically. To do what you appear to be trying to do, you could use an else block after your for block, e.g. you could replace your inner loop with this:

for line2 in iter(unvalues.readline, ""):
    if line == line2[:6]:
        print "%s: Matching %s to %s for date %s" % (i, line, line2[:6], line2[6:14])
        break
else:
    print "%s: No match for %s" % (i, line)

The else block is executed when the for loop finishes without the break statement getting hit.

However, you may well find that one of the other approaches using sets is quicker.

Answer 7

from collections import defaultdict

unvalues = open("file1.txt", "r").readlines()
newfunds = open("file2.txt", "r").readlines()

unvals = defaultdict(int)

for val in unvalues:
    unvals[val] = 0

for line in newfunds:
    line = line.strip()

    if line[:6] in unvals.keys():
        unvals[line[:6]] += 1

for k in unvals.keys():
    if unvals[k] == 0:
        print "Match Not Found For %s" % k

might give you a good starting point for what you want to achieve, without being terribly messy. This gives you the performance advantage of only looping through each data set individually.

As a quick addendum, if you want line numbers, rather than building a counting variable outside the loop and incrementing it, try this instead:

for i, line in enumerate(newfunds):

enumerate() basically zips a sequential integer iterator with your list to produce the desired result without unnecessary counting operations.

Answer 8

Another approach using sets

keys = set(line[:6] for line in open('file.txt'))
missing = set(value[:6] for value in open('file2.txt') if value[:6] not in keys)
if missing:
   print "Keys Missing " + ', '.join(missing)