Problem with for loop iterations over data from a queue

Question

I'm having a problem reading and processing data from my server. I've spent hours debugging this and it seems as though the problem has nothing to do with the server but is really just an issue with one of my for loops. Its difficult to explain this without actually seeing the data from the server, but I'll do my best to explain this.

Basically I have a server that when I send a command, it returns to me 5 different XML strings sequentially. My program parses this data into Python dictionaries and stores them in different class attributes. As TCP is a stream, I usually don't receive each string separately. What usually happens is I'll receive the first string, then the next 4 strings in groups of 2. So theres normally 3 different iterations of the for loop. However, the last string fails to get processed and stored in an attribute. Here's the logic containing the for loop in question:

 def xmlParser(self, newData):
     print newData
     for string in newData:
         if 'a' in string:
             print 'string1 being processed'
             xmlToDictionary(string, 'string1')
         elif 'b' in string:
             print 'string2 being processed'
             xmlToDictionary(string, 'string2')
         elif 'c' in string:
             print 'string3 being processed'
             xmlToDictionary(string, 'string3')
         elif 'd' in string:
             print 'string4 being processed'
             xmlToDictionary(string, 'string4')
         elif 'e' in string:
             print 'string5 being processed'
             xmlToDictionary(string, 'string5')

The print statements are solely for debugging purposes. My first reaction to the last string not being processed was that it simply wasn't being received. However, by printing newData I can verify that all the strings are received and sent to this function. The newData is a list containing the strings to be processed. For the 3 different iterations, newData looks like this

newData = ['string1']
newData = ['string2', 'string3']
newData = ['string4', 'string5']

Its 'string5' that isn't processed, and I don't believe its because of an exception being raised in xmlToDictionary() because the string 'string5 being processed' isn't printed. And if I include an else statement in the function, that isn't executed either. Its as if the for loop just refuses to perform a second iteration on the last group of data. To test things further, I modified the server script to send 'string5' before 'string4' and, to my surprise, completely fixed the issue and caused 'string5' to be processed and stored as a dictionary. In this situation, newData looked like this

newData = ['string1']
newData = ['string2', 'string3', 'string5']
newData = ['string4']

I also tried modifying the server script to stop sending 'string4' altogether. This also was successful in allowing 'string5' to be processed. The for loop only performed two iterations and newData looked like this

newData = ['string1']
newData = ['string2', 'string3', 'string5']

I tried to put all this together into determining what the problem is, but I'm totally confused. I've never encountered something like this before. If I can't come up with a solution, I'll either leave the script to send 'string5' before 'string4' or I may simply combine all the strings on the server and have it sent as one big string. Regardless, I would still like to determine what is going on here since it may indicate a deeper problem with my program.

Answer 1

"As TCP is a stream, I usually don't receive each string separately "

You can't reliably and predictably receive them separately. TCP makes things into a single stream of data. That's what TCP/IP must do. It must buffer to create a "stream" of data, not matter what you try to do at each end to send or receive things "separately". Separately has no meaning when you use TCP/IP.

You must break up the strings at your end using some feature of the string. Syntax. Punctuation. Something.

Usually, what we do is combine the 5 XML strings into a single XML message.

<message>
    <string1>...</string1>
    <string2>...</string2>
    <string3>...</string3>
    <string4>...</string4>
    <string5>...</string5>
</message>        

Parse the one message, find your five strings. It's much simpler. And it fits with the way TCP/IP actually works.

Answer 2

Hi,

I just tried your code with the following result:

def xmlParser(newData):
     for string in newData:
         print string
         if 'a' in string:
             print 'string1 being processed'
         elif 'b' in string:
             print 'string2 being processed'
         # ...
         elif 'e' in string:
             print 'string5 being processed'

newData = ['string1']
xmlParser(newData)
newData = ['string2', 'string3']
xmlParser(newData)
newData = ['string4', 'string5']
xmlParser(newData)

And here is the output:

$ python test.py
string1
string2
string3
string4
string5

The loop will process exactly all the element if there is no exception throw inside xmlToDictionary. If you want to verify if throw an exception, try this code:

def xmlParser(self, newData):
     print newData
     for string in newData:
         try:
             if 'a' in string:
                 print 'string1 being processed'
                 xmlToDictionary(string, 'string1')
             # ...
             elif 'e' in string:
                 print 'string5 being processed'
                 xmlToDictionary(string, 'string5')
          except:
             print "excetion error"

Maybe the test condition is not exactly what you want:

if 'a' in string:

Hope I helped you

Answer 3

Well, this is a stumper. Especially since you have not posted the real code, but your own concept of how the code should be working. Since there is nothing wrong with this concept, there is nothing wrong with your example.

Still, I copied out your code, and then tried to perturb it to get your results. I think the most likely culprit is that you have some undesirable side effect occurring in your code that processes string4's. Your description sounds very much like what happens when a list is modified while it is being iterated over. Here's one example that would cause the exact behavior you describe:

elif 'd' in string: 
    print 'string4 being processed' 
    # undesirable side effect occurs here
    newData.remove(string)
    xmlToDictionary(string, 'string4') 

Try sending a sequence like [string4, string1] and see if string1 gets through. If not, there is a strong chance that some kind of side-effect is coming from this chunk of code.