How to compare unicode strings with entity ref to non-unicode string

Question

I am evaluating hundreds of thousands of html files. I am looking for particular parts of the files. There can be small variations in the way the files were created

For example, in one file I can have a section heading (after I converted it to upper and split then joined the text to get rid of possibly inconsistent white space:

u'KEY1A\x97RISKFACTORS'

In another file I could have:

'KEY1ARISKFACTORS'

I am trying to create a dictionary of possible responses and I want to compare these two and conclude that they are equal. But every substitution I try to run the first string to remove the '\97 does not seem to work

There are a fair number of variations of keys with various representations of entities so I would really like to create a dictionary more or less automatically so I have something like:

key_dict={'u'KEY1A\x97RISKFACTORS':''KEY1ARISKFACTORS',''KEY1ARISKFACTORS':'KEY1ARISKFACTORS',. . .}

I am assuming that since when I run

S1='A'
S2=u'A'
S1==S2

I get

True

I should be able to compare these once the html entities are handled

What I specifically tried to do is

new_string=u'KEY1A\x97RISKFACTORS'.replace('|','')

I got an error

Sorry, I have been at this since last night. SLott pointed out something and I see I used the wrong label I hope this makes more sense

Answer 1

Why not the obvious .replace(u'\x97','')? Where does the idea of that '|' come from?

>>> s = u'KEY1A\x97DEMOGRAPHICRESPONSES'
>>> s.replace(u'\x97', '')
u'KEY1ADEMOGRAPHICRESPONSES'

Answer 2

You are correct that if S1='A' and S2 = u'A', then S1 == S2. Instead of assuming this though, you can do a simple test:

key_dict= {u'A':'Value1',
        'A':'Value2'}

print key_dict
print u'A' == 'A'

This outputs:

{u'A': 'Value2'}
True

That resolved, let's look at:

new_string=u'KEY1A\x97DEMOGRAPHICRESPONSES'.replace('|','')

There's a problem here, \x97 is the value you're trying to replace in the target string. However, your search string is '|', which is hex value 0x7C (ascii and unicode) and clearly not the value you need to replace. Even if the target and search string were both ascii or unicode, you'd still not find the '\x97'. Second problem is that you are trying to search for a non-unicode string in a unicode string. The easiest solution, and one that makes the most sense is to simply search for u'\x97':

print u'KEY1A\x97DEMOGRAPHICRESPONSES'
print u'KEY1A\x97DEMOGRAPHICRESPONSES'.replace(u'\x97', u'')

Outputs:

KEY1A\x97DEMOGRAPHICRESPONSES
KEY1ADEMOGRAPHICRESPONSES