Make encoding uniform before comparing strings in PHP

Question

I'm working on a feature which requires me to get the contents of a webpage, then check to see if certain text is present in that page. It's a backlink checking tool.

The problem is this - the function runs perfectly most of the time, but occasionally, it flags a page for not having a link when the link is clearly there. I've tracked it down to the point of visually comparing the strings in the output, and they match just fine, but using the == operator, php tells me they don't match.

Recognizing that this is probably some sort of encoding issue, I decided to see what would happen if I used base64_encode() on them, so I could see if doing so produced different results between the two strings (which appear to be exactly the same).

My suspicions were confirmed - using base64_encode on the strings to be compared yielded a different string from each. Problem found! The problem is, I don't have any idea how to solve it.

Is there some way I can make these strings uniform based on the outputted text (which matches), so that when I compare them in php, they match?

Answer 1

Without application code it's difficult to say what's happening.

Try using trim() on the strings to remove trailing whitespace, which is invisible to the naked eye.

You may find strcmp gives better results as well.

Answer 2

what about running both through a sanatizing filter (if you have php >5.2.0). I don't know that it will do anything, but it may.

http://www.phpro.org/tutorials/Filtering-Data-with-PHP.html#12

Answer 3

Try mb_strstr() and trim(), as pointed by dcaunt.

Answer 4

You could try using the Dom Extension to PHP. On creating a new Dom Document you can specify the encoding of the underlying document / webpage. According to This website, internally everything is done in UTF-8. You could then find the dom nodes you were interested in, and compare the Text Content of the node

If you were not using webpages, with an associated specified character encoding, I would suggest using the multibyte functions, in particular mb_detect_encoding and mb_convert_encoding

Answer 5

I'm not entirely sold on your belief that it is the encoding. PHP is going to internally store all its strings in the same format. Could you try this code? It will compare the ascii value of each character in both strings, which might reveal something you're not seeing by visually comparing the strings.

$str1 = ...;
$str2 = ...;

if(strlen($str1) != strlen($str2)) {
  echo "Lengths are different!";
} else {
  for($i=0; $i < strlen($str1); $i++) {
    if(ord($str1[$i]) != ord($str2[$i]) {
      echo "Character $i is different! str1: " . ord($str1[$i]) . ", str2: " . ord($str2[$i]);
      break;
    }
  }
}

Answer 6

If you can't reliably get the encoding, you can use mb_convert_encoding.

$string1 = mb_convert_encoding($string1, 'utf-8', 'auto');
$string2 = mb_convert_encoding($string2, 'utf-8', 'auto');

If you can determine the encoding (from the http headers or meta tags) you should specify the encoding instead of using "auto."

$string1 = mb_convert_encoding($string1, 'utf-8', $encoding1);
$string2 = mb_convert_encoding($string2, 'utf-8', $encoding2);