The png could have the wrong gamma or color space, and it's getting converted on load, affecting the result. Some versions of Photoshop had a bug where they saved pngs with the wrong gamma.
What are you doing prior to the code posted?
PNG is a compressed format, using the deflate algorithm (See Section 5 of RFC2083), so if you're just doing binary reads, you're not looking at actual pixels.
I checked a couple of values from your screenshot.
The input pixels:
are very similar to the corresponding output pixels.
Looks to me like you are XORing two images that are closely related (inverted in each color channel), so, necessarily, the output is near 0xffffff.
If you were to XOR two dissimilar images, perhaps you will get something more like what you expect.
The question is, why do you want to XOR pixel values?