At minimum, you need to be comparing the top N bits against known-private or known-unused network blocks. And the minimum of that is the RFC 1918 private networks:
10/8
172.16/12
192.168/16
(This notation means that if you mask off the top 8 bits and get "10", it's in the first private block. 12 bits == 172.16 is the second, etc.)
There are many more blocks you could detect.
The easiest example is 127/8, which are all loopback addresses. You probably know about 127.0.0.1, but in fact all 127.x.y.z addresses refer to the loopback interface on your machine.
A more obscure example is 5/8, which was assigned but never used on the wide Internet, so it's been appropriated by Hamachi. You would want to be careful about testing for this, though, if there's a chance that your program could be run on a machine with access to the Hamachi network.
Another set of addresses you'll probably want to ignore are the various multicast ranges. You should only ever be sending UDP packets to those, never TCP connections.
If you want to get really aggressive, you could keep up to date on the latest assignments published by IANA, ignoring IPs from blocks that haven't been assigned yet. You'd want to keep that list frequently updated, though.