There are some basic heuristics you could use, to try to detect it.
The most basic thing would be to look at the originating host and headers of the previous emails and see if they match previous ones. If the poster's email domain uses SPF you can verify the IP address has permission to send on behalf of that domain. It's easy to spoof an email - it's harder to spoof an email and actually have it originate from a subnet that matches where your previous mail has come from. But that can be subverted, as well.
First off, the email you post to should be random and unrelated to any display name. This keeps random strangers on the net from knowing where to post in the first place.
You could also go far more sophisticated and implement some sorts of machine-learning-ish things (or even more straightforward heuristics) to pick up on style, but that would be much tougher to get right.
Or just be lame, and upon receipt of a post, fire back an email to the poster with a link to verify and/or a special token to send back as a subject (if they're posting from a phone, etc. with no browser).