If you need the validation in the UI I would go with a custom field type.
If you need the the check upon creation in the library I would go with an eventreceiver and ItemUpdating/ItemAdding events.
Or maybe a combination...
If this is what you want, let me know and I´ll provide some code examples for you (if you need that).
UPDATE:
If you need an example of a custom field type see this post. You could in fact combine that with some JQuery if you for example add a special css class to the rendered control and then add you JQuery to validate it client side.
I don´t know if you need an example of an event reciever or how to add one, but here´s one link for you anyways:
Add eventreceiver extension method example