The only impact on performance is visible in administration tools which need to fetch info from domains. If you have a database with, let's say, 500 tables having 20 columns on average, that's 10000 domains. If you use custom domains, you might have 50 or so. So, admin tool will load table and column definitions much faster.
What you really should be concerned is logic. Having domains makes sure you don't mismatch some columns' datatypes which could create problems with foreign keys for example. Also, it makes easier to change some domain datatype globally: for example, some time ago I decided to change datatype of CustomerIDs in Customer table in one of my databases. It is referenced in about 50 foreign keys. With domains, it was as easy as dumping SQL script and changing the definition of domain. If there were no domains, I would have to do search&replace on a huge SQL script - which is of course error prone.