If your application has a connection back to the server it's trivial, just generate random tokens, store them in a database, and require the application to check back with the server before running. However some customers may find this requirement unacceptable. (I personally will never buy any software with an internet activation requirement. I want to buy software, not rent it.)
For keys with genuineness checkability without having to have a connection back to the server:
On the server, generate a random unique token.
Use a public key cryptography scheme (eg. RSA) to sign the token with a private key that is kept secret to the server.
Encode the unique token and the signature together using some binary-to-text scheme (eg. base64, or just using 2-9A-KMNP-Z
symbols for safer typability). The encoded combination is your key or serial.
Embed the public key matching the server's private key in each copy of the application, and have the application prompt for a key at install time. It can split the unique token and the signature, and use the public key to verify that the signature is valid for that token.
This approach requires you bundle some crypto libraries with your software, and most algorithms use quite long signatures, for security, which will make your serial numbers pretty tedious to type in. It's OK if you expect users to copy-and-paste the numbers.
For these reasons many software packages use a less-secure key validation scheme where the checking algorithm is wholly built into the client. This might be a hash, eg. the last four hex digits of the serial must match the lowest two bytes of the SHA1 hash of the rest of the serial combined with a ‘secret’ key. However, since the key has to be bundled in the application, it's possible for a hacker to look at the app code and extract the ‘secret’ key, allowing them to write their own key generators.
That's why for some programs you will see ‘keygens’ available: the app used a less-secure checking algo which left enough information in the application to allow the cracker to reproduce the key-making process. For programs with a more secure setup based on public-key crypto or internet activation, you typically see a ‘cracked’ version of the application instead, where the checking code has been altered/removed.
...which kind of demonstrates that whatever you do, you're still not going to be able to enforce honesty. So don't worry too much about it.