Well, traditionally serial numbers are serial ... numbers. So the first example off the production line has sn 0001 then the next one is 0002 and the next one is 0003. I think that most people can work out that algorithm.
I think you're actually asking about product keys, which use a similar mechanism to public key message signing - the product key is the encrypted value, the program has a public key which allows it to verify that the key is valid, but only the software vendor has the secret key to 'sign' the product key. The wikipedia article on digital signatures has the general mechanism; the only proviso is that for a key to be entered by the user it has to be quite a bit shorter than a PGP one.
If you are restricted to a very short serial number, then it's unlikely to be big enough to store the result of a typical signing mechanism, in which case it's quite common to just use some variant of checksum on it. That has the disadvantage of being easy to reverse engineer - it's security is because the algorithm is 'secret' rather than due to any cryptographic properties. Each product would have its own algorithm, and they usually get cracked quite quickly.
If you have 5 blocks of 5 characters, you have 36^25 combinations, which is bigger than 2^128, so could use one of the standard digital signature algorithms which generates a 128 bit, then convert that value to base 36.