Keep things simple to begin with. If you need more complexity, or tougher security, then do that after you've got your basic framework developed.
First step. Look at some simple encryption algorithms like the Tiny Encryption Algorithm. Get comfortable with encrypting and decrypting a file using a single key.
http://en.wikipedia.org/wiki/Tiny_Encryption_Algorithm
Next step. You will need lots of keys, a database to store them and some kind of customer id to associate with them. To begin with, lets assume that the customer id is a 6 digit number starting at 100000, the key is 128 bits in hex format, and the database is a simple text file. At this point, you should have no problems making a program to generate a batch of keys for the next 100 customers.
Third step. Now you have a database, so set up some scripts or an application so that you can encrypt a specific file, producing multiple copies, one for each customer ID, encrypted with their key.
Fourth step, write a short simple app that takes a file, decrypts it with a key, then loads and runs the resulting executable file. After the app terminates, delete the decrypted file.
Now the little app that decrypts and runs has to somehow be produced in numerous versions, one for each customer with their key in place. Best way to do that is to generate a random number the same size as the key, and place it as a constant in the code. Then compile it. At this point a tool like grep can find where it is in the executable. Write another small app to read in the decryptor app replace the tag with a real key, and write out a version of the decryptor that you will distribute with the matching decrypted application.
After this, you might want to look at better decryption algorithms than TEA, but don't use something too strong that would run afoul of export laws.
Note that none of this will prevent determined thieves, but it should ward of casual ones and you will learn a lot. Better security can be had with an application that stores critical parts of itself on an encrypted USB key so that there is never a decrypted copy on the hard drive. And remember, any time you think that you have figured out a way to improve the encryption or code protection, you are very likely to be wrong. Much greater minds than ours, have spent their entire lives studying encryption and the general rule is that when you deviate a bit from the accepted algorithms and usages, you end up reducing, not increasing, the security.