This is something I've always wanted to learn. When I design a system (software or hardware) where multiple components communicate with each other, how can I implement some simple encryption or other features in the protocol for some basic security?
I can change the protocol anyhow, since I have low-level access down to the series of bytes that are transmitted / received.
Edit: I'm interested in embedded programming, simple over-the-wire encryption for protocol security.
OAuth may be interesting for you, in order to permit access control between various APIs.
If you purely want an encrypted channel of data, it's better to just layer that ontop of your protocol; go over SSL or SSH.
Don't implement basic encryption in the sense of designing it from the ground up. It is VERY hard to correctly implement encryption properly.
Instead, concentrate on using encryption available on your platform... what is your platform?
EDIT:
See this SO post for a list of encryption alternatives, some of which should work quite well in an embedded environment (e.g. CryptoPP and TomCrypt).
The simplest solution by far is to just use a SSL socket instead of a normal one, a la HTTPS/FTPS style.
There is no difference at all in the protocol between HTTP and HTTPS, the only difference is that they operate on different ports (80 vs 443 normally), and one uses a normal socket, and the other uses a SSL encrypted socket.
OAuth is definitely a consideration. In addition to that, you may want to consider WS-Security. By transmitting via a Web Service, you keep everything platform and language agnostic, and adding WS-Security means that the encryption and decryption are on the application layer (end-to-end). So as long as you trust that the messages are safe once they leave the sender, you know that they will stay safe until the client actually receives the message (this means that it won't be decrypted by the server and THEN passed to the client/receiver, but AFTER the receiving application has the message).
So basically you would OAuth to secure/authenticate the various applications, and WS-Security to secure the messages passing between them.
You say "basic security" but that doesn't mean much. What is your threat model? What kind of attack are you defending against? If you are transmitting credit card data, then you're going to need to use robust encryption, such as RSA. But I can't think of an example that needs "less" protection. If you're worried about hackers disassembling your code, it's already game over -- if it's interesting or valuable, they'll hack it. If not, you wasted time implementing it.
If you are doing embedded programming then where is the encryption concern?
As long as you are staying on the same processor, for example, encryption is useless.
If you are concerned between processors on the device then encryption can be difficult, as a dsp won't necessarily have the spare space for any encryption of any complexity.
If you want fast then a symmetric algorithm is your best bet, and there are many that you can do that are good, such as Blowfish or IDEA. You can store a unique symmetric key, so if they can take the device apart, only one key will be discovered, but each device should have it's own key.
Just tie each key to the serial number, so that if you are communicating with a server then pass the serial number with the packet and the webserver can look up the correct symmetric key and decrypt it quickly.
If you wanted to do a fast hardware encryption, for my MSEE thesis I developed an encryption family that uses arbitrary order calculus, which would be difficult to determine the key as it can be in the hardware circuit using microstrip, as there is no processing, it would be tied in directly before the antenna and everything going over it would be encrypted.
There are many things to be considered in designing a secure system. Just to mention a few:
I suggest you read Bruce Schneier's material for starters.