Here are the rules for public key encryption:
- A public key may be used to encrypt a message that only the corresponding private key can decrypt.
- A private key may be used to encrypt a message that only the corresponding public key can decrypt.
So, it works both ways. The public key being public isn't a problem.
The reason it works is a lot more complicated and involves some interesting math.
Now, OTR doesn't exactly use standard public key encryption. Their algorithms are based on public key encryption but have a small twist in them to allow either party to convincingly deny the conversation ever happened.
Also, OTR isn't secure if you don't actually talk to the person. It's susceptible to a man-in-the-middle attack. This isn't because the encryption can be broken or anything like that. No, a man-in-the-middle attack is where someone interposes themselves between you and the party you want to talk to.
If you are A and you are trying to talk to person B and there's a man in the middle, M, then the attack works like this.
You start talking to B, but unbeknownst to you, you start talking to M who is pretending to be B. When you start talking to M, M initiates a conversation with B pretending to be you. You encrypt all of your stuff to M (who you think is B). M decrypts it, and re-encrypts it to B (who thinks M is you).
If M can always stay in the middle, none of your conversations with B will be secure. If you can ever talk to B without M in the middle, OTR will complain that Bs public key has changed (which is basically because you were really using Ms public key and are just now getting Bs real public key). Of course, OTR might also be complaining because M is trying to interpose (him/her)self and the key you think is the right one really is.
The way to defeat a man-in-the-middle attack is for you and B to talk in some way where you can be certain for other reasons that you're talking to B. Then you can share information about your public key with B, and B can share information about h(is/er) public key with you. You can do this in a coffee shop or over the phone. It doesn't matter if anyone overhears the details, just that you know that it's the person you expect giving them to you.
There is also a fancier way to do this involving secrets that only you and B know. OTR has a mode for asking a question design to elicit a secret in such a way that M cannot provide the proper answer for you, only B can. M will be able to overhear the answer B provides, but will not be able to give it to you.