My experience is in C++, but the CryptoAPI library called underneath by both languages is the same so my advice should still work. It's just syntactical sugar that's different between the two.
First, you don't have to use the machine certificate store or the registry. You don't need a store at all. You just need to acquire a crypto context that includes the Microsoft Enhanced Provider, and uses the RSA Full provider.
You need a key pair. You can create a PRIVATEKEYBLOB structure yourself, if you want, but when I was playing with the library, I found it pretty easy to use the CryptoAPI to generate a key with CryptGenKey (make sure it's exportable) which I then exported using CryptExportKey using the PRIVATEKEYBLOB format, which will include the private and public keys. Now you have a formatted blob of memory that contains an RSA key pair. You can mess with this all you want.
Anyway, once you have stuck your key pair into a formatted PRIVATEKEYBLOB, you use CryptImportKey to import it back into the provider. After this, the provider is ready to encrypt and decrypt.
Now, call CryptEncrypt() and pass it the data you want encrypted. It'll use the public key to encrypt it. Pass the results back in to CryptDecrypt() and it'll decrypt it with the private key.