If you want in-game purchasing with Flash I think the only reasonable thing to do is use SSL. You are being cheap about it but don't want your customers to be cheap (that is, you want them to pay for your game but you won't pay for their security). Thats is quite unreasonable IMO. You would only secure the transaction part (Flash requests and sends sensitive data to an https
url... the rest goes through normal channels).
You could look around for payment solutions using Flash. I wouldn't recomend a custom-made "encryption" solution since you don't seem to be a security expert.
Take a look at this tutorial on using PayPal with Flash (although it is not AS3 but can be easily converted and I have not tested it and am not sure if this fits your particular application).
By the way, AMFPHP can handle binary send/receive. You also send/receive what you need in a given moment so CPU usage can be kept at a minimum.