If you use a socket (and practically building your own NIO server, maybe based on Apache MINA) it is possible to obtain a boost on performance but in my opinion you will have to invest a huge amount of work on it. I know some people using this approach but for a very particular application (a Flash game which was supposed to have zounds of users). They spend one month building it.
Another solution (if you plan to use sockets) is to use some products which are offering RTMP channels (LCDS, WebOrb) but they are not free and in my opinion is does not worth it if you plan to use them just for RPC calls over a socket.
For most of the cases the RPC approach based on POST should be good enough. The AMF serialization is quite compact (I wrote here a small benchmark and you can find another resources on web).
Also the Flash Player is sending your RPC request in one batch - for example if you call in your Flex code 3 methods one after another all of them are going to be grouped in only one POST call.