You can create a server in (almost?) any language that supports sockets and manage your clients with that.
From a flash perspective you could use ruby, but I don't know how fitted ruby would be for the task...I mean, you can make a PHP socket server but it would not handle much stress.
An alternative to SmartFoxServer could be open sourced Red5, written in Java.
If you want to know about flash built in capabilities in handling p2p:
Flash p2p: Everett Church
In Flash Player 10 adobe added Live media (RTMFP) support- Media was always sourced from the publishing peer. In Flash Player 10.1 beta they introduced groups and application-level multicast.
However, you would still need a server (Stratus) to handle introductions and manage active connections. ( Also, I regard it as geared towards flash video, Stratus does not support shared objects or scripting. So with Stratus, your applications will be clients communicating directly with each other.)
Edit:
After re-reading your question and the answer provided by Beanish, I think the first time I got your question wrong. So I just re-wrote my answer...