I see Red5 which looks nice but it really lacks documentation and since I'm new to using sockets it would be a hassle.
I'm wanting to make a flash chat with audio/video (though just text will work at first) and I enjoy ES but the cheapeast license is $700! I'm a solo developer so the cost is too high.
The main factor is just having good enough documentation for a new comer.
(I use the flash CS4 IDE if that matters)
Try Union. IT's written by one of the best actionscript programmers out there, Colin Moock. The options seem affordable and it's well documented as well.
SmartFoxServer is a good choice. I played around with it a few years ago with AS2, so might be work a look.
They offer a free lite version, but it does not support AS3. I think you need to get the basic plan for that. The pricing plans do look well priced though.
They have some nice documentation and plenty of example.
SmartFoxServer seems to be targeted at multi-player games, but it could be used for your needs.
EDIT*
Just found these Beta AS3 classes for SmartFoxServer. I haven't tried it out for my self though.
EDIT 2*
Another option would be to make your own. It's probably easier than you think if you have a few programming languages under your belt. Of course, this is the kinda thing that can suck your time away.
There are a bunch of tutorials out there for building your own socket server, and some are even specifically for flash. Here are some of the best:
I would definitely suggest using C# .net if you have Windows hosting. It would definitely be the simplest to build and would have the best performance. I would stay away from PHP for anything bigger than a chat app, but if that's all you have then go for it.
You can get a free flash socket policy server from https://fsps.rpath.org. I built this appliance for internal use at rPath where I work, but I also maintain it on our free rBuilder Online service. Just click on the Home tab, and download the type of appliance you want (ISO, vmware image, etc.). The Appliance is a full system image including the OS (in this case rPath Linux 2) plus the fsps application.
The policy service resides in the fsps package and runs on the standard Adobe port of 843. By default, it only opens up access to port 443 on the system (https). The policy file served up lives in /usr/share/fsps if you want to change it.
just a quick note: union is free for 1000 users. see: http://www.unionplatform.com/?p=1115