Introduction: I want to develop the chat client that user can chat on the browser and I use the protocol call xmpp. Because of HTML5 web socket not yet available I try flash xmlsocket instead.
Problem: I cannot connect to the server via browser. I'm not sure why, but I think that it is the problem of the server configuration.
Question: Which is the best jabber server suitable for this job?
Most probably this is a permission issue. Either that or you've just configured the wrong host/port to connect to. Flash is not allowed to connect to other hosts than the one it was gotten from itself. You have to explicitely allow flash connections on the receiving side of the request (so on the chat server that you are connecting to). Google for crossdomain.xml to get more info.
ejabberd sounds like a suitable option for you. ejabberd is xmpp server written in erlang and is used quite widely. Many of the well known web based im services like www.meebo.com etc are known to be running on ejabberd. It allows you to install other transports which would let you enable talking to users of other protocol like yahoo, msn, icq etc.
You can also have a look at the xiff action script library by ignite realtime. It is an xmpp client library in action script. If you use this library you would just need to implement the ui components.
Hum... flash is ok, but you'd be better of using something like BOSH, which is basically an HTTP layer over XMPP.
Ejabberd would be a good server, as it supports BOSH, I don't know about OpenFire or Tigase (but I'd say they do). Other servers should be looked at carefully because they don't seem to have a "dynamic" community.
You will need a lot of Javascript, and for that, I can recommend StropheJS, which is probably the very best library out there today.
We have created a MUC (Multi-user chat room) client called Aristochat that works in the browser recently at Superfeedr. You can find the code on Github and an example here.