is it possible for me to build a server in java so that i can let my different flash clients communicate with each other?
You can use any HTTP server of your choice.
If you want functions like data pushing, you might want to try Blaze DS.
BlazeDS is the server-based Java remoting and web messaging technology that enables developers to easily connect to back-end distributed data and push data in real-time to Adobe Flex and Adobe AIR™ applications for more responsive rich Internet application (RIA) experiences.
The buzzwords aside, it has good support for data pushing features and is tuned for interacting with Flex applications.
Dont think you understand the SmartfoxServer at all.
Would suggest you read the SmartfoxServer documentation completely first.
Last but not the least, Completely disagree with the statement SmartfoxServer being "a bit awkward"
P.S: Very interesting.
Smartfox Server programmer at Cochin?