I'll pick some points of your very broad question and answer them. Note that there are other points where others may be of more help...
First, a GSM modem is probably not the way you'd want to go as they usually don't allow for concurrency. So unless you just want one user at the time to use your service, you'd probably need another solution.
Also, think about cost issues - at least where I live, providing such a service would be prohibitively expensive using a normal GSM modem and a normal contract - but this is drifting into off-topicness.
The next issue will be to get voice data from the client to the server (which will relay it to the phone system - using whatever practical means). Pure browser based functionality won't be of much help, so you would absolutely need something plugin based.
Flash may work, seeing they provide access to the microphone, but please don't ask me about the details. I've never done anything like this.
Also, privacy would be a concern. While GSM data is encrypted, the path between client and server is not per default. And even if you use SSL, you'd have to convince your users trusting you that you don't record all the conversations going on, but this too is more of a political than a coding issue.
Finally, you'd have to think of bandwidth. Voice uses a lot of it and also it requires low latency. If you use a SIP trunk, you'll need the bandwidth twice per user: Once from and to your client and once from and to the SIP trunk. Calculate with 10-64 KBit/s per user and channel.
A feasible architecture would probably be to use a SIP trunk (they optimize on using VoIP as much as possible and thus can provide much lower rates than a GSM provider generally does. Also, they allow for concurrency), an Asterisk box (http://www.asterisk.org - a free PBX), some custom made flash client and a custom made SIP client on the server.
All in all, this is quite the undertaking :-)