Is there any good c++ source codes or api for handling phone lines like understanding tone signals. For example i like to find out if the person enters 3 (it's likely that this is done using it's tone sound).
Do i need a special modem for this purpose or it can be done using only standard modems.
DTMF is the term you are looking for: http://en.wikipedia.org/wiki/Dual-tone%5Fmulti-frequency
Whether you can process incoming DTMF tones with a particular modem depends on whether the modem supports it. If it does there will be an AT command to manage it, both for issuing outgoing DTMF tones and being notified of incoming DTMF tones.
There are some examples here: http://www.tek-tips.com/viewthread.cfm?qid=24275&page=1
The specific AT commands might depend on your particular modem.
If your modem can't handle DTMF itself and you're interested in being able to inject DTMF tones into the outgoing audio stream / recognise and extract DTMF tones from the incoming audio then you'll need some DSP support; that's not going to be something you can just do in C++.
There seem to be lots of TAPI DTMF decoders and sample code on the net, which will lift you one level from the hardware.