I remember years ago my friend and i were playing command and conquer red alert and there was a mode were we put the others phone number and the game would dial up and connect. What was this called? and where can i find resource to program for this?
Dial-up Networking perhaps. You will have to learn how to control the modem. I remember there were some commands that looked like this: ATH0++ which was how you could make the modem do different things. Perhaps that will give you something to search for.
This resource looks kind of helpful: http://www.activexperts.com/activcomport/tutorials/modem/
One issue you might find is that there are two types of modems generally. One is an actual modem which is connected to your serial port. The other is what is typically known as a "winmodem" which is usually in a PCI slot and didn't have all of the functionality on the hardware but instead used the hardware drivers which typically only worked in Windows. MODEM stands for "MOdulator DEModulator" which means it just converts a digital signal to analog and vice versa.
In essence, it seems that if you can figure out how to program to the serial/com ports on your computer, you should be able to access the modem.
Another interesting link: http://en.wikibooks.org/wiki/Serial_Programming:Modems_and_AT_Commands
Have a look at TAPI (Telephony API). In Windows world there is a set of APIs in the OS for this (http://msdn.microsoft.com/en-us/library/ms737219(VS.85).aspx). The AT command set (Hayes commands) can also be used without TAPI in Windows if you treat your Modem as a COM port and send AT commands to that COM port (that's what actually TAPI does) but it isolates you from their different variants and also running initialization and other commands in a particular order.