We've been requested to create an Online payment processing system the like of Paypal for our national use.
does an open source version of this exist ? (so I could study it and maybe improve on it)
are there any books/resources/materials that could be useful ?
How can I go about taking on such a huge task ?
I'd think your teachers want you to learn how to plan, not how to copy, so don't look for a reference implementation, instead learn how to think about a problem.
The trick to solving any large problem is breaking it down into small problems.
So do this.
You will also want to consider testing it, and making sure it delivers all the functionality you need.
Once you start thinking about a problem with a pencil and paper in your hand, it becomes very easy, IMHO :)
Well, first of you need to have solid knowledge the way your country handles money in terms of laws (VAT, refunds, and so on) so it might be help if you tell us that.
Second, this is not a product it's a service so you need a staff to support your users. You also need to make deals to process credit cards, e-checks and so on. And since you're dealing with possibly large amounts of money, you also need to financial and law advice (read this E-Gold statement and the Wikipedia entry). You also need to take security very seriously, both virtual and physical, so you'll need to contract several different teams to independently analyze and audit your system.
This is really a broad question, I'd suggest you read all the Wikipedia entries about PayPal and other processing systems and then explain your exact problem in a little more detail (though you might really want to keep some stuff secret, since this is a public accessible website).
To start with first determine who is going to process your credit card transactions, as they can also most likely do debit cards.
To process these yourself is a pain as there are standards that have to be met, and you end up needing to pay for the privilege. Unless you will be processing an enormous amount of transactions you are better off using another company that already can process them.
To start with, just design the system using something like Payflow, to do the actual processing.
Once you are up and running, and you have dealt with the PCI standards to protect credit card data, then you can look at perhaps trying to phase out your payment gateway and do it yourself, but, that should only be if you determine that you are processing in sufficient volume that it is with the additional development and resource costs to do it yourself.
You will need to have a signed certificate, to ensure people that your site is safe, and ensure that you have strong encryption to protect the credit card data, and make certain that the passphrase or symmetric key is not on the computer, but is kept only in memory that will never be swapped to a hard drive, otherwise someone could steal it, if they can copy the hard drive.
http://www.allbusiness.com/sales/internet-e-commerce-securelectronic-transaction/2310-1.html
+1 each to @James Black and @eyze for their answers. This is not a minor undertaking, and unless you work for a company that is already affiliated with the credit processing network in some way, you're in for a lot of work and a lot of compliance issues. Their answers were good enough that I don't have a lot to add, but I would like to add this.
We looked at working with a company that already is in the business of authorizing credit card transactions, but they work primarily with POS systems and terminals, rather than as an Internet gateway. We wanted to stick with them for processing web site payments, since we use them for our stores. In essence, since they didn't function as an Internet payment gateway, we would need to write our own payment gateway using them in the background. After weeks of research, we came to the conclusion that even though this was technicially within our capability, and even though we have the knowledge of PCI and other applicable standards that this is something best left to companies that do this as their primary business. We'll be going with one of the pre-existing gateways.
Also, to answer your specific questions:
I do not believe anything open source exists. The backbone on which credit card processing is done is so sensitive and such a target for attackers that there is a very real need to limit the knowledge of how to process cards on a need-to-know basis. (I lost count of the number of non-disclosure agrements I was presented with just to research the idea.)
For the same reason, I doubt you will find much in the way of books, etc.
If you're working with a company that already processes cards, then you're a step ahead, but if you're trying to break into the business you are going to face huge hurdles.