A bit more info would be welcome, but I'll give it a try.
I assume you are running with FormsBasedAuthentication? If so, you would need to write the logic that creates the user in the database. The existing ASP.NET Membership Provider should meet all your needs. After that, you would need to add the newly created user to the appropriate sharepoint groups (this can also be done using the sharepoint object model).
The first step though would be to create your "become a member" page and make that accessible anonymously. Add a webpart / control to that page that handles the payment and then when succesful creates the ASP.NET Membership Provider user. Then adds that user to the appropriate sp groups. The easiest way to restict access would be to create a subsite of the main welcome site (root of the sitecollection), which has it's own rights and groups. make the root site accessible for anyone, allow access to the subsite for paying users only.