Using the sharepoint master page in another app is not going to work. It has a bunch of sharepoint specific controls on it that your custom app won't recognize. It is much easier to put all your app's logic (i.e. all controls now in an ASPX page) in a USerControl and use those on ASPX pages in SharePoint.
Then to be able to use these controls you need to do either of these options:
- Then deploy either by putting the .ascx files in the CONTROLTEMPLATES folder in SHarePoint's 12 hive and the DLL in the bin folder / Global Assembly cache. If you go for the BIN folder you would probably need to write CAS Policy files though. This is the most secure method.
- use the SmartPart.
I'm guessing you are very new to SharePoint , but want to urge you to consider packaging all this up in a SharePoint Solution (a .wsp file) using a tool like WSPBuilder.
Also I would encourage you to read up on SharePoint Development / Deployment Best Practices check out this Checklist.