What do I need to do to ensure that the files are secured during submission?
During submission? If your fear is that a man-in-the-middle attack between you and the user is going to leak the contents of the documents, the only solution is to use HTTPS (SSL). You will need to purchase a server cert from a CA.
What type of security I need to put on my server to ensure that the files are going to be secure/hacker proof?
Protect the server from being hacked in general by not opening (or firewalling) unnecessary services on the public internet, and ensuring any admin services (eg. SSH) are suitably protected by strong account passwords or keys. Don't run FTP!!
Avoid common security errors in the web application itself that might let users through security measures. SQL injection, cross-site-scripting, cross-site request forgery: this is a big topic! OWASP is a good place to start.
Don't put the files in the web root to be served by the web server. Instead serve them through your own script which controls access in a suitable way for your application. You also need to serve them with ‘Content-Disposition: attachment’ and/or serve them through a different hostname to your site, in order to prevent content-sniffing attacks posing a cross-site-scripting risk (in particular for IE users).
Then they need to enter the temp password, and another two boxes are going to ask for the new password(new password, confirm password).
This sort of thing is a common approach. But typically it is done through a separate token sent in mail (usually as a parameter in a clickable link) that allows the password to be changed, rather than a temporary password as such. Certainly the user's main password shouldn't immediately be changed in response to a Forgotten request, otherwise you'd be able to harrass another user by constantly changing their password.