Methods for Automatic User Uploads

Hello all,

I realize that this question is slightly server related and could be posted on ServerFault, but I'm more asking about a programming-related way to handle this situation, so please forgive me if it seems a bit server-specific.

We currently have a web-based app that is running on Apache/PHP on a Linux server. We use PostgreSQL as our DB backend. Every night, our users have automatic uploads that drop their files onto our server. We then import their specific data (about users and schedules) into our PostgreSQL database. It works fairly well, but it's not very scalable or secure as we have to create a user account on our server for each client that needs to drop off data.

I am currently rewriting the software from the ground up and will handle this much better in the future. I'm hoping to get some ideas from your answers about the best method to do this, however, until this new software is written, we need something to get us by.

I am currently setting up a new server to host our application (due to some recent failures) so I get to design this however I want to make it easier. I can make some code changes, or add some new methods for our users delivery of their files, but it must be secure and easy to automate on both Linux and Windows machines.

What are your suggestions for allowing clients to upload potentially large files securely, easily and automatically, but without us having to create a separate user account on our server? I am obviously open to a PHP-based solution (or another language if it's more appropriate) and would definitely like to hear your thoughts on it regardless as I will be rewriting the software to function more like an API in regards to data uploads.

I don't necessarily need a web server involved, but we have a web-based application, so if the users simply sent a POST request to our server and uploaded their data, it would be easier for them to automatically upload without having to use sftp or scp.

Topher Fangio 2009-11-16 20:04:03

More specifically, it would be much more automated. We wouldn't need to setup different user accounts for each client. They would already be in the database and have an API key.

Topher Fangio 2009-11-16 20:07:16

So, technically speaking, they already have user accounts... Why not just have your application manage those accounts? The reason I persist is that sftp is geared towards sending large data streams, restarting failed requests, etc. HTTP isn't. Sure, you can do it with HTTP, but that's not ideal.

Chris Lively 2009-11-16 20:10:23

Incidentally, what do you consider as "large"? Maybe I'm thinking bigger than you are.

Chris Lively 2009-11-16 20:11:25

@Chris Lively - Yes, the users currently have accounts, but I would rather them not have any access to the server if possible. The size of the files is anywhere from a few megabyts to possibly a gigabyte depending on our client's size. However, these can be tarred or zipped, so that would decrease the size dramatically.

Topher Fangio 2009-11-16 20:22:55

I lied...I just looked at the files. The maximum is 1.8MB untarred. So, maximum untarred is like 12MB maybe. And by tarred, I mean tarred and gzipped.

Topher Fangio 2009-11-16 21:02:25

In this case, I might consider using a web service based on REST or something similiar. Pass the credentials at the same time as the file data. I still think sftp is best because it's easy and even boneheads can work it manually if necessary.

Chris Lively 2009-11-20 14:26:06

ansaurus

tags:

views:

answers:

Methods for Automatic User Uploads

related questions