Remote java program execution using ftp, very large dataset on remote machine - program to data vs data to program

Hi all,

I am developing a java based application; its pertinent requirements are listed below

Large datasets exist on several machines on network. my program needs to (remotely) execute a java program to process these data sets and fetch the results
A user on a windows desktop will need to process datasets (several gigs) on machine A. My program can reside on the user's machine. He will execute my program from his machine and initiate the dataset processing on remote machine(s)
Instead of getting the dataset over the network from the remote machine to his machine, he will execute the program on the remote machine and fetch results
The user may have open access to the other machines but ftp is the requirement
Data should not be brought through network to the user's machine.
Users have windows OS

My question(s)

How can I perform this kind of remote process execution ? Any ideas?
I am looking at hadoop; I am working on Windows XP. I was unable to get hadoop working for a single node cluster; I am unable to find good documentation. I therefore haven't quite tested hadoop. Any comments on if I am on the right track?
Any links any of you has found useful for installation of hadoop and trouble shooting?

Thanks in advance for any responses. Do please let me know if I should provide any more/specific details.

-jv

Thanks a lot. Will check into this

2010-01-15 21:42:11

I haven't delved deeply into using RMI yet; but I am thinking that each machine on the network could have the application ("server" (remote) code + "client" code) running on its jvm. So each machine is a client and server and this is how they can talk to eachother. Does that make sense?

2010-01-15 21:58:30

There really isn't a client and server with RMI, and you could certainly have class A call methods on class B, and then have class B return the favor. Or you could just have class B return the dataThis is Remote Method Invocation, so anything you can do with a method, you can do with RMI.

Jim Barrows 2010-01-15 22:41:18

Thanks Jim. Have started working on my RMI based architecture.

2010-01-20 20:48:02

Cool.. could you check mark the answer then :)

Jim Barrows 2010-01-20 22:47:16

Sure Jim, as soon as figure out some more details (early next week).

2010-01-22 16:02:18

Thanks a lot. Will check into this.

2010-01-15 21:41:16

Between a web service and a RMI implementation, which might be faster and light weight. I don't mean to ask questions without doing my research first but I thought an expert's opinion can help guide in right direction.

2010-01-15 22:01:00

RMI will be faster. That's Java byte code to java Byte code. Web services are typically XML to java byte code. That's I put them last in my list of other things.Web services work well when you have, or can reasonably expect to have non-java clients talking to you. In your case, as I understand it.

Jim Barrows 2010-01-15 22:44:11

The speed difference between RMI and web services (if any) will severely drown in the speed of the network. If you don't believe me, measure!

Thorbjørn Ravn Andersen 2010-01-16 00:53:39

Thanks a lot. Working on RMI

2010-01-20 20:46:22

ansaurus

tags:

views:

answers:

Remote java program execution using ftp, very large dataset on remote machine - program to data vs data to program

related questions