Using Cloud/Distrubted computing to share processor time - possibilities and methods

Question

My question is one I have pondered around when working on a demanding network application that would explicitly share a task across the network using a server to assign the job to each computer individually and "share the load".

I wondered: could this be done in a more implicit manner?

Question

Is there a possibility of distributing processor intensive tasks around a voluntary and public network of computers to make the job run more efficiently without requiring the job's program or process to be installed on each computer?

Scenario

Lets say we have a ridiculously intensive mathematics scenario where I am trying to get my computer to calculate every prime factorization break down for all numbers from 1 to 10,000,000 and store them in a database (assuming I have the space and that the algorithms are already implemented in their own class, program, dynamic link library or any runnable process.)

Now it would be more efficient to share this burdening process across a network or on a multi-core super computer, however these are both expensive. To my knowledge you would require a specifically designed program to run the specific algorithm and have the program installed across the said cloud/distributed computing network whilst you have a server keep track of what each computer is doing (ie. what number they are currently calculating the primes for).

Conclusion

Overall:

• Would it be possible to create a cloud program / OS / suite where you could share processor time for an unspecified type of process?

• If so how would you implement it, where would you start?

• Would you make an OS dedicated to being able to run unspecified non-explicit tasks or would it be possible to do with a cloud enabled program installed on volunteers computers volunteers who were willing to share a percentage of their processor clock to help the general community).

• If this was implementable, would you be a voluntary part of the greater cloud?

I would love to hear everyone's thoughts and possible solutions as this would be a wonderful project to start.

Answer 1

Would you trust someone else's code to run on your computer?

Its more practical to not ask their permission: ;)

I once wrote a programming competition solver written with Haxe in a Flash banner on a friend's fairly popular website...

Answer 2

I have been dealing with the same challenge for the last few months. My conclusions thus far:

The main problem with using a public network (internet) for cloud computing is in addressing the computers through NATs and firewalls. Solving this is non-trivial.

Asking all participants to open ports in their firewalls and configure their router for port-forwarding is generally too much to ask for 95% of users and can pose severe security threats.

A solution is to use a registration server where all peers register themselves and can get in contact with others. Connections are kept open by server and communication is routed through server. There are several variations. However, in all scenario's, this requires vast resources to keep everything scalable and is therefore out of reach for anyone but large corporations.

More practical solutions are offered by commercial cloud platforms like Azure (for .Net) or just the .Net ServiceBus. Since everything runs in the cloud, there will be no problems with reaching computers behind NATs and firewalls, and even if you need to do so to reach "on-premise" computers or those of clients, this can be done through the ServiceBus.

Answer 3

I would not expect a program which allow "share processor time for an unspecified type of process". An prerequisite of sharing processor time is that the task can divided into multiple sub tasks.

To allow automatic sharing of processor time for any dividable task, the key of solution would be an AI program clever enough to know how to divide a task into sub task, which does not seems realistic.