Any good distributed agent/service models for .NET?

Question

I'm looking for tools that implement the distributed agent/service model ... I guess we could also call this grid or cloud computing, but I'm not sure the term is exactly analagous. A distributed agent model would be one where developers build agents that are dispatched to a framework and the framework executes the agent somewhere in the distributed environment.

Specifically, I'm looking for a framework that support dynamic distribution across the grid/cloud, but replaces the transient agent with a more persistent service. The goal here is to minimize the amount of time and effort it requires to deploy a service into the grid/cloud container.

FWIW, I think Digipede gets pretty close to this, but I'm interested in what else is out there. I'm under the impression that assembly distribution would have to be considered in order for the solution to be viable.

Answer 1

There's a project at Microsoft called Dryad. Here's a good podcast with more info: http://www.dotnetrocks.com/default.aspx?showNum=378 It uses LINQ and is supposed to be easy to use. Microsoft has it doing some production stuff for Microsoft's AdCenter. Here's the research page. http://research.microsoft.com/research/sv/Dryad/

Dryad is quite expressive. It completely subsumes other computation frameworks, such as Google's map-reduce, or the relational algebra. Moreover, Dryad handles job creation and management, resource management, job monitoring and visualization, fault tolerance, re-execution, scheduling, and accounting.

P.S. I don't see a download, but if you contact them and you are serious you could probably get it.

Update: it's now available for download here: http://research.microsoft.com/en-us/downloads/03960cab-bb92-4c5c-be23-ce51aee0792c/default.aspx

Answer 2

I've been looking into a similar thing for my own work. So far I've found Digipede, Alchemi and NGrid, but they all need a SQL database as a grid manager except for NGrid which isn't as mature.

Answer 3

Take a look at our Appistry Enterprise Application Fabric (EAF) product. We have an EAF Community Edition license that is free, and good for five servers and/or ten cores. You can use the license for production or whatevever and it is perpetual (no end date). It's supported through our Peer2Peer developer network.

EAF takes a distributed network of N servers or virtual machines and makes them act as a single cloud resource for your Plain Old .NET objects and code. You package your .NET objects, and then deploy your .NET objects to the fabric with a single call. The fabric automatically provisions the code to all the servers, and makes it available to run. Your application and objects are versioned, so if you deploy an update, the fabric makes sure automatically that all servers are running the latest version. The theme here is that you don't administer individual boxes.

Your code running on the fabric's cloud environment runs as persistent services (as you desire), and not as transient things. The services are monitored and managed by the fabric's self-healing and self-organizing infrastructure. However, you do not write code to make this happen. It comes from the environment itself. Again, your code are just PONOs.

Clients call on your objects/services in the fabric through either .NET remoting or using our APIs. The code you deploy on the fabric does not have to use any special APIs, nor does it "know" its running in a distributed manner. Your code just automatically gets load balancing, scaling, and reliability with no single points of failure. The fabric is self-organizing and self-healing and works around hardware and other failures. Your code is free to utilize external resources like databases, but the fabric does not require a database.

Cool stuff to have built! :-) EAF is in production at many customers, and our latest release has a lot of new .NET enhancements. The Peer2Peer community is active and helpful, and again, the community edition license is free and does not expire.

So, specifically addressing what seems to be your requirements: 1. Automated deployment, versioning, and provisioning of .NET assemblies on a cloud? Yes 2. .NET objects becoming persistent services distributed across a cloud? Yes

Answer 4

Voyager from Recursion Software provides a mature framework for many styles of distributed architectures.

Answer 5

Qizmt is used by MySpace for distributed MapReduce operations. It's analogous to Hadoop from the Java world. Both projects are based on MapReduce by google. It runs on normal Windows boxes, unlike dryad which requires Windows Server High Performacne Computing (HPC)

Plus it's open source. Which is nice. http://code.google.com/p/qizmt/

Answer 6

Hi guys,

This may be a little late to the party - but we're the developers of Alchemi. We've developed a better version of the framework to address its limitations.

You might want to check out the Utilify Platform (www.utilify.com) - which is conceptually similar to Alchemi - with a more flexible model.

You do need a grid manager, and a database - however, you could use the free SQL Server Express version and it is tested to work on Windows Vista, XP, 2003.

We're happy to help you in using Utilify for your application.

Regards, Krishna