Workflow system

Question

I've just started at a new communications company, and we're looking at a workflow / intranet system to manage jobs and processes.

Basically, we receive data files from clients which we then process through our systems.

1. Receive data file (FTP, Email, etc)
2. Process data file (either generic script with data mapping to the file, or bespoke ETL package). Adds address values
3. Create printstream (send processed data file into a postscript / PDF composition engine), or create email output
4. Send output to production floor (copy to printer input stream, mailing machines)
5. Process other streams (e.g. send emails / faxes, upload to e-Archive)
6. Update internal systems (e.g. warehouse stock, invoicing)

We also have a lot of other internal business processes (e.g. reprocessing damaged output, processing dead/returned mail).

I'm trying to keep all elements separated. Some will be off the shelf (e.g. printstream composition, email sending / management, CRM). Some will be built in house (e.g. reprocess damaged output).

But, I'm looking for something to tie it all together, and put the business workflow processes in. E.g. scheduling jobs, kicking off data processing tasks in sequence and managing errors. A lot of this will have human steps. Also, put in SLA management and business activity monitoring / reporting.

One key requirement soon is for automated file receipt and processing (i.e. directory watching and matching to client / application).

I'm keen for something that's easy to manage and maintain (e.g. adding in new steps to a workflow, or conditional logic, or whatever).

I realise this is a big job, and at the moment we're focusing on each individual component and putting manual processes in place until we get a system to manage it. We don't want to design a gargantuan bespoke system to tie it all in, but would rather look at buying some kind of workflow or integration system.

Any suggestions? I've had a look at Biztalk, but not sure if it's overkill or not suited for internal-only systems. Another product I've been exposed to is Sagent Automation, but it looks a little pokey.

-- EDIT --

Forgot to mention, our existing skillset is largely Microsoft. So anything in Microsoft technologies / .Net based would be preferable. But if there's a fantastic product, we're not adverse to upskilling

Answer 1

Check out Apache's Active MQ. It implements the Java Message Service 1.1 specification, layers on a servlet API, and has tons of features that should address your requirements. You can also layer on Camel, which adds a rich implementation of many enterprise integration patterns.

Typically, JMS messages are persisted in a transactional database, which can be configured to give you extremely high degrees of fault tolerance (eg, RAID, master-backup database machine pairs, multiple copies of transaction log files). On top of the database can go multiple, load-balanced app server machines running Active MQ, to give you scalability and high-availability. I think you'll find that you can write your components in a very decoupled fashion if you use Active MQ as your common message bus.

In JMS, when a message is de-queued by a consumer, the consuming process must later confirm that the message was successfully handled. If a confirmation does not come in in time, the JMS system will revive the message so another consuming process can attempt to handle it. This means you can run multiple copies of your application to gain reliability and fault tolerance.

Take a look at O'Reilly's Java Message Service, 2nd Edition, which just came out this week.

A different avenue would be to look into BPEL (Business Process Execution Languge).

Edit: I'm not very familiar with Microsoft offerings, but MSMQ seems like the equivalent to JMS.

You should be able to use ActiveMQ in a Microsoft environment. They claim to support "cross language clients" like "C# and .NET". And even if that should be problematic, since ActiveMQ has a Java servlet-based API for queueing and de-queuing messages, the outside world only has to be able to make HTTP requests to the ActiveMQ server. That should limit the amount of learning your team would have to do. Good luck, this sounds like an awesome project!

Answer 2

SharePoint has a workflow engine that works very well. You can build your workflow using SharePoint designer or Visual Studio 2008. It uses Windows Workflow, which is similar to BizTalk (if not the same engine), but without BizTalk's other services that may not be necessary for your application.