Business Rule Management Systems
An BRMS (of the type sold by ILOG) allows you to set up a set of rules and uses an algorithm such as Rete to match based on the conditions, generate assertions (which can in turn be matched by other rules) or take actions based on these rules. These can be embedded in application and used to support configurable business logic.
Business Process Management Systems
A BPMS (such as MS Workflow Foundation or any of the BPEL engines on the market) steps an application through a series of stages in a workflow. These stages can be assigned to different people. A BPMS can use an embedded BRMS to make decisions on where to direct a workflow. For example, ILOG makes a plugin that allows its rules engine to be used with MS Workflow foundation.
The difference between the two
A rule-engine based application uses an embedded rules engine (BRMS) with a mechanism to read the data record it is working with and hooks for actions directed by the rules engine to be executed by the application. It may (for example) read an XML DOM, matching based on the content of nodes within the DOM and recording assertions as other nodes. Other API's are possible. The application can also have callbacks registered where matches can trigger actions where the rules engine invokes callbacks into the application.
A workflow-aware application will present different screens to one or more users under the control of a process that is defined in the workflow engine. This engine can be external to the application, and a workflow can hop across more than one application. The workflow engine may also manage explicit compensating rollbacks and error handling for applications that cannot directly participate in distributed transactions.
The two items are not mutually exclusive. It is possible to have a workflow-aware application that has an embedded rules engine. It is also possible to have a rules engine driving business logic and instantiating or making branching decisions in a workflow. Many workflow managers come with a rules engine for precisely this reason.