The first step in BPM processes is often ad hoc: A customer fills in a request form, an employee reports an issue or a salesperson enters a new sale. You can’t predict when exactly they will occur, but as soon as the task is executed, the business starts working on fulfilling the job. These ad hoc process tasks are an integral part of the business process, or are they not?
Business and IT often tend to disagree at this point. Lets analyse what makes this situation such a problem. What does the business expect from a BPM process, and what does IT expect?
The business perspective
From a business point of view, every step in the business process must be clear. Business analysts need to write down what is entered in this task, and verify that it contains the data required for the rest of the process. The screen-flow might be defined here, the actor is written down and a RACI matrix can be defined. Every task has clear in- and outputs. Certain conditions may apply, and tools might be needed to fulfill the task. This is the domain of the business analysts, and provides handholds for auditors and managers.
As such this first task is important to include in the BPM flow.
The IT perspective
From IT perspective however, this is a very awkward situation. IT gets involved to automate the workflow, but where does the workflow start? The business process magically begins just before the user starts her work. How does the system know it needs to send a task to the user? If the user is not part of the organisation, can we even send a task to the user? What if the user halfway decides not to fill in the request, do we still have a process then? What if she continues later on?
The system can’t predict that there will be a process, so it has to wait until the user finalizes the task. Once the task is committed, the process starts and the system takes control. This implies that the first task is not part of the workflow. The best way for IT to build this, is to have a data-entry screen in some user interface application. The user interface is free format, and starts the remainder of the workflow by sending a start message.
We could compromise by defining a manual task directly after the process start. This task will not be executed by the workflow engine, but can be used as a placeholder for the information that belongs to the ad hoc process task at the start of the process. The user will enter data using some external front-end application which acts as the manual user task.
This leaves us with two separate types of tasks
- Ad hoc tasks, which are modeled as manual tasks in the BPMN language and aren’t converted to the workflow. They are implemented in the UI layer, and send a message to the workflow engine, which needs a message event to receive the information.
- Assigned user tasks, which are modeled as plain user tasks in BPMN. These have a user, role or group that needs to complete the task, and are the bread and butter of the workflow.
Since we model both tasks, the business can have a complete view of the process. The fact that we don’t execute manual tasks means that we can automate the workflow. If we use the same user interface application to wrap both the ad hoc tasks and the workflow tasks, the integration can be done seamless and the end-users will never know the difference.
You know about good BPM design, about separation of long-lived flows and short-lived interactions. You want to move your forms outside of your long-lived flows, preferably in a dedicated GUI product, but the business requires you to put the forms in the BPM flow. Either because they want one system to handle all interaction, they need the persistence in every step of the user-interaction, or maybe because they need the transaction log for legal reasons. How do you solve this dilemma?
Where to put user forms in BPM
User forms are a good example of a short-lived interaction. The interaction can be persisted, but once the user has completed her task, the flow finishes. This type of interaction-flow typically changes a during its lifetime. The layout of the forms may change, extra fields may get added and the order of the forms in one interaction may even change. This fast-changing piece of code doesn’t belong with the stable code for a business process: the core process almost never changes.
Fast changing code doesn’t belong together with a stable business process
The solution is quite simple. We build at least two layers of process flows: the top layer consists of one long-lived flow that defines the basic steps of the business process. The layer below that consists of a short-lived flow per user task. These processes are packaged separately so that the version control on these packages can be done independently.
The business process defines the high level steps. When it needs a user interaction, it calls an other process and delegates the interaction this short-lived one. All the forms are in the short-lived process. The long-lived process starts the short-lived one, and waits for its completion. Once the sub-process has completed, the main process continues on.
- all interactions are inside the BPM engine: everything is logged and handled in a consistent way as required by the business.
- all forms are inside short-lived processes. If a form is changed, the old versions are gone within a couple of hours or maybe days. No longer do you need to support dozens of different form versions because the processes are still running.
- versioning and deployment of forms can be done independent of the business processes
There is one big pitfall here: if you move the forms into a sub-process, you will feel the urge to define an interface between the main and sub-process that passes the data from the main process to the sub-process and back. Don’t do this!
If you define an interface that passes along the actual data, you are binding your main process to the model of the data used in the forms. You have now moved your dependency, instead of removing it.
Instead, you need to store the actual data somewhere else, for example inside a database. You just pass a reference to the data when you start or end the sub-process. All bindings to the data-model are now removed from the main process. The main process can use the data when required, but does not need to know the shape of the data.
Don’t pass data from the main process to the subprocess and back, but only pass references.
Keep your data out of your BPM process, but store references instead. Put your data in a database, that is what it is for!
If you keep fine grained, often changing elements such as forms and data outside of your BPM flow, you end up with a more maintainable design. The BPM process itself just defines what steps to take, but should not define how to do them. Instead it delegates to subprocesses and services that perform the task at hand.
At the start of any large business automation project (after the analysis of the business domain, but before the design starts) you should always ask yourself the question: will I be using case management or classic BPM? This decision has a huge impact on the final result of your BPM project. But what is the difference between case management and classic BPM? And what will be the impact of choosing either solution?
Business Process Management originally was a method of analyzing and optimizing the processes in a business, as opposed to focusing on the individual tasks. By looking at the tasks as a part of the whole, synergy between tasks can be found, and improvements can be made. When this principle was elevated to the IT-era, it became a method that defines all steps in a business process, and made those steps executable.
At the core of this BPM philosophy is the thought that all cases are the same (or one of only a few possible options). Every process therefore goes through exactly the same steps. The content may vary, but the process stays the same. Classic BPM defines the order in which the steps occur. It is a great tool that ensures all necessary tasks have been finished. In the mean time it is excellent at keeping record of what has been done, by whom and when.
Simple BPMN flow
The mayor advantages of using this type of BPM are predictability, simplicity and visibility. A process is at a certain step, we know who is responsible for that step, and we can find important statistics such as workload of the employees. All possible actions at a certain point in the process are known to us, because everything is defined upfront in the process flow.
This upfront definition of all possible orders of steps is also the problem with classic BPM. In simple, straight-forward flows, we can do this easily. With longer, complex flows it quickly becomes near impossible. Cases may involve completely different steps, and share very little. Imagine the business process of requesting an official permit for building a house or factory. This type of permit needs to be investigated by many different departments: does it comply with the local plans, are there prior applications that were rejected, does it pose environmental risks, etc.
If you’d put all of these steps in sequence, it would take too long to evaluate the plan. But running all possible steps in parallel can become a big tangle of dependencies. What if the output of one department influences the flow for the other department? Do we need to model all these types of relations in our big business process?
Case Management Example in BPMN
Instead of focusing on the process, we could also focus on the subject. This is what case management does. Case management consists of many short processes that all operate on a central piece of business information: the case. Each process changes the state of the business information, and each state change might trigger new processes. For example, a high-over environment scan might show that an expert needs to investigate the risks to the groundwater. The high-over scan changes the state on the case, and a new business process is started for the environmental expert to make a report. There is no hard link between these two processes, so they don’t need to know about each-other. This makes maintenance a lot easier.
Synergy with complex event processing
In the classic BPM example, the process is predefined and once started, runs it course with minimal outside interaction. Though it is possible to receive signals from outside, this is not often used in BPM. The idea was to define all steps, including the steps that lead up to the signal, so why would you need a signal from outside? It means that you have missed a part that should have been in the model.
In case management, we have an external source: the case . This business object needs to be monitored, and every state change might trigger a new process. Monitoring an object for changes is exactly what complex event handling is good at. We can monitor all changes on the object. When certain combinations of signals are found, we start a new BPM process to handle the situation. The business process itself doesn’t need the complex logic of monitoring the state, as it would have in the classic BPM example. The complexity is shifted towards a rule engine, which is easier to work with, and dedicated for this kind of complexity.
Reduced complexity with short-lived processes
The move from one big BPMN flow to many short-lived BPM processes with a shared case object makes the design a lot easier. The complex decision logic is removed from the BPM flow, making the design simpler and more maintainable. The short-lived BPM processes are also to be preferred over long-lived processes, because the governance of long-lived processes tends to be a lot harder. Imagine finding a bug in a process. When the process lives for 3 hours max, we know that after installing the fix, the bug will have left the system on its own after 3 hours, and we don’t have to keep track of it anymore.
Now if the process remains active for 3 months, we would need to keep track of it for all that time. For every change on the system during those 3 months, we need to think of this particular bug and its impact. For every incident report, we need to cross-check it with this bug. This might sound trivial, but in a large, fast-changing environment there might be a hundred small bugs that all demand a bit of attention. Added up, the support organisation might get bogged down by it. Reduction of the complexity is so very important to keeping the support organisation flexible.
When to use case management
Case Management isn’t a magical solution to all BPM problems. It fits a certain area of the BPM domain. When your are facing a straightforward workflow, case management might not be a good solution. The simplicity of the classic BPM can make communicating about it much easier. The BPMN flow is easier to comprehend, and holds all the possible interactions.
Only when the flow becomes more complicated, with parallel tasks and many conditions, case management begins to shine. Taking the complexity out of the BPMN design, and into a stateful business object, allows us to manage the problems. Moving the complexity into a rule engine reduces complexity in the flows even more. Most import thing to take away here is that you need to avoid complexity in long-living processes, since they are hard to govern. Case Management is a good tool to accomplish this.
People often wonder makes a good BPM process design. Should a process include many small steps, or should it be a few large steps? Do we avoid user actions, or do we make extensive page-flows as part of our process? I find that these things are not what makes a process design good or bad. Good design is mostly about avoiding pitfalls. Here are some tips that will make your BPM processes better.
Separation of business data and process state
The most important step for good BPM process design is to remove all your business data. You don’t need to keep the address of your customer in your BPM database. It introduces only problems, because what happens if your customer moves while your BPM flow is in the middle of the process? Instead you only want references to entities in your BPM. Everything else must be moved to a service elsewhere in your enterprise environment.
Keep the focus on the process: which steps do I need to take, and which decisions do I need to make? The business data is there to facilitate the flow, but instead of pulling the data inside and holding it inside the process, you need to access it on-demand and only store the answer of your question, not the data that leads up to the answer. You may want to log the decision making, but you should not keep the data itself for later use.
Not so long living processes
We often state that BPM is for long-living process as contrary to the normal, short-living EAI and SOA services. This is however a major pitfall. Long-living in this context means that it the process can be suspended for a while, it is able to survive a system reboot and may wait for user input for a while. It doesn’t mean that the process should live for months or years. It would be a very bad BPM process if you designed a BPM process that starts when somebody becomes a customer, and ends when she leaves. The process would potentially live for many years, pulling along the burden of many years of patches, changes and upgrades. In the end, every process would be unique: the combination of upgrades and the state at which the upgrade happens will make it so that no two processes share the same history. A system of many unique processes is maintainable, and impossible to govern.
Instead we should have multiple short lived business processes that focus on a certain state change: somebody becomes a customer. This state change is always about a business entity, and this entity will be the part that continues on after the process ends. The BPM process will be long-living, but in this context, that may be 1 hour for the administrative steps to finish, or a few days while the credit check is pending, etc.
This process will now live for a few days, making maintenance and government much easier.
Move forward, not backwards
An efficient business process never returns to a previous step. Your customer intake should include all information that is required for your BPM process, so that you don’t need to call the customer again for more information. The same holds for the other steps in your flow. Once a user has finished her input, the flow should move on and not return to her.
It should also hold for business services: you go to a service and complete your interaction in one, atomic step. This avoids a lot of pain, because otherwise it could happen that the business data is modified inbetween your steps, making your second step invalid. Handling these types of errors is one of the most costly parts of BPM maintenance.
In process design, this means that the decision making may need to occur during the interaction with said user. An extra screen with detailed questions may popup if a certain condition applies later in the business flow. By pulling this decision to an earlier moment in the flow, we can avoid going back to the user for follow up questions. This avoids stalls in the process, such as the original user being on holiday or otherwise unable to pick up this specific task.
Finally it also holds that a loop should not occur in the normal flow. You can have a loop that handles every item in an array, but you should avoid a loop that performs the same steps for the same business entity again and again. If such a business logic is required, it should be multiple business process instances started after eachother on the same business entity, not one big process.
There is however one exception: during error handling, a retry is very common, and in this case it is perfectly acceptable do loop back to a failed step to retry it.
Keep it simple stupid
Designers tend to overthink the BPM process. Many of the details can and should be handled outside of the stateful BPM process. Decision making for example can be handled in the BPM code, but it is harder to maintain than a simple service that provides a yes/no answer. The code in the BPM flow should then only test the answer of the service, and not do the underlying comparison itself. The business logic of these business rules tends to change often, while the BPM flow normally stays the same. A stateless service can be modified easily without issues with backward compatibility and job migrations, while a stateful BPM flow has all of this pain embedded. Keep the hard stuff outside of the BPM code, and only pull the results in the BPM scope.
Good BPM process design is not easy. It is mainly about avoiding the common pitfalls. The ones listed here are by no means complete, but they are often overlooked by the normal trainings you get by the BPM software development firms. The normal trainings are about what is possible, but they tend to forget that it may be unwise to do so. Don’t go blindly on what the tools offer: to them BPM is a hammer and every problem a nail. You should know when to use the hammer, and use it well.