The first step in BPM processes is often ad hoc: A customer fills in a request form, an employee reports an issue or a salesperson enters a new sale. You can’t predict when exactly they will occur, but as soon as the task is executed, the business starts working on fulfilling the job. These ad hoc process tasks are an integral part of the business process, or are they not?
Business and IT often tend to disagree at this point. Lets analyse what makes this situation such a problem. What does the business expect from a BPM process, and what does IT expect?
The business perspective
From a business point of view, every step in the business process must be clear. Business analysts need to write down what is entered in this task, and verify that it contains the data required for the rest of the process. The screen-flow might be defined here, the actor is written down and a RACI matrix can be defined. Every task has clear in- and outputs. Certain conditions may apply, and tools might be needed to fulfill the task. This is the domain of the business analysts, and provides handholds for auditors and managers.
As such this first task is important to include in the BPM flow.
The IT perspective
From IT perspective however, this is a very awkward situation. IT gets involved to automate the workflow, but where does the workflow start? The business process magically begins just before the user starts her work. How does the system know it needs to send a task to the user? If the user is not part of the organisation, can we even send a task to the user? What if the user halfway decides not to fill in the request, do we still have a process then? What if she continues later on?
The system can’t predict that there will be a process, so it has to wait until the user finalizes the task. Once the task is committed, the process starts and the system takes control. This implies that the first task is not part of the workflow. The best way for IT to build this, is to have a data-entry screen in some user interface application. The user interface is free format, and starts the remainder of the workflow by sending a start message.
We could compromise by defining a manual task directly after the process start. This task will not be executed by the workflow engine, but can be used as a placeholder for the information that belongs to the ad hoc process task at the start of the process. The user will enter data using some external front-end application which acts as the manual user task.
This leaves us with two separate types of tasks
- Ad hoc tasks, which are modeled as manual tasks in the BPMN language and aren’t converted to the workflow. They are implemented in the UI layer, and send a message to the workflow engine, which needs a message event to receive the information.
- Assigned user tasks, which are modeled as plain user tasks in BPMN. These have a user, role or group that needs to complete the task, and are the bread and butter of the workflow.
Since we model both tasks, the business can have a complete view of the process. The fact that we don’t execute manual tasks means that we can automate the workflow. If we use the same user interface application to wrap both the ad hoc tasks and the workflow tasks, the integration can be done seamless and the end-users will never know the difference.
You know about good BPM design, about separation of long-lived flows and short-lived interactions. You want to move your forms outside of your long-lived flows, preferably in a dedicated GUI product, but the business requires you to put the forms in the BPM flow. Either because they want one system to handle all interaction, they need the persistence in every step of the user-interaction, or maybe because they need the transaction log for legal reasons. How do you solve this dilemma?
Where to put user forms in BPM
User forms are a good example of a short-lived interaction. The interaction can be persisted, but once the user has completed her task, the flow finishes. This type of interaction-flow typically changes a during its lifetime. The layout of the forms may change, extra fields may get added and the order of the forms in one interaction may even change. This fast-changing piece of code doesn’t belong with the stable code for a business process: the core process almost never changes.
Fast changing code doesn’t belong together with a stable business process
The solution is quite simple. We build at least two layers of process flows: the top layer consists of one long-lived flow that defines the basic steps of the business process. The layer below that consists of a short-lived flow per user task. These processes are packaged separately so that the version control on these packages can be done independently.
The business process defines the high level steps. When it needs a user interaction, it calls an other process and delegates the interaction this short-lived one. All the forms are in the short-lived process. The long-lived process starts the short-lived one, and waits for its completion. Once the sub-process has completed, the main process continues on.
- all interactions are inside the BPM engine: everything is logged and handled in a consistent way as required by the business.
- all forms are inside short-lived processes. If a form is changed, the old versions are gone within a couple of hours or maybe days. No longer do you need to support dozens of different form versions because the processes are still running.
- versioning and deployment of forms can be done independent of the business processes
There is one big pitfall here: if you move the forms into a sub-process, you will feel the urge to define an interface between the main and sub-process that passes the data from the main process to the sub-process and back. Don’t do this!
If you define an interface that passes along the actual data, you are binding your main process to the model of the data used in the forms. You have now moved your dependency, instead of removing it.
Instead, you need to store the actual data somewhere else, for example inside a database. You just pass a reference to the data when you start or end the sub-process. All bindings to the data-model are now removed from the main process. The main process can use the data when required, but does not need to know the shape of the data.
Don’t pass data from the main process to the subprocess and back, but only pass references.
Keep your data out of your BPM process, but store references instead. Put your data in a database, that is what it is for!
If you keep fine grained, often changing elements such as forms and data outside of your BPM flow, you end up with a more maintainable design. The BPM process itself just defines what steps to take, but should not define how to do them. Instead it delegates to subprocesses and services that perform the task at hand.
People often wonder makes a good BPM process design. Should a process include many small steps, or should it be a few large steps? Do we avoid user actions, or do we make extensive page-flows as part of our process? I find that these things are not what makes a process design good or bad. Good design is mostly about avoiding pitfalls. Here are some tips that will make your BPM processes better.
Separation of business data and process state
The most important step for good BPM process design is to remove all your business data. You don’t need to keep the address of your customer in your BPM database. It introduces only problems, because what happens if your customer moves while your BPM flow is in the middle of the process? Instead you only want references to entities in your BPM. Everything else must be moved to a service elsewhere in your enterprise environment.
Keep the focus on the process: which steps do I need to take, and which decisions do I need to make? The business data is there to facilitate the flow, but instead of pulling the data inside and holding it inside the process, you need to access it on-demand and only store the answer of your question, not the data that leads up to the answer. You may want to log the decision making, but you should not keep the data itself for later use.
Not so long living processes
We often state that BPM is for long-living process as contrary to the normal, short-living EAI and SOA services. This is however a major pitfall. Long-living in this context means that it the process can be suspended for a while, it is able to survive a system reboot and may wait for user input for a while. It doesn’t mean that the process should live for months or years. It would be a very bad BPM process if you designed a BPM process that starts when somebody becomes a customer, and ends when she leaves. The process would potentially live for many years, pulling along the burden of many years of patches, changes and upgrades. In the end, every process would be unique: the combination of upgrades and the state at which the upgrade happens will make it so that no two processes share the same history. A system of many unique processes is maintainable, and impossible to govern.
Instead we should have multiple short lived business processes that focus on a certain state change: somebody becomes a customer. This state change is always about a business entity, and this entity will be the part that continues on after the process ends. The BPM process will be long-living, but in this context, that may be 1 hour for the administrative steps to finish, or a few days while the credit check is pending, etc.
This process will now live for a few days, making maintenance and government much easier.
Move forward, not backwards
An efficient business process never returns to a previous step. Your customer intake should include all information that is required for your BPM process, so that you don’t need to call the customer again for more information. The same holds for the other steps in your flow. Once a user has finished her input, the flow should move on and not return to her.
It should also hold for business services: you go to a service and complete your interaction in one, atomic step. This avoids a lot of pain, because otherwise it could happen that the business data is modified inbetween your steps, making your second step invalid. Handling these types of errors is one of the most costly parts of BPM maintenance.
In process design, this means that the decision making may need to occur during the interaction with said user. An extra screen with detailed questions may popup if a certain condition applies later in the business flow. By pulling this decision to an earlier moment in the flow, we can avoid going back to the user for follow up questions. This avoids stalls in the process, such as the original user being on holiday or otherwise unable to pick up this specific task.
Finally it also holds that a loop should not occur in the normal flow. You can have a loop that handles every item in an array, but you should avoid a loop that performs the same steps for the same business entity again and again. If such a business logic is required, it should be multiple business process instances started after eachother on the same business entity, not one big process.
There is however one exception: during error handling, a retry is very common, and in this case it is perfectly acceptable do loop back to a failed step to retry it.
Keep it simple stupid
Designers tend to overthink the BPM process. Many of the details can and should be handled outside of the stateful BPM process. Decision making for example can be handled in the BPM code, but it is harder to maintain than a simple service that provides a yes/no answer. The code in the BPM flow should then only test the answer of the service, and not do the underlying comparison itself. The business logic of these business rules tends to change often, while the BPM flow normally stays the same. A stateless service can be modified easily without issues with backward compatibility and job migrations, while a stateful BPM flow has all of this pain embedded. Keep the hard stuff outside of the BPM code, and only pull the results in the BPM scope.
Good BPM process design is not easy. It is mainly about avoiding the common pitfalls. The ones listed here are by no means complete, but they are often overlooked by the normal trainings you get by the BPM software development firms. The normal trainings are about what is possible, but they tend to forget that it may be unwise to do so. Don’t go blindly on what the tools offer: to them BPM is a hammer and every problem a nail. You should know when to use the hammer, and use it well.