Wolfgang asked me the following question today:

Is there a log filter in ProM that splits an event log in two logs? The first log should contain all instances that executed a specific activity ‘X’. The second log contains all the other instances, i.e., those that did not execute activity ‘X’.

Because I get the same question more often, I figured that I share it here with you so that more people can benefit from it.

The answer is that there is no log filter that does that. Instead the LTL Checker plug-in can be used. Here is a step-by-step description:

1. Open the log and select the LTL Checker from the Analysis menu
2. Then select formula ‘eventually_activity_A’
3. Provide the name of the activity as parameter A (so that would be ‘X’)
4. Press the button Check formula
5. Choose Export -> Correct instances -> Efficient MXML.GZ Export from the menu to export the log with all process instances that contain ‘X’
6. Use Export -> Incorrect instances -> Efficient MXML.GZ Export to export those process instances that do not contain ‘X’

Here is a screenshot, where I export a log that contains only cases for which activity ‘Inbound Email’ was performed at some point in time.

There are a whole lot more ways to split up the event log. Check out the LTL Checker manual, which is located in the documentation folder of your ProM installation!

We continue our interview series with the “godfather” of process mining, Wil van der Aalst. He has written an enormous number of books and papers, but he mainly shines by his consistent quality. Wil is a full professor at Eindhoven University of Technology and the head of the Process Mining group at TU/e.

Wil talks about how process mining research was started ten years ago, how he defines process mining today, what it takes to be successful as a scientist, and — last but not least — how to escape military service.

As announced by Kurt Jensen on the Petri Net mailing list, Carl Adam Petri passed away on 2 July.

I am very sorry to inform you that the farther [sic] of our research area, Carl Adam Petri, has passed away a few days ago. Carl Adam Petri has had a tremendous influence on the theory of concurrency and many other areas of science. We are all very grateful to him for laying the foundation for our flourishing research area.

Petri, a German mathematician and computer scientist, was a pioneer of research in concurrent systems. He invented Petri nets, a formalism to describe and analyze dynamic, concurrent behavior. Any researcher working in process modeling or process mining, among many other fields, knows what we owe to his contributions.

George Varvaressos initiated a discussion on process mining in the BPTrends LinkedInGroup (you can join the group here). In this discussion, Paul Harmon commented on process mining:

I think its an important technical development. I suspect, however, that its too technical to gain widespread use as a stand-alone product. That isn’t to say that there won’t be early adopters that will use it and benefit, but I suspect that most managers will encounter process mining as a utility in a BPMS product. They will be using an existing BPMS suite working to model and improve an existing process, and realize that their BPMS tool provides them with process mining capabilities they can use.

Now, he may be right about that future Business Process Management Systems (BPMS) will have a process mining (checklist?) feature. But I am convinced that process mining has the potential to be much more than just an add-on component. Here is why.

Surely, it is interesting to collect existing logs and understand how the process works before designing and implementing a new, process-centric IT solution. It reminds me of the idea to wait until dirt tracks emerge around a new building complex before putting the actual walkways in concrete. But you can rethink the systems that support the execution of processes more profoundly.

In any BPMS, there is a natural trade-off between flexibility and support. The more support you want to give, the more you run the risk to over-specify and unnecessarily limit flexibility. But how often does over-specification actually stem from the desire to have transparency about the business process, from the desire to know what is going on?

Wanting transparency is not the same as wanting control, in the sense of preventing undesirable behavior to happen in the first place. Control and flexibility are also a natural trade-off. But flexibility and transparency do not need to be.

The mining and monitoring of processes based on good quality, ‘after-the-fact’ data can provide transparency about the actual process without limiting flexibility. Where appropriate, process mining technology can help to move towards more human-centered IT solutions that are still process-oriented and let managers stay in control by providing them with a clear picture of the process reality.

The role of internal audit departments is to help organizations ensure effectiveness and efficiency of operations, reliability of financial reporting, and compliance with laws and regulations in an independent and objective manner.

Due to scandals and the economic downturn, there have been ongoing discussions that internal audit must be proactive and redefine its value. While PricewaterhouseCoopers’ Internal Audit study 2009 still emphasized the need to do more with less, the 2010 study sounds somewhat less urgent. But regardless of whether the pressure is high or moderate, process mining fits into trends such as continuous auditing, where the use of technology plays a key role.

In contrast to other Computer Aided Audit Tools (CAATs), process mining provides an explicit process perspective. This EDP-Auditor article (in Dutch) describes an example and the possibilities of process mining in the auditing field. [Update: This IEEE Computer article on 'Auditing 2.0' describes a new auditing framework based on process mining.] Overall, one can foresee that process mining tools will be one of the many in the auditor’s tool box.

Here are 6 reasons why process mining should be interesting for auditors:

1. Audit the actual process reality

Process audits are still too often just based on Interviews, What if analyses, and Design reviews that review the intended but not necessarily the real process. Today’s business processes are supported by ERP en WFM-applications that are too complex to understand but also record detailed information about the execution of these processes. Process mining can be used to make the actual transaction flows visible by evaluating these IT audit trails in an automated fashion.

2. Test entire data populations

To improve effectiveness in search for errors or unusual transactions internal audit should test entire data populations automatically. Process mining techniques such as LTL checking and Conformance checking can be used to verify the compliance to rules (e.g., segregation of duty constraints) or prescribed procedures based on the actual process execution records.

3. Make control processes visible

Also built-in controls such as authorization steps are usually reviewed on a design level. For example, there are tools that verify whether people currently have conflicting access roles that may put the organization at risk (but not whether there are conflicting roles at different points in time). An automatic mining of the control processes can help to audit the effectiveness of these controls by making visible when these built-in controls take place, who performed them, when controls lead to rejection, etc.

4. Targeted audits

Especially in large organizations audits are still performed based on a yearly audit plan. Conducting audits on a more targeted basis helps to concentrate on higher-risk areas. But it also requires continuous data analysis and needs to be facilitated by technology. Process mining can be leveraged in the context of such a continuous monitoring infrastructure to do quick scans and bring potential problems to attention.

5. Improve auditing process

To improve the efficiency and quality of the auditing process itself, there are several tools that support the auditor’s workflow and make sure that all tasks are done and found issues are resolved. By analyzing the logs of these audit support systems one can go a step further and evaluate the efficiency and quality of the audit process in an objective way.

6. Add value by delivering business insight

Internal auditors are independent of the operational side and often report directly to the CEO of the company. But although their role is also to monitor the efficiency of operations, actual business insight is usually only delivered on an ad-hoc basis. Process mining can be used to detect bottlenecks and other inefficiencies in the actual business processes, which can then be shared with the relevant stakeholders to expand the overall value of the audit function.

Let me know where you agree or disagree, and how you would rank these points in importance!

When I watched this presentation for the first time, I liked it immediately¹. After having thought about it some more, I think I know why.

First, I think he obviously has a point. We’ve all had that sneaking suspicion, at some point in our lives, that carrots and sticks may not be the be-all and end-all of motivation techniques. Some people seem to get by with just the minimum, while others are burning the midnight oil with no external trigger whatsoever. And under pretty similar circumstances, that is.

Second, the production quality is just stunning. You rarely see a great topic so masterfully illustrated for maximum impact. Okay, it’s post-produced and all that, and you could never possibly deliver something like this live and in PowerPoint. But still, there’s a benchmark for you.

Third and last, it’s downright inspiring². I don’t know how to put it in any other way, but it just makes you want to get going and do something meaningful.

It has been a real blessing for me that, for most of my life, my motivation to do stuff has been overwhelmingly intrinsic, and greatly satisfying — and I still have a whole lot of work that is inspiring, meaningful, and important to me ahead.

So would you please excuse me while I get right to it…³

1. There’s also a closely related TED talk by the same speaker right here, if you can’t get enough.
2. Also, make sure to check out the other RSA talks (Personally, I also liked Philip Zimbardo, of Stanford Experiment fame).
3. As always, you can of course shoot right back in the comments

Some weeks back I told you about the Demo Track at this year’s BPM Conference in Hoboken, US. Well, it turns out that there was probably a little less interest than expected, but in any way the deadline for submissions has been extended to 30 May.

Like I have told you the last time around: Software implementations and demos are a vital part of BPM research. BPM often gets accused of being too much of an “ivory tower” science, where little scientific results actually get to influence the professional practice. We here at Fluxicon attempt to fix at least some part of that perception, most of all when it relates to process mining. However, it is up to all of you to go prove that BPM researchers can not only draw visions of a beautiful future, but also lay the first stepping stones towards it, and put your code where your mouth is!

Anyway, again the happy news: The deadline for submission of demo proposals has been extended to 30 May. If you have some code you would like to showcase, get those few pages written and go have yourself some fun in NYC!

One of the main benefits of process mining is that it can help you understand how your process really works based on hard evidence. Process mining uses log data from IT systems, which allows an objective reconstruction of the process flows.

On the other side of the spectrum are manual process documentation activities, which are in the domain of process consultants. Interviews are time-consuming and often difficult due to different interpretations of reality:

• Employees may have problems to articulate how they work in a complete fashion. Because they do these things every day, many aspects of their work will seem obvious to them.
• They are likely to describe only the ’sunny day scenarios’ or ‘happy flows’, and forget about the exceptions.
• Or, they might emphasize the exceptions and things they are not happy with in a disproportionate way.

Requirements analysis resorts to more objective elicitation methods such as observing employees to reveal hidden assumptions. Process mining goes further by objectively analyzing past process executions over a potentially large time frame. One can then either focus on the “happy flows” or look at the complete picture, including all the exceptions, depending on the desired level of detail.

Nevertheless, the goal of process mining is not to eliminate talking to people — quite to the contrary. It can facilitate discussions in workshops and interviews by providing a starting point. So, you don’t have to start from an empty sheet. Talking to people and doing process mining are complementary.

To make it more concrete, I collected 7 reasons to do process mining as a consultant:

1. Come up with first results quickly

Since the traditional process discovery activities can take several weeks or months, there is a considerable time in the beginning of the project where your client may not see the value that you deliver. Being able to do a process mining quick scan of just one of the most important processes, and to come up with first results and hypotheses quickly, can help to increase trust and engage people for the remainder of the project.

2. Ask “Why” instead of “What”

Because process mining provides an objective reference on how things are done, you can focus on understanding the “why” of the process. The reasons for why people work the way they do can rarely be uncovered using observation or data analysis. However, understanding the root causes of inefficiencies also on the human level is crucial to successfully implement organizational change. Focus on the “why” and maximize your value by digging deeper than you normally could.

3. Get a head start within new domains

Most consultants are specialized to assist clients in specific domains. In domains they have worked before they know how to approach their job, since processes in many domains are quite similar. Having a tool to quickly understand the process landscape of an unfamiliar domain can help you when taking on assignments in new domains. Also, such a tool can increase the productivity especially of junior analysts, right from the start.

4. Skip workshops in politically difficult sitations

Sometimes politics prevent you from bringing everyone to the table who should be there for a process documentation project. People may refuse to collaborate because of internal power struggles, because they don’t see the use, or because they don’t see a problem. In such situations, the use of log data to understand how the process works can help to skip interviews and process discovery workshops, and to still be able to perform a process analysis and deliver results.

5. Underpin the credibility of your results

Knowing the ‘as-is’ process is essential for being able to decide which route should be taken to achieve a goal. Unlike with typical query tools, where you need to know the question in advance, process mining can provide you with a complete picture of what is happening. Furthermore, having an objective basis for your problem analysis gives you credibility and helps to underpin the correctness of your conclusion. Nobody can say they don’t believe you when you can prove your points with hard data.

6. Help your client to justify changes within the company

Sometimes, your client is with you — but other people in the company don’t believe in the project and just dismiss the results as “not true”. What can you do? By providing your clients with an objective reference you can put them in a stronger position and help them succeed at what they want to achieve within their own organization.

7. Compare “before” and “after”

Especially for process improvement projects you may want to demonstrate the effect of the implemented changes. For example, you can show how the process has been streamlined after a change (and one month of new data collection) by comparing the “before” and “after” images. Ideally, the process performs much better now, and you can use process mining to communicate these results.

So, this was my list of seven reasons for doing process mining as a consultant. Let me know whether you agree or disagree, and tell me what I am missing!

I just found an article called Engineer Thinking on Matt Gemmel’s blog. If you are at all involved in the development of GUI software, you should definitely read the whole thing. Here’s a choice quote, though:

If you’ve exposed underlying complexity or unnecessary choice in your software because you see those things as inevitable, it’s because your job isn’t finished. If you’re going to write GUI software for other people to use, do it properly, and treat those people like human beings instead of software engineers. If you want to expose complexity to the user and wash your hands of it, write command-line tools – or utilities that are used exclusively by other machine processes.

You can’t have it both ways. Writing GUI software is for people who strive for excellence not only in the “software” part but in the “GUI” part too.

Yeah, we have all been there before: We have some really nice code on our hands, so let’s just whip up a quick GUI and get it out there. However, your users will recognize these UIs for what they actually are: Laziness and, ultimately, an expression of utter contempt for your users’ time.

Every minute you put into polishing the usability of your UI will be rewarded manifold — every time a user accomplishes his task faster and with less hickups. And please don’t stop there: Streamline the entire experience, from the installation procedure down to those error dialogs you think they’ll never see¹.

True, it’s a lot of hard work, and it can really get on your nerves. But if you truly want to deliver great software, there’s really no way around it. And one thing is for sure: Your users will notice.

1. Here’s a hint: You are most certainly wrong…

For every entrepreneur, there is something which is extremely important: The elevator pitch. The scenario is that you find yourself in an elevator with a well-known investor. How do you best use the little time you have on the elevator, to explain what you do to the investor, and hopefully make him interested in your project?¹

So, I thought that’s also a nice challenge for the blog: Explain process mining in as few words as possible.

It works best using an example. Take a look at the following log from the ProM tutorial below. This log fragment contains information about four process instances (cases 1-4) in a process that handles traffic fines. Each row corresponds to an event that happened for one of these cases (Case ID column) and states which activity was performed (Task Name column), who performed it (Originator column), and when it happened (Timestamp column).

Now, the core idea of process mining is to use these log data to automatically discover a process model that shows the flow of the traffic fine process. The resulting model is depicted below. It is read from left to right and shows that after a fine is entered in the system, the bill is sent to the driver. If the driver does not pay the bill, a reminder is sent (potentially multiple times). When the bill is paid, the case is archived.

The shown example is very simple, but business processes are often really complex because people in different departments handle large numbers of cases throughout various steps and IT- systems. So, obtaining an overview about the actual process is difficult.

In addition to automatically discovering the actual process flow, you can also visualize the handover of work among people in the organization, analyze the timing behavior of the process and highlight bottlenecks, deviations from intended process flows, and so on.

I think that works well as an elevator pitch for process mining². So, as a challenge, why not try this: In the comments, elevator-pitch what you do in as few words as possible, so that other readers may want to get in touch!

And of course, if you’d like me to give you the Fluxicon elevator pitch, please let me know

1. By the way, this is also a very helpful concept when you are giving a presentation — whether you present at an academic conference, to your colleagues, or to a customer: Try to elevator-pitch your message in the beginning, in one minute or less. You will be amazed at how more engaged your audience will be!
2. That is, given a high enough building, of course…