Process Mining Challenges

Process Mining 2024

A big part of this year’s Process Mining Camp was to collect and discuss the challenges we all experience when applying process mining. Some challenges are generic. Others depend on your organization or emerge from your current skill set.

Traditionally, there has been a division between data-oriented and process-oriented disciplines.

For example, data science and BI teams have a lot of data and statistics skills. But they are not responsible for the process and often don’t understand it. On the other hand, business process experts recognize things like rework when they see them, but they are not used to leveraging the data that is available. The same applies to auditors who know how to watch out for various risks in the process but might not yet be able to handle large data sets.

In fact, process mining bridges this gap by marrying process and data-driven skills. And you, the process miners, are doing this! By using data to understand processes.

However, you can also look at your own skill gaps to improve further. For example, if your background is in BI or data science, we recommend familiarizing yourself with methodologies like process analysis or lean six sigma, and picking up skills like change management. Or are you more coming from the process analysis or audit side? Then, it can be really valuable to develop some additional data skills.

Data Challenges

Process Mining 2024

In the Data Skills workshop at this year’s camp we did just that. Together, we transformed and improved a data set step-by-step. We also discussed best practices for the data challenges that we collected the day before.

Overall, the most frequent data challenges that were mentioned fall into the following four groups.

DC1: No data or no access to the data

Some data is not captured. After all, most IT systems did not think about process mining when developing their data structures. Essential data fields might be missing completely (for example, instead of the history of status changes only the last status is captured) or partially (for example, there might not be a single identifier from beginning to end if multiple systems are involved).

Sometimes, getting access to the data can be difficult. For example, an external provider might manage the IT system, and data access is not included in your contract. Your own IT team might have no capacity to help you due to competing priorities. Furthermore, legacy systems and a lack of knowledge of all processes can make it difficult to know where to look for data in the first place.

Best practices

  • Know the data requirements for process mining.

  • To get access to the data, involve & inspire higher management by sharing process mining success stories that illustrate the benefits in monetary terms.

  • Develop a good relationship with data owners.

  • Involve data experts early in your process mining project and try to make them enthusiastic about what you are trying to do.

  • Work together with the data science team to extract and make it suitable for process mining.

  • Team up with your internal audit team as auditors can make their data extraction a priority.

  • Renegociate contracts with external providers to make sure they provide you with your data.

  • Make sure to include data requirements for new IT systems in the future.

DC2: Lack of data skills

Even if you have data, it might not be in the right format. Furthermore, it can be challenging to extract the data due to its volume or because of your data skills. You might need to merge data from different sources. Finally, the scope of your process determines the scope of your data, and you need to have an understanding of how the target data should look after your preparation.

Best practices

  • Focus first on the core data and extend it further in a second step.

  • Start with a small sample and scale up your data size later.

  • Learn to translate the process questions in the right data combination.

  • Avoid repeated copy and paste in Excel and develop data skills, making the data extraction and preparation repeatable.

  • Keep in mind that developing an event log is an iterative process (don’t try to do it right the first time).

DC3: Data quality problems

As with any data analysis technique, data quality problems can make your analysis harder or impossible. Garbage in = garbage out. Especially manually collected data is often unreliable. Furthermore, a lack of precision in the timestamps can create wrong orderings in the events.

Best practices

  • Don’t skip the data validation session.

  • Use the data quality checklist to spot (and clean) common data quality problems in process mining.

  • Understand how the data is collected during the process execution to foresee mismatches between data and reality.

  • Use the data that you have rather than waiting for the perfect data.

  • Check how much of your data remains after the cleaning step to see how representative it still is.

  • Make event data quality a priority in your data architecture.

  • Improve data quality if you can influence the data collection in the future.

DC4: Security and privacy concerns

If process mining is still new in your organization, you may encounter a reluctance to share data. Reasons can include security and privacy concerns and a fear of misusing the data. The rules regarding employee and customer data vary across different countries/entities.

Best practices

  • Make sure you consider privacy, security, and ethics questions from the start.

  • Connect to your legal team to understand which rules apply to you (e.g., GDPR).

  • If you are in Germany or another country that prioritizes workers’ rights, involve the workers’ council at the beginning of the project.

  • Minimize data to what you really need.

  • Address privacy concerns and create an ethical charter.

  • Foster a collaborative culture that helps people to speak the truth about their process.

  • Data ownership and access should be a priority. Create a data access policy and a privacy guideline for process mining, and make event data part of your data governance.

  • Anonymize data to hide sensitive information if needed.

  • Be aware that even anonymized data may be traced back to individuals (adjust your level of protection to the type of data and the situation).

  • Instead of cloud solutions, you can use a local process mining tool like Disco or ProM, where your data is not uploaded but remains in your organization.

Process Challenges

Process Mining 2024

In the analysis workshop, we learned how to separate incomplete cases and how to identify the standard process and its deviations based on patterns in the data.

Overall, the most frequent process challenges that were mentioned fall into the following four groups.

PC1: Dealing with complexity

Certain processes, such as customer journeys or healthcare processes, are particular complex. For example, clickstream sequences on websites and patient pathways are very individualized. So, they contain a lot of variants. But also other processes seem quite chaotic when you first look at their complete picture in the process mining tool.

Best practices

  • Remove incomplete cases, explore different perspectives, and separate data issues from process issues before you start with the analysis of your process.

  • To find structure in the chaos, apply simplification techniques.

  • Look at individual cases and variants to see example scenarios of the process.

  • Identify events that indicate the beginning and end of the process as well as important steps in the middle. Then, use the Milestone simplification method to visualize the process just based on these milestones (hiding the rest) to get an overview.

  • If you have a reference process (it can be just a whiteboard drawing from a subject matter expert), find those referenced milestones in the data.

PC2: Lack of process mining skills

Learning how to read the process map, how the sliders work, and which filtering mode does what is essential. Luckily, working with the process mining tool can be practiced, and you will gain more experience over time. If you are unsure, you can always look at the user guide or follow a training.

Best practices

  • Always double-check whether your process mining results are in line with your expectations. For example, if you filter completed cases based on an endpoint and end up with just 20% of the cases from the initial data set, inverse the endpoint selection to inspect the incomplete cases to see if you have missed valid endpoints.

  • To find specific activities or events, you can search them in the process map or the cases view.

  • Click on paths in the process maps that seem odd to you and look at example cases.

  • Use the synchronized animation to compare process behavior for different segments.

  • Use TimeWarp if your SLAs are based on working time rather than calendar time.

  • Make copies when you explore different analyses to preserve your current views.

  • Keep your workspace organized by renaming and re-ordering data sets.

  • Use the recipes to save and reuse filter combinations.

PC3: Lack of process mining methodology

In addition to learning how to use the process mining tool, you also need a plan for how to set up your project. Which process do you choose, and how big should you make it? We are currently putting together the essential ingredients in this 12-step project guide.

Best practices

  • Find multiple processes to mine and eyeball the data for the low-hanging fruits.

  • Take a small process to prove value fast.

  • Define an objective (important for data selection and to avoid getting lost in the data).

  • Look at the process with the most pain points or the largest business value.

  • Embed your process mining practice into your existing way of working.

  • To ask the right questions, identify the “What keeps you up at night?” and ask “If you had a magic wand and could make a wish for this process, what would it be?”.

  • Keep in mind that the questions need to be made more precise once you start answering them with the data.

  • If there are no targets for a process yet, you can use process mining for an initial measurement and define the ambition for, e.g., throughput times or other SLAs.

PC4: Missing domain knowledge

You can do a lot of exploration by just looking at the patterns in the process and inferring the most likely meaning from the names of the event labels. But ultimately, you will hit a wall and need domain knowledge of the process behind it. If you are not a subject matter expert yourself, you will need access to someone who can answer your questions.

Best practices

  • Involve stakeholders early.

  • Have a close connection with the business and select projects based on sponsorship.

  • Try to understand the process yourself before consulting the expert. Keep a list of all the questions that you encounter along the way.

  • Create a multi-disciplinary team (see the skills and roles that are needed for your process mining project here).

  • Understand the process behind the data. For example, follow the process manager for a day (shadowing) to see a “day in life.”

  • Don’t be afraid to ask “stupid” questions. Your outside perspective can be precious!

Organizational Challenges

Process Mining 2024

In the third and last workshop, we examined the organizational challenges and brainstormed best practices for addressing them.

The most frequent process challenges that were mentioned fall into the following five groups.

OC1: Stakeholder management and sponsorship

Just as important as data availability is good support from the team responsible for the process. Who do you need to involve? How do you ask the right questions? And how do you manage the right stakeholders?

Best practices

  • Identify a process owner, engage in dialog, and ask “Are you aware how the reality deviates from the expected process?”

  • Quick workshop/demo to show, not tell, advantages.

  • Discuss with auditors about the risks to look for.

  • Derive “right” stakeholders from the use case.

  • Use classical stakeholder map to classify stakeholders.

  • Involve: Budget holder, CFO/controller, IT manager, process owner (see also skills and roles).

  • Engage with stakeholders early and make them part of the project from initiation. Communication: Explain the why, how, when, and what.

  • If you sense a lack of trust, have an honest dialogue and answer questions that those hesitants might have. Show how it works, and be transparent and honest!

OC2: Business case & value

If your company is not yet familiar with process mining, they might not see the benefits (yet). Finding the proper business case can be challenging and is often a chicken-and-egg problem: How do you know how much value you can realize before looking at the data?

Best practices

  • Use experiences from the community to show process mining examples that are relevant to your organization.

  • Tailor the business case to your use case. For example, auditors can save time on audits or avoid risks or fraud cases in the future.

  • Identify company-/business issues (where KPIs are not met) and ask customers for the biggest pain.

  • Choose the process where data is easily obtained to show the advantages quickly.

  • Balance effort & reward when evaluating (accounting for the skill of the organization).

  • Close the loop after your project: Measure the impact of the change to demonstrate the value you have delivered.

  • To sustain improvement, define performance metrics and build a program to track them regularly. This helps prevent the process from falling back into old patterns.

  • While privacy is not easily quantifiable in business value, you can include it in your business case as an avoided risk.

OC3: Adopt methodology (and drive change)

Having the insights from your process mining analysis does not do anything yet! You need to change the process to realize the benefits. We discussed how to manage this change and how to expand after your initial project.

Best practices

  • “Start small” & be clear about the goals & impact. Have regular follow-ups & checks.

  • Be open about the change and leverage the change management professionals in your organization.

  • Demonstrate the value of process mining (before vs after).

  • Standardize your process mining practice before expanding.

  • Appoint process mining champions per business unit.

  • Team up with other groups, such as the BPM group, lean six sigma experts, or the data science team to develop an integrated approach and join forces.

OC4: Connecting to strategy

Connecting your process mining initiatives with your company strategy is a good way to align them with your organization’s goals. For example, process mining can support the digital transformation journey that many companies are still making.

Best practices

  • Create a clear and concrete use case that is aligned with your company’s goals.

  • Thoroughly understand the company strategy as an analyst.

  • Link your project to cost savings, efficiency, and operational excellence.

  • Link strategy with the business case.

  • Introduce and use OKR to break down company strategy (objective) into process mining tasks (key requests).

OC5: Develop a process mining capacity within the organization

How do you take the next steps after your first process mining projects? You want to skill up the organization, get new assignments, stay up to date, and build a community that shares what they have learned.

Best practices

  • To build up expertise, start small and perform multiple pilot projects.

  • Celebrate successes and use your successful projects to market the added value of process mining and get more people interested.

  • Offer awareness training for leadership positions. Make people aware of the possibilities in all layers (bottom-up/top-down).

  • Go to Process Mining Camp :)

  • If you don’t have enough internal resources, hire a consultant to help you with your first projects. Make sure you can take over what they did for you after the project.

  • Participate in hands-on process mining training.

  • Schedule meetups with peers. Document knowledge & best practices.

  • Promote process mining by nominating process mining champions.

  • Start an innovation lab.

  • Read and research about what is new in the field.

  • Sign up for newsletter.

Process Mining Café this Wednesday

Process Mining 2024

In the upcoming Process Mining Café tomorrow, this Wednesday, 10 July, at 15:00 CEST (Check your timezone here), we will review what we learned from this year’s Process Mining Camp. Come join us!

As always, there is no registration required. Simply point your browser to fluxicon.com/cafe when it is time. You can watch the café and share your thoughts while we are on the air, right there on the café website.

If you want to be reminded, you can:

— See you at the café!

Anne Rozinat

Anne Rozinat

Market, customers, and everything else

Anne knows how to mine a process like no other. She has conducted a large number of process mining projects with companies such as Philips Healthcare, Océ, ASML, Philips Consumer Lifestyle, and many others.