You Need To Be Careful How You Measure Your Processes

Spurious Correlations

Everyone knows the saying that you can lie with statistics. One of the themes around the responsible use of statistics is that correlation does not imply causation. For example, the above graph from the Spurious correlations book illustrates how ridiculously unrelated things can be correlated.

Another problem that is less frequently mentioned is that you get what you measure. This is the inverse take on the popular “you can’t know what you don’t measure” and hints at the fact that the way you measure influences your results.

To understand the you get what you measure problem take a look at the following process from a customer service department at a large Internet company. It shows the contact moments that customers had with the support team over various channels (phone, web, email, chat).

The key metric that was used in the team to monitor the service performance was the First Contact Resolution Rate (FCR). The FCR measures how many of the customer problems the team could solve within the first contact with the customer, for example, without the customer having to call back again. In the process map below you can see that out of 21,304 inbound calls only 540 resulted in repeat calls. The overall FCR was an impressive 98%.

Customer Service Process with Service Request as Case ID (click to enlarge)

However, the process mining analysis was done based on the Service Request number as a Case ID. The Service Request ID is a unique identifier that is automatically assigned to each new service case by the Siebel CRM system. A deeper analysis revealed that all service requests were closed pretty quickly - typically within up to 3 days.

If the customer did call back after 3 days, a new service request was opened. So, the process above shows the flow of the service requests, but it does not show the real service process the customers went through.

To shift the perspective, the same data was then imported again into Disco. This time, the Customer ID was used as a Case ID. You can see how the process changes if you look at it from this new perspective.

Only 17,065 cases were in reality started by an inbound call. Over 3,000 were actually repeat calls (only counted as new service requests). With this new view the true FCR dropped to 82%.

Customer Service Process with Customer ID as Case ID (click to enlarge)

The customer service example demonstrates how the perspective that you take on the process influences the results. And while Disco allows you take different views on the process very quickly, it is your responsibility as a process mining analyst to make sure you explore these different views and think about how you should look at the process.

The initial, service-request based analysis was being done from the perspective of the measured KPI, which, in fact, may have influenced the behavior of the agents in the call center in the first place: If you are measured based on how few call-backs you get, you are inclined to close those service requests just a little more quickly.

However, from the customer perspective this leads to a worse experience, because they have to repeat all their information details and describe the problem again. It would be better for them if the agent would look up and re-open their case. So, also from a process management perspective you often get what you measure. And if the KPIs that are used to evaluate the performance of the employees do not encourage the right behavior that you want in your process then you are in trouble.

As a process miner you need to be careful to take contextual factors like how people are measured, and what their incentives are, into account when you asses a process in your organization. Otherwise you won’t get the full picture.

Anne Rozinat

Anne Rozinat

Market, customers, and everything else

Anne knows how to mine a process like no other. She has conducted a large number of process mining projects with companies such as Philips Healthcare, Océ, ASML, Philips Consumer Lifestyle, and many others.