This is Flux Capacitor, the company weblog of Fluxicon.
You can find more articles here.

You should follow us on Twitter here.

Process Mining for Usability Tests 5

You might have noticed: Products—and especially consumer electronics—are becoming more and more complex. As a result, people are not always able to deal with these complexities and usability becomes a distinguishing factor in brand reputation and customer satisfaction.

Process mining is a new technology that makes invisible process flows visible by analyzing existing log data in a bottom-up manner. Earlier, we have seen how process mining can be applied to the test process of ASML and an HR process. But can process mining also help to improve the usability of products?

Usability Testing

In the context of the Master’s project of Pieter Hofstra, together with Jeroen Keijzers, Yuan Lu, and Ton Weijters, we investigated the applicability of process mining for usability tests1.

Usability tests, for example first-use consumer tests, can help to get early feedback from the field while there is still time to adapt the product before releasing it to the market. Traditional usability measures include mostly static information, for example:

The results typically do not reflect the temporal aspects of the test data. So, in this project, we looked at how process mining can be used to get insights into the actual user behavior.

Experimental Setup

For the project, a group of 29 Dutch volunteers (from age of 22 to 66) participated in a usability test for a new television. 19 participants were male and 10 were female. The usability test took place in a simulated living room to make them feel at home as much as possible.

The participants were asked to complete the following three tasks:

  1. Channel selection. After installation of the television, channel RTL 7 has been automatically programmed on channel 25. The participants were asked to put RTL 7 on channel 7.
  2. Dual screen. The ‘Dual screen’ function is innovative in comparison with previous versions of the product. It is one of the features promoted by marketing to sell the product. The participants were asked to watch the channels NEDERLAND 2 and NET 5 simultaneously.
  3. Digital picture. Another function that is new in comparison with previous versions of the television is the ‘Digital picture’ function, which allows to view digital pictures from a USB stick on the television screen.

The “correct procedures” to solve each of these three usability tasks is shown in the picture below. Further down, you can see the process models of the actual user behavior for the middle task (Dual Screen).

(Process models of the optimal user behavior for solving each of the three television usability tasks.)

The entire experiment took about 15-40 minutes per person, depending on the participants’ performance. During the experiment, the participants and the television screen were captured on a video camera. From these video recordings, an event log of the actual usage behavior was created semi-automatically. You can find more details about the event log creation in Pieter’s Master thesis.

Results

One of the goals of the study was to assess the effect of a consumer’s (product) knowledge on usability. People with high product knowledge are assumed to be more familiar with the product, and to have more experience in using it.

The test participants were divided into ‘High Knowledge’ (13 people) and ‘Low Knowledge’ (16 people) groups based on their knowledge ratings in the questionnaire2. Process mining was done on the usage logs of these two groups separately.

(Process model discovered for the ‘High Knowledge’ group performing the Dual Screen task. The numbers and coloring indicate frequencies.)

Look at the behavior of the ‘High Knowledge’ group performing the Dual Screen task above. One can nicely see the paths that were taken from the start to the end of the task.

(Process model discovered for the ‘Low Knowledge’ group performing the Dual Screen task. Compared to the ‘High Knowledge’ group this model is more complex showing more variability among the group of users.)

In the model above it is very visible that the people in the ‘Low Knowledge’ were even further away from the optimal solution. One person even had to give up.

I find this is a nice example that through the visualization of actual user behavior it is possible to reveal usage patterns, which provide both qualitative and quantitative feedback.

Not only for Consumers

Also in a business context usage behavior can be crucial. For example, in call centers there is an increasing use of analytics for operational performance management. Agents often have to switch between 4-6 different applications (Siebel, SAP, etc.) while handling a call. Desktop analysis tools can analyze the key strokes of an agent and the resulting insight can be used to build an abstracting layer on top of the actual applications that matches the typical call flow.

Do you see other examples where understanding user behavior is important? Let us know in the comments.


  1. See details in P.P.H.J. Hofstra. Analysing the Effect of Consumer Knowledge on Product Usability Using Process Mining Techniques. Master’s thesis, Eindhoven University of Technology, Department of Industrial Design, Eindhoven, The Netherlands, 2009.  
  2. The knowledge level of each participant was measured by asking them questions assessing their “familiarity” and “expertise” with televisions and computers.
    For example, participants had to react to statements such as “Compared to most other people, I know less about televisions/computers” (familiarity) and “I usually talk with friends and colleagues about new developments regarding televisions/computers” (expertise) on a 5-point Likert scale (ranging from total disagreement to total agreement). For more details see Pieter’s thesis.  

Comments (5)

Another impressive case study which demostrates where process mining can be applied to and what you can gain with it.
Thanks.

Thank you! Feedback like yours helps us to understand what kind of articles people would like to read on our blog. So, please keep letting us know what you find interesting and what not so much.

[…] this is a blog entry about a usability testing case study, but really the part I liked best about the […]

Great article!

I found it to be a great case study for PM:)

Curious to know what the master thesis author does now with Process mining?

THanks!

Thanks, Anton! I agree.

I don’t think that Pieter is actively working with process mining at the moment.

But of course we are on it 🙂


Leave a reply