What is Usability Testing?

A researcher observes users who perform certain tasks on a product or service. The goal is to identify potential problems.

Moderated

When this research is executed face-to-face and researcher and participant are in the same room (or in the same call) we call it moderated: the participant and the researcher interact directly with each other.

• The advantage of moderated calls is that the research can act based on the observations and inputs and ask in-depth questions where needed
• The disadvantage is a more complex scheduling procedure as time and location must be agreed between researcher and user

Unmoderated

The user executes the tasks on his own. There is no direct interaction with a researcher.

• The advantage of an unmoderated testing is that the user can choose time and place on their own. This can be useful if you do research with participants that are hard to access (different time zones, professions such as medical doctors / nurses).
• As the researcher doesn't need to be present it is less time consuming and you can include a larger sample in your research.
• The disadvantage is loss of control on the research side as you cannot ask follow-up questions when needed. You also miss in-depth-insights and reasoning why something is not working.

When to do usability testing?

This research method is helpful at any stage in the product or service development process. You can use it to test your assumption early on with a prototype, to get early feedback from first users (also called: Friends and Family testing) or at any other point where you want to check if and how your solution work for your users.

As it is time consuming, usually a smaller group of users (between five and ten) is approached. Therefore the results are not statistically significant but you get a deep understanding on the usage and interaction with your solution.

If you do remote moderated / unmoderated sessions and you have the possibility to record it, usability testings are an excellent method to provide the whole development team and stakeholders with direct customer feedback.

Challenges

• Usability testings are time consuming (planning, scheduling, execution, analysis, documentation). Especially scheduling can take quite some time.
• There might be additional travel expenses if you shadow activities on-site in the natural environment.
• Different participants might execute the same tasks in very different ways. It is important to record differences and adjust the sample size based on those observations. As a rule of thumb a research can be considered as done once you get foreseeable results.

How to do it?

Planning

As for any research you start with the research plan:

• Research question (objective) and expected outcomes
• Participants needed for this research
• Stakeholders
• Research methodology
• Timeline and scheduling plan
• Location
• Pilot testing
• Budget, risk and challenges
• Research team

Observers

If observers will be present that have never participated in your research make sure they know in detail how the session will be run.

I always share a guide to participate in fieldwork and inform them how they can pass over questions to me during the interview. In the worst case you have observers that start asking questions or give answers where you want to understand what is not understandable to the participant. This kind of interactions can kill a session and render the insights useless.

Usability testing script

To make your research comparable you create s script with the exact wording of the tasks you want to execute. It is important that the scenarios are not changing and stay comparable with each other during a research.

Sometimes, it makes sense to do this research in phases. That means, you keep the same structure for the first phase but you improve for example your prototype thereafter and test the improved version in phase two. This gives you the opportunity to not only learn about the status quo but also about improvements based on your insights.

For a usability testing it is recommended to prepare the questions upfront (structured session). But it can also make sense to keep it a little bit more open and adapt your questions based on the participant (semi-structured). The important thing is to define the key questions in that case as you need to translate qualitative to quantitative data. If the order is not so important you can also consider a question map instead of an script.

Scheduling

Be sure to keep as a minimum 25–30 minutes between interviews. You will need it to make additional notes and have a short break between sessions. If you do remote sessions plan longer breaks to accommodate for technical problems.

Script for a moderated session

Greetings

This is also where you set the stage emotionally.

We are very glad that you agreed to provide us feedback…

Expectation management

• Explain briefly why are you doing this research.
• Provide some reasons why you invited this participant in particular.
• Inform how the session will be run and how long it will take. If you need users to share their screen or even open their mailbox make sure to have it mentioned upfront.

We are having this call today to get early feedback on a new feature we have planned for our application…  

With those improvements you will be able to accomplish…

We have invited you because you have requested a similar feature in the past and we would like to understand how this feature will help you in your daily work.

Confidentiality agreement

In case it is needed to sign a non-disclosure agreement you can either do this upfront to the session or make it part of the setup phase. In my experience participants are also happy to hear that we take care about their data and they also want to know with whom this data is shared. Therefore, don't be afraid and highlight also the benefits the participants get out of this agreement as this works mutually.

To help us with our notetaking I kindly ask you if we can record this call…

The recording will only be used for transcription and deleted afterwards…

This is also part of our confidentiallity agreement. I will put the document in the chat. 

Basically, we are sharing new feature ideas with you, and you agree that you will keep this information confidential.

On the other side, we confirm that we keep your information confidential. We will anonymize your inputs before we share our aggregated learnings with other stakeholders…

Please let us know if you have any questions before we start…

Warm-up question

Include a short pre-task question to get a better understanding of the person you are talking to and give the participant a possibility to set the stage from their side. This is also relevant data to confirm assumed personas.

Topics you might ask:

• Job title and area of work"
• Devices and digital tools used to accomplish something
• Devices and tools used in different environments (at home, at work)
• What kind of tasks that are relevant for your service / product are done digitally or non-digitally?

Please tell us a little bit about your tasks and your role within the company: / with regard to…

Which of the following tasks, if any, do you prefer to do online:

Research tasks

This is the main part of the research:

• You give open-ended tasks with only the necessary minimum information
• You observe the user how they accomplish the task
• In order to understand how users approach the solution you ask them to think aloud. The participant is required to express verbally what she is thinking. It helps, to provide the participant with a very simple example what you expect, especially if they don't speak the language natively in which the interview is conducted.
• In case of questions by the participant don't provide an immediate answer. Rather ask them what they would do in the situation when this question arises and they are alone. If they ask about a certain functionality that is not part of the application (or prototype) you can ask what they would expect. Sometimes, it doesn't add anything to the research to give a lengthy answer and you just wanted to understand if something was understood by the participant. In that case you might also confirm that their reply was correct (even if it wasn't) as you got your insight and you don't loose time.
• If you use a prototype make sure the first page of the prototype doesn't contain relevant research data yet and you can use it to explain what you will expect from the participant.
• In my experience it helps to emphasize once more that you are not testing the customer and that is rather the other way around. You want to understand if the assumed solution works for your user and if you got it right.
• Don't be afraid of pauses. It can happen that participants need some time. Don't try to overcome the silence (as it might feel awkward). The participant will respond and most of the time those pauses trigger additional information that was not expected. If you break the pause you might prevent the participant from documenting something that is on her mind.

I will send you a link to the prototype in the chat and kindly ask you to open that link and share your screen with us.

What you see in front of you is a prototype of …

Prototype means that not all functions that you see are clickable. 

In this prototype you are an international company with the name ABC. Your name is…

I will give you a couple of small tasks now.

Please fulfill the process each time until you think, you are done. 

Please inform us, when you are done. 

We kindly ask you to think aloud and tell us what you see.

For example: “I think this button will do this or that”. In case there are issues with the prototype itself I will support you.

Important: We are not testing you. We are here to learn from you. We want to understand if our solution proposal meets your demands.

Please do / check / find…

Tell me what you see and what you can do here…

What do you exect will happen if…

How did you feel / find doing…

Closing

Is there anything else you would like to add that can help building this product / service?

How would you descripe your overall experience with this product / service?

Thank you for your time and valuable input.

If time allows you can also include a System Usability Scale (SUS) or Net Promoter Score (NPS) to measure satisfaction after they have accomplished the tasks.

Analysis

It might make sense to map your findings along the User Journey of the tasks if you have multiple observers and researches. Especially to share observations and not only information that was verbalized. Easiest way is to print out the screens and ask everyone to add their comments to collect findings.

If it was possible to record the session it helps to re-watch and transcribe the sessions. A lot of insights you gain when you go through the whole process once more, especially when you do a transcription. As transcribing can be very time consuming there are also excellent free AI driven multilingual tools that can be used for this purpose such as WhisperScript by Open AI (GitHub: Whisper).

Documentation

When it comes to documentation (and later presentations) simple spreadsheets have proven to be an ideal method to map qualitative data to make it measurable. To protect participants you pseudonymize the data and don't use real names or other personal information in your documentation.

Usually, I create a Microsoft Excel sheet where I document insights on the horizontal axis for each participant (and target segment / persona). Demographics are recorded as needed.

On the vertical axis I group and document each task and what was said and observed by the observers. This data is kept as neutral as possible (for a human :-) This gives a good overview what and how often something was mentioned by participants.

On the bottom of the vertical axis I add actionable insights and recommendations that feed into presentations.

Outcomes

As deliverable you can expect deep insights on how your solution works for users who are performing the specified tasks. You will understand what works and what doesn't work and might need further research.

In a moderated session you will usually discover 80% of usability problems even it a non-representative sample of only ten participants. In case you are flexible you run the research until you can foresee the results / behavior. If you go for an unmoderated testing you aim for a high number of participants to get a representative sample as you lack the possibilty of an in-depth-conversation.