User testing is a great tool for understanding how users interact with a system and is one of the cornerstones of a user centered design process. In this article we are going to review some of the biases that can impact your findings, and will break user testing down into its component parts, analysing each in turn. By being aware of these biases we can identify when they occur and look to limit their effects. Photo credit to: Emma b
For this article we will break the environment down into two parts; the physical environment and the social environment. First lets review the physical environment. The testing environment is one variable that can have an affect on user behaviour. To minimise these effects look to test in an environment similar to that which the user would typically be using the system in. The second aspect of the environment is the social setting. Ordinarily testing will require a facilitator sitting in a room with the user and presenting them with various tasks, there may also be people observing remotely or a via two way mirror. We will look at the effects this type of social environment can have on user behaviour. One bias that research has identified is that when participants know they are being observed they are likely to be much more vigilant, spending more time reading instructions and completing tasks, than they would otherwise. This is known as the Hawthorne effect. There has been much discussion around the causality of this effect and whether it is purely the act of observation that causes the change in behaviour. From personal experience I have observed users persevering with tasks much longer in a formal user testing scenario than when approached in a more informal way. So whatever the reason for the change in behaviour, it is important to understand that user behaviour in testing will not always exactly mirror behaviour in the real situation.
The selection process, and getting the right participants, is an important aspect to getting real results from testing. If you haven’t recruited participants that truly represent your real end users then you can generate a selection bias. This bias will be accentuated the more unique the user base and/or the more novel the task being tested. For example there would be little benefit testing an ultrasound system with users who have no medical training. Although general usability issues are likely to be experienced by most users, specific behaviours can differ greatly dependant on many factors such as: users previous experience, skills, age etc.
When running a test, there will be at least one facilitator in the environment with the participants. The facilitator may also have defined the testing tasks and in some cases had a hand in designing the system being tested. The facilitator of the test is likely to have preconceived ideas of what they think is right or wrong with the system they are testing. This can be beneficial as they have an understanding what areas to focus on but this can lead to them being biased towards the behaviour that matches their assumptions. This issue can be exacerbated when facilitators test their own work, as they are likely to have a stronger bond with the system they are testing. As a facilitator you should try and stay objective, keeping an open mind to the users behaviour. Remember participants don’t just pick up on what the facilitator says, they can also recognise other cues such as the stressing of certain words, phrasing of questions and even body language.
The tasks that are chosen for the testing probably have the biggest impact on user behaviour, as the user will respond and try to complete the given tasks. Selecting the right tasks and scenarios is therefore essential to identifying usability issues. Lets move on to look at the various biases that can have an effect on the outcome of the tasks and scenarios you set your users:
It has been found that participants emphasise issues depending on when they happened. Participants are more likely to give emphasis to events that have most recently occurred and also events that happen at the beginning of a test. Remember, first impressions count, but so do last impressions! In psychological experiments a for these biases is to vary the order of the tasks you give the participant. However this may not always be possible in user testing due to linear and interlinked nature of certain tasks. For example: It wouldn’t make sense to test the confirmation page of a checkout process before first going through the checkout process.
A selection of different biases that can be triggered by the tasks, or the facilitator. Examples of these include: Providing the user with supplementary information: This can occur when a task gives the user information they wouldn’t necessarily have available were they completing the task on their own. An example could be when tasked with finding a call to action on screen, If the task includes a specific term then users may scan the page until they find the term, but would they have been looking for that term without prompting? Be aware of how users respond to the information they are given and try not to provide extra information that the user wouldn’t have available if they were completing the task on their own. The phrasing of questions can introduce further variables, try to phrase questions neutrally so not to lead the user towards a specific answer.
If you ask a user to complete a task they will assume there is a way to complete it, and will therefore invest more effort into finding a solution. One way to potentially combat this would be to add tasks that cannot be completed, although you would have to consider the ethical implications of such an approach, also this would likely introduce further behavioural changes later in the test.
Participants are usually given a cash incentive for taking part in testing. The exact amount will depend on many factors including the duration of the testing and the availability/expertise of the participants. Offer too little and it can be hard to recruit, or you may have participants not show up. On the other end of the spectrum, there have been some reports that offering too high an incentive can lead to participants trying too hard to earn it.
User testing is an invaluable tool to gaining insight into user behaviour. By being aware of the biases that can affect the validity of your results, you can look at ways to minimise their effects, taking them into consideration when analysing your data.
Arrigo Sacchi has always been a bit of a footballing legend to me and I was recently reading an article about his footballing principles and experiences. A lot of the methods he applies to football coaching feel as though they can be applied in other areas such as user experience design. In this post we will introduce Sacchis principles and draw parallels to the realm of user experience design.
For those of you who don’t know Arrigo Sacchi is a highly successful football coach. Back in the late 80s he won back to back European cups with AC Milan. Starting managing his local team, he worked his way up the leagues and had spells with the Italian national team, Parma and both Madrid teams.
So let’s take a look at what coaching principles Sacchi advocated during his career. His most important concept is probably that the whole is greater than the individual. He strongly believed that teams shouldn’t be built around individual and that creating unity within the team is essential. We will look at practical examples of these footballing principles and how they can be applied to user experience design.
One of the experiences Sacchi talks about occurred during his time at Real Madrid was the “proliferation of specialists”, this was during the galacticos era, when Real were focussed on signing superstar players. He goes on to explain how the individuals became more important than the collective. You had players such as Zidane and Figo who weren’t known for their tracking back. This led to a reactive system of having to employ a defensive midfielder to cover for their lack of defensive capability. His belief was that this policy was reactionary and didn’t multiply the players qualities. His belief is that the tactics employed should exponentially multiply the players qualities.
This concept can be seen in the different facets of design. For example take the aesthetic design of an application, this should be in harmony with the applications usability. If you have an application that is designed primarily from an aesthetic perspective then the usability considered retrospectively, you are taking a reactionary approach to design and as Sacchi found, you aren’t multiplying the qualities of the design but limiting them. If you take all the elements of a design, they should be working towards a clear goal. An underpinning of sound usability principles can work in cooperation with the aesthetic design to communicate the desired concepts to the user.
Another comparison that can be made here is that of marquee signings. We have clubs that have become play things of their rich owners, who want to make a statement and what better way to make a statement than signing a world renowned player? This approach is at odds with the tactical holistic approach Sacchi prescribes.
We can see similar effects created in the realm of design, when design trends are blindly followed without understanding whether they are being used in the appropriate context. The cause in this case could be the client/designer has seen the latest trend and wants to employ it (parallax, flat design etc). The principles/players in isolation may well have positive attributes, but you need to consider their attributes within context of the specific application/team.
By looking to other areas for validation you can often find similar principles that can be applied within an unrelated field. These parallels can provide fresh impetus to your design thinking and help refresh principles you may have forgotten.
Special thanks to Jonathan Wilson for his articles in the Guardian including: Big name transfer comfort food for clubs