While user testing is generally driven by the designer's desire to validate with users that a design makes sense, statistical analyses of where users are dropping out can help identify trouble spots to focus on.

The first step in user testing is to determine which users are appropriate to participate. In order to match the persona as closely as possible, we usually use a screener. The survey must be carefully constructed to not give away what answers we expect or desire, since everyone filling it out knows there's a gift card waiting for them at the end of an hour of just discussing.

User testing aims to create a situation like a real context of use. The key to this is to avoid giving the user any hints or information that they would not receive from the interface, associated marketing materials, or the person who introduces them to the software (if any). In practice, this means avoiding referring to the interface itself as much as possible, and asking open-ended questions.

This informal user testing script shows how to do this by focusing on user goals, not on tasks.

Next, we develop a specific protocol for how to carry out the test. This helps standardize the tests so that results are more controlled across users, testers, and other factors. Due to the inherent fuzziness of such tests, I view the protocol as a way to ensure that information about how the interface works is revealed to users in the same order each time (and not before we observe the user's assumptions) - not to specify the tester's every word and gesture.

Due to the history of software development and the low incidence of great user experience design, most users are used to thinking of themselves as dumb and unable to properly understand software. In order to help users skip apologizing for their misunderstandings, and in order to make them more confident about expressing their thoughts and assumptions, I like to give an introduction letting them know that if it's difficult or confusing, it's not their fault.

At BitTorrent, we opted to avoid filming users' faces to respect their privacy and make it easier to share data amongst our teams.

During the tests, another team member usually sits in to take notes. It's faster to aggregate observations over multiple users in text format, compared to watching videos multiple times. This is also a great opportunity for engineers and other non-designer team members to participate and see how real users think and behave.

To share knowledge with the rest of the implementation team, I like to create several short clips like this one. This insight is illustrative of user behavior which the team wasn't aware of before, and led to a major refactoring of the interface.

In the clip, we see the user fundamentally misunderstand the flow of the dialog. After pasting a key, she overwrites it by pressing "generate", losing the connection she was attempting to create. This throws off her understanding of the second field on the screen, also.

A testing analysis and summary document is a great way to capture a bunch of details that aren't worth the time it takes to create and watch video clips.