There’s a scene in Hot Shots! Part Deux (shut up, we all have our guilty pleasures) where Charlie Sheen’s character (Topper Harley) rescues Rowan Atkinson’s character (Dexter Hayman) from prison. The conversation goes something like this:

Topper: Dexter, I’m here to rescue you.
Dexter: You don’t understand. I can’t walk.
Topper: Why?
Dexter: They tied my shoelaces together.
Topper: Bastards!

The joke is funny, of course, (shut up, it is funny!) because of the ridiculous nature of the claim that tying someone’s shoelaces together can somehow stop them from walking around. We look at the situation from the outside and think they’re idiots – don’t they realize they can just untie Dexter’s shoelaces?

I often think of this scene when I hear designers defend their decisions by insisting that users will “figure it out”. I hear statements like “it’s not our fault that they can’t use this feature”, and I think about users with their shoelaces tied together, unable to move. We look at them with pity in our eyes – if they could only see the obvious and untie the knot, they would have no trouble using the site.

You Are Not The User

But of course, that’s not how it works. We think users are stuck because they aren’t untying their shoelaces, while they’re actually knee deep in the cement of poor usability we put them in. We can make T-Shirts that say “I am not the user” and wear them all day long, but somehow we still manage to find a way to blame them when something goes wrong. Not cool.

We will never be able to design web sites that don’t confuse users unless we observe them using our sites, and fix the issues that uncovers. We cannot think like our users – as designers we are simply too close to the product, and way too proficient in all things web. It reminds me of something Douglas Adams once said:

A common mistake that people make when trying to design something completely foolproof is to underestimate the ingenuity of complete fools.

There is a great post on the Agile Bits blog that talks about the difficulties of designing security systems. In one part they discuss the problem of designing for users and sum up the issue perfectly:

Security systems (well, the good ones anyway) are designed by people who fully understand the reasons behind the rules. The problem is that they try to design things for people like themselves — people who thoroughly understand the reasons. Thus we are left with products that only work well for people who have a deep understanding of the system and its components.

Stepping Into Their Shoes

We have to be able to step out of this cocoon of deep understanding, and the only way to do that is to regularly observe users as they make their way through our applications. Whether you take your laptop to a coffee shop and ask random people to give you a few minutes of their time, or set up full-scale usability tests, the payoff of uncovering usability issues on your application is so worth the time. What’s the upside, you ask? Matt Gemmel sums it up really well:

The biggest (and most lucrative) set of customers is ordinary people, without a computing degree or specialist knowledge. These are people with no interest in specific technologies, but only in how easily they can finish today’s tasks without reading the manual. Apple caters to that market; companies who loudly proclaim their device supports CSS3 and MPEG4 and SDHC don’t even understand that it exists.

If we can get into the heads of those ordinary people who use our products every day we’ll be able to meet their needs so much better. I agree with Jeff Gothelf on this one: test everything, regardless of its polish or fidelity:

Increasing your time with customers throughout the design and build process improves the outcome of your project by continually nudging the interface in a more appropriate direction. As an added side benefit, you also begin to build a user-centric culture within the company if it didn’t already exist – a huge plus.

I’ll end with the words of Jeffrey Zeldman in Style versus design, because it articulates so well why this is such an important issue:

Not enough designers are working in that vast middle ground between eye candy and hardcore usability where most of the web must be built.

Most of all, I worry about web users. Because, after ten-plus years of commercial web development, they still have a tough time finding what they’re looking for, and they still wonder why it’s so damned unpleasant to read text on the web — which is what most of them do when they’re online.

Let’s realize that the problem is a little more complex than untying shoelaces. Better yet, let’s realize it’s our problem if users get stuck, not theirs. And best of all, let’s allow them to help us fix it.

I recently attended a brand presentation where the video below was shown. It’s pretty funny, and also a perfect example of how interactive products and consumer-generated content should fundamentally change our traditional views of customer loyalty. Loyalty in our current environment is fostered through repeated great (user) experiences, not just through advertising and coupons.

But even though I like the general point the video is trying to make, I think it stops a little short of the real issue. It is saying that we should listen to our customers better. But that’s not enough — we need to understand customers in ways they don’t even understand themselves, and then build experiences that meet unmet (and sometimes unconscious) needs through repeated, positive experiences that deepen the customer-company relationship.

Uncovering these needs happens not just through “Voice of the Customer” research programs, but also through more contextual research efforts like ethnography and contextual inquiries (combined with validating quantitative research). I believe this is where traditional Market Research programs like NPS (Net Promoter Score) only tell a part of the full brand loyalty story (albeit an important part, for sure).  There is evidence that the tide is turning on this topic as the field of HCI (Human-Computer Interaction) becomes more mainstream and user experience research techniques become more accessible.

There is a powerful synergy in discovering how to deepen true customer loyalty through collaborative efforts between Market Research and User Experience Research, and we need to bring these two disciplines closer together (this view is also very much in line with the thinking described in the excellent Adaptive Path essay The Long Wow).

A recent discussion on a user experience forum I participate in turned to the topic of A/B testing.  I really enjoyed the conversation so I wanted to reiterate some of the points I made, and expand on it a little bit as well.  It’s not my goal to define A/B testing here but to share my opinion on its use.  I believe that even though A/B testing can be extremely valuable to help identify the best iteration of a site or a particular page, it should never be used in isolation.

Since A/B testing is relatively cheap to do and the results are so compelling, companies are in danger of adopting a “test and learn” culture where pages are just A/B tested with no additional user input.  That would be the wrong way to go.  A/B testing shouldn’t be used on its own to make decisions, it should always be used in conjunction with other research methods — both qualitative (such as usability testing, ethnography) and quantitative (such as desirability studies).

A/B testing is an important method in the research toolkit because it can give you information that usability testing on its own cannot.  The main goal of A/B testing is to see how business metrics move up and down depending on the version of the page — click through rates, checkout rates, purchasing rates, etc.  You can’t see that with usability testing alone.  But as Kohavi et al. point out in their paper Practical Guide to Controlled Experiments on the Web, A/B testing has some major limitations:

• Quantitative Metrics, but No Explanations. It is possible to know which variant is better, and by how much, but not why.  In user studies, for example, behavior is often augmented with users’ comments, and hence usability labs can be used to augment and complement controlled experiments.
• Short term vs. Long Term Effects. Controlled experiments measure effects during the experimentation period, typically a few weeks.   It is wise to look at delayed conversion metrics, where there is a lag from the time a user is exposed to something and take action. These are sometimes called latent conversions.
• Primacy and Newness Effects. These are opposite effects that need to be recognized. If you change the navigation on a web site, experienced users may be less efficient until they get used to the new navigation, thus giving an inherent advantage to the Control. Conversely, when a new design or feature is introduced, some users will investigate it, click everywhere, and thus introduce a “newness” bias.
• Features Must be Implemented. A live controlled experiment needs to expose some users to a Treatment different than the current site (Control). The feature may be a prototype that is being tested against a small portion, or may not cover all edge cases.  Nonetheless, the feature must be implemented and be of sufficient quality to expose users to it.
• Consistency. Users may notice they are getting a different variant than their friends and family. It is also possible that the same user will see multiple variants when using different computers (with different cookies).

As with most things, it is important to use A/B testing responsibly.   Since every research/testing method comes with its own limitations, a combination of methods is the only way to get the full picture and make the right decisions.