Menu

Links and articles about technology, design, and sociology. Written by Rian van der Merwe.

User testing and long-term product planning

Steve Barnett makes some great points on long-term planning in Plans, Details, Dates, and The Future1. I especially like the point about how user research fits into planning:

Before development starts on a new bit of work, you should be building prototypes and doing user testing with them. This always results in some changes to the plan, and often results in rather large charges. You can’t plan what these changes will be: you don’t know until you’ve done your user testing.


  1. And bless his heart for using an Oxford comma. 

The problem with surveys

Erika Hall speaks so much truth in her post On Surveys:

If you are treating a survey like a quantitative input, you can only ask questions that the respondents can be relied on to count. You must be honest about the type of data you are able to collect, or don’t bother.

My first role at eBay, years ago, was as a quantitative user researcher1. We ran surveys to measure satisfaction with different areas of the product over time. If that period taught me anything, it’s that surveys are extremely useful when combined with analytics as well as qualitative user research (triangulation), and pretty useless when looked at in isolation. There just isn’t enough context by itself.


  1. One of my early experiences at eBay was getting to work one morning and discovering that Peter Merholz wrote a scathing blog post about a survey I was running. This was my second month on the job, so I was pretty sure I was going to get fired. The worst part of it was that he didn’t have the full context, so his criticism wasn’t even valid. We were doing a controlled experiment where each group saw only one of the images in the survey, and the “likelihood to purchase” question was just a decoy as an introduction. We weren’t trying to get absolute numbers of likelihood to purchase (that would be ridiculous) — we were comparing responses to different pages to figure out what iconography would be best for ratings (stars, bars, or check marks). Subsequent questions were more specific about the ratings aspect. It went all the way up to our VP of Product and my manager had to write an explanation. I was mortified. I still sometimes wake up in the middle of the night in a cold sweat, screaming “survey!!!!!!!!!” 

A framework for empathy in design

The Paradox of Empathy is such a great post by Scott Jenson that I pretty much want to quote the whole thing. But I’ll stick with just this one gem, and encourage you to read it in full. It is a fantastic exploration of empathy in design, and includes a framework for making empathy part of our everyday work in a very practical way:

Designers will be the first to admit that not every empathic observation leads to a miraculous insight. However, it’s called “Design Thinking” for a reason: it’s how we process and explore, taking a complex problem and breaking it down before we build it back up. Product managers seem to expect a designer to walk up to a product, say something brilliant, and drop the mic. Experienced designers deeply understand a simple fact: design isn’t a deliverable, it’s a process. A process paved with dozens of small empathic observations that lead you, slowly, iteratively to a better product.

The problem for us designers is that our fellow teammates don’t always think this way and unfortunately, we as a community don’t reflect on this difference. It’s ironic that designers are passionate about how a product interacts with people but not how they themselves interact with their team.

Twitter text shots, and what design wants

I’ve been spending a lot of time thinking about how product design decisions aren’t neutral. The way we design a product has a direct effect on how people use it. This is obvious, but I think we often forget the real implication: Design wants something from its users. And we are the architects of those wants. We have a direct impact on user behavior, and we need to recognize the weight of that responsibility.

Let’s look at some recent product changes on Twitter as an example.

In October 2013, Twitter introduced more visual tweets, with photo previews within the timeline. It’s almost hard to imagine now, but before they introduced this you had to click on a link before you could see a photo.

This change had an immediate effect on how people used the product. I’m sure some of it was intentional — people began to tweet a lot more photos. Some of it was probably not intentional but still made sense: social media marketers caught on to the fact that if they attach a photo to an article tweet, they’ll get more attention since the tweet will take up more screen real estate. A little annoying, but ok, so far so good.

But then there was what must have been a fairly unexpected behavior change. People started to use screen shots of text to bypass Twitter’s 140 character limit. At first, only a few people did it. But then publishers and marketers started to notice, and it took off:

Some have even come up with guidelines for the best way to stand out in these “textshots”:

Following an exchange with MG Siegler a while back, I settled on a specific textshot style: sans-serif text with a sepia background pulled from Pocket. The idea of using the app’s sepia theme for these came from MG, who noticed that yellow screenshots had more contrast in Twitter’s native apps.

I’ll admit, the temptation to do this is strong. Earlier this week I used a textshot and it became my most retweeted tweet ever:

Nice, right? Win!

But wait… Let’s step back for a moment and have a look at the metrics on that tweet of mine:

Tweet activity

Almost 25,000 impressions, with a 0.9% click-through rate. That’s worse than a crappy banner ad, and it’s the sum-total of the amount of traffic I sent to Scott’s excellent post. As Derek Thompson points out in The Unbearable Lightness of Tweeting:

Is the social web just a matrix of empty shares, of hollow generosity? As Chartbeat CEO Tony Haile once said, there is “effectively no correlation between social shares and people actually reading.” People read without sharing, but just as often, perhaps, they share without reading. […]

There used to be a vague sense that Twitter drives traffic, and traffic drives renown (or fame, or pride, or whatever word defines the psychic benefit of public recognition). Instead, the truth is that Twitter can drive one sort of renown (there are some people who are Twitter-famous), and traffic affords a different psychic currency. But they are nearly independent variables.

All of this culminated in Medium’s Text Shots announcement yesterday:

Text shots

So there you have it. There is now a very real chance that most of our Twitter timelines will become nothing but screenshots of Medium articles that no one reads. That doesn’t help Medium, it doesn’t help authors, and it frankly doesn’t help us to experience and learn, which is kind of the point of reading. This trend does help Twitter, though. Quoting from The Unbearable Lightness of Tweeting again:

In the last month, I’ve created nearly 2 million impressions for Twitter. Whether that is good for my Twitter persona and my pride is a qualitative question whose answer resides outside the bounds of an analytics dashboard. But it is quantitatively not a good deal for The Atlantic. Something I already suspected has now been made crystal clear: 99 percent of my work on Twitter belongs to Twitter.

Twitter is a business, and impressions are how they make money, so this isn’t inherently evil or wrong. But Twitter is, if nothing else, not what we think it is. Not to get too curmudgeonly about “early Twitter”, but there was something amazing about the 140 character limit. Something about the constraint that brought out people’s creativity. And because it was all text, timelines were easy to scan. Now, all of that is different.

Putting all my personal feelings about this trend (and its implications on traffic and reading) aside, it’s time I get to the point. This fundamental change in the way Twitter is used can all be traced back to a single, fairly simple design decision back in 2013: expanding photos natively in the timeline. Without that change, none of this would have happened.

As designers we can’t possibly know how all the ways our decisions will affect behavior in a product. But we have to, at the very least, recognize that design has an opinion, and that it wants people to behave a certain way. I like the way Jared Spool phrases this:

Over the last year, we’ve started explaining design as “the rendering of intent.” The designer imagines an outcome and puts forth activities to make that outcome real.

We have a responsibility to do our best to ensure design wants things that are good for users as well as the business. We have to think ahead as much as possible, because what design wants is up to us. And once it wants the wrong things, it might be too late to change.

Don’t just design features, design systems

Rune Madsen wrote a really interesting post on how our design methods need to change when we work in software (as opposed to print). He explains in the post On meta-design and algorithmic design systems:

So what is meta-design? In a traditional design practice, the designer works directly on a design product. Be it a logo, website, or a set of posters, the designer is the instrument to produce the final artifact. A meta-designer works to distill this instrumentation into a design system, often written in software, that can create the final artifact. Instead of drawing it manually, she is programming the system to draw it. These systems can then be used within different contexts to generate a range of design products without much effort.

I’ll add my vote for the need to spend much more effort on design systems (like Atomic Design) upfront, to standardize (and eventually speed up) later development.

The managing/making dance

In my latest column for A List Apart I discuss our obsession with “managerial tracks” in career development, and propose something different. From Managing and Making: It Doesn’t Have to Be One or the Other:

I think we need a career system that encourages people to oscillate between individual contributor roles and manager roles. Maybe we provide “manager sabbaticals” where a manager becomes an individual contributor on a team for six to nine months. Maybe when a manager goes on vacation, an individual contributor takes on their role for a period of time (or for the duration of an entire project). I don’t know exactly what this looks like yet, but I think it’s important for us to figure it out.

I explain why in the article. I’d also like to point out that my original title for the piece was “Managing and Making: It Doesn’t Have To Be A Dirty Dance”, and it featured this photo:

Dirty Dancing Croc

I’m really glad my editor talked me out of it. It’s not good to make jokes that only people who grew up in the 80s will appreciate.

It’s a good joke, though.

Bots and the law

Kashmir Hill asks an interesting question: Who do we blame when a robot threatens to kill people?

Last week, police showed up at the home of Amsterdam Web developer Jeffry van der Goot because a Twitter account under van der Goot’s control had tweeted, according to the Guardian, “I seriously want to kill people.” But the menacing tweet wasn’t written by van der Goot; it was written by a robot.

He goes on:

Bots will be bots. They won’t know if they’re doing something wrong unless we program them to realize it, and it’s impossible to program them to recognize all possible wrong and illegal behavior. So we’ve got challenges ahead. In the short term, [Clément Hertling, a Paris-based university student who wrote the software that powered the bot] suggested Twitter — and any other platforms bots might live on — could solve the offensive speech problem by allowing bots to self-identify in an obvious way as bots. “That would allow people (law enforcement included) to ignore what they say when it becomes problematic.”

This issue only gets scarier as the question expands to wondering what happens when we put self-driving cars in morally ambiguous situations.

The problem with dogfooding

Dogfooding

Another possible origin is the president of Kal Kan Pet Food, who was said to eat a can of his dog food at shareholders’ meetings.

Eating your own dog food on Wikipedia

In software circles dogfooding is usually encouraged — and for good reason. When companies force themselves to use their own products extensively it can have some great benefits. Since internal users visit even the darkest, most neglected corners of a product, bugs tend to be found and fixed much faster than waiting for reports from end users. Employees are also often the most critical users of their own products, so they are driven to make it better and will often have some fantastic ideas.

However, I’ve started to see a dangerous side effect of dogfooding that can sneak up and do huge damage to a product if we’re not careful. Here’s the problem. If we use our own products all the time we become such overly advanced users that we’re eventually unable to separate our own needs for the product from those of our target market.

Most users of our products don’t eat, sleep, and drink the ins and outs of the features we slave over every day. They dip in and out to buy something, or upload something, or quickly edit a photo. So in a really unfair and ironic turn of events, the more we use our own products the less we’re able to think like our users. It sucks, but it’s unfortunately the way it is.

We all repeat the mantra “I am not the user” to each other, but we tend to forget that the way to solve that particular problem is not to morph into advanced users, but to spend more time with actual users. It’s already so hard for us to put ourselves in the minds and shoes of our users. Dogfooding makes this even harder by lulling us into a false sense of security that maybe — just maybe — we are the user after all.

So, be careful. By all means use the crap out of your product — you wouldn’t be pouring your life into it if you didn’t love it. But once you start demanding features that would make your life easier, stop yourself. Be critical of your own motivations. Look at analytics and figure out if yours is a common use case or something reserved for the 1%. Contact your friendly neighborhood user researcher and ask them if they’ve validated the need for said feature. Hold on to your demand with open hands, and let it go if you realize it’s not something that would improve the product for target users.

In short, keep dogfooding — but don’t assume that all dogs will like the new flavors you’re proposing.

Social media and our insatiable desire for approval

Jon Ronson looks into online shaming (and the lives it destroys) in a brilliant piece of journalism called How One Stupid Tweet Blew Up Justine Sacco’s Life:

Eventually I started to wonder about the recipients of our shamings, the real humans who were the virtual targets of these campaigns. So for the past two years, I’ve been interviewing individuals like Justine Sacco: everyday people pilloried brutally, most often for posting some poorly considered joke on social media. Whenever possible, I have met them in person, to truly grasp the emotional toll at the other end of our screens. The people I met were mostly unemployed, fired for their transgressions, and they seemed broken somehow — deeply confused and traumatized.

The conclusion is worth pausing over and pondering:

But perhaps [Sacco] had now come to understand that her shaming wasn’t really about her at all. Social media is so perfectly designed to manipulate our desire for approval, and that is what led to her undoing. Her tormentors were instantly congratulated as they took Sacco down, bit by bit, and so they continued to do so. Their motivation was much the same as Sacco’s own — a bid for the attention of strangers — as she milled about Heathrow, hoping to amuse people she couldn’t see.

We really are terrible at giving people grace — and the benefit of the doubt.

Are they users or people?

In Don’t Say ‘Cyclists,’ Say ‘People on Bikes’ Sarah Goodyear explains how some deliberate language changes turned a serious conflict in Seattle into a civil debate. Here’s what they did:

“Though the group made no secret of their biking advocacy, they didn’t brand themselves as biking advocates,” writes [PeopleForBikes blogger Michael Andersen]. “They branded themselves as neighborhood advocates.”

[Seattle Neighborhood Greenways] also developed a list of new ways to talk about their concerns and promoted it in handy chart form. Instead of “cyclists,” they suggest, use “people on bikes.” Instead of “drivers,” “people driving.” Instead of technical traffic-engineering terms such as “pedestrian/hybrid beacon,” say “safer ways to cross busy streets.” Replace “pedestrians” with “people walking.”

The result?

[Tom Fucoloro of Seattle Bike Blog] says that talking about streets in a way that emphasizes the common humanity of all users, rather than dividing them into tribes with warring interests, has made a real difference in the way Seattle’s planners discuss possible changes to streets with the community. As a result, he says, the discussion has become much more civil. And Seattle has been installing protected bike lanes (don’t say cycletracks!) at a steadily increasing pace.

I wonder if we need something similar in our industry. Instead of “users,” perhaps we should talk about “people who use websites.” After all, we’re supposed to be all about emphasizing humanity in our products.