(Photo by Chase Hoffman on Flickr)

Usability studies (qualitative)

You sit down individually with anywhere between 5 and 20 people (preferably all in your target audience, though this isn’t vital), and give the subject a task to complete.

You then observe how they complete this task and ask open-ended questions along the way.

Good for:

• Testing preliminary designs and drafts. Your can even test hand-drawn drafts.
• Understanding your customer’s behavior and reasoning. During the test you can (and should) ask your subjects why they reacted the way they did.
• Testing out structural changes and see how people respond to them. Because you can observe how people use your site and can ask them questions about it, you get a good idea of what information structure makes the most sense to your users.
• Uncovering points of failure. Is a certain form field overlooked constantly?
• Unveil your customer’s expectations. Often your customers will expect your site to work in a different way than it was designed to.

Pitfalls:

• Don’t care about your subject’s opinions. The sample size is waaaay too small to provide any meaningful data on stuff like what colors to use (use a/b split testing instead), so ignore whether your test subjects like the design or not.
• Be careful not to influence your subject during the test. Leading questions and constant interruptions will result in bad data.

When?

These studies should be done during initial drafts and before final design.

A/B split testing (quantitative)

You show different versions of the same page to those people visiting your site, and then measure which version converts the most visitors (convert can mean anything from buying a product, signing up for a newsletter, contacting you, etc.).

Once you have a statistically significant number, you choose a winner and show that version to everyone. You then repeat the process, testing something else. Continual improvement.

Good for:

• Testing specific features. It can be anything:
  • Copy: should the button on the product page read “Buy” or “Add to cart”? Should we write long or short product descriptions?
  • Color: should the headline be black, blue or red? Should we color-code our menus?
  • Images: should we use the image with the baby or the one with the half-naked lady or the one with the 30-something man reading a book? Should the image be big or small?
  • Price: how much do sales decrease if we raise the price by 30%? Do they even decrease? Perhaps they increase because people perceive it to be better.
  • …
• Getting insight on how your customers actually behave. The data is based on what your customers actually did in real life, not what they said they would do during an artificial study.
• Proving one thing works better than another. You will with high statistical probability be able to say that green headlines are better than blue for this site, and be able to show real data to back up your statement.

Pitfalls:

• Can be technically difficult to set up because it is a complex matter juggling around with multiple versions of the same page and making sure the same user sees the same version every time he visits that page.
• Requires a decent amount of visitors to yield statistically valid data which can be a problem for brand new sites from unknown brands.
• Many will neglect to do a/b split testing because it is done post-launch and takes patience to conduct.
• You need clear goals to test. If your goal isn’t directly related to selling a product, you need to look at all parameters and not just “time on site” or “pages visited” as your indicator for success.

When?

These studies should be done after launch (whether it was launch of the entire site or just a specific feature that made its way into production), ideally in a never-ending cycle.