UMN Usability Test Recap (for Drupal.org front page)

Hey, folks!

Here's a barfing of text I made on the plane home yesterday about our findings/experience with the UMN usability testing last week. Members of the team, feel free to edit this, else I'll continue to pick away at it today/tomorrow.

TITLE: Report from the University of Minnesota Usability Testing

Summary

From May 17 - 19, 2011, in advance of DrupalCamp Twin Cities, a number of Drupal community members met at the University of Minnesota usability lab in Minneapolis to perform a round of formal usability testing on Drupal. This is the fourth such test we as a project have done, and the first targeting the new Drupal 7 release.

The great news is that most of the changes we made in Drupal 7 seemed to test very well; Drupal was no longer confusing participants with basic conceptual hurdles like where the front-end vs. back-end of their site was and how to create an "About us" page, and for the most part the places in the administrative interface that people needed to go to perform various tasks were very clear; much less blind clicking around.

The bad news is that now that some of these basics have been dealt with, we've uncovered a whole new layer of challenges that first-time Drupal site builders encounter, including some that were pretty surprising. Modules, blocks, and content types are really difficult, as are many of the obscure words that we use *(e.g. does anyone without an art degree know what a "Tryptich" is? :)).

This report talks about why usability testing is important, goes over the methodology we used, reviews previous findings,

Why usability testing?

Methodology

We focused on the "Jeremy" persona. [LINK TO PERSONAS]

Something about the Help desk.

List of tasks we tested with.

These participants were all people directly in Drupal's target audience. These weren't your parents, these weren't random bums off the street. These were site builders who already used alternative tools such as Dreamweaver and WordPress, who knew HTML, and who understand how the web works.

Previous findings

In 2008 and 2009 our community did several rounds of formal usability testing on Drupal core and a few key contributed modules, at the University of Minnesota (report) and University of Baltimore (report, report). These tests were primarily focused on the "old" Drupal 6 user interface.

During these tests, we uncovered a number of enormous conceptual problems with Drupal, some of the biggest being:

• Separation of front-end vs. back-end: Participants could not tell if they were seeing what their users were seeing, or if they were looking at an administrative page that only they could see. The front-end and back-end of Drupal out of the box looked identical, so most participants experienced complete disorientation throughout the test.
  [IMAGE]
• Overwhelming number of administrative options exposed, with no visual indication of importance: On the main administrative landing page (/admin), the sheer number of options exposed there was completely overwhelming to new users. Worse, links that are referenced very often when building and maintaining a Drupal site, such as "Modules" and "Content", had exactly the same visual importance as never-used options such as "Post settings".
  [IMAGE]
• Content creation process fraught with issues: All sorts of problems occurred during the content creation process:
  • The distinction between "Page" and "Story" wasn't clear. Most people, especially those coming from Dreamweaver or similar, think of "Page" as being the entire browser window from top left to the bottom right. "Story" was thought of as perhaps being a multi-page article, but no one was really sure. Worse, the entry forms for both types are identical (simply a Title/Body field), making the distinction even more difficult to discern. The long descriptions didn't help, either, in the rare cases they were read.
    [iMAGE]
  • The visual prominence of both the "Menu settings" fieldset and the "Split summary at cursor" link drew undesired attention to these options, and left many participants thinking they were required parts of the content creation process. The fact that the menu placement drop-down included the "Navigation" menu, which had all of the site's links made this first step even more confusing.
  • Finally, collapsed fieldsets were problematic, in two different ways. Some participants thought that they were links that would take them off the page and they'd lose their data. Many others clicked and opened every. single. fieldset. to look at the options therein. Doing this created an even more confusing page with too many options.
  • If they created a content type such as a Page, that was not promoted to the front page, and didn't add it to the primary navigation, their content was instantly lost, with no hope of finding it again. This was incredibly frustrating for participants, many of whom were convinced that Drupal had just destroyed their data. They didn't know to go to the "node orphanage" (admin/content/node) to find their content.

How did we attempt to fix these issues in Drupal 7?

In Drupal 7, both the usability team and the D7UX initiative attempted to solve a lot of these larger problems. We added things like:

• "Seven" admin theme and Overlay module for better front-end/back-end separation: These two features work in tandem to create a very stark contrast between a "user" view and "admin" view, to hopefully provide better context to new users.
  [IMAGE]
• Toolbar module and task-based re-organization of administration links: By making all top-level admin links visible at the top of all pages, and by moving some of the most frequently-accessed tasks in Drupal (e.g. "Modules", "Content") to the top, we hoped to eliminate the initial overwhelming feeling coming from new site builders. We also added "Add content" and "Find content" to the prominent Shortcut bar, in order to help with the "node orphanage" problem.
  [IMAGE]
• Contextual links module, to provide an easier way to configure page elements for the visually-oriented. Instead of having to stop what you're doing when you decide you want to change the title of a block, and go through Administer >> Site building >> Blocks, find the block, configure its settings, and so on, contextual links let you jump directly to the block's configuration page right from where you see it. Then when finished, you can close the overlay and get right back to what you were doing on the front-end.
• Improvements to content creation: We renamed "Page" and "Story" to "Basic page" and "Article" to help make the initial distinction between these types more clear. Article also has a "Tags" and "Image" field to help further visually distinguish the two. We replaced dozens of collapsed fieldsets with the "vertical tabs" pattern, which shows without clicking into things what options exist and what they're currently set to. We also greatly decreased the visual importance of summary view, and moved Menu settings to one of the advanced settings, along with an easy way enable it, if desired.

About the new usability tests

What tested well?

So, in summary, all of the major usability problems found in Drupal 6 appear to be either vastly improved or non-existent in Drupal 7. Hooray! :D

What tested poorly?

The flip-side of fixing all of our previous issues is that we uncovered the next layer of confusion. :) Here are a list of things that were utterly baffling, now that you can actually navigate and do basic content creation tasks in Drupal without much difficulty:

• The distinction between "sidebar" content and "primary" content made absolutely no sense. Content is content. When asked to create a sidebar block, every. single. participant went for the "Add content" link. Unsure which option to choose, they went for "Basic page" and scoured the vertical tabs options for a way to add it into the sidebar. When they eventually ended up at the help desk and were informed they needed to re-create their content all over again using a different interface in a different place (Structure >> Blocks), they were all pretty frustrated.
• Drupal's Terminology is obscure. Terms like "Block" and "Module" and "Content types" and "Tryptich" either have no meaning, or a pre-conceived meaning that is different from what Drupal uses these words for. For example, when asked what the word "Module" meant, a few participants thought that a module would be a small piece of content that shows up in the sidebar or on the front page (probably coming from a Joomla! background). When users encounter words they do not understand, they generally do not click them. As a result, Drupal can come across as having limited functionality.

  We should explore the possibility of renaming terms to things that are more universally recognized (most people thought of "Blocks" as "Boxes", for example), or at least improving tooltips and other descriptions to help point people in the right direction. As one egregious example, the title text for the "Modules" link is "Enable or disable modules". We'd be far better served if it were something like "Extend Drupal's functionality with add-on modules," or similar).

• Lack of realistic previews makes building sites in Drupal extremely challenging. The preview provided by Color module is what participants really were craving to see absolutely everywhere. Blocks have no preview, and the visual reference of where regions are located are disconnected from the place where you assign blocks to regions, and also lacks realistic site content.
  [IMAGE]

  Menus also have no preview; simply text descriptions that give no indication where visibly in the page their items will appear. Content does have a preview button, but the preview you get is shown in the admin theme and overlay, not in the context of your actual user-facing site. This was frustrating and confusing.
  [IMAGE]

• While Drupal takes a "content-first" approach to site building, most people take a structure- and/or appearance-first approach. When looking at the mockup, the natural inclination for many participants was to start building out the navigation structure first. You run into issues if you do this, because you need to enter something for "Path" and have no idea what that could be. More visually-oriented users also want to start with picking a theme that looks close to what they want, because without that they can't tell if they're on the right track. The abysmal search options for new themes on Drupal.org makes such an approach impossible.
• New users to Drupal do not realize you can extend it, and so Drupal is seen as having limited functionality. If users found their way to the modules page, they didn't see that they could add new options there; they thought the options presented were what Drupal could do, and were completely lost as to what to do when "webform" wasn't in the list. (It was surprising to people that such basic website functionality wasn't included out of the box.)

  This is a horrifying problem, because the entire point of Drupal is that you can extend its functionality with modules.

• Workflow of finding and adding a new module is seen as overwhelming. Once a help desk call pointed them at the "Install new module" link, new problems began. Being taken off-site to Drupal.org was extremely jarring, as was the (apparent) requirement to download a module to the desktop and then upload it to the server to install it. People craved an in-app browser for modules from Drupal.org.

  Additionally, the filtering options presented for modules are very overwhelming. The "search" box, arguably the most valuable thing when you have the general idea what you're looking for, is located far down the list and was missed by most participants. Instead, they honed in on the "Categories" drop-down, which was completely useless for them.
  [IMAGE]

  Finally, people couldn't make the association between version numbers and how to get that on their site. They generally just picked the first thing in the list, which happened to be 7.x on webform, but wasn't on "Announcements", which another participant searched for.

• The concept of "Content types" isn't remotely clear. There's no indication on the Add content screen that you can create additional content types besides those listed. As a result, when asked to add a new "Announcement" content type to the site, most went right for the Modules page, and hilarity ensued.
• Lack of visual way to place content on the page was a huge detriment. People really craved some sort of "edit in place" mode as a visual way of controlling their site layout. They tried to click on region names in the block demonstration page to add content there. They tried to take a node outlined by the contextual links border and drag it to the sidebar to use it as a block. They tried to click and drag to resize sidebars. The lack of this functionality made Drupal appear extremely clunky.

Overall

• Tag cloud

• Graphs

• Not one user could complete these basic tasks without calling the help desk.

We still have work to do.

Next steps

We found over 100 issues during this round of usability testing, some of them major, some more minor (and more easily fixable). These are captured in raw form in the "Issues Analysis Matrix" spreadsheet [LINK], and in the issue queue under the "UMN 2011" tag [LINK]. We need help on these with designs, with code, and with discussions involving the larger community on what it means to fix them, and how.

Check out the Usability group if you'd like to participate in these discussions, as well as the Usability Community Initiatives page for a listing of high-impact issues that need some folks to work on them.

We also need more folks to help with further usability testing, both formal and informal. What we tested at University of Minnesota only scratches the surface of what Drupal core can do (for example, Taxonomy, Search, adding Themes from Drupal.org...), and there are also another 9,000 contributed modules out there as well. You can read more about guerilla usability testing at [LINK??]

So please get involved, and help us make Drupal 8 (and Drupal 7, through contributed modules) the easiest to use version of Drupal yet! :D