One of the most difficult parts of executing a successful round of usability testing can be recruiting participants. You may need to show your designs to people who fit a certain professional persona, or hear feedback from people with a particular technical skill set. Once you’re in touch with testers who fit the bill, you’ll have to navigate their busy schedules, and disparate locations, to organize their test sessions.
But consider this: There are dedicated conferences for almost every profession, technology, and interest under the sun. Attendees are gathered in a single place for a set time to devote their attention to whatever domain it is they all have in common. If that domain is relevant to what you’re building, it’s a user researcher’s paradise. You’re surrounded by your users, and potential users, at a moment when they’re focused on the very context you all have in common.
Acquia builds tools for Drupal developers, who throw some of the most dynamic and exciting open source tech events around. The Drupal community’s flagship conferences, DrupalCons, happen multiple times a year around the world.
I was lucky enough to attend DrupalCons in Los Angeles and Barcelona this year to run usability tests on a product redesign Acquia’s UX team is working on.
Having run the "Conference Usability Testing Gauntlet" twice now, I’ve come away with plenty of process knowledge, which I’ve collected here.
I hope these tips will help you make the most of a conference setting, resulting in ample, helpful, feedback from your target users.
Usability Test Basics
My basic kit consists of two laptops. Yep. That’s it. You can even get away with one if you’re just going to give it to your tester and observe over her shoulder. But two is nice if you want to use screen-sharing to give your tester some space and still get an up-close and personal view of her screen interactions.
At Acquia, we’ve also tested mobile prototypes at DrupalCons. We used an iPod Touch to mimic the screen size and OS of an iPhone. I used an extra-long USB cable to tether the iPod to my laptop, so users could hold the test “phone” closer, and I could still observe their interactions on my screen.
Take the time to think about the best setup for your test, but here’s what’s worked well for me:
- A moderator laptop: Alongside the screen share window, where I can observe my tester’s interactions, I run Google Docs for note-taking. I start with a master test plan document, and make a copy for every test session I moderate, which helps keep my notes in line with the test tasks for each session.
- A test laptop: I keep files, folders and other apps to a minimum to avoid distraction. I also put a shortcut to the prototype file right on the desktop, or bookmark a Web-based UI in the browser I’ll be testing in, so I’m not fumbling to reopen the prototype at the start of a test.
- Remember to turn off any apps that run in the background but might send distracting notifications during a test.
- Kill background apps that might drain your battery. Power is a premium at most conference centers, so you’ll want all the juice in your laptops to be dedicated to your testing.
And two suggestions for additional gear:
- Laptop leash or lock: If you have a place where it’s safe to lock up your test laptop (at DrupalCon, I locked mine at the Acquia booth a few times while I walked the floor recruiting) this is handy.
- Backpack: The most comfortable way to carry a usability test lab around with you for a few days.
Sign-ups, Sign-ups, Sign-ups
If you want to maximize the number of users you hear from, a recruitment plan will save you from wandering the exhibit hall floor trying to approach busy folks (this works too, of course!)
Find allies! If you have other colleagues attending the conference, tell them about your testing activities so they can connect you with people they meet.
In addition to your own business cards, small hand-outs outlining the usability test goals and process are nice to give out to potential testers.
Pro Tip: Create a virtual signup sheet. I use YouCanBook.Me, a great service that allows people to see your available times and snag an appointment. It has all kinds of handy built-ins for handling time zones, sharing the calendar, and letting you easily communicate with folks who register for a test. I printed my custom YouCanBook.me URL on the back of my business cards, so I can hand one over any time I met an interested potential tester. That way they can visit it at their leisure and find a time during the conference that works best for them.
Stationary versus Mobile Set Up
Let’s compare the value of setting up shop in one place versus wandering around and testing where your testers are.
Stationary: You can optimize your space. Find somewhere close to the main events but quiet enough so that you can have a conversation with your tester. Find a bench, table or other spot where you can easily maneuver to see what your user is doing. If you can score a power outlet, even better. Drawback: If you’re parked in one place, you’ll need to get your users to come to you / know where you are. Prepare for that.
Mobile: You can optimize your time. If you’re carrying your usability lab with you, you can recruit testers on the exhibit hall floor and test with them right there when you find someone with free time. Being visible like this can even help you get more folks aware of what you’re doing, and get them interested in testing. Drawbacks: It can be a little uncomfortable, and you might have a hard time finding a place to sit. Scope out areas ahead of time, like extra seating at the edge of the venue, or a quiet lobby area you can pop into. Keep audio quality in mind if you’re recording; it can be tough to record in a noisy conference hall.
Screen Sharing & Recording
Webex gives you browser-based screen sharing and screen recording. This is great, no arching your neck to peer over your tester’s shoulder. However, it’s Wi-Fi dependent. If you’re using a conference Wi-Fi signal, which may be taxed by hundreds of other attendees, this might not be the best option.
Camtasia provides a screen and audio record that’s local to your tester’s laptop. Hit “record” before they start testing, and you’re good to go.
If you’re using a browser-based screen recorder, don’t forget: your recordings are saved to the service and not your computer! Download those files if you want to watch ‘em on the plane home without Wi-Fi.
Respect the Event & the Community
I’ve saved this for last, but it should come first in your considerations of usability testing at a conference or any setting. Make sure you’re mindful of the fact that the conference itself comes first.
DrupalCon is an epic event meant to support Drupal users in learning, sharing, teaching, and coding Drupal -- all together in a shared space. As a UX researcher, I’m thrilled to talk to these folks, but I want to balance that with respecting their time and their work during the week. The same goes for the organizers of the event where you’re testing: they’re doing the work that brought you all together in the first place, so be sure that your testing activities aren’t a disruption.
Check in with conference organizers or volunteers before and during the event to make sure you’re good to go with your test plan. During the event, chat with volunteers working the floor. Ask where it’s okay to hang out if you want to chat with testers or potential testers, and make sure your setup isn’t in the way. And hey, don’t forget to give back: if you’re using a certain space, offer to help volunteers if they’re setting it up at the beginning of the day, or cleaning it up at the end. It’s just more fun that way.
Acquia’s UX Team ran full rounds of usability testing at DrupalCon North America and DrupalCon Europe this year. The focused feedback on our designs from so many Drupal developers let us iterate and improve in line with what our core users want. To UXers out there, I hope these little details will make your next big-event usability test session a success.
And to DrupalCon: thanks to all of the attendees who volunteered as testers.
See you next year!
One of the most difficult parts of executing a successful round of usability testing can be recruiting participants. You may need to show your designs to people who fit a certain professional persona, or hear feedback from people with a particular technical skill set. Once you’re in touch with testers who fit the bill, you’ll have to navigate their busy schedules, and disparate locations, to organize their test sessions.Acquia Developer Center November 4, 2015 May 13, 2016