How Acquia Tests Software (via uTest)

I started at Acquia 6 months ago, having previously worked for larger, process-heavy corporations that sold packaged proprietary, software with long release cycles. Our QA teams consisted of over 50 people (sometimes a lot more), and were always the long pole in the process, whether it was Agile or Waterfall.

Not so at Acquia.

At Acquia, I manage a lean QA team of 4 people (we have another team that tests usability), and we support 5 products. We work in an Agile environment, release every 3 weeks, and meet our quality goals for each sprint.

Our QA testing strategy at Acquia is to perform tests on agile user stories (akin to use-case tests or acceptance tests in Behavior Driven Development). Our goals are to:

  • Define the behavior of the system, and not have a previously-coded system define the behavior for us
  • Test failure cases so that they won’t affect production
  • Stress systems through performance and load automation
  • Mix automated and manual testing methods, as they’re complementary (machines are fast and consistent, but people have brains and are unpredictable)

We do this by:

  • Listing the scenarios that must succeed for a product to be complete
  • Writing automated tests to perform basic success and failure operations
  • Engaging a crowd-sourced manual testing platform to examine our product in more depth

How can we do this with only a team of 4? uTest’s crowdsourced testing platform lets us leverage over 45 testers a month, without exceeding my start-up budget.

“Coming together is a beginning; keeping together is progress; working together is success.” – Henry Ford

Our relationship with uTest started in the fall of 2009, with approximately one test cycle a month. During our initial engagements, we learned how to work together in an Agile environment where code changes occurred throughout the testing cycle. Then, with each test cycle iteration, we began to synchronize our efforts, and become more successful. uTest started by being only a manual testing platform; to being a service that could execute test cases, analyze a product, and write test cases; to having mature test team leads that make our engagements quite “effortless”.

uTest puts a lot of effort into our relationship, which is evident in how they listen to our feedback, respond immediately to our concerns, and collaborate with us on solutions to our problems. I feel like they’re extensions of our QA team, working together with us to ensure that common and expected user-performed tasks succeed 100% of the time.

Currently, we have a dedicated team of testers (that includes a test lead and a project manager). Our testers test our products, in addition to our regular test cycles. Dedicated testers develop test cases, perform functional testing, and provide bug-fix verification.

We’ve learned the following lessons while we’ve worked with uTest:

  1. Make sure you know what you want, and articulate it well to others.
  2. You can have the most passionate testers in the world, but they can’t read your mind. (See #1)
  3. Good testers will learn and adapt to your testing methodology.
  4. You’re never as ready for a release as you think you are. We run with the expectation of being able to release more or less any time, but when we prepare for a uTest cycle, we come to realize there’s more work to do.

As Archimedes said, “Give me a lever long enough and a fulcrum on which to place it, and I shall move the world.” Our relationship with uTest is just the lever we need as we meet our current testing requirements, and as we look forward to more products and services to test.