The proliferation of and reliance on touch screen smartphones has changed the way consumers perceive and use their mobile devices. Consumers today are visiting mobile sites for far more than just a quick check-in or update. Indeed, they expect mobile sites to be more fully functional and feature the ability to quickly browse and search, a quick call to action, with limited scrolling and assurance of security. Mobile sites have evolved into a critical consumer touchpoint that impacts retention and conversion and ultimately the overall value of a brand. With 24/7 access, a mobile site is an essential element of a cross-platform, full digital strategy, allowing brands to deliver marketing or unique experiences in a totally different way from other channels. That’s why it is crucial to quantitatively measure the user experience on a mobile site to understand behaviors on the channel, track KPIs, ensure the user experience is optimized, and identify pain points and strategies for ongoing enhancements to drive overall engagement and provide an exceptional user experience.

The technology to assess a mobile site user experience has evolved substantially to allow for a more robust survey process. Recently, we have had a few opportunities to partner with Fortune 500 companies to help them evaluate their mobile sites. Executing quantitative UX research on a mobile site is significantly different from conducting traditional online surveys on a desktop computer. Here are some key best practices to foster success for your next project:

Prior to Launching the Survey

  • Establish Limits. Restrict the scope of your research objectives since you will be employing a shorter survey that will be less expansive in terms of the number of metrics you can capture. If a shorter survey is not sufficient though, mobile site surveys offer the flexibility to have rotated / blocks of questions which only a subset of users respond to. The responses can be amalgamated together, but this approach requires a larger sample size for full sufficient sample sizes across questions.
  • Know Your Site Traffic. Understand site- and page-specific traffic (number of unique vs. return visitors) to determine realistic sample and segment size goals, field length, whether an incentive might be needed, and designate the pages for the survey to “pop-up”. Web analytics data can help us provide reasonable expectations and predictions.
  • Know Your Max. Establish clear sampling parameters up front in terms of maximum sample numbers allowed, especially if employing an incentive to limit the pay out in case an overwhelming response.
  • Timing Matters. As with traditional sites, designate the sample rate (i.e. every n’th visitor receives a survey pop-up) and the time on the page needed prior to the survey initiating (5 to 10 seconds is a good starting place to allow the user to have some experience). Both these elements can be easily adjusted in field to facilitate achieving desired sample goals.

Introducing the Survey

  • “Sell” Your Survey. Ensure the survey “pop-up” invitation on screen is unobtrusive and has branding consistent with the mobile site so that the survey looks and feels like part of the site visit. Employ a short, engaging invite that catches users’ attention and is honest about the survey length (e.g., “Congratulations! We would love to know your opinion about our site. Please take a moment to answer a very quick survey to help us improve our mobile site.”)
  • Get the Right Device. Capture the device type in the survey (e.g., smartphone, tablet, MP3) and prevent those on a feature phone or desktop from taking the survey. This can be automated on the back end to ensure the only the correct respondents are captured.

Developing the Survey

  • Short and Sweet. Limit the survey length to 5 to 7 minutes at most, since your respondents tend to be more time constrained and often on-the-go when taking a survey on their mobile devices.
  • Keep It Simple. For an enhanced and engaging survey experience on a device that limits scrolling, use very simple and concise wording in questions and attribute text, restrict ratings scales to 5 to 7 points, have only one question per screen, and keep overall branding simple.
  • Keep It (Mostly) Closed-Ended. Limit the number of “other specifies” and “open-end” questions to 2 to 3 to minimize typing on screen and the potential for errors and survey drop-off.
  • Capture Context. Ask users where they are taking the survey and what tasks they are performing to capture valuable situational information and understand mobile site expectations and experiences.

Continuous vs. Pulse?

As with traditional desktop online surveys, the decision to institute a continuous versus pulse tracking methodology is dependent upon a host of contextual factors. Continuous should be employed when there is:

  • Heavy site traffic to have sufficient survey responses over time. AnswerLab can work with you to determine the numbers needed and estimate the response rate and field time.
  • A desire to relate in-market activities (e.g., advertising, news, promotions) to survey responses.
  • A need to monitor trends before, during, and after ad campaigns/events (cause & effect) and reduce the impact of potential seasonality
  • Capture unanticipated competitive and other activities.

Overall, mobile surveys are an extremely important, relatively lower cost, flexible, easy-to-implement, and valuable source of user feedback, offering real time data with dashboards. Brands that survey their mobile sites and use those insights to invest in and drive strategic decisions to enhance the user experience will develop a competitive advantage and ensure their brand is strong. Clients we have worked with have successfully used mobile site surveys to gain strategic insights into a host of areas, including determining how the site impacts their brand, the extent to which design changes have impacted the overall user experience, and identifying key drivers of call to action for ongoing enhancements.

If you want to learn more about measuring website user experience generally, check out our Guidelines for that. If you have general questions about when to apply which mobile UX research methods, see these Guidelines.