Cool stuff and UX resources

< Back to newsletters

Introduction

Have you ever – just for fun – played around with a photograph in Adobe Photoshop? It's an incredibly powerful tool. You can change lighting...the colors...the composition...even the content. (Remember the royal wedding pictures where Prince William wasn't smiling? So they doctored the photo...?) And we haven't even started talking about the dozens of filters that you can use to turn your photo into a watercolor from outer space with a fuchsia-to-lime mezzotint and a Gaussian blur for extra effect... In the hands of a graphics artist, Photoshop is an amazing tool. In the hands of a curious novice, it's a fun toy. We (I consider myself part of this group) can do some very basic things effectively. But for most novices, Photoshop becomes an unproductive (albeit entertaining) distraction because it is TOO powerful. The myriad of possibilities is distracting. Entrenched in the tools, we lose track of the fact that we lack the essential understanding of the subtleties that allow real graphics artists to do their thing.

Electronic survey tools present a similar problem. In recent years, a number of powerful survey tools have emerged on the Web. Some of these tools make it easy to post very sophisticated surveys and collect data on the Web. Distribution is cheap. Data entry, verification, and descriptive analysis can be completely automated. Just-in-time information survey help (e.g., links to definitions of terms) and multimedia resources can be embedded directly into the survey. Since contingent question presentation can be handled invisibly, there is no more: "If you answered, 'Yes,' skip to question 26b". Further, using electronic mail to invite participants to the survey, transitions participant recruiting from a pull to a push activity.

These are all very powerful tools. You can literally implement a survey in a few minutes. It is easy to forget, in this domain of apparently instantaneous productivity, that there is an entire field of knowledge that supports the design of effective and meaningful survey data collection. As with Photoshop, the power of the tool distracts us from the essence of the task.

Toyota Prius

Research on effective electronic surveys

A recent survey paper by Andrews, Nonnecke and Preece (2003) highlights the challenges of developing effective surveys. Based on a comprehensive review of the literature, they discuss several areas of concern for researchers creating electronic surveys. They include sampling and participant selection, and survey design.

Click here to provide feedback...

It is not unusual today for Web content managers to simply add a link to their home page that says: "Give us feedback. Participate in a survey." Who clicks on these links? Why?

Effective sampling – or getting an appropriately large and representative set of participants to respond – presents one of the greatest challenges for effective survey implementation in any mode. If the sample isn't representative, the survey results are not very meaningful. So this is a very important concern.

Achieving an appropriate representative random sample with a reasonable return rate is never trivial. Unfortunately, the nature of the Internet makes accessing a representative sample virtually impossible. Research suggests that individuals who participate in Internet surveys use the net more frequently and have better Internet skills than those who opt out (Kehoe & Pitkow, 1997). They are just different. While the male-female gap has largely disappeared (NUA, 2001), electronic survey participants tend to have higher incomes and education levels and tend to be more homogeneously Caucasian than the general population of Internet users. The demographic gaps (economics, age, ethnicity) between on-line and off-line populations are even larger (Yun & Trumbo, 2000).

Since there is no common registry of Internet users, there is no way to identify all possible users (sometimes called "the sampling frame"). As such, participant selection is necessarily nonrandom from the offset. Surveyors must remember that generalizing the results to off-line (or even the whole on-line) population may not be viable.

Available tools notwithstanding, knowing who you want to survey, and how you will reasonably access and get responses from that group, is critical to developing a meaningful survey. Just adding a link typically won't get you there.

Couper (2000) suggests implementing strategies such as intercept sampling to achieve a pseudo-random sample. In intercept sampling every nth visitor is invited to participate in the survey – like an exit poll. Anecdotal evidence suggests that inviting individuals to participate via a pop-up with a single "Would you...?" question followed up with an e-mail message containing a link to the actual survey yields increased response rates. It's a start.

Surveys that feel like fishing expeditions usually are

Getting people to finish the survey is also a challenge. Have you been overwhelmed in a survey by an entire page of 1-7 ratings and just bailed? Surveys that appear long and mentally intensive typically don't get finished. Andrews, Nonnecke and Preece (2003) note that it is important to make the survey appear easy and usable. They review several research-driven findings to increase the likelihood that respondents will complete the survey:

  • Make sure the survey topic is of interest to the invitees
  • Customize the survey to the target population (e.g., language, expectations of multimedia use, etc.)
  • Assure privacy and confidentiality of an individuals' responses
  • Warn participants how long the survey will take (and be honest about it)
  • Use an appropriate subject line in e-mail invitations
  • Clearly provide contact information and access to the surveyors
  • Make automated passwords easy to enter and use
  • Pilot your survey on different platforms so you know that your participants are seeing what you intend them to see
  • Separate the invitation from the survey itself
  • Send reminder e-mails to non-responders

Piloting the survey

In addition to their literature review, Andrews, Nonnecke and Preece (2003) review a case study of the development of a survey. Their goal was to determine whether effective and thoughtful design would allow researchers to elicit survey responses from 'lurkers' or non-active participants in electronic communities. (Yes, it does.) More interesting though, are the (re)discoveries these trained researchers experienced as they piloted their survey through several stages of development.

Stage 1: The Colleague Test
Candidate questions were drafted and reviewed over four iterations within the survey team.

Research team observations:

  • Writing effective questions is not easy.
  • Balancing brevity, tone, and accuracy requires feedback.
  • Redundant questions and those not directly related to the topic should to be deleted.

Stage 2: The Cognitive Test 
The entire survey system was usability tested. Participants read the invitation, completed the survey and participated in retrospective interviews using a think-aloud procedure.

Research team observations:

  • Writing effective instructions, questions, and privacy reassurances requires feedback from outside the survey team.
  • Physical interactions with the survey (e.g., scrolling a long set of questions) has a significant impact on survey usability.
  • Effects of sequence and grouping are group dependent. Survey developers' intuitions are not necessarily representative of survey takers' experiences or conceptual hierarchies.
  • Typographic elements like category headings help participants map the sections and content of the survey.
  • Participants tend to move directly to the survey without reading the text. Instructions must be scannable.

Stage 3: Live Test
Pilot the survey on a small scale to validate the sampling procedure and investigate the response rate. Test-drive analysis procedures.

Research team observations:

  • You don't know exactly what data you wish you had collected until you collect some. Having an opportunity to review and make minor changes in the data-collection process in this stage can result in large differences in the effectiveness of the data collected overall.

Stage 4: Clean Up
Final grammar and format checks.

Owning Photoshop doesn't make you an artist

Creating surveys that meaningfully collect data from the right set of participants requires a keen and thoughtful effort built on an understanding of the subtleties of survey design. The strategies above provide some key insights to issues and concerns of implementation; they do not begin to touch on what it means to write effective survey questions.

Remember that photograph that you stretched and colored and filtered in Photoshop? What did you do with it? As with Photoshop, novices can pull off some trick maneuvers with electronic survey tools. But to develop serious, longitudinal data-collecting tools you need to invest time in understanding the domain, not just accessing tools.

References

Andrews, D., Nonnecke, B., Preece, J. (2003). Electronic Survey Methodology: A Case Study in Reaching Hard-to-Involve Internet Users, International Journal of Human-Computer Interaction. Volume 16, Issue 2, Page 185-210.

Couper, M. P. (2000). Web Surveys: A Review of Issues and Approaches. Public Opinion Quarterly. 64(4), 464-494.

Dillman, D. A. (2000). Mail and Internet Surveys: The Tailored Design Method, New York: John Wiley and Sons.

Kehoe, C. M., and Pitkow, J. E. (1996). Surveying the Territory: GVU's Five WWW User Surveys. The World Wide Web Journal 1(3), 77-84.

Yun, G. W. & Trumbo, C.W. (2000). Comparative Response to a Survey Executed by Post, E-mail, & Web Form. Journal of Computer-Mediated Communication 6(1).

Message from the CEO, Dr. Eric Schaffer — The Pragmatic Ergonomist

Leave a comment here

Subscribe

Sign up to get our Newsletter delivered straight to your inbox

Follow us

© 1996-2018 Human Factors International, Inc. All rights reserved | Privacy policy | Terms and conditions for public training courses

Privacy policy

Reviewed: 18 Mar 2014

This Privacy Policy governs the manner in which Human Factors International, Inc., an Iowa corporation (“HFI”) collects, uses, maintains and discloses information collected from users (each, a “User”) of its humanfactors.com website and any derivative or affiliated websites on which this Privacy Policy is posted (collectively, the “Website”). HFI reserves the right, at its discretion, to change, modify, add or remove portions of this Privacy Policy at any time by posting such changes to this page. You understand that you have the affirmative obligation to check this Privacy Policy periodically for changes, and you hereby agree to periodically review this Privacy Policy for such changes. The continued use of the Website following the posting of changes to this Privacy Policy constitutes an acceptance of those changes.

Cookies

HFI may use “cookies” or “web beacons” to track how Users use the Website. A cookie is a piece of software that a web server can store on Users’ PCs and use to identify Users should they visit the Website again. Users may adjust their web browser software if they do not wish to accept cookies. To withdraw your consent after accepting a cookie, delete the cookie from your computer.

Privacy

HFI believes that every User should know how it utilizes the information collected from Users. The Website is not directed at children under 13 years of age, and HFI does not knowingly collect personally identifiable information from children under 13 years of age online. Please note that the Website may contain links to other websites. These linked sites may not be operated or controlled by HFI. HFI is not responsible for the privacy practices of these or any other websites, and you access these websites entirely at your own risk. HFI recommends that you review the privacy practices of any other websites that you choose to visit.

HFI is based, and this website is hosted, in the United States of America. If User is from the European Union or other regions of the world with laws governing data collection and use that may differ from U.S. law and User is registering an account on the Website, visiting the Website, purchasing products or services from HFI or the Website, or otherwise using the Website, please note that any personally identifiable information that User provides to HFI will be transferred to the United States. Any such personally identifiable information provided will be processed and stored in the United States by HFI or a service provider acting on its behalf. By providing your personally identifiable information, User hereby specifically and expressly consents to such transfer and processing and the uses and disclosures set forth herein.

In the course of its business, HFI may perform expert reviews, usability testing, and other consulting work where personal privacy is a concern. HFI believes in the importance of protecting personal information, and may use measures to provide this protection, including, but not limited to, using consent forms for participants or “dummy” test data.

The Information HFI Collects

Users browsing the Website without registering an account or affirmatively providing personally identifiable information to HFI do so anonymously. Otherwise, HFI may collect personally identifiable information from Users in a variety of ways. Personally identifiable information may include, without limitation, (i)contact data (such as a User’s name, mailing and e-mail addresses, and phone number); (ii)demographic data (such as a User’s zip code, age and income); (iii) financial information collected to process purchases made from HFI via the Website or otherwise (such as credit card, debit card or other payment information); (iv) other information requested during the account registration process; and (v) other information requested by our service vendors in order to provide their services. If a User communicates with HFI by e-mail or otherwise, posts messages to any forums, completes online forms, surveys or entries or otherwise interacts with or uses the features on the Website, any information provided in such communications may be collected by HFI. HFI may also collect information about how Users use the Website, for example, by tracking the number of unique views received by the pages of the Website, or the domains and IP addresses from which Users originate. While not all of the information that HFI collects from Users is personally identifiable, it may be associated with personally identifiable information that Users provide HFI through the Website or otherwise. HFI may provide ways that the User can opt out of receiving certain information from HFI. If the User opts out of certain services, User information may still be collected for those services to which the User elects to subscribe. For those elected services, this Privacy Policy will apply.

How HFI Uses Information

HFI may use personally identifiable information collected through the Website for the specific purposes for which the information was collected, to process purchases and sales of products or services offered via the Website if any, to contact Users regarding products and services offered by HFI, its parent, subsidiary and other related companies in order to otherwise to enhance Users’ experience with HFI. HFI may also use information collected through the Website for research regarding the effectiveness of the Website and the business planning, marketing, advertising and sales efforts of HFI. HFI does not sell any User information under any circumstances.

Disclosure of Information

HFI may disclose personally identifiable information collected from Users to its parent, subsidiary and other related companies to use the information for the purposes outlined above, as necessary to provide the services offered by HFI and to provide the Website itself, and for the specific purposes for which the information was collected. HFI may disclose personally identifiable information at the request of law enforcement or governmental agencies or in response to subpoenas, court orders or other legal process, to establish, protect or exercise HFI’s legal or other rights or to defend against a legal claim or as otherwise required or allowed by law. HFI may disclose personally identifiable information in order to protect the rights, property or safety of a User or any other person. HFI may disclose personally identifiable information to investigate or prevent a violation by User of any contractual or other relationship with HFI or the perpetration of any illegal or harmful activity. HFI may also disclose aggregate, anonymous data based on information collected from Users to investors and potential partners. Finally, HFI may disclose or transfer personally identifiable information collected from Users in connection with or in contemplation of a sale of its assets or business or a merger, consolidation or other reorganization of its business.

Personal Information as Provided by User

If a User includes such User’s personally identifiable information as part of the User posting to the Website, such information may be made available to any parties using the Website. HFI does not edit or otherwise remove such information from User information before it is posted on the Website. If a User does not wish to have such User’s personally identifiable information made available in this manner, such User must remove any such information before posting. HFI is not liable for any damages caused or incurred due to personally identifiable information made available in the foregoing manners. For example, a User posts on an HFI-administered forum would be considered Personal Information as provided by User and subject to the terms of this section.

Security of Information

Information about Users that is maintained on HFI’s systems or those of its service providers is protected using industry standard security measures. However, no security measures are perfect or impenetrable, and HFI cannot guarantee that the information submitted to, maintained on or transmitted from its systems will be completely secure. HFI is not responsible for the circumvention of any privacy settings or security measures relating to the Website by any Users or third parties.

Correcting, Updating, Accessing or Removing Personal Information

If a User’s personally identifiable information changes, or if a User no longer desires to receive non-account specific information from HFI, HFI will endeavor to provide a way to correct, update and/or remove that User’s previously-provided personal data. This can be done by emailing a request to HFI at hfi@humanfactors.com. Additionally, you may request access to the personally identifiable information as collected by HFI by sending a request to HFI as set forth above. Please note that in certain circumstances, HFI may not be able to completely remove a User’s information from its systems. For example, HFI may retain a User’s personal information for legitimate business purposes, if it may be necessary to prevent fraud or future abuse, for account recovery purposes, if required by law or as retained in HFI’s data backup systems or cached or archived pages. All retained personally identifiable information will continue to be subject to the terms of the Privacy Policy to which the User has previously agreed.

Contacting HFI

If you have any questions or comments about this Privacy Policy, you may contact HFI via any of the following methods:
Human Factors International, Inc.
PO Box 2020
1680 highway 1, STE 3600
Fairfield IA 52556
hfi@humanfactors.com
(800) 242-4480

Terms and Conditions for Public Training Courses

Reviewed: 18 Mar 2014

Cancellation of Course by HFI

HFI reserves the right to cancel any course up to 14 (fourteen) days prior to the first day of the course. Registrants will be promptly notified and will receive a full refund or be transferred to the equivalent class of their choice within a 12-month period. HFI is not responsible for travel expenses or any costs that may be incurred as a result of cancellations.

Cancellation of Course by Participants (All regions except India)

$100 processing fee if cancelling within two weeks of course start date.

Cancellation / Transfer by Participants (India)

4 Pack + Exam registration: Rs. 10,000 per participant processing fee (to be paid by the participant) if cancelling or transferring the course (4 Pack-CUA/CXA) registration before three weeks from the course start date. No refund or carry forward of the course fees if cancelling or transferring the course registration within three weeks before the course start date.

Cancellation / Transfer by Participants (Online Courses)

$100 processing fee if cancelling within two weeks of course start date. No cancellations or refunds less than two weeks prior to the first course start date.

Individual Modules: Rs. 3,000 per participant ‘per module’ processing fee (to be paid by the participant) if cancelling or transferring the course (any Individual HFI course) registration before three weeks from the course start date. No refund or carry forward of the course fees if cancelling or transferring the course registration within three weeks before the course start date.

Exam: Rs. 3,000 per participant processing fee (to be paid by the participant) if cancelling or transferring the pre agreed CUA/CXA exam date before three weeks from the examination date. No refund or carry forward of the exam fees if requesting/cancelling or transferring the CUA/CXA exam within three weeks before the examination date.

No Recording Permitted

There will be no audio or video recording allowed in class. Students who have any disability that might affect their performance in this class are encouraged to speak with the instructor at the beginning of the class.

Course Materials Copyright

The course and training materials and all other handouts provided by HFI during the course are published, copyrighted works proprietary and owned exclusively by HFI. The course participant does not acquire title nor ownership rights in any of these materials. Further the course participant agrees not to reproduce, modify, and/or convert to electronic format (i.e., softcopy) any of the materials received from or provided by HFI. The materials provided in the class are for the sole use of the class participant. HFI does not provide the materials in electronic format to the participants in public or onsite courses.