Sampling from Online Panels
Sampling from Online Panels
- Luzi ShiLuzi ShiCriminology and Criminal Justice Department, University of Rhode Island
- , and Sean Patrick RocheSean Patrick RocheSchool of Criminal Justice and Criminology, Texas State University
Summary
Since the 2010s, online surveys have become a popular method among criminologists. Often these surveys are conducted with the assistance of private survey research companies, which gather large groups of people (i.e., respondents) who have indicated a willingness to share their opinions on a variety of issues. These panels of potential respondents vary in size and quality. Researchers planning to collect survey data via these online panels must also consider probability versus non-probability sampling methods. Probability samples provide stronger assurances that sample statistics—particularly, univariate point estimates—are generalizable to broader populations (e.g., adult Americans). They are also often very expensive, although this is somewhat dependent on the size and complexity of the proposed project. Two popular providers of probability samples of the American public are the Ipsos Knowledge Panel and the AmeriSpeak Omnibus panel. In criminology and criminal justice, researchers have used online probability panels to study a variety of topics, including behaviors regarding firearms, attitudes toward policing, and experiences of violence.
Non-probability samples present a budget-friendly alternative but may be less generalizable to populations of interest. Since 2010, these samples have become especially popular in the criminological literature and are much more commonly used than online probability samples. Findings from non-probability online surveys often yield remarkably similar relational inferences (e.g., correlations) to those obtained from probability samples. However, non-probability samples are generally unsuitable for providing generalizable univariate point estimates. Some of the leading providers of non-probability samples from panels are YouGov, Qualtrics, and Lucid. As of 2024, YouGov uses a matched opt-in sample with a more sophisticated sampling design, while Qualtrics and Lucid provide quota samples. Researchers may also directly recruit non-probability samples of respondents via crowdsourcing platforms, such as Amazon Mechanical Turk, or services that incorporate those platforms into their own business model, such as CloudResearch. Research suggests that platforms with more sophisticated sampling procedures tend to yield more accurate results. Consequently, matched opt-in samples such as YouGov are approximately twice as expensive as Qualtrics samples and are many times more expensive than crowdsourcing platforms. Finally, it should be noted that the demographic composition of online samples, even those that have been simply crowdsourced, tend to be more diverse than typical in-person non-probability samples used in criminology and criminal justice research (e.g., college students).
Subjects
- Research Methods
Introduction
Survey research companies, both those serving academic customers and those conducting market research for private firms, often gather large groups of people who have indicated a willingness to share their opinions on a variety of issues (Evans & Mathur, 2005). These groups are called panels. Since 2010, sampling from online panels has become a popular method among criminologists. The first two sections, “Online Probability Samples” and “Online Non-Probability Samples,” introduce some commonly used online platforms, discuss the opportunities and challenges of these online sources, and address questions that peer reviewers may raise to authors who conduct online surveys. The final section, “Considerations When Choosing an Online Survey Platform” concludes with some best practices for collecting high-quality data and suggests further readings for researchers who are interested in using online panels.
As with more traditional administration methods, and depending on their research question and budget, researchers who plan to collect survey data via online panels may consider probability or non-probability sampling methods. Two popular probability samples of the American public are the Ipsos KnowledgePanel and the AmeriSpeak Omnibus panel. The “Online Probability Samples” section describes the sampling procedure used by these two companies and ways to access each platform. Note that both panels require very large monetary investments.
Non-probability online samples present a common, and much more budget-friendly, alternative. Once the internet became a ready source for survey participants, non-probability panels and other online non-probability samples spread rapidly. Non-probability samples usually rely on convenience and/or quota sampling. With convenience sampling, researchers may post their web survey link via social media or promote the survey via email lists or organizational websites. Researchers may also recruit respondents via crowdsourcing platforms, such as Amazon Mechanical Turk (MTurk), or services that incorporate those platforms into their own business model (e.g., CloudResearch, formerly known as TurkPrime). The “Online Non-Probability Samples” section presents criminological research that has used MTurk and other convenience samples and discusses the limitations of using such methods. With quota sampling, researchers may purchase responses from non-probability panels via vendors like YouGov, Qualtrics, and Lucid, including quotas for demographic characteristics, such as race and ethnicity, gender, and age.
It should be noted that a tacit assumption when using many panels is that the researcher’s goal is to generalize sample findings to the adult population of the United States. While this is true for many researchers, particularly those interested in public opinion on crime and criminal justice topics, it is not necessarily true for all criminological research inquiries. Scholars may employ these platforms to obtain samples of use as proxies for narrower populations of interest (e.g., judges, prosecutors, defendants). Researchers should share their intentions with the administrators of survey panels and be cognizant that their population of interest may substantially differ from panel respondents, both demographically and in terms of training and life experiences. While demographic characteristics may be possible to emulate using an online panel, differences in training and life experience could be considerably more difficult if not impossible.
The use of online samples, especially non-probability samples, presents both benefits and challenges. Online samples are generally easier, quicker, and cheaper to collect than traditional survey samples. Vendors can easily integrate with, and may even require, the use of powerful survey design programs, such as Qualtrics CoreXM. While beyond the current discussion, these programs allow for customization and randomization at multiple levels, providing tools far beyond what most single researchers, or even small teams, could reasonably accomplish outside of major survey research centers and/or with other survey modes (e.g., mail, telephone). At the same time, these design tools can require individual paid subscriptions or university-wide (i.e., “enterprise level”) partnerships to use, and they may require a substantial amount of training and expertise to use to their full potential.
Perhaps most importantly, many scholars may wonder: Who are the respondents for these surveys? Are they paying attention, or being truthful? In an age of growing artificial intelligence, the question might asked, Are they human beings at all? The opacity of many online sample vendors’ business operations is a cause for concern among many scholars. While this discourse has largely been absent in the field of criminology, researchers in political science, psychology, and economics have spent substantial time investigating trends in data quality across various platforms and developing best practices to identify inattentive and unnatural behaviors in survey responses. In summary, readers who already have a basic understanding of survey methodology will find this to be a practical guide to the various online panels, and sampling methodologies, currently being employed by criminological researchers.
Online Probability Samples
Probability sampling, where each element in a population has an equal or at least known chance of being selected, is the basis for much of inferential statistics (design-based inference; see Neyman, 1934) and provides researchers with a theoretical basis for assuming that results will generalize to the target population. In the mid-20th century, many scholars considered traditional data collection methods such as face-to-face interviews, in conjunction with a probability sampling procedure, to be the “gold standard” of survey research (Igo, 2007). By the 1970s, with the swiftly growing popularity of home telephones and random digit dialing (RDD) methods, the designation of “gold standard” shifted to telephone surveys (Yeager et al., 2011). However, by the early 2000s, the rise of telemarketing companies, criminal scams committed over the phone, the invention and rapid proliferation of the cell phone, and declining response rates combined to substantially degrade the efficacy of telephone surveying.
Since 2000, internet-accessible devices have become ubiquitous, and many research organizations have transitioned from using traditional RDD telephone surveys and mail surveys to online probability-based panels (Olson et al., 2021). Some of the online panels are accessible to individual researchers for primary data collection as well. Two popular online probability samples in the United States are the Ipsos KnowledgePanel and the AmeriSpeak Omnibus panel.
The Ipsos KnowledgePanel is conducted by Ipsos Public Affairs, LLC and is the largest national probability-based panel in the United States (University of Michigan, 2019).1 In 2021, there were approximately 60,000 panelists from approximately 52,000 households (Barlas, 2021). Ipsos recruits respondents primarily using address-based sampling (ABS) via the U.S. Postal Service’s Computerized Delivery Sequence File. Participants receive mail invitations (e.g., invitation letters, postcards, and follow-up letters) and phone calls if a landline telephone number can be matched and found. Recruited panelists must complete a Core Profile survey to answer questions on their basic demographics and household compositions before taking surveys from Ipsos clients. Ipsos provides web-enabled devices or internet access to households without the internet so that the samples are not limited to those who already have access to the internet. Many recruited respondents then take online surveys as a next step. The sample demographic composition is designed to resemble that of the U.S. population using stratified sampling. Individual researchers usually conduct primary data collection via Ipsos KnowledgePanel via the KP Omnibus service, which sends out combined surveys from different researchers to 1,000 adults every week.2 The same process of recruitment, profile, and invitation to a specific survey can be found for other probability and non-probability panels reviewed herein.
The AmeriSpeak Omnibus panel is administered by the National Opinion Research Center (NORC) at the University of Chicago. The sample uses probability ABS frames to recruit panelists and is designed to be representative of households from all 50 U.S. states and the District of Columbia.3 Respondents are recruited using a series of the United States Postal Service (USPS) mails and contacted by NORC’s telephone research center if there is a matched telephone number. Among people who do not respond during the initial recruitment stage, a new recruitment package containing an enhanced offer is sent out by FedEx to a stratified random sample, followed by face-to-face visits from NORC field interviewers. Once recruited, respondents must register their personal profiles either by registering online or by telephone. NORC sends out its omnibus survey to 1,000 U.S. adults on a biweekly schedule and delivers the survey data in 1 week.4
Other online probability panels include the Understanding America Study held by the University of Southern California and the American Life Panel run by RAND.5 There are also online probability-based panels in Europe, such as the LISS panel in the Netherlands and the GESIS panel in Germany (Olson et al., 2021). The list is not exhaustive and will undoubtedly change over time. Readers are encouraged to check the latest literature and pricing rates before accessing any of the panels described here.
Research comparing data collected via online probability samples to RDD telephone surveys finds that results from online probability samples, even without high response rates, may be highly accurate and comparable to results from traditional telephone surveys (Yeager et al., 2011). Chang and Krosnick (2009) compared results from the same survey administered via RDD telephone interviewing versus an internet probability sample and an Internet non-probability sample. The authors found that the internet probability sample outperformed the traditional telephone survey, with less survey satisficing, less random measurement error, and less social desirability bias, and suggest that it is the “optimal combination of sample composition accuracy and self-report accuracy” (p. 641). The survey platforms may also provide weights for researchers to adjust for any nonresponse and noncoverage biases that occur for the specific panel selection (e.g., Wojcik & Hughes, 2019). Additionally, as the respondents are recruited from an existing panel, the survey results are quickly obtained over the Internet.
In light of this reliability and speed, government agencies sometimes use online probability panels to understand public attitudes and behaviors on emerging and emergent social issues. For example, the Centers for Disease Control and Prevention (CDC) used both AmeriSpeak Omnibus and Ipsos KnowledgePanel for surveys of the American public on the topic of COVID-19 vaccinations (Nguyen, Lu, et al., 2021). The CDC subsequently compared respondents’ vaccination reports from administration records and the two online probability panels. The resulting report (Nguyen, Srivastav, et al., 2021) suggests that 38% of Ipsos and 22% of NORC respondents answered the survey questions. After weighting the samples, the vaccination rates reported by Ipsos KnowledgePanel were similar to those of the administration records, with a 0.7 percentage point difference; the estimates from AmeriSpeak Omnibus were higher than the administration records by 6.7 percentage points (Nguyen, Srivastav, et al., 2021).
In criminology and criminal justice, researchers have used online probability panels to study a variety of topics, including behaviors regarding firearms, attitudes toward policing, and experiences of violence. For example, Hill and colleagues (2021, 2022) used data from AmeriSpeak Omnibus surveys to understand gun ownership and related behaviors. Kravitz-Wirtz et al. (2021) used data from Ipsos KnowledgePanel to understand public opinion about violence and firearms. Jipguep-Akhtar et al. (2021) used data from AmeriSpeak to understand perceptions of law enforcement during the pandemic. Iverson et al. (2022) used data from Ipsos KnowledgePanel to understand the experiences of intimate partner violence of women veterans.6
Probability-based panels are costly and may be out of financial reach for many individual researchers. Costs are incurred by the number of questions asked, the estimated time in minutes the entire survey will take respondents (respondents typically offered at a discounted rate when purchasing larger samples), and additional handling costs. For example, as of 2023, Ipsos KnowledgePanel charges US$1,000 for a single close-ended question that is considered simple and straightforward.7 Similarly, AmeriSpeak charges US$1,000 for each close-ended question and $1,200 for each open-ended question (see Greaves, 2017).8 According to data from the RAND website (2023), a 10-minute interview may cost US$32,000, with US$3.25 per interviewee minute for the first 500 respondents, US$2.75 per interviewee minute for the next 500 respondents, US$2.00 per interviewee minute for respondents beyond 1,000, and an additional US$2,000 handling costs per survey.9
Because of the high costs of probability-based panels, they are often used to field multi-client omnibus surveys. Omnibus surveys mean that the survey combines sets of questions from multiple clients and researchers, with respondents answering questions on different topics in one survey. Researchers who would like to use an omnibus may want to consider the following two issues. First, because they are usually long and include different types of questions, respondents may feel the questions are burdensome to answer (Phillips & Stenger, 2022), and this may lead them to break off from the survey. Second, the ordering of sets of questions may influence responses and their quality. If respondents change their answers, even their responses to experimental manipulation, as the result of viewing a prior set of questions, this constitutes a confound (Shadish et al., 2002). Specifically, it is a type of multiple treatment interference and could have severe consequences for the generalizability of the findings. Fortunately, ordering effects like this can be alleviated in the aggregate by randomizing the ordering of the sets of clients’ questions in the omnibus survey. Researchers should confirm with the panel administrators that this sort of randomization is being conducted. They can also request to have a randomization indicator variable included in the metadata provided by the panel administrator.
Online Non-Probability Samples
Like the online probability panels described in the previous section, there are many online non-probability online panels operated by private companies. Web-based professional panels, including those offered by Qualtrics and Lucid, allow the use of quota sampling. This contrasts with YouGov Panel, which is a matched opt-in sample with a more sophisticated sampling design. Researchers may also recruit respondents using convenience sampling via crowdsourcing platforms such as MTurk. There are also emerging tools that help researchers recruit respondents from crowdsourcing platforms such as CloudResearch, and crowdsourcing platforms that primarily focus on academic research tasks such as Prolific Academic.
Qualtrics Panel
Many researchers may already be familiar with Qualtrics because of its suite of tools to design online surveys. In addition to the survey design service, Qualtrics also offers a service to help researchers recruit respondents. Qualtrics maintains a network of pools of research participants, who are recruited from various sources through advertisements on TV, radio, and mobile games; membership referrals; and social networks (Miller et al., 2020). When researchers contract with Qualtrics, they may choose a specific demographic composition of the sample; existing panelists may then receive a targeted survey invitation based on their demographic characteristics on file (Belliveau et al., 2022). Qualtrics assigns a program coordinator to help researchers program the survey (Johnson, 2021). Depending on the requested sample size, data collection via Qualtrics may take anywhere from a few days to a couple of weeks; Belliveau et al. (2022) collected 2,575 responses over 1 month. Criminologists have also used Qualtrics panel to collect survey data (e.g., Bolin et al., 2021; Hazen & Brank, 2022; Hickert et al., 2024; McLean & Nix, 2022; Moule et al., 2019, 2022; Nam et al., 2024; Shi et al., 2022; Silver & Shi, 2023).
Research showed that samples recruited using the Qualtrics Panel were demographically and politically diverse and approximated that of a nationally representative sample on most demographic variables, especially when weights were applied (Boas et al., 2020). Researchers also reported high transparency of Qualtrics when reporting back to the researchers concerning the number of contacts that they sent out and the quota sampling method that they used (Miller et al., 2020). Nonetheless, researchers should note that, by its nature, data from the Qualtrics Panel are not representative of the U.S. population on all possible variables of interest but rather common demographic variables only.
Compared to other services (e.g., Lucid Theorem, Lucid Marketplace), the costs of Qualtrics Panels are relatively high. Boas et al. (2020) paid US$5 per response and were informed the respondents received about one-third of the payment.10 Miller et al. (2020) reported that Qualtrics charged US$6.50 per response. The exact amount paid to respondents is difficult to calculate, as Qualtrics participants are often compensated with coupons, gift cards, and other types of rewards in addition to cash (Douglas et al., 2023).
Lucid Theorem and Marketplace
Lucid has emerged as a new online platform for data collection, especially for researchers with small budgets (Coppock & McClellan, 2019; Graham, 2020). It links interested survey researchers and survey response providers (Coppock & McClellan, 2019). Respondents answer demographic screening questions before entering the survey. Lucid then uses quota sampling to recruit respondents based on the specific request of researchers. According to Lucid,11 their respondents come from a very wide variety of suppliers and are recruited via “ads and promotions across various digital networks, search, word of mouth and membership referrals, social networks, online and mobile games, affiliate marketing, banner ads, offerwalls, TV and radio ads, and offline recruitment with mail campaigns.” Lucid has two options for researchers: Lucid Theorem and Lucid Marketplace. Lucid Theorem is a self-service platform where researchers can launch their Qualtrics surveys and pay US$1.50 for one complete response. Lucid Marketplace assigns a one-on-one survey programmer to researchers, who helps launch the survey, and charges US$2.00 for one complete response, with a US$1,500.00 minimum threshold.12 The early 2020s have seen a variety of criminological research using Lucid samples to understand public opinion about crime and criminal justice (e.g., Drakulich & Denver, 2022; Shi, 2023; Vaughn et al., 2022; Wu, 2023).
A major concern about all online samples, but particularly those from Lucid, is respondent inattentiveness. Analyses by Ternovski and Orr (2022) show that the percentage of respondents passing all attention check questions (ACQs)13 in Lucid surveys declined in 2020, to as low as 15.5%. The authors also showed that respondents who failed ACQs may provide unreliable demographic information and meaningfully different answers to the survey questions. In comparison, some publications using MTurk samples showed a much higher ACQ passing rate; for example, Nix et al. (2021) included two ACQs and had 78% and 91% of the respondents passing each ACQ. An additional concern is that respondents in Lucid Marketplace are not directly compensated by Lucid but instead by their original supplier. It is unclear how problematic this practice is and what proportion of respondents, if any, are going uncompensated for their participation. Researchers using Lucid Marketplace should make inquiries regarding this practice if they, or their organization’s institutional review board, have concerns regarding the practice.
YouGov Audience Panel
The YouGov Audience Panel is a matched-in sample, which differs from the previously mentioned professional panels in that it uses a two-stage sample selection method: in the first stage, it uses strata proportional in size to the U.S. population to sample its active panelists based on their demographic characteristics including age, gender, race, and education; in the second stage, the sample is matched using a synthetic sampling frame (Ansolabehere & Rivers, 2013). A synthetic sampling frame is a subsample from larger samples that reflect the demographic distribution for the target population and is used as a template for the analytic sample; in the case of YouGov, the synthetic sampling frame (SSF) is drawn from the American Community Survey (Mercer et al., 2017). Sampling matching may lower errors that occur during sample selection through a model-based approach (Mercer et al., 2018). YouGov also provides weights based on propensity score matching and post-stratification to further adjust for matching; some criminological publications using YouGov panels have used weights provided by the platform (e.g., Metcalfe & Pickett, 2022; Socia et al., 2021; Wozniak et al., 2022).
Matched opt-in samples such as YouGov are approximately twice as expensive as Qualtrics samples and are many times more expensive than other crowdsourcing platforms. Moreover, researchers have noted that for some surveys, although YouGov generates the sample and administers the survey, the panelists may come from other vendors and sources (Enns, 2022). As such, researchers may want to have clear communication with the program manager about the source of the sample and pricing before contracting the service.
MTurk
MTurk is a crowdsourcing platform or an online labor market where people can request and pay registered workers to complete a variety of human intelligence tasks (i.e., HITs; see Buhrmester et al., 2011). Researchers can promote their online survey as a HIT and recruit respondents via MTurk using convenience sampling.14 Importantly, unlike some other services, when using MTurk researchers set the price point for completion. Payment via MTurk goes to the workers, with Amazon receiving a percentage of the proceeds. Horton and Chilton (2010) found that the median of the lowest “acceptable” wage on MTurk was US$1.38 per hour, taking into consideration that most HITs were simple tasks and could be completed within a short time. In 2010, Paolacci and co-authors launched a 3-minute survey and paid each participant just US$0.10. While larger amounts of compensation may help recruit hundreds of respondents in hours (Buhrmester et al., 2011), researchers have not found higher payment to be related to data quality for U.S.-based workers (Litman et al., 2015).15 Some researchers have raised ethical concerns that workers do not receive fair pay (Chandler & Shapiro, 2016), although respondents to traditional random digit dialing telephone surveys also typically do not receive any remuneration at all. And even if many workers may accept a HIT at a low rate, researchers may want to maintain a good reputation on the platform by offering higher wages, as workers often exchange their experiences working for a requester via various online forums and blogs (Paolacci et al., 2010).
For researchers who would like to collect a sample from the United States, they may choose to invite workers nationwide to complete the survey. However, respondents are more likely to be younger, be politically more liberal (Berinsky et al., 2012), be female, and be more educated, as well as have a lower income than the general U.S. population (Paolacci et al., 2010). Black and Hispanic populations are also underrepresented among workers (Levay et al., 2016). If the selection bias of the sample is related to the outcome variable of the study, this may threaten the validity and statistical inference of the study (Pasek, 2016).
Another common concern for researchers using MTurk is data quality. Researchers have pointed out that some respondents participated in many surveys (i.e., non-naivete) and might be familiar with survey design and study materials, which might contaminate subsequent study outcomes (Chandler et al., 2013, 2015). Respondents might also be distracted and multitask while taking online surveys (Clifford & Jerit, 2014), although this concern is not limited to MTurk alone. Some researchers recommend restricting the survey to only workers who have a higher than 95% approval rate in previous HITs (Peer et al., 2014), while others do not find evidence that this status is associated with worker performance (Rouse, 2020). Finally, data quality on MTurk, and indeed all online platforms, may change over time; for example, Chmielewski and Kucker (2020) documented a panic in 2018 when researchers found computer programs (“bots”) could automatically complete HITs and workers could bypass location restrictions, leading to lower quality of data if researchers did not use screeners or validity checks.
Although MTurk samples have several limitations associated with convenience sampling, it is still gaining popularity among criminologists, possibly due to its relative inexpensiveness and fast turnaround in data collection (Thompson & Pickett, 2020). A review of studies done via MTurk shows that with 7,300 workers available for HITs at any given time, most HITs can be completed within half a day (Aguinis et al., 2021). Researchers also find that respondents recruited via MTurk can pay attention to experimental stimuli (Berinsky et al., 2012) and produce quality data on established tasks (Shank, 2016). Perhaps most importantly, convenience samples collected via MTurk are highly likely to be more demographically diverse than college student convenience samples (Berinsky et al., 2012) and convenience samples recruited via social media postings (Casler et al., 2013).
CloudResearch
CloudResearch (formerly TurkPrime) developed a service that interfaces with MTurk to make it easier for researchers to use. These improvements include
excluding participants on the basis of previous participation, longitudinal studies, making changes to a study while it is running, automating the approval process, increasing the speed of data collection, sending bulk e-mails and bonuses, enhancing communication with participants, monitoring dropout and engagement rates, providing enhanced sampling options . . .
(Litman et al., 2017, p. 432)16
CloudResearch uses screener questions to run both an MTurk Toolkit, which vets respondents directly from MTurk, and a PrimePanel, which recruits respondents from MTurk and other panel data sources (Berry et al., 2022). Berry et al. (2022) compared results from PrimePanel, a vetted MTurk sample, and two non-probability panels (Qualtrics and Kantar), and found the vetted MTurk sample produced higher quality data overall.
Douglas et al. (2023) collected data from MTurk, CloudResearch, Prolific, and Qualtrics, as well as a college student sample. The authors reported that the cost per participant was lowest for CloudResearch and Prolific (approximately US$2.00), doubled for MTurk (US$4.00), and quadrupled for Qualtrics (US$8.00). Respondents recruited via CloudResearch and Prolific yielded data of higher quality in the study as well, including a higher likelihood of passing ACQs and following survey instructions. Litman et al. (2021) and Peer et al. (2021) also compared data collected via CloudResearch and other online platforms. However, both sets of authors were affiliated with CloudResearch and Prolific, respectively, and readers should be aware of this potential conflict of interest.
Prolific Academic
Prolific is an online crowdsourcing platform that maintains its own participant pool and allows researchers to screen participants’ characteristics for their study (Peer et al., 2017). Unlike MTurk, which promotes various tasks, Prolific primarily caters to research needs such as online experiments and has clear rules about payment and pre-screening filters, as well as the option for longitudinal studies (Palan & Schitter, 2018). Prolific collects participants’ demographic information including age, gender, and nationality during the registration stage and researchers can use this information to screen participants (Lee et al., 2022) and as part of the data analysis (Marreiros et al., 2017). As Prolific is based in the United Kingdom, many studies using Prolific service have recruited respondents from this country. For example, Anwyl-Irvine et al. (2020) paid their participants £8.70 per hour (approximately US$10.60). However, recruitment of U.S.-based participants is also possible; for example, Lee et al. (2022) paid respondents US$6 for a 40-minute-long survey.17 Researchers may also request the sample demographics to represent the population distribution.
Considerations When Choosing an Online Survey Platform
Comparing different types of online non-probability samples, a 2016 Pew Research Center report argues that not all non-probability samples perform equally well. Rather, platforms with more sophisticated sampling procedures (e.g., matched opt-in samples) tend to yield more accurate results (Kennedy et al., 2016). Zack et al. (2019) found that the Qualtrics Panel outperformed MTurk when considering univariate and multivariate observational social science research. However, there are also examples that show MTurk data outperforming panel data (Kees et al., 2017) and being more representative than quota samples (Redmiles et al., 2019). Examining variables related to criminal justice and criminology, researchers find that regression results derived from convenience samples and quota samples tend to produce relations that are similar in direction but different in magnitude, compared to results from nationally representative samples (Thompson & Pickett, 2020). In contrast, the matched opt-in sample (e.g., YouGov) tends to produce regression results that are both similar in direction and comparable in magnitude with results from probability-based samples in criminological research (Graham et al., 2021).
Survey researchers comparing online probability and non-probability samples have found that the non-probability samples tend to have more bias and yield less accurate results, although crucially they examined univariate estimates exclusively (MacInnis et al., 2018; Yeager et al., 2011). Because all panelists or respondents in non-probability samples are volunteers, the non-probability samples are more likely to recruit respondents who are more interested in volunteering and problem-solving in their communities (Kennedy et al., 2016). And because the non-probability samples are less representative of the diverse population, estimates based on racial and ethnic minorities tend to be less accurate (Kennedy et al., 2016). Findings from non-probability samples have low external validity and generalizability, especially regarding observational findings (Thompson & Pickett, 2020). It should be noted that even among probability sampling designs, respondents are ultimately participating voluntarily, and thus self-selection can still be problematic, especially given that the initial recruitment response rate usually varies between 5% and 15% (American Association for Public Opinion Research, 2022). The issue here is whether someone is self-selecting into the sampling frame in the first place.
Despite the limitation of non-probability samples, the advantage of online non-probability samples is that prices, especially those of the unmatched opt-in samples, are much more affordable for individual researchers. The demographic composition of online non-probability samples also tends to be more diverse compared to the typical in-person non-probability samples used in criminology and criminal justice research (e.g., college students). This can help researchers recruit a diverse nationwide sample, which may yield valid results, particularly for those conducting experimental research (Weinberg et al., 2014). Simmons and Bobo (2015) also showed that the differences in results from online non-probability (YouGov) and probability samples were usually not substantial. Non-probability samples may be especially useful for researchers conducting experiments, as researchers have found that results from survey experiments usually generate treatment effects that are comparable to those found in probability samples (Berinsky et al., 2012; Coppock, 2019; Coppock & McClellan, 2019; Coppock et al., 2018; Mullinix et al., 2015).
Note that the platforms and panels introduced herein are not exhaustive; for example, criminologists have also used other professional panels, such as the SurveyMonkey Audience Panel (e.g., Pickett et al., 2013; Shi, 2022). Ultimately, when choosing the appropriate online survey platform, researchers must take into consideration their research design, the affordability and speed of data collection, and the quality of data, as well as what (if any) specific demographic composition is preferred or needed. Often, these considerations have a substantial bearing on one another (Dillman et al., 2014). Researchers should also visit the official website of the services and contact the platform for updated price quotes and sampling methods.
Further Reading
- American Association for Public Opinion Research. (2022). Data quality metrics for online samples: Considerations for study design and analysis.
- Ansolabehere, S., & Schaffner, B. F. (2014). Does survey mode still matter? Findings from a 2010 multi-mode comparison. Political Analysis, 22(3), 285–303.
- Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368.
- Chang, L., & Krosnick, J. A. (2009). National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), 641–678.
- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons.
- Evans, J. R., & Mathur, A. (2005). The value of online surveys. Internet Research, 15(2), 195–219.
- Krosnick, J. A., Presser, S., Fealing, K. H., Ruggles, S., & Vannette, D. L. (2015). The future of survey research: Challenges and opportunities. The National Science Foundation Advisory Committee for the Social, Behavioral and Economic Sciences Subcommittee on Advancing SBE Survey Research, 1–15.
- Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138.
- Olson, K., Smyth, J. D., Horwitz, R., Keeter, S., Lesser, V., Marken, S., Mathiowetz, N. A., McCarthy, J. S., O’Brien, E., Opsomer, J. D., Steiger, D., Sterrett, D., Su, J., Suzer-Gurtekin, Z. T., Turakhia, C., & Wagner, J. (2021). Transitions from telephone surveys to self-administered and mixed-mode surveys: AAPOR Task Force report. Journal of Survey Statistics and Methodology, 9(3), 381–411.
- Simmons, A. D., & Bobo, L. D. (2015). Can non-full-probability Internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387.
- Thompson, A. J., & Pickett, J. T. (2020). Are relational inferences from crowdsourced and opt-in samples generalizable? Comparing criminal justice attitudes in the GSS and five online samples. Journal of Quantitative Criminology, 36, 907–932.
- Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The science of web surveys. Oxford University Press.
- Tourangeau, R., Groves, R. M., Kennedy, C., & Yan, T. (2009). The presentation of a web survey, nonresponse and measurement error among members of web panel. Journal of Official Statistics, 25(3), 299–321.
- Walter, S. L., Seibert, S. E., Goering, D., & O’Boyle, E. H. (2019). A tale of two sample sources: Do results from online panel data and conventional data converge? Journal of Business and Psychology, 34, 425–452.
- Weinberg, J. D., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310.
References
- Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837.
- American Association for Public Opinion Research. (2022). Data quality metrics for online samples: Considerations for study design and analysis.
- Ansolabehere, S., & Rivers, D. (2013). Cooperative survey research. Annual Review of Political Science, 16, 307–329.
- Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020). Gorilla in our midst: An online behavioral experiment builder. Behavior Research Methods, 52, 388–407.
- Barlas, F. M. (2021). Representative research: Assessing diversity in online samples? [Webinar]. University of Michigan.
- Belliveau, J., Soucy, K. I., & Yakovenko, I. (2022). The validity of Qualtrics panel data for research on video gaming and gaming disorder. Experimental and Clinical Psychopharmacology, 30(4), 424–431.
- Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368.
- Berinsky, A. J., Margolis, M. F., & Sances, M. W. (2014). Separating the shirkers from the workers? Making sure respondents pay attention on self‐administered surveys. American Journal of Political Science, 58(3), 739–753.
- Berry, C., Kees, J., & Burton, S. (2022). Drivers of data quality in advertising research: Differences across MTurk and professional panel samples. Journal of Advertising, 51(4), 515–529.
- Boas, T. C., Christenson, D. P., & Glick, D. M. (2020). Recruiting large online samples in the United States and India: Facebook, Mechanical Turk, and Qualtrics. Political Science Research and Methods, 8(2), 232–250.
- Bolin, R. M., Applegate, B. K., & Ouellette, H. M. (2021). Americans’ opinions on juvenile justice: Preferred aims, beliefs about juveniles, and blended sentencing. Crime & Delinquency, 67(2), 262–286.
- Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5.
- Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160.
- Chandler, J., Paolacci, G., & Mueller, P. (2013). Risks and rewards of crowdsourcing marketplaces. In P. Michelucci (Ed.), Handbook of human computation (pp. 377–392). Springer.
- Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139.
- Chandler, J., & Shapiro, D. (2016). Conducting clinical research using crowdsourced convenience samples. Annual Review of Clinical Psychology, 12, 53–81.
- Chang, L., & Krosnick, J. A. (2009). National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), 641–678.
- Chmielewski, M., & Kucker, S. C. (2020). An MTurk crisis? Shifts in data quality and the impact on study results. Social Psychological and Personality Science, 11(4), 464–473.
- Clifford, S., & Jerit, J. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1(2), 120–131.
- Coppock, A. (2019). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research and Methods, 7(3), 613–628.
- Coppock, A., Leeper, T. J., & Mullinix, K. J. (2018). Generalizability of heterogeneous treatment effect estimates across samples. Proceedings of the National Academy of Sciences of the United States of America, 115(49), 12441–12446.
- Coppock, A., & McClellan, O. A. (2019). Validating the demographic, political, psychological, and experimental results obtained from a new source of online survey respondents. Research & Politics, 6(1), 1–14.
- Denver, M., Pickett, J. T., & Bushway, S. D. (2017). The language of stigmatization and the mark of violence: Experimental evidence on the social construction and use of criminal record stigma. Criminology, 55(3), 664–690.
- Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). John Wiley & Sons.
- Douglas, B. D., Ewell, P. J., & Brauer, M. (2023). Data quality in online human-subjects research: Comparisons between MTurk, Prolific, CloudResearch, Qualtrics, and SONA. PLOS ONE, 18(3), e0279720.
- Drakulich, K., & Denver, M. (2022). The partisans and the persuadables: Public views of black lives matter and the 2020 protests. Perspectives on Politics, 20(4), 1191–1208.
- Enns, P. K. (2022). Do you know where your survey data come from?
- Evans, J. R., & Mathur, A. (2005). The value of online surveys. Internet Research, 15(2), 195–219.
- Graham, A., Pickett, J. T., & Cullen, F. T. (2021). Advantages of matched over unmatched opt-in samples for studying criminal justice attitudes: A research note. Crime & Delinquency, 67(12), 1962–1981.
- Graham, M. H. (2020). Self-awareness of political knowledge. Political Behavior, 42(1), 305–326.
- Greaves, L. (2017). An investigation into the feasibility of an online national probability panel study in New Zealand. University of Auckland Press.
- Hazen, K. P., & Brank, E. M. (2022). Do you hear what I hear? A comparison of police officer and civilian fairness judgments through procedural justice. Psychology, Crime & Law, 28(2), 153–178.
- Hickert, A., Shi, L., & Silver, J. R. (2024). Is compassion the flip side of punitiveness? Incorporating COVID-19 crisis in experimental vignettes to examine support for visitation and vaccination in prison. Journal of Experimental Criminology, 20, 1–22.
- Hill, T. D., Dowd-Arrow, B., Ellison, C. G., Garcia-Alexander, G., Bartkowski, J. P., & Burdette, A. M. (2021). Sexual dysfunction and gun ownership in America: When hard data meet a limp theory. American Journal of Men’s Health, 15(5), 1–11.
- Hill, T. D., Dowd-Arrow, B., Ellison, C. G., Hale, L., McFarland, M., & Burdette, A. M. (2022). Gun ownership, community stress, and sleep disturbance in America. Sleep Health, 8(2), 161–166.
- Horton, J. J., & Chilton, L. B. (2010, June). The labor economics of paid crowdsourcing [Cambridge, Massachusetts, USA.]. Proceedings of the 11th ACM Conference on Electronic Commerce.
- Igo, S. E. (2007). The averaged American: Surveys, citizens, and the making of a mass public. Harvard University Press.
- Iverson, K. M., Dardis, C. M., Cowlishaw, S., Webermann, A. R., Shayani, D. R., Dichter, M. E., Mitchell, K. S., Mattocks, K. M., Gerber, M. R., & Portnoy, G. R. (2022). Effects of intimate partner violence during COVID-19 and pandemic-related stress on the mental and physical health of women veterans. Journal of General Internal Medicine, 37(Suppl. 3), 724–733.
- Jipguep-Akhtar, M. C., Dickerson, T., & Bradley, D. (2021). When crises collide—Policing a pandemic during social unrest. Journal of Community Safety and Well-Being, 6(3), 97–103.
- Johnson, L. (2021). Exploring factors associated with pregnant women’s experiences of material hardship during COVID-19: A cross-sectional Qualtrics survey in the United States. BMC Pregnancy and Childbirth, 21(1), 1–14.
- Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s Mechanical Turk. Journal of Advertising, 46(1), 141–155.
- Kennedy, C., Mercer, A., Keeter, S., Hatley, N., McGeeney, K., & Giminez, A. (2016). Evaluating online nonprobability surveys. Pew Research Center.
- Kravitz-Wirtz, N., Aubel, A., Schleimer, J., Pallin, R., & Wintemute, G. (2021). Public concern about violence, firearms, and the COVID-19 pandemic in California. JAMA Network Open, 4(1), e2033484.
- Lee, S. J., Ward, K. P., Lee, J. Y., & Rodriguez, C. M. (2022). Parental social isolation and child maltreatment risk during the COVID-19 pandemic. Journal of Family Violence, 37, 813–824.
- Levay, K. E., Freese, J., & Druckman, J. N. (2016). The demographic and political composition of Mechanical Turk samples. SAGE Open, 6(1), 1–17.
- Litman, L., Moss, A., Rosenzweig, C., & Robinson, J. (2021). Reply to MTurk, prolific or panels? Choosing the right audience for online research. SSRN.
- Litman, L., Robinson, J., & Abberbock, T. (2017). TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences. Behavior Research Methods, 49(2), 433–442.
- Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on Mechanical Turk. Behavior Research Methods, 47(2), 519–528.
- MacInnis, B., Krosnick, J. A., Ho, A. S., & Cho, M. J. (2018). The accuracy of measurements with probability and nonprobability survey samples: Replication and extension. Public Opinion Quarterly, 82(4), 707–744.
- Marreiros, H., Tonin, M., Vlassopoulos, M., & Schraefel, M. C. (2017). “Now that you mention it”: A survey experiment on information, inattention and online privacy. Journal of Economic Behavior & Organization, 140, 1–17.
- McLean, K., & Nix, J. (2022). Understanding the bounds of legitimacy: Weber’s facets of legitimacy and the police empowerment hypothesis. Justice Quarterly, 39(6), 1287–1309.
- Mercer, A., Lau, A., & Kennedy, C. (2018). For weighting online opt-in samples, what matters most. Pew Research Center.
- Mercer, A. W., Kreuter, F., Keeter, S., & Stuart, E. A. (2017). Theory and practice in nonprobability surveys: Parallels between causal inference and survey inference. Public Opinion Quarterly, 81(S1), 250–271.
- Metcalfe, C., & Pickett, J. T. (2018). The extent and correlates of public support for deterrence reforms and hot spots policing. Law & Society Review, 52(2), 471–502.
- Metcalfe, C., & Pickett, J. T. (2022). Public fear of protesters and support for protest policing: An experimental test of two theoretical models. Criminology, 60(1), 60–89.
- Miller, C. A., Guidry, J. P., Dahman, B., & Thomson, M. D. (2020). A tale of two diverse Qualtrics samples: Information for online survey researchers. Cancer Epidemiology, Biomarkers & Prevention, 29(4), 731–735.
- Moule, R. K., Jr., Burruss, G. W., Jaynes, C. M., Weaver, C., & Fairchild, R. (2022). Concern, cynicism, and the coronavirus: Assessing the influence of instrumental and normative factors on individual defiance of COVID-19 mitigation guidelines. Crime & Delinquency, 68(8), 1320–1346.
- Moule, R. K., Jr., Fox, B. H., & Parry, M. M. (2019). The long shadow of Ferguson: Legitimacy, legal cynicism, and public perceptions of police militarization. Crime & Delinquency, 65(2), 151–182.
- Mullinix, K. J., Leeper, T. J., Druckman, J. N., & Freese, J. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2(2), 109–138.
- Nam, Y., Wolfe, S. E., & Nix, J. (2024). Does procedural justice reduce the harmful effects of perceived ineffectiveness on police legitimacy? Journal of Research in Crime and Delinquency, 61(1), 124–167.
- Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. Journal of the Royal Statistical Society, 97, 558–606.
- Nguyen, K. H., Lu, P., Meador, S., Hung, M. C., Kahn, K., Hoehner, J., Razzaghi, H., Black, C., & Singleton, J. A. (2021). Comparison of COVID-19 vaccination coverage estimates from the household pulse survey, omnibus panel surveys, and COVID-19 vaccine administration data, United States, March 2021. Centers for Disease Control and Prevention.
- Nguyen, K. H., Srivastav, A., Razzaghi, H., Williams, W., Lindley, M. C., Jorgensen, C., Abad, N., & Singleton, J. A. (2021). COVID‐19 vaccination intent, perceptions, and reasons for not vaccinating among groups prioritized for early vaccination—United States, September and December 2020. American Journal of Transplantation, 21(4), 1650–1656.
- Nix, J., Ivanov, S., & Pickett, J. T. (2021). What does the public want police to do during pandemics? A national experiment. Criminology & Public Policy, 20(3), 545–571.
- Olson, K., Smyth, J. D., Horwitz, R., Keeter, S., Lesser, V., Marken, S., Mathiowetz, N. A., McCarthy, J. S., O’Brien, E., Opsomer, J. D., Steiger, D., Sterrett, D., Su, J., Suzer-Gurtekin, Z. T., Turakhia, C., & Wagner, J. (2021). Transitions from telephone surveys to self-administered and mixed-mode surveys: AAPOR Task Force report. Journal of Survey Statistics and Methodology, 9(3), 381–411.
- Palan, S., & Schitter, C. (2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.
- Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5(5), 411–419.
- Pasek, J. (2016). When will nonprobability surveys mirror probability surveys? Considering types of inference and weighting strategies as criteria for correspondence. International Journal of Public Opinion Research, 28(2), 269–291.
- Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163.
- Peer, E., Rothschild, D. M., Evernden, Z., Gordon, A., & Damer, E. (2021). Data quality of platforms and panels for online behavioral research. SSRN.
- Peer, E., Vosgerau, J., & Acquisti, A. (2014). Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 46, 1023–1031.
- Phillips, A., & Stenger, R. (2022). The effect of burdensome survey questions on data quality in an omnibus survey. Journal of Official Statistics, 38(4), 1019–1050.
- Pickett, J. T., Ivanov, S., & Wozniak, K. H. (2020). Selling effective violence prevention policies to the public: A nationally representative framing experiment. Journal of Experimental Criminology, 18, 387–409.
- Pickett, J. T., Mancini, C., & Mears, D. P. (2013). Vulnerable victims, monstrous offenders, and unmanageable risk: Explaining public opinion on the social control of sex crime. Criminology, 51(3), 729–759.
- Redmiles, E. M., Kross, S., & Mazurek, M. L. (2019). How well do my results generalize? Comparing security and privacy survey results from MTurk, web, and telephone samples. In 2019 IEEE symposium on security and privacy (SP) (pp. 1326–1343). IEEE.
- Rouse, S. V. (2020). Reliability of MTurk data from masters and workers. Journal of Individual Differences, 41(1), 30–36.
- Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Cengage Learning.
- Shank, D. B. (2016). Using crowdsourcing websites for sociological research: The case of Amazon Mechanical Turk. The American Sociologist, 47(1), 47–55.
- Shi, L. (2022). Public opinion about# MeToo victims and offenders: A nationwide experiment. Violence Against Women, 28(12–13), 3154–3173.
- Shi, L. (2023). Is cross-group friendship related to less perceived criminal threat of immigrants and less negative attitudes toward immigrants to the United States? International Criminal Justice Review.
- Shi, L., Silver, J., & Hickert, A. (2022). Conceptualizing and measuring public stigma towards people with prison records. Criminal Justice and Behavior, 49(11), 1676–1698.
- Silver, J. R., & Shi, L. (2023). Punishing protesters on the “other side”: Partisan bias in public support for repressive and punitive responses to protest violence. Socius, 9, 1–16.
- Simmons, A. D., & Bobo, L. D. (2015). Can non-full-probability Internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387.
- Socia, K. M., Rydberg, J., & Dum, C. P. (2021). Punitive attitudes toward individuals convicted of sex offenses: A vignette study. Justice Quarterly, 38(6), 1262–1289.
- Ternovski, J., & Orr, L. (2022). A note on increases in inattentive online survey-takers since 2020. Journal of Quantitative Description: Digital Media, 2, 1–35.
- Thompson, A. J., & Pickett, J. T. (2020). Are relational inferences from crowdsourced and opt-in samples generalizable? Comparing criminal justice attitudes in the GSS and five online samples. Journal of Quantitative Criminology, 36, 907–932.
- University of Michigan. (2019). National poll on healthy aging, 2019.
- Vaughn, P. E., Peyton, K., & Huber, G. A. (2022). Mass support for proposals to reshape policing depends on the implications for crime and safety. Criminology & Public Policy, 21(1), 125–146.
- Weinberg, J. D., Freese, J., & McElhattan, D. (2014). Comparing data characteristics and results of an online factorial survey between a population-based and a crowdsource-recruited sample. Sociological Science, 1, 292–310.
- Wojcik, S., & Hughes, A. (2019). Sizing up Twitter users. Pew Research Center.
- Wozniak, K. H., Pickett, J. T., & Brown, E. K. (2022). Judging hardworking robbers and lazy thieves: An experimental test of act- vs. person-centered punitiveness and perceived redeemability. Justice Quarterly, 39(7), 1565–1591.
- Wu, S. (2023). Public opinion on compensating different types of wrongful convictions. State University of New York at Albany.
- Yeager, D. S., Krosnick, J. A., Chang, L., Javitz, H. S., Levendusky, M. S., Simpser, A., & Wang, R. (2011). Comparing the accuracy of RDD telephone surveys and Internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly, 75(4), 709–747.
- Zack, E. S., Kennedy, J., & Long, J. S. (2019). Can nonprobability samples be used for social science research? A cautionary tale. Survey Research Methods, 13(2), 215–227.
Notes
1. For accessing the Ipsos Omnibus service, readers may visit https://www.ipsos.com/en-us/solutions/public-affairs/knowledgepanel-omnibus
2. For a comprehensive understanding of the sampling and recruitment process, readers may visit https://www.ipsos.com/sites/default/files/ipsosknowledgepanelmethodology.pdf
3. For a technical overview of the specific sampling methods, readers should refer to the following report: https://amerispeak.norc.org/content/dam/amerispeak/research/pdf/AmeriSpeak%20Technical%20Overview%202019%2002%2018.pdf
4. For accessing the platform, readers may visit https://amerispeak.norc.org/us/en/amerispeak/our-capabilities/amerispeak-omnibus.html
5. For more information about Understanding America Study (UAS), readers may visit https://uasdata.usc.edu/index.php. For more information about the American Life Panel (ALP), readers may visit https://www.rand.org/research/data/alp.html.
6. Please note that the IPSOS KnowledgePanel was previously part of the GfK Group (formerly known as Knowledge Networks). Criminologists have commissioned the GfK Group to collect data from nationally representative samples in the United States (e.g., Denver et al., 2017; Metcalfe & Pickett, 2018; Pickett et al., 2020).
7. IPSOS defines such a question as an individual question unit. For a question with a checklist, it can include up to 10 response categories; for a question with a rating scale, it can include up to four attribute statements. The rates were obtained in mid-2023.
8. The cost of AmeriSpeak’s close-ended questions is reduced to US$800 each after the first three questions. Customers may get the respondents’ demographic information, including age, gender, education, race, ethnicity, and income with no additional costs (Greaves, 2017).
9. Please note that these quotes are costs for single survey questions. The costs for a complete survey design would be much higher.
10. In comparison, the authors paid US$1.38 per response when recruiting respondents via MTurk, which included a 40% overhead fee paid to Amazon.com.
11. https://support.lucidhq.com/s/article/Sample-Sourcing-FAQs
12. Information about Lucid survey methodology can be found via Lucid websites. Information on pricing is learned via communication with Lucid sales representatives and support teams. Pricing may have changed since the spring of 2021 when the information was gathered.
13. ACQs help researchers check if respondents are reading the survey questions carefully; for researchers conducting experiments, it is particularly important to know if respondents are picking up the experimental stimuli (Nix et al., 2021). Other researchers have suggested that researchers conduct analyses using both full samples and samples stratified on attention (Berinsky et al., 2014). If only a small number of respondents can pass the ACQs, it means that researchers may need to collect a larger sample than budgeted, so that the stratified sample on attention can have a sufficient number of respondents and enough statistical power for a supplementary analysis.
14. For a detailed description of the procedure, readers may refer to Shank (2016).
15. A higher payment may increase the data quality for India-based workers (Litman et al., 2015).
16. For how to set up a research study using CloudResearch, readers may read through Litman et al. (2017). For pricing and other services (such as PrimePanels) offered by CloudResearch.
17. For more information about payment and participant characteristics, readers may visit https://www.prolific.co/researchers