Study Duration

To reiterate, experience sampling employs repeated measures. This means participants complete the same measures multiple times over a given period. For a daily experience sampling study, that period is often every day for one to three weeks. There are studies that have run for four weeks, but they are taxing on participants (Ilies & Judge, 2002; Koopman, Rosen, et al., 2020; McClean et al., 2021). As we discuss in the subsequent paragraph, two weeks seems accepted as the prototypical duration (e.g., Dimotakis et al., 2011; Gabriel et al., 2021; Scott & Barnes, 2011). However, there are some reasons to prefer three weeks if possible (e.g., Koopman, Lin, et al., 2020; Koopman et al., in press; Lanaj et al., 2016; Yang et al., 2016). Importantly, this decision about how many weeks has conceptual and empirical implications.

Conceptually, at issue is that experience sampling is a way to assess participants’ “lived-through” experience (Weiss & Rupp, 2011, p. 83). To do so, the study must be conducted (a) during a “normal,” “typical,” or an otherwise representative window of each participant’s life, and (b) for long enough to capture a generalizable picture of the same. These are obviously conditions that cannot be directly measured. For the former, typically the authors must make an informed assumption (e.g., if the sample is U.S. accountants, then April may not be a good time for a study, though even here that would depend on the phenomenon of interest). For the latter, while authors generally do not explicitly justify the duration of their study, when they do, it typically involves citing Reis and Wheeler (1991) who stated that “we have found that 1 to 2 weeks is the optimal record-keeping duration” (p. 286) and later said that “the 2-week record-keeping period is assumed to represent a stable and generalizable estimate of social life” (p. 289). Note that these authors were primarily interested in socializing patterns among college students, and not work interactions, thus there is nothing inherently definitive or authoritative in that particular assertion. This is one reason why a three-week design might be beneficial, as this is a longer window of participants’ lives. Similarly, this is why one-week designs might be limited, though this does not mean such designs are not publishable (e.g., Bakker & Xanthopoulou, 2009; Matta, Scott, et al., 2020; Sonnentag, 2003).

Empirically, aside from statistical power (i.e., the more weeks of data collected, the more data points for the analysis), there is another consideration which is that the analysis of experience sampling data typically involves estimating person-level intercepts and slopes (i.e., random effects). The estimation of these effects is heavily dependent on the number of within-individual data points. For a more complete explanation of this issue, please consult Snijders and Bosker (2012). For models of even moderate complexity, the number of random effects to be estimated can be significant. Thus, having as high an average number of such points as possible is desirable (hence the added value of a third week of data collection).

Surveys Per Day

Beyond the number of days, researchers must also decide how many surveys per day to require of participants, and when to send out such survey(s). This is a critical decision. Notwithstanding the logistical elements this decision influences (e.g., more daily surveys generally equate to greater cost and an increase in the difficulty of running the study), it also affects how well the data can test the research question. As an example, if the question involves how start-of-work states influence behavior across the day, this design should probably involve at least two daily surveys. Very broadly speaking, it is useful to think of the day in four distinct windows or “chunks”: start-of-work, mid-work, end-of-work, and after-work. These partitions represent theoretically relevant and operationally well-defined periods of time.

Most commonly, experience sampling studies seem to sample from either two or three of these windows. Administering only one survey per day is risky—if the theoretical model includes mediation, such a design largely mimics a same-time/same-source study, which is conceptually and empirically problematic (Kozlowski, 2009). This is not to say examples do not exist (e.g., Matta et al., 2017; Menges et al., 2017); however, there were arguably mitigating circumstances in both. Matta and colleagues’ contribution focused on the moderation of the first stage effect, whereas Menges and colleagues obtained an objective measure of each participant’s daily performance. It is also common to not exceed three surveys per day, but there are recent studies that—for reasons specific to their research questions—used either four (Anicich et al., 2021) or five (Frank et al., in press) daily surveys. The key here is keeping the surveys short and making sure that the design is relevant to the phenomenon. Also note that for studies using a weekly or longer design, typically participants will do only a single survey and therefore this survey will need to have all study measures on it. If testing mediation with a weekly design, it will be important to separate at least one of the measurement periods from the other two. Thus, it is common to collect at least five to six weeks of data (Matta, Sabey, et al., 2020; Schaubroeck et al., 2018).

Frame of Reference for Daily Measures

This point follows directly from the discussion on the number of surveys, as the time frame over which participants are asked to respond must be in alignment with when the surveys are administered. Participants are asked to consider their experiences, states, or behavior over some period of time. Experiences reflect some environmental condition or an event that has happened. Typically, these are measured by framing the items as “since arriving at work today,” “since the previous survey,” “today at work,” “last night,” “over the past week” (if using a weekly design) and so on. It will usually be some clear and definable period of time that creates explicit temporal boundaries (e.g., Rosen et al., 2016). Behaviors are often worded the same. Indeed, because behavior is often a dependent variable, temporal boundaries are critical to arguing that the actions in question follow the antecedent experience or state. States reflect a person’s cognitive or affective circumstances. Often these are framed as “right now” (e.g., Johnson et al., 2014), but it is not uncommon to frame them similarly as with experiences (e.g., Gabriel et al., 2018). Notably, Puranik et al. (2021) found a correlation of .86 between the same measure framed as “right now” versus “since arriving at work.” While this suggests the wording of state measures may not be much of a concern from an empirical standpoint, this is very important from a conceptual standpoint.

Consider a model such as that from Methot et al. (2021) in which the independent variable (small talk—an experience) is measured as “since arriving at work” and the mediator variable (positive emotion—an affective state) is measured as “at the present moment”—both are collected on the same survey. Theoretically, these authors argued that small talk will be associated with subsequent positive emotion. While the authors could likely also have worded positive emotion as “since arriving at work,” their choice of wording should more effectively separate the independent variable and mediator in the minds of the participant (i.e., an experience they had earlier and a state they feel right now). In fact, had the authors worded positive emotions as “since arriving at work,” they may be less able to respond to a criticism that positive emotions could lead one to engage in small talk, as the measurement time periods would have been concurrent. This issue is arguably alleviated with the wording they adopted, as it would not make sense conceptually to argue that how a person feels “right now” could have influenced their experiences over the past several hours. Similarly, adopting “since arriving at work” for both measures leaves researchers more open to omitted variable problems (maybe there was a work event that led to both higher positive emotions and more small talk). While changing the wording does not, of course, solve the omitted variable problem, it does make it somewhat less likely as the change in temporal frame would likely attenuate the relationship of the omitted variable to at least one of the other two.

Notwithstanding these arguments as to why this approach may be theoretically valid (particularly when worded as the authors did and presuming that the dependent variable is measured on a separate survey or from some alternate source), researchers may still experience some pushback from reviewers who could argue that the measures should have been temporally separated. Yet beyond imposing an additional burden on researchers (and study participants), this design aspect can be potentially damaging to the inferences that can be drawn. Consider the following model: an employee’s enacted helping is associated with positive affect. Helping is measured mid-work at 12 p.m. with the stem “since arriving at work today” (thus covering behavior during the past three to four hours). Positive affect is measured at the end-of-work at 4 p.m. as one’s state “right now.” Thus, this model proposes that one’s behavior between 8 a.m.–12 p.m. affects how that person feels at precisely 4 p.m. Yet note that this design does not include any measurement of what the employee felt or did between 12 p.m. and 4 p.m. This design makes it hard to rule out whether some other experience could have occurred in that intervening time period that was more proximally related to the reported positive affect, and puts the burden on the researchers to be prepared to rule out a myriad of other alternative explanations or contributing factors. And importantly, the greater the within-individual variability in the outcome variable, the more severe this problem would be, as it would illustrate greater susceptibility of this construct to daily antecedent constructs. In contrast, had the authors measured positive affect at 12 p.m. with the same “right now” stem, then the phenomenon and subsequent state would be better linked—this is precisely how these variables were measured by Koopman et al. (2016).

However, this discussion should not be read as indicating that the measurement of an intervening state should always be framed as “right now.” Instead, there is an alternate scenario that is also common. Imagine a model which proposes negative affect is associated with enacted incivility. The authors collect both negative affect and enacted incivility on the same end-of-work survey. If negative affect is measured “right now,” and enacted incivility is measured “since the previous survey,” there is a conceptual problem. Temporally, what occurred “since the previous survey” occurred before “right now,” yet empirically these are modeled in reverse. Perhaps authors could use this argument from Puranik et al. (2021) to say the relationship would have been the same, but it would be better to not be in this situation at all. This leads to a recommendation for study design purposes. When an experience or behavior is thought to affect a subsequent state (and both are measured on the same survey), then it may be better to measure that experience or behavior as retrospective over a previous period of time and that state as “right now.” If a state is thought to affect a subsequent behavior or situation (and both are measured on the same survey), however, then it may be better to measure both as retrospective over a previous period of time.

Alternate Scheduling Approaches

Note the discussion thus far has been about schedule-based (or interval-contingent) sampling. There are alternatives to this (see Dimotakis et al., 2013), but they have become less common over time. For example, surveys can be sent (semi)randomly through the day, but this can create problems particularly for capturing behaviors or experiences, as the timing of surveys is often to ensure that participants have had opportunity to have the experience or enact the behavior in question. Another alternative is trigger-based (or event-contingent) scheduling, in which no fixed number of surveys are given—rather, individuals are asked to fill out a survey any time a triggering event occurs. These may have demand effects and lead to hypothesis guessing, as the particular event needs to be explained to participants so they can complete a survey when the experience occurs. However, it is worth noting that technological advancements such as location tracking or heart rate monitoring can be used to trigger surveys, which may allow for increasingly sophisticated study designs such as sending a survey after interactions with a supervisor or coworker, or when a person is experiencing physiological strain.

Resources (Monetary)

Perhaps one of the most significant constraints associated with experience sampling studies are the resources required to administer them. Monetary resources are often what people think of first, and these costs can be significant. It is not uncommon to pay participants up to $60–$70, and Gabriel et al. (2019) noted that the average study tends to be about 83 people (after attrition, which suggests that researchers should ideally aim for at least a starting sample size of 100–125). Thus, such a study may cost between $4,980 and $8,750, though it is common with experience sampling to pay “per survey,” so not everyone earns the top-line number.

However, this presumes the study will only need self-reports. If other-reports are to be employed as well (e.g., examining spillover or crossover between spouses), then the cost can increase. If the relationship of interest is unidirectional (i.e., an employee whose experiences spillover to their spouse), then it is common to ask spouses only to complete a single daily survey each evening (for which they may be paid up to $25–$30; Lanaj et al., 2018). But an alternative research question may acknowledge the working status of both the employee and spouse (e.g., explicitly focusing on dual income couples and asking the spouse to respond to one or more surveys at work). In this case, both individuals would be recruited as primary participants in the study—a design that necessitates further discussion.

To begin, a question that arises is deciding what the unit of analysis is (i.e., the individual or the dyad). If it is the dyad, then analytically this design is essentially identical to the one where spouses completed only a single evening survey, but instead of paying spouses $25–$30 for those reports, they will have been paid $60–$70. This design thus doubles the top-line number, as well as the sample size. This is not inherently problematic if the budget can accommodate it—indeed, many relationships that occur daily are small, so the added statistical power may be beneficial. In fact, this thinking can be leveraged to increase sample size because a problem with experience sampling can be finding enough people to participate. If a sample is found, then recruiting a working spouse (or even a coworker) to participate as well is an effective way of increasing the sample size. Note that analytically, the employee and other (spouse or coworker) likely cannot be treated as fully independent in the analysis, as they are clustered in their own dyad (i.e., both people can be an “employee” analytically, but as a spousal couple or coworker dyad, they are not entirely independent). This can be addressed analytically with a cross-nested approach or clustered standard errors that account for the nonindependence (for an example; see Yoon et al., 2021).

Resources (Nonmonetary)

Resources reflect more than money, however—indeed, the above discussion also reveals that accessing participants willing to do an experience sampling study is critical. Some authors have used university staff as participants (e.g., Ilies et al., 2010; Koopman, Rosen, et al., 2020). Others use social media to recruit either from their own personal and professional networks (e.g., Butts et al., 2015), or from particular interest groups (e.g., Gabriel et al., 2020). For business schools, a fruitful source is part-time MBA students (e.g., Rosen et al., 2021), particularly for questions involving leadership processes (e.g., Johnson et al., 2014). Plus, using an MBA sample has the added benefit of reducing the financial burden of the study. Beyond that, industry contacts can be quite helpful (e.g., Liu et al., 2017; Spieler et al., 2017; Tang et al., 2022; Uy et al., 2017).

One resource that may be overlooked is having a survey platform to host the study. Many universities have contracts with platforms such as Qualtrics or SurveyMonkey, but even these can vary based on how many surveys can be hosted, how many responses can be obtained, and other relevant considerations. Any limitations of the university contract can create barriers to running these studies—an issue that compounds if the university does not have a contract at all. There are also specific platforms geared towards experience sampling studies such as Expiwell, Piel Survey, or Movisens. However, discussion of these can blur the line between monetary and nonmonetary resources. These platforms can be expensive but can simplify some aspects of running an experience sampling study and allow researchers to collect more than just survey data.

Study Waves

The vast majority of experience sampling studies are conducted in a single wave, which greatly simplifies the administration of the study. But this is not an operational requirement—large experience sampling datasets can be compiled from separate “mini” ESM studies conducted serially with different groups of participants (i.e., the participants are different across studies, but they are participating in the same “study,” albeit at different chronological points in time). For example, the first set of fifty participants complete their surveys during the first three weeks of the study; the second set of fifty participants complete their surveys from the third to fifth week of the study; the third set of fifty complete their surveys from the fifth to seventh week of the study, and so forth. This would be most useful when researchers want to assess not only the “typical” experience of a group of individuals, but how the nature of this relationship might change in response to an ongoing phenomenon. Fu et al. (2021) adopted this technique (what they termed a shingled ESM approach) using four waves of experience sampling data with a one week overlap in between (so Wave 1 was weeks 1 to 3; Wave 2 was weeks 3 to 5, and so forth) in their examination of employee stress responses to the ongoing COVID-19 pandemic.

Use of Other-Reports

There may be good reasons to use other-reports when feasible. For one, introducing another source is a way to alleviate some common-source variance concerns associated with a common rater (Podsakoff et al., 2003)—though, we have more to say on this point in the subsequent paragraphs. Additionally, others can be used to report on some stable aspect of the job that is relevant to the focal employee’s responses (e.g., as a between-individual moderator; Gabriel et al., 2019). Indeed, others have been argued to potentially be a more valid source of measuring employee personality (e.g., Oh et al., 2011). A practical rationale is that it can be desirable to show that the effects being hypothesized are observable in the “real world” (e.g., that employees are noticeably engaging in citizenship, that task performance is higher, etc.). Thus, for scholars used to seeing other-reports in papers on task performance (e.g., Le et al., 2011), citizenship (e.g., Halbesleben et al., 2009), deviant behavior (e.g., Deng et al., 2018), work–family conflict (e.g., Wilson et al., 2018) and so on, it may appear questionable to see experience sampling papers published using self-reports (e.g., Guarana et al., 2021; Liu et al., 2015; Mitchell et al., 2019; Rosen et al., 2021; Sherf et al., 2019). This point is understandable.

Yet it is not constructive to suggest that experience sampling studies that do not include other-reports are lacking in rigor. Rather there are theoretical, empirical, and logistical issues that are often ignored for what at times seem to be a broad critique of experience sampling in general rather than a pointed focus on whether other-reports are appropriate for the phenomenon, research question, and sample at hand. One response to this criticism that has been addressed in prior reviews of experience sampling is that while other-reports are often used in between-individual research to reduce common-source variance attributed to response tendencies, this issue is not as germane for experience sampling data (for an explanation of this point, see Beal, 2015; Gabriel et al., 2019). Beyond this, there are other features of experience sampling which may make the use of other-reports deficient that have not been fully elucidated to this point.

To illustrate, consider the following. Imagine you were asked to report on how much a colleague helps others, in general. You could likely answer this question easily, based on both your own experiences with that person and what you have heard from other colleagues. This is precisely how such a question is intended to be answered. But now extend the thought experiment. Think how well you could answer that question about how much this colleague helps others with the following prompts: over the past six months; over the past one month; over the past one week; over the past day; and over the past four hours. With each successive narrowing of the time frame, you may have found it slightly harder to accurately assess your colleague’s helping behavior as your opportunities to observe that person correspondingly narrowed. Perhaps you have been with that person all morning and could easily report on their behaviors either today or in the past four hours (as would be typical prompts for an experience sampling study). Yet ask yourself—could you have accurately assessed that behavior each day for the past five to fifteen days? Likely the answer is no, as you probably have separate offices and are often engaged in noninterdependent work. Now multiply that by the 80–100 or so employees in the typical experience sampling study and the problem with other-reports should begin to become clear. Simply put, a given coworker’s ability to assess an employee’s behavior over short periods of time each day for one to three weeks in a manner that adequately covers the construct’s content domain is questionable.

This has the potential to be problematic because study participants often want to be helpful (or are reluctant to not at least attempt to provide information as they wish to be paid). Thus, even if they have not observed the employee whose behavior they are supposed to report, they may still answer the questions. For example, for measures of helping, a participant could write something to the effect of “while I haven’t seen Jane today, she is the most helpful person in the office,” and rate each of the items as a “5.” The reverse happens for deviance (i.e., “Jane would never do these things”) along with ratings of “1” for each item. It goes without saying that, as the coworker has not observed Jane today, those reports may or may not be accurate for that day—Jane could have conformed to these typical modes of conduct or had an “off” day that resulted in her being less helpful than usual (and perhaps cyber-loafing more, or being more uncivil than she usually is). Yet if such responses are present in the data, this can make the measurement of the construct invalid (Gabriel et al., 2019).

Now to some extent, this situation described above can be mitigated by asking coworkers about the extent to which they have observed the employee that day, as these cases can perhaps be eliminated from the data. Open ended questions can also be of help here, as illustrated in the example about Jane. However, even this can be unclear because it is difficult to define all the ways in which people become aware of a person’s behavior. Perhaps the coworker did not explicitly observe the focal employee’s helping or deviant behavior, but a third coworker mentioned it. This may be a valid case that could otherwise get screened out. There is also nuance when a coworker has had low to moderate opportunity to observe the employee. At this point, it may be quite difficult to ascertain whether those opportunities correspond with the employee having an opportunity to enact the particular behavior (e.g., helping). For example, those observations could come while the two people were on break, or while the employee was being helped with a task by the coworker. Thus, while the coworker has observed the employee for some time, there was no opportunity for the employee to enact helping behavior of their own. If that coworker responds with “1s” for those questions, the report’s accuracy is unknown. A final concern here is that “opportunity to observe” is not synonymous with “ability to recall.” It is fair to question how much attention people really pay to the behaviors of others at work. Just because a coworker has been in the presence of the focal employee for four hours does not mean that the employee’s behaviors during that time are salient when the coworker is asked to report on them.

Also note that opportunity to observe can also be confounded with the research question. Consider a study on how workload influences subsequent behavior. If an employee has high workload, they may stay isolated in their office, which may reduce opportunities to be observed by their coworker. There are other potential problems here as well. For example, citizenship behavior has been argued to be driven, in part, by impression management motives (which also may vary daily; Klotz et al., 2018). Thus, whether one’s citizenship was visible to others on a given day may be driven by unmeasured variables. Conversely, consider deviant behavior, which has been shown to be often unobservable by others (Carpenter et al., 2017). If the designated other-reporter has not seen this behavior from the focal employee, then it becomes difficult to know if that was because the employee did not engage in that behavior, or just did not see it because this behavior is often not observed. For these reasons, using an other-report has the potential to introduce considerable heterogeneity into the responses that can result in both Type I and Type II errors.

The best scenario for a coworker report is a situation in which the researchers can know for certain that the employees and coworkers have plenty of opportunities to observe each other. For example, Trougakos et al. (2015, p. 230) noted that coworkers should work in “close physical proximity” to the employee (see also Tang et al., 2020). However, such conditions are rare, and reviewers should not expect that authors will have this level of knowledge and control over the ability of coworkers to provide accurate reports. Note also that this situation is likely even more fraught for supervisor reports, as employees and supervisors may often go multiple days without direct interaction. However, this is likely less of a concern regarding reports from one’s spouse/partner with whom they cohabitate, as these individuals do often interact for multiple hours each day (though this is not a given and can be assessed—see our upcoming discussion on this point in the ‘Experience Sampling Study Administration’ section). Obviously, spouses cannot report on employees’ work behaviors, but if behavior at home might proxy (i.e., if an employee’s helpfulness at home can test the mechanism nearly as well as helpfulness at work), this could be seen as appropriate and thus high in both utility and rigor.

In sum, authors are encouraged to obtain other-reports (coworker, supervisor, spouse, etc.) when possible and appropriate. However, there are aspects of experience sampling that not only make these reports difficult to collect, but also may make those reports less appropriate. Thus, a measure of skepticism regarding their accuracy is generally warranted and any perspective that experience sampling is less robust if other-reports are not obtained should be balanced against the theoretical and logistical challenges of obtaining those reports. Ultimately, it is the research question and study context that should drive empirical decisions, and it is often a mistake to discount the one person present each time a behavior in question was enacted—the actual focal employee. Often, they are the only person capable of answering particular questions (for exemplars of this argument, see McClean et al., 2019; Puranik et al., 2021). But even when behavior is potentially observable to others, the discussion to this point reveals the myriad theoretical and logistical complications that arise from using other-reports. For this reason, reviewers and editors are encouraged to recognize that adding other-reports (a) adds to the cost of what is already an expensive data collection method, (b) may not solve an empirical problem, and (c) provides data that may be difficult to interpret, is potentially construct-invalid, and may introduce additional sources of error. Other-reports should be seen as a more conservative test of a hypothesis when present (if they are appropriate to the question and context), but their absence should not, by default, be used as evidence of lack of rigor—rather, study design should be evaluated holistically based on its ability to address the research question.

Sample and Screening

Sample selection issues are not unique to experience sampling—no matter the study design, the sample must be appropriate to the phenomenon of interest and there needs to be an expectation that there will be sufficient variance to analyze. However, some of these issues can become trickier with experience sampling. Consider a study on surface acting towards coworkers. First, if the recruitment process is relatively open (e.g., recruiting through some type of list-serv, social media, alumni network, etc.), it is possible that employees who tend to work on solitary tasks and only interact with coworkers sparingly during the week could sign up. These individuals may be acceptable in a between-individual study that asks the extent to which participants surface act towards coworkers “in general” or even “over the past week,” as they may have sufficient opportunity over those time frames to engage in that behavior, and some employees may be more likely to surface act than others (thus, there should be sufficient between-individual variance). However, an experience sampling study could be problematic, as there may be multiple days in which employees do not interact with their coworkers.

The above scenario can easily occur, as participants can be dispersed among different jobs, organizations, and geographies. Thus, sign-ups should be screened. For example, it is possible that the relationships of interest could differ for workers who are employees of an organization versus independent contractors. If this is a potential concern, independent contractors should be screened out in the sign-up process, or at minimum this information should be collected and examined during data analysis. The same goes for factors such as frequency of coworker or supervisor interactions. Other issues to consider include whether a person has one versus multiple supervisors, whether people work in-person versus remote versus a mix, how many hours per day employees work (and is this stable versus variable), and so on. Researchers must be thoughtful about the sample and how well it fits with the design and research questions. Otherwise, participants can be frustrated by repeatedly responding to inapplicable questions, and researchers will spend money, effort, and time to get data that cannot (or should not) be used.

The point about initial screening also applies to the recruitment of the person providing other-reports (if applicable). If the study design involves a coworker-report, this is the time to emphasize the necessary characteristics of this person (e.g., someone the employee sees or talks to on a regular basis, someone on their work team, their most frequent collaborator at work, etc.). Not all employees will follow these requirements in choosing that person, but this is still a useful place to start. Similar wording may be relevant if asking for supervisor-reports, as employees might have more than one person whom they might consider a supervisor. Whomever the source of the other-report, researchers should be the ones to send the recruitment email instead of providing a link for the employee to pass along, as this eliminates an opportunity for the employee to simply do the survey themselves as the designated other.

Sample Information

At this point, it is important to ensure participants’ schedules fit the design of the study, though note that this part is less important if participants are known to have uniform schedules. That is, researchers should confirm that participants will be where they “should” be when surveys are sent (i.e., at work if the survey should be completed at work, or at home if the survey should be completed at home). There are different ways in which this information can be both collected and implemented in the study design, and these various options are well-represented in the extant literature. It may be enough to ask participants to confirm they work typical hours (i.e., arriving at work between 6 a.m.–10 a.m., leaving work between 3–7 p.m.). Alternatively, participants can be asked to indicate their typical schedule in discrete windows (i.e., asking them if they arrive between 6 a.m.–6:29 a.m., 6:30 a.m.–6:59 a.m., 7 a.m.–7:29 a.m., and so on). This may seem hyper-specific, but there are some benefits (see the next section on ‘Survey Planning’) though this does not imply that studies that do not take this approach are in any way flawed.

Expectations should be reinforced when obtaining this information (i.e., being very explicit about the fact that they will be asked to complete one or more surveys per day for a number of days). To this point, it may be helpful to explain what each daily survey is doing (e.g., “Daily Survey #1 is intended to capture your experiences before starting work for the day,” “Daily Survey #2 is intended to capture your experiences over the first half of your workday,” etc.). By explaining the purpose of each survey, participants may better understand why the surveys are being sent at different times and could even motivate them to complete their surveys as it treats them as a partner in the data collection process (Gabriel et al., 2019). If the design entails surveys to be completed at home each evening, the participant should be asked if they prefer a different email for that survey as not everyone checks their work email at home.

Survey Planning

Some experience sampling designs send surveys with relatively broad availability windows (e.g., 6 a.m.–10 a.m., 10 a.m.–2 p.m., 3 p.m.–7 p.m.). This design is easy to implement, however, there are potential drawbacks. Imagine two participants (one of whom arrives at 7:30 a.m. and the other at 9 a.m.) who each complete that first survey at 9:30 a.m. If that survey intends to capture experiences that happen early in the workday, then this worked for the first participant, but not the second. However, this can be screened for by asking participants when they arrived at work, which allows it to be accounted for during data analysis (i.e., controlling for time at work, or creating a decision rule (say, one hour) and excluding cases for those who have been at work for shorter). Another potential problem with this design arises if a participant does the first survey at 9:30 a.m. and the second survey at 10:15 a.m., as not much time has passed.

An alternative is to create semi-customized schedules for each participant. This requires detailed information about each participant’s day (i.e., it would be important to ask the hyper-specific questions noted in the ‘Sample Information’ section). Also, this requires recognizing that each participant’s day is not necessarily the same (i.e., people may work four days per week, have standing meetings Tuesday at 2 p.m., or have an appointment next Friday at 9 a.m.). Thus, beyond getting the information about each person’s typical schedule, it would be necessary to ask participants about idiosyncratic conflicts during the study window. With this information, it is possible not only to place participants in groups that receive their survey links at different time points (i.e., Group 1 receives their link at 6:45 a.m., Group 2 receives their link at 7:15 a.m., and so on), but also participants can be moved between different groups based on their idiosyncratic conflicts (i.e., Sarah is normally in Group 1, but next Wednesday she will get to work later than normal and so she will be in Group 3 that day).

Regardless of how surveys are scheduled, they should be set to deactivate within a specified amount of time to prevent surveys from being completed out of order (Daniels et al., 2009). Consider a study in which links to the daily surveys are sent at 8 a.m., 12 p.m., and 4 p.m. If the window is the same for each person, the survey has to stay active during the entire window but can at least be closed before the next survey is to be sent. But if surveys are customized to participants, the window can be tightened. For example, the participant can be given a specified amount of time (e.g., two hours) to complete each survey. As such, this participant has until 10 a.m. to finish the first survey, while a different participant who does not arrive at work until 9:30 a.m. would have until 11:30 a.m.

Measure Selection

Experience sampling research has tended to use shortened measures for the daily surveys. This practice originated from the need to keep surveys as short as possible given the repeated measures nature of experience sampling and the fact that items which may make sense for a scale that assesses someone’s behaviors in general (i.e., “goes out of the way to make newer employees feel welcome in the work group”; Lee & Allen, 2002) are not applicable to repeatedly assessing a person’s behavior over the past four hours each day for three weeks. Importantly, the majority of experience sampling research does report estimates of measure internal consistency (Gabriel et al., 2019), and those measures are generally held to the same standard as other research, which suggests that even the typically shortened measures often used with experience sampling are generally reliable. One point to note though is that internal consistency cannot simply be calculated directly from the raw measures due to the nonindependence of the reports. One approach that has been common in the literature is to calculate a daily internal consistency value (as each person should only have a single value per day) and take the average. A more recent alternative that Gabriel et al. (2019) have called attention to, however, is to calculate this value strictly from the within-individual variance of the responses (Geldhof et al., 2014).

Confirmatory factor analyses are also generally presented and held to similar standards as with any other research. Yet these analyses, even if supportive, do not preclude the potential for the selected items to be deficient in terms of capturing the content domain for the construct. One option to alleviate this concern is to follow an approach used by Rosen et al. (2019) in which a supplementary study is conducted that administers the whole measure to a sample and examines the correlation between the short- and long-form. Also, a new procedure from Cortina et al. (2020) provided a more data-driven process for shortening measures. Again, however, the research question should ultimately be the primary driver of these decisions.

Scale Anchors

While it may seem desirable to leave anchors as they were originally specified for the scale, anchors do need to be tailored to the context in which the measure is to be used and in line with the temporal frame of the study. One option is to change anchors to agree/disagree. An alternative sometimes used for behavioral measures specifically is to use counts (i.e., 1 = “0 times,” 2 = “1 time”), or to use anchors such as 1 = “not at all” to 5 = “very much.” What is critical is that authors be thoughtful about how the anchors fit with the context and research question at hand.

Data Structure

Recall that number of surveys per day and the frame of reference for the measures in those surveys lead to how the data is subsequently structured and described. Imagine 100 participants complete three surveys per day for 15 days: 4,500 surveys. However, 4,500 is not necessarily the final sample size—this depends on the research question and analytic needs. First, there are actually two sample sizes that need to be discussed. The first is the number of participants, which in this example is 100. This is referred to as the “Level 2 N.” The second refers to the number of cases for analysis, which will be referred to as the “Level 1 N.” If each survey were to provide the full information needed for the analysis, then the Level 1 N would be 4,500. But this is not common—typically, each survey provides one or more variables for the overall model (i.e., some experience at T1 influences some state at T2, which influences some behavior at T3). In this situation, a complete set of surveys is needed to create a day-level case for the analysis, in which case the Level 1 N is a function not of the number of surveys, but the number of study days (1,500).

Even then, the final usable sample size is not guaranteed to be equal to the total complete days available. For example, if lagged variables are needed (for example, the independent variable was collected the previous night), the sample size might be considerably reduced. Consider a case where a participant fills out all surveys on days 1, 3, and 4. Although three days of data are present, only one case (that was collected on day 4) has the necessary lagged variable from the previous day. A full set of surveys completed on days 1, 2, and 3 would provide two valid observations, while a full set of surveys completed on days 1, 3 and 5 would provide none.

Data Screening

Data screening is not unique to experience sampling; however, given the amounts of money that participants can earn for these studies, the incentives to cheat may be higher. If the recruitment method is relatively open and participants are signing up with web-based email addresses (e.g., Gmail, Yahoo, etc.) then it is critical that researchers have some means for verifying that participants are both unique and who they say they are (for example, employed adults). One option is to not use electronic payment methods (i.e., gift card codes) for these samples. Instead, requiring a home address to send a check or gift card code can make it more onerous for people to register multiple times. Alternative options involve either calling participants to verify their identity (e.g., Greenbaum et al., in press), or requiring participants to sign a tax document (e.g., W-9) to validate their identity—something that is not prohibitive with the increasing availability of electronic document signing programs (and may even be required by some universities). These concerns are lessened for samples that rely on participants from a known entity (i.e., company, university, etc.) and where participants use their company email addresses. Similarly, this is not feasible for online subject pools, as anonymity is fundamental to their design. Additionally, these sites have processes in place to ensure that individuals register only once, though this does not obviate the need to carefully screen those sign-ups. Other methods of data screening such as those suggested by Meade and Craig (2012) can be implemented at this stage as well.

Missing Data

Missing data is commonplace with experience sampling studies. Some “missingness” exists simply due to participants not completing any surveys on a given day. In general, this is not a problem, though some have advocated to only retain people who completed some minimum number of days of the study. Gabriel et al. (2018, p. 92) spoke directly to this point, explaining that “three data points per person are statistically needed to appropriately model within-person relationships.” The point is that when analyzing nested data (which is how these data are structured; Bryk & Raudenbush, 2002; Dimotakis et al., 2013), each person has a regression line representing their intra-individual relationship between the variables. A line cannot be drawn with one case, and for two, that line would fit the data perfectly and thus have no statistical error. Therefore, three has often been used as a cut-off, though there is variance in both whether studies implement a cut-off, as well as what that cut-off is (i.e., there is a trade-off between retaining more data overall versus increasing the stability of the intra-individual relationship by only keeping data from participants who provided a greater number of cases).

Missing data can also occur when participants complete some but not all of the surveys on a given day. For example, participants may get to work late and thus miss a morning survey or might leave work particularly early or late and miss an evening survey. This could also come from a coworker who does not complete a survey (or did not interact with the participant that day). How this data is treated depends on where it is positioned in the research model. It is most common (though not universal) to specify main hypothesized paths with random slopes as can be seen in supplemental syntax provided by Gabriel et al. (2019). If this analysis is conducted as a path analysis (e.g., with Mplus), it will not run with missing data on a predictor (i.e., independent variable or mediator). For missing data on a dependent variable (or, for variables modeled with fixed slopes, which is typical for control variables), the analysis will by default be conducted with a “full-information” algorithm (Newman, 2014). This is preferable to listwise deletion (e.g., Enders, 2010; Graham, 2009; Larsen, 2011), and so has become a common way to handle this situation (e.g., Hill et al., 2020; Lennard et al., 2019, in press; McClean et al., 2021; Rosen et al., 2021; Tang et al., in press; Trougakos et al., 2015).

Experience Sampling or Longitudinal/Latent Growth?

We discussed in the earlier section on ‘Repeated Measures’ that a core assumption behind experience sampling is that the data collection occurs during a “typical” window in people’s lives. This property is critical because otherwise group-mean centering (the typical centering approach used for experience sampling data; Dimotakis et al., 2013) would technically violate causality, as responses from later points in the study influence the deviation of prior responses from the person’s “typical” level. Thus, a critical assumption is that the individual and the within-individual phenomenon under investigation is not inherently changing over the course of the study (i.e., there are fluctuations, which is what makes the phenomenon within-individual, but there is no growth or change). For this reason, longitudinal or latent change approaches are inappropriate for experience sampling data, and reviewers should neither request these methods be employed, nor suggest that the analysis is otherwise flawed for not using these approaches.

In contrast, if the phenomenon is growing or changing, the typical analytic approach for experience sampling data would be inappropriate, as group-mean centering would remove variance that has not yet occurred, which is obviously problematic. In this situation, the data should be examined with an approach that is explicitly designed to detect and predict change or growth patterns. To put it simply, if the research question involves fluctuations around a stable mean, group-mean centering, variance partitioning, and other such typical approaches to analyzing, then experience sampling data should be used. If instead growth is the effect predicted, longitudinal or other change methods should be used. These approaches are not substitutable.

Effect Size and Variance Explained

It is good practice to calculate variance explained in the endogenous variables. However, the effect sizes and corresponding percentages of variance explained in experience sampling are often somewhat small. For this reason, the practical impact of the research could be questioned. Yet this omits the critical realization that this question involves a relationship that occurs between two constructs over a very short period of time (e.g., over the past several hours, right now, etc.). As already discussed, experience sampling studies are conducted during normal and common periods of people’s work lives. It is unlikely that such studies would have large relationships—imagine if every time the person in the office next to you was in a better mood than usual, they immediately came to your office and tried to help with the analyses that you were running. Alternatively, imagine that each time that person felt more angry than usual, they walked office-to-office being rude and uncivil.

These examples are extreme, but illustrate the point. There are myriad contextual and idiosyncratic influences on what employees experience, feel, or do in the tight windows examined in experience sampling that are averaged out when examining the same phenomenon over longer periods of time (e.g., over the past week, over the past month, or in general). Over relatively short periods of time during generalizable periods of an employee’s life, the norm should be small effect sizes, and even here, note that the word “small” may be inappropriate because it is inherently comparing the size of experience sampling effects against those derived from studies with completely different designs (Bosco et al., 2015; Cohen, 1992).

To the extent that these effects are considered small, it should be noted that, for example, the correlation between antihistamine usage and symptom relief is .11 and that between anti-inflammatory drugs and pain relief is .14 (Funder & Ozer, 2019). This leads us to the more important question to ask—not whether these effects are small, but whether they are meaningful. Funder and Ozer (2019, p. 161) provide the statistics noted above in service of making the argument that if a relationship that occurs over a short time period can be reliably shown, “its influence could in many cases be expected to accumulate into important implications over time or across people even if its effect size is seemingly small in any particular instance.” If theoretically-derived relationships can be found in just a few hours, the fact that employees spend upwards of 2,000 hours at work each year suggests the relationship may potentially compound and be important in the long-run (Abelson, 1985; Cascio & Boudreau, 2008; Prentice & Miller, 1992).

In sum, it is good practice for authors to report some index of effect size (e.g., variance-explained) for their hypothesized relationships. Regardless, editors and reviewers are encouraged to refrain from criticizing the size of those effects for not meeting their own implicit standard of how large an effect “should be” in order to be practically meaningful. Often, this criticism is wielded as a bludgeon against which authors have no meaningful counter, given that effect sizes viewed as small are the norm with this research. At a minimum, effect sizes for experience sampling studies should not be compared with effect sizes derived from experiments or studies with a longer frame of reference for the relationships under investigation. These effects are not comparable and so holding experience sampling effects sizes to this standard is unreasonable. Instead, a study that would be helpful is one similar to that of Bosco et al. (2015) which categorizes the types of relationships that have been examined and empirically indexes “typical” effect sizes.