Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence

Christa L. Lilly, Luciana C. Braga, Emily Jones, Sean Bulger, Kibum Cho, Eloise Elliott

American Journal of Sports Science and Medicine

Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence

Christa L. Lilly1,, Luciana C. Braga2, Emily Jones3, Sean Bulger3, Kibum Cho3, Eloise Elliott3

1Department of Biostatistics, West Virginia University School of Public Health, Morgantown, WV, USA

2Department of Kinesiology, California State University Chico, Chico, CA, USA

3Department of Physical Education Teacher Education, West Virginia University College of Physical Activity and Sports Sciences, Morgantown, WV, USA


Purpose: The purpose of this study is to present a longitudinal planned missing design used to conduct a physical activity intervention in public middle schools (6th – 8th grades) in a rural Appalachian county. Method: Program outcome measures were collected at 13 points, with 33% participant random selection, and complete measures of demographic and anthropometric variables. Results: Of the 4,621 randomly selected participants over the three-year span, missing after random selection was highest for pedometers (74.68%), but also high for PACER (44.47%) and 3DPAR (41.9%). No differences were found on demographic or anthropometric variables, suggesting missing completely at random (MCAR) data on the planned missing data. Participants with missing data after random selection were more likely to be older, male, and in the 8th grade, suggesting missing at random (MAR) data. Conclusions: Results suggest the use of the planned missing design allowed for the feasible evaluation of the intervention using modern missing data analysis to account for MCAR and MAR data.

Cite this article:

  • Christa L. Lilly, Luciana C. Braga, Emily Jones, Sean Bulger, Kibum Cho, Eloise Elliott. Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence. American Journal of Sports Science and Medicine. Vol. 4, No. 4, 2016, pp 83-93.
  • Lilly, Christa L., et al. "Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence." American Journal of Sports Science and Medicine 4.4 (2016): 83-93.
  • Lilly, C. L. , Braga, L. C. , Jones, E. , Bulger, S. , Cho, K. , & Elliott, E. (2016). Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence. American Journal of Sports Science and Medicine, 4(4), 83-93.
  • Lilly, Christa L., Luciana C. Braga, Emily Jones, Sean Bulger, Kibum Cho, and Eloise Elliott. "Planned Missing Design in a School-Based Physical Activity Intervention for Early Adolescence." American Journal of Sports Science and Medicine 4, no. 4 (2016): 83-93.

Import into BibTeX Import into EndNote Import into RefMan Import into RefWorks

1. Introduction

School leadership and personnel realize their role in promoting health and achievement of the whole child, which includes providing students with educational foundations of health, and opportunities to participate in health-enhancing activities throughout the school day [1, 2]. The school setting is an optimal environment for the implementation of interventions targeting youth physical activity as adolescents spend a large portion of time in school [3]. As a result, the past two decades have been marked by a significant growth in school-based physical activity interventions across the United States along with an increase in local, state, and federal support for these types of initiatives [4]. Many researchers are now focusing on evaluating the effectiveness of school-based physical activity interventions, and how to best assess program outcomes. These efforts aim to improve health promotion programs in schools with the ultimate goal of enhancing students’ positive behaviors and attitudes toward physical activity as well their physical fitness levels [5].

There are some challenges in assessing physical activity interventions in the school setting. Participants’ compliance with assessment protocols, school schedule, staffing, equipment costs, and attrition can create barriers to the adequate, valid, and reliable assessment of physical activity in youth. Physical activity is challenging to assess across all ages, but particularly so for early adolescence. During this age, not only do physical activity levels start to decline, they begin to decline at different ages by gender. According to a recent meta-analysis, this decline can start as early as age 9 in girls whereas the decline was higher in boys in the 13-16 age group [6].

The choice of selecting a particular measure of physical activity depends on the study design, available resources, and the age of participants. Within middle school students, the most commonly used assessment methods to obtain physical activity data in the school environment are often subjective (e.g., self-report questionnaires, interviews, proxy reports, logs/diaries, and direct observation) rather than more expensive and time-consuming objective measures (e.g., pedometers, accelerometers, heart rate monitors, and criterion-referenced fitness tests) [7]. With each of these physical activity measures, whether subjective or objective, there are potential limitations that can affect the validity and reliability of the data. In this study, a combination of three different objective and subjective measures were used; thus, we explore limitations inherent to each of these measures, with particular detail related to missing data.

Motion sensing equipment such as the latest, technologically advanced pedometers (along with their more expensive counterpart accelerometers) are often hailed as a valid, cheap, and easy-to-use measure of objective physical activity assessments in schools [7, 8]. Despite these strengths, pedometer use also has some major limitations to consider. Notably, particularly among adolescent and early-adolescent children, compliance (i.e., lack of consistency in wearing the monitors during data collection periods, lost pedometers, or tampering with or resetting the pedometer) becomes a major issue. Because of this, many researchers will only include data in which the pedometer has been worn a minimum of 4 days, which can result in >30% sample exclusion [9]. Causes of missing data for accelerometer were studied within a large Chinese sample, where they found that collecting data during the weekend days was more likely to lead to missing data, and that the 1st day had the lowest amounts of missing data, while the last day had the most missing data. Finally, they found that male participants had slightly higher amounts of missing data than did their female counterparts for most days [10].

Aerobic capacity is often used as an indicator of overall physical fitness. The Progressive Aerobic Cardiovascular Endurance Run (PACER) test is an objective aerobic capacity test component of the FITNESSGRAM, a criterion-referenced test that assesses health-related fitness. However, measurement issues present themselves in the school environment, including social desirability bias that causes participants to “opt out” or not perform to potential. Children are asked to run laps until they feel like stopping and are given credit for “attempting” a single lap, and thus may stop prior to giving an accurate measurement of their aerobic capacity. Voss and Sandercock [11] note that among healthy volunteers in a high school setting, most achieved maximal effort; however, there were still some individuals where they noted substantial below maximum effort, likely due to “the product of social pressure felt by girls to not out-perform their peers or not wishing to continue running alone during the test” (p. 60).

The 3-Day Physical Activity Recall (3DPAR) is a validated subjective (self-report) instrument that is designed to measure usual moderate and vigorous physical activity among adolescents [12]. This measurement is accomplished by prompting the student to recall what physical activity they did, for how long, and at what intensity. To improve the accuracy of the recall, each day is segmented into 34 30-minute time blocks, and commonly performed activities grouped into 55 categories. Each activity is then rated by intensity level. The recall is completed in one reporting session for three days of the previous week. The task of recall is quite complex, which could cause inaccurate reporting or lack of reporting (i.e., causing missing data); indeed, research has found validity of the instrument decreases as the recall period increases from 1 to 3 days [12, 13]. Sirard and Pate [7] note that self-report questionnaires greatest limitation as a measure of estimating physical activity in children is the subjectivity inherent when individuals are asked to respond to questions about their behavior, resulting in recall errors, deliberate misrepresentations, social desirability and other biases.

Thus, physical activity interventions in the public school systems, particularly for early adolescents, are often clouded by assessment problems that are not corrected for in the final analysis model, and may only be included in study limitations, or ignored completely. For example, McNamara, Hudson, and Taylor [14] note in a review of the literature on pediatric pedometer use, pedometer compliance is an under-reported but important issue.

1.1. Missing Data: Not all Missing Data are Created Equal

Missing data and clarifying the reasons for that missing data can be a unique challenge when often missing data are based on the targeted outcome, a type of data called “missing not at random” or MNAR. If, for example, an adolescent wearing a pedometer elected not to wear it because they knew they did not walk as much as their peers, this would be considered MNAR data and difficult to adjust for within intervention designs. Failing to adjust for MNAR data would make it impossible to determine whether the results of the physical activity intervention are actually due to the intervention or to the limited sample that remained after drop-out, and may completely reverse the actual effects had the data not been missing.

Alternatively, missing data can take two other forms, including missing completely at random (MCAR) or missing at random (MAR). When missing data is MCAR, then the reason for missing is completely unrelated to the missing values. MCAR is the most ideal type of missing data; for example, if an adolescent misplaced their pedometer then the reason that adolescent pedometer is missing is not due to their fitness level. This is a strong assumption, and it is far more common to see the next preferred type of missing data, MAR. MAR occurs when the outcome is not missing due to the outcome after controlling for some other variable in the analysis. For example, MAR occurs if the missing pedometer data was due to gender (perhaps females are more likely to not wear their pedometers), but within each gender the probability of missing data on pedometers is unrelated fitness.

1.1.1. Analyzing Missing Data

When we can account for the reason for missing data using these missing data patterns if MCAR or MAR, we consider this information “ignorable” [15]. When missing data becomes ignorable, modern missing data techniques such as Full Information Maximum Likelihood (FIML) [16], Restricted Maximum Likelihood (REML) [17], or Ignorable Likelihood (IL) [18] methods, including Multiple Imputation (MI), could be used to recover the power of lost data, reduce error inflation, and avoid bias introduced by utilizing traditional forms of missing data approaches [19]. In the event of MNAR, more complex analysis is needed as the outcome would need to be jointly modeled with the association between the outcome and probability of response using techniques such as pattern mixture modeling.

To make matters more confusing, a single dataset can have more than a single mechanism of missing data, which can include both ignorable and nonignorable missing data mechanisms. But if we can group data by the mechanism for missing, we can mathematically account for the missing information. For example, let’s say we know data is missing both completely at random and also due to gender within the pedometer dataset. Using Ender’s [20] notation based on Rubin’s [15] work for the three types of missing data mechanisms (MCAR, MAR, and MNAR) we demonstrate the importance of grouping reasons for missing data. The conditional distribution of the missing data indicator matrix is denoted as R, and is a binary matrix showing 0 for missing values. Y indicates the data values, with subscripts for denoting observed (Yobs), missing (Yobs), and complete (Ycom) data. Finally, denotes the parameter that describes the relationship between R and the data. The probability of the reason for missing can be reduced to for MCAR (the probability of missing data on pedometers is unrelated to pedometer steps), for MAR (the probability of missing data on pedometers is related to some observed characteristic, like gender), and the full function for MNAR (the probability of missing data on pedometers is related both to some observed and unobserved pedometer data). MAR would depend on the Ycom variable associations with the reason for missing. Grouping subsets of the pedometer data based on the missing data mechanism, defined now as G, could then contribute to the individual probability for the reason for missing via for . Determining the true distribution for each G would improve imputation ability to account for each type of missing. Please additionally see Little & Rubin [21] for a more complete understanding of probability functions within the use of missing data patterns.

1.1.2. Planned Missing Data Design

In an effort to reduce missing data across all assessment types, planned missing data designs are often recommended [22] with the concept that randomly selected data points would reduce MNAR and allow MCAR or MAR assumptions to stay true. Planned missing designs combined with modern missing data analysis methods allows for ways to cut costs regarding assessments, improve data quality, and reduce participant fatigue and practice effects [23]. While planned missing designs can take several different forms [22] including giving a longer measure to a small subsample of a larger study (two-method measurement designs), we focus on two designs in particular: a multi-form designs and longitudinally-designed missing measurement time points (wave missing designs) [24].

Multi-form designs attempt to reduce the number of items or questions that participants fill out by creating multiple forms that contain subsets of the total items. Typically, this type of design includes the three-form design [25], which divides assessment items into 4 subsets including a common block X and three partial blocks (A, B, C). The X block is administered to all participants first and allows for testing potential missing at random (MAR) assumptions. Thus, this X block usually includes demographic information or some other primary outcome. The three partial blocks are then combined into three combinations (AB, BC, and AC) and administered to one-third of study participants to allow sufficient overlap between information. Multi-form designs are considered ideal when specific conditions are met; according to Rhemtulla and Little [23] these include: 1) The ideal battery of assessment is too long otherwise (e.g., due to time or attention constraints), 2) it is possible to increase sample size (due to likely decreases in power in this type of design), 3) the research focus is at the group rather than individual level, and finally 4) the sample size is sufficiently large. Once these requirements are met, multi-form designs can take on many forms and are limited only by the practical necessities of the research design. This type of design is often used in cross-sectional studies and can be easily extended to multiple time points.

In contrast to the multi-form design, wave missing designs occur when individuals are planned to have data missing on one or more occasion in a longitudinal design. For example, if a longitudinal evaluation measures all participants at baseline, but randomly excludes a group of participants over a series of repeated measurements, this would be considered a wave missing design form.

Interestingly, this wave missing type of design can be combined with a multi-form design at each measurement occasion (Little & Rhemtulla, 2013). Ideally, any of the planned missing designs strives to maximize covariance coverage (or the amount of available information due to overlap between variables) while minimizing the fraction of missing information (or the amount of missing information).

1.2. Summary and Study Goals

In summary, assessments of physical activity interventions often have data missing, and this missing data is rarely taken into account (or even reported). This is particularly problematic because data can be missing due to the outcome of physical activity (a type of missing data called MNAR), and therefore impact interpretation of the results of these studies. Researchers conducting physical activity interventions should 1) attempt to decrease MNAR data by using a planned missing design, 2) report missing data mechanisms and patterns, and 3) analyze the data using statistical analysis appropriate to the missing data mechanism.

We can examine the presence and absence of data in the planned missing design to see if there are missing data patterns. These missing data patterns have different probability functions related to the occurrence of MCAR, MAR, and MNAR data. If we plan for data to be missing, then there should be no discernable missing data pattern beyond our intended planned missing (the presence of missing data would not be linked to fitness level, for example). Even with planned missing data, we often see several patterns of missing data emerging at the same time.

The goal of this study was to present a longitudinal planned missing design, using a modified wave-missing multi-form design, which enhance power and outcomes for a physical activity (PA) intervention conducted in public middle schools (6th – 8th grades, range of ages 10-16, average age 13) in a rural Appalachia county in the USA. In order to use several physical activity measures, including the costly pedometers, the decision was made to use random sampling across collection periods (resulting in partial blocks of the PA measures) but complete X block measures on all participants, thus resulting in a modified wave-missing, multi-form design. A final goal of this study was to define which type of missing data group an individual had at a given time point, assuming the missing data type was independent by time point. Specifics of project design and set-up, including grant requirements, are included in detail under methods. Detailed descriptions of the missing data and types of missing data are included in results. Finally, strengths and limitations of this design, including barriers found in this project, are presented in the discussion.

2. Methods

2.1. Grant Requirements

Data from this article were based on a three-year school-based health intervention funded by the Carol M. White Physical Education Program (PEP) grant awarded in 2011. Details of the PA intervention and general descriptive statistics of the study outcomes (e.g., [26]) can be found elsewhere. Institutional approval of the protocol was obtained, allowing “opt-out” informed consent, wherein parents of participating students were provided with the opportunity to sign a document (or call the project manager) if they did not wish their child to participate in the study. This information was provided via letters sent home with the child at the beginning of every school year. Children could also decide to not participate at any point during the evaluation.

Within the required PEP grant evaluation components, specific data collection and design elements were required. Grant requirements mandated evaluation over three years of intervention to include 13 collection periods. This timeframe included a “baseline” collection period prior to any intervention activities, four other collection periods of evaluation data the first year, four collection periods the second year, and four collection periods the third year.

Greenbrier CHOICES project was one of the few selected among PEP awardees to be permitted random sampling across the assessments. The random selection was conducted in two middle schools and included all 6th, 7th, and 8th grade attendees (except for those who opted-out of the project). A master list of all students including identification allowed longitudinal connections. The master list was updated each year to account for incoming 6th grade students, those leaving or entering the school district over the summer months, and those exiting 8th grade. An eligible student list was maintained from the master list to account for those who opted out of the study. For every collection period, random selection was conducted using a random selection Excel macro made available by PEP upon approval of random sampling, wherein after noting the total number of eligible students, random selection of 33% of the numbers between 1 and that total number were given. These numbers were then linked to row number within the eligible list. A dummy-coded variable was included in the dataset to denote whether the participant was randomly selected or not for the given evaluation time point. Complete measures were available to all participants each year for Body Mass Index (BMI) and general demographic information, including age, gender, school and classroom. PEP also included an excel macro that attempted to account for the missing design within reporting the primary outcomes. This method would not be appropriate, however, for the final assessment and research piece of the project as the excel macro only adapted the final number of those meeting a given grant outcome requirement (ie, grant required Government Performance and Results Act of 1993 (GPRA) goals of number of participants obtaining 60 minutes of PA a day, meeting age-appropriate cardiovascular fitness levels, and eating at least two fruits and three vegetables daily) to make reported numbers consistent across PEP grant awardees, and did not use a modern missing data approach to make this correction for random sampling among the few selected PEP sites.

2.2. Adaption of the Multi-form Design

PEP grant requirements permitted us the use of randomized participant selection. To better account for the inevitable loss of power and missing information using this design, we planned to conduct a modified wave missing, multi-form planned missing design. Although related to a classic multi-form design, the three-form design was not conducted. The common block X was administered to all participants each year (demographics and BMI assessment; see collection periods 2, 6 and 10 in Table 1).

Table 1. Planned Missing Design, n=2,075

However, all evaluation outcome measures (instead of the one-third of the evaluation measures in the three-form design) were given to the randomly selected participants each collection period. Thus, there were only 2 different forms (X block and outcome measures), but the outcome measures were given to only one third of the eligible participants at any data collection period. Since each of the 13 collection periods over the three-year period also included participants who were entering and leaving the middle school system, this design type allowed for some individuals to have data spanning three years but did not likely permit any student to complete all 13 evaluation periods (Table 2).

Table 2. Demographics of Participants and Descriptive Statistics (Unique Cases Presented by Year, Total Unique n = 2,075)

2.3. Measures

Measures on the common block X were reported by the school and included gender, grade, and birthdate. Student age at collection period was calculated using the school reported birthdate for each collection period. Height and weight, used for BMI calculations, were collected by school nurses and calculated by evaluation team using CDC EpiInfo nutstat software, v. 9.02.

Other evaluation outcome measures included the number of steps each day of five-day pedometer use recorded by the pedometer, the number of PACER laps run on evaluation day as recorded by an evaluation team member, and a self-reported number of moderate-to-vigorous half hour activity blocks each day of a 3DPAR. A complete description of evaluation measures and procedures, along with more specific details about these measures can be found elsewhere (e.g., citation omitted for blind review). For the purpose of this study, a total number of PACER laps completed were used to indicate PACER laps; the intraclass correlation over the three-year span indicated consistency within individuals (ICC=0.67). Pedometer steps were summed for each day of the five recorded days; Cronbach α of the five days (year 1 n=416 α= 0.67, year 2 n= 134 α= 0.55, year 3 n=148 α=0.80) indicated sufficient reliability that items could be combined for a scale score, although year 2, in particular, was below the desired α= 0.70. Likewise, number of half-hour activity blocks (range: 0 to 32) were added together for each day of the 3DPAR; cronbach alpha of the three days (year 1 n=575 α= 0.71, year 2 n= 618 α= 0.76, year 3 n=803 α=0.81) indicated sufficient reliability that items could be combined for a scale score.

2.4. Planning Missing Data Management

Data were kept in an Excel spreadsheet by a data manager. Spreadsheets were kept separate for each evaluation collection period. At the end of the grant, a longitudinal database was constructed using SAS. First, data were stored in “long” rather than “wide” format, permitting multiple rows per participant rather than a single row. This process was done due to a large number of evaluation items (particularly for the 3DPAR and pedometers). Each year, the master list for randomization was used and combined with a single variable denoting collection period (1 through 13; complete for all participants in the master list) and another variable denoting whether the participant was randomly selected during that collection period (0 for not selected, 1 for selected, plus an indicator for missing from the master file). Missing data were recorded for each individual for each time point as missing based on random selection, missing on the random selection variable, and selected but missing in a given evaluation measure. Data were then combined with the evaluation assessments. BMI assessments were included in collection periods 2, 6, and 10 as the screenings occurred closest to those evaluation dates. Otherwise, evaluations were matched according to ID and collection period.

2.5. Statistical Analysis

All statistical analysis and data management for this particular project were conducted in SAS 9.4. Counts and percentages are reported for some cases and observations for describing categorical data along with missing data patterns of data. Means and standard deviations are used to describe continuous data.

Sensitivity analysis between random selection, and random selection and missing were conducted on variables included for all individuals, including BMI percentile, age at collection period, grade, and gender. BMI percentile, collected on most individuals and categorized as an X block variable, was anticipated to be associated with all PA outcomes but was only correlated with PACER (Table 2). If missing data were linked to different BMI percentiles, we assumed data to be MNAR (i.e., missing due to the outcome) instead of MAR or MCAR. For example, if students were selected to participate in PACER laps but a subgroup of those selected decided to not participate because they were not as fit, we should see associations between BMI percentiles and the variable noting the absence of their PACER laps. If present and missing data were not significantly different based on other X block variables, we assumed MCAR on the random selection variable (planned missing data) only. In other words, if selected students do not participate in the PACER laps, that variable denoting missing would not be linked to any outcome or other demographic variable, and the presence of missing data are truly random. Other differences on X block variables linked to missing data was considered MAR. For example, if 6th grade students were selected to participate in PACER laps but were not able to because of a field trip, the reason for their missing data would not be linked to their fitness level, but instead, be associated with the students’ grade. The analysis to determine reason for missing included chi-square, t-tests and general linear model analysis of variance tests depending on variable type between the variable noting presence or absence of missing data if selected on each outcome measure (i.e., unplanned missing data for PACER, pedometers, and 3DPAR) and each X-block variable (BMI, age, grade, and gender.)

Two types of missing data patterns are reported using SAS software. This includes PROC CALIS, the SAS package used for modeling FIML missing data and PROC MI, used to model MI missing data. Since BMI (available on all participants) were assigned to collection periods 2, 6 and 10, these collection periods were selected to present missing data results. Missing data models included X block demographic (gender, grade and age, i.e., no missing data) as well as continuous primary grant outcomes; results are presented for: (1) BMI percentile, (2) total pedometer steps by day (for 5 days), (3) number of moderate-to-vigorous physical activity half hour blocks by day (for 3 days) and (4) number of PACER laps run. Results included: (1) the most common missing data patterns (in other words, what missing values tended to co-exist), (2) covariance coverage (or the proportion of participants with complete data for each combination of measures) with minimum desired covariance coverage of 10% [20], and (3) fraction of missing information (FMI; or the proportion of a parameter’s sampling error due to missing data) with desired FMI below 20% [20]. After analysis by collection period, missing data analysis was conducted on year data, in which collection periods were averaged for grant outcomes within years.

3. Results

A total of 315 students opted-out of the project over the three-year span, for a final response rate of 91.53%. Thus, the final dataset included 15,278 rows, with 2,075 unique students included. Of the 15,278 observations, 689 were missing information about the random selection (4.5%) due to declining to further participate in the study after previous participation, and 4,621 (31.67%) were selected for random participation. The random sampling was done across all the students in the grades together, rather than within each grade, and did result in relatively even representation of the grades (Table 1). 150 students were selected for measurement every year for all three years, 646 had measures for two years, and 1,262 had a single year of data measure. Demographics (block X variables) for the unique sample can be found in Table 2. No missing data was recorded for gender, grade, and student age at collection period; however, 1,202 BMIs were missing over the 3-year span (7.87% missing). Students missing on this X-block variable tended to be slightly older (average age 13.33 versus 13.11, p < 0.0001) but no differences were found on gender or grade.

As expected, no differences were found based on random selection on any of the X block variables, including grade (p = 0.928), gender (p = 0.562), age (p = 0.98), BMI percentile (p = 0.24), or BMI z-score (p= 0.11), suggesting MCAR on the planned missing data.

Analysis of the additional 4.5% missing on the random selection variable revealed no differences among including gender (p = 0.654), BMI percentile (p = 0.50), or BMI z-score (p= 0.27). However, those with attrition on the random select variable were more likely to be older (p < 0.001; M = 13.61, SD = 0.79) than those who were either randomly selected (M = 13.11, SD = 0.99) or those who were not randomly selected (M = 13.11, SD = 0.98), and more likely to be in 8th grade (p < 0.001; 52.54%) than those either randomly selected (31.57%) or not randomly selected (31.71%). Results suggest the presence of MAR data due to attrition based on age in our middle-school sample.

Further exploration was done to examine associations between random selection and missing on other grant outcomes. Of the 4,621 selected participants over the three-year span, missing on selected was highest for pedometers (n = 3,451, 74.68%), but also surprisingly high for PACER (n = 2,055, 44.47%) and 3DPAR (n = 1,936, 41.9%). Table 3 expands on these three outcomes in particular. Of particular note, participants with missing data after random selection were more likely to be older, male and in the 8th grade. However, differences were not significant on the X block variable BMI percentile, again suggesting the presence of MAR rather than MNAR.

Table 3. Description of Missing Data if Selected on Grant Outcomes (n = 4,621 Observations)

For collection periods 2, 6 and 10, select grant outcomes are presented in Table 4. Notably, for these three collection periods missing data patterns showed the most common pattern was the presence of X block variable information and absent on all others (56%-64.8%). The next most common pattern was the presence of X block variables except BMI percentile (7.7%-29.6%) and missing only on pedometer data (2.2%-11.6%). The average proportion of coverage of covariances ranged from 0.13 to 0.21; smallest covariance coverage was 0.036 (pedometer steps days 1 and 5, collection period 6). Ideally, we would have complete data on roughly .33 (since we had roughly 33% participation) with preferred lowest covariance coverage of .10. While the average covariance coverage is within that range, the item covariance coverage is below this preferred amount. The largest fraction of missing data was 0.972 (pedometer steps day five at collection period 10), indicating a large proportion of the sampling error to be due to missing data.

Table 5 presents the grant outcomes collapsed into years 1 through 3. On average, responses were present for a single observation for 11.81%-44.59% of evaluation day measures, and responses were present for more than a single observation for 0.7%-27.88% of evaluation day measures. Again, presence of all X block variables and missing on all others was the most common missing data pattern (21.8%-30.5%), followed by missing on BMI for the X block variables (0.5%-6.4%) and missing only pedometer data (highest for year 3: 45.8%). Average proportion coverage of covariances ranged from 0.29-0.47. Smallest covariance coverage was 0.097 (BMI percentile and pedometer steps day 5, year 2). The largest fraction of missing data was 0.96 (pedometer steps day five at year 3). The proportion of missing data, covariance coverage and fraction of missing information were all improved by collapsing within years over single collection period information.

Table 5. Missing Data Information by Year

4. Discussion

In general, use of the planned missing design, coupled with PEP grant requirements, allowed feasible evaluation of the PA intervention in a public, middle school setting in rural Appalachia. With this design, further analysis on the longitudinal dataset can involve advanced modeling techniques using any modern missing data method, including FIML for use in structural equation modeling, REML for use with general linear mixed modeling, and MI for use with any general analysis method.

Results from an in-depth examination of the type of missingness on X block and PACER test, in particular, suggest the presence of MAR based on demographic rather than outcome variables. In other words, we believe that students were missing on outcome variables due to older and male, rather than actual fitness or physical activity levels. Grouping should thus include two different MAR groups, one based on being older (to account for attrition on the random selection variable) and another based on being male and in 8th grade (to account for missing on evaluation outcome measures). Unfortunately, the other outcome measures of 3DPAR and pedometer steps were not significantly correlated with BMI at the item level, minimizing MAR and MCAR inferences based on the data. Future data should include PACER as an X-block variable, which is correlated with other PA outcome data.

Despite the use of the planned missing design, high rates of unplanned missing data were consistent across evaluation pieces and particularly high for pedometer use. Thus, this modified wave missing multi-form design, combined with MAR type missing data, resulted in low covariance coverage and high FMI, making it more difficult to recover missing information using a modern missing data analysis method.

Collapsing within years improved covariance coverage and fraction of missing information, generally above the desired .10 for covariance coverage and near or below the .20 for FMI except for pedometer use. This type of collapsing may be desirable for improved imputation fit, except that collapsing the data voids the assumption of independence of reason for missing (i.e., an individual student may now have missing data due to both MAR of difference types and/or MCAR, a mixing of group G in the probability function). Since collapsing the data voids this vital assumption of imputing missing data, it would not be recommended for use.

4.1. Strengths

Results indicate some recovery of missing data information due to MAR and MCAR (and not due to MNAR) with this planned missing design and thus improved power over ignoring missing data. Examination of why the unplanned missing data occurred additionally allows grouping so that recovery of missing data will be more successful using imputation or modern missing method techniques. These results can inform other physical activity interventions that occur in middle school settings so they can account for unplanned as well as planned missing data.

Consistent with the previous literature on this type of study design, costs were much lower due to equipment purchasing of pedometers, hiring substitute teachers during assessments, and costs of scanning responses or hand entering into databases. Also, data quality was improved due to smaller datasets for cleaning purposes. Assessment fatigue, always an issue with so many assessments and measures in such a short span of time, particularly for young adolescents, was reduced.

It is important to note that the constant evaluation of data collection procedures during regular team meetings over the three-year period and consequent refinement of the data collection protocols contributed to reduced assessment fatigue and management of missing data. Further, on-site data collection and data management across the grant period was facilitated by a consistent group of research team members, which helped to ensure research protocols were adhered to, and necessary modifications or efficiencies were made accordingly.

4.2. Barriers, Limitations and Next Steps

Despite the planned missing design, some challenges remained, including the continued high percent of unplanned missing data. First, we found a small percent (7.8%) missing data on the X-block variable of BMI. Although there was a difference in age, this difference was minimal (13.33 vs. 13.11 years). We believe this small percent to be generally due to missing school or other missing at random reasons. Perhaps other researchers could consider including teachers and school personnel in the BMI screening to improve upon this participation rate. Research shows that teachers’ participation in the screening process to be vital to student engagement through the reinforcement they provide to the students about the importance of screening results and the impact on their health [27]. Additionally, we were surprised that BMI was not significantly correlated with the item level pedometer steps and 3DPAR activity blocks; thus, we recommend including PACER as an X-block variable in the future.

Pedometer use, in particular, continued to be plagued with assessment fatigue, equipment failure and loss, low reliability, and participant burnout, all resulting in a high percentage of unplanned missing data, consistent with accelerometer results in the literature [10]. For example, a member of the evaluation team found one pedometer hanging off a guardrail next to a road near one of the schools during project year two. Although pedometers have been identified as useful tools to measure daily physical activity, limiting the use of pedometers to the school hours may ensure a more controlled administration of the instrument and limit misuse of the tool and would likely improve reliability of the measure across the five-day span as well. However, due to the PEP grant requirements, students were asked to wear the pedometers outside of school for consecutive five-day periods, which resulted in a considerable number of lost pedometers. The evaluation team also identified pedometers distribution (e.g., coordinating randomly selected participants within the structure of the school day), maintenance (e.g., battery replacement), and monitoring participant wear time and compliance as challenges to the use of this measure. Finally, we hypothesize that the drop in pedometer reliability in year 2 may be in part caused by a protocol change, wherein students were previously asked to keep hand-written step sheets in addition to wearing the pedometer, which perhaps reminded students to more reliably wear the pedometer.

Challenges associated with missing data on the PACER test included student motivation and attrition. Members from the evaluation team frequently commented on students’ hesitancy in taking part of the test and in attempting to run as many laps as possible. This reluctance seemed particularly problematic amongst females and/or students who are less physically fit. A few strategies were implemented to reduce attrition and increase student motivation to complete the PACER test to the best of their abilities. First, the evaluation team introduced a rewards system for students who ran the highest amount of laps [28]. Second, substitute teachers were hired on data collection days in order to allow physical education teachers to get involved in the testing process as motivators and role models. Because the teachers had a stronger rapport with the students, they were able to encourage their participation through positive reinforcement and modeling (e.g., teachers completing the PACER test with students). Third, the evaluation team established a consistent testing protocol that minimized attrition due to peer pressure. The protocol requested that all students remained in the testing area during the testing instructions. This strategy allowed for evaluation members and physical education teachers to reach all students when explaining the purpose of the test and soliciting their participation.

In regards to the 3DPAR, the evaluation team members noticed that the students needed focused instruction and diligent supervision while completing the paper-pencil 3DPAR instrument otherwise resulting in unplanned missing data. After the initial data collection windows, the evaluation team refined the routine and protocol for 3DPAR administration that aimed at reducing missing data and increasing the accuracy of information from the students. For example, evaluation members began verifying each 3DPAR for completion and asking students to provide missing information when necessary.

Another critical issue affecting data collection was inclement weather. It is important to note that for any measurement of physical activity in youth, especially when data is collected in the school setting, seasonal variation (e.g., climate, holidays, school testing schedule) might contribute to some missing data. For example, in grant year three one school missed a pedometer data collection period due to inclement weather. Further, a number of out-of-school days due to snow forced the evaluation team to make last minute changes in collection schedule, which affected on-site staff availability, required additional adjustments to school and testing schedule, and contributed to increased assessment fatigue. Attrition due to dropouts and participants moving out of the school occurred regardless of the planned missing design, resulting in some MAR data. Hence, it is critical that researchers account for those factors when planning data collection in schools.

General reporting of descriptive statistics within the PEP guidelines did not use a modern missing method to account for the random selection of participants.

Missing data is best recovered on the per-item basis – however, due to the sheer number of items included (particularly for 3DPAR and pedometers), in may be most feasible only to recover on the final variables rather than each individual item. Additionally, while collapsing within years improved covariance coverage and reduced FMI, this type of aggregation violates the assumption of independence of type of missing data (i.e., an individual could now have MCAR and MAR simultaneously), making recovery of that type data via MI more difficult. On the other hand, to better estimate the habitual activity of the students, combining the data for an entire year may be preferable.

Future studies may wish to consider a more traditional three-form design (with forms AB, AC, and BC) rather than conducting all evaluations on just a third of the participants each collection period for improved covariance coverage and decrease in fraction of missing information [29]. Current research [30] highlights the importance of improving the overlap of data across measures, participants, and waves within a planned missing design. Regarding lessons learned, we would revise this study’s planned missing design to include all students providing data on the PACER at each wave. PACER was correlated with all PA measures. This would have provided a stronger covariance coverage and a lower fraction of missing information while keeping the more expensive measurements still reduced and its completion randomly assigned.

4.3. Conclusion

Despite these limitations, this planned missing design was successfully utilized with pre-adolescents and young adolescents, over a three-year span, and (with some modifications) is recommended for other researchers in the field. Other researchers should note the low covariance coverage and high FMI that resulted from randomization using PEP grant requirements; thus, one of the recommended modifications includes using PACER as an X-block variable in order to improve wave overlap. Sensitivity analysis of the unplanned missing demonstrated the unplanned missing data were likely due to MAR rather than MNAR; these results can be used to group type of missing to better adapt modern missing methods to recover the missing information. Other interventionists may wish to oversample older males, in particular, to improve covariance coverage and FMI and recover information due to MAR in middle school PA programs.

Physical activity interventions often are plagued with assessment problems that include large amounts of missing data. This study builds upon research examining the causes of missing data in physical activity in children (e.g., [10]), including examination of the problematic missing not at random (MNAR) data type. This articles adds to the field by presenting a method for reducing missing data (a planned missing data design) with supportive findings from a project utilizing the method. Results support the argument that researchers conducting physical activity interventions should 1) attempt to decrease MNAR data by using a planned missing design, 2) report missing data mechanisms and patterns, and 3) analyze the data using statistical analysis appropriate to the missing data mechanism.


The authors would like to thank Dr. William Neal for his support of the project, as well Dr. Michael Regier for his invaluable assistance in developing the probability formulas, and Ms. Amanda White for her data management role during the project. Many thanks to the administrators, teachers, and students who participated in this project.

Funding Statement

This study was primarily supported by the Carol M. White Physical Education Program (PEP) grant (PR/Award #: Q215F110479) from U.S. Department of Education in 2011. The grant information is available at “” Additional support was provided by the WV Bureau for Public Health for the CARDIAC project, and by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number U54GM104942. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies.


[1]  Pate RR, Davis MG, Robinson TN, Stone EJ, McKenzie TL, Young JC, et al. Promoting physical activity in children and youth: A leadership role for schools. Circulation. 2006;114(11):1214-24.
In article      View Article  PubMed
[2]  Strong WB, Malina RM, Blimkie CJ, Daniels SR, Dishman RK, Gutin B, et al. Evidence based physical activity for school-age youth. Journal of Pediatrics. 2005;146(6):732-7.
In article      View Article  PubMed
[3]  Trost SG, Loprinzi PD. Exercise-promoting healthy lifestyles in children and adolescents. J Clin Lipidol. 2008;2(3):162-8.
In article      View Article  PubMed
[4]  Sallis JF, McKenzie TL, Beets MW, Beighle A, Erwin H, Lee S. Physical education's role in Public Health: Steps forward and backward over 20 years and HOPE for the future. Res Q Exercise Sport. 2012;83(2):125-35.
In article      View Article
[5]  Brown T, Summerbell C. Systematic review of school-based interventions that focus on changing dietary intake and physical activity levels to prevent childhood obesity: An update to the obesity guidance produced by the National Institute for Health and Clinical Excellence. Obes Rev. 2009;10(1):110-41.
In article      View Article  PubMed
[6]  Dumith SC, Gigante DP, Domingues MR, Kohl HW, 3rd. Physical activity change during adolescence: A systematic review and a pooled analysis. International Journal of Epidemiology. 2011;40(3):685-98.
In article      View Article  PubMed
[7]  Sirard JR, Pate RR. Physical activity assessment in children and adolescents. Sports Medicine. 2001;31(6):439-54.
In article      View Article
[8]  Scruggs PW. Middle school physical education physical activity quantification: A pedometer steps/min guideline. Res Q Exercise Sport. 2007;78(4):284-92.
In article      View Article
[9]  Hohepa M, Schofield G, Kolt GS, Scragg R, Garrett N. Pedometer-determined physical activity levels of adolescents: differences by age, sex, time of week, and transportation mode to school. Journal of Physical Activity and Health. 2008;5 (Suppl. 1):S140-52.
In article      View Article  PubMed
[10]  Zhuang J, Chen P, Wang C, Huang L, Zhu Z, Zhang W, et al. Characteristics of missing physical activity data in children and youth. Res Q Exercise Sport. 2013;84 (Suppl. 2):S41-7.
In article      View Article  PubMed
[11]  Voss C, Sandercock G. Does the twenty meter shuttle-run test elicit maximal effort in 11- to 16-year-olds? Pediatric Exercise Science. 2009;21(1):55-62.
In article      View Article  PubMed
[12]  Pate RR, Ross R, Dowda M, Trost SG, Sirard JR. Validation of a 3-day physical activity recall instrument in female youth. Pediatric Exercise Science. 2003;15(3):257-65.
In article      View Article
[13]  Mcmurray RG, Ring KB, Treuth MS, Welk GJ, Pate RR, Schmitz KH, et al. Comparison of two approaches to structured physical activity surveys for adolescents. Med Sci Sport Exer. 2004;36(12):2135-43.
In article      View Article
[14]  McNamara E, Hudson Z, Taylor SJ. Measuring activity levels of young people: The validity of pedometers. British Medicine Bulletin. 2010;95:121-37.
In article      View Article  PubMed
[15]  Rubin DB. Inference and missing data. Biometrika. 1976;63(3):581-90.
In article      View Article
[16]  Enders CK. The impact of nonnormality on full information maximum-likelihood estimation for structural equation models with missing data. Psychol Methods. 2001;6(4):352-70.
In article      View Article  PubMed
[17]  Berk K. Computing for incomplete repeated measures. Biometrics. 1987;43(2):385-98.
In article      View Article  PubMed
[18]  Little RJ, Raghunathan T. On summary measures analysis of the linear mixed effects model for repeated measures when data are not missing completely at random. Statistics in medicine. 1999;18(17-18):2465-78.
In article      View Article
[19]  Graham JW. Missing data analysis: Making it work in the real world. Annu Rev Psychol. 2009;60:549-76.
In article      View Article  PubMed
[20]  Enders CK. Applied missing data analysis. New York, NY: Guilford Press; 2010. xv, 377 p.
In article      
[21]  Little RJA, Rubin DB. Statistical analysis with missing data. 2nd ed. Hoboken, N.J.: Wiley; 2002. xv, 381 p. p.
In article      View Article
[22]  Little TD, Rhemtulla M. Planned missing data designs for developmental researchers. Child Dev Perspect. 2013;7(4): 199-204.
In article      View Article
[23]  Rhemtulla M, Little T. Tools of the trade: Planned missing data designs for research in cognitive development. Journal of Cognitive Development. 2012;13(4).
In article      View Article  PubMed
[24]  Graham JW, Taylor BJ, Olchowski AE, Cumsille PE. Planned missing data designs in psychological research. Psychol Methods. 2006;11(4):323-43.
In article      View Article  PubMed
[25]  Jorgensen TD, Rhemtulla M, Schoemann A, McPherson B, Wu W, Little TD. Optimal assignment methods in three-form planned missing data designs for longitudinal panel studies. Int J Behav Dev. 2014;38(5):397-410.
In article      View Article
[26]  Jones E, Bulger S, Braga L, Cho K, Lilly C, Neal W, Elliott, E. Initial findings of a multi-component school-based PEP grant intervention: Greenbrier CHOICES. 2016 SHAPE America Convention; April 2016; Minneapolis, MN.2016.
In article      
[27]  Stalter AM, Chaudry RV, Polivka BJ. Facilitating factors and barriers to BMI screening in schools. Journal of School Nursing. 2010;26(4):320-30.
In article      View Article  PubMed
[28]  Buckworth J, Lee RE, Regan G, Schneider LK, DiClemente CC. Decomposing intrinsic and extrinsic motivation for exercise: Application to stages of motivational readiness. Psychology of Sport and Exercise. 2007;8(4):441-61.
In article      View Article
[29]  Rhemtulla M, Jia F, Wu W, Little TD. Planned missing designs to optimize the efficiency of latent growth parameter estimates. Int J Behav Dev. 2014;38(5):423-34.
In article      View Article
[30]  Little TD, Jorgensen TD, Lang KM, Moore EW. On the joys of missing data. Journal of Pediatric Psychology. 2014;39(2):151-62.
In article      View Article  PubMed
  • CiteULikeCiteULike
  • MendeleyMendeley
  • StumbleUponStumbleUpon
  • Add to DeliciousDelicious
  • FacebookFacebook
  • TwitterTwitter
  • LinkedInLinkedIn