Understanding the Youth Survey Data

Data can help us better understand the experiences of young people, identify areas of strength and concern, and guide decisions about where to focus resources and action. While no dataset can capture every dimension of youth well-being, these data provide valuable insights that can help communities in Western Massachusetts learn, collaborate, and improve supports for young people over time. These data are unique in sharing student experience for Western MA counties and therefore add great value to our region.

At the same time, the data are not perfect, no data is. These data were developed from young people filling out surveys, not all students answered them, and survey methods varied across counties. To draw accurate conclusions, it is best to consider these data alongside other sources of information, local knowledge, and the limitations described below.

How does this data help our region?

Although the state of Massachusetts surveys young people across the Commonwealth, the way that the data are collected (at a random sample of schools) means that we cannot look at the results at a local or regional level. This is where the surveying efforts of local organizations comes in. Learn more about local surveying organizations participating in the Hub on our Methods page.

The organizations and coalitions who survey young people in their communities do so because it’s important to have data that represents the voices of the young people they serve. Local youth surveying efforts are critical to understanding what is really happening in our young people’s lives. The data are used for action in countless ways, including:

To advocate for or against policy
To change curricula
To support professional development for school staff
In grant applications to support funding for initiatives that support youth
To engage with young people and support an interest in data
In reports, such as the Youth Mental Health Roadmap for Western Massachusetts and Community Health Needs Assessments for hospitals in all four counties in Western MA

Another major benefit of data from local youth health surveys is that local surveying efforts are more able to be flexible by adding or modifying questions to reflect local priorities and emerging issues. For example, PHIWM expanded the series of questions on gambling after the opening of the MGM casino and added questions about belonging in school based on feedback from youth. All of the surveying organizations on the Hub have added questions focused on social media and screen time to keep tabs on technology’s impacts on young people. This flexibility makes the data even more valuable for our region, as it provides information about key local issues, and can cover topics not covered in larger surveys.

What should I know to draw accurate conclusions from the data?

It’s important to recognize that students’ experiences are greatly impacted by a broad range of factors, including school and community conditions, racism, classism, ableism, heterosexism, and gender-based discrimination. You may see different responses for groups of students in one county or across the region overall. This does not mean that living in one location or being part of one group (e.g., race, grade, gender) versus another is the reason for the difference. When interpreting the data, we encourage users to consider the broader systems and environments that shape health and well-being.

What factors could impact whether our data accurately represents student experience?

These data were developed from survey questions (see our methods to learn more). There are several factors that can make us more or less confident that our data accurately captures the full student experience in a specific datapoint shown on the Youth Health Data Hub:

How many students took the survey?

Not all students took the survey, so the results may not reflect everyone’s experiences. The fewer students who took the survey (what we call the “response rate”), the more concerned we are about how accurately our results represent the experiences of all students at the school.
The students who miss taking the survey (whether because their parent/guardian opted them out, they chose not to take it, or they were absent that day) may have experiences that are different from the students who took the survey. For example, students who are chronically absent, and more likely to miss taking the survey, may be at higher risk for some health outcomes and behaviors than students who consistently attend school. This could cause our results to be an under- or over-estimate of student experience as a whole.
This is not unique to us. Nationally, the percent of students who complete the Youth Risk Behavior Survey have been on a steady decline since 2011. Most organizations who conduct youth surveys, like the CDC and Bach Harrison, consider a 60% response rate as the minimum we can consider to be a good representation of student experience.

How many students answered the specific question?

We show results when at least 30 students answered the question. However, when only a small number of students answered the question, even one or two different answers can noticeably change the results. Because of this, we should be less confident in this result and be cautious in how we use it. We recommend using additional sources of information alongside this result to understand student experience.

Do all students respond accurately?

We can’t know. One of the limitations of asking survey questions is that people are not perfect at remembering their life experiences. Survey takers may also not answer all questions honestly. This is human nature and is a limitation of all survey research.

Can we understand why students responded a certain way?

Because we asked all our questions at the same point in time, we can’t say one thing (or response) caused another. We don’t know what came first. You may hear this concept referred to as “correlation does not equal causation.”

For example, we can’t look at ice cream sales from Flayvors or the Sugarloaf Frostie alongside water-related accidents in the Connecticut River, then say that eating or selling ice cream leads to more accidents. These things may be correlated, or associated, because people eat ice cream and swim or kayak in the summer, but we can’t say that one caused the other.

Why shouldn’t I compare the data across counties?

We can’t be confident that our data always show us the reality of all student experiences, as we discussed above. There are other reasons, too, why we need to be careful and avoid comparisons across counties.

Many groups across Western MA survey their communities, and each organization’s survey methods are somewhat different. These differences mean that we cannot directly compare across counties, because they could have an impact on how students choose to respond to any given question. Some of these differences include:

Different question text: Although we tried to pick similar questions to develop each metric, sometimes they were not the same. It is important to think about the full question text and whether the way things were asked may have made a difference in students’ responses. The full question text is available at the bottom of the map slider on each topic page or in our data dictionary. Here is an example of the different survey questions we used to make the vaping metric:
How old were you when you first: vaped nicotine or used an e-cigarette? (Hampshire)
How old were you when you first used electronic cigarettes, e-cigarettes, vape pens, or e-hookahs? (Berkshire, Franklin/North Quabbin)
Have you ever, even once, used: Electronic cigarettes, e-cigarettes, vape pens, or e-hookahs? (Partial Hampden)

Survey timing: Not every organization surveys at the same time. Schools may administer their surveys in the fall, winter, or spring. Things like seasonal depression, anticipation of summer break, vacation, world events, and more could impact how students respond. Because the timing is different by county, we can’t be sure that the differences we see are not (at least partially) due to the timing of the survey.

Data cleaning methods: Each organization has different data cleaning protocols. “Data cleaning” refers to the process of making sure the data is in a usable form for analysis. When some surveying organizations take away more responses while cleaning than others, that could change the results. For example,
Some organizations may omit respondents who answered less than 95% of the survey, while others may omit respondents who answered less than 75% of the survey.
Some organizations may check whether responses make sense. This usually includes thinking about whether students responded differently about their behavior across multiple questions. For example, did a student say that they both never vaped, and vaped in the past month?