2024 Psychometric Testing Plan

OMB Control Number 0910-0932 

Expiration Date 05/31/2027

National Youth Tobacco Survey (NYTS)

Psychometric Testing Plan

Contract No. 75F40120A00017-75F40125F19048

Submitted to:

Center for Tobacco Products

Food and Drug Administration

Submitted by:

RTI International

P.O. Box 12194

Research Triangle Park, NC 27709-2194

March 27, 2023

RTI Project No. 0218678

Public reporting burden for this collection of information is estimated to average 30 minutes per survey, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: Office of Operations, Food and Drug Administration, Three White Flint North, 10A-12M, 11601 Landsdown St., North Bethesda, MD 20852, PRAStaff@fda.hhs.gov, ATTN: PRA (0910-0932).

Contents

Introduction 1

Segmentation Approach for Cognitive Testing 2

Recruitment Plan 2

Interview Guides 3

Cognitive Testing Analysis and Data Management 4

Introduction

This effort involves cognitive testing of the National Youth Tobacco Survey (NYTS). In cognitive testing, an interviewer administers survey questions to a respondent while collecting additional verbal information about survey responses with the goal of either evaluating the quality of the responses or determining if the wording is generating the intended information.¹ The results of the cognitive interviews may lead to revisions of survey questions and/or changes in question sequencing and overall questionnaire structure that will enhance response rate and data quality.

While the Food and Drug Administration (FDA) is partnering with the RTI/Deloitte Team to perform psychometric testing of the NYTS in 2026, the agency may reevaluate the need and feasibility of such testing due to potential time constraints. Below provides our current approach to conducting cognitive testing, previously approved by OMB. Constructs we plan to test are outline in Table 1 and specific items are in Attachment N5.

Table 1. Cognitive Interview Constructs

Constructs for cognitive interviews	Items
E-cigarettes	E-cigarette module E-cigarette or vape terminology E-cigarette substances (THC, CBD, Wellness vapes) E-cigarette use intensity/frequency E-cigarette ceremonial/tradition vs. commercial use
Socio-demographic status	Sexual orientation Family affluence Parental education level
Social Determinants of Health (SDoH)/Health Equity	Neighborhood environment Discrimination

There are currently 160 questions in the survey to be considered for testing which may change as new questions are considered. Interviews will take approximately one hour, but time will be based on the number of questions determined to be used in cognitive interviews. The cognitive testing will involve testing unstandardized items or items that have been identified as potentially problematic and review of each item and discussion facilitated through cognitive probes or questions related to the respondent perceptions of the survey items. Any items that have not changed, are not unstandardized, or were not identified as potentially problematic will not be include in our cognitive test. Our testing will include factors such as clarity of survey instructions and understanding of questions. Cognitive interviewing techniques will be guided by strategies that include respondent narrative, verbal probing techniques, and “think-aloud” interviewing techniques.

This document represents the psychometric testing protocol. The protocol consists of the following sections.

1. Segmentation Approach

2. Recruitment Plan

3. Interview Guides

4. Analysis & Data Management

Segmentation Approach for Cognitive Testing

Following the approved protocol, RTI/Deloitte proposes five population segments with associated selection prioritization to ensure representation across 24-30 cognitive interview participants. We will use a specialized web panel of parents/guardians used on several studies. We will select approximately 24-30 youth to participate in two rounds of testing and offer an incentive aligned to OMB approval per interview. All 24-30 participants will be asked all of the cognitive interview questions during both rounds. When recruiting, we will request that all participants be willing to engage in two rounds of cognitive testing to participate. Both rounds of cognitive testing will be conducted remotely. Two rounds of cognitive testing will be conducted with the same 24-30 participants. The second round of cognitive testing will use similar methods as the first, sampling the same 24-30 participants, and re-testing the items that were revised between rounds 1 and 2. This will also enable us to calculate test/re-test reliability for items that are not altered between rounds of testing to confirm the efficacy of changes made to survey items after round one. The five population segments include: school grade level (youth in grades 6 through 8, youth in grades 9 through 12); ever use tobacco, race/ethnicity (American Indian or Alaskan Native, Asian, Black or African American, Hispanic, Native Hawaiian or Pacific Islander, or white); low socioeconomic status (household income less than 250% the federal poverty level); and sex. This segmentation will provide adequate representation by grade level to draw meaningful conclusions for youth in middle or high school. Although we intend to recruit a diverse sample, segmentation is subject to change based on the ability to recruit the target population we have identified in Table 2. We will aim to oversample participants of minority race and ethnicity and participants with low socioeconomic status (SES) to provide additional insights into challenges experienced by vulnerable populations.

Table 2: Target Recruitment Sample

	Middle School Male	Middle School Female	High School Male		High School Female
N=24-30	N=12-15			N=12-15
Tobacco use	Mix of ever use/never use
Race/ethnicity	Mix of white/non-white
Income level	250% Below federal poverty line (FPL)

Two screening surveys will be administered to collect demographic and tobacco product use status data (one for parents, and one for students—following parental consent). These data will be used to identify respondents who qualify for one or more segments of interest. First, we will invite parents to complete the screener survey about their children. Once we achieve a sufficient response from parents, we will first move to balance each segment by grade level, sex, race/ethnicity, and SES. Following student screening, we will diversify each segment using tobacco product use status. [See the next section on recruitment for more detail on how RTI/Deloitte will achieve diverse segmentation in the sample population.] By monitoring the representation across each segment during recruitment, we will be alerted to attempt to fill gaps in underrepresented segments and, as a result, adjust our recruitment efforts. In selecting respondents to invite to participate in the interviews, we will prioritize diversification of selected respondents by tobacco product use status, race/ethnicity, SES status, and sex but there is not a guarantee we will reach our target recruitment segmentation sample.

Recruitment Plan

RTI/Deloitte will recruit respondents in grades 6 through 12 (age approximately 11 to 18 years) through two stages. Stage 1 ensures national reach, while Stage 2 allows more targeted supplementing of specific groups, if necessary.

Stage 1 involves recruiting children who are in grades 6 through 12 from the nationwide web panel. RTI/Deloitte will work with panel administrators to identify the most cost-effective approach that matches testing goals for recruiting participants from adult panels. Interview participants will be recruited through adult panelists who have children in their household (which is a characteristic included in the panel profile). These parents/guardians will be prompted to complete a screener survey which will allow us to identify those with children aged 11 to 18. Parents with children aged 11 to 18 who are deemed eligible will then be sent further information about the study, as well as a form to obtain consent for their child to participate. After obtaining parental consent, students will then be sent an invitation to complete a youth screener survey so that RTI/Deloitte can obtain additional relevant information and draw on this information to achieve the desired segmentation when selecting students to invite to participate. Students who are selected will be sent a consent form to complete prior to participation. All students who complete the screener but are not selected will be informed about this. The screener surveys, as well as the consent and assent forms are written for a 6^th grade reading level to ensure ease of access to information for both parents and youth.

If there are any gaps in age groups or other segments not found using a panel, we can supplement with Stage 2 of our recruitment strategy. This next step includes working with another recruitment firm to address any gaps in recruitment, if necessary.  We will identify a firm that has experience recruiting hard-to-reach populations including youth, ethnic minority populations, and people who use drugs. Firms like this use a national panel to recruit participants, and conduct extensive outreach with national, regional, and local organizations to supplement their efforts to support specific outreach efforts. We will evaluate which firm will provide the most cost-effective approach for testing goals before using them. We then again obtain parental consent, student assent, and conduct a short youth screener with all participants identified through the recruitment firm. This two-stage approach helps ensure a cognitive testing sample that fills each of the segments discussed above in the Segmentation Approach.

Interview Guides

Cognitive interviews will involve a review of new items, unstandardized items or items that have been identified as potentially problematic. Interview guides will be used to facilitate discussion through cognitive probes or questions related to the respondent’s perceptions of the survey items. Trained interviewers will build rapport and achieve a working flow with participants, during an interview that is no longer than two hours.

Based on their responses to key gate items in the e-cigarette module (e.g., having used e-cigarettes in the past 30 days), the number of items respondents see during the interview will vary for that section. We have included required probes for some individual items – some items that present more difficult cognitive tasks do have required probes, while others will be probed at the section-level. Additionally, interviewers will be instructed to probe whenever a respondent hesitates or seems unsure about an item, using generic probes laid out in the interviewer guides. At the same time, interviewers will ensure that interviews can be completed within the two hours allotted.

Interviews will be conducted virtually through Zoom video calls, and each interview will be attended by the interviewer and notetaker. Participants will be provided with instructions for accessing the meeting via their browser. All interviews will also be audio recorded and transcribed through Zoom. Interviewers will use the Zoom screen share feature to show participants the relevant survey items. Respondents will be shown items using a version of the web questionnaire that has been programmed to present only the items identified to be addressed during cognitive interviews. After introductions and a simple practice question, the interviewer will guide the respondent through the items. The interviewer will instruct the respondent to read and verbally respond to each question. The interviewer will enter the responses and advance through the survey. Interviewers will stop at the end of each section to probe. Interviewers will similarly proceed through the guide, entering participant responses to items, and using specific, generic, or emergent probes for in-the-moment probing on items that appear challenging for the respondent.

Cognitive Testing Analysis and Data Management

The following represents a preliminary analysis plan. Early in the data collection process, RTI/Deloitte will submit to FDA expanded plans for data analysis and reporting for cognitive interviews.

As a first step in qualitative data analysis, the analysis team will use an abductive coding process to develop a codebook to guide thematic analysis. The initial codebook will contain deductive codes, which will be developed based on the interview items and cognitive goals. In addition, the team will conduct an inventory of the data through review of a subset of the transcripts and notes to assess emerging themes and inform the development of inductive codes designed to capture issues or stories that may be unexpected, important to include, and/or unlikely to be captured by the deductive coding scheme. The team will develop inclusion and exclusion criteria for codes, provide explicit guidance for applying the codes, and ensure that codes are both comprehensive and mutually exclusive to avoid misclassification of data. This initial codebook will be provided to FDA for review and feedback.

All data from the interviews (i.e., transcripts and notes) will be imported into MAXQDA, which will be used to organize and manage the data and to conduct analysis in an iterative fashion. As noted, initial codes will be developed based on interview items and cognitive goals. In addition, the software program will facilitate the development of additional codes as the analyses are conducted. The software program will support the development of a coding system and coding process by allowing the text to be marked for subsequent search, retrieval, and classification. The software will facilitate interpretation of the data, identification of themes, tracking of representative quotes and text segments, and reporting of results. Analysts will review the transcripts and notes and select and code sections of text to organize responses according to analytic categories.

To ensure intercoder reliability, all coders will undergo a brief training prior to beginning live coding. Next, coders will code approximately 10% of the transcripts (to be randomly selected). As a team, coders will then review and resolve any differences in coding across these transcripts. Coders will also periodically and proactively share their ongoing coding experience, questions, and potential tweaks. Coders will also regularly share and discuss new codes and/or themes as these emerge. Any new codes and/or themes will be shared with FDA before formally being added to the codebook.

Upon completing the coding, analysts will prepare output for analysis by selecting a code and/or codes, retrieving the relevant text, and saving the output in files for analysis. The coding software will allow analysts to organize and pull codes across all interview items, cognitive goals, as well as by respondent types. Analysts will examine all responses and/or categories in the output to identify patterns and themes in the data.

All documents produced and/or collected before, during, and after each interview (e.g., screener data, audio files, interview notes, transcripts) will be stored in password-protected electronic files accessible only to Deloitte project team members. Data files will be stored on a secure drive by respondent ID, a five-digit code used to track all data collection. Respondent names, or other identifiers, will never be used. The separate, task specific, secure drive will only be available to a limited number of project staff working on the analysis.

The data storage ‘system’ consists of the Deloitte corporate laptops of those individuals authorized to participate in the study (estimated to be 6-10 individuals), and a secure shared drive. Each laptop is its own boundary and requires 2-factor authentication to log in. The laptop operating systems are Windows 10. The laptops will run Microsoft Office software which may be used for list data. De-identified response data will be captured to MS Office documents and then uploaded to MAXQDA for analysis. Microsoft Teams will be used to capture audio recording of interviews. Data sharing between team members within Deloitte will happen via the secure folder, as will delivery of results to the FDA when done. All list data, response data, and audio recordings will be wiped from the laptops at the conclusion of the project using drive-wipe software with multiple passes.

At the conclusion of the project, analyzed and deidentified data (audio files and transcripts) will be delivered (disassociated from any PII) to FDA.

1 Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.