SS Part B_1-6-12 (REVISED 2-16-12)_(Alternate) Clean (1)

Evaluation of the National Tobacco Prevention and Control Public Education Campaign

Supporting Statement: Part B

Centers for Disease Control and Prevention

National Center for Chronic Disease Prevention and Health Promotion

Office of Smoking and Health

Health Communications Branch

February 16, 2012

Project Officer: Robert Alexander, Ph.D.

Health Communications Specialist

Phone: (770) 488-1212

Fax: (770) 488-5939

Email address: RAlexander@cdc.gov

Centers for Disease Control and Prevention

4770 Buford Highway, Mailstop K-50

Atlanta GA 30341

Contents

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

B.2. Procedures for the Collection of Information

B.3. Methods to Maximize Response Rates and Deal with Non-response

B.4. Tests of Procedures or Methods to be Undertaken

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

List of Attachments

A-1. Public Health Service Act

A-2. Family Smoking Prevention and Tobacco Control Act

A-3. PPHF

B-1. Federal Register Notice

B-2. Summary of Public Comments and CDC Response

C-1. Smoker Baseline Questionnaire

C-2. Smoker Follow-Up Questionnaire

D-1. Non-smoker Baseline Questionnaire

D-2. Non-smoker Follow-Up Questionnaire

E-1. Screening and Consent Process

E-2. Email Notifications and Reminders

F-1. Knowledge Networks Panel Recruitment Procedures

F-2. Knowledge Networks Privacy and Confidentiality Procedures

G-1. RTI IRB Approval Notice

H-1 and H-2. Reference Articles on Knowledge Networks

I-1. Primary and Secondary Outcome and Survey Items

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

We will conduct a two-wave longitudinal survey of adult smokers and adult non-smokers in the United States to facilitate repeated measures on variables of interest. The first survey will be fielded during early February/March 2012 before the launch of a national campaign on smoking and health, serving as a baseline assessment. The second survey will be conducted approximately three months later at the end of the campaign. The pre-post design will facilitate analysis of relationships between individuals’ exposure to the campaign and pre-post changes in outcomes of interest. This longitudinal design will allow us to calculate baseline-to-follow-up changes in campaign-targeted outcomes for each study participant. We hypothesize that if the campaign is effective, the baseline-to-follow-up changes in outcomes should be larger among individuals exposed to the campaign more frequently (i.e., dose-response effects).

The primary study sample will consist of 3,000 baseline and follow-up sets of interviews among smokers in the U.S. from the Knowledge Networks panel. in addition to 2,000 baseline and follow-up sets of interviews of non-smokers, also from the KN panel. Evidence on the accuracy of self-reported data from the KN panel has been demonstrated in prior research, notably in two recent studies published in Public Opinion Quarterly (Chang & Krosnick, 2009 (Attachment H-1); Yeager, Krosnick, & Chang et al., 2011; Attachment H-2). These studies explicitly examined the comparison between KN panel survey results and results from RDD telephone and opt-in non-probability Web panels. Yeager et al. (2011) conducted an experiment by administering the same survey instrument to multiple samples which included seven non-probability Internet platforms and two probability-based survey platforms which included a RDD telephone survey and a probability-based Internet survey. Although it was not directly named, the Knowledge Networks KnowledgePanel was the probability-based Internet survey used in this study. This study showed that the KN panel was the most accurate in terms of primary demographics even compared to RDD telephone surveys. KN interview cases (unweighted) were on average 2.47 percentage points different than Census benchmarks compared to an average 3.43 percentage point difference for RDD telephone surveys. This may be due to improved coverage of cell phone only households in the KN address-based sampling frame which are usually excluded from RDD landline telephone surveys. The overall conclusion of these studies is that the foundations of statistical sampling were sustained in both types of probability samples (RDD telephone and KN panel data) and these data yield quite accurate results even when response rates are not especially high.

A secondary sample of 2,000 smokers will be recruited to the KN web survey from Survey Sampling International (SSI), a leading provider of online sampling in the U.S. SSI is a global online sampling provider consisting of a large ongoing panel of participants as well as participants from online communities, social media, and other partners affiliated with SSI. The KN panel will be the primary source of sample and will be used to generate a singular estimate of this outcome across a large, geographically and socio-demographically diverse sample of smokers and nonsmokers in the U.S. However, the KN panel alone may not be sufficient to assess awareness at more granular levels, particularly within smaller geographic areas. In order to obtain estimates of campaign awareness by smaller geographic regions, more sample is needed, particularly in areas with relatively small population densities. The additional sample of smokers from the SSI panel will help address this gap. This additional sample will be concentrated, to the extent possible, in geographic areas with less coverage by the existing KN panel. These data will be used to boost the overall sample sizes of specific geographic regions of interest that would otherwise not be sufficiently represented in the KN panel alone. The combined KN and SSI sample will therefore be used to provide estimates of campaign awareness (and other outcomes) at more granular levels than the U.S. as a whole.

The SSI sampling frame will be identified with pre-existing panel profile information, similar to the profile information that KN uses to pre-identify its own panelists for specific studies. That is, we will be able to pre-identify panelists from SSI who reside in certain geographic areas, based on existing geographic profile variables among SSI panelists. Geographic areas that will be oversampled will be based on the final media plan for campaign delivery which will help us identify any geographic areas of interest that are not well-covered by the KN panel alone. Generally, we will draw as many SSI panelists as possible from these areas to yield a minimum of 200 completed interviews per sampled area. In cases where larger regions are sampled that have larger existing SSI sample, panelists will be selected randomly from these areas to yield this many completed interviews. It is important to note that survey procedures for the supplemental sample of smokers including screening, consent, and survey completion will all occur within the KN-administrated survey just as with the primary smoker and non-smoker samples from the existing KN panel.

All data collected for this study will be weighted for analysis. KN will weight all data to facilitate separate analysis of KN-only sample (for singular estimates of a large, geographically and socio-demographically diverse sample as described above) and for analysis of pooled KN and SSI sample (for estimates of smaller geographic regions). Weights for the KN sample are calculated using a standard post-stratification weighting procedure that adjusts for survey non-response as well as non-coverage. This weighting procedure also applies a standard post-stratification adjustment based on demographic distributions from the most recent October 2010 data from the Current Population Survey (CPS). Benchmark distributions for Internet access used in this weight are obtained from the most recent (October 2009) special CPS supplemental survey measuring Internet access. Weights for the pooled KN and SSI sample are generated by first adding the SSI cases to the independently weighted KN sample. This combined sample is then reweighted using the weighted KN sample as its benchmark, resulting in sample that is similar to the KN panel in terms of its weighted demographic profile. As noted elsewhere in this information collection request, limitations of the pooled SSI and KN data must be acknowledged. While this data will provide valuable additional information to CDC about campaign awareness in these smaller geographic areas, the opt-in nature of the SSI sample limits our ability to project those results to those areas generally.

Study sample sizes were determined through power analyses that were conducted to determine the necessary number of interviews to detect specific relationships between self-reported campaign awareness and outcomes of interest. For purposes of this study, we examined existing evaluation literature and research to determine the expected effect sizes on the outcome of making a quit attempt. Based on these analyses, we have powered the study to detect an underlying odds ratio of 1.20 between self-reported campaign awareness and the likelihood of a quit attempt. This power analysis is based on KN sample sizes only, given that overall U.S. estimates will be derived from the KN panel alone. Previous media evaluations of statewide campaigns have demonstrated relationships of this magnitude between self-reported campaign awareness and the likelihood of a quit attempt. We have conservatively powered the sample to detect this effect at 80% power among KN smokers in the sample. For non-smokers we have reserved sufficient sample to detect this same effect on other outcomes relevant to non-smokers at the standard 80% power level.

Because there will be participant attrition between the baseline and follow-up surveys, we must collect enough interviews at baseline to yield the desired sample sizes at follow-up. Based on data from previous longitudinal studies we have conducted among smokers and from the Knowledge Networks panel, we conservatively anticipate a baseline-to-follow up retention rate of approximately 73% among existing Knowledge Networks panelists and 25% among the supplemental sample of smokers from outside the panel. Hence, we will collect a total of 4,100 interviews of Knowledge Networks smokers at baseline to yield a retained two-wave sample of 3,000 smokers at follow-up. The supplemental sample of off-panel smokers will contain approximately 7,500 baseline interviews and 2,000 interviews at follow-up. Because non-smokers are a higher-incidence population in the U.S. compared to smokers, we expect higher overall participation and study retention between waves. Based on our examination of data from previous studies with Knowledge Networks that have involved longitudinal designs with nonsmoking adults, we anticipate an approximate longitudinal retention rate of 75% among this population. Based on this attrition rate, we will collect 2,666 baseline interviews of non-smokers to yield a retained two-wave sample of 2,000 non-smokers at follow-up.

We anticipate that the initial cooperation rate from study invitations to complete the initial screenings will be approximately 70% among Knowledge Networks smokers and non-smokers and approximately 30% among the SSI sample. Based on this initial cooperation rates for screening, we anticipate that a total of 5,860 Knowledge Networks smokers, 3,800 Knowledge Networks non-smokers, and 25,000 off-panel smokers for the supplemental sample will complete the introductory screenings for this study. Therefore, the total number of unique respondents in this information collection is 34,660.

All decisions about assumptions that guided our power analysis were intended to err in favor of a larger sample size to safeguard for the possibility of being able to detect small effect sizes from the campaign. These assumptions increased our confidence that smaller effects produced by The Campaign than those found by previous prevention programs would be reasonably detected using the sample sizes we identified. As noted earlier, our sample design is also based on conservative assumptions about survey response. Thus our estimates of longitudinal retention rates should be viewed as “worst case” scenarios that if hold true, would still ensure sufficient sample sizes to reasonably detect small campaign effects.

It should be noted that while the KN panel’s recruitment procedures are designed to approximate a nationally representative sample, the limitations associated with the panel decrease our capacity to draw nationally representative conclusions about either smoking-related knowledge and behavior or the impact of the campaign on long-term quit rates in sub-populations. Although KN panelists must be invited to participate and cannot volunteer on their own, there may be systematic differences between individuals who choose to join an ongoing internet panel and the type of individuals who do not wish to participate in either an internet panel and/or over an ongoing committee. Furthermore, our estimates for smaller geographic areas are more limited with respect to representativeness because we will only be adding smokers who agreed to be in a separate ongoing panel, and we anticipate the response rate will be about 1/3 of that expected for the KN panel (i.e., 25% for the supplemental panel and 75% for the KN panel). Therefore, evaluation results must be interpreted with appropriate caution regarding our ability to generalize the findings to the national population of smokers and nonsmokers.

The KN-only sample will provide valuable information on the knowledge and behavior of a geographically and socio-demographically varied population of smokers and nonsmokers in the U.S., as well as any differences in knowledge and behavior after exposure to an intensive communication campaign. More granular conclusions about the awareness (and change in awareness) of smokers in certain geographic regions will be made possible by the supplemental panel sample, although the conclusions drawn from the combined sample will be more limited with respect to population representativeness than the KN sample due to a variety of methodological limitations.

The evaluation design used allows CDC to estimate the potential for this type of three-month national campaign to reach a large portion of the population, to gauge change in knowledge and immediate behaviors of smokers and nonsmokers, and to generate hypotheses about potential differences in responsiveness by geographical region. Study design limitations decrease our capacity to draw nationally representative conclusions about either smoking-related knowledge and behavior or the impact of the campaign on long-term quit rates in sub-populations. However, the design is the best available solution to CDC’s evaluation objectives, within the time, cost, and feasibility constraints noted above.

The following table provides a summary of respondents, by type, panel (source), and information collection (form name).

Type of Respondent	Information Collection/Form Name	Number of Respondents
Adult smokers, ages 18-54 (KN Panel)	Screening and Consent Process (Smokers)	5,860
	Smoker Baseline Questionnaire	4,100
	Smoker Follow-Up Questionnaire	3,000*
Adult non- smokers, ages 18-54 (KN Panel)	Screening and Consent Process (Non-smokers)	3,800
	Non-smokers Baseline Questionnaire	2,666
	Non-smoker Follow-Up Questionnaire	2,000**
Adult smokers, ages 18-54 (Supplemental Off-Panel)	Screening and Consent Process (Smokers)	25,000
	Smoker Baseline Questionnaire	7,500
	Smoker Follow-Up Questionnaire	2,000***
		34,660****

* Subset of original 4,100 KN Panel smokers collected in the Baseline Survey. ** Subset of original 2,666 non-smokers collected in Baseline Survey. *** Subset of original 7,500 off-panel smokers collected in Baseline Survey. ****Total number of unique respondents calculated as total respondents who complete introductory screenings.

B.2. Procedures for the Collection of Information

When the study is assigned to the sampled email addresses, individuals will receive email notification that the survey is available for completion. Nonrespondents will receive two e-mail reminders from Knowledge Networks requesting their participation in the survey. See Attachment E-2 for study email notifications and reminders. The surveys will be self-administered and accessible any time of day for a designated period. Participants can complete the survey only once. Study screeners will be used to determine study eligibility, including information on current smoking behavior. Eligible participants will include smokers and non-smokers in the U.S. and participants will be allowed to complete the survey in either English or Spanish. The Spanish language surveys will be identical in terms of items, question wording, and substantive meaning. The Spanish translations will be done in a culturally competent manner and all survey items will be cross-checked with Spanish-speaking adults. The Spanish language surveys will be provided upon OMB approval of the content of this information collection request. Informed consent will be sought from participants for participation in the Web survey. Participants will consent by selecting the appropriate link on the Web screen. A detailed description of Knowledge Networks’ panel recruitment methodology is provided with this submission (Attachment F-1).

We estimate that 11,600 smokers must be recruited to complete the Smoker Baseline Questionnaire (Attachment C-1) in order to yield 5,000 completed post-campaign Smoker Follow-Up Questionnaires (Attachment C-2).

We estimate that 2,666 non-smokers must be recruited to complete the Non-smoker Baseline Questionnaire (Attachment D-1) in order to yield 2,000 completed post-campaign Non-smoker Follow-up Questionnaires (Attachment D-2).

B.3. Methods to Maximize Response Rates and Deal with Non-response

The following procedures will be used to maximize cooperation and participation in this study:

 Participants from the KN panel will be offered 5,000 Knowledge Networks bonus points (equivalent to $5 cash) for completion of the baseline survey and 15,000 bonus points (equivalent to $15 cash) for completion of the follow-up survey. Thus, a total incentive of $20 will be offered to KN participants who complete the entire 2-wave study. The incentive schedule for SSI panelists is $1 for completion of the baseline survey and $3 for completion of the follow-up survey, respectively. This incentive structure is intended to recognize the time burden placed on them, encourage their cooperation, and to convey appreciation for contributing to this important study.

• Email reminders (Attachment E-2) will be sent to all sampled participants who do not complete their assigned survey within a given period of time after it is assigned. A second round of email reminders will be sent to nonresponders who do not complete the survey once the initial email reminder is delivered.

 An attempt will be made to locate participants who leave the Knowledge Networks panel before the end of this study. Location efforts will include mailings of refusal conversion materials designed to persuade participants to complete the study. In addition to using mailed refusal conversion materials, Knowledge Networks may also conduct telephone-based refusal conversion, contacting each non-responders via telephone.

 Knowledge Networks will provide a toll-free telephone number to all sampled individuals and invite them to call with any questions or concerns about any aspect of the study.

 Knowledge Networks data collection staff will work with RTI project staff to address concerns that may arise.

B.4. Tests of Procedures or Methods to be Undertaken

Prior to launching the baseline survey, we will field an eight-case pretest of the survey instrument. This survey will be identical to the instrument that will be used in this evaluation and approved by OMB with the exception of a few additional questions to assess overall clarity of instrument questions and respondent’s opinions on any aspects of the survey that were not clear. The purpose of the pilot test will be twofold: (1) to assess technical aspects and functionality of the survey instrument, and (2) to identify areas of the survey that were either unclear or difficult to understand. Once this pretest is completed, Knowledge Networks will create a data file for analysis by RTI International. This data file will contain diagnostic data on average time of survey completion, survey completion patterns (e.g., are there any concentrations of missing data?), and other aspects related to the proper function of the survey. We will also examine data on pilot test measures that will be used to assess the clarity of item wording and ease of understanding. Although this pretest will be conducted, such pretests rarely result in changes to the instruments. Therefore, we do not expect or plan to have any changes made to the instruments.

In addition to the aforementioned eight-case pretest, RTI will conduct rigorous testing of the online survey instrument prior to its fielding. RTI researchers will have access to an online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and in specification with instrument approved by OMB.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals inside the agency have been consulted on the design and statistical aspects of this information collection as well as plans for data analysis:

Tim McAfee, MD, MPH

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5709

Email: mtt4@cdc.gov

Terry Pechacek, PhD

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5592

Email: txp2@cdc.gov

Diane Beistle

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5066

Email: DBeistle@cdc.gov

Robert L. Alexander Jr., PhD, MPH, CHES

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-1212

Email: Ria8@cdc.gov

Jami L. Fraze, PhD, MEd, CHES

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5186

Email: Jnf0@cdc.gov

Bob Rodes, MS, MBA, MEd

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-5748

Email: Rur9@cdc.gov

Jeffrey McKenna, MS

National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K40

Atlanta, GA 30341

Phone: (770) 488-5131

Email: Jwm0@cdc.gov

Karen Debrot, DrPH, MNS, RD

Office on Smoking and Health

Centers for Disease Control and Prevention

4770 Buford Highway NE, Mailstop K50

Atlanta, GA 30341

Phone: (770) 488-1037

Email: Bol6@cdc.gov

The following individuals outside of the agency have been consulted on the questionnaire development, statistical aspects of the design, and plans for data analysis:

Kevin C. Davis, MA

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 541-5801

Email: kcdavis@rti.org

Jennifer Duke, PhD

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 485-2269

Email: jduke@rti.org

Donna Vallone, PhD

Legacy Foundation

1724 Massachusetts Avenue, NW

Washington, DC 20036

Phone: (202) 454-5783 
Email: dvallone@legacyforhealth.org

April Brubach

FDA, Center for Tobacco Products

9200 Corporate Boulevard

Rockville, MD 20850

Phone: (301) 796-9214

Email: april.brubach@fda.hhs.gov

The following individuals will conduct data collection and analysis:

Kevin C. Davis, MA

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 541-5801

Email: kcdavis@rti.org

Jennifer Duke, PhD

RTI International

3040 Cornwallis Road

Research Triangle Park, NC 27709

Phone: (919) 485-2269

Email: jduke@rti.org

References

Abreu, D.A., & Winters, F. (1999). Using monetary incentives to reduce attrition in the survey of income and program participation. Proceedings of the Survey Research Methods Section of the American Statistical Association.

Chang L. & Krosnick J.A. (2009). National surveys via RDD telephone interviewing versus the Internet: comparing sample representativeness and response quality. Public Opinion Quarterly. 74(4):641-678

Shettle, C., & Mooney, G. (1999). Monetary incentives in U.S. government surveys. Journal of Official Statistics, 15, 231-250.

Southwell B, Barmada C, Hornik R, et al. (2002).Can we measure encoded exposure? Validation

evidence from a national campaign. J Health Commun, 7:445-453.

Yeager D.S., Krosnick J.A., Chang L., et al. (2011). Comparing the accuracy of RDD telephone

surveys and Internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly. 75(4):709-747.