CCBHC Evaluation- BSCA OMB_SSB_7 10 24

Evaluation of the Certified Community Behavioral Health Clinic Demonstration in Accordance with the Bipartisan Safer Communities Act

Supporting Statement – Section B

Submitted: XX XX, 2024

Program Officials/Project Officers

Judith Dey, Ph.D. and Laura Jacobus-Kantor, Ph.D.

U.S. Department of Health and Human Services

Office of the Assistant Secretary for Planning and Evaluation

200 Independence Avenue SW, Washington DC 20201

Section B - Collection of information employing statistical methods

1. Respondent universe and sampling methods

Virtual interviews with state Medicaid and behavioral health officials: The evaluation team will identify the appropriate state respondents based on their knowledge of state demonstration leadership and organizational structure. They will also consult with demonstration leadership within CMS and review state websites as needed.

Virtual interviews with CCBHC site leadership: The evaluation team will conduct interviews with the CEO, medical director, and any other key staff recommended by the CCBHC at up to 15 clinics per year in years three and five. The evaluation team will collaborate with ASPE to purposively select a diverse group of CCBHCs in states that encompass the demonstration’s different payment models, organizational structures, and implementation contexts. The team will use responses to past and new CCBHC surveys and input from internal subject matter experts to identify clinics with characteristics of interest. The final list of CCBHCs will likely reflect factors such as geographic location; a mix of prospective payment system models available to demonstration states; the types of partnerships between CCBHCs and other organizations; and other characteristics of interest.

Virtual focus groups with clients: The evaluation team plans to recruit clients from the same CCBHCs at which it conducts interviews and conduct up to four focus groups in year five. The team will leverage the relationships developed with CCBHC leaders during the virtual interviews to help identify clients for the focus groups. Recruiting clients from the same CCBHCs that participate in virtual interviews is an efficient way to help ensure this data collection activity also reflects a balance of CCBHC characteristics, and allows for triangulation of responses across both a CCBHC’s staff and clients. If there are more than four clinics interested in helping to recruit for focus groups, the evaluation team will consider applying additional selection criteria to choose among them.

Strategies for recruitment of clients for focus groups will be tailored to each clinic, and the evaluation team will consider potential bias introduced by the recruitment procedures. The team will work closely with selected CCBHCs to develop an initial sampling frame and procedures for inviting clients to participate (for example, recruiting among participants receiving services added as a result of CCBHC certification). Prior to the focus group, the evaluation team will work with CCBHCs to obtain informed consent from clients (using consent materials we supply). The evaluation team expects to have CCBHCs obtain written consent from ten to twelve clients. Of those clients, the team assumes that approximately eight will ultimately attend, given their experience with typical no-show rates for focus groups among this population. Ultimately, the evaluation team plans to include up to eight clients in each focus group, for a total of approximately 32 clients. If more (or fewer) people attend than expected, the evaluation team will adjust its sampling approach over time.

CCBHC surveys: The evaluation team will ask all CCBHCs participating in the demonstration to submit the survey in the fourth and fifth years of the evaluation. The team will not use sampling methods.

2. Procedures for the collection of information

Virtual interviews with state officials: The evaluation team will conduct semi-structured virtual interviews with state Medicaid and behavioral health officials (mental health and substance use disorder agency officials) in each active demonstration state in each evaluation year to gather information on demonstration progress at different stages of implementation. Two researchers will conduct each interview, with one leading the interview and the other taking notes. The interviews will be conducted using WebEx and will be recorded and professionally transcribed.

Virtual interviews with CCBHC site leadership: The evaluation team will conduct an interview by telephone with the CEO and medical director, and any other key staff recommended by the CCBHC at up to 15 clinics per year in evaluation years three and five. Two researchers will conduct each interview with one leading the interview and the other taking notes. The interviews will be conducted using WebEx and will be recorded and professionally transcribed.

Virtual focus groups with clients: The evaluation team will conduct up to four focus groups with clients of CCBHC services during the fifth year of the evaluation. Two researchers will conduct these focus groups with one leading the focus group and the other taking notes. The focus groups will be conducted using WebEx or Zoom and will be recorded and professionally transcribed.

CCBHC survey: The evaluation team will conduct the CCBHC survey in the fourth and fifth evaluation years. The survey will be conducted through Confirmit, a user-friendly online platform. The survey team will review daily reports to assess completion rates overall and by state and work closely with state officials to encourage participation among CCBHCs. The survey will include skip patterns to reduce burden on CCBHC respondents and improve consistency of data collection.

3. Methods to maximize response rates and deal with nonresponse

As recipients of CCBHC demonstration funding, states are expected to demonstrate high levels of cooperation and be fully engaged as partners in the evaluation, encouraging CCBHCs to collaborate with the evaluation team regarding the interviews, focus groups, and submission of surveys. The evaluation team will work with state officials and CCBHC leadership to identify a mutually convenient date and time to conduct the interviews. For the focus groups, the evaluation team will work with CCBHCs to identify incentives that will encourage attendance at focus groups and schedule them at dates and times that will allow client participation. The evaluation team will also work closely with state officials to strongly encourage clinic participation in the survey by enlisting state demonstration leaders to send reminders and provide feedback to state officials on response rates.

4. Tests of procedures or methods to be undertaken

There will be no official pre-test for the interview protocols, focus group protocols, or the survey because these instruments are similar to instruments used without difficulty in previous evaluations of the CCBHC demonstration and other similar evaluations the evaluation team has conducted.

5. Consultants on statistical aspects of the design and people who will collect and analyze the information

In September 2023, ASPE awarded a task order to Mathematica and its subcontractor, the RAND Corporation, to design and conduct the evaluation (Task Order number HHSP233201500035I/ 75P00123F37074). Mathematica, Inc. designed the evaluation in conjunction with ASPE officials Judith Dey (Contracting Officer’s Representative) and Laura Jacobus-Kantor. The evaluation design team includes Allison Wishon, Kate Stewart, Stefanie Pietras, Rachel Miller, and Jonathan Brown (Mathematica) and Joshua Breslau, Michael Dunbar, and Courtney Kase (the RAND Corporation). The evaluation will be conducted by Mathematica, the RAND Corporation, and Advocates for Human Potential under contract with ASPE.