7/30/24
|
Benjamin
D. Hoffman, MD, FAAP
|
Necessity
and Utility of the Proposed Information Collection for the Proper
Performance of the Agency’s Functions
AAP
recognizes that there is significant variance in the structure,
funding, history, and implementation of PMHCA programs across all
states and territories and hopes the collected data will be
comparable across the multitude of PMHCA programs while still
considering these differences.
|
7/31/2024
|
JBS
has constructed a detailed Program Implementation Database that
captures detail on individual PMHCA and MMHSUD awardees’
programs, combining data from various sources, including awardee
applications and narrative reports, survey data, HRSA-required
reporting measures, and evaluation capacity-building calls. This
Database allows us to record and analyze key differences such as
program structure, funding, history, size, reported
implementation facilitators and barriers, and target population.
|
Ways
to Enhance the Quality, Utility, and Clarity of the Information
to be Collected
Additionally,
we recommend that HRSA consider using “mental and
behavioral health” in place of “behavioral health”
as well as “infant, child, and adolescent” instead
of “child and adolescent” in any survey language.
These relatively simple changes will improve comprehensive data
collection across the full range of populations and services
that PMHCA programs engage.
AAP
recognizes that training and education are part of the federal
PMHCA program goals, and we are interested in whether and how
programs are successfully enacting this goal in practice. We
recommend that HRSA clearly define what PMHCA program activities
are considered training for the purpose of the program
evaluation. Our members expressed some confusion about what this
may refer to, so additional clarity will likely improve data
fidelity.
AAP
has some potential concerns about HRSA’s plan to assess
changes over time in participating health practitioners’
capacity to address patients’ mental and behavioral health
and access to mental and behavioral health care through
screening. While relationships with PMHCA programs can improve
primary care providers’ familiarity with different mental
and behavioral health treatment paths, PMHCA programs are not
necessarily intended or prepared to train providers on how to
conduct mental and behavioral health screenings. PMHCA programs
are typically most valuable after a primary care provider has
identified the need for mental or behavioral health
interventions and consults the PMHCA program for further care,
not in conducting initial screenings.
To
the extent feasible, it is important that the data collection is
optimized for the various participants engaging with the program
models. We recommend that HRSA ensure that the data collection
is conducted in a manner that is clear and relevant for the full
range of anticipated survey and interview respondents, which
vary from pediatric primary care providers to community resource
partners and program champions.
AAP
would also recommend that HRSA collect the data in such a way
that insights can be gained regarding rural, urban, and suburban
access to care. If appropriate, we would also be interested in
information about the distance patients and families need to
travel to access mental and behavioral care recommended by PMHCA
teams. Are PMHCA programs considering potential barriers such as
distance and travel time when issuing recommendations in a
consultation? This is especially relevant for programs located
in areas considered mental and behavioral health deserts.
Thorough data collection about these barriers and others will
contribute to improved understanding of the existing access gaps
and better prepare HRSA and other stakeholders to take targeted
actions to close those gaps.
|
After
consideration of definitions from the American Medical
Association, American Psychiatric Association, the National
Alliance on Mental Illness, the Substance Abuse and Mental
Health Services Administration, and the World Health
Organization as well as input of from the PMHCA Impact Study
External Partner Group (EPG) from a separate impact study,
"behavioral health" was selected as the most
parsimonious and accepted term. The "child and adolescent"
terminology is not used in the surveys for the HRSA Evaluation
of the MCHB PMHCA and MMHSUD Programs Project.
For
the program evaluation, we collect survey data from health
professionals and practices on the number of trainings attended
and modality for training received from the PMHCA and MMHSUD
programs (i.e., in-person training event, webinar, self-study
with program resources, video conferencing, learning
collaborative [e.g., Project ECHO, Project REACH], other
training modality). HRSA also collects data on the total number
of providers trained as well as the number of trainings held by
topic, mechanism used (e.g., in-person, web-based), and type of
training materials used.
For
the evaluation, capacity is being operationalized as health
professionals’ BH knowledge, skills, practice, and
attitudes. Specific evaluation questions related to changes over
time in access to behavioral health services have been revised
to focus on change in knowledge and skills; screening,
assessment, treatment, and referral; attitudes about providing
behavioral health care; and how change over time differed based
on (1) frequency and modality of program access and (2)
treatment location, demographics, and treatment settings. These
evaluation questions will be able to measure screening behavior
change separately from other measures of provider capacity and
describe how screening and other capacity measures differ in
various contexts.
Different
strategies will be used to optimize data collection for the
various participants engaging with the program models. The
respondent universe for the Health Professional (HP) and
Practice-Level Surveys will comprise identified enrolled and/or
participating HPs and practice managers from all 2021, 2022, and
2023 PMHCA awardees and 2023 MMHSUD awardees. The respondent
universe for the Program Implementation Survey and
semi-structured interview (SSI) will comprise program
implementers (e.g., program directors [PDs]/principal
investigators [PIs] from all 2021, 2022, and 2023 PMHCA awardees
and 2023 MMHSUD awardees. The respondent universe for the
Behavioral Health Consultation Provider SSI, Care Coordinator
SSI, and Champion SSI will comprise 1 representative per
stakeholder group for each of the 67 PMHCA and MMHSUD The
Community-Based and Other Resources SSI will be a case study
with up to 50 community-based and other resources
representatives across all 67 PMHCA and MMHSUD programs.
HRSA
is collecting data in the HP Survey about the setting(s) in
which HPs’ patient populations live and in the
Practice-level (PL) Survey about practice setting, with response
options for questions in both surveys including urban, suburban,
rural, frontier. The PL Survey also collects whether the
practice is located in a federally designated medically
underserved area and a federally designated rural area. Finally,
both the HP Survey and the PL Survey collect the zip code of the
primary clinical practice setting. Specific data on travel to
access mental and behavioral care are not collected, though we
collect qualitative data from Care Coordinator SSIs. Relevant
data from the Care Coordinator SSIs include (1) barriers or
challenges to making referrals and it is possible that travel
may be one of the barriers identified, (2) types of resources
they have connected with to support clients and the geographic
areas in which the resources cover, (3).their systems or
databases used for managing resources and specific information
for each resource, which may include location of services and if
the resource provides telehealth services, and (4) the types of
support offered to facilitate a successful referral and it is
possible that taking into account travel time to resources prior
to making referrals would be an identified strategy discussed by
care coordinators.
|
Use
of Automated Collection Techniques or Other Forms of Information
Technology to Minimize the
Information
Collection Burden
AAP
supports the use of easily accessible automated collection
techniques as these technologies lower the collection burden and,
when the techniques include objective measures, increases the
validity of the measures.
|
The
evaluation of the MCHB PMHCA and MMHSUD programs will follow a
multimethod approach. Data collection methodologies for this
evaluation will use surveys (i.e., web-based, email) and virtual
SSIs (e.g., Microsoft Teams, Zoom). All technology used for the
survey administration (i.e., web-linked survey administered via
email and via survey platform) will meet federal requirements for
Section 508 accessibility.
We
selected the data collection methods for the evaluation because
they will reduce participant burden while providing the
evaluation with necessary data. Offering a web-based survey
reduces burden to participants by eliminating the time it takes
to write responses on a paper-based, mail-in survey. In addition,
having participants respond to an online survey eliminates the
time needed to mail back a paper-based survey. This reduces the
burden for respondents participating in interviews via a
web-based platform (e.g., Microsoft Teams, Zoom) because they
will not have to write down responses to the questionnaires or
travel to participate in an in-person interview.
Using
protected electronic data is the most secure form of data
management because it eliminates the possibility of either paper
documents or data being lost in transit or delivered to an
incorrect location.
|
7/30/24
|
Ashaki
M. Jackson, MFA, Ph.D.
|
The
necessity and utility of the proposed information collection for
the proper performance of the agency's functions
The
MAMA’S PROMISE team (referred to as PROMISE throughout)
finds the proposed mixed evaluation plan to be extensive. While
tools are not yet available, the approach – to evaluate
practitioners’ and implementation staffs’ workflows –
seems to be robust in understanding how grantees provide
services.
|
7/31/2024
|
The
evaluation design includes outcome and process evaluation, using
a mixed-methods design, with primary and secondary quantitative
and qualitative data collection activities across all HRSA MCHB
awardees. In addition, data are collected from various program
stakeholders including awardee PDs/PIs); enrolled/participating
HPs and practices; program champions; community resource partner
representatives; behavioral health consultation providers, and
care coordinators.
|
|
|
The
accuracy of the estimated burden
PROMISE
considers the estimated burden feasible with a timeline and ample
lead time to alert participants. We do question, however, what
options a grantee might have if they do not employ a likely
respondent. For example, PROMISE does not currently staff a
Champion. Are evaluation questions and the related estimated
burden eliminated for that staff, or will the evaluation team ask
similar questions of other staff, increasing their time burden?
|
|
If
the grantee does not employ a likely respondent, their program
will not be required to participate in that data collection
activity. For example, grantees were not required to have program
champions (and for data collection purposes, we have requested
that the identified program champions not be employed by the
awardee programs). However, in talking with awardees over time,
many discussed the value of having a program champion supporting
program implementation and sustainability. Again, as noted above,
if there is no program champion (or other likely respondent), the
program will not be required to participate in that data
collection.
|
|
|
Ways
to enhance the quality, utility, and clarity of the information
to be collected
PROMISE
offers these initial thoughts without full awareness of the
evaluation tools:
We
find the breadth of roles included in the evaluation promising
to more broadly understand how roles contribute to optimal
implementation. It is unclear, however, how participants’
voices will be reflected in this evaluation, if at all. PROMISE
considers it important to ask patients:
The
extent to which they think their provider sought the most
accurate, current intervention for their care;
The
extent to which patients knew and were comfortable with their
diagnoses being discussed to aid their care;
Perceived
care quality for their diagnoses; and
Clarity
of information shared by their provider if the PROMISE hotline
consultation, for example, yielded a change in care management.
We
believe that care receipt and a patient’s perception
thereof is part of implementation. We should note that PROMISE is
housed in a data-driven unit where participants are familiar with
study participation. Compensation for their time (e.g., gift
cards) is a standard that we build into our budgets.
Related
to evaluation implementation: to prepare likely
respondents for
the evaluation, it would be useful for the evaluation team to
brand HRSA-MMHSUD programming in marketing leading up to
evaluation activities so that providers can more easily respond
to questions. Given the numerous grant-funded programs
throughout the region, the evaluation team and grantees might
mention verbally and/or in materials that services are part of a
HRSA-MMHSUD initiative so that when the evaluation team recruits
participants will be more easily able to identify the
HRSA-MMHSUD grantee (e.g., PROMISE) and engage in conversation.
|
|
Patient-level
data collection is not included in the scope of the HRSA
Evaluation of the MCHB PMHCA and MMHSUD Programs Project.
To
prepare likely respondents for the evaluation surveys, we have
developed a promotion packet of materials with
branding guidance and customizable messages for awardees to use
to (1) increase HP and practice engagement with their programs
and (2) encourage participation in evaluation surveys. These
materials include emails, PowerPoint slides, graphics, and
social media messages. Additionally, the HP, Practice-Level,
and Program Implementation Surveys will be customized for each
program with the program name and logo, as applicable.
|
|
|
The
use of automated collection techniques or other forms of
information technology to minimize the information collection
burden
We
welcome software and strategies that will help the evaluation
while being minimally invasive to daily activities. Availability
polls that automatically create calendar appointments on days
evaluation participants note they are available; QR codes
embedded in email invitations that lead to surveys; automatic
email and calendar reminders; online text that is accessible to
screen readers; and the option to audio record verbal responses
to be transcribed later by the evaluation team are also helpful
tools that shorten the logistical time required for evaluation
activities.
Peripherally,
we would like to offer that shared grantee software to collect
provider and patient data of interest would have been useful to
this effort. Just as HRSA Healthy Start has ChallengerSoft
software available to its grantees for use via purchase of a
license, HRSA MMHSUD might consider one standard software build
that allows grantees to purchase user seats.
|
|
The
evaluation of the MCHB PMHCA and MMHSUD programs will follow a
multimethod approach. Data collection methodologies for this
evaluation will use surveys (i.e., web-based, email) and virtual
SSIs (e.g., Microsoft Teams, Zoom). All technology used for the
survey administration (i.e., web-linked survey administered via
email and via survey platform) will meet federal requirements
for Section 508 accessibility.
We
selected the data collection methods for the evaluation because
they will reduce participant burden while providing the
evaluation with necessary data. Offering a web-based survey
reduces burden to participants by eliminating the time it takes
to write responses on a paper-based, mail-in survey. In addition,
having participants respond to an online survey eliminates the
time needed to mail back a paper-based survey. This reduces the
burden for respondents participating in interviews via a
web-based platform (e.g., Microsoft Teams, Zoom) because they
will not have to write down responses to the questionnaires or
travel to participate in an in-person interview.
As
noted above, survey data collection for the evaluation of the
MCHB PMHCA and MMHSUD programs will be primarily through
web-linked survey administered via email and via survey
platform. Data that awardees are required to collect to meet
HRSA reporting requirements as part of the cooperative
agreements are collected through HRSA’s Electronic
Handbooks (EHB).
|