B1. Development of Concepts, Methods, and Design
The Consumer Assessment of Healthcare Providers and Systems (CAHPS®) is developed by the Agency for Healthcare Research and Quality (AHRQ), a division of the Department of Health and Human Services (HHS). It is the industry standard for measuring patient experience and satisfaction. CAHPS surveys are utilized by a wide variety of public programs, including Medicare, Medicaid, TRICARE, Veterans Affairs (VA), and states. In addition, health plan accrediting bodies all require commercial health plans to conduct a CAHPS survey of their enrollees in each line of business as part of the accreditation process. The National Quality Forum is the endorsement body for healthcare quality and customer experience measures and has endorsed CAHPS through its rigorous process.
There is no alternative customer experience survey in health care that is as thoroughly vetted, methodologically sound, widely known and utilized as the existing CAHPS survey for health plans.
Obtaining OMB approval and a control number for CAHPS surveys is done by all the federal entities listed in the previous paragraph, for example the Centers for Medicare and Medicaid Services’ (CMS) Medicare Advantage and Prescription Drug Plan CAHPS® Survey MA PDP CAHPS OMB number 0938-0732.
As part of its Federal Employees Health Benefits (FEHB) Program and the Postal Service Health Benefits Program (PSHB) Plan Performance Assessment (PPA), OPM is requiring all FEHB Carriers and PSHB Carriers to utilize the CAHPS commercial survey using a certified CAHPS vendor. The PPA ties FEHB and PSHB Carriers’ profit to performance on key metrics (48 CFR 1615.404-70).
The AHRQ CAHPS surveys provide a uniform and industry accepted survey vehicle to assess member satisfaction. According to the AHRQ website:
The CAHPS Health Plan Survey is a tool for collecting standardized information on enrollees' experiences with health plans and their services. It was designed to support consumers in assessing the performance of health plans and choosing the plans that best meet their needs. Health plans can also use the survey results to identify their strengths and weaknesses and target areas for improvement. Survey results can be used to:
Support consumers in assessing the performance of health plans and choosing the plans that best meet their needs.
Identify the strengths and weaknesses of health plans and target areas for improvement.
OPM uses the Adult Commercial Survey. The survey utilizes a sampling methodology as required by the survey licensor, the National Committee for Quality Assurance (NCQA), outlined in extensive detail in the HEDIS Volume 3: Specifications for Survey Measures includes surveys and protocols for the CAHPS 5.1H Survey. Information can also be found on the AHRQ website. Recommended data collection modes include mail, email and telephone, and protocols have been developed for each mode.
Two modes of survey administration are allowed during the national implementation to give facilities options in how they would like to administer the survey, based on their goals and resources. These two modes are described below:
Mail-only Mode
Mailing of the questionnaire and cover letter to all sampled members.
Reminder postcard mailing
Second mailing of the questionnaire and cover letter to sampled members who do not respond to the first questionnaire mailing within 5 weeks
Second reminder postcard mailing
Third mailing of the questionnaire and cover letter to sampled members who do not respond to the first or second questionnaire mailing within 3 weeks of the second questionnaire mailing.
Mixed Mode (Mail with Telephone Follow-up)
Mailing of the questionnaire and cover letter to all sampled members.
Reminder postcard mailing
Second mailing of the questionnaire and cover letter to sampled members who do not respond to the first questionnaire mailing within 5 weeks
Second reminder postcard mailing
Telephone follow-up with all sampled patients who do not respond to one of the questionnaire mailings. A maximum of six telephone contact attempts per sampled patient will be made to complete the survey.
NCQA provides a standardized internet data collection protocol enhancement for optional use by health plans and survey vendors (prior approval is not required). Survey vendors may add an email component to the internet enhancement which allows vendors to email the cover letters, surveys and reminders to members with valid email addresses. The email component is optional (prior approval is not required). Specifications for the standardized HEDIS CAHPS internet enhancement are included in HEDIS Volume 3: Specifications for Survey Measures.
AHRQ versions of the survey are administered by CMS as part of their public reporting and reimbursement programs. The CAHPS surveys utilized by CMS have received PRA approval. CAHPS is also utilized by many private health insurance purchasers and is part of the NCQA accreditation process.
B2. Collection of Data
Under the FEHB and PSHB contract, FEHB and PSHB Carriers must administer CAHPS consistent with the AHRQ guidelines required by NCQA. OPM requires all FEHB and PSHB Carriers to use a certified CAHPS vendor when administering the CAHPS Survey.
To become an NCQA-Certified survey vendor, “an organization must demonstrate that it has the capabilities, experience and expert personnel to collect and report survey results accurately. NCQA issues an annual request for proposal (RFP) for new organizations to apply for certification.”
AHRQ first launched the CAHPS Survey in 1995 to develop standardized surveys that organizations can use to collect comparable information on patients’ experience with care, and to generate tools and resources to support the dissemination and use of comparative survey results to inform the public and improve health care quality.
The AHRQ CAHPS surveys provide a uniform and industry accepted survey vehicle to assess member satisfaction.
Development of the CAHPS Surveys
The development process for CAHPS surveys emphasizes scientific rigor and frequent input from patients and other key stakeholders. Major steps include literature reviews and environmental scans, focus groups with patients, input from health care providers and other key stakeholders, cognitive testing of survey questions and reporting measures, and field testing. This process is designed to ensure that the survey will generate valid and reliable data to meet the information needs of health care consumers, care providers, health plans, purchasers, and policymakers.
To achieve the program’s goals, AHRQ funds private research organizations with proven expertise in survey design and evaluation, public reporting, and quality assessment and improvement. The current organizations are RAND and the Yale School of Public Health. These research organizations work with AHRQ and other Federal agencies to develop, test, and maintain the surveys and to produce resources that support and evaluate their use in reports and improvement initiatives.
Collectively, these organizations are known as the CAHPS Consortium.
Uses of CAHPS Survey Results
Once AHRQ’s CAHPS Consortium releases a CAHPS survey into the public domain, it is available for any organization to use for its own purposes. The results of CAHPS surveys are typically used to monitor and drive improvements in patient experience with care and to better inform consumers about health care providers in their area. Some organizations incorporate the survey results into programs that reward or recognize health care providers for providing high-quality care.
The target population for OPM’s CAHPS Survey is FEHB and PSHB members who are eighteen and older without Medicare as their primary coverage and were enrolled in a FEHB or PSHB health plan. The estimated target population is 2.27 million people. This sample frame is the eligible population and is required by AHRQ and NCQA. The anchor date to be eligible is December 31 of the measurement year (i.e., the calendar year before the survey is administered).
AHRQ and NCQA require a systematic sampling method. The sample is drawn by the certified survey vendor. The steps, as shown in Section 3 of NCQA’s HEDIS 2023 Volume 3, Specifications for Survey Measures, are:
Step 1 |
Determine the minimum required sample size (MRSS) from Table S-3. If the number of eligible members in the sample frame is ≤ MRSS, include the health plan’s entire eligible population in the sample (steps 2–7 are not needed in this case). |
Table S-3: Minimum Required Sample Sizes
Survey Type |
Required Sample Size |
Adult Commercial |
1,100 |
Adult Medicaid |
1,350 |
Child Medicaid |
1,650 |
Step 2 |
Determine the oversampling rate based on anticipated disenrollment and response rate (refer to Oversampling section). |
Step 3 |
Determine the final sample size (FSS). The FSS includes the MRSS (from step 1) plus oversample (from step 2) and is calculated by the following formula: FSS = MRSS + (MRSS ´ oversampling rate) (round up to the next whole number), where MRSS = the minimum required sample size. For example, if the MRSS is 1,100 and a 10 percent oversample is needed, FSS = 1,100 + (1,100 ´ 0.10) = 1,210. |
Step 4 |
Using the validated sample frame, sort the list of eligible members (EM) in alphabetical order by last name, first name, date of birth and address. Sort EMs from Z to A. Note: Sort order applies to all components. Sort all fields by descending order (i.e., last name descending, first name descending, date of birth descending, address descending). |
Step 5 |
Calculate N = EM/FSS. Round down to a whole number. Determine
N, which is used in the formula to determine which member will
start your sample. N = EM/FSS where EM = the eligible member population (the number of eligible members in the sample frame) and FSS = the final sample size (step 3). |
Step 6 |
Calculate START = (RAND ´ N). Before choosing members, determine the member to start with (START). It is important that the sample be selected from a single pass through the member list. START can have many values and still allow only one pass.
|
|
NCQA provides survey vendors with list of Random Numbers (RAND) each with a value between 0 and 1. Survey vendors use the RAND to calculate the starting point from which to draw the final sample. Calculate the number from which to start drawing the final sample as follows: START = (RAND ´ N) (round per the .5 rule to the nearest whole number greater than 0), where RAND = the random number. |
Step 7 |
Select the sample, choosing every ith member using the formula: ith member = START + [(i-1) x (EM/FSS)] (rounding [(i-1) x (EM/FSS)] per the .5 rule to the nearest whole number greater than 0).
For i = 2,3,4, …, FSS where EM = the eligible member population (e.g., the number of members in the sample frame). FSS = the final sample size (step 3).
Starting with the member corresponding to the number START, choose every ith member until the FSS is met.
If the oversample was underestimated and the sample falls below the MRSS, the oversampling rate must be adjusted and a new sample must be pulled.
|
|
Small numbers: A health plan with very small numbers of eligible members (where denominators of less than 100 are expected) should contact NCQA to determine specific HEDIS survey reporting requirements. The health plan can submit concerns through the NCQA PCS system at https://my.ncqa.org.
|
|
Oversampling: A health plan must oversample if it cannot eliminate disenrolled members from membership files; correct addresses and, when appropriate, telephone numbers; and provide updated, accurate sample frames to the survey vendor by the required date. For example, a health plan that receives December membership updates from purchasers in January may need additional time to enter updates into its membership files. Similarly, a health plan that does not receive December membership updates from purchasers until late February may be unable to update membership files before generating the sample frame. The oversampling rate must be a whole number (e.g., 7%). Health plans must oversample enough to guarantee that sufficient eligible members are surveyed to meet the required sample size. Health plans may also oversample to obtain a greater number of completed surveys at the end of the survey administration. For example, the health plan may oversample if it has a prior history of low survey response rates; if it anticipates that a significant number of the telephone numbers in the membership files are inaccurate; or, if after reviewing the information in Appendix 7: General Recommendations for Oversampling for Survey Measures, it does not expect to achieve a denominator of 100 for most survey calculations. If the oversample was calculated correctly, the members remaining in the sample will be greater than or equal to the MRSS.
|
The surveys collected and processed by NCQA-Certified survey vendors, and are submitted through the NCQA Portal, and access is restricted by NCQA. NCQA validates the submission, determines benchmarks for each measure, and provides results to health plans and purchasers. Respondents are informed about the data collection in the instructions for the survey. OPM contracts with a vendor to further analyze and validate the aggregated results, and then produces a dataset for OPM’s use. The surveys are administered annually generally between January and May.
Applicable FEHB and PSHB Carriers are required to report their CAHPS results, and many report the results for their entire book of business of which FEHB or PSHB is a part. Participation in the survey is entirely voluntary. The burden on participants is not considered onerous, as the estimated time to complete the survey is fifteen minutes.
The results are aggregated and used as part of the formula to determination of FEHB or PSHB Plans profit factor. In addition, aggregated CAHPS survey results are provided on two OPM webpages to help inform selection of FEHB health plans. Aggregated CAHPS survey results are provided on one OPM webpage to help inform selection of PSHB health plans. Specific Carrier scores are reported within a range for all platforms.
B3. Processing and Editing of Data
Standard data cleaning and analysis procedures are performed by the NCQA Certified Vendor. The vendor is responsible for validating the results, including making sure that the sample protocol is followed, and all other requirements are followed. Only then is the data submitted to NCQA. The OPM vendor does additional validation and analysis under a separate contract with OPM. Currently, the OPM vendor is a NCQA-Certified survey vendor.
B4. Production of Estimates and Projections
The CAHPS sampling protocols are set by NCQA and applied by the NCQA-Certified vendor for each health plan. This ensures that they are statistically valid, and comparisons of results can be made across all carriers. NCQA-Certified vendors identify any biased results, and those results are excluded from use. For a specific measure identified as biased, the results are not used in the analysis or for the calculation of profit factor. They are also not made available to consumers.
NCQA auditors review the sample for material bias. If this bias is found, they do not approve the sample frame for use. For the survey sample frame, material bias is caused by a (+/-)10% difference between the eligible population and the survey sample frame. For example, if an error in the generation of the sample frame results in the exclusion of more than 10% of eligible members, the sample frame is considered materially biased, and the auditor does not approve the sample frame. The health plan must correct errors and regenerate the sample frame, or it receives an audit result of Not Applicable (NA) for the survey and any measure collected through that survey.
B5. Data Analysis
Basic survey analysis techniques (e.g., comparing to commercial benchmarks produced by NCQA for CAHPS, calculating frequencies, patterns, etc.) will be used to interpret and explain the data provided by respondents. FEHB results will be used in the FEHB Plan Performance Assessment, and ranges for each FEHB Plan are posted on the OPM website (https://www.opm.gov/healthcare-insurance/healthcare/plan-information/compare-plans/quality) and included in the results for the FEHB Plan Comparison Tool (https://www.opm.gov/healthcare-insurance/healthcare/plan-information/compare-plans/). PSHB results will be used in the PSHB Plan Performance Assessment, and ranges for each PSHB Plan are posted on the OPM Decision Support Tool (Home - Postal Service Health Benefits)
B6. Review and Evaluation Procedures
As noted above, the survey tool was developed by AHRQ and is the industry standard, used for multiple government healthcare programs including Medicare and Medicaid, by NCQA for accreditation, by states for their healthcare programs, and throughout the industry. PRA approval has been granted on numerous occasions for CAHPS in CMS programs.
B7. Data Dissemination
Aggregated reporting on the CAHPS survey results are provided on two OPM webpages to help inform selection of FEHB and PSHB health plans. Specific carrier scores are reported within a range.
No PII is reported on the CAHPS results. The results are aggregated, and no individual responses are reported.
B8. Contact Person(s)
Meredith Gitangu, Senior Policy Analyst
Healthcare and Insurance
U.S. Office of Personnel Management
Meredith.Gitangu@opm.gov
Johanna Kalin, Branch Chief
Healthcare and Insurance
U.S. Office of Personnel Management
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Sobek, Lauren M. |
| File Modified | 0000-00-00 |
| File Created | 2025-11-01 |