one time State Framework Questions

OMB Control No.: 2020-0031

Approval expires: 11/30/08

State Review Framework Evaluation

Survey Questions for State and Local Agencies

2/5/2021

Q1 Overarching Evaluation Question: Improvement of state program

consistency:

1. Is it your perception that the SRF process has improved consistency in core enforcement activities across programs in your state? Across states in your region? Why?

2. Were your state’s policies, levels of activities and data completeness found to be consistent with national policies?

   1. What changes or improvements were recommended? Were they made?

3. What other steps has your state taken to better align with national policy?

Q2 Overarching Evaluation Question: Improvement in consistency of EPA oversight of state programs:

1. Is it your perception that the EPA Region followed the SRF process?

2. Did the state feel that the report accurately reflected state performance?

3. Did the use of the 12 elements of the SRF as a standard for evaluating performance lead to EPA Regional oversight that was:

   1. Fair?

   2. Consistent across programs? Across states?

   3. Does it promote a level playing field across states?

4. Did the state feel that there were subjective factors that interfered with the process?

5. Did the regional reviewers appear knowledgeable about the SRF? About national policies?

Q3 Overarching Evaluation Question: Collaboration between EPA and states in implementing the SRF:

1. Did the EPA Regional review team work collaboratively with your state to implement the review? Did they do the following in a collaborative manner and tone:

   1. Conduct a preliminary conference call and other pre-review activities

   2. Discuss schedule and process for the review

   3. Discuss relevant documents and data to prepare for the on-site review

   4. Conduct a preliminary data analysis and review of data metrics

   5. Conduct an on-site review that included:

      1. Entrance meeting

      2. Close out meeting

   6. Share draft report (findings and recommendations)

   7. Review and address state comments on draft report

   8. Attach state’s final comments to the final report

   9. Discuss a plan for implementing the recommendations

   10. Negotiate plans for improvement into PPGs or other agreements

2. Did the report identify and recognize the state’s accomplishments and best practices?

3. Did you agree with the recommendations in the SRF report? If not, how were problems resolved? Was the process fair?

4. Do you feel that you have a better understanding of the federal enforcement program?

5. Do you feel that the Region has a better understanding of the state’s program?

Q4 Overarching Evaluation Question: Improvement in efficiency and effectiveness of SRF review process:

1. What components of the review were especially valuable or important? Which were not?

• Pre-review activities

• Offsite review activities

• Onsite review activities

• Draft report

• Final report and follow-up (see Attachment 1)

2. What steps or different approaches can be taken to improve the effectiveness and efficiency of the SRF?

3. What steps can be taken to reduce the cost of the reviews?

4. Did your state attend any SRF training? Did it help the state to understand national policies, targets and goals that were used to gauge performance?

5. Was the SRF training beneficial? How could it be improved?

Q5 Overarching Evaluation Question: Value derived by states and locals from SRF review and approach:

1. What do you see as the value of the SRF reviews to your state’s enforcement program?

2. From your perspective, were the advantages to the regions from conducting a consistent oversight system nationally?

3. What do you see as the value of the SRF to the national program?

4. What improvements could be made to enhance the value of the SRF reviews?

5. How would you like to see the information that is gathered through the SRF process used?

Q6 Overarching Evaluation Question: Use of differential oversight in future strategies:

Differential Oversight means applying different levels, types and degrees of oversight based on performance.

1. How can the results of the SRF review be used to effectively implement a differential oversight system?

2. What are the components of a differential oversight system that you would like to see implemented?

3. How would the levels of differential oversight be defined?

   1. What should be the results or benefits of good performance?

      1. Current Menu of benefits:

         1. Reduce frequency of reviews up to a 3 year cycle

         2. Reduce frequency of other oversight activities

         3. Conduct joint inspections to train rather than oversight inspections

         4. Ability to do self-evaluation and play more participatory role

         5. Provide flexibility in how a state applies its resources to allow the inclusion of state priorities, while still maintaining a balanced program

         6. Get recognition or offset credit for alternative approaches

         7. Coordinated approach to national initiatives (negotiate ability to lead cases)

         8. EPA provides extra funds for state priorities

         9. EPA promotes state as national expert in demonstrated areas

         10. State participates in national policy or regulatory efforts

         11. Increase availability of EPA specialized training opportunities.

         12. EPA provides public recognition of state program

   2. What should be the results of under performing?

4. What is the appropriate cycle for conducting SRF reviews for all states? Within that cycle, how much flexibility should exist for regions to handle states or programs differently that perform well?

Attachment 1

List of review components for Q4, Question 1

Pre-review activities

1. RA contacts state commissioner to set tone and context for review.

2. Region forms review team and prepares for review

3. Regional team has expertise required to conduct the review

4. Prior to review, Region provides the state with introductory letter explaining review process, schedule, etc.

Offsite review activities

5. Region and state identify any relevant reviews (within 2 years) to prevent duplication.

6. Query OTIS for data metrics, perform preliminary data analysis and share with state

7. Allow states to provide additional data

8. Regional team assesses all relevant state documents (PPA/PPGs, etc.)

9. Optional: negotiate additional program areas (i.e., assistance under Element 13)

Onsite review activities

10. Region and state participate in entrance meeting

11. Region and state determine the files to be reviewed

12. Region and state participate in exit meeting. Region presents preliminary significant findings and discusses timing for draft and final report.

Draft report

13. Region prepares and distributes draft report

14. State and OECA provide comments on draft report

15. Region and state work to reconcile differences in draft report

16. Region responds to OECA comments and makes changes to the report

Final report and follow-up

17. Final report includes comments from the state

18. Final report includes executive summary

19. Final report includes sufficient detail on data and file metrics, other state data and contains findings, conclusions, and recommendations

20. Final report is submitted to OECA and posted on Tracker

21. Region and state negotiate follow-up through PPA, grant agreements, etc

BURDEN STATEMENT

The OMB Control Number and expiration date must appear on the front page of an OMB-approved form or survey, or on the first screen viewed by the respondent for an on-line application. The rest of the burden statement must be included somewhere on the form, questionnaire or other collection of information, or in the instructions for such collection.

OMB Control No.: 2020-0031

Approval expires: 11/30/08

The public reporting and recordkeeping burden for this collection of information is estimated to average 1.96 per response. Send comments on the Agency's need for this information, the accuracy of the provided burden estimates, and any suggested methods for minimizing respondent burden, including through the use of automated collection techniques to the Director, Collection Strategies Division, U.S. Environmental Protection Agency (2822T), 1200 Pennsylvania Ave., NW, Washington, D.C. 20460. Include the OMB control number in any correspondence. Do not send the completed survey to this address.

5