Att D - Assessment Strategy

Attachment D. NPHII Assessment Strategy

NPHII Assessment Strategy; Data Sources; Timing of Measurement and Notes

FINAL

11/16/2012

Outcome 1: Accreditation Readiness

Overarching Assessment Questions	Sub-questions	Specifics	Data Sources
To what extent has NPHII supported increased readiness of its grantees for accreditation by the Public Health Accreditation Board (PHAB)?	In what ways have NPHII grantees addressed the PHAB prerequisites?	Development of prerequisites % that have current prerequisite as defined by completion within the previous 5 years (each/all) % that completed prerequisite(s) in past year	Annual Assessment APR/IPR Years 2 and beyond Workplan Years 3 and beyond
		Use of prerequisites In what ways (if any) are each of the pre-requisites being used by the grantee organizations?	APR/IPR Years 2 and beyond Qualitative?
	To what extent have NPHII grantees completed self-assessments against the PHAB standards? For which domains and standards have NPHII grantees identified gaps?	Self-assessment against PHAB standards % of grantees that have completed self-assessments What is the extent of gaps identified: Overall By domain/standard	Annual Assessment Work plan Year 3 APR/IPR Years 2 and beyond
		Closing gaps against standards % of grantees conducting efforts to close gaps against standards Prioritized domains to address Outcomes of efforts– standards met
	In what ways has NPHII supported grantee processes necessary to prepare for accreditation?	% of grantees engaging in specific accreditation preparation processes such as: Developing a timeline for agency’s application to PHAB’s accreditation program Developing a “roadmap” to agency’s application to PHAB’s accreditation program Organizing agency workforce and documentation for accreditation Conducting communications with staff / leadership about accreditation That have submitted a statement of intent to pursue accreditation through PHAB	Annual Assessment APR/IPR
	In what ways has NPHII advanced accreditation readiness with other organizations in the grantee’s jurisdiction?	Support from grantees to other health agencies in grantee jurisdiction (e.g., LHDs, tribes) – % of grantees providing support Mechanisms/types of support provided (e.g., training and TA, funding such as mini-grants) # of jurisdictions receiving each type of support from grantees Funding: Amount of support to jurisdictions—potentially in total $ or Avg/jurisdiction Results achieved	Annual Assessment IPR/APR PIM-Network evaluation? Qualitative?
	In what ways have the various components of NPHII contributed to grantee readiness for accreditation?	How has each component of NPHII contributed to grantee accreditation readiness? What is the perceived value of each of the components? Components: PIM Knowledge of /ability to advance accreditation readiness PIM Network TA/CBA Grantee use and perceived effectiveness of TA to advance accreditation readiness Training (including grantee meeting) Guidance	Qualitative PIM Network Evaluation TA/CBA Evaluation APR/IPR
	Which organizational factors facilitate grantees’ accreditation readiness?	Aspects of organizational QI maturity Leadership support Other elements/domains from the QI maturity tool Performance Management Capacity PIM Educational background Tenure/turnover Existence of a performance management office Placement of the performance management office within the organization Training of other agency staff in accreditation-readiness related topics or skills	Qualitative Annual assessment (maturity tool) APR/IPR
	What challenges and successes have grantees experienced when implementing activities to improve accreditation readiness?	Challenges, successes, and lessons learned that could inform NPHII program improvement and PIM activities	Qualitative

Outcome 2: Increased Efficiency / Effectiveness through Quality Improvement

Overarching Evaluation Questions	Sub-questions	Specifics	Data Sources
To what extent has NPHII supported improved efficiency and effectiveness of grantees’ program-specific and/or agency-wide operations?	How are grantees addressing efficiencies through QI initiatives?	What factors influence the selection of specific initiatives? How are initiatives implemented (tools/methods used)? % of grantees addressing efficiencies through QI initiatives Overall By efficiency outcome (e.g., time saved, money saved)	APR/IPR: Could add as data elements captured with measures in APR/IPR for Years 3 and beyond Annual Assessment Year 2 Qualitative?
	How are grantees addressing effectiveness through QI initiatives?	What factors influence the selection of specific initiatives? How are initiatives implemented (tools/methods used)? % of grantees addressing effectiveness through QI initiatives Overall By effectiveness category (e.g., increased reach)	APR/IPR: Could be data elements captured with measures in APR/IPR for Years 3 and beyond Annual Assessment Year 2 Qualitative?
	What outcomes were achieved related to increased efficiencies / effectiveness?	Outcomes for QI initiatives % of grantees demonstrating success in achieving outcomes of efficiency or effectiveness Efficiency: % of grantees demonstrating successful achievement of efficiencies Overall For each outcome (e.g., time saved, money saved) Actual results of initiatives (data on time saved, money saved, etc.) Effectiveness: % of grantees demonstrating successful achievement of effectiveness outcomes Overall For each outcome (e.g., increased reach, satisfaction) Actual results of initiatives (data on increased reach, etc.)	Annual Assessment Year 2 APR/IPR: May be some data in Year 2 APR performance measures; Performance Measures Year 3 and beyond
	In what ways has NPHII advanced quality improvement activities and outcomes within other organizations in the grantee’s jurisdiction?	Support from grantees to other health agencies in grantee jurisdiction (e.g., LHDs, tribes) – % of grantees providing support Mechanisms/types of support provided (e.g., training and TA, funding such as mini-grants) # of jurisdictions receiving each type of support from grantees Funding: Amount of support to jurisdictions—potentially in total $ or Avg/jurisdiction Results achieved	Annual Assessment APR/IPR PIM-Network evaluation? Qualitative?
	In what ways have the various components of NPHII contributed to quality improvement activities and outcomes?	How has each component of NPHII contributed to quality improvement activities and outcomes? What is the perceived value of each of the components? Components: PIM Proficiency in QI-related competencies PIM Network TA/CBA Grantee use and perceived effectiveness of TA to advance quality improvement outcomes Training (including grantee meeting) Guidance	Qualitative PIM Network Evaluation TA/CBA Evaluation APR/IPR Annual Assessment
	Which organizational factors facilitate grantees’ positive QI outcomes?	Aspects of organizational QI maturity Leadership support Other elements/domains from the QI maturity tool Performance Management Capacity PIM Educational background Tenure/turnover Existence of a performance management office Placement of the performance management office within the organization Training of other agency staff in QI methods/tools	Qualitative Annual Assessment (QI maturity tool) APR/IPR
	What challenges and successes have grantees experienced when implementing activities to improve efficiency and effectiveness?	Challenges, successes, and lessons learned that could inform NPHII program improvement and PIM activities	Qualitative

Outcome 3: Increased Performance Management Capacity

Overarching Evaluation Questions	Sub-questions	Specifics	Data Sources
To what extent has NPHII supported the implementation of performance management in grantee organizations?	How have grantees implemented performance management systems?	% of grantees that have established each component % of grantees that have established / are maintaining a complete systems	Annual Assessment
		% of grantees using data / reports from performance management systems for select purposes In what other ways are grantees using performance management systems?	Annual Assessment Qualitative?
	In what ways have grantees strengthened their performance management capacity?	To what extent has PIMs’ proficiency in select performance management-related competencies improved? % of grantees that have increased performance improvement capacity as evidenced by: % of grantees with QI councils/committees % of grantees with an agency QI plan % of grantees that have spread PM through their organization as evidenced by: % of grantee staff with training in PM Number of staff dedicated to performance management? % of grantees with a dedicated PM office Location of office within the agency’s organizational structure	Annual Assessment APR/IPR
	In what ways has NPHII advanced performance management capacity and activities with other organizations in the grantee’s jurisdiction?	Support from grantees to other health agencies in grantee jurisdiction (e.g., LHDs, tribes) – % of grantees providing support Mechanisms/types of support provided (e.g., training and TA, funding such as mini-grants) # of jurisdictions receiving each type of support from grantees Funding: Amount of support to jurisdictions—potentially in total $ or Avg/jurisdiction Results achieved	Annual Assessment APR/IPR PIM-Network evaluation? Qualitative?
	In what ways have the various components of NPHII contributed to the implementation of performance management?	How has each component of NPHII contributed to the implementation of performance management? What is the perceived value of each of the components? Components: PIM Proficiency in performance management-related competencies PIM Network TA/CBA Grantee use and perceived effectiveness of TA to advance performance management capacity Training (including grantee meeting) Guidance	Qualitative PIM Network Evaluation TA/CBA Evaluation Annual Assessment APR/IPR
	Which organizational factors facilitate grantees’ implementation of performance management?	Aspects of organizational QI maturity Leadership support Other elements/domains from the QI maturity tool Performance Management Capacity PIM Educational background Tenure/turnover Existence of a performance management office Placement of the performance management office within the organization Training of other agency staff in performance management topics/skills	Qualitative Annual Assessment (QI maturity tool) APR/IPR
	What challenges and successes have grantees experienced with the implementation of performance management?	Challenges, successes, lessons learned that could inform NPHII program improvement and PIM activities	Qualitative

Beyond specific NPHII requirements

Overarching Evaluation Questions	Sub-questions	Specifics	Data Sources
In what ways has NPHII resulted in, or influenced, activities and outcomes beyond specific NPHII cooperative agreement requirements?	What are the unintended outcomes of NPHII?	What are the unintended positive outcomes of NPHII at grantee organizations? What are the unintended negative outcomes of NPHII at grantee organizations?	Qualitative
	What activities beyond cooperative agreement requirements have been conducted?	What additional activities have grantees implemented using NPHII funds (e.g., QI projects beyond minimum requirement, accreditation- or performance-management related activities)? What additional activities have occurred that were inspired or influenced by NPHII (e.g., locally-funded hiring of additional PM/QI staff, additional adoption of PM/QI)?	APR/IPR Qualitative
	What is the value-added of the PIM to the grantee organization as a whole?	What is the breadth of the roles/responsibilities filled by PIMs? To what extent have PIMs been integrated into the operations of their organizations? To what extent are PIM-related functions understood within the grantee organization?	Qualitative

Additional question to be pursued towards the end of the cooperative agreement / evaluation:

1. Which of the NPHII components are essential to sustaining the achievement of NPHII outcomes?

   1. PIM

   2. PIM Network

   3. TA/CBA

   4. Training (including grantee meeting)

   5. Guidance

Cross-cutting issues to be explored for relevant evaluation questions

• Context/stratification

• Grantee type (STLT)

• Funding level (has to be anchored in baseline/starting point)

• Starting point along continuum of PM and QI maturity

• Governance structure

• Free-standing versus super-agency

• Executive and legislative influences

3