Evaluation_PLC_Survey_SWA_TA_2025

OMB Control Number: 1205-0436

Expiration Date: 11/30/2026

`

Evaluation Peer Learning Cohort

Participant Survey

2025

According to the Paperwork Reduction Act of 1995, persons are not required to respond to this collection of information unless it displays a currently valid OMB control number and expiration date. Public reporting burden for this collection of information is estimated to average 15 minutes/60 hours per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.  The obligation to respond to this collection is voluntary. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, Employment and Training Administration, 200 Constitution Ave. NW, N5641, Washington DC 20210 & ATTN: Division of Research and Evaluation or Wayne Gordon at gordon.wayne@dol.gov and reference the OMB Control Number 1205-0436. Note: Please do not return the completed survey instrument to this address.

PROGRAMMER NOTES

Frequently Used Fills

In the boxes below, please list fills that are repeated frequently in your questionnaire requirements. These must come from a single source (whether from a preload or a question). The fills specified here do not need to be specified in the fill condition box each time they appear in a question.

Fill	Source / Condition	First Used at question #:
[TOPIC_1- 5]	Fill from Preload File: Topic_1-5 For OMB review: Topics and the number of topics may vary by cohort based on cohort needs. Potential topics include: Designing actionable evaluations Data collection and management Merits of different evaluation designs Equitable evaluation practices Analysis planning methods Evaluation reporting Securing funding Evaluation culture and awareness	C3

Introduction

The Employment and Training Administration (ETA), in the U.S. Department of Labor (DOL) is conducting a survey to obtain feedback on the Evaluation Peer Learning Cohort (EvalPLC) technical assistance (TA) activity, under which training and coaching on evaluation research were provided to state teams. The goal of this TA was to build states’ capacity to conduct evaluation research, as required the Workforce Innovations and Opportunity Act (WIOA), and articulated in the regulations at 20 CFR 682.220.

This survey is being distributed to all state workforce and partner agency representatives who participated as team members in EvalPLC activities (coaching, webinars, action planning, etc.). As a team member, your responses are vital for helping ETA to understand the effectiveness of the training and coaching provided and to identify other strategies to improve future EvalPLC sessions.

The survey should take about 15 minutes and asks about your:

1. role in your organization,

2. perceptions of your state team’s growth in building evaluation capacity through the EvalPLC TA

3. experience in participating in the EvalPLC TA

4. comments and suggestions about improving future EvalPLC TA

Please note that all information you provide will be kept private to the extent permitted by law. You will not be required to provide personal identifiers such as your name or address and your personal information will not be disclosed or identified in any report.

Your responses will be very much appreciated as ETA strives to improve the TA provided to states. Thank you, in advance, for completing this survey.

--The Research Team

Instructions

You are about to enter the survey. To go back a page, please use the survey’s “previous” button, not your browser’s “back” button.

Your answers will be saved each time you click “Next” and will be saved until the next time you log in. The survey takes about 15 minutes to complete.

Please answer all the questions the best you can. There are no right or wrong answers, and your responses are completely confidential. In reports responses will be presented as aggregate responses to protect individual identities. If you have questions about this survey, please contact EvalPLC@Mathematica-mpr.com for assistance.

Thank you for your feedback!

SECTION A.
Background Information

A1. Identify and select your state workforce agency’s region.

MARK ONE ONLY

[this will be a dropdown box]

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your response to this question is very important. Please select a response

A2. Please check the box that describes your role or roles within your organization.

Role

MARK ALL that apply

1  Executive Leadership

2  Finance/Budgeting/Accounting

3  Program Administration

4  Evaluation/Research/Performance/Data Collection

99  Other (specify) (STRING 255)

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your response to this question is very important. Please select a response

A3. Please check the box(es) of the WIOA program(s) which are your prime responsibility.

Program

MARK ALL that apply

6  Adult Title I

7  Youth Title I

8  Adult Education Title II

9  Wagner-Peyser Title III

10  Rehabilitation Services Administration Title IV

11 Other (specify) (STRING 255)

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your response to this question is very important. Please select a response

A4. Please select the option which best describes your participation in the EvalPLC cohort.

Select one only

 I am currently participating in a EvalPLC Cohort 1

 I participated in a EvalPLC Cohort in the past 2

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your response to this question is very important. Please select a response

SECTION B.
Peer Learning Cohort on Evaluation (EvalPLC) State’s Growth Areas

Your state team completed two assessments, the Evaluation Readiness Assessment and Evaluation Design and Implementation Assessment covering 10 key elements of evaluation planning activities. The categories in the next items were included in the Evaluation Readiness and Evaluation Design Assessments you completed at the beginning of the EvalPLC

B1. In which of the following areas (required for evaluation planning) did your state team experience growth during [IF A4 = 2; or after your participation in] the EvalPLC?

HOVER TEXT FOR “substantial growth”

Substantial growth: By substantial growth we mean that your team has experienced significant improvements in this particular area.

HOVERTEXT FOR “some growth”

Some growth: By some growth we mean that your team has experienced small or marginal improvements in this particular area.

(Please mark one per row)

	MARK ONLY ONE PER ROW
	Substantial growth in this area	Some growth in this area	No growth in this area, even though we would have liked to grow in this area	No growth in this area because it was not an area we focused on
Evaluation Readiness
a. Evaluation Culture and Awareness HOVERTEXT: familiarity with available resources; cross agency and partnerships that support evaluations; considering effective evaluation strategies with partners	1 	2 	3 	4 
b. Funding Strategies HOVER TEXT: using Governor’s statewide set aside funds for core programs; using discretionary grants to develop data infrastructure; pursuing additional funding through competitive grant programs; incorporating evaluation requirements into funding and procurement requirements	1 	2 	3 	4 
c. Data Management HOVER TEXT: accessing cross-agency administrative data through a centralized entity; participating in state or region data sharing agreements; leveraging federal data collection activities	1 	2 	3 	4 
d. Staff Skills, Capacity, and Knowledge HOVERTEXT: Having an in-house research and evaluation unit or staff; being familiar with evaluation types; knowing how to select design, conduct market research, articulate personnel requirements, and develop labor estimate for evaluations	1 	2 	3 	4 
e. Strategic Planning HOVERTEXT: Involving all key agencies in planning; developing an ‘evidence portfolio’ on subject areas of interest	1 	2 	3 	4 

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your responses to this question are very important. Please select a response for each row.

CREATE HOVER TEXT WITH DEFINITIONS OF “substantial growth” AND “some growth” AS WELL AS EACH CETEGORY NOTED IN ITEM.

If A4 = 1 AND B1. a, b, c, d OR e = 3

B1a. You indicated no growth even though your team would have liked to grow in this area in some areas under “Evaluation Readiness”, what additional supports would you need to grow in these areas?

(STRING)

B2. In which of the following areas (required for evaluation planning) did your state team experience growth during [IF A4 = 2; or after your participation in] the EvalPLC?

HOVER TEXT FOR “substantial growth”

Substantial growth: By substantial growth we mean that your team has experienced significant improvements in this particular area.

HOVERTEXT FOR “some growth”

Some growth: By some growth we mean that your team has experienced small or marginal improvements in this particular area.

(Please mark one per row)

	MARK ONLY ONE PER ROW
	Substantial growth in this area	Some growth in this area	No growth in this area, even though we would have liked to grow in this area	No growth in this area because it was not an area we focused on
Evaluation Design
a. Evaluation Design and Research Questions HOVERTEXT: developing logic models; performing evidence reviews; selecting research questions; engaging partners and key stakeholders; selecting evaluation design and timeline; considering pilot study	1 	2 	3 	4 
b. Data Collection and Analysis Plan HOVERTEXT: ability to handle collected data; identifying data sources, elements, and access; developing plan for primary data collection; developing an analysis plan which describes how data sources will be used and a plan for sampling	1 	2 	3 	4 
c. Evaluator Selection HOVERTEXT: Considering in-house, partner, or third-party evaluator; crafting an RFP for third-party evaluators; establish a proposal review committee for third party evaluators	1 	2 	3 	4 
d. Participant Rights HOVERTEXT: protecting privacy of participants; receiving IRB approval	1 	2 	3 	4 
e. Evaluation Reporting HOVERTEXT: designing deliverables to disseminate results to different target audiences; meeting with stakeholders to report results; planning to make reports publicly available; planning to create a public-use data file	1 	2 	3 	4 

PROGRAMMER: HARD CHECK IF NO RESPONSE; Your responses to this question are very important. Please select a response per row

CREATE HOVER TEXT WITH DEFINITIONS OF “substantial growth” AND “some growth” AS WELL AS EACH CETEGORY NOTED IN ITEM.

If A4 = 1 AND B2. a, b, c, d OR e= 3

B2a. You indicated no growth even though your team would have liked to grow in this area in some areas under “Evaluation Design”, what additional supports would you need to grow in these areas?

(STRING)

SECTION C.
EvalPLC State Team’s Technical Assistance Experience

C1. The items in this section ask about your experience with EvalPLC overall.

The following grid presents five different response fields that range from “strongly agree” to “strongly disagree.” Please mark one response in each row below.

	MARK ONLY ONE PER ROW
	Strongly disagree	Disagree	Neither disagree nor agree	Agree	Strongly agree
OVERALL EVALPLC EXPERIENCE:
a. My knowledge of evaluation research increased because I participated in the EvalPLC.	1 	2 	3 	4 	5 
b. After the EvalPLC, I felt more prepared to educate others on the types of evaluations.	1 	2 	3 	4 	5 
c. After the EvalPLC, I felt more prepared to identify possible areas for research	1 	2 	3 	4 	5 
d. After the EvalPLC, I felt more prepared to participate in the design of different types of evaluation research	1 	2 	3 	4 	5 
e. My state team’s collaboration on evaluation practices was enhanced after participating in the EvalPLC.	1 	2 	3 	4 	5 
f. I learned from other state teams how to address various challenges and develop options for evaluation research.	1 	2 	3 	4 	5 
g. If asked, I would recommend participating in a similar EvalPLC cohort to another state team.	1 	2 	3 	4 	5 

If A4 = 1

C2. The items in this section ask about your experience with virtual group events as part of EvalPLC.

The following grid presents five different response fields that range from “strongly agree” to “strongly disagree.” Please mark one response in each row below.

	MARK ONLY ONE PER ROW
	Strongly disagree	Disagree	Neither disagree nor agree	Agree	Strongly agree
VIRTUAL GROUP EVENTS:
a. The EvalPLC events offered my state team information we would not have had access to otherwise.	1 	2 	3 	4 	5 
b. The objectives of each EvalPLC event that occurred were clearly articulated and presented	1 	2 	3 	4 	5 
c. The subject matter expert (SME) presentations increased my understanding of the topics and issues discussed.	1 	2 	3 	4 	5 
d. Our state team wanted even more time to hear from SMEs during EvalPLC events.	1 	2 	3 	4 	5 
e. In future EvalPLC events I would recommend inviting the same SMEs to present.	1 	2 	3 	4 	5 
f. My state team wanted even more time for peer-to-peer exchanges during each event.	1 	2 	3 	4 	5 

If A4 = 1

C3. What two topics covered in group EvalPLC events were the most helpful or relevant to your state team?

Select up to two

 [TOPIC_1] 1

 [TOPIC_2] 2

 [TOPIC_3] 3

 [TOPIC_4] 4

 [TOPIC_5] 5

 None of the topics were helpful or relevant to my state team 99

PROGRAMMER NOTE: ONLY ALLOW UP TO TWO OPTIONS TO BE SELECTED

C4. Are there any topics that were not covered in group EvalPLC events [IF A4 = 2; or during your EvalPLC experience, generally,] that would have been helpful or relevant to your state team? If yes, please describe topics that would have been helpful.

(STRING)

 _NA None

If A4 = 1

C5. The items in this section ask about your coaching experience.

The following grid presents five different response fields that range from “strongly agree” to “strongly disagree.” Please mark one response in each row below.

	MARK ONLY ONE PER ROW
	Strongly disagree	Disagree	Neither disagree nor agree	Agree	Strongly agree
COACHING EXPERIENCE:
a. The coaching support for my state team added value to the EvalPLC experience.	1 	2 	3 	4 	5 
b. The coach was knowledgeable about research and evaluation.	1 	2 	3 	4 	5 
c. The coach was able to answer my state team’s questions about evaluation.	1 	2 	3 	4 	5 
d. The amount of involvement that the coach had with my state team was about right.	1 	2 	3 	4 	5 
e. The time spent on the capstone project development by the team was valuable	1 	2 	3 	4 	5 
f. The time spent on the capstone project development by the team was productive.	1 	2 	3 	4 	5 
g. The capstone project developed by the team during this cohort will be (or was) a useful resource for my state.	1 	2 	3 	4 	5 
h. Based on my experience with EvalPLC coaches, my team is able use the evaluation toolkit independently	1 	2 	3 	4 	5 

C6. Based on your experience with the EvalPLC capstone project, what knowledge or skills, related to evaluation readiness have you been able to apply to evaluation work in your state?

Select all that apply

 Familiarity with available evaluation resources 1

 Creating cross agency and partnerships that support evaluations 2

 Considering effective evaluation strategies with partners 3

 Using Governor’s statewide set aside funds for core programs 4

 Using discretionary grants to develop data infrastructure 5

 Pursuing additional funding through competitive grant programs 6

 Incorporating evaluation requirements into funding and procurement requirements 7

 Accessing cross-agency administrative data through a centralized entity 8

 Participating in state or region data sharing agreements 9

 Leveraging federal data collection activities 10

 Having an in-house research and evaluation unit or staff 11

 Being familiar with evaluation types 12

 Knowing how to select design, conduct market research, articulate personnel requirements, and develop labor estimate for evaluations 13

 Involving all key agencies in planning 14

 Developing an ‘evidence portfolio’ on subject areas of interest 15

 Other (Specify) 99

Specify (STRING 255)

C7. Based on your experience with the EvalPLC capstone project, what knowledge or skills, related to evaluation design, have you been able to apply to evaluation work in your state?

Select all that apply

 Developing logic models 1

 Performing evidence reviews 2

 Selecting research questions 3

 Engaging partners and key stakeholders 4

 Selecting evaluation design and timeline 5

 Considering pilot study 6

 Ability to handle collected data 7

 Identifying data sources, elements, and access 8

 Developing plan for primary data collection 9

 Developing an analysis plan which describes how data sources will be used and a plan for sampling 10

 Considering in-house, partner, or third-party evaluators 11

 Crafting an RFP for third-party evaluators 12

 Establish a proposal review committee for third-party evaluators 13

 Protecting privacy of participants 14

 Receiving Institutional Review Board (IRB) approval 15

 Designing deliverables to disseminate results to different target audiences 16

 Meeting with stakeholders to report results 17

 Planning to make reports publicly available 18

 Planning to create a public-use data file 19

 Other (Specify) 99

Specify (STRING 255)

SECTION D.
Other Feedback and Comments

Finally, we would like to give you an opportunity to provide any additional information you think may be useful.

If A4 = 1

D1. What do you think about the format of the EvalPLC program (e.g. virtual meeting format, the frequency and timing of group sessions and coaching sessions, number of attendees)? What, if any, recommendations would you make to strengthen the format?

(STRING)

D2. What do you think about the content covered in the EvalPLC? What, if any, recommendations would you make to strengthen the content?

[If A4 = 2; Thinking of your research and evaluation experiences after the Eval PLC, what would you have been helpful to have been covered during the Eval PLC?]

(STRING)

D3. What additional areas of support should the EvalPLC provide to state teams?

(STRING)

 _NA None

D4. Please describe any challenges you faced with the EvalPLC.

(STRING)

 _NA None

D5. If you would like the EvalPLC Technical Assistance team to contact you about your experience, please type in your e-mail address in the text field below:

(EMAIL ADDRESS)

Thank you for taking the time to complete this survey. We will use your answers to provide feedback to enhance the TA provided through the Peer Learning Cohort on Evaluation (EvalPLC) by the DOL-ETA.