Grant Agreement Phase

Strengthening Mobility And Revolutionizing Transportation (SMART) Grant Program

Exhibit C (PROJECT EVALUATION PHASE): Evaluation Plan Guidance for SMART Grant Recipients

USDOT will provide SMART Grant recipients with guidance on the type of information that should be included in their Evaluation Plans. The following tables highlights the type of guidance that will be provided.

Evaluation Plan Topics

Project Overview

• Describe the project and highlight the technologies being deployed

• List the project stakeholders (project team, partners, evaluation team) and describe roles and responsibilities, particularly with respect to completing the evaluation

• Summarize what constitutes end-of-project successes

• Provide a deployment and evaluation schedule in terms of months and years; include project milestones

Evaluation Goals/Objectives and Evaluation Questions

• Describe project evaluation goals and/or objectives and associated evaluation questions (or hypotheses)

• Develop at least one evaluation question for each goal or objective; multiple specific, evaluation questions are better than a few general ones

Performance Measures (PMs) (see Table 2)

• Identify one or more outcome based performance measure(s) for each evaluation question

• Ensure (describe how) you are meeting the performance measures prescribed in BIL.

• Develop system performance measures that measure whether the technology is functioning as intended (i.e. to verify the functionality of the technology).

• Ensure (describe how) your PMs are measurable within the scope of the evaluation. If targets are described, ensure they are appropriate

• Think about the unit of analysis (metric) needed for your analysis and the geographic scope.

• Describe the data sources for each PM (include existing data sources as well as primary data collection). If your agency is uncertain about the data sources or elements, indicate what data you would need to measure the PMs and note that updates to the plan will include more details on “X”.

Evaluation Methodology (see Table 2 below)

• Describe the method(s) that will be used to address each evaluation question (likely a mix of quantitative and qualitative methods)

  • Describe the experimental design, as appropriate (before-after; treatment-control)

  • Describe potential confounding factors

• Ensure the evaluation design enables the measurement of the proposed PMs; identify the specific data elements that are required

• Describe any limitations or risks associated with the method or the data elements

• If multiple technologies are deployed, be clear how the different technologies will be evaluated; consider organizing the evaluation plan by “Use Cases.”.

Data Collection Procedures and Data Management

• Describe how the data will be collected, including any plans for a pilot

  • For example, for surveys, plan should include: general method of recruitment; sample size; potential survey topics

  • For field studies, plan should include: location, data collection frequency, data collection period

• Address data management (e.g., data logging and transmission to the evaluation team (if applicable); data storage; data access and privacy protection; data fusion (if applicable), data quality checks, etc.) [note: for existing systems, less detail may be needed]

1