Form NHTSA Form 1178 NHTSA Form 1178 NHTSA.gov Usability Study

Usability Study NHTSA.gov October 15, 2012

OMB Control Number 2127-0682 Expiration Date 2/28/2015

A federal agency may not conduct or sponsor, and a person is not required to respond to, nor shall a person be subject to a penalty for failure to comply with a collection of information subject to the requirements of the Paperwork Reduction Act unless that collection of information displays a current valid OMB Control Number.  The OMB Control Number for this information collection is 2127-0682.  Public reporting for this collection of information is estimated to be approximately 55 minutes per response, including the time for reviewing instructions, completing and reviewing the collection of information.  All responses to this collection of information are voluntary.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: Information Collection Clearance Officer, National Highway Traffic Safety Administration, 1200 New Jersey Ave, S.E., Washington, DC, 20590

NHTSA.gov Usability Study Scenarios

NHTSA.gov Website Usability Study

Moderator Scenarios

Usability Study Scenario

Notes

Participant:__________________________________ Briefing Thanks and introduction. Today we will be looking at a vehicle safety website. The owner of this site wants to ensure it is an easy-to-use resource for consumers, so they’d like to get some feedback on consumers’ use of the website. Explain the scenarios Will be asked to complete a series of tasks on the website I’ll ask you to read a scenario out loud Think out loud as you try to complete the task (Play-by-Play) The purpose of this session is to make the website as user-friendly as possible, so please be honest and remember that there are no wrong answers! Video Consent Team members observing Recording for note-taking purposes only May end the session at any time May take a break at any time Any questions? Before we begin, I have a few questions about your experiences: When I say “vehicle safety”, what comes to mind? What are some topics you believe should be included on a website about vehicle safety? What information is most important? Have you looked for information on vehicle safety in the past year? When you’re doing research in this area, what type of information are you looking for? What information on vehicle safety would help you in your daily life? Where (do/would) you go for information on vehicle safety? How important is a vehicle’s safety performance to you when you are purchasing a vehicle? Are you looking for information about features, or actual vehicle performance in a crash? Testing Notes: All respondents will see Scenario 1 first Scenarios 2-5 will be randomized

		Notes
	Scenario #1 – Evaluate the website homepage Key Objectives Evaluate the overall ease of use of the homepage. Gather feedback on the overall visual design. Assess users’ knowledge of sharing and subscription features. Task 1: General homepage review The purpose of this task is to obtain initial impressions of the home page. This will lend insight into whether or not they would use the site in their research if they came to it via search engine. You’re on the internet looking for information about vehicle safety and click on a link that takes you to the page shown here. Spend a few minutes exploring the page, then I’ll ask some questions. Please stay on the homepage at this time, we’ll explore additional sections later. Intended User Steps: Scroll up and down on the homepage providing feedback on what they see. No other actions needed at this time. [http://www.nhtsa.gov/] Supporting Questions: Tell me your first impression of this page. What website is this? Who runs this website? How do you know? On a scale of 1 to 5 (5 being most confident), how confident or not confident are you? Have you visited this website before? What were your reasons for visiting the website? Do you remember how you found the website? How was your experience using the website? What type of content do you expect to find on this site? Is the page design appealing or unappealing? Is the content easy or difficult to read? Would you say the page is well organized or cluttered? What, if anything, do you find confusing about this page? Looking only at the homepage, on a scale of 1 to 5 (5 being the easiest), how easy do you believe this website is to use? What would improve the ease of use of the website? Task 2: Connect with NHTSA The purpose of this task is to understand whether or not users would connect with NHTSA beyond this site and how easy or hard it is to do so. You are looking to receive ongoing information from NHTSA. How would you go about establishing this connection? Intended User Steps: Sign up for Email Updates at the top of the page [https://public.govdelivery.com/accounts/USDOTNHTSA/subscriber/qualify?email=Enter%20Email%20Address] -or- Click on either Facebook or Twitter icons -or- Click on RSS Feeds at the bottom of the page[http://www.nhtsa.gov/Misc/RSS] -or- Click on Contact NHTSA at the bottom of the page [http://www.nhtsa.gov/Contact] Supporting Questions: How easy or hard was it to find a way to connect with NHTSA? If you wanted to receive more information from NHTSA, what would be your preferred method to receive that information? If you wanted to receive information about updates to this specific website, what is your preferred method of receiving these updates? Do you use RSS feeds? What do you use for an RSS reader? How would you go about subscribing to RSS feeds from this website? How likely are you to use social networking tools to connect with NHTSA? What would motivate you to connect with NHTSA? Where would you go to sign up for email updates? On a scale of 1 to 5 (5 being most likely), how likely would you be to sign up for email updates?

Scenario #2 – 5-Star Safety Ratings Key ObjectivesEvaluate the ease of accessing 5-Star Safety Ratings. Evaluate the 5-Star Safety Ratings search tool ease of use. Determine the likelihood that a user returns to SaferCar.gov to obtain information on 5-Star safety ratings. Task 1: Identify information on the 5-Star Safety Ratings program. The purpose of this task is to gauge how easy or hard it is to navigate to the safety ratings tool. You’re talking with a friend about their new vehicle and they mentioned that it received a 5-Star crash rating from the government. You’re curious to find more information on these ratings and discover that NHTSA is the agency responsible for the 5-Star Safety Rating program. Using this website, please find information about these ratings. Intended User Steps: Click on 5-Star Crash Ratings image under Key Issues on NHTSA.gov homepage [http://www.safercar.gov] Click on the 5-Star Safety Ratings image on SaferCar.gov homepage [http://www.safercar.gov/Safety+Ratings] -or- Click on 5-Star Crash Ratings image on NHTSA.gov homepage [http://www.safercar.gov] Click on Vehicle Shoppers [http://www.safercar.gov/Vehicle+Shoppers] Click on 5-Star Safety Ratings item in left menu [http://www.safercar.gov/Safety+Ratings] -or- Click on 5-Star Crash Ratings image on NHTSA.gov homepage [http://www.safercar.gov] Click on Vehicle Shoppers [http://www.safercar.gov/Vehicle+Shoppers] Click on 5-Star Safety Ratings link on landing page [http://www.safercar.gov/Safety+Ratings] Supporting Questions: How easy or hard was it to identify the 5-Star safety ratings from the homepage? Based on what you see here, what are the 5-Star safety ratings and how are they derived? You may have noticed you were taken to a different website. How much, if at all, did this impact the ease of finding this information? How do you think this search tool will work? What, if any, additional information do you need about this topic? Based on what you see, what is the difference between “1990 – 2010” 5-Star Safety Ratings and “2011 – Newer” 5-Star Safety Ratings? What, if anything, would you change about how information is presented on this page? Would it be more helpful for you if this information is in a PDF format, video format or presented on a separate page on this website? Task 2: Find the 5-Star Safety Ratings of User’s Primary Vehicle The purpose of this task is to gauge how easy or hard it is to find the safety ratings and supporting information for a specific vehicle. All this information has left you wondering what the ratings are for the vehicle you drive most often. Using this site, search for these 5-Star Ratings. Tell me aloud the steps you take to do so. Intended User Steps: Click 5-Star Crash Ratings image under Key Issues [http://www.safercar.gov] Click Vehicle Shoppers [http://www.safercar.gov/Vehicle+Shoppers] Click 5-Star Safety Ratings on the side menu Click 1990-2010 Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/1990-2010+Vehicles] -or- Click 2011-Newer Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/2011-Newer+Vehicles] Click Search by Model Select make, model & year -or- From the homepage, click 5-Star Crash Ratings under Key Issues [http://www.safercar.gov] Click Vehicle Shoppers [http://www.safercar.gov/Vehicle+Shoppers] Click 5-Star Safety Ratings on the side menu Click 1990-2010 Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/1990-2010+Vehicles] -or- Click 2011-Newer Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/2011-Newer+Vehicles] Click Search by Class Select your vehicle class -or- From the homepage, click 5-Star Crash Ratings under Key Issues [http://www.safercar.gov] Click Vehicle Shoppers [http://www.safercar.gov/Vehicle+Shoppers] Click 5-Star Safety Ratings on the side menu Click 1990-2010 Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/1990-2010+Vehicles] -or- Click 2011-Newer Vehicles [http://www.safercar.gov/Vehicle+Shoppers/5-Star+Safety+Ratings/2011-Newer+Vehicles] Click Search by Manufacturer Select your vehicle manufacturer Supporting Questions: How easy or hard was it to find the ratings for this vehicle? What, if anything, would you change to make this easier? Tell me about the ratings for this vehicle. How do you feel about the way the information is organized? Would you say it is well-organized or cluttered? How do you feel about the level of content on this page? (too much, too little, too in depth, not in depth enough) What, if anything, would you change about this page to make it easier for you to use? Is there anything on this page that is unclear or confusing? Return to homepage. [http://www.nhtsa.gov/]
Scenario #3 Identify Data on Recalls and/or Investigations Key Objectives Evaluate the ease of identifying the recall search tool on NHTSA.gov. Evaluate the ease of conducting a search on vehicle recalls. Evaluate the quality of content obtained through a vehicle recall search. Determine the likelihood of returning to NHTSA.gov for future information vehicle recalls. Where do you typically see, read or hear about vehicle recalls? Have you ever had a vehicle that was part of a recall? [YES] – What did you do for more information? [NO] – If you heard something about a recall for a vehicle you own, what would you do to find information? Task 1: Find general information on safety recalls The purpose of this task is to identify how easy or hard it is for users to find information about vehicle recalls. On the news, you heard about a recall on a vehicle. After hearing this, you decide you want to understand what these recalls mean and why a vehicle or its parts may be recalled. Using this site, find information on safety recalls. Tell me aloud the steps you take to find this information. Intended User Steps: Click Vehicle Safety [http://www.nhtsa.gov/Vehicle+Safety] Click Recalls & Defects [http://www.nhtsa.gov/Vehicle+Safety/Recalls+&+Defects] -or- Hover Vehicle Safety menu and click Recalls & Defects [http://www.nhtsa.gov/Vehicle+Safety/Recalls+&+Defects] Supporting Questions: How easy or hard was this to find? Does this page provide you with the information you need about this topic? Is this content easy or difficult to understand? Is there too much information, or too little information? Return to the homepage. [http://www.nhtsa.gov/] Task 2: Search for recall and potential investigation information for a specific vehicle The purpose of this task is to identify how easy or hard it is for users to search for a specific recall from the homepage. The recall you heard about was for a 2006 Honda CR-V. Someone you know drives this vehicle and you want to find specific information about the recall. Tell me aloud the steps you take to find this information. Intended User Steps: Click Search next to Latest Recalls in right menu [http://www-odi.nhtsa.dot.gov/recalls/recallsearch.cfm] Using the search tool enter 2006 Honda CR-V Click Retrieve Recalls [http://www-odi.nhtsa.dot.gov/cars/problems/recalls/results.cfm] Supporting Questions: How easy or hard was it to find information on recalls or potential investigations for this vehicle? Does the vehicle have an existing recall or potential recall investigation? What, if anything, would you change to make this tool more helpful for you? Is there any information missing that you would want to see when searching for recalls? You’ll notice you were brought to a different website to conduct this search. Overall, how does that impact how easy or hard this task was to complete? Return to the homepage. [http://www.nhtsa.gov/] Task 3: Search for new recalls The purpose of this task is to identify how easy or hard it is for users to search for new recalls from the homepage. Since you’re hearing all this information about recalls, you want to know what vehicles or parts have been recalled so far this month. Tell me aloud the steps that you take to find this. Intended User Steps: Click Recalls for Current Month link on homepage [http://www-odi.nhtsa.dot.gov/recalls/latestRecalls.cfm] Supporting Questions: How easy or hard was it for you to find information about recent recalls? What, if anything, would you change to make this easier? Do you believe this page is well-organized or cluttered? Is the content easy or difficult to read? Is there any other information or content you would have expected to see during this task? On a scale from 1 to 5 (5 being most likely) how likely would you be to return to NHTSA.gov for vehicle recall information in the future? Task 4: Evaluate Vehicle Safety menu The purpose of this task is to understand the relevance of the navigation menu Vehicle Safety. Before we move on, I’d like you to take a look at what else is available under Vehicle Safety. Mouse over the Vehicle Safety menu item. Supporting Questions: Do the items that fall under each make sense? Is there anything missing under the menu item, or do these items cover the information you would expect to find? Return to the homepage. [http://www.nhtsa.gov]
Scenario #4: Crash Avoidance Technologies Evaluate the ease of accessing the crash avoidance technologies information. Evaluate the value of the information provided on NHTSA.gov Determine the likelihood that a user will navigate to SaferCar.gov to obtain additional information on crash avoidance and crash protection technologies. Task 1: Identify Crash Avoidance Technologies You were recently talking with a friend who bought a new model year vehicle. They mentioned the vehicle has a great crash avoidance feature called Electronic Stability Control. Using this site, find more information on technologies like this. Tell me aloud the steps you take to find this. Intended User Steps: Click Research [http://www.nhtsa.gov/Research] Click Crash Avoidance [http://www.nhtsa.gov/Research/Crash+Avoidance] -or- Hover Research menu and click Crash Avoidance [http://www.nhtsa.gov/Research/Crash+Avoidance] -or- Click Laws and Regulations [http://www.nhtsa.gov/Laws-Regs] Click Electronic Stability Control (ESC) [http://www.nhtsa.gov/Laws+&+Regulations/Electronic+Stability+Control+(ESC)] -or- Hover Laws and Regulations menu and click Electronic Stability Control (ESC) [http://www.nhtsa.gov/Laws+&+Regulations/Electronic+Stability+Control+(ESC)] Supporting Questions: How easy or hard was it to identify where the crash avoidance and crash protection safety technologies live on the homepage? How likely would you be to click on it if you were looking for crash avoidance technologies? [NO] – Is there a way this could be done better? Clearer? How could the page be better organized in a way that makes it easier to find these technologies? Does this page provide you with the information you need about this topic? Did you notice if the page links back to SaferCar.gov? Where would you click for additional related info? What, if anything, would you change to make this page more useful for you? Task 2: Evaluate Research menu The purpose of this task is to understand the relevance of the navigation menu Research. Before we move on, I’d like you to take a look at what else is available under Research. Mouse over the Research menu item. Supporting Questions: Do the items that fall under each make sense? Is there anything missing under the menu item, or do these items cover the information you would expect to find? Return to the homepage. [http://www.nhtsa.gov]
Scenario #5 – Child Safety Key Objectives Evaluate the ease of navigating back and forth between Parents Central and NHTSA.gov. Determine if content is written in a way that consumers understand. Gather input on additional functionality, links or content that should be made available. Task 1: Find a Child Inspection Station The purpose of this task is to gauge how easy it is for a user to find a child seat inspection station. Your child just grew out of their child seat, so you bought a new one and you want to make sure you properly install it. Using this website, find a child car seat inspection station near your hometown. Intended User Steps: Click “Search” under Child Seat Inspection Station Locator in blue box on right of the homepage [http://www.nhtsa.gov/cps/cpsfitting/index.cfm] Enter state or zip code and click “Search on Map for Stations” [http://www.nhtsa.gov/cps/cpsfitting/ak/map/FindFitting.cfm?q_State=&q_Zip=] -or- Click Driving Safety [http://www.nhtsa.gov/Driving+Safety] Click Child Safety [http://www.nhtsa.gov/Safety/CPS] Click “Car Seat Inspection Station Near You” [http://www.nhtsa.gov/cps/cpsfitting/index.cfm] Enter state or zip code and click “Search on Map for Stations” [http://www.nhtsa.gov/cps/cpsfitting/ak/map/FindFitting.cfm?q_State=&q_Zip=] -or- Click Child Safety image on homepage [http://www.nhtsa.gov/Safety/CPS] Click “Car Seat Inspection Station Near You” [http://www.nhtsa.gov/cps/cpsfitting/index.cfm] Enter state or zip code and click “Search on Map for Stations” [http://www.nhtsa.gov/cps/cpsfitting/ak/map/FindFitting.cfm?q_State=&q_Zip=] Supporting Questions: How easy or hard was this to find? What, if anything, would you change to make this easier to find? Is the information provided easy to understand? Are there any additional search options that would make this easier for you? For what reasons did you choose to use (map/list) view? Return to the homepage. [http://www.nhtsa.gov] Task 2: Register your Child Safety Seat The purpose of this task is to gauge how easy it is for a user to register a child safety seat. Now that you’ve had the car seat inspected, you want to register the car seat. Tell me aloud the steps you take to do so. Intended User Steps: Click Child Safety [http://www.nhtsa.gov/Safety/CPS] Click Register Your Child Seat [http://www-odi.nhtsa.dot.gov/cars/problems/recalls/register/childseat/csregfrm.pdf] -or- Click “Register Your Child Seat to Receive Recall Notifications” link in blue box on right of the homepage [http://www-odi.nhtsa.dot.gov/cars/problems/recalls/register/childseat/index.cfm] Return to Child Safety page. [http://www.nhtsa.gov/Safety/CPS] Supporting Questions: How easy or hard was this to find? Does this page provide you with the information you need about this topic? Based on the information provided, how comfortable are you with registering a child safety seat? What information is necessary to register a child safety seat? On a scale of 1 – 5 (5 being the easiest), how easy or hard was it for you learn how to register a child safety seat? Using the same scale, if your child safety seat’s brand isn’t listed on the page, how easy or hard is it to register your child safety seat? What additional information, if anything, would make it easier for you to understand how to register a child safety seat? How do you feel about the level of content on this page? You may have noticed that you were taken to a different website—does this make it easier to use, harder to use or does it make no difference? Return to the homepage. [http://www.nhtsa.gov]

Debrief

• Overall, on a scale of 1 to 5 (1 is difficult, 5 is easy), how would you rate the ease of use of this website? What reasons do you have for providing this rating?

  • [IF NOT 5] In general, what, if anything, could be done to improve your rating?

• What would motivate you to visit this website?

• Have you visited other websites on this topic?

  • [IF YES] How is the information at NHTSA.gov similar or different from the information you’ve seen at other sources dedicated to the topic of vehicle safety?

• Is there any other information or tools we could add to the site to make it more useful for you? [MODERATOR PROBE FOR 2 KEY CHANGES]

• Do you have any questions?

Before we finish, I’d like you to fill out this sheet. For each measure, please circle the number that best corresponds with the rating you would provide NHTSA.gov based on what you saw today.

Based on what you saw today, please circle the rating you would give NHTSA.gov on each of the following measures.

ORGANIZATION	1 Cluttered	2	3	4	5 Well-organized
INFORMATION	1 Uninformative	2	3	4	5 Informative
READABILITY	1 Difficult to read	2	3	4	5 Easy to read
APPROACHABILITY	1 Intimidating	2	3	4	5 Inviting
RELEVANCE	1 Irrelevant to me	2	3	4	5 Relevant to me
ATTRACTIVENESS	1 Unattractive	2	3	4	5 Attractive
INTEREST	1 Uninteresting	2	3	4	5 Interesting
EASE OF USE	1 Difficult to find what I need	2	3	4	5 Easy to find what I need

NHTSA Form 1178

1