Case Study: Veteran Appointment Scheduling App
DISCLAIMER: In compliance with my Non-Disclosure, Intellectual Property, Copyright and Social Media agreements, the information provided in this case study is limited to material that has been previously published in public forums, as approved by my customer and/or my employer. While this case study represents my work and the work of my team, I am extremely limited in what I can share on this website. Graphics in this case study are intentionally obtained from public sources. If this case study interests you, please connect with me privately for additional discussion.
Summary
VA Online Scheduling (formerly known as the Veteran Appointment Request (VAR) App) allows Veterans who are in the Department of Veterans Affairs (VA) health care system to self-schedule and request primary care appointments.[1] Our team worked with the program office responsible for this app to improve usability throughout the design, development, piloting and phased deployment.
The Challenge
This app was created by combining the features of several previous standalone apps to enable patients to enter information that had formerly been handled by VHA scheduling staff. This provided patients greater flexibility in making appointments. However, the merging of features produced a product with a potentially fragmented user experience. Our team identified this concern as part of the initial usability evaluation required to deploy VA apps. We then engaged with the program office to evaluate the app and iteratively provide feedback required to improve the overall user experience and usability of the product throughout the rollout process.
The Approach
This project evolved to address the needs and concerns of the program office as the product was developed. Our approach consisted of meeting the program office where they were in the app development process, understanding their needs for evaluation and delivering on those needs to materially improve the user experience in concert with the development team’s cycles. Consequently, our team evaluated V2.1 of the product during the initial pilot, provided recommendations for improvement and then performed a follow-up evaluation of a revised V3.0 product.
The Solution
Veteran Appointment Request App on Smartphone [2]
The final V2.1 evaluation consisted of a diary study to collect information surrounding the process to obtain login credentials, formative remote unmoderated usability testing of core tasks and administration of the System Usability Scale (SUS) and follow-up moderated remote interviews to collect overall feedback and ask probing questions. Ranked usability findings were used to produce actionable recommendations for improvement.
Subsequently, the program office and development team issued a new version, VAR 3.0, which addressed critical and major findings. Our team then performed a second round of usability testing on the new version. Consistent with our approach to continuous learning, our team applied lessons learned from the initial evaluation and designed a study to assess the same core tasks while alleviating study constraints.
The follow-up V3.0 evaluation consisted of moderated in-person usability testing using mobile devices consistent with those owned by the participant but running a pre-tested configuration to ensure stability during the testing. Questionnaires and administration of the SUS were also done in person, moderated using a standardized questionnaire instrument. Usability findings were identified data analysis techniques similar to those used in the V2.1 study. Identical metrics were collected for comparison and actionable recommendations were provided to the development team for additional improvements.
The Impact
The average task success rate leaped from an average of 29% to 83% between the two versions.
Many of the measures collected for both the V2.1 study and the V3.0 study indicated notable improvement between the two versions. Participant satisfaction with the app, as expressed in the System Usability Score (SUS), rose from 79.3 for VAR 2.1 to 88.5 for VAR 3.0. Additionally, while 80% of participants said they would recommend VAR 2.1 to other Veterans, 100% of VAR 3.0 participants said they would recommend the app. Most notably, the average task success rate leaped from an average of 29% to 83% between the two versions. Additionally, by including three sequential scheduling tasks of the same type in VAR 3.0, our team was able to examine the impact of learning by the participants. The majority of participants completed all three scheduling tasks and 84% felt that the app got easier to use with practice. Following these series of studies, the development team and program office continued to stay engaged with our team for continuous improvement of the user experience. This product team now values timely user research insights to drive their design decisions and alleviate program office concerns.
Details of this Case Study
| When | 2015-2016 |
| My Role | Study Co-Lead |
| Our Team | Nancy Wilck, John Brown, Jane Robbins |
| Our Methods | Diary Study, Unmoderated Usability Testing, Remote Moderated Interviews, System Usability Scale (SUS), In Person Moderated Usability Testing, Interviewing |
| Our Tools* | Userzoom, Morae, SurveyMonkey, Mr. Tappy, Microsoft Office (Word, Excel, PowerPoint) |
* Methods and tools listed are limited to those employed by our team for user research. Other tools were used by the development team for other purposes, such as user interface design and code management.
Publications About this Case Study
Information in this case study was presented as a featured article in the VHA Human Factors Quarterly Newsletter (Winter 2017) (PDF).
Veteran Appointment Scheduling App Information
- Learn more about Veteran Online Scheduling at the VA App Store. If you are a Veteran receiving care at one of the pilot locations and you have a DS Logon Level 2 (Premium) Account credentials, download the app and try it for yourself!
- View a User Manual summarizing the basic features that were studied during this project.
Press Releases
- The American Legion: The VA Launches Scheduling App (Released: 5/4/2017)
- Eastern Oklahoma VA Healthcare System: Veterans Appointment Request App Now Available (Released: 2/27/2017)
- Atlanta VA Health System: Use the New Veteran Appointment Request (VAR) App to Simplify Scheduling Appointments with Your VA Care Team (Released: 5/4/2017)
Footnotes
[1] See the VA Online Scheduling User Manual provided by the VA for more details on the capabilities of this app.
[2] Image borrowed from The American Legion’s announcement of the VAR app deployment in May 2017.
