The Problem
Introducing design validation
Throughout the history of Avenio Connect’s development, there was a lack of formal UX presence - several designers briefly consulted on the project, but there was an absence of a dedicated design lead. This meant that much of Connect’s decisions were formed under assumptions from stakeholders, such as how lab technicians would interact with the product. Because of this, there were many foundational questions left unanswered. How do we know our product is solving actual users’ needs if we don’t perform formal user testing? How can we know what experiences we need to improve upon? How do we know with confidence that our hypothesis/assumptions are true?
Design goals
Conduct and end-to-end observation study to provide a benchmark of Avenio Connect’s usability. This allows us to understand the overall experience of our lab technicians, provide a base value for future studies to measure against, as well as make recommendations in future design iterations and studies
Business goals
Both the FDA and IEC 62366-1 require formative and summative usability evaluations to ensure medical devices are effective and safe. This study is required for FDA submission.
Study Overview
Study logistics
Test setup
Design: Formative usability evaluation
Evaluation scope: End-to-end Avenio Connect flow
Materials: Avenio Connect, Avenio Edge, third party instruments
Location: Santa Clara and Pleasanton Roche sequencing labs
Date: May 30 – June 29, 2023
Depending variables: use errors, usability issues, user quotations, system usability scale
Participants
Number of participants: 8
Gender: 5 female, 3 male
Working experience: Avg 10 years
Familiarity with Capstone IVD (Roche assay):
6 expert, 2 comfortable
Familiarity with Avenio Connect software: 1 expert,
6 comfortable, 1 novice
Work areas: 8 research and development
Lab setup
All participants are required to wear lab coat, goggles, mask, latex gloves, and shoe covers.
Several participants opted to also use plastic laptop covers to avoid having to wipe down their computers with sanitizing wipes once exiting the lab.
Multiple steps require pipetting samples into wells or tubes while using their laptops to calculate and enter data or read instructions.
Contextual inquiry session breakdown
Each session was segmented into five sections: Intro, observation of sequencing steps, post-session surveys, System Usability Scale (SUS) survey, and wrap-up.
1. The intro explained the purpose of conducting a formative study according to FDA guidelines.
2. Within the observation, we focused on moments where participants struggle to complete a task, areas in which participants completed tasks unconventionally, as well as tasks that participants completed with ease.
3. For the post-session surveys and SUS survey, we asked participants to provide feedback on the ease of use of each step throughout the sequencing process.
4 & 5. Finally, our wrap-up allowed us to reflect on the session and discuss things that we may not have covered.
Each session lasted for 3-4 days and was recorded and annotated.
How did this affect strategy and culture
Throughout the history of Avenio Connect’s development, there was a lack of formal UX presence - several designers briefly consulted on the project, but there was an absence of a dedicated design lead. This meant that much of Connect’s decisions were formed under assumptions from stakeholders, such as how lab technicians would interact with the product. Because of this, there were many foundational questions left unanswered. How do we know our product is solving actual users’ needs if we don’t perform formal user testing? How can we know what experiences we need to improve upon? How do we know with confidence that our hypothesis/assumptions are true?
Evangelize UX research
Generated excitement amongst other product teams to also perform their own comprehensive usability studies
Encourage iterative design process
The amount of tickets (45) filed as a result of the study created a sense of urgency and encouraged stakeholders to have iterative research often throughout the development cycle.
Data driven design decisions
This study provided a benchmark of not only the overall user experience of Avenio Connect, but the user experience at every individual step of the NGS process. This allows the team to make informed design decisions moving forward and eliminates much of the assumption regarding our users’ mental models.
Analysis
The findings
All notes taken throughout each session were hand-coded to turn qualitative data into quantitative data. Each session was transcribed and separated into issues, findings, and comments. Those entries were later given severity levels to help determine issue priority. Through the issues categorized as catastrophic, we discovered that some of the major assumptions made earlier in the product development did not align with our users’ mental model.
Primary problem
Connect enforced a rigid methodology for genomic sequencing, forcing all samples to move through each step at the same time. Study participants perceived Connect as excessively limiting due to abundant safety measures, thereby constraining their sequencing processes. Every phase of the sequencing flow had to comply with Connect's stringent timetable and methods, ignoring potential unforeseen lab circumstances. This inflexibility compelled users to make premature decisions, often without full foresight. Consequently, lab technicians risked data accuracy or significant financial loss on samples and reagents due to these predetermined directives.
See how we iterated on our design to solve this issue:
Other findings
Understand the environment you’re designing for
It’s difficult to interact with a UI (that was designed for a monitor) on a laptop when users are constantly moving throughout the lab wearing latex gloves.
Data entry is repetitive and tedious
When each sample order contains at least 24 samples, there is too much time spent performing administration work when users could be continuing their lab steps instead.
Guidance prevents mistakes
“I wish there were instructions on what to do next.” Users were often confused how to proceed to the next step of the lab sequence.
Errors are costly. How do we help users fix them?
Mistakes are hard to address and errors are costly and time consuming. Also, areas that allow manual user inputs tend to also not allow users to edit to fix mistakes