Virtual Reality Usability Study
January to March 2018 | 10 weeks
UX Research
Overview
For my Usability Studies course at the University of Washington, I was given the opportunity to partner with 360Stories to design and implement a research plan with a team of three students.
The research plan focuses on user research methods from screening surveys design, scenario-based VR explorations, and attitudinal questions. I collected both quantitative and qualitative data for analysis, and synthesized actionable, data-driven recommendations to our NYC-based stakeholder.
What is 360Stories?
360Stories is a 360 degree photo-tour application designed specifically for travelers. The app blends high quality 360 degree photographs with locally-sourced history and context to create an immersive tour experience in the virtual reality space, allowing users to explore and get a feel of their destination cities in advance.
Project Scope
Our stakeholder from 360Stories defined the following project scope:
Evaluate the user interface that supports experiencing engaging stories inside a virtual environment, before testing in-flight prototypes with real travelers at Air France and Royal Dutch Airlines.
Assess users’ interests of in-app purchases such as booking hotel accommodations and local tours within the curated virtual reality space.
My Role
As a UX Researcher and Project Manager on this project, I:
Established the goals and milestones with our stakeholder via consistent communication.
Formulated the screening questionnaires and VR scenario-based questions.
Recruited and scheduled participants for usability testing sessions, and led sessions in multiple roles of moderating, note-taking, photographing, and video-recording.
Synthesized data into findings, and presented deliverables to our stakeholder.
Collaborators
Executive stakeholder, UX researchers
Research Objectives
What are users’ goals when they use this application?
Do users find the app easy to achieve their goals?
What are the users’ attitudes toward making purchases in VR?
Our Target Users
Tech-savvy tourists and business travelers, especially those who frequent boutique hotels and travel by airplane.
People who are planning an upcoming trip and enjoy sharing their trip experiences.
The Challenges
“How do we test our participants in VR mode when we cannot see what they are seeing? How, then, do we follow them on their journey inside the VR space?”
Desired Target Audience vs. Reality: Our stakeholder defined a specific test demographic as males between the ages of 25-35, frequent international business travelers.
Testing Equipment & Technology Blindspot:
Our team’s VR equipment is not compatible with the prototype file provided by our stakeholder.
No access to a physical, VR-specific facility for testing sessions.
360Stories’ web version does not provide an immersive enough experience.
Solutions
Recruit a convenience sample of participants who meet baseline requirements of being tech-savvy and enjoys traveling, while meeting our project timeline.
Prime and introduce our participants in exploring the 360Stories’ desktop application before entering the VR space.
Employ the ‘Think Aloud’ technique to encourage participants in sharing what they see and how they feel in the VR space.
Usability Test Design & process
Scenario 1: Website Experience
This scenario is designed to introduce our participants to the 360Stories’ website, using several existing features to navigate around Rijks Museum in Amsterdam.
Key tasks include:
Navigate using Scrolling View
Navigate using Filters & Search
Navigate using Map
Scenario 2: VR Experience
This scenario is designed to familiarize our participants with 360Stories’ VR space.
With Google Glass, participants are virtually immersed inside the Rijks Museum. After exploration, participants are tasked to find where a mystery object is displayed in the room.
Methodology & Findings
A total of six usability testing sessions were conducted, each lasting about 45 minutes, with an equal mix of both male and female participants.
Both quantitative and qualitative data data were captured during sessions, with an emphasis on observations such as participant statements and feelings toward making purchases in VR.
Our team assessed priorities of impact using the following metrics:
Impact = Frequency x Severity
Frequency = Number of participants who experienced the issue
Severity scored as follows:
4 = Critical, prevents completion of the task
3 = Major, difficulty in completing the task
2 = Minor, inconvenience in completing the task
1 = Cosmetic or opportunity for improvement
High Priority Finding 1: Default Map View
All six participants weren’t able to find the easternmost place in the city because the default map view doesn’t show all available places.
High Priority Finding 2: Fatal Error
On the web version, three participants experienced a Fatal Error after using the back button repeatedly to return to Homepage.
High Priority Finding 3: Navigation Tools
In VR mode, three participants found navigation more difficult because the hover-over text is no longer available.
Recommendation:
Consider modifying the default zoom view to include all available places within the selected city.
Recommendation:
Consider periodic back-end de-bugging and website maintenance.
Recommendation:
Consider keeping consistency with navigation tools in website and VR.
Would participants make purchases in VR?
YES
“I’d be willing to make a purchase directly from inside the VR app; it needs serious improvements.” — P2
“Yes, as long as I could put my payment information in once; I’d like everything to be contained inside the app, like Groupon.” — P3
NO
“I would be interested, but won’t be able to do it. I would do it the traditional way.” — P1
“Making purchases would be a good side feature, but I wouldn’t make a purchase in VR.” — P4
“I’d be more comfortable to make purchases on the web.” — P5
“I’d go straight to the hotel website to find deals.” — P6
How might we encourage more purchases in VR?
Overall, our participants see potentials in making purchases in the virtual reality space. I speculate that an advanced VR equipment and purchasing mechanism embedded in the app may increase participants’ comfort level for making purchases in the virtual reality space.
Reflections on this project
In retrospect, I especially learned that,
Use constraints to guide and refine the usability test design. Despite perceived constraints preventing us to evaluate how participants truly feel in the VR space, they did not limit our research approach but rather led to new ways to paint a vivid picture of our participants’ experience using not just one scenario, but two.
Quantitative data doesn’t always necessarily speak the truth. Although the results we gathered show that participants preferred to make purchases in the web than the VR experience, much due to VR’s infancy in the online purchasing footprint; however, from the qualitative perspective, when we dug into participants’ emotional cues, the overall reaction from participants is that participants are excited about VR as a new platform for the online realm, and that they were overwhelmingly positive about the potential that VR brings.