Star Wars: Jedi Challenges is a smartphone-powered augmented reality experience by Disney and Lenovo. The product hardware and gameplay both were originally intended to be developed in-house as the next phase of Disney Playmation. During this R&D phase, I led fit testing and playtesting efforts, explored the competitive Star Wars game space, and worked on game design, control schemes, and AR menu design.
Star Wars Competitive Analysis
As part of our iterative R&D efforts, I researched and analyzed the Star Wars digital landscape, everything from browser games to Lego Star Wars to VR experiences. This accomplished the following:
Provided an overview of what games and experiences were already out there as a reference for our teams.
Identified key commonalities and design patterns that could be leveraged while iterating on ideas for future phases of development
Unearthed gaps in the current landscape that we could potentially turn into opportunities.
Early on, we had planned for the product to include a wristband which would enable the player to carry out gesture-based moves. During fit tests, I would walk kids through trying on a few versions of a wristband. I was looking for comfort, quality of fit, and ease of use in putting it on and taking it off. I quickly realized that you can’t just ask a child which was their favorite, because the answer you’ll usually get is “all of them.” Instead, I found that asking the kids to rank the options from best to worst led to more critical thinking on their part and more honest, helpful answers for our team.
I also conducted a series of internal tests for our team to gain a better understanding of the needs around IPD (interpupillary distance) calibration. Measuring the distance between the wearer’s pupils is what head-mounted VR/AR displays use to position camera output clearly, reducing eye strain. Seeing as a user may not be able to see anything when they first don the headset, I wanted to make sure we were able to lead a user through the calibration process without visual instruction.
Augmented reality UI & Control Schemes
Considerations and requirements:
Must be able to function without any peripherals connected
Minimize how much the player needs to use it, since overuse could lead to risk of (a) arm fatigue and (b) knocking tracking out of alignment
Needs to be usable with a saber in one hand
Pause and IPD calibration are critical functions that must be easy to access (and must be able to access IPD calibration even if visuals are misaligned)
May need physical volume controls depending on how audio is implemented
Headset will act as a console, used to access system controls and settings > separation of system menu and game menu, as in any modern console
Will mostly be used at start/end of gameplay session. Mid-play usage should primarily be only if something has gone wrong. This will minimize necessary usage.
Having the buttons located on top may mean the user is more likely to brace the entire headset with their thumb, which could lead to a higher likelihood of stability.
However, if the player uses (for example) their left hand to press the right button, that could result in increased instability.
One possible solution is to have buttons that curve along the edges so the player can press on the bottom or side.
A wheel or similar bidirectional control (e.g. touchpad) with two directions enabled (i.e. back/forward or left/right) will let user scroll through options or calibrate IPD with one hand
Risk: depending on how we implement IPD calibration, this may not be as intuitive as left and right flippers.
With L1 as both open menu and select, if IPD calibration is the first option in the menu, then the user can just double-click L1 to go directly to calibration
If the more commonly used controls are all on one side (with R1 being only back/cancel) the player will not have to reach across themselves/switch hands for saber as often
STATE: in-game, no menus open
L1: opens system menu (also pauses gameplay)
R1: TBD (gameplay-specific?)
L2: wheel or toggle to adjust volume
STATE: system menu is open
L2: Scroll (and left/right controls for IPD calibration)
System Menu INFORMATION ARCHITECTURE AND DESIGN
System Menu Hierarchy
Volume of music/dialogue/etc.
Set up connections (saber, marker)
Battery levels (phone, saber)
Do not disturb toggle (phone)
Timer (tracks how long you’ve been playing, could prompt user to take a break for eye strain)
Change user profile
System Menu + HEADSET Use Cases
New user onboarding:
Assume peripherals aren’t set up yet
Start with IPD calibration
Teach gaze selection
Marker and saber setup
Player is starting to feel eyestrain/visuals are looking blurry while playing: double-click L1 on headset to go directly to IPD calibration
Saber stops working: headset menu > saber settings to troubleshoot connection
Music is too loud and player can’t hear dialogue: headset menu > audio settings
Parents are yelling at player to turn it down right now: volume wheel (or audio settings)
Sibling’s turn to play: headset menu > change user profile
Friend comes over and wants to try it before they get their own:
If player 1 doesn’t want friend to affect their progress: headset menu > change user profile > guest
If friend has played before and knows how it works: headset menu > IPD
If friend has never played before: headset menu > help > reset FTUE
AUGMENTED REALITY LIGHTSABER PROPOSAL
Considerations and requirements
Must not break with the look of the actual lightsaber (Lucas requirement)
Controls must be accessible without too much maneuvering (ideally with one hand), while still not easy to hit accidentally during combat
Proposal 1: Saber as Motion Controller
Saber will act as a controller specifically for gameplay (all system controls in headset)
Gives player six degrees of motion, more intuitive for interaction with AR elements
Current concept has a saber activate/deactivate button, as well as a grip squeeze control (possibly for use with the Force).
Optimally, most controls in this concept would be displayed visually in game (such as tapping on AR elements with your saber for selection, physically moving/walking, etc.)
Proposal 2: Saber with Physical UI (see mockups)
Saber will act as a controller specifically for the gameplay
Physical controls can be broadly applicable to different types of gameplay (since we don’t have that defined yet)
Does not have to be consistent for future Playmation franchises – can be specific to Star Wars
STATE: in-game, no menus open
A: open in-game menu (also pauses gameplay)
B: TBD (could be for activating force powers)
D: activate saber (toggle or hold)
STATE: system menu open
A: TBD (game design needed)
D: TBD (game design needed)