Star Wars: Jedi Challenges is a smartphone-powered augmented reality experience by Disney and Lenovo. The product hardware and gameplay both were originally intended to be developed in-house as the next phase of Disney Playmation. During this R&D phase, I led fit testing and playtesting efforts, explored the competitive Star Wars game space, and worked on game design, control schemes, and AR menu design.


Star Wars Competitive Analysis

As part of our iterative R&D efforts, I researched and analyzed the Star Wars digital landscape, everything from browser games to Lego Star Wars to VR experiences. This accomplished the following:

  • Provided an overview of what games and experiences were already out there as a reference for our teams.

  • Identified key commonalities and design patterns that could be leveraged while iterating on ideas for future phases of development

  • Unearthed gaps in the current landscape that we could potentially turn into opportunities.

^ Back to top


Early on, we had planned for the product to include a wristband which would enable the player to carry out gesture-based moves. During fit tests, I would walk kids through trying on a few versions of a wristband. I was looking for comfort, quality of fit, and ease of use in putting it on and taking it off. I quickly realized that you can’t just ask a child which was their favorite, because the answer you’ll usually get is “all of them.” Instead, I found that asking the kids to rank the options from best to worst led to more critical thinking on their part and more honest, helpful answers for our team.

I also conducted a series of internal tests for our team to gain a better understanding of the needs around IPD (interpupillary distance) calibration. Measuring the distance between the wearer’s pupils is what head-mounted VR/AR displays use to position camera output clearly, reducing eye strain. Seeing as a user may not be able to see anything when they first don the headset, I wanted to make sure we were able to lead a user through the calibration process without visual instruction.

Back to top

Augmented reality UI & Control Schemes

Headset Proposal

  • Considerations and requirements:

    • Must be able to function without any peripherals connected

    • Minimize how much the player needs to use it, since overuse could lead to risk of (a) arm fatigue and (b) knocking tracking out of alignment

    • Needs to be usable with a saber in one hand

    • Pause and IPD calibration are critical functions that must be easy to access (and must be able to access IPD calibration even if visuals are misaligned)

    • May need physical volume controls depending on how audio is implemented

  • Headset will act as a console, used to access system controls and settings > separation of system menu and game menu, as in any modern console

  • Will mostly be used at start/end of gameplay session. Mid-play usage should primarily be only if something has gone wrong. This will minimize necessary usage.

  • Having the buttons located on top may mean the user is more likely to brace the entire headset with their thumb, which could lead to a higher likelihood of stability.

    • However, if the player uses (for example) their left hand to press the right button, that could result in increased instability.

    • One possible solution is to have buttons that curve along the edges so the player can press on the bottom or side.

  • A wheel or similar bidirectional control (e.g. touchpad) with two directions enabled (i.e. back/forward or left/right) will let user scroll through options or calibrate IPD with one hand

    • Risk: depending on how we implement IPD calibration, this may not be as intuitive as left and right flippers.

  • With L1 as both open menu and select, if IPD calibration is the first option in the menu, then the user can just double-click L1 to go directly to calibration

  • If the more commonly used controls are all on one side (with R1 being only back/cancel) the player will not have to reach across themselves/switch hands for saber as often


Control Scheme

STATE: in-game, no menus open

L1: opens system menu (also pauses gameplay)
R1: TBD (gameplay-specific?)
L2: wheel or toggle to adjust volume

STATE: system menu is open

L1: Select/Confirm
R1: Back/Cancel
L2: Scroll (and left/right controls for IPD calibration)


System menu wireframe (v1)

System menu wireframe (v1)

System Menu Hierarchy

  • IPD Calibration

  • Visual Settings

    • Brightness

    • Tint

  • Audio Settings

    • Overall volume

    • Volume of music/dialogue/etc.

  • Devices

    • Set up connections (saber, marker)

    • Battery levels (phone, saber)

    • Do not disturb toggle (phone)

  • Help

    • Tutorials

    • Tips

  • Session

    • Timer (tracks how long you’ve been playing, could prompt user to take a break for eye strain)

    • Change user profile

    • Quit

System menu wireframe (v2)

System menu wireframe (v2)

System Menu + HEADSET Use Cases

  • New user onboarding:

    • Assume peripherals aren’t set up yet

    • Start with IPD calibration

    • Teach gaze selection

    • Marker and saber setup

    • Saber/in-game tutorials

  • Player is starting to feel eyestrain/visuals are looking blurry while playing: double-click L1 on headset to go directly to IPD calibration

  • Saber stops working: headset menu > saber settings to troubleshoot connection

  • Music is too loud and player can’t hear dialogue: headset menu > audio settings

  • Parents are yelling at player to turn it down right now: volume wheel (or audio settings)

  • Sibling’s turn to play: headset menu > change user profile

  • Friend comes over and wants to try it before they get their own:

    • If player 1 doesn’t want friend to affect their progress: headset menu > change user profile > guest

    • If friend has played before and knows how it works: headset menu > IPD

    • If friend has never played before: headset menu > help > reset FTUE


  • Considerations and requirements

    • Must not break with the look of the actual lightsaber (Lucas requirement)

    • Controls must be accessible without too much maneuvering (ideally with one hand), while still not easy to hit accidentally during combat

  • Proposal 1: Saber as Motion Controller

    • Saber will act as a controller specifically for gameplay (all system controls in headset)

    • Gives player six degrees of motion, more intuitive for interaction with AR elements

    • Current concept has a saber activate/deactivate button, as well as a grip squeeze control (possibly for use with the Force).

    • Optimally, most controls in this concept would be displayed visually in game (such as tapping on AR elements with your saber for selection, physically moving/walking, etc.)

  • Proposal 2: Saber with Physical UI (see mockups)

    • Saber will act as a controller specifically for the gameplay

    • Physical controls can be broadly applicable to different types of gameplay (since we don’t have that defined yet)

    • Does not have to be consistent for future Playmation franchises – can be specific to Star Wars


STATE: in-game, no menus open
A: open in-game menu (also pauses gameplay)
B: TBD (could be for activating force powers)
C: select/confirm
D: activate saber (toggle or hold)

STATE: system menu open
A: TBD (game design needed)
B: Back/Cancel
C: Select/Confirm
D: TBD (game design needed)