SAAXR (Magic Leap, 2019)

 Project SAAXR: an AR DJ platform at the AT&T Shape Hackathon

The concept

A prototype for an AR-enabled music distribution and remixing platform, creating new avenues for real-time collaboration as well as new ways for artists and brands to be in conversation with their fans.


Platform

Working on a Magic Leap was a joy. The SDK and developer tools are polished, clear, and insightfully designed--plus, one doesn’t have to remove the headset to locate one’s coffee.

The mechanics

Each song arrives as a capsule that contains a miniature interactive world as well as a number of collectable “stems”—the individual audio tracks that make up a song. Users are able to find these capsules at special events, pop-up shops, or purchase them directly from the service. They are thereby able to modulate individual properties of the tracks, exploring the interplay between them in the original song, as well as add individual stems into their personal collections to create entirely new tracks with.

Design challenges

Because the initial project brief was much larger than a team of four could tackle in a weekend, our first challenge was deciding on a scope that would showcase the important elements of experience and leave us with a clear path forward for continued development. I was able to bring my experience as both a developer and designer to accurately budget for the features we could achieve in the timeframe, and to refine the XR UX elements to be more effective in an immersive space. My main design contributions to the project were a prototype of an interaction system, and reworking the initial concept to feel native in space.

Creating a clear visual vocabulary

An app must be understood to be used, and one of the principal tasks of XR design is making the user worry as little about the interface as possible. To this end, I reworked the initial design of similarly shaped objects to a both more unified and more diverse schema. Initially, the containers for the tracks and the container for the user’s own collection were to be uniform for the sake of visual consistency. I identified that this might cause some users confusion, and might prevent them from attaching a sense of ownership to their collections. Seeing this, we were able to adapt an existing briefcase asset into a mixing board / storage case, and come up with a design for the song capsules that allows for complete customization by the artist.

Crafting a robust interaction system

One of the goals of the project was to foster a conversation between artist and fan, and what better way to do this than giving the fans some access to the artist’s creative process. We wanted fans to be able to take individual song stems and tweak them, arrange them diegetically in the scene, apply filters and other modulations. We also discovered new opportunities for storytelling: by letting listeners interact with the visual representation of the song, artists are able to communicate visual narratives in any style or theme. Our prototype demonstrated this with a small town that grows from the song’s container to fill up the room; within the town are dozens of interactable elements that animate as the controller sweeps over them, and others that react as realtime audio visualizers.


Technical challenges

A (very) Work-in-Progress video.

Perhaps the biggest hurdle we faced was a lack of FMOD support in Lumin. Coordinating a dozen 3D audio sources on mobile hardware, especially with the complexity of interaction we included, while lacking the syncing and event interface of FMOD took a lot of work. We considered using Wwise but between me handling all the programming and app creation, and the junior Unity dev handling the animation and asset management, we didn’t have time to read up on it.

My takeaways

  • Developing for the Magic Leap gave me a new appreciation of what the platform can be used for--its comfort factor and the beauty of its interface, even after dozens of hours of on-and-off use, was a real taste of what AR integration into everyday life will be like.

  • Much brand content of today is in reference to experience--pictures of things that have happened; physical items that reference, but are not themselves, art. As XR permeates life, I expect to see a lot more brand content that is itself an experience.

  • One of the force multipliers of AR adoption will be a boom in persistent content. There is nothing like looking at a magical apparition that somebody else actually sees.