Introduction

When someone listens to music, they want to escape, to inspire, and to engage. Music as an art form and cultural activity has been made and enjoyed by people in many ways since its creation. The type of music experience we are familiar with today is music streaming services such as Spotify, iTunes, and Tidal. As mobile or desktop/laptop based music player applications, these music streaming services have replaced traditional physical media based listening experiences such as CDs, records, and tapes. With the shift of media, music playback devices such as the radio, tape deck, reel-to-reel, and turntable, have been dematerialized for today’s most common setup: a smartphone with earbuds. Working with my project team, we aim to create a new modern music experience from the ground up, and subsequently, ideate what that music experience would be in a VR/AR/mixed reality society. We ask ourselves, “how can new forms of interaction such as voice, gesture, AR and VR, AI/machine learning, the design of physical objects and interactive environments be incorporated into the design of a next generation music experience?”

The Challenge

Market

While the music market is saturated, it has always been growing, evolving and changing since its creation. Existing as a $47 billion industry today, music is as relevant as ever and it appears to be a market that will never dry up.

Research

Compared to my previous projects, the research for this design is done uniquely. Normally, I conduct research to better identify and pinpoint problems within a problem, however, this project involves conducting research to help my team and I ideate solutions and new experiences. Rather than conducting in-depth analysis on problems, we will be observing how people use music in their everyday lives. Throughout the research, we kept these three questions top of mind:

Sticky Notes

To begin our research, we reviewed music in its entirety from the top and looked downwards in order to have a very broad view of the issues and solutions. On the contrary, if we started from the from the bottom and looked upwards, we would have a fairly narrow view of what we could see. Having a top to bottom view provides us with hundreds of decisions at different crossroads to get to a bottom. By going from abstract to concrete rather than from concrete to abstract, we are able to look at music from outside the box allowing us to work towards innovation. To apply this concept of Abstraction Hierarchy, we examined music by posing three abstract questions:

What? - In what forms can it heard?
How? - How does it work?
Why? - What is the purpose of music?

Picture Storyboard

Next, each member of the team examined a present or past experience of listening to music. We documented those experiences as a photo-based storyboard that captured place actors and interactions. We wanted to observe people enjoying music in everyday life hoping to provide insights into what their listening experience could be. Examples that teammates and I observed were people listening to shows on the radio, listening to records, watching someone making music, hearing a band play, observing someone compiling a mix-tape for a friend, and watching a peer commute to the supermarket. One of the photo-based storyboards of an experience is depicted below:

Storyboards

Upon completing our brief research which guided our ideation, we made six to eight frame storyboards of how we envision what the future of the music listening experience might be like:

Music Discovery

After our research, ideation, and observation, we decided to pursue interactive music discovery as the future of the music listening experience.

Brainstorm

With a solution in mind, we brainstormed further:

Solution

The outcome of our work is represented as Noted, a way for people to discover new music while exploring new places.

Noted is a map-based app that leads users to destinations, hole-in-the-walls, and experiences they would not be aware of without access to the app. Noted does this by leading users to a song at specific destinations.

Additionally, Noted allows users to drop the song they are listening to at their current location creating a Notesphere. Users drop songs in order to share their experience with other users who might visit the same spot at a future time, whether it be in a month or in a year.

While there are many other features we ideated in the brainstorming phase and intended to include in the app, the two features discussed are the core of Noted. For a clearer understanding of Noted, we describe specific scenarios of how the app can be used in the section below. Furthermore, towards the end of this write-up, there is a video demonstrating Noted in action.

Scenarios & Magical Moments

In this section, I write about four magical moments Noted could provide users:

Kevin reaches the peak of his hike and opens Noted. He notices there is a Notesphere right around the corner and decides to walk to it. He reaches a beautiful view he did not know was there and plays the song left by another user a year ago. Standing there, looking at the view, and listening to the song, Kevin is experiencing the exact same nice moment that the other user experienced a year ago.
Sadie is in France for the first time when she decides to use Noted. Noted shows her around the foreign city and takes her to spots she otherwise would not have visited. Additionally, as she travels through the city, she listens to the popular songs the locals have dropped around, helping Sadie to get more immersed in French culture.
Jon’s father has recently passed away. He checks on his late father’s Noted account to see all the songs he has dropped around the world. Jon plans a trip to travel to all of his father’s dropped songs in order to experience similar experiences that his father did.
The year is 2035 and Rachel sets her Noted app to 2018. Rachel walks around and listens to all of the songs that were dropped in 2018 in order to get a taste of what music was like back then.

Interaction Model

Presented below is an interaction model that maps out our magical moment #4:

Sketches

Sketches of our early ideas for the application:

Early Concept Video

Early concept of our ideated music experience app, "Noted:"

Early Iteration

After testing, discussing and further ideating, we created wireframes and early iterations of our mobile application:

Style Guide

Final Product

After a lot of hard work and careful consideration the final product, screens and video user story is born:

Home Page

3D View & Add Song

Notifications & Music Player

User Flow

Video Experience User Story

To properly explain Noted and the importance of sharing not only music but also experiences, we created a video user story and prototype:

Mixed Reality

Since our project began with us attempting to ideate the future of music listening and music experiences, we wanted to take Noted and our project one step further. Our next step to accomplish this goal was to imagine and design Noted for AR/VR and mixed reality. We develop a revised interaction model of Noted that incorporates AR/VR interactions and create a new experience walkthrough that showcases the potential integration of AR/VR interaction elements into Noted. That resulted in:

It would take a lot for Noted VR to be practical right now, but with the proper scalability of technology, it will become a possibility.

Noted VR Video Experience

We return to the field to make another video to show the potential of Noted VR:

Takeaways & Reflection

I have taken away a lot from this project because of the unique way my team and I went about conducting it. As I stated above, in all of my previous projects, I began with a problem in mind and conducted research to pinpoint the not so obvious roots of that problem. Although, in this project, I started with a modern experience, and engaged in research aiming to discover the future of that experience. This process led to a unique research, ideation and design thinking experience for my entire team. I am looking forward to participating in more projects where I can design the future of interactions rather to simply solve a problem (which I still love too)!

Second, this was my first time designing a product for VR and mixed realities. I believe that the concept of user experience will move beyond just tech, such as websites and applications, and migrate into other fields which lack it. With new interaction paradigms such as AR, AI, and improved wearables beginning to take root, how people interact with their environments and our products will completely change, allowing designers to create connections in ways we never could before. As our world grows more comfortable with AR and AI, there will be a blurring of the line between online and offline experience for the user. My long term goal as a designer is to integrating user experience into an offline space to create deeply empathetic experiences that people from all backgrounds can utilize. Through this project, I feel I have taken the first steps toward this goal, strengthening my desire to marry design and mixed reality.

In a future exploration of this project, I would like to further develop Noted using more of the ideas we ideated in the brainstorming phase such as “routes” and playlists. Lastly, I want to give credit to my three teammates for assisting with this project. I thoroughly enjoyed working with them because they challenged me to be a better designer. Being able to brainstorm, develop, and grow ideas with them took the project to new heights. Overall, I learned a great deal working with my team, and am grateful for their dedication and hard work.