Future Of Visual Design: How Designers Can Evolve Their Skills For AR, VR & MR

In the next decade, the world of design will undergo a massive revolution. Virtual reality (VR), augmented reality (AR), and mixed reality (MR) will fundamentally transform the work that many designers do.

What will you need to succeed in this new world?

In this article, we’ll give you a sneak preview of where VR, AR, and MR are headed, the skills you’ll need, and how to get started on your new journey.

The Future of VR, AR, and MR

What Are VR, AR, and MR?

When you hear the term VR, think of the Matrix — or at least for the foreseeable future, a lower resolution, less immersive version of it.

With VR, designers place users in a virtual environment where users interact with solely within that environment.

In contrast, with AR, users interact with virtual objects inside the real world.

MR allows users to interact with virtual objects either in the virtual world or in the real world.

How do you collectively refer to VR, AR, and MR? Usually they’re known as extended reality or XR.

How XR Will Transform Society

In the coming years, VR, AR, and MR will have a deep and profound impact on society.

This fall, for example, the Cleveland Clinic Lerner College of Medicine is going to switch to teaching anatomy using MR.

And thousands of public school classrooms have begun experimenting with teaching physics, chemistry, history, and other topics using VR or AR.

Entertainment will also be transformed by XR.

It’s not surprising that video game makers have been at the forefront of exploring how XR can make games more immersive. What they’ve accomplished so far is just the tip of the iceberg.

Even an area as mundane as meetings will go through a profound change.

Today it’s possible to work remotely with someone using Zoom. But it’s a little clunky, especially if you have more than a handful of participants.

XR will allow people to collaborate across geography as if they were in the same room.

Not only will this transform how corporations and conferences work, but it will also allow community activists and local officials to collaborate on, say, stopping climate change in a way that would be almost unthinkable today.

XR Hardware 

One of the challenges of XR design is at the hardware and software are a moving target.

Currently, unless you can afford to spend big bucks, XR headsets are pretty limited in what they can do and how immersive they are.

But in the next few years, that’s going to change.

For example, today one of the most powerful widely used headsets is Microsoft’s Hololens 2. 

Microsoft is betting heavily on using Hololens to transform the corporate world, and they are closely working with an impressive roster of companies across all industries.

Today a Hololens will set you back $3500. Microsoft is racing to cut production costs to get it to a consumer price point.

Apple also seems to be betting heavily on making a headset.

And other giants like Facebook are frantically working to transform existing consumer headset into a powerhouse.

As soon as one of these fierce competitors figures out how to manufacture a untethered headset that’s as powerful as Hololens for less than $400, it’ll set off an XR goldrush.

XR Software

There’s also fierce competition around XR software strategies.

Today, there are essentially two ways to create an XR experience.

You can use a platform like Unity to create an app.

Or you can use an open source library such as A-Frame or Babylon.js to create a WebXR experience that can be used by most headsets and anyone with a modern Firefox or Chrome browser.

We are also beginning to see custom apps for building XR as well as efforts by companies like Amazon to create their own platform for XR experiences.

At this point, it’s anybody’s guess as to which approach or approaches will win out. 

For example, it used to be that if you wanted to create a powerful immersive experience, you had to create an app. But WebXR has been catching up fast, and what you can do in the browser these days is truly impressive.

What Skills Will You Need?

What skills will you need to flourish in the coming XR era? 

Here’s a list of the areas where you’ll need to level up.

3D

Depending on kind of design work you currently do, you may not have much experience working in 3D.

And unless you’ve built games, you probably don’t have experience working with 3D objects and scenes that a user can experience from multiple perspectives.

With XR, that’s going to change. To be a skilled XR designer, you’ll have to be fluent at creating 3D assets.

Designing for 3D immersive environments brings lots of new challenges. 

Let’s say you’re designing a VR postcard.

Adding 3D text is pretty easy. But it’ll take a little practice before you can be confident it’ll look good to users who see it from a variety of angles.

Becoming skilled at creating effective 3D assets such as text will be particularly challenging in the next two years; we don’t yet have a large body of best practices for 3D are tools that support these practices.

The good news is that mastering 3D assets is a twofer. Creating 3D assets for XR will help you today, and down the road it’ll be useful in another field that’ll take off in the next decade: digital fabrication.

Animation

If the 3D scenes you create are going to be truly immersive, most of them will need motion.

You’ll want to design doors that open and close, characters that can walk or sit, trees that move in the breeze.

Luckily, every year there are more free and affordable tools that can substantially reduce the amount of work it takes to animate the 3D assets you create.

And the number of free and affordable 3D assets you can find online that come pre-animated is also growing.

So over time, you can expect to have access to a substantial number of resources that make it easier to do increasingly sophisticated animation, freeing you up to either more quickly build your creations or push the boundaries of what you can do with animation.

Sound

The next time you walk around at the place you work or in your neighborhood, take a moment and really pay attention to what you’re hearing.

Or turn on your favorite 3D shooter and listen for how they use sound, from the noise you make when you run to the sound of firing your favorite weapon to the background music.

We don’t normally notice, but sound is a critical part of how we experience the world.

Sound gives us feedback on what we are doing. It also helps us locate ourselves in space as we move.

If you’re designing an augmented reality a coffee cup, you don’t have to worry about sound. But you will if you want the VR representation of a coffee shop to feel immersive.

As XR takes off, most of the time designers won’t have to design soundscapes from scratch. But sound will be another critical piece of the design puzzle.

Controllers

When a designer is creating elements of a website or an app that a user can control, the rules of the road are pretty settled. But with XR, it’s the Wild West.

As of now, there isn’t a standard across most platforms for how users can manipulate an XR environment.

Most platforms have at least some suggested rules. But designers have wide latitude.

To make design even more complex, the physical options a user has can vary widely from platform to platform.

We are beginning to see some norms come into place.

For example, odds are that an XR environment will try to mimic natural gestures, such as pushing buttons, picking up objects, nodding for yes, etc.

But for the most part, there are no hard and fast rules — certainly nothing like the standard UI on a Mac, PC, or iPhone.

So at least for the next few years, designers will bear more responsibility for how users interact with XR.

User Experience Design (UX)

A poorly designed user experience on an app or website can frustrate and even drive away users.

But with XR, badly designed UX can (virtually) smack you in the face or even make you throw up.

So is important as UX is with your design work today, it’ll be far more critical for your XR work.

Because XR is designed to be immersive, it brings with it unique UX challenges.

For example, if you’re building a VR app, you’ll need to think about how to reduce the chance that your users will get nauseous.

Typically, designers try to provide a fixed reference point, such as a dashboard or a horizon line. And the more a scene contains movement, the more helpful it is for users to have some control over it.

You’ll also need to think about the scale of the space you’re creating.

If it’s too small, some users may get claustrophobic.

If it’s too tall, they may experience vertigo.

Finally, designers will have to take ergonomics into account. If you design an environment which encourages users to repetitively move their arms in a way that creates strain, you can accidentally end up injuring them.

Luckily, Google, Microsoft, Oculus, and others have been developing UX guidelines and standards for XR, so you won’t be starting from scratch.

 

Performance

As a developer, you’ve probably already had to spend at least some time worrying about performance. If you’re creating graphics for a website, for example, you don’t want to create so many large graphics that the page loads slowly.

But with XR, performance moves from being a sideshow to one of your main headaches.

Because XR is so personal and immersive, your clients will always want your graphics and effects to be as detailed and realistic as possible. But the more bandwidth your design chews up, the more sluggish your users’ experience will be.

That’s why one of the main challenges developers will wrestle with is finding the right balance between detail and speed.

You might be thinking, once we get to 5G won’t this problem go away?

Yes, for a minute. And then designers and their clients will think of 20 new ways to chew up more bandwidth, and we’ll be right back to where we started.

Once XR takes off, you’ll spend the rest of your career alternating between being excited about the latest tech and/or tricks that give you more speed to play with and being frustrated with the cool new effects and techniques that take it away.

Coding And a Tiny Bit of Math

Today, it’s not uncommon for designers to know a little bit of coding. With XR, being able to code a bit will become even more useful.

For example, say you want to represent how much of a backlog someone has on their virtual desk by the height of a stack of papers. Odds are that’ll require a tiny bit of scripting.

Similarly, if you’ve created a little animated koala bear and you want to run the animation that makes their eyes bug out when a user picks the wrong choice, you’ll need coding to connect them.

Tired of using thermometers as a sign of fundraising progress? Whip up a flower whose animations you can run to show it at different stages of growth, then add a bit of coding to tie it to your client’s membership fundraising so far.

Coding can also be used to create truly unique effects using special-effects libraries, generate a unique forest using a terrain library, etc.

Over time, some of these tasks will have smart UI’s that take away the need for coding. But if you want to create a unique design, a bit of coding will always be helpful.

To do XR scripting, you also need to remember or relearn a little  math.

Luckily, most of the important math you’ll need to be comfortable regularly using is pretty simple — addition, subtraction, multiplying, and dividing.

But when I first started working with XR, it was a little bit of a shock when, for example, I needed to use a little math to dynamically place one object next to another. Call it the revenge of my high school math teacher.

The Threat and Opportunities of AI

Over the next decade, XR will create lots of new opportunities for designers. But as XR generates the need for a lot of design work, machine learning and AI will be snapping at the heels of designers.

For example, last year researchers unveiled StyleGANs, a type of generative adversarial network that can “switch up an image’s style.”

Give StyleGANs the right set of pictures, and it can generate variations on everything from portraits to new watch styles.

Similarly, Nutella ran a campaign in which a machine learning algorithm generated 7 million unique versions of Nutella’s iconic jar that were sold in Italy and elsewhere.

And last March, Nvidia unveiled an early version of a tool that can turn a crude sketch of a landscape into a photorealistic picture in seconds.

Over time, it’s quite likely that as machine learning gets more sophisticated, a lot of routine design work will no longer require a designer. And XR is likely to accelerate this trend.

Talk to any programmer who works in XR, and they’ll tell you it takes an impressive amount of design.

As XR becomes more sophisticated, the need for graphic design will only grow over time — so much so that any big tech company or start up that can automate a lot of design work will make billions.

But that doesn’t mean designers are going to become obsolete.

If anything, machine learning and other forms of AI may create more opportunities for designers — at least for designers who know how to take advantage of them.

In our modern society, people crave novelty.

So the more AI makes it easier for nondesigners to produce good quality routine design work, the more people and corporations will seek ways to stand out from the crowd.

Rather than worrying about the work AI will take away from them, smart designers will look for ways to leverage new AI tools to produce work that their clients can’t make themselves or purchase anywhere else.

If anything, smart designers will see AI as an opportunity to expand the role they play in their workplace or with their clients.

The easier it is to produce routine design work, the more nondesigners need designers to help them understand what makes for great design.

So in the future, good designers won’t have to spend hours painstakingly creating yet another round of slightly customized buttons. Instead, they’ll be able to focus more on the parts of their work that got them excited about design in the first place.

How to Get Started

Now that you know all the skills you’ll need to master, you may be feeling overwhelmed.

If so, I’ve got good news for you. 

It’s not that hard to get started. And growing your skills and confidence over time isn’t rocket science.

Here’s how.

1) Get Some Gear and Play

The first thing you need to do is to start playing with XR.

If you can afford it, your best bet is Occulus Quest.

For around $400, you get an all-in-one VR headset that’s optimized for gaming and that comes with two controllers that let you manipulate a VR environment using your hands.

Is $400 too pricey for you? For $15 or less you can get a version of Google Cardboard.

Pop your smart phone into a Google Cardboard and you can start surfing VR.

Google Cardboard is limited in what it can do, and it doesn’t come with controllers. But you can still get a lot out of it when you’re first getting started.

If you can afford to seriously invest in your XR skills, your best bet is the Hololens 2. 

Hololens is a high-powered XR platform that’s the closest you can get to where XR is headed. As mentioned earlier, it costs a whopping $3,500, although $99 a month developer version is due out this year.

 You may also be able to experiment with custom AR apps on your smart phone. Newer versions of the iPhone and Android have limited but functional AR support built-in, and both have a reasonable number of apps.

Regardless of which platform you choose, you should also definitely check out WebXR, which runs in most modern browsers.

All you need to do is use a recent version of Google Chrome or Firefox to visit A-Frame‘s homepage or blog and Babylon.js‘ demos page or Twitter feed, and you’ll find lots of examples to explore. You can move around in them using your mouse and arrow keys.

Once you’ve got your gear, it’s time to play around.

Although it’s tempting to jump right in and start building your own creations, take the time to explore what other people have built.

And take notes. Lots and lots of notes.

What do you like? What don’t you like? And why?

Use the list of skills above to organize your thoughts. You can learn a lot about 3D, motion, sound, UX, etc. if you pay attention to what you’re experiencing.

Then show your friends, family, and acquaintances examples you like and dislike, and learn from their reaction.

Make sure that some of the people who give you feedback have a very different background or perspective from yours. You’ll pick up XR design a lot faster if you incorporate what you learn from several audiences.

2) Start Small

Now you’re ready to dive in. How should you begin?

If you were blown away by a bunch of elaborate XR demos, it’s understandable if your first impulse is to build something big and cool.

Resist that impulse.

As we saw above, getting good at XR design requires a ton of skills. If you try to take a bunch of them at once, you’re going to be overwhelmed.

Also, you will learn so much more and gain so much more confidence if you build 3 to 5 small projects instead of one big project.

So, how do you start small?

You could surf a site like Sketchfab, which has lots of free 3D assets.

Find a simple 3D asset — e.g., a puppy or a spaceship or a slice of pizza — download it, and try tweaking it.

Or if you look at several low res owls on Sketchfab and think, I could do a better job, make one from scratch.

Whatever you decide to design, create at least one high-res and one lo-res asset; it’s a good way to start getting a feel for performance trade-offs.

When you first get started, you don’t need to spend a lot of money on tools. There are lots of free or cheap tools you can use.

Blender, for example, is a terrific, powerful 3D design and animation tool that’s free and that runs on Mac, Windows, and Linux. And there are plenty of terrific articles and videos online on how to use it.

For each simple 3D asset you create, upload it to a site like Sketchfab and send it out to your friends or colleagues.

Once you’ve created a handful of 3D assets — or if you’re already experienced with 3D — now it’s time for a slightly more complex project.

For example, a number of people in the WebXR community have created VR postcards. Because of the way WebXR works, you can get a copy of the postcard’s source code by just using View Source in your browser.

So, find a simple postcard you like and try changing some of the graphics or text, tweaking the colors and positioning, etc.

If you get stuck, don’t spend too much time banging your head against the wall by yourself. Both A-Frame and Babylon.js’ communities are very friendly and helpful; just make sure you treat people’s time with respect by paying attention to the norms about how to ask a question.

Once you’ve mucked around for a bit with someone else’s postcard, try creating one from scratch.

Or maybe take a lyric from one of your favorite’s bands and create a simple scene that shows it off or illustrates it.

If you’re happy with the end result, tweet it out to the band’s main Twitter feed. Who knows? Maybe they’ll like it and retweet it.

3) Start Building a Portfolio

Now that you have a clue as to what you’re doing, it’s time to start building your first portfolio. Here’s one strategy you might find useful.

In the next few years, every corporation, nonprofit, and government agency is going to need to start exploring XR.

So, brainstorm about how your favorite coffee shop, tattoo parlor, the place you get your hair cut, local event you like to attend, or a local or national nonprofit you admire might get their feet wet with XR.

Sketch out some ideas. Play around a bit.

And then show them what you’ve done and see if they are interested in having you build a simple XR experiment for them for free or barter.

You could try creating an app. But you’re probably better off using WebXR.

Almost everyone you’d be interested in helping has a website. So most folks who visit their website will be able to explore what you’ve built.

Why start with an organization or event?

Because at this point in your XR design journey, you will learn so much more if you are working with a client.

It’s also easier to get paying clients or get hired as an XR designer if you’ve got happy clients you can use as a reference.

Another way to go: offer to help a small open source project or library.

A lot of small projects are driven by developers who have a little experience with design. They could certainly use some help from a skilled designer.

For example, if someone has created an A-Frame or Babylon.js terrain generator library or special-effects library, they could almost certainly use some more eye-catching demos.

How would you find these projects? You could browse the platform’s website for a specific library that interests you, or you could just ask on their community sites.

Regardless of what projects you decide to take on, small and well executed is better than big and complicated —  especially when you are new and don’t know how much work design will take.

4) Team Up

There’s an awful lot to learn to be able to create great XR designs. But you don’t have to do it alone.

Are you a skilled designer who’s nervous about coding? Or you can do it but it’s not your idea of a good time?

Odds are you or one of your friends or colleagues knows some developers who would love to work in XR but are intimidated by the design elements. So why not team up?

Or maybe you know someone who’s skilled at mixing sound or likes composing music. Why not get them involved?

Teaming up is especially a good idea when you get to the point where you’re ready to experiment with machine learning libraries that can be used to generate graphic design.

A lot of these libraries are built in languages like Python, where it’s easy to whip up a custom library on top that hides some of the geekier stuff, letting you focus on the parts where a designer can really soar.

If you’re going to spend a good chunk of your career working in XR design, you’ll almost certainly be working in teams. If you start sooner, you could substantially speed up your learning.

The Time to Start Your Adventure Is Now

XR is going to revolutionize the design world. The sooner you get your feet wet, the easier it will be to get up to speed and take advantage of some of the earliest opportunities.

And working in XR can be a lot of fun. It often feels like you’re playing with magic. 

So why not begin your journey today?