History of Virtual Reality, Part 2: AR/VR in Digital Entertainment

Virtual Reality enjoys a long and exciting history, but the bulk of that excitement has occurred within the last decade. My first installation in this series introduced you to VR’s early trailblazers and the technological limitations they faced.

Today, we’ll meet the pioneers of modern-day digital entertainment. These teams successfully incorporated Virtual Reality and Augmented Reality into video games and other entertainment products. From Palmer Luckey’s Kickstarter campaign for the Oculus Rift to the prototypes and developer kits that followed, these are the events and players that got us here.

Oculus Rift smashes Kickstarter records and introduces believable Virtual Reality tech

In 2012, the Virtual Reality industry entered the limelight and showed up on every high-tech consumer’s radar. Palmer Luckey offered his prototype to the Kickstarter crowd, and smashed the crowdfunding platform’s records. This product, Oculus Rift, raised a whopping $2.4 million from over 9,500 people around the world.

This campaign was a profound statement for the digital entertainment industry. The tech for believable VR was here, and consumers were ready to part with their money to try it out.

Consumers weren’t the only ones to turn their head towards Oculus. Big players at the time, such as id Software’s John Carmack (now CTO of Oculus), Epic Games’ Cliff Bleszinski (now cofounder of Boss Key Productions), and Valve’s Gabe Newell; all endorsed the campaign when it went live.

A developer kit soon followed

Oculus released their first developer kit on March, 29, 2013. The kit sported an RGB display with a resolution of 1280×800 (640×800 per eye) at a refresh rate of 60Hz. It featured a horizontal Field of View of 90 degrees, which is more than twice any previous head-mounted display (HMD). The kit had swappable lenses and allowed adjusting of the focal distance in the headset.

7,500 Kickstarter backers received these developer kits, and a public version was available later for $300.

The Oculus dev kits allowed developers to add rudimentary VR support to games. While this first edition only featured 3 degrees of freedom (rotational tracking via gyroscopes), it gave an immediate feeling of presence which was unparalleled by any preceding technology.

Later that summer, Oculus showed off an HD prototype HMD that featured a display with twice the resolution as the initial developer kit. This prototype showed that even small, incremental improvements could yield head-turning progress for the technology.

Crystal Cove HMD

The following year, at CES, Oculus unveiled their latest prototype to the world: Crystal Cove.

Crystal Cove was Oculus’s first publicly-shown prototype to feature rudimentary positional tracking. This made it the first HMD with 6 degrees of freedom (rotation and position on 3 axis each). To achieve this, the headset was fitted with multiple IR emitting diodes which would be tracked by an external sensor that faced the user. The diodes were placed in such a way that the tracking camera could determine how the headset might be oriented by the “constellations” made by the diodes.

This was the origin of Oculus’s so-called Constellation Positional Tracking System.

Crystal Cove later launched the second Oculus Development kit, dubbed the DK2. This new Headset boasted a low-persistence display of 1080p resolution (960×1080 per eye) and a refresh rate of 75Hz. Oculus shipped over 100,000 units, making the DK2 widely adopted by developers around the world.

Positional tracking proved to be an essential feature of believable VR.

Sony iterates on Augmented Reality

Based in Tokyo, Japan, entertainment giant Sony started working on Augmented Reality in 1999, when Dr. Richard Marks joined the company’s American arm and began conceptualizing the EyeToy. Sony released the EyeToy for the PlayStation 2 in 2003.

The EyeToy wasn’t necessarily a sales sensation, but it was the first offering that introduced Sony to the Virtual Reality scene. This peripheral featured sophisticated blob-tracking algorithms to find certain parts of the human body from the camera image. After a calibration, the camera was able to take relative scale into account in order to approximate the distance the tracked object was from the camera.

From EyeToy, to Eye, to Move…

Dr. Marks persisted despite lukewarm reception, and released the PlayStation Eye in 2007. The Eye failed to rally the fervor that earlier digital entertainment products from Sony had celebrated. Similar to its older brother the EyeToy, the PS Eye would focus mainly on Augmented Reality gameplay experiences.

The tracking power of the Eye would give birth to the PlayStation Move, a motion controller accessory said to be designed with future head-mounted technology in mind.

…To Morpheus, to PlayStation VR

At the Game Developer’s Conference in 2014, Sony announced their Project Morpheus. Morpheus, later to be known as PlayStation VR, was a headset that Sony began development on shortly after the release of the Move. At the conference, Engineer Anton Mikhailov noted that his team’s work on the project had begun three years ago, predating the funding of the Oculus.

The PlayStation VR’s position in the market had two major advantages over its competitors: price, and a pre-existing install base. 32 million PS4 consoles had already been sold prior to the release of the $400 PlayStation headset. So, at $800, PlayStation’s all-inclusive VR system looked better than a Vive headset for $800 or $600 for the Oculus.

The low cost-barrier for this first generation of VR headsets made the PlayStation solution a consumer hit. As a result, PSVR outsold Vive units in the first quarter of 2017 by four times.

Motion capture with Microsoft

Microsoft announced Kinect (originally revealed as “Project Natal”) at E3 in 2009. The demonstration included the sensor and the ever-ambitious Project Milo, conceived by designer Peter Molyneux of Lionhead Studios. Early in prototyping, the goal was to track up to four human bodies, with a feature extraction of 48 skeletal joints at 30Hz. And, the unit would feature an onboard chip capable of running the tracking algorithms in the sensor.

But on release, Microsoft scrapped the onboard chip and left the Xbox console to do the heavy lifting of interpretating the data from the sensor. The final version of the Kinect, released for the Xbox 360 in 2010, was only capable of tracking two users and extracting information for about 20 joints.

Trying to Kinect

The sensor array in the Kinect made it a very capable low-cost motion capture system. This helped to lower production costs of independently-produced games.

Microsoft didn’t stop after their first attempt, either. When they announced the Xbox One in 2013, they revealed Kinect’s successor. The Xbox One included the Kinect 2, giving the console voice recognition functionality. Now, players could issue console commands without using a controller.

Unfortunately, there was very little developer support for games utilizing the new Kinect. Microsoft eventually removed it from the base model Xbox One, and lowered the entry price for the console.

Here comes the HoloLens

In 2015, Microsoft unveiled their internal “Project Baraboo,” later known as the HoloLens. The HoloLens is a Mixed Reality headset heavily inspired by the Kinect sensor. HoloLens utilizes a lot of Kinect’s technology to incorporate tracking of the user’s surrounding space and gesture recognition.

HoloLens was the first headset to augment the user’s field of view with holograms projected onto the glass visor of the headset. And, it was the first headset to feature inside-out tracking to keep track of the user in 3D space.

This method of tracking differs from the Oculus and Vive, which have external sensors monitoring the available “playspace” that the user can occupy. The headset was the first step that Microsoft took into the modern VR space. HoloLens stands as the predecessor to Microsoft’s 3rd-party manufactured Mixed Reality headsets, released in late 2017.

Mixed Reality with inside-out tracking

Microsoft is pushing the use of inside-out tracking for Virtual and Augmented experiences, and they’ve partnered with leading manufacturers to do it. They’re aiming to build a network of mixed reality headsets that run on the software stack in the Windows Mixed Reality platform.

This inside-out methodology effectively allows simulations to be aware of the environment surrounding the user by combining data from multiple input sensors and cameras. Then, the virtual space is laid out accordingly. This is different from outside-in tracking, which forces the user to conform their physical space to match the virtual one.

With the additions of wireless functionality and self-contained headsets, the Virtual Reality market is looking to take some exciting turns in 2018. Next, in part three of this series, we’ll greet the future and I’ll share how I think it’ll change computing for the average person.

---

Augmented Reality Emerging Tech Innovation Virtual Reality