360 Video & Virtual Reality (VR) in JW Player Part 1 : State of the Industry

JW Player is proud to announce support for 360 Video playback and VR in our HTML5 player and native mobile SDKs, providing a smoother more performant experience for your iOS and Android apps.

In Part 1 of our blog series, we will present our perspective on the current state of 360 Video & VR technology and its emerging standards.  In Part 2 next week, we will explain how you can publish 360 videos and VR experiences on your own websites and apps using JW Player tools.

You can also visit our JW Player 360 Video / VR demo page to try it out right now!

jw-player-360

Calibrating Terms : 360 Video vs VR

Since VR & 360 Video have gained global awareness and become more mainstream with consumers and the media, the newness and lack of standards are lending to some confusion in a “Wild West” of terminology & technology.  The terms 360 Video, VR, live-action VR, CG VR, AR, MR, are often used interchangeably and incorrectly.

In order to alleviate this confusion, we will define some terms if only to be concise in our language within the scope of our 360/VR blog series.  To accomplish this we scoured the web and spoke with some veterans in the industry, some that have been pioneering VR & 360 Video since before Google VR or Facebook 360 even existed, to help give us clarity.

360 Video, VR, Magic Window, Live Action VR

360 Video is a term best used to describe the content or medium itself: a video that offers a 360 degree spherical view around the viewer, and “during playback the user has control of the viewing direction.” But there are many ways to watch 360 Video content.

360 Video can be watched on desktop browsers, mobile browsers, and native apps, in what Google calls a magic windowmode.  This is the traditional rectangular video player on your screen that users can typically watch on their mobile phone in two ways: tap & drag on the video itself (like on desktop); or just move the phone to change perspective as they view a particular slice of the 360 degree video; but the view is limited to this small window to the virtual world.

But 360 Video can also be watched using virtual reality.  VR is the immersive experience you have when wearing a special head-mounted display (HMD) like Oculus Rift, and both your actions and perspective in the virtual environment are controlled by the movement of your body.  Diving one level deeper, a popular type of VR is 360 Video filmed with a camera/rig (also known as live-action VR) where the viewer is tethered to the camera’s 360-degree view of the world and moves where the camera moves.  

On the other hand, a viewer can also enjoy an untethered experience created artificially using digital computer-generated VR (CG VR).  CG VR gaming experiences like those on HTC Vive and the upcoming Sony Playstation VR allow users to freely explore the virtual space and can choose where to move untethered by any camera.

In addition to VR there is also augmented reality (AR) and mixed reality (MR).  The Foundry offers a great article that highlights the differences, and we will touch upon them at the conclusion of this post.  But in the meantime, how does all this 360 Video technology work?

360-infographic

360 Video Playback Technology

360 Video playback on a device requires the following basic components:

  • A stitched 360 video in a specific layout.  Stitching software is commonly packaged with 360 camera hardware, and typically outputs the video in an equirectangular layout.
  • A sphere rendered in 3D space, with an imaginary viewer in the middle.
  • Some method to control this imaginary viewer’s perspective.  On desktops/laptops that is your mouse and keyboard, and on mobile devices the gyroscope, accelerometer and touchscreen.

Internally, 360 Video is decoded like any other video – but where a conventional video player directly draws the decoded video frames to your screen, a 360 video player uses them to texture the sphere in 3D space.  After every decoded frame the video player updates this texture and this allows video playback on the sphere.

On mobile phones, the viewer’s perspective can be controlled by the device’s gyroscope and accelerometer.  As each frame is being rendered, data is read from the device’s sensors and used to determine the viewer’s orientation (yaw, pitch, and roll).  On desktops and laptops a simpler conventional mouse-controlled view is used to track only yaw and pitch.

Emerging Standards in 360 Video

Currently, the most commonly used projection to store and serve 360 videos is the equirectangular projection.  But, the equirectangular layout is not ideal – it contains redundant information due to the distortion in the polar areas.  Moreover it cannot be encoded as optimally as other projections such as the Cube Map.  Facebook engineers have written a really interesting blog post about the problems with the equirectangular layout and how they can be tackled.

We are investigating ways to optimize the delivery and playback of 360 videos and are very encouraged by Fraunhofer Institute’s approach of solving the bandwidth problems that arise with the delivery of high quality 360 Video.  They developed a technique to use the tiles feature of the HEVC codec to optimize delivery of 360 Video.  This technique allows streaming of 16K 360 Video in a 4K stream, we believe that this will really push the 360 Video experience to the next level.

OpenGL/WebGL, WebVR, 360 Video Metadata, & 360 Video Ads

VR relies heavily on OpenGL and WebGL technology.  OpenGL is an API for rendering 2D & 3D vector graphics and has been around for many years – its first release was in 1992.  More recently OpenGL was introduced to the browser in the WebGL standard.  Within the context of VR these API’s are responsible for doing the actual graphics rendering work.  OpenGL is supported natively on Android and iOS whereas WebGL is supported in every major browser.

Another emerging standard we are excited about is WebVR, an experimental JavaScript API that provides access to VR devices in the browser.  WebVR is an initiative by Mozilla’s VR team and Google.  WebVR works in conjunction with the WebGL API, it provides the data required to properly render graphics for VR devices.  After we integrated WebVR into the JW Player HTML5 VR plugin to support popular HMDs (eg. Oculus Rift, HTC Vive), we found that performance was decent, but was still buggy and unstable across the browsers.  One nice advantage is that the WebVR polyfill allows us to support “magic window” mode and VR mode on mobile web browsers similar to our native SDKs.

In addition we are paying close attention to Google’s initiative for spherical video metadata inside of MP4 (ISOBMFF) and WebM (Matroska) multimedia containers.  Storing spherical video metadata within the actual multimedia containers will make playback of 360 videos on the JW Player easier as we can read how the player should be set up from the video file itself instead of asking developers to supply this data.  Another advantage of having the spherical video metadata inside the video is that content in JW Player will always be displayed the way the content creator intended it.

Finally, since 360 Video can also be useful to create immersive advertising experiences, we are keeping a close eye on IAB’s Standard Ad Unit portfolio.  IAB is working on creating 360-degree video ad units that will allow players to properly handle this new ad format and allow players to correctly display both traditional ads and 360 Video ads while still maintaining a good viewer experience.  We see a lot of opportunities in 360 Video for advertising.  Many brands like Nestle, Coke, and Volvo are starting to experiment with this format and we’d like to support them with our player so that publishers can monetize their 360 videos.

Challenges for the Industry

Every technology or industry that has explosive growth inevitably comes with challenges.  In an upcoming blog post, we’ll touch upon technical challenges such as performance and stability, but here we’d like to highlight two industry challenges for virtual reality: fragmentation and storytelling.

Mobile Fragmentation

We see fragmentation in the mobile market for VR. Currently there are two major competing VR platforms on mobile: Facebook and Samsung with their Oculus/Gear VR ecosystem; and Google with its Cardboard/Daydream ecosystem. Unfortunately, we have found that apps made for one ecosystem are not compatible with the other.  Since we have initially chosen to support Google’s VR ecosystem, our JW VR SDKs are not yet compatible with Oculus’s native ecosystem.  We expect the WebVR standard to solve this cross-platform issue for browsers, but it doesn’t look like this issue will be solved for native apps any time soon.

360 Video Storytelling

Last and most importantly, how will 360 Video and VR avoid the fate of VR from the ‘80s & ‘90s and 3D TV? In our opinion the answer lies in the adage “Content is King.”  In addition to content, there are other factors that we’ll discuss further in Part 2, but this begs an even bigger question. Can 360 Video / VR be an effective means of storytelling – a medium compelling enough to entice future filmmakers and be more than just a fad?

The answer is still unclear, with strong opinions from both sides.

The Guardian published an article quoting Steven Spielberg saying “I think we’re moving into a dangerous medium with virtual reality.  The only reason I say it is dangerous is because it gives the viewer a lot of latitude not to take direction from the storytellers but make their own choices of where to look.”

Other articles explain in their opinion why VR storytelling does not currently work and the challenges of VR’s non-linear format.  The difficulty is that user’s don’t know where to look, they risk missing the action, and VR is passing the workload buck to their viewers.  The traditional approach of a linear narrative, with connected sequences of events carefully crafted to tell a specific story envisioned by the filmmaker, is being challenged by the freedom of 360 Video.

But many filmmakers, publishers, and organizations like the New York Times and United Nations, have jumped on this new format and are telling empathy-driven stories through 360 Video/VR.  They are exploring new possibilities with live-action VR such user interactivity, merging real-life and virtual reality together with AR/MR, and are helping to pioneer the future of storytelling by capturing their audience and keeping them engaged.

“Storytelling in VR and 360 video is fundamentally different from traditional film and therefore needs to be approached as such. The goal is to have one connecting theme throughout the entire experience, while also providing a secondary and tertiary narrative, with the viewer having agency over which to follow. To me, this is the real the magic of VR storytelling.”   – Max Bolotov, Executive Producer at Koncept VR.

Are You Ready to Publish 360 Video & VR?

So if you’re a publisher or content creator with a library of 360 videos that you want to publish on your websites and mobile apps, what do you do?  Check out our JW Player 360 Video / VR demo page, try out our new JW VR Plugin & VR SDKs, and leave us a comment to let us know what you think.

Stay tuned for Part 2 of our 360 Video blog series where we share details about JW Player’s new 360 Video / VR product solutions and how we’re helping to make this emerging technology super simple and easy for everyone to use.  In the meantime, enjoy our JW Player HTML5 VR Plugin demo below!