Encoding HLS with Amazon Elastic Transcoder

Update: JW7 is now available. Check it out here.

HLS is a powerful streaming protocol that Apple introduced in order to allow for live streaming on iOS devices. JW Player introduced HLS support in Flash too.

If you are looking to encode only a couple of videos, you could consider converting all videos manually, but if you have a larger library, it is important to replace any manual steps with a good encoding pipeline. Amazon’s Elastic transcoder has all the important components you need to create a transcoding pipeline using Amazon’s API’s.

The goal of this tutorial is to build an encoding pipeline that looks in one bucket for origin files and then uploads the converted HLS stream and thumbnail images to another bucket. This tutorial assumes that you have already signed up for Amazon S3 and Amazon Elastic Transcoder and can access these services from your AWS console.

Setting up storage buckets

You need an incoming S3 bucket; Go to the S3 console to create a new bucket. Choose a region that is as close to you as possible in order to optimize upload speed. After this, proceed to make two more output buckets, one for thumbnails and one for HLS streams.

By default, Elastic Transcoder makes transcoded videos private. For this tutorial, we’ll change these permissions to make the output streams/thumbs public. Do this by adding a permission for the Amazon S3 Group “All users” to open/download:

Setting up a transcoding pipeline

The first time you log on into your Elastic Transcoder console you will see a page that explains the three main components of Amazon’s Elastic Transcoder: pipelines, presets and jobs.

For now each components will be addressed as needed. The first thing you need to do is to create a pipeline. A pipeline connects your In bucket to your Out buckets. Setting up a pipeline is really easy. In the Elastic transcoder dashboard simply go to the Pipelines section, create a new pipeline and then select the origin bucket and the target buckets for streams and thumbnails.

By default, Elastic transcoder has 5 HLS system presets that give you a good range of qualities to begin with. You can create presets for additional formats and sizes as needed.

Creating a job

Now that the pipelines are set up, you can start to process a video. To process a video upload it to one of the incoming S3 buckets. Now go to the Jobs section of the Elastic Transcoder dashboard, create a new job, select the pipeline that you uploaded the video to, and, for the input key, choose this video. The output key prefix should be a unique identifier that indicates this video.

In the Output Details you now select an HLS preset. For the segment duration Apple recommends that you use 10 second segments. Each output also needs their own output key that indicates what preset created it.  For this example you will use three quality levels: HLS 400K, HLS 1M and HLS 2M.

Finally (at the bottom of the job creation interface) you can create a master playlist that links to all individual HLS streams. You will need this master playlist to set up an adaptive streaming player later on.

Now press the “create new job” button and wait for the video to process. If you like the settings you had for one job and wish to use the same settings for a new job, you can copy a job in the jobs list and the same job options will be applied to new job. You should change the input key to be that of your new input file, the output key and/or output prefix.

Generating preview thumbnails

Amazon’s transcoder job presets create very small preview thumbnails. You’ll want a nicely large one though as poster image. That can be achieved by customizing a Preset. Simply select one of the job presets you used and press the “Copy” button. You will now end up in the preset editor. At the bottom, you will find the Thumbnail settings:

Choose a JPG image (or PNG if your content is motion graphics) and an interval that makes sense for your content. Note Amazon cannot yet create a single thumbnail. Next, dimensions of 640×360 provide a good tradeoff between filesize and crispness. Bigger looks better, but loads slower. The sizing and padding policies should be fine as-is.

Now, in order to create thumbs, simply include this Preset with your jobs. The thumbnails will automatically end up in your S3 thumbnail Out bucket.

Demo and Wrap

You’ve now created a transcoder pipeline with presets that render an adaptive HLS stream and high-quality preview thumbnails. Let’s see what this output looks like in real life, by creating a basic video tag with the HLS master playlist and one of the JPG thumbnails:

<video height="270" width="480" controls
  poster="http://d3mk7n3cl27pjb.cloudfront.net/bigbuckbunny/bunny-00004.png"
  src src="http://d1s3yn3kxq96sy.cloudfront.net/bigbuckbunny/index.m3u8">
  <p>Your browser does not support HLS playback in HTML5.</p>
</video>

Please note that this demo will only work on the browser that supports HLS playback in the video tag: Safari for iOS and Mac. In a follow-up tutorial, we’ll look into using JW Player to play this HLS stream on other browsers and devices. We’ll also look into using Cloudfront instead of S3 for the delivery, so your streams are delivered faster and cheaper.