moving pictures - web 2.0 expo nyc

Moving PicturesImplementing Video on Flickr

Cal Henderson

Web 2.0 Expo NYC, 17th September 2008 2

Hello


Flickr

• Large scale kitten sharing website

• Started 2003, launched 2004– 5 years old this december

• Almost 3 billion photos


Enter: Video

• Video was added this year– Launched April 2008

• Many hundreds of thousands of videos uploaded

• Many millions of playbacks


“Video? That’s just like photos!”

-Me, before Flickr Video


12 4 Steps

• 4 main tasks– Uploading– Transcoding– Storage– Serving & Playback


1. Upload


Simple upload

• Web forms– Just like any other file

<form action="/uploadify/" method="post" enctype="multipart/form-data">

<input type="file" name="fred" /> <input type="submit" value="Go!" />

</form>

• But files are large / huge


Issues

• Two components for uploading:– Sending from the client– Receiving on the server

• Both of these present problems


Sending from the client

• Multiple options– Simple form– Flash– Desktop app


Simple Forms

• Pros– Very easy to implement– Works on every browser out of the box

• Cons– Upload progress is harder– ‘Slow’– Select a single file at once


Flash

• Pros– Upload progress is easy– ‘Fast’– Multi select of files

• Cons– Harder to implement– Flash isn’t quite ubiquitous


Desktop App

• Pros– Upload progress is easy– ‘Very fast’– Multi select of files– Drag and drop

• Cons– Hard to develop– Hard to deploy (relative to the web)


Making Progress

• Upload progress– Not impossible with plain forms– Just need to be able to query the upload

progress via AJAX

• Multiple machines– The VIP issue– Enter Perlbal


Making Progress

Browser

Web 1

Web 2

1. Browser starts upload

2. Web server broadcastsprogress via UDP

3. Browser queries progressvia AJX call

Loadbalancer


Receiving on the server

• File uploads are slow– Much slower than serving pages

• Apache processes are heavy– Waste of resources

• Use a poll based server (like jetty)


Receiving on the server

• Or, use a buffering layer– Perlbal is great for this

• Or a lightweight Apache– E.g. w/ mod_proxy

Browser Buffer ServerSlow Fast


But wait

• It’s not just the first step that’s slow

• Moving files around between servers is slow– Do it out of band

• Asynchronous jobs are in order anyway– Do it!


2. Transcode


Transcode?

• Why transcode at all?

• Input comes from many sources– Point & shoots– DV Cams– Mobile devices– Video editing software

• All in different formats


So many formats

• But very few of these formats can be played back cross platform– Without special software or hardware

• Formats are designed to do one thing well– They don’t always manage even that

• Transcoding puts all videos on an equal footing


Video file basics

• Most file types are really just containers– MOV, FLV, AVI

• The data inside can be in multiple formats– We call these codecs (encoder + decoder)

• Files contains multiple ‘streams’– Both audio and video


Interleave

• Audio and video are often interleaved– Hence AVI

• A video file looks like this:– Headers– Video chunk– Audio chunk– Video chunk– Audio chunk– Etc


Compress

• Raw video files are huge– A bitmap for every frame– Rarely used, even in post production

• At 30 fps, that gets crazy pretty quickly

• We don’t need to store every frame– Static backgrounds don’t change (much)

between frames


Compresssss

• Intraframe– Treat each frame as a picture– Compress it (just like JPEG)– DCT (Discrete Cosine Transform)

• Interframe– Store the differences between frames– Treat the pixels as a 3D array to be

compressed


The IPB

• Three frame types: I, P & B

• Intra coded pictures– A full raw frame

• Predicted pictures– Based on a single reference frame

• Bi-predictive pictures– Based on two or more reference frames


IPBIPBIPBIPB

• Reference frames may be I, P or B

• P & B frames may contain a mix of image data and motion vector displacements

• I frames require the most bits– Then P frames– Then B frames


Bad terminology

• We should really say picture– (Not frame)– Because of interlacing

• Really, we encode fields not frames– Picture is the general term

• And H.264 contains ‘slices’– Sub-regions of the field– ‘Macroblocks’ & ‘Artifacts’


I-Frames

• Also called Key Frames

• Allow easy random seeking

• Twice a second for Digital TV & DVDs

• More widely spaced online


Seekable


Oh, and audio too

• We can worry less about this– Older problem, well solved

• MP3 is pretty good– Who cares how it works?

• Syncing is the only issue– Presentation Time Stamps (PTS) and Decode

Time Stamps (DTS) in MPEG-2


Flash! Woah-oh!

The big question:

Flash?


Non-flash sites• QuickTime• Windows Media• This is gradually disappearing

• Flash player is ubiquitous• Compression is good enough• Interactive too

• But no 3D/VR as with QuickTime :(


The Flash Player

• Flash Player 6– March 2002

• Video: Sorenson Spark (H.263)• Audio: MP3

– Or ADPCM / Uncompressed– Or Nellymoser Asao


Second Generation

• Flash Player 7– August 2005

• Video: On2 TrueMotion VP6• Audio: MP3


The hot shit

• Flash Player 9 (update 3)– December 2007

• Video: H.264 (MPEG-4 Part 10)– w/ container formats from MPEG-4 Part 14

• Audio: AAC (MPEG-4 Part 3)

• Plus 3GPP Timed Text (MPEG-4 Part 17)


TrueMotion VP6

• Proprietary• Reasonable compression• Created by On2

– Patented– Probably illegal for GPL code

• YouTube uses it for lower quality and old streams


H.264• Not proprietary• Good compression• MPEG Standard

– Open, but patented– Patent licenses from the MPEG LA– Unclear how this applies to (L)GPL code– But probably badly

• YouTube using it for higher quality streams

• iPhones and AppleTV


Software

• Open source transcode tools

• FFmepg– libavcodec for VP6– Probably illegal – dubious– Also pretty shoddy– Can only decode H.264


More software

• MEncoder– libmpcodecs uses libavcodec

• VLC– libvlc uses libavcodec

• So basically the same– Different muxing, same codecs


Free H.264?

• Unfortunately, not really

• x264 is the only usable one– It’s pretty good– MEncoder can use it– Still limited in options at this point– Again, dubiously legal


Non-free tools• Flash encoder

– Not automatable

• On2 FlixEngine– Creators of VP6– Windows or Linux– Some support for H.264

• Rhozet Carbon Coder– The new hot shit– Good H.264 support– Windows


Choices

• Video codec• Resolution• Bitrate (VBR, CBR)• Keyframes• Audio codec

– Channels– Bit depth– Sampling rate


Doing it at scale

• Not really a problem• Very easily parallelizable

• Amazon EC2 is awesome here– Exactly what it was design for– Grow/shrink as needed– But, per-CPU software licensing


3.Store


Easy!

• Really, just like photos– But with bigger files

• Same disk layout as any other serving

• But the serving part is slower


But..

• Remember the files are huge

• Operations take time– More likely to fail halfway through– Checksums are your friend

• Do it all asynchronously


4. Serve & Playback


The choice

Streamingvs

Progressive


Streaming

• Pros– Easily seekable– Live feeds

• Cons– Special server software– Slower to start– Firewall troubles


Progressive download

• Pros– Just use a web server– Play offline– Firewall/proxy friendly

• Cons– Harder to seek ahead (but not impossible)


Streaming tech

• Non flash stuff– We’ll ignore that– You’re using flash, right?

• RTMP– Real-time Messaging Protocol– Proprietary (thanks Adobe!)


RTMP

• RTMP - Raw TCP socket stuffs• RTMPT – RTMP tunneled over HHTP

– For firewalls, etc

• Flash Media Server– previously Flash Communication Server

• Wowza Pro– $1000/server


Open source

• It’s not all bad

• Red5– Java implementation of RTMP server– Mostly feature complete– Beta quality, but usable in production– Facebook


Progressive

• Used by the majority of large sites

• Very simple!

• Seekable with server support


Seeking

• Serve the FLV starting at a different point

• Just add a simple FLV preamble before seeking into the file

• Simple to do in PHP, Perl, etc• mod_flvx for Apache• mod_secdownload for lighttpd


5. Other considerations


Review

• Videos are slow• Expensive to review

• Review grids– Doesn’t cover audio


Not enough?

• Social tools are useful here


Summary


Summing up

• Flash makes sense– For uploading too

• H.264 is probably your best bet today• Transcoding software still costs money

– Unless you’re willing to take on the risk• Progressive download is basically

awesome


The end!


Awesome!

These slides are available online:iamcal.com/talks/

moving pictures - web 2.0 expo nyc

Technology