Fall Premieres: Media for Apple Platforms in

Late 2017Chris Adamson • @invalidname

CocoaConf Next Door • June, 2017

Agenda

• HEVC / H.265

• Apple Music API

• Changes to Existing Frameworks

• HTTP Live Streaming

• Developments outside of Apple

HEVC / H.265

H.265• "High Efficiency Video Codec"

• Approximately double the data compression (or much better quality) of H.264 at same bitrate

• Expected use-case: 4K video streaming

• Work began in 2010, standard approved in 2013

• Versions 2, 3, and 4 approved since then

Standardized Mac / iOS support Major Uses

MPEG-1 1993 QuickTime 2.5 Video CD

MPEG-2 1996 QuickTime 6 (optional) DVD, Digital TV

H.264 2003QuickTime 7

(MacOS X 10.4), iPhone OS 1

Blu-Ray, Digital TV, Streaming

H.265 2013 macOS High Sierra, iOS 11 ???

Patent Encumbrances• HEVC contains technology covered by hundreds

(thousands?) of patents

• Patent Pools: MPEG-LA, HEVC Advance, Velos Media

• Don't even cover everything (Technicolor SA)

• Much costlier than H.264 licenses (max $25M/yr [MPEG-LA] $40M/yr [HEVC Advance])

VP9• Google-supported competitor to HEVC

• Royalty-free

• Only patents (🤞) are held by Google

• Supported by 75% of browsers (desktop + mobile)

• Used extensively by YouTube for delivery to non-Apple platforms

AV1

• Royalty-free codec from Alliance for Open Media (AV1 means "AOMedia Video 1")

• Founding members: Amazon, Cisco, Intel, Google, Microsoft, Mozilla, Netflix

• Successor to VP9

• Planned for release in 2017

http://www.streamingmediaglobal.com/Articles/ReadArticle.aspx?ArticleID=118062

Then I asked where HEVC fit in. Aaron explained that Netflix currently deployed HEVC primarily on Smart TVs, and saw the codec as integral to its HDR strategy. For computers and mobile, however, H.264 and VP9 are Netflix's primary codecs, and the focus of most of its current research, which will soon include AV1.

I asked about the future of HEVC given the imminent release of AV1, and the fact that Netflix was a founding member of the Alliance for Open Media. Ronca laughed, explaining that Netflix was still encoding in VC1 format for some older platforms, and that HEVC is important for their UHD/HDR experience, though he noted that the HEVC license uncertainty is an ongoing concern.

NAB 17 Codec Roundup

Jan Ozer HEVC will do well in broadcast, no doubt. Still not available in any browser, iOS, and Netflix prefers VP9/AV1 over HEVC for Android. VP9 gets you most browsers and many smart TVs and OTT boxes (like Roku 4), so it's the smart money UHD codec if you don't need HDR.

http://www.streamingmediaglobal.com/Articles/ReadArticle.aspx?ArticleID=118062

High Dynamic Range (HDR)

http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/HDR-The-More-Bits-the-Better-118643.aspx

HDR: The More Bits, the Better Apple's new Macs feature 10-bit HEVC decoding for HDR video viewing. To understand why that's important, it's worth a look at an old technology—film cameras

By Tim Siglin

Posted on June 6, 2017

The aperture itself determines how deep or shallow the focus will be, which is where we get the terminology for depth of field that's now being tossed about for consumer smartphones. The Apple iPhone 7 "depth effect" feature makes use of two lenses to create a digital version of depth of field.

HEVC in code• AVVideoCodecType.hevc

• AVAssetExportPresetHEVC1920x1080, …3840x2160

• AVOutputSettingsPreset.hevc1920x1080, hevc3840x2160

• kCMVideoCodecType_HEVC

• kVTCompressionPropertyKey_ProfileLevel values: Profile0_Level10, Profile0_Level45, Profile3_Level45

Apple Music API

AppleMusic TODOs

• Web API (MusicKit) for getting Apple Music artists/albums/playlists, getting user's preferences

• Working with Store Kit to actually access those items

• Playing with Media Player

MusicKit web service

• APIs: artists, albums, songs, playlists, search, charts, recommendations

• Must have a developer token to call

• Must get a user token to access personalized features

https://api.music.apple.com/{version}/catalog/{storefront}/{api}?[params]

Subscription UI

• If user isn't an Apple Music subscriber, you can show the subscription view, optionally showing specific content

• Can embed your affil token

Playback

• MPMediaPlayer

• Can create MPMediaItem (local songs), MPMusicPlayerStoreQueueDescriptor (Apple Music songs), MPMusicPlayerPlayParametersQueueDescriptor (personalized collections)

Changes to Existing Frameworks

AVAudioEngine• Offline Rendering

• Auto shutdown

• Completion callbacks (.dataConsumed, .dataRendered, .dataPlayedBack)

• Important: AUGraph (Audio Toolbox) to be deprecated in 2018

Also

• watchOS recording

• AUAudioUnit "preferred view configuration" API

• Support for FLAC and Opus audio codecs

• Spatial audio formats

HTTP Live Streaming

HLS updates• HLS approved for an actual RFC

• Format war with MPEG-DASH continues

• HEVC supported (of course), fMP4 only (not .ts)

• New subtitle standard ISMC1

• Nice but esoteric features (variables in playlists)

• FairPlay is still very much a thing. 🤐

Beyond Apple Park

ReplayKit

• Screen Capture + Livestreaming API

RTMP HLS

https://github.com/shogo4405/GPUHaishinKit.swift

https://github.com/shogo4405/GPUHaishinKit.swift

Fall Premieres: Media for Apple Platforms in

Late 2017Chris Adamson • @invalidname

CocoaConf Next Door • June, 2017