Michael Naimark Visiting Associate Arts Professor

Interactive Media Arts NYU Shanghai

VR / AR FundamentalsINTM-SHU  280C - 1   Topics in New Media & Entertainment Fall 2017

a practical guide to the big ideas

Week 8: Production Intro

Your Mission: Exploring (real) people close-up in (artificial) constructed environments

VR Experience Lab — Layout & Proposal — Instructions — Downloads — Hosting

Ongoing Assignment — 1 new title reviewed, tagged, and rated per week — DUE ON WEDNESDAYS — Use Resources for Links & Leads (I will)

VR / AR Fundamentals

But First

VR / AR Fundamentals

Production

Your Mission: Exploring (real) people close-up in (artificial) constructed environments

WHY:

— Close-up “intimate” people imagery in VR is awesome — Close-up “intimate” people imagery in VR is hard — Using 360 opens up new possibilities — Between Cinematic VR / CGI Games

VR / AR Fundamentals

Exploring (real) people close-up in (artificial) constructed environments

BASICS: — Real people shot with cameras as close as 1 meter away — Must be stereoscopic or volumetric — Must have spatial sound

— Artificially over or under populated foreground — Artificially modified or replaced background — CGI foreground or background components

— Shot with stationary camera — People mic’ed individually

— Synced movement and speech — Processing in video and/or CGI

— Interactive possibilities

VR / AR Fundamentals

Exploring (real) people close-up in (artificial) constructed environments

WORKFLOW: Capture: — Needs to be stereo or volumetric (not necessarily 360, can be composted later) — High-end VR 360 camera like Jaunt or Z Cam V1 Pro or Insta360 Pro — Stereo camera or stereo pair of cameras (Zed Cam) — Kinect + DSLR (DepthKit) — Tango or equivalent — Audio: mic’ing people individually + spatial ambience capture

Processing: — Video compositing (e.g., After Effects) — Depth maps (Mettle) — Point clouds and game engines (Unity? Unreal?) — Spatial audio

User Interaction: — Selective compositing — Branching

Examples

VR / AR Fundamentals

Exploring (real) people close-up in (artificial) constructed environments

1816 x 908

(Ring around the Rosie silent palendrome) “Loop back and forth” in Keynote

“Credible” hyper-image composite. Everyone was shot separately. Soft looping for ambient movement.

(Don’t Be Evil.) “Loop” in Keynote

“Incredible” hyper-image composite. [audio]

1816 x 908

(Ring around the Rosie) Jim

Solos minus fall Chorus with fall

freeze (I’ll FTB in Keynote)

Everyone was mic’ed and shot separately. Various spatial audio mixes. [audio]

1816 x 908

http://medium.com/@michaelnaimark/vr-cinematography-studies-for-google-8a2681317b3

VR / AR Fundamentals

Exploring (real) people close-up in (artificial) constructed environments

TASKS / TWO-DAY RESEARCH:

— High-End cameras (Michael) — Cameras, Mics, Lighting, and Space @IMA / NYUSH — Zed Cam — Depth Kit — Mettle — Digital compositing — Spatial Audio processing — Moving video/point clouds into game engines — Moving spatial audio into game engines — Selective compositing and branching in game engines

— Everyone else: find examples of — — people close-ups in VR titles — — hybrid video/CGI in VR titles