michał kwiatek it-is1 media archive status 29/10/2007

Michał Kwiatek IT-IS 1

Media Archive Status

29/10/2007

Motivation1. Summarise the past

a. DV MOV 2 AVI conversionb. Recent data migrationc. FLV external encoderd. Recent configuration changese. Recent changes of the Media Archiver softwaref. Test of HD master format

2. Convince you that at present Media Archivecan grow without active help from IT-IS

3. Discuss the futurea. Space requirementsb. WMV playback on CDSc. FLV format on CDSd. HD video master format


Overview

NB: No master file

is encoded if Media Archiver cannot read itsmetadata

3Michał Kwiatek IT-IS

1.The Past


DV MOV to AVI conversion

• How to convert DV MOV files into DV AVI files

• Impossible on a Mac?• Found, bought (~150 CHF) and tested:

FireStore DV Conversion Suite• Converted 39 files provided• It’s been decided to keep both DV MOV

and DV AVI files in the archive• Handed over to the Video team


After 2007-10-10 CERNDATA26A CERNDATA26B CERNDATA27A CERNDATA27B CERNDATA28A CERNDATA28B CERNDATA29A CERNDATA29Bcapacity 1862 2793 1862 2793 1862 2793 1862 2793free 1099 3 707 431 862 1453 777 1088Audio 450 Photo/ConversionSettings 1 Photo/Masters 550 Photo/Public 5 Photo/Public2 1 Photo/Workspace 605 Software 1 Video/Masters/Conferences 1085 1705Video/Masters/Movies 660 1340 0 Video/Masters/Rush 340 Video/ConversionSettings 1 Video/Public 237 Video/Public2 622 Video/TestMasters 62 Video/TestSlaves 5 Video/Workspace 2790 TempVideoWorkspace 1740

On 2007-10-02 CERNDATA26A CERNDATA26B CERNDATA27A CERNDATA27B CERNDATA28A CERNDATA28B CERNDATA29A CERNDATA29Bcapacity 1862 2793 1862 2793 1862 2793 1862 2793free 1099 3 707 1053 1240 2453 0 3Audio 450 Photo/ConversionSettings 1 Photo/Masters 550 Photo/Public 5 Photo/Public2 1 Photo/Workspace 605 Software 1 Video/Masters/Conferences 2790Video/Masters/Movies 1862 Video/Masters/Rush 340 Video/ConversionSettings 1 Video/Public 237 Video/Public2 622 Video/TestMasters 62 Video/TestSlaves 5 Video/Workspace 2790 TempVideoWorkspace 1740

Video Data Migration 03.10.2007


Legend:ok, don't touchcan put more dataneed to free some spacemoved from tosplit betweenvideo quota ownersaudio quota ownersphoto quota ownersoperations quota owners

2008 growth estimates Video/Masters/Movies 2000Video/Masters/Conferences 1000Video/Masters/Rush 1000Video/Workspace 0TempVideoWorkspace 0Video/Public2 200 Total 4200

Recent configuration changes

• New worker node dedicated to photos

• Retry erroneous slave files after 4 hours instead of 1 day; abandon after 8 days:


Try

4 hours

Try

4 hours

4 hours

Try

4 hours

4 hours

4 hours

Try

...

8 days

Recent changes in the Media Archiver software

• Notify CDS only when all slave files are ready (complex – have to synchronise on master files to be sure of semantic correctness)

• Implemented a “DFS offline” check• Support for external encoders


FLV External Encoder

• We’ve tested ffmpeg.exe to generate FLV videos: ffmpeg.mplayerhq.hu/download.html

• Current statistics: 100% OK(the only 5 errors were related to bad master files)

• IT-UDS has complete control over:– The version of ffmpeg used– How it is invoked– Generation logsMichał Kwiatek IT-IS 9

http://ffmpeg.mplayerhq.hu/download.html

Tests of HD master formats

• We’ve looked at H.264 MPEG-4 codec (AVC)

• We’ve decided it is still too early


2.The Present


Formats supported by Media Archiver

Master formats

Video Audio Photo

DV AVI WAVE JPEG

MPEG-2 DVD-quality


Slave formats

Video Audio Photo

WMV WMA JPEG

JPEG MP3

? FLV ? OGG

Formats stored in Media Archive


Master formats

Video Audio Photo

DV AVI WAVE JPEG

MPEG-2 DVD-quality

TIFF

HDV MOV RAW

DVCProHD MOV

DV MOV

Current funtional needs are satisfied

Media Archive interventions

• More space needed

• Media Archiver does not work according to specification


• Metadata not present or not found

• Bad master file

• Specification changes

IT-IS IT-

UDS

Current statistics

• Photo:– ERROR 1737– INSYNC 17425

• Audio:– INSYNC 543– NEW 1

• Video:– BUSY 3– ERROR 4– INSYNC 563– MSTRCHD 19


3.The future


Space requirements in 2008


Video/Masters/Movies 2000

Video/Masters/Conferences 1000

Video/Masters/Rush 1000

Video/Workspace 0

TempVideoWorkspace 0

Video/Public2 200

Total 4200

WMV playback on CDS

• Playback windows should be bigger• Higher resolution file should be used• Streaming server should be used

instead of the HTTP server

http://it-multimedia.web.cern.ch/it-multimedia/test/webcast/templates/Artificial_Intelligence/test/default_CDS_WMV_vs_improved_WMV_4_3.htm





FLV on CDS

• Let’s see how it works:

http://it-multimedia.web.cern.ch/it-multimedia/test/webcast/templates/Artificial_Intelligence/test/improved_WMV_vs_flash_4_3.htm

• Implemented as external encoder, no support from IT-IS for the encoder





External Encoder Requirements

• It supports all current (and future!) master formats

• It is reliable:– Always works for valid files (and returns

0)– Returns non-zero for invalid files– Does not get stuck


(Minor) problem with ffmpeg


Future master format requirements

• be supported and easy to read,• have good chances to stay supported and

easy to read in the future,• be recognised by external customers of the

archive: television companies (for video files) and newspapers and publishing houses (for photo files),

• be the format in which the teams work and save their work,

• be easy to process automatically with the available, well-supported tools


H.264 requirements for WMV

• directshow filters needed:– mp4 splitter - takes the files and splits

the video and audio into two – H.264 codec


4.Discussion


michał kwiatek it-is1 media archive status 29/10/2007

Documents