challenges of quality control for file-based video · 2017-08-07 · content services - mezzanine...

Technical Brief

Challenges of Quality Control forFile-Based Video Storing and manipulating compressed, file-based video/audio affords tremendous speed

and savings but Operations is discovering how difficult it is to ensure correctly repurposed

files are reaching broadcast and content-on-demand audiences. When formats and files

accumulate, automated Quality Control (QC) and other workflow integration techniques

can improve efficiency. There are many questions to ponder as you think about your own

file-based quality control strategy.

Challenges of Quality Control for File-Based Video Technical Brief

2 www.tektronix.com/cerify

Opportunities or Considerations for Monetization Your cost?

The cost of missed commercials - both refunds and ‘free’ replacement

The ‘brand cost’ of transmitting poor quality content

The cost of dead air time

The opportunity cost of answering/fixing problems after they have occurred -

often involving staff management time if an important content supplier is involved

Quantifying the costs of lost subscribers

Reducing the churn rate in digital mastering

Giving refunds for a download with poor quality

Reducing the number of people in the workflow

Reducing end to end file time

The cost of rejected content

Whether you are a Broadcaster, Content Services Provider or Content Aggregator, today you have access to more file-based digital content than ever before. Format requirementsfor new delivery mechanisms are limited only by the audiencesyou choose to serve. Archives are growing every day andstandards that you may need to soon support are still evolving.

Factors that could help differentiate one provider from another will likely include the perception of quality that theconsumer sees.

Ensuring and optimizing file-based content quality meansevaluating and reacting to media quality inside your networkand understanding the impact your content can have onother elements in the ecosystem as well. It not just the audio,video and metadata that matter now, format and syntax are critical.

Throughout the video delivery chain, participants are evolvingtheir workflow to support all digital environments. But becausemore and more content is compressed and archived in oneformat and then re-purposed to another format, archives areanything but homogenous, and working to maintain controlover your facility s output can be more challenging than ever.

This white paper examines the key challenges facing managerswho deal with the quality control of file-based video.

Can I still get away with simply spot-checking selected content?As you will read, there are many reasons to check file-basedquality - at a number of stages. Even if you begin with highquality video, compression and transcode failures at any pointmay cause transfers to stall, set top decoders to crash oreven dead air. This can be costly. While every business is different, please consider your own situation and assign a dollar figure for each category to the right.

Key Concepts

Many Operations Managers feel that the explosion of video files, format

requirements and delivery choices is becoming unmanageable.

Unblinking visual QC of incoming content is not comprehensive, fast,

or scalable and has consequences for missing something important.

A formalized content agreement between suppliers can reduce costly

churn (rework of content) and often push the QC process upstream,

saving you additional time and money.

Automated, file-based QC can trap errors that humans cannot even

see, scale with content growth, increase service quality, and get better

leverage out of existing QC staff.

Today, is visual inspection enough?In the past, the most common approach to QC was to havea small staff of people visually review the content (visualinspection). Even with a waveform monitor, these checks are,well visual and subjective. Realistically, you should onlyexpect your QC staff to be able to see two main categories of technical impairments:

Analog parameters of signal levels, like luma and chroma levels.

Quality levels like black sequences, freeze frame, blockiness, loss of audio, video and audio playtime.

This visual inspection approach has proven effective whenreviewing relatively small volumes of video content. Butregardless of the strength of your QC staff, there are humanfactors to be considered during visual inspection:

Visual and audio errors are easily missed, just by blinkingor losing concentration for a second.

Reviewers have a range of skill levels, experience andtraining which results in considerable differences amongerrors found by different observers.

Staying objective is very difficult, especially over long periods of time, even while viewing similar content.

Some content may have special considerations (e.g. adult entertainment).

It is tiring to guarantee human visual inspection, day after day, week after week.

Equipment used in visual inspection may differ by QC station or site, leading to inconsistent results.


3www.tektronix.com/cerify

MPEG stream errors.



What about the crucial elements that aperson cannot detect?A human cannot look inside the file at the details. Only byanalyzing each file with automated systems designed to do so, can you detect the kinds of problems that occur in file-based video. These can include:

Incorrect play time — measured with frame accuracy.

Putting the audio on the correct channels (or omitted altogether).

The wrong format of the content has been provided.

Incorrect stream setup (e.g. three seconds of audio silenceis required at the start but is not present).

Compliance to various industry de-facto standards. Thestream is correct and legal, but still not what the clientneeds (e.g. H.264 instead of MPEG2).

Missing required data for closed captioning.

Transport Stream and multiplexing errors.

Missing metadata used by an automation system.

Incorrect bit rate for the video or audio.

Encoding quality errors, where the encoder produces a series of blocky video frames.

MPEG encoding syntax errors, which can occur due to multiple mux/de-mux operations, or anencoder/transcoder blip.

Errors in the syntax of the video and audio elementary streams.

Any one of these items could catastrophically impact thequality of what the viewer sees and hears — or doesn t seeand hear.

Of course, there are areas to check for which humans areessential - checking for inappropriate content, as an example.However, if all the technical aspects are good, this checkingcan almost invariably be done either on a quick sample basis ( is it really this movie ) or a very fast 10x speed scanthrough to quickly find any scenes which might require further attention.

What errors are hiding on the server?



How will you manage the transition todigital with lots of new formats?As broadcasters, content services providers and contentaggregators exchange more and more content, ContentInterchange workflows have emerged. There are a variety ofQC approaches, depending on the workflow. What sufficedfor QC in a tape-based workflow (i.e. simply viewing the tape)will not be enough in a file-based workflow.

Even the simple act of viewing a compressed file requiresdecoding. After decoding to baseband, whether or not problems are detected, whether any amount of external correction may have been applied; the audio and video must be re-encoded. There is a great chance that thisprocess will introduce errors:

The file must be recompressed to the same video standard MPEG-2, MPEG-4/AVC, VC-1, etc.Alternately, any transcoding must be done with care.

It must keep the same parameters, which are sometimesset manually over a range of frames to get the optimumappearance.

Software-based transcoders may introduce freeze framesor skipped frames to meet strict bitrate budgets.

The compressed video will need to be re-multiplexed withthe correct audio and metadata.

The metadata might need to be updated to reflect anychanges or editing that occurred.

The point is that as facilities rely more on file-based videosources, it becomes even more important to be sure thatwhat is stored will be useable when it comes time for playback. Figure 1 shows a Content Interchange workflow.Let s examine how three different companies exchange content, and their typical approach to QC today withoutautomated verification.

Automation System

Source

Digital

Tape

Live Event

Transcode

Quality Control & Monitoring

Human?

Spot Check?

No QC?

Monitor?

Archive

StorageContentDelivery

Playout

Download

Stream

Disc Media

CustomerUpload

(ftp)

NearLine

Ingest

Capture

Conversion

Transcode & Repurpose

Digital Rights

Management

Figure 1. Content Interchange Workflow.



Figure 2. Content Services - Content Interchange Workflow.

Content Services - Mezzanine Ingest

Video at a Content Service Provider may follow two or morepaths through the steps the workflow.

In the example in Figure 2, the input source might be a Tape.It passes through Ingest and a mezzanine level (high bit ratedigital master like 50 M/bps MPEG-2) is created. Next, the file is Transcoded to an end services platform (format) for aclient. There might have been visual inspection at the initial

ingest process, which is often watching for tape hits on theencode process, but the visual inspection does not showwhat the encoder is doing. Is the tape being captured correctly? Has the encoder been correctly configured for the tape format? Regardless, the file is placed intoNearline Storage.

Automation System

Digital

Tape

Transcode

Ingest

Capture

Conversion

Live Event

SourceQuality

Control & Monitoring

Human?

Spot Check?

No QC?

Monitor?


Playout

Download

Stream

Disc Media


CustomerUpload

(ftp)

Archive

NearLine

Digital Rights

Management



Figure 3. Content Services - Content Interchange Workflow.

In a second path, shown in Figure 3 the input source mightbe a Digital File. The file is Transcoded, and visual inspectionis a spot check of tops and tails (the beginning and the end).The file is Uploaded by FTP to the client.

The growth in content volume leads to transcoding more and more files. This change in transcoding volume has adirect impact on the QC strategy. Do you have time to QCeverything at each step?

A basic assumption is that when ingesting the original tape,there was 100% QC (via visual inspection) and you caught allthe errors. If you don t catch the errors, the errors will be in allof the transcodes. Then it s up to the spot check to catchthese errors. This can be an expensive process if you catch

the errors at the end - it may be too late to re-ingest the original source. 100% QC at initial ingest is necessary to prevent the ripple of faulty content downstream. By the time that you get to the repurposing process, the only errorsshould be the ones introduced by the transcode process.

Being forced to repeat the digital mastering process is oftenreferred to as churn , and it truly is the cost of failure. Goingthrough this process once, taking a tape to digital file, couldinclude a standards conversion, adding letterboxing, andadding close captioning. Depending upon the cost and timepressure of having the correct file, the number of times thatthe digital mastering process is repeated could make the difference between making and losing money.

Digital Rights

Management

Automation System

Digital

Tape

Transcode

Ingest

Capture

Conversion

Live Event

SourceQuality


Human?

Spot Check?

No QC?

Monitor?

Archive


Playout

Download

Stream

Disc Media

NearLine


CustomerUpload

(ftp)



Figure 4. Content Aggregator — Content Interchange Workflow.

Content Aggregator – Multiple Transcodes and Streaming

In the example show in Figure 4, the Source is a Digital file(perhaps transferred in by FTP), it is Transcoded, possiblyincluding a standards conversion (e.g from HD to SD), the QCis 100% visual inspection and file might go through the QC-back-to-transcode loop several times in order to achieve the

required quality. This is because running at such low bit rates, transcodes often don t work perfectly the first time, and especially on fast action sequences. When it looks goodenough visually, it moves to the last step of Content Deliveryin this case a Stream for playout.

Digital Rights

Management

Automation System

Digital

Tape

Transcode

Stream

Ingest

Capture

Conversion


Live Event

SourceQuality


Human?

Spot Check?

No QC?

Monitor?

Archive


Playout

Download

Disc Media

NearLine

CustomerUpload

(ftp)



Figure 5. Broadcaster — Content Interchange Workflow.

Broadcaster – Multiple File Movements

In Figure 5, the source is a Tape or Live Event, which isIngested to a mezzanine format (like MXF MPEG-2). Clearlythe biggest challenge is the ingest process. There is typicallyno human QC. Having to go through the ingest process againon a live event may be impossible. Having a QC strategy that tells you that you either got it right the first time, or what exactly needs to be addressed, is critical. Lastly the

file is sent to Near Line storage and from there to Playout. Often, the file is moved from Near Line or the air server back to archive if it is not to be used again within a certaintime frame.

In all these examples, doing only visual QC, and just onceduring the processes, leaves the door open to costly reworkor make-goods downstream.

Automation System

Digital

Tape

Transcode

Ingest

Capture

Conversion

Live Event

SourceQuality


Human?

Spot Check?

No QC?

Monitor?

Archive


Playout

Download

Stream

Disc Media

NearLine


CustomerUpload

(ftp)

Digital Rights

Management



How will you manage the expansion of your business and maintain yourreputation for quality? Broadcast Operations are reaching critical mass. The volumeof video is multiplying as business units continue to re-formatand re-purpose video for new revenue streams. Some broadcasters say they are exponentially growing content while only linearly growing QC. But faced with a large growthin channels and services, scaling your QC strategy with the increase in content volume can become difficult. Somepotential strategies are:

Scale down from full QC of all material to perhaps viewing the beginning, middle and end of programs or spot checks.

Check one program out of ten (sampling).

Leave the checking to the next consumer.

How do you monitor the quality of many new channels when there are different formats and quality levels required for terrestrial, satellite, cable, VoD, and IPTV? Once youdecode and re-encode to a different format, how do youmake sure that the quality remains intact? Have you found

a way to check each different version required for SD/HD, for internal archives, third party licensees, and internationalframe rate versions? There are a variety of other factors thathave an impact on your brand quality as you expand yourbusiness. Here are some examples of real world challenges:

Repurposing - A music channel is straining to fully check only its high bit-rate encoding of incoming master (mezzanine)files, but has not yet found a way to check each different version required for their internal archives, third partylicensees, several international versions, VoD, etc.

Time - Sometimes, there is just not enough time. A majorlate-night talk show must be edited and reformatted to be onsyndication servers and third party platforms by the very earlymorning. Even if you can still use people for checking video,you may not be able to hire the right talent at the right time of day.

Another very popular network show is anticipated by its viewers each week. However, for reasons of security, the program is not given to the network until two hours beforeairing. Again, this must be repurposed to their website andother VoD networks within 12 hours.



Scalability - Broadcasters are both centralizing equipmentand decentralizing QC. This means, that they want all QC hardware centralized to control costs, but wish to decentralize the place where the work can be done. Thisfrees up expensive real-estate and allows more freedom in contracting out QC. Content Services companies have work groups all over the world — decentralizing is the key.

Automation System Integration - Automation systems are constantly improving their ability to track and move filesfrom ingest to playout. QC workflow integration with assetmanagement systems will help leverage your investment andfacilitate scaling — while maintaining consistent quality levels.

Interoperability - How do you make sure that all of theencoding and mastering equipment in your company all have the same configuration? There is always a need to monitor equipment configurations like the settings onencoders and decoders.

Establishing a new content vendor - It may take months to get the digital mastering correct for a new VoD system or web platform. How do you test your content before it isrejected while on-line?

Who would you rather do businesswith? Someone who guarantees filesmeet a specification or someone whorequires you to test each one yourself? Due to the amount of new and repurposed content, ContentInterchange continues to accelerate. Communicating anddocumenting your requirements for file content requirementsbetween content providers and content users can be difficult.

You may already embrace the concept of a ContentConformance Agreement (CCA), but you may call it by another name. Chances are that you don t have it writtendown anywhere — just a verbal agreement. You may have

several; one for each client. Unless you have an automatedsystem, how do you enforce this? It is not possible to enforcea subjective agreement — especially if there are elements in anagreement that will be missed in visual inspection. It has tobe non-subjective. You have to be able to reject it and push it back if there is ever a disagreement.

Ideally, checking the file against an agreed upon CCA wouldallow you to have an automated content filter to evaluate yourincoming content. Table One gives an example of the contentparameters in a CCA for correct file configuration and qualityof a feature-length movie for full format VoD.

Using a CCA could make the difference between contentbeing accepted or rejected both upstream and downstream.

Table 1. CCA for a feature-length movie for full format VoD.

Category CCA Parameter

Video Standard MPEG-2

Profile & Level Main

Play Time Greater than 60 min

Horizontal & Vertical Resolution 720/480

Frame Rate 29.97 fps

Bit Rate 3 – 3.5 Mbps

Display/Aspect Ratio 4:3

Color Depth 4:2:0

Black frames at start, Min 2s black at start;

end or during video Min 2s black at end

Letterbox and Pillarbox checks Disallowed

Blockiness Not greater than 75%

Luma Limit Violation None



The Solution: Tektronix Cerify¤

The industry choice for automated QualityControl of file-based video.

The Tektronix Cerify is a fully automated system for verifyingand checking file based content. Cerify is integrated into a network, and automatically checks the correctness of file-based content against defined standards, at many stages.

The multiple levels of testing mean your content:

Will play: it is syntactically correct and correctly encoded

Can be transmitted: the transmission parameters are correct

Is technically legal: gamut, audio range etc. are within limits

Is structured correctly: audio on the right channels, correct length, etc.

Quality is good: video is not blocky, audio is not clipped

A summary the operational benefits of Cerify is listed in Table 2.

Figure 6. Content Workflow with automated testing at multiple stages.

Automation System

Source

Digital

Tape

Transcode Archive

StorageTranscode & Repurpose

Quality Control & Monitoring

ContentDelivery

Playout

Download

Stream

Disc Media

NearLine

Ingest

Capture

Conversion

Live Event

CustomerUpload

(ftp)

Digital Rights

Management



Operations Challenge Cerify Attribute Cerify Operations Benefit

Verifying impairments that QC staff can’t see. Automated file verification with syntax checking Now you have an insurance policy against

Insuring repeatability in the content QC process. ‘dead' air time. Cerify works unattended 24/7

Content has security considerations. to perform over 800 tests on all the invisible

Content has short turn around time before playout. aspects of compressed video/audio files.

Automated QC allows you to make your QC

staff more efficient by having them concentrate

on the 5% of the “bad” files, rather than

spending their time watching the 95% of

“good” files.

Verifying that the content meets the requirements Content filter templates Establishing a Content Conformance Agreement

for correct file configuration and quality. that is service content specific reduces rejected

content and churn rate. Consistently audit the

performance of your content suppliers.

Insuring that all of the encoding and mastering Content filter templates A content filter makes it easy to monitor the

equipment has the same configuration. interoperability of equipment configurations

like settings on encoders and decoders.

Will automated QC integrate with my existing System Integration Cerify speaks to existing automation systems

automation system and server network? via Ceri-Talk™. This allows the automation

system to automatically control processing of

files by Cerify.

The Cerify Developer Community insures that

Cerify can have a seamless QC workflow

between major server, encoder and automation

system manufacturers.

Reducing churn rate – determining why the Audit trail Cerify logs error detail as the content is

content was rejected. checked. Diagnose and troubleshoot errors

down to the frame level.

Scaling the QC staff. Scalability Start with one Cerify unit and analyze up to 4

files in 2X real time. Grow your system by

clustering three or more Cerify units to match

the required throughput and still allow for

future growth.

I need to centralize my digital mastering Accessibility The Cerify GUI is web-based so multiple users

and decentralize my QC. at the same site, or at different sites, can

access the testing and results using a standard

web browser.

Table 2. Operational benefits of Tektronix Cerify.



ConclusionWhile the interchange of content continues to grow exponentially, visually inspecting program content fails toidentify costly problems. In fact, visual inspection of incomingfile-based video content as a means of Quality Control is notcomprehensive, fast or scalable.

Server based, automated file verification provides a contentfilter that can catch the errors that people would normallymiss, and gives you a way to uniformly check the conformanceof the content that you accept. A specific content filter, withyour unique program requirements can be used to establish a Content Conformance Agreement (CCA) with content suppliers and customers. Documented CCA results canreduce rejected content, create an audit trail, and increasethe quality of content viewed by the consumer.

Workflow integration is not painful. You do not need thatchange everything that you do. Using automated file-basedQC solutions can reduce your end-to-end file time, reduce the number of people in the workflow, and reduce churn byhaving your content accepted the first time.

Automated Quality Control can scale with content growth,increase service quality, and get better leverage out of existingQC staff.

Call a Cerify Systems Specialist today to find out how fastCerify can pay back your investment. Call your local accountmanager or e-mail [email protected]

For Further Reading:www.tektronix.com/cerify

For Further InformationTektronix maintains a comprehensive, constantly expandingcollection of application notes, technical briefs and otherresources to help engineers working on the cutting edge oftechnology. Please visit www.tektronix.com

Copyright ' 2008, Tektronix. All rights reserved. Tektronix products are covered by U.S. and foreign patents, issued and pending. Information in this publicationsupersedes that in all previously published material. Specification and pricechange privileges reserved. TEKTRONIX and TEK are registered trademarks of Tektronix, Inc. All other trade names referenced are the service marks, trademarks or registered trademarks of their respective companies. 08/08 EA/WOW 2NW- 21358-0

Contact Tektronix:ASEAN / Australasia (65) 6356 3900

Austria +41 52 675 3777

Balkans, Israel, South Africa and other ISE Countries +41 52 675 3777

Belgium 07 81 60166

Brazil (55) 40669400

Canada 1 (800) 661-5625

Central East Europe, Ukraine and the Baltics +41 52 675 3777

Central Europe & Greece +41 52 675 3777

Denmark +45 80 88 1401

Finland +41 52 675 3777

France +33 (0) 1 69 86 81 81

Germany +49 (221) 94 77 400

Hong Kong (852) 2585-6688

India (91) 80-22275577

Italy +39 (02) 25086 1

Japan 81 (3) 6714-3010

Luxembourg +44 (0) 1344 392400

Mexico, Central, South America and Caribbean 52 (55) 54247900

Middle East, Asia and North Africa +41 52 675 3777

The Netherlands 090 02 021797

Norway 800 16098

People’s Republic of China 86 (10) 6235 1230

Poland +41 52 675 3777

Portugal 80 08 12370

Republic of Korea 82 (2) 6917-5000

Russia & CIS +7 (495) 7484900

South Africa +27 11 206 8360

Spain (+34) 901 988 054

Sweden 020 08 80371

Switzerland +41 52 675 3777

Taiwan 886 (2) 2722-9622

United Kingdom & Eire +44 (0) 1344 392400

USA 1 (800) 426-2200

For other areas contact Tektronix, Inc. at: 1 (503) 627-7111

challenges of quality control for file-based video · 2017-08-07 · content services - mezzanine...

Documents