skyfire log files100411

Skyfire Rocket Logs Guidance and Outline

Robert Oberhofer October 4, 2011

Skyfire Labs, Inc. 779 E. Evelyn Ave.

Sunnyvale, CA 94041

© 2011 Skyfire Labs, Inc

Copyright and Trademarks The information in this document is subject to change without notice and describes only the product defined in the introduction of this documentation. This document is not an official customer document and does not take responsibility for any errors or omissions in this document. This document is intended for the use of Skyfire Labs Inc.(Skyfire) personnel and their customers. No part of this documentation may be used, reproduced, modified or transmitted in any form or means without the prior written permission of Skyfire.

The information or statements given in this documentation concerning the suitability, capacity or performance of the mentioned hardware or software products are given “as is” and all liability arising in connection with such hardware or software products shall be defined conclusively and finally in a separate agreement between Skyfire and the customer.

IN NO EVENT WILL SKYFIRE BE LIABLE FOR ERRORS IN THIS DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS INTERRUPTION, BUSINESS OPPORTUNITY OR DATA, that might arise from the use of this document or the information in it.

THE CONTENTS OF THIS DOCUMENT ARE PROVIDED "AS IS". EXCEPT AS REQUIRED BY APPLICABLE MANDATORY LAW, NO WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT, ARE MADE IN RELATION TO THE ACCURACY, RELIABILITY OR CONTENTS OF THIS DOCUMENT. SKYFIRE RESERVES THE RIGHT TO REVISE THIS DOCUMENT OR WITHDRAW IT AT ANY TIME WITHOUT PRIOR NOTICE.

This document and the product it describes are considered protected by copyrights and other intellectual property rights according to the applicable laws.

Copyright © Skyfire Labs, Inc. 2011. All rights reserved.

© 2011 Skyfire Labs, Inc PAGE 3

Contents

Contents

Overview 5

Generic Log Elements 6

Rocket Controller Log 7 LogMetaDataStats ...................................................................................................................... 7 MP4 MetaData .......................................................................................................................................... 8 FLV MetaData 9 .......................................................................................................................................... VideoDetectionStats ................................................................................................................... 11 Field Definitions 11 ........................................................................................................................................

Rocket Optimizer Log 14 VideoTranscoderStats ................................................................................................................ 14 Field Definitions 14 ........................................................................................................................................ VideoServerStats ....................................................................................................................... 17 Field Descriptions .................................................................................................................................... 18

Overview


Overview The Skyfire Rocket solution has two main components:

Rocket Controller

Rocket Optimizer

The main functions of the Rocket Controller is to:

Provide deep video packet inspection capability

Pre-determine whether a video can and will be transcoded

Handle Adaptive Bitrate Video requests

Determine the video optimiztion profile that applies for a specific video and flow and inform the optimzier of the respective parameters.

Maintain state of ongoing video transactions

Provide video related logging information – even if the video is not optimized/transcoded.

The main function of the Rocket Optimizer is to:

Instant, Real-time Transcode the video

Thus with regards to logging it will be necessary to review and correlate information from both the Optimizer and the Controller to get a comprehensive picture of video activity (whether optimized or not).

The log files for the Controller and Optimizer can be found on each server under:

/var/log/SkyController.log

/var/log/SkyOtpimizer.log


Generic Log Elements

Generic Log Elements The following is an example of a log line:

[09:08:2011:13:56:14.394233] 0.1 SkyController qa-ctlr004 !NOTIFY [3225-

1169942848] ../src/Common/MP4Utils.cpp:236 LogMetaDataStats():236 lc=6,

MetaData: MetaDataFormat=MP4, MetaDataFound=true, creation_time=3303238425,

duration=38998, modification_time=3303238425, timescale=600, volume=256

It contains the following generic elements common to all log lines:

Date and Time: [09:08:2011:13:56:14.394233]

Log Version: 0.1

Process creating the log file: SkyController

Server name: qa-ctlr004

Log Level of the entry: !NOTIFY

Unique ID (TBD): [3225-1169942848]

Source File & line generic entry: ../src/Common/MP4Utils.cpp:236

Type of Entry / Function: LogMetaDataStats()

Rocket Controller Log


Rocket Controller Log By default, the Rocket Controller will be run in production with log level set to !Notify.

When set to the !Notify level, all messages of level !Notify, !WARN and !Error will be visible.

For analytics purposes, the following are the two main log lines of interest:

LogMetaDataStats

VideoDetectionStats

The purpose of ‘LogMetaDataStats’ is to provide Meta Data contained in the headers of a specific video file. This data is useful for videos that might not be transcoded.

The purpose of ‘VideoDetectionStats’ is to inform about the decision process within the Controller.

LogMetaDataStats Metadata may or may not be present in the log file. The system merely attempts to capture the metadata found.

The following two examples are log lines for a LogMetaDataStats event:






1232881984] ../src/Common/FLVUtils.cpp:584 LogMetaDataStats():584 lc=21,

MetaData: MetaDataFormat=FLV, MetaDataFound=true, duration=180.768000,

width=512.000000, height=288.000000, videodatarate=495.000000,

framerate=30.000000, videocodecid=7.000000, audiodatarate=64.000000,

audiosamplerate=2097152000.000000, audiosamplesize=16.000000, stereo=true,

audiocodecid=10.000000, filesize=12622924.000000

As can be seen these entries, depending on the type of video file processed, the metadata stats will contain very different levels of detail.

MetaDataFormat=MP4

This field indicates that the source file format is MP4, which is a complex format. Because MP4 is complex, the Controller does not always have enough data available to complete a full inspection of all the headers contained within. Thus the meta-data available might be limited.

NOTE For MP4 ,the Optimizer has more detailed information available in its log files.



MetaDataFormat=FLV This field indicates that the source file is FLV. FLV stands for “FLash Video”, and is used only for video that has an Adobe Flash video player as its destination. FLV has a simpler metadata structure than MP4, which allows the Controller to provide full more metadata detail.

NOTE Flash Video players can also support MP4 video format.

MP4 MetaData

The MP4 MetaData log entry contains the following information elements:





Field Definitions Field MetaDataFound

Description Should always be true, otherwise there will be no log entry.

Value(s) True | False

Field: creation_time

Description The timestamp from when the video file was created by the publisher.

Value(s): Time in ms from 1 Jan 1970 (TBD)

Field: Duration

Description Length of the video in ms

Value(s): Integer

Field: modification_time

Description Timestamp of the last time the video has been modified. Normally this is the same time as creation_time.

Value(s): Time in ms from 1 Jan 1970 (TBD)

Field: Timescale

Description TBD

Value(s): Integer



Field: Volume

Description Volume of the video. Max = 256.

Value(s): Integer from 1 to 256

FLV MetaData

The FLV MetaData log entry contains the following information elements:

[09:16:2011:00:37:37.912094] 0.1 SkyController vzw-flv003 !NOTIFY [6875-

1099671872] ../src/Common/FLVUtils.cpp:544 LogMetaDataStats():544

lc=863620, MetaData: MetaDataFormat=FLV, MetaDataFound=true,

duration=10.449000, width=240.000000, height=160.000000,

videodatarate=250.000000, framerate=12.000000, videocodecid=2.000000,

audiodatarate=31.250000, audiosamplerate=11025.000000,

audiosamplesize=16.000000, stereo=true, audiocodecid=2.000000,

filesize=392352.000000

Field: MetaDataFound

Description Should always be true

Value(s): True

Field: Duration

Description Duration in seconds and fraction of seconds

Value(s): Decimal string

Field: Width

Description Width of video in pixels

Value(s): Integer

Field: Height

Description Height if video in pixels

Value(s): Integer

Field: Videodatarate

Description Bitrate of the video substream in kbps




Field: Framerate

Description Frames per second


Field: videocodecid

Description ID of the used codec in the format TBD XXX

Value(s): Integer

Field: Audiodatarate

Description Bitrate of the audio substream in kbps

Value(s): Decimal String

Field: audiosamplerate

Description Same rate of the audio substream in Hz. The previous sample code suggests that the metadata was incorrect.


Field: audiosamplesize

Description Number of bits per sample.


Field: stereo



Field: audiocodecid



Field: filesize





VideoDetectionStats These message lines give key information regarding video detection.

The following is an example of a VideoDetectionStats log line:

[09:20:2011:12:27:34.200701] 0.1 SkyController sfo-ctlr002 !NOTIFY [17178-

1198274880] ../src/SkyController/VideoAnalysis.cpp:1201 OnDataReceived():

1201 lc=6838, VideoDetectionStats:

sessionID=5cbb91d0f69e4e22aefae1e7791fc3d5, contentType=video%2fx-flv,

srcFileFmt=FLV, Player=Flash, srcVideoCodec=On2VP6, srcAudioCodec=Unknown,

transcoderAvail=true, srcBitrate=909547, bitrateThreshold=773000,

bitrateTarget=350000, minOptimizationTest=passed, srcDuration=1.79s,

srcSize=203625, bypass=false,

redirectURL=http%3a%2f%2f209.118.181.135%3a4243%2fskyRequest_5c1d62ea-f592-

4322-abdd-

5c10e1f76486%2fskyWidth_480%2fskyHeight_320%2fskyBitrate_350%2fskyMinComp_15

%2fskyABRMinComp_30%2fskyMinCompTest_1%2fskyTranscode_1%2fskyOuttype_flv%2fs

kyCookie_%2fskyStatsId_sfo-

ctlr002%2fskySessionId_5cbb91d0f69e4e22aefae1e7791fc3d5%2fskyVideo_l.yimg.co

m%252fcv%252feng%252ffox%252f110920%252ff%252fxfactor_to_1000.flv,

srcURL=http%3a%2f%2fl.yimg.com%2fcv%2feng%2ffox%2f110920%2ff%2fxfactor_to_10

00.flv,

UA=Mozilla%2f5.0%20(compatible%3b%20MSIE%209.0%3b%20Windows%20NT%206.0%3b%20

Trident%2f5.0%3b%20FunWebProducts)

Field Definitions

Field: sessionID

Definition The unique identifier as programmed in the Controller API's flow ID field

Value(s): Customer defined variable.

Field: contentType

Definition The HTTP content type of the payload

Value(s): See the HTTP specification

Field: Player

Definition The player used by the client.

Value(s): Flash | Non-Flash. Note: it is possible for Flash players to render MP4 content, but a Non-Flash player cannot render FLV content.



Field: transcoderAvail

Definition The transcoder health check at the time the video was detected

Value(s): True | False. If true, the transcoder was online If false, the transcoder was offline

Field: minOptimizationTest

Definition The optimization test result.

Value(s):

Passed | Failed | Deferred Passed indicates that video is detected and its format is deemed worthy of optimization, given the current congestion level. Failed indicates that video is detected and its format is deemed not worthy of being optimized, given the current congestion level. Deferred indicates that not enough information was available to make the optimization decision, so the Optimizer makes the decision at a future date.

Field: bypass

Definition Whether or not the video optimization was bypassed.

Value(s):

False | variable False indicates the video will be optimized Variable: Any other value provides a text description of why we do not intend to optimize the video. Typical variable values might be: transcoder_unavailable url-blacklisted min-optimization-failed

Field: redirectURL

Definition When video was found, the client redirected to this URL.

Value(s): Variable URL format

Field: UA

Definition The User Agent of the client application

Value(s): Variable string



When the field M3U8Playlist=true, the following fields are included:

Field: M3U8Type

Definition The type of variant playlist that was found

Value(s):

HLS:variants: HLS:mixed | HLS:unknown An HLS:variant indicates a pure variant playlist is detected HLS:mixed indicates a combined variant playlist and content playlist in a single file is detected HLS:unknown indicates an HLS format is detected, but that no other determination could be made.


Rocket Optimizer Log

Rocket Optimizer Log When conducting analytics for the Optimizer, the two fields of most interest are:

VideoTranscoderStats

VideoServerStats

VideoTranscoderStats This field provides optimization information.

A typical example of this log entry is:

[10:03:2011:16:25:39.758386] 0.1 SkyOptimizerServer qa-opt102.dc.skyfire.com

!NOTIFY [28889-

1119922496] ../src/SkyOptimizerServer/SFTranscoderWrapper.cpp:1270 DumpSt

atistics():1270 lc=21283, inst={82447.82444.82441.8} VideoTranscoderStats:

sessionId=92c2acfc5f234a08bda4c69b711da6d1, statsId=qa-flv002,

srcDuration=80.05s, playTime=79.74s, fetchedInBytes=37882522,

processedInBytes=37882522, outBytes=6241702, optimizationRatio=83.53,

srcDimensions=1920x1080, srcBitrate=3786.07kbps, srcVideoCodec=H.264,

srcAudioCodec=AAC, srcContainer=MP4, outDimensions=852x480,

outBitrate=626.00kbps, outVideoCodec=H.264, outAudioCodec=AAC,

outContainer=FLV, inputFileWaitTime=0.00s, outputFileWaitTime=0.01s,

sourceFileFirstDataTime=0.01s, firstDataInToOut=2.58s, srcURL=qa-

web001.dc.skyfire.com%2fhosted_vids%2fmp4s%2fMain_HiRes_1080P.mp4

Field Definitions

Field: sessionID



Field: statsId

Definition The hostname of the controller that intercepted the video.


Field: srcDuration

Definition The duration in seconds of the original non-transcoded source video.

Value(s): Variable, floating point



Field: playTime

Definition The duration in seconds that the customer actually watched the transcoded video.

Value(s): Variable, floating point.

Field: fetchedInBytes

Definition The total amount of video, in bytes, fetched by the Optimizer.

Value(s): Variable, Integer

Field: processedInBytes

Definition

The optimized portion of the video sent to the client (in outBytes size). The optimizer fetches content ahead of the current position from the internet; in order to calculate the true optimization ratio of the currently delivered content, the following calculation is used:

optimizationRatio = 1- (outBytes/processedInBytes) in %


Field: srcBytes

Definition The size in bytes of the original non-transcoded data in bytes the user received.

Value(s): Variable, integer

Field: outBytes

Definition The amount of output transcoded data in bytes the user received.

Value(s): Variable, integer.

Field: optimizationRatio

Definition The average optimization percentage of video transcoded so far.

Value(s): Variable, integer.

Field: srcDimensions

Definition The dimensions of the original non-transcoded source video.

Value(s): Integer x integer

Field: srcBitrate



Definition The average bitrate in kbps of the original non-transcoded source video.

Value(s): Variable, floating point.

Field: srcVideoCodec

Definition The video codec of the original non-transcoded source video.

Value(s): Variable, free-form string, may be unknown

Field: srcAudioCodec

Definition The audio codec of the original non-transcoded source video.

Value(s): Variable, free-form string, may be unknown

Field: srcContainer

Definition The container of the original non-transcoded source video.

Value(s): Either FLV or MP4

Field: outDimensions

Definition The dimensions of output transcoded data

Value(s): Integer x integer

Field: outBitrate

Definition The average bitrate in kbps of the transcoded video


Field: outVideoCodec

Definition The video codec of the transcoded video.

Value(s): Variable, free-form string, should always be H.264

Field: outAudioCodec

Definition The audio codec of the transcoded video.

Value(s): Variable, free-form string, should always be AAC

Field: outContainer

Definition The container of the transcoded video.



Value(s): FLV | MP4. The field should never be MP if the scrContainer is FLV.

Field: inputFileWaitTime

Definition The time the SkyTranscoderHost spends waiting for input data to transcode. If this number is high, it indicates the Optimizer can transcode faster than the network can feed it data from the origin server..


Field: outputFileWaitTime

Definition The time the SkyTranscoderHost spends waiting to push output data to SkyTranscoderServer.


Field: sourceFileFirstDataTime

Definition

The approximate time from when the server received the request to transcode to the time it received the first packet of source data from the origin server. Unlike the field, inputFileWaitTime, this field does not include mid-download stalls.


Field: firstDataInToOut

Definition

The time from when the server receives the first input data from the origin server to when transcoded sends the first output data to the client. The fields, sourceFileFirstDataTime + firstDatainToOut = the time from the initial request to the first packet of output data.


Field: srcURL

Definition The URL to the original non-transcoded video.

Value(s): Variable, URL

VideoServerStats Typical log line looks like:



[09:27:2011:11:37:54.51804] 0.1 SkyOptimizerServer qa-opt101

!NOTIFY [20970-

1170946368] ../src/Server/HTTPVideoServer.cpp:110 DumpStatistics():110

lc=25, inst={1086.1083.1107.8} VideoServerStats:

sessionId=ff6ff7b4bcac31d5, statsId=qa-flv001, servedVideo=true,

rejectedVideo=false, sourceRequestTimeout=false, dataSentTimeout=false,

transcoderFail=false,

srcURL=server7.rooftopcomedy.com%2fm%2f2010%2f10%2frivera_thai_mexican_or_no

t_HD_DRT_PMSTR_deint.mp4,

UA=Mozilla%2f5.0%20(Windows%20NT%206.1%3b%20WOW64%3b%20rv%3a6.0.2)%20Gecko%2

f20100101%20Firefox%2f6.0.2

Field Descriptions

Field: sessionID



Field: statsId

Definition The hostname of the controller that intercepted the video.


Field: servedVideo

Definition Indicates whether the server served the video or not.

Value(s): True | False. True indicates normal operation; False indicates no transcoded data was sent to the client.

Field: rejectedVideo

Definition Indicates whether or not the server rejected the video

Value(s):

True | False. True indicates the Optimizer rejected the video, triggering a redirect back to the source. This condition usually means the content failed our minimum optimization criteria. It does not indicate a problem with the Skyfire transcoder. Fale indicates normal operation.

Field: sourceRequestTimeout

Definition Indicates whether the server received content from the origin server or not.

Value(s): True | False. True indicates the origin server did not respond to the request for data, indicating a problem at the source. False indicates normal operation.

Field: dataSentTimeout



Definition Indicates whether the transcoder did not send data to the client.

Value(s): True | False. True indicates the transcoder did not provide data to the client, indicating a problem with the Skyfire transcoder. False indicates normal operation.

Field: TranscoderFail

Definition Indicates whether the transcoder detected a failure in the process of converting the video.

Value(s): True | False. True indicates the transcoder was not able to process the original video, indicating a problem with the Skyfire transcoder. False indicates normal operation.

Field: srcURL

Definition The URL to the original non-transcoded video.

Value(s): Variable, URL

skyfire log files100411

Technology

copyright skyfire labs

use of skyfire labs

skyfire rocket solution

skyfire personnel

rocket controller log

rocket optimizer log

skyfire rocket logs

specific video