media extensions
DESCRIPTION
Media extensions. Marcin Stankiewicz and Matthew Howard Development - Multimedia Platform Team Richard Fricks Windows Scenario Adoption Team Session 4-104. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/1.jpg)
Media extensionsMarcin Stankiewicz and Matthew HowardDevelopment - Multimedia Platform TeamRichard FricksWindows Scenario Adoption TeamSession 4-104
![Page 2: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/2.jpg)
Engaging with rich media—whether watching a movie, video chatting, or playing music—is one of the most prevalent and enjoyable things we do on our PCs today.
![Page 3: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/3.jpg)
Drag picture to placeholder or click icon to add
![Page 4: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/4.jpg)
Architecture overviewMedia Foundation pipelineMedia extensionsVisual Studio 2012Media source demoStreaming media to JavaScript
Agenda
![Page 5: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/5.jpg)
Media Application Architecture
Media Foundation
Media enginePlayback and preview
Windows Store app
Capture engineCapture
Sharing enginePlayTo
Transcode
Windows Runtime (WinRT)
C#MediaElement
JavaScript – HTML5Video tag, audio tag
Media namespace• Capture• Transcode• PlayTo• PlaybackManager• MediaExtensionMana
ger
![Page 6: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/6.jpg)
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
Media app architecture
Media enginePlayback and preview
Capture engineCapture
Sharing enginePlay To
Transcode
Media Foundation
Media source
Decoder Effect SinkEncoder
![Page 7: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/7.jpg)
Media app architecture
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipelineMedia source
Decoder Effect SinkEncoder
http://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx
• Can be either native C++ DLL or a native C++/CX DLL• The DLL must ship in your app package• You can call WinRT APIs but you can use ONLY those desktop APIs that are
marked for Windows Store apps:
![Page 8: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/8.jpg)
Registration processMedia source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
Windows.Media.MediaExtensionManager
.RegisterAudioDecoder()
.RegisterVideoDecoder()
.RegisterAudioEncoder()
.RegisterVideoEncoder()
.RegisterSchemeHandler()
.RegisterByteStreamHandler()?
![Page 9: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/9.jpg)
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
Effects are registered using the class your code uses to process the media:MediaElement AddAudioEffect() VideoTag msInsertAudioEffect()
AddVideoEffect() msInsertVideoEffect()
MediaTranscoder AddAudioEffect() AudioTag msInsertAudioEffect()
AddVideoEffect()
MediaCapture AddEffectAsync()
Registration process: Effects
![Page 10: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/10.jpg)
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
Sinks are supported only by MediaCapture:MediaCapture StartRecordToCustomSinkAsync()
StartPreviewToCustomSinkAsync()
Registration process: Sinks
![Page 11: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/11.jpg)
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
Windows.Media.MediaExtensionManager
.RegisterAudioDecoder()
.RegisterVideoDecoder()
.RegisterAudioEncoder()
.RegisterVideoEncoder()
.RegisterSchemeHandler()
.RegisterByteStreamHandler()?
.RegisterAudioDecoder()
.RegisterVideoDecoder()
.RegisterAudioEncoder()
.RegisterVideoEncoder()
.RegisterSchemeHandler()
.RegisterByteStreamHandler()
Registration Process: Media source
![Page 12: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/12.jpg)
ByteStreamHandler• New container format (WebM, MKV)
SchemeHandler• Generating your own streams• Media stored in a location not
accessible to inbox ByteStream• Custom streaming protocol
Which to use?
![Page 13: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/13.jpg)
IMFByteStreamHandlerInterface
Method DescriptionBeginCreateObject Begins an asynchronous request to create a media source from
a byte stream.CancelObjectCreation Cancels the current request to create a media source.
EndCreateObjectCompletes an asynchronous request to create an object from a URL. Completes an asynchronous request to create a media source.
GetMaxNumberOfBytesRequiredForResolution
Retrieves the maximum number of bytes needed to create the media source or determine that the byte stream handler cannot parse this stream.
![Page 14: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/14.jpg)
HRESULT BeginCreateObject( [in] IMFByteStream *pByteStream,[in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown **ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );
IMFByteStreamHandlerInterface
MediaElement.Source = new Uri(“video.mv4");
![Page 15: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/15.jpg)
HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );
typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;
IMFByteStreamHandlerInterface
![Page 16: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/16.jpg)
SchemeHandlers and Byte-StreamHandlers
Byte-StreamHandler
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
![Page 17: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/17.jpg)
IMFSchemeHandlerInterface
Method DescriptionBeginCreateObject Begins an asynchronous request to create an object from a
URL.CancelObjectCreation Cancels the current request to create an object from a URL.
EndCreateObject Completes an asynchronous request to create an object from a URL.
![Page 18: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/18.jpg)
HRESULT BeginCreateObject( [in] LPCWSTR pwszURL,[in] DWORD dwFlags,[in] IPropertyStore *pProps,[out] IUnknown
**ppIUnknownCancelCookie,[in] IMFAsyncCallback *pCallback,[in] IUnknown *punkState );
IMFSchemeHandlerInterface
MediaElement.Source = new Uri("myscheme://square");
![Page 19: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/19.jpg)
HRESULT EndCreateObject( [in] IMFAsyncResult *pResult,[out] MF_OBJECT_TYPE *pObjectType,[out] IUnknown **ppObject );
IMFSchemeHandlerInterface
typedef enum MF_OBJECT_TYPE { MF_OBJECT_MEDIASOURCE,MF_OBJECT_BYTESTREAM,MF_OBJECT_INVALID } MF_OBJECT_TYPE;
![Page 20: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/20.jpg)
SchemeHandlers and Byte-StreamHandlers
Byte-StreamHandler
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
![Page 21: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/21.jpg)
SchemeHandlers and Byte-StreamHandlersSchemeHandler
Byte-StreamHandler
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
ByteStream
![Page 22: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/22.jpg)
What is a ByteStream?
Represents a byte stream from some data source:• Local file• Network file• Some other source
This interface supports typical stream operations such as reading, writing, and seeking.
![Page 23: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/23.jpg)
IMFByteStream Interface
Method GetCurrentPositionBeginRead GetLengthBeginWrite IsEndOfStreamClose ReadEndRead SeekEndWrite SetCurrentPositionFlush SetLengthGetCapabilities Write
![Page 24: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/24.jpg)
SchemeHandlers and Byte-StreamHandlersSchemeHandler
Byte-StreamHandler
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
ByteStream
![Page 25: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/25.jpg)
SchemeHandlers and Byte-StreamHandlersSchemeHandler
Byte-StreamHandler
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
Media Foundation pipeline
ByteStream
SourceResolver
![Page 26: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/26.jpg)
Media Foundation pipeline
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
SchemeHandler
Byte-StreamHandlerByteStream
SourceResolver
![Page 27: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/27.jpg)
Media Foundation pipeline
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
SchemeHandler
Byte-StreamHandlerByteStream
SourceResolver
![Page 28: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/28.jpg)
Media Foundation pipeline
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
SchemeHandler
Byte-StreamHandlerByteStream
SourceResolver
![Page 29: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/29.jpg)
Marcin Stankiewicz Development - Multimedia Platform Team
Demo: Building a media extension• How to build an extension using C++/CX• The proper use of WRL to implement a
SchemeHandler• Instantiating a media source from a SchemeHandler • Using the PropertySet class to implement two-way
communication between the app and the extension• PropertySet change notification to support dynamic
updates• Direct3D rendering
![Page 30: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/30.jpg)
Recap of Demo• How to build an extension using C++/CX• The proper use of WRL to implement a
SchemeHandler• Instantiating a media source from a
SchemeHandler • Using the PropertySet class to implement two-
way communication between the app and the extension
• PropertySet change notification to support dynamic updates
• Direct3D rendering
![Page 31: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/31.jpg)
Media Foundation pipeline
Media source
Video decoder
Video effect
Video encoder Video sink
Audio decoder
Audio effect
Audio encoder Audio sink
Video
Audio
SchemeHandler
Byte-StreamHandlerByteStream
SourceResolver
![Page 32: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/32.jpg)
Matthew HowardDevelopment - Multimedia Platform Team
Demo: Streaming media to an app• How to implement a sink media extension for
audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques
![Page 33: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/33.jpg)
Recap of Demo• How to implement a sink media extension for
audio• Capturing uncompressed audio• Processing audio buffers in a JavaScript app• Writing audio to a wave file• Asynchronous processing techniques
![Page 34: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/34.jpg)
Need more information?
Come to our question-and-answer session!3-106 Chalk Talk: Media appsThursday 4:15pm, B33 Hood
![Page 35: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/35.jpg)
Media-related sessions3-117 10/31/2012 17:15300 - Advanced B33 Hood Key technologies for building advanced media apps3-122 11/1/2012 14:30 300 - Advanced B33 Hood Media app case studies3-106 11/1/2012 16:15 300 - Advanced B33 Hood Chalk Talk: Media apps
![Page 36: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/36.jpg)
Resources
Windows 8 media overviewhttp://blogs.msdn.com/b/b8/archive/2012/06/08/building-a-rich-and-extensible-media-platform.aspx
Media extensions SDK samplehttp://code.msdn.microsoft.com/windowsapps/Media-extensions-sample-7b466096
Windows 8 multimedia desktop APIs available to Windows Store appshttp://msdn.microsoft.com/en-us/library/windows/apps/hh452756.aspx
SchemeHandlers and ByteStreamHandlershttp://msdn.microsoft.com/en-us/library/windows/desktop/aa371872(v=vs.85).aspx
Writing a Custom Media Sourcehttp://msdn.microsoft.com/en-us/library/windows/desktop/ms700134(v=vs.85).aspx
![Page 37: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/37.jpg)
• Develop: http://msdn.microsoft.com/en-US/windows/apps/br229512
• Design: http://design.windows.com/
• Samples: http://code.msdn.microsoft.com/windowsapps/Windows-8-Modern-Style-App-Samples
• Videos: http://channel9.msdn.com/Windows
Resources
Please submit session evals by using the Build Windows 8 appor at http://aka.ms/BuildSessions
![Page 38: Media extensions](https://reader033.vdocuments.net/reader033/viewer/2022061610/5681610a550346895dd0554a/html5/thumbnails/38.jpg)
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.