1

The Java Media Framework: javax.media

• Access to devices & media files• Resources

– Sun JMF web site• http://java.sun.com/products/java-media/jmf/index.jsp• http://java.sun.com/products/java-media/jmf/2.1.1/apidocs/

– Sample chapter from book, “Essential JMF - Java Media Framework” Gordon & Talley, 1998

– Terrazas et al. (2002). Java Media API’s. SAMS Pub.

2

Topics• General contents of JMF• Digital video primer• Classes

– java.awt.image.BufferedImage• Abstract Window Toolkit class

– java.awt.image.ColorModel– javax.media.format.RGBFormat

• Pixel representation • Classes

– javax.media.Format;– javax.media.MediaLocator;– javax.media.protocol.DataSource;– javax.media.CaptureDeviceManager;

3

Digital Video Primer

• Cameras return “frames” of visual information• Each frame is a 2D matrix

– Components of these arrays are pixels• Usually integers

– Dimensions: height by width• E.g., 160 x 120 pixels

• Frames are returned from the camera at a certain rate– I.e., the frame rate– E.g., 15 frames per second

• Frames are often represented in programs as one-dimensional arraysint[] frame; // in Java

4

Frames as 1D Arrays

int [] frame = null;

// code to get frame from camera…

int x = 10, y = 20; // get some pixel

// 0 <= x <= width-1; 0 <= y <= height-1

// (0, 0)-- upper left of image

int pixel = frame[y * width + x];

// assumes a row oriented representation

5

Class java.awt.image.BufferedImage• An Image comprised of a data buffer for image,

plus color model information• Represents color or grayscale images in various

formats– E.g., TYPE_BYTE_GRAY, TYPE_USHORT_GRAY,

TYPE_USHORT_565_RGB, TYPE_INT_RGB

• Methods includepublic ColorModel getColorModel()

• ColorModel (java.awt.image.ColorModel) describes the format of the image

public BufferedImage getSubimage(int x, int y, int w, int h)

public int[] getRGB(parameter list…)

6

Classjava.awt.image.ColorModel

• Abstract class used to construct representations of colors (color models) for images

• Methods include– public abstract int getRed(int pixel)– public abstract int getGreen(int pixel)– public abstract int getBlue(int pixel)

7

Class javax.media.format.RGBFormat

• The RGBFormat class of the javax.media.format package– Subclass of VideoFormat

(javax.media.format.VideoFormat)

• “32-bit packed RGB stored as 32-bit integers would have the following masks: redMask = 0x00FF0000, greenMask = 0x0000FF00, blueMask = 0x000000FF.” (From JMF API)

8

Pixel Representation

• Pixels as returned from the BufferedImage class getRGB method– Represented using lower 24 bits of int’s

• High order byte: Red• Middle byte: Green• Low-order byte: Blue

• This format because the Java programs we are using set up the image formats this way– RGBFormat (javax.media.format.RGBFormat)

image formats were obtained from the camera

9

Example

• 1) Given a pixel in RGBFormat as described, use bit manipulation operations to extract out the R, G, and B components

• 2) Use bit manipulation operations to create a new RGBFormat pixel given 8 bit R, G, and B components

10

Outline of Using JMF to Access USB Camera Devices• Determine available camera devices

– CaptureDeviceManager.getDeviceList

• Obtain a DataSource– Use Manager to create this DataSource, based on

a MediaLocator

• Obtain a Processor– From Manager based on the DataSource

• Obtain a PushBufferDataSource– From the Processor

11

Classjavax.media.CaptureDeviceManager

• Methodspublic static CaptureDeviceInfo getDevice(java.lang.String deviceName)

• Get a device by name

public static java.util.Vector getDeviceList(Format format)• Returns a Vector containing a list of CaptureDeviceInfo

descriptions of devices available on the computer of the format given

– Formats returned may be more than that specified– E.g., if a given device supports YUV and RGB formats, then all

formats (including both YUV and RGB) will be returned in the result of the getDeviceList call

• If Format is null, returns the list of all available devices on the system

12

Classjavax.media.CaptureDeviceInfo

• Methods includepublic Format[] getFormats()

• Returns an array of Format objects• Specific Format’s available for the video device

public MediaLocator getLocator()

13

Class javax.media.Format

• Format class of javax.media package • Constructor

Format(java.lang.String encoding)

• Device/media specific description of a media format– E.g., for a video (visual) device

• "RGB, 160x120, Length=38400, 16-bit, Masks=31744:992:31, PixelStride=2, LineStride=320, Flipped”

– E.g., for an a audio device• "PCM, 44.1 KHz, Stereo, Signed"

14

Class javax.media.MediaLocator

• Constructor– MediaLocator(java.lang.String locatorString)

• Locator strings are similar to URLs

• Example locatorString– “vfw://0”

• This is for my USB web camera

• MediaLocator’s can describe filese.g.,MediaLocator m = new

MediaLocator(“file://media/example.mov”);

15

Classjavax.media.Manager - 1

• Access point for obtaining system dependent resources– Player’s, DataSource’s, Processor’s, DataSink’s,

TimeBase

• DataSource– Object used to deliver time-based multimedia data

that is specific to a delivery protocol.• Examples of protocols: http, ftp, file

– Once you have a “usable” (formatted) DataSource, you can display the media (via a Player), or manipulate the information (via a Processor)

16

Class javax.media.Manager - 2• Manager methods include

public static DataSource createDataSource(MediaLocator sourceLocator) Returns a data source for the protocol specified by the MediaLocator.

When you call createDataSource on the media locator of a video capture device (obtained from the CaptureDeviceManager), the returned DataSource will implement the CaptureDevice interface

public static DataSource createCloneableDataSource(DataSource source)- create a DataSource that can be cloned; this enables DataSource to be processed by different tasks

• public static Processor createProcessor(DataSource source)

- Processors are used in controlling the processing of media

17

Classjavax.media.protocol.DataSource

• An abstraction for media protocol-handlers. DataSource manages the life-cycle of the media source by providing a simple connection protocol.

• Methods include– DataSource(MediaLocator source)– connect– start, stop

• Start and stop data transfer

18

Interface CaptureDevice

• A capture device is a DataSource of type PullDataSource, PullBufferDataSource, PushDataSource or PushBufferDataSource. It also implements the CaptureDevice interface… A CaptureDevice DataSource contains an array of SourceStream's. These SourceStreams provide the interface for the captured data streams to be read. (From JMF API)

• Methods include– public FormatControl[] getFormatControls()

19

InterfaceFormatControl

• Objects implementing the FormatControl interface can be used to set the Format of the CaptureDevice to the Format desired

• Methods– public Format setFormat(Format format)

• Returns null if the format is not supported. Otherwise, it (typically) returns the format that's actually set.

20

Interface javax.media.Processor

• Processes and controls time-based media data. Extends the Player interface. Unlike a Player, which processes data as a "black box" and only renders data to preset destinations, a Processor supports a programmatic interface enabling control over media data processing and access to output data streams.

• Processing performed by a Processor is split into three stages:– Demultiplexing - into separate tracks of data– Data transcoding - of each track into other formats– Multiplexing - to form an interleaved stream

• Data transcoding and multiplexing processes are programmable.

21

Interface javax.media.Processor• Processor is a Player is a Controller• Methods include

DataSource getDataOutput()get the output DataSource from the Processor

public void addControllerListener(ControllerListener listener)

• From Controller interface• Events get posted to the listener, I.e., the

public void controllerUpdate(ControllerEvent event)

method gets called for that listener

public void realize()• Constructs media dependent portions of Controller.

public void prefetch()public void start()public int getState()

• States: Unrealized, Configuring, Configured, Realizing, Realized, Prefetching, Prefetched, and Started.

22

Controller Life Cycle• Controller’s have five resource-allocation states:

Unrealized, Realizing, Realized, Prefetching, and Prefetched.

• These states provide programmatic control over potentially time-consuming operations. For example, when a Controller is first constructed, it's in the Unrealized state. While Realizing, the Controller performs the communication necessary to locate all of the resources it needs to function (such as communicating with a server, other controllers, or a file system). The realize method allows an application to initiate this potentially time-consuming process (Realizing) at an appropriate time.

23

Classjavax.media.protocol.PushBufferDataSource

• A data source that manages data in the form of push streams. The streams from this DataSource are PushBufferStream’s and contain Buffer objects

• We we use getDataOutput to get a DataSource from our Processor, and this DataSource is actually a PushBufferDataSource

• Methodspublic abstract PushBufferStream[] getStreams()

• Return streams that this source manages. The collection of streams is entirely content dependent. The ContentDescriptor of this DataSource provides the only indication of what streams can be available on this connection.

• For our USB camera capture device, there is only one PushBufferStream

24

Interface javax.media.protocol.PushBufferStream

• A read interface that pushes data in the form of Buffer objects. Allows a source stream to transfer data in the form of a media object. The media object transferred is the Buffer object as defined in javax.media.Buffer. The user of the stream will allocate an empty Buffer object and pass this to the source stream in the read() method. The source stream allocates the Buffer object's data and header, sets them on the Buffer and sends them to the user.

25

Interface javax.media.protocol.PushBufferStream

• Methods includepublic Format getFormat()

public void

setTransferHandler(BufferTransferHandler transferHandler) • Register an object to service data transfers to this stream.

public void read(Buffer buffer)• Read from the stream without blocking.• (Not documented regarding what happens if you read and

there is no data available-- Is the buffer empty or is an IOException raised).

26

Interface javax.media.protocol.BufferTransferHandler

• Implements a callback from a PushBufferStream.

• A PushBufferStream needs to notify the data handler when data is available to be pushed.

• Methods includepublic void

transferData(PushBufferStream stream)• Notification from the PushBufferStream to the handler

that data is available to be read from stream. The data can be read by this handler in the same thread or can be read later.

27

Classjavax.media.Buffer

• A media-data container that carries media data from one processing stage to the next inside of a Player or Processor. Buffer objects are also used to carry data between a buffer stream and its handler.

• Maintains information including the time stamp, length, and Format of the data it carries, as well as any header information that might be required to process the media data.

28

Classjavax.media.util.BufferToImage

• A utility class to convert a video Buffer object to an AWT Image object– you can then render the Image to the user

display with AWT methods.

• Methods includepublic BufferToImage(VideoFormat format)public java.awt.Image createImage(Buffer buffer)

Converts the input Buffer to a standard AWT image and returns the image.

29

Class java.awt.image.BufferedImage• An Image comprised of a data buffer for image, plus

color model information– A subclass of java.awt.Image

• Represents color or grayscale images in various formats– E.g., TYPE_BYTE_GRAY, TYPE_USHORT_GRAY,

TYPE_USHORT_565_RGB, TYPE_INT_RGB

• Methods includepublic ColorModel getColorModel()

• ColorModel (java.awt.image.ColorModel) describes the format of the image

public BufferedImage getSubimage(int x, int y, int w, int h)

public int[] getRGB(parameter list…)

(From Above)

30

Capturing the Webcam Images

PushBufferStream PushBuffer; // created above

Buffer ImageBuffer = new Buffer(); BufferToImage ImageConverter =

new BufferToImage((VideoFormat) (PushBuffer.getFormat()));

// assume a new frame is readyPushBuffer.read(ImageBuffer);BufferedImage InputImage =

((BufferedImage) (ImageConverter.createImage(ImageBuffer)));

int[] CurrentFrame = InputImage.getRGB(0, 0, VideoWidth, VideoHeight, null, 0, VideoWidth);

From ImageProcessingThread.java (Lab1)