theoretical foundations of multimedia chapter 3 hardware that enables multimedia n input and output...

Theoretical Foundations of Multimedia Chapter 3

Hardware that Enables Hardware that Enables MultimediaMultimedia

Input and Output DevicesInput and Output Devices

Virtual Reality DevicesVirtual Reality Devices

Modems and Network InterfacesModems and Network Interfaces


Input and Output DevicesInput and Output Devices

MonitorsMonitors Speakers and MIDI interfacesSpeakers and MIDI interfaces VR helmets and immersive displaysVR helmets and immersive displays Keyboards and OCR devicesKeyboards and OCR devices Digital cameras, scanners, & CD-ROMsDigital cameras, scanners, & CD-ROMs MIDI keyboards and microphonesMIDI keyboards and microphones Video cameras and frame grabbersVideo cameras and frame grabbers Mice, track balls, joysticks, and VRMice, track balls, joysticks, and VR gloves and gloves and

wandswands

OutputOutput

InputInput


MonitorsMonitors

A simplified cathode ray tube (CRT)A simplified cathode ray tube (CRT)


MonitorsMonitors

Pixel — a picture element; a dot Pixel — a picture element; a dot of color on the screenof color on the screen

Three different phosphors at Three different phosphors at each pixel to create the coloreach pixel to create the color

RGB (RGB (Red, Red, green, green, blueblue)) CYM (CYM (CyanCyan, , yellowyellow, , magentamagenta))


MonitorsMonitors

Raster ScanningRaster Scanning


MonitorsMonitors

Refresh rate — the frequency atRefresh rate — the frequency at which which the phosphors are excitedthe phosphors are excited

Normally the refresh rate is givenNormally the refresh rate is given in Hertzin Hertz For flicker-free images 75 Hz orFor flicker-free images 75 Hz or faster is faster is

desirabledesirable The refresh rate for a projector needs The refresh rate for a projector needs

to be coordinated with the monitorto be coordinated with the monitor


MonitorsMonitors

The digitized image to be displayed The digitized image to be displayed must be must be stored in a bufferstored in a buffer

The stored image is said to beThe stored image is said to be “bit-“bit-mapped,” because, for mapped,” because, for monochrome monochrome images, the map images, the map used just one bit per used just one bit per pixelpixel

Multimedia monitors use 24 bitsMultimedia monitors use 24 bits per per pixel (8 for each color); canpixel (8 for each color); can define define >16 million colors>16 million colors


A Good Multimedia MonitorA Good Multimedia Monitor

Large enough for comfortableLarge enough for comfortable viewing, viewing, probably 15” or greaterprobably 15” or greater

Pixel size of no more than 0.28mmPixel size of no more than 0.28mm Refresh rate of at least 75 HzRefresh rate of at least 75 Hz Capable of displaying 24-bit colorCapable of displaying 24-bit color Designed for the CPU and operatingDesigned for the CPU and operating systemsystem Ergonomically comfortable andErgonomically comfortable and

attractiveattractive


Speakers and Speakers and MIDI InterfacesMIDI Interfaces

Storage of digitized sound filesStorage of digitized sound files Reproduction via digital-to-analog Reproduction via digital-to-analog

conversion sent to a loudspeakerconversion sent to a loudspeaker Built-in speakers often do not haveBuilt-in speakers often do not have sufficient sufficient

fidelityfidelity Low-powered (3- to 5-watt) externalLow-powered (3- to 5-watt) external

speakers or head-phones will serve a speakers or head-phones will serve a single single user and provide excellent fidelityuser and provide excellent fidelity


Storage of synthesizer command files Storage of synthesizer command files Create the sounds by sending theCreate the sounds by sending the

commands to a synthesizercommands to a synthesizer Musical Instrument Digital InterfaceMusical Instrument Digital Interface

(MIDI) standard (1982)(MIDI) standard (1982) MIDI includes both a hardware and a message MIDI includes both a hardware and a message

standardstandard

Speakers and Speakers and MIDI InterfacesMIDI Interfaces


Speakers and MIDI InterfacesSpeakers and MIDI Interfaces

MIDI hardware standard defines MIDI hardware standard defines cables, connectors, circuits, cables, connectors, circuits, andandelectrical signalselectrical signals

MIDI message standard defines aMIDI message standard defines a Device number for multiple device systems Control segment that gives an instrument-specific

command Data segment containing the information needed by

that instrument for that command


Alphanumeric KeyboardsAlphanumeric Keyboards

For entering commands, text, For entering commands, text, and dataand data Each key is a switch that closesEach key is a switch that closes when it is when it is

depressed, sending adepressed, sending a code to the CPUcode to the CPU The arrangement of the keys mayThe arrangement of the keys may

varyvary The most common is QWERTYThe most common is QWERTY Another arrangement is DvorakAnother arrangement is Dvorak


Choosing a KeyboardChoosing a Keyboard

Does it include Does it include allall of the needed of the neededcharacters, including command characters, including command keys?keys?

Is it ergonomically comfortableIs it ergonomically comfortableand safe, preventing repetitive and safe, preventing repetitive stress stress syndrome?syndrome?


Optical Character Recognition Optical Character Recognition (OCR)(OCR)

Hardware — scans the text imageHardware — scans the text image Software — systematically checksSoftware — systematically checks the the

entire image for patterns ofentire image for patterns of light and light and dark that it recognizesdark that it recognizes as alphabetic, as alphabetic, numeric, or punc-numeric, or punctuation characterstuation characters

OCR software entails pattern OCR software entails pattern recognition, a sophisticated recognition, a sophisticated logic logic problemproblem


E e E E e e E E e Ee e E E E e e E e e

It is relatively easy for a human to recognize each of these characters as the letter “e.” For the pattern recognition logic in OCR software, this is very difficult.

Optical Character Recognition Optical Character Recognition (OCR)(OCR)


Digital Cameras and ScannersDigital Cameras and Scanners

Real Image Real Image —— a portion of what is physically present in nature

Digital Image Digital Image — — a representation of a real image in which individual points are encoded to represent the wavelength and intensity of light at that point

Still Image Still Image — — a single snapshot of an instant; may be real or digital

Motion Image Motion Image — — a sequence of images that, when viewed consecutively at the appro- priate rate, gives the impression of continuous motion; may be digital or analog


ScannersScanners

Schematic Drawing of a ScannerSchematic Drawing of a Scanner


Digital CamerasDigital Cameras

Schematic Drawing of a Digital CameraSchematic Drawing of a Digital Camera


Digital Cameras and ScannersDigital Cameras and Scanners

Quality of the optics and the scanning Quality of the optics and the scanning mechanism, which determines focusmechanism, which determines focus

Precision of the photosensitive cells, whichPrecision of the photosensitive cells, whichdetermines the accuracy of the encodingdetermines the accuracy of the encoding of of intensity and wavelength dataintensity and wavelength data

Resolution of the instrument in dots perResolution of the instrument in dots per inch, inch, which determines graininesswhich determines graininess

Amount of storage available, which deter-Amount of storage available, which deter-minesmines the total size of an image that can the total size of an image that can be be digitizeddigitized

Image quality depends on the:Image quality depends on the:


Inputting ImagesInputting Images

Memory required to store a 5” x 7” snapshotMemory required to store a 5” x 7” snapshot

Dots/inch resolutionof snapshot image

100300600

1200

Bytes requiredfor storage

1.05 Mb9.45 Mb37.8 Mb

151.2 Mb

Assuming no compression, 24 bits per pixel

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.


















Video Cameras and Video Cameras and Frame Frame GrabbersGrabbers

Video cameras are similar to digital camerasVideo cameras are similar to digital cameras Except that a video camera takes image Except that a video camera takes image after image after image

continuouslycontinuously The output from many video cameras isThe output from many video cameras is

analog and requires digitizing circuitryanalog and requires digitizing circuitry to make the to make the image usable in a computerimage usable in a computer

Digital camcorders are now available Digital camcorders are now available Frame grabber software allows the capture ofFrame grabber software allows the capture of a a

single still image from the video streamsingle still image from the video stream Frame grabbed images are of rather lowFrame grabbed images are of rather low

resolution, however, <80-90 dots/inchresolution, however, <80-90 dots/inch


Microphones and Microphones and MIDI KeyboardsMIDI Keyboards

For input of soundFor input of sound Microphones capture sound waves from theMicrophones capture sound waves from the air air

as an analog signalas an analog signal The analog signal must be digitized to be The analog signal must be digitized to be stored and stored and

then replayed by the computerthen replayed by the computer Digitizing at <10,000 Hz is adequate forDigitizing at <10,000 Hz is adequate for

speech; 20,000 Hz is needed for musicspeech; 20,000 Hz is needed for music MIDI keyboards usually look like piano key-MIDI keyboards usually look like piano key-

boards with extra switches and controlsboards with extra switches and controls MIDI keyboards encode and transmit musical MIDI keyboards encode and transmit musical

informationinformation according to the MIDI standard according to the MIDI standard


Inputting Positional Inputting Positional InformationInformation

MiceMice TrackballsTrackballs Track padsTrack pads JoysticksJoysticks Drawing tabletsDrawing tablets



Specifying a point on a surface Specifying a point on a surface requires two dimensions, asrequires two dimensions, aswith latitude and longitudewith latitude and longitude

A third dimension could be added, A third dimension could be added, as with as with altitudealtitude

For multimedia, what is commonly For multimedia, what is commonly needed needed is position on the monitor is position on the monitor in terms of left-in terms of left-right (X) and up-right (X) and up- down (Y) distancesdown (Y) distances



X and Y coordinates are obtainedX and Y coordinates are obtainedrelative to a fixed point, usuallyrelative to a fixed point, usually one one corner of the screencorner of the screen

The coordinates are entered inThe coordinates are entered in analog analog form as output from roll-form as output from roll- ing wheels ing wheels inside a device such inside a device such as a mouseas a mouse

The analog values are digitized toThe analog values are digitized tospecify the X and Y coordinatesspecify the X and Y coordinates


The Mechanism of a MouseThe Mechanism of a Mouse


Using a Drawing TabletUsing a Drawing Tablet


Created in part using a drawing tabletCreated in part using a drawing tablet © Janet Anderson

Collage by Janet AndersonCollage by Janet AndersonQuickTime™ and a

Photo - JPEG decompressorare needed to see this picture.


are needed to see this picture.QuickTime™ and a

























Photo - JPEG decompressorare needed to see this picture.QuickTime™ and a



CD-ROMs, DVDs, and CD-ROMs, DVDs, and Video DisksVideo Disks

Media for external storage and Media for external storage and transport of datatransport of data Compact disk—read-only memory Rewritable compact disk (CD-RW) DVD Video disk (laser disk); analog

format


CD-ROMsCD-ROMs

Digital formatDigital format Write once, read many timesWrite once, read many times A rewritable version (CD-RW) is available,A rewritable version (CD-RW) is available, but but

not in common usenot in common use Information is “written” by burning tiny Information is “written” by burning tiny

holes in the disk surface with a laserholes in the disk surface with a laser The hole pattern is read by a laser and inter-The hole pattern is read by a laser and inter-

preted as the bits comprising the datapreted as the bits comprising the data Can store 600-700 megabytes of data; aboutCan store 600-700 megabytes of data; about

300,000 pages of double-spaced text, or300,000 pages of double-spaced text, or more than more than an hour of high fidelity soundan hour of high fidelity sound

Theoretical Foundations of Multimedia Chapter 3Creating Creating

Multimedia Multimedia CD-CD-

ROMsROMs Requires a hard disk large enough to storeRequires a hard disk large enough to store

~650 megabytes of data to be written~650 megabytes of data to be written to the to the CDCD

Requires a CD-ROM recorder that writes theRequires a CD-ROM recorder that writes thedata to the blank CD using a laserdata to the blank CD using a laser

The developer creates the multimedia mater-The developer creates the multimedia mater-ial, stores it on the hard disk, and then ial, stores it on the hard disk, and then tests it as tests it as completely as possiblecompletely as possible

When the material is in final form, it isWhen the material is in final form, it iswritten to the blank CD as if it werewritten to the blank CD as if it werebeing copied from one disk to anotherbeing copied from one disk to another


Video Disks orVideo Disks orLaser DisksLaser Disks

Much larger than a CD-ROM; ~12” in diameterMuch larger than a CD-ROM; ~12” in diameter Hold ~54,000 video frames per sideHold ~54,000 video frames per side Hold ~30 minutes of video per sideHold ~30 minutes of video per side Read-onlyRead-only Analog formatAnalog format Requires a conversion board to be used with a Requires a conversion board to be used with a

computercomputer Excellent for large-scale, video-based multimedia Excellent for large-scale, video-based multimedia

projectsprojects


Virtual Reality DevicesVirtual Reality Devices Non interactive

Slow image update rate

Simple image

Nonengaging content and presentation

No sound

Basic Screen display

Low resolution image

Monoscopic image

Small field of view

No head tracking

No body motion sensing

No tactile feedback

Highly interactive

Fast image update rate

Highly complex image

Highly engaging content and presentation

Three-dimensional sound

Head-mounted display

High resolution image

Stereoscopic image

Full field of view

Full head tracking

Full body motion sensing

Full tactile feedback

Factors affecting the degree of immersion in virtual reality


NCSA’s CAVENCSA’s CAVE

QuickTime™ and aVideo decompressor



NCSA’s CAVENCSA’s CAVE

• Virtual Reality Room with stereo glasses and magnetic head/hand tracking

• Fully immersive using three of four walls to display the graphics

• Uses an SGI Power Onyx with Reality Engine 2 software


NCSA’s ImmersaDeskNCSA’s ImmersaDesk

QuickTime™ and aCinepak decompressor


Miniature version of NCSA’s CAVE





Drafting-table format virtual prototyping device





Uses CAVE’s stereo glasses and magnetic head/hand tracking





Semi-immersive, fills the user’s field of vision





Uses an identical SGI Power Onyx with the same Reality

Engine 2 software as the CAVE


VR Head-Mounted DisplayVR Head-Mounted Display


VR Head-Mounted DisplayVR Head-Mounted DisplayLimitations:Limitations: Liquid Crystal Displays (LCDs)Liquid Crystal Displays (LCDs)

pixels not as small as a CRT pixels not as bright as a CRT cannot change as quickly as a CRT short focal distance makes precision, high

resolution, and rapid response even more essential Muscle receptor feedback confusionMuscle receptor feedback confusion

light rays indicate “distant” muscles indicate “very close”


VR Head-Mounted DisplayVR Head-Mounted Display Parallax — the apparent change in position of a Parallax — the apparent change in position of a

stationary object when viewed from slightly stationary object when viewed from slightly different positionsdifferent positions

A person’s eyes each see a slightly different view of A person’s eyes each see a slightly different view of an objectan object

As the brain receives these two images, it interprets As the brain receives these two images, it interprets the the distance to the object in terms of the the the distance to the object in terms of the difference in position of the object in the two imagesdifference in position of the object in the two images

Parallax can be used to fool the brain into “seeing” Parallax can be used to fool the brain into “seeing” images as being at various distancesimages as being at various distances


DemonstratingDemonstratingParallaxParallax

Pencil

Looking at a pencil aligned What is seen using with the corner of a room both eyes

What is seen with What is seen withright eye covered left eye covered


StereoscopeStereoscope

Courtesy of Special Collections, M. I. King Library, University of Kentucky


Parallax Problems withParallax Problems withVR Head-Mounted DisplaysVR Head-Mounted Displays

Images may not be perfectly realistic, Images may not be perfectly realistic, especially especially with motion imageswith motion images

When the observer’s head moves and the eyes are When the observer’s head moves and the eyes are refocused, muscle receptor feedback data does not refocused, muscle receptor feedback data does not correlate with visual cuescorrelate with visual cues

The perspective is always that of the camera, never the The perspective is always that of the camera, never the viewer’s eyesviewer’s eyes

A viewer motion feedback mechanism is needed to A viewer motion feedback mechanism is needed to change the perspectivechange the perspective

This all contributes to “cybersicknessThis all contributes to “cybersickness””


VR Aural OutputVR Aural Output

Refer to the discussion in chapter 2 regarding the Refer to the discussion in chapter 2 regarding the perception of soundperception of sound

Two key factors Two key factors Localization Identification

The brain interprets differences in the signals it receives The brain interprets differences in the signals it receives from the two ears in a manner analogous to binocular from the two ears in a manner analogous to binocular visionvision

For multimedia sound to be completely realistic, it requires For multimedia sound to be completely realistic, it requires head-position sensing feedback and enormous head-position sensing feedback and enormous computational power — not practical for most multimediacomputational power — not practical for most multimedia


VR Input DevicesVR Input Devices

The terminology of three-dimensional motion

y

x

z

Origin

Roll

Yaw

Pitch


VR Position SensingVR Position Sensing

A point in space is defined in terms of A point in space is defined in terms of distance along three mutually perpen-distance along three mutually perpen-dicular axes, usually termed X, Y, and Zdicular axes, usually termed X, Y, and Z

Motion is defined in terms of changes in Motion is defined in terms of changes in position, which requires six parametersposition, which requires six parameters

Devices that can sense and record Devices that can sense and record motion are motion are termed six-degrees-of-freedom (6-DOF) devicestermed six-degrees-of-freedom (6-DOF) devices


VR Position SensingVR Position Sensing Sensor output from a 6-DOF device can beSensor output from a 6-DOF device can be

continuous polled, or sent only upon request

Parameters to consider in evaluating a Parameters to consider in evaluating a tracking device:tracking device: Lag or Latency — the delay between the actual time of the

motion and when it is available as input data; should be <50 mSec

Update rate — Rate at which measurements are made; should be as fast as possible

Precision or accuracy of the measurements Range over which the sensors operate Rejection of interference


VR Voice InputVR Voice Input Speech RecognitionSpeech Recognition Complications due to variations inComplications due to variations in

Pitch Timbre Volume Speed of Delivery Inflection Accent

Natural language processingNatural language processing


Natural Language ProcessingNatural Language Processing

Put out the light.Put out the light. Turn off the light.Turn off the light. Close the light, please.Close the light, please. The light, turn it off.The light, turn it off. Please, shut the light.Please, shut the light. Kill the lights.Kill the lights.


Natural Language ProcessingNatural Language Processing

Only the son praised his sister.Only the son praised his sister. (The rest of the family did not.)

The only son praised his sister.The only son praised his sister. (There was just one son.)

The son only praised his sister.The son only praised his sister. (He never found fault with her.)

The son praised only his sister.The son praised only his sister. (But never anyone else.)

The son praised his only sister.The son praised his only sister. (He had just one sister.)

The son praised his sister only.The son praised his sister only. (In this instance, he praised no one but her.)


Modems andModems andNetwork InterfacesNetwork Interfaces

Serial and ParallelSerial and Parallel Serial — the bits arrive sequentially Parallel — the bits arrive simultaneously

Character encodingCharacter encoding ASCII — American standard code for information

interchange EBCDIC — Extended binary-coded decimal

interchange code Unicode


ASCII and UnicodeASCII and Unicode

ASCII is limited because it is only a 7- ASCII is limited because it is only a 7- or 8- or 8-bit code; even using “escape sequences” only a bit code; even using “escape sequences” only a small number of characters can be encodedsmall number of characters can be encoded

Unicode is a 16-bit code that can Unicode is a 16-bit code that can encode encode many primary scripts plus many primary scripts plus special special character sets known as character sets known as secondary scriptssecondary scripts


Unicode ScriptsUnicode ScriptsPrimary scripts:Primary scripts:

Arabic Georgian Hebrew MalayalamArmenian Greek Hiragana OriyaBengali Gujarati Kannada PhoneticBopomofo Gurmkhi Katakana TamilCyrillic Han Latin TeluguDevanagari Hangul Lao Thai

Secondary scripts:Secondary scripts:

Numbers General diacriticsGeneral symbols Miscellaneous symbolsGeneral Punctuation Technical symbolsDingbats Mathematical symbolsPresentation formsArrows, blocks, box drawing forms, and geometric shapes



Start bitsStart bits Opposite bit from the system idle state Necessary to alert the receiver to the beginning of a new

character

Stop bitsStop bits Provide a short delay at the end of each character to

give the receiver enough time to convert from serial to parallel

Error-checking codesError-checking codes Parity bits, CRC bits, etc. Discussed in chapter 6



Transmission rateTransmission rate Internal transfer rates are much faster than data rates

over networks The interface needs to “interrupt” the

computer when it has new data, not keep it from doing other processing while data is being received

Transmission formTransmission form Connection via a telephone line requires a modem

(MOdulator-DEModulator) to translate the internal data transfer format into an audio signal, and vice-versa



Use of modems and telephone lines Use of modems and telephone lines for connectivityfor connectivity

theoretical foundations of multimedia chapter 3 hardware that enables multimedia n input and output...

Documents

color n

data n

desirable n

qwerty n

screen n

excited n

buffer n

keyboard n