multimedia beta-version p?v=jejuanpc2va#t=20
Post on 01-Apr-2015
217 Views
Preview:
TRANSCRIPT
CS338
MultimediaBeta-version
http://www.youtube.com/watch_popup?v=jEjUAnPc2VA#t=20
Overview -1
Required backgroundC , C++ or JavaCS240 (Data Structures) or CS212 (for Engineers)
This is NOT a graphics course (see CS460/560)
Labs Data compressionElements of graphics in OpenGLPhoto manipulation
GIMP Steganography
Creating & Rendering models
© D.J. Foreman 2009 2
Topics (general order of coverage)
Compression Encoding/Decoding Codecs Huffman encoding
example lab! Working with Images
Image creation Photography Drawing RIB (Renderman Interface
Bytestream) Modeling• OpenGL – lab!• Makehuman, 3DCanvas,
etc. lab! Image manipulation
Gimp, Photoshop, etc. lab! Stereoscopic & other 3D Rendering lab!
Bitmaps Ray tracing
Animation Hand drawn Program generated• Stop-motion• 3D CGI• Alice lab!• DirectX & OpenGL
Hardware acceleration RIB revisited Shaders
Steganography lab! Additional topics (time
permitting) Augmented reality Geospatial data systems
© D.J. Foreman 2009 3
Software we might use (all free)
Modelers and environments Ayam – ortho & 3D 3DCanvas – 3D Blender – ortho & 3D Makehuman - 3D
Renderers Aqsis Pixie – no GUI, file input
© D.J. Foreman 2009 4
5
Compression
© D.J. Foreman 2009
Codecs
A device or program for encoding and/or decoding a digital data stream or signal
Encoding A->D for storage or transmission Decoding D->A for playback Lossy vs. lossless Raw uncompressed Pulse Code Modulation
(PCM) audio, a digital representation of an analog signal where the magnitude of the signal is sampled regularly at uniform intervalsE.g.; (44.1 kHz, 16 bit stereo, as represented on an
audio CD or in a .wav or .aiff file) is a standard across multiple platforms.
© D.J. Foreman 2009 6
Codecs - 2
May emphasize aspects of data Video:
motion vs. colorAudio:
latency (cell phones) vs. high-fidelity (music)Data size:
transmission speed or storage size vs. data loss
© D.J. Foreman 2009 7
Codec References
Forensic Computing: A Practitioner's Guide by T. Sammes & B. Jenkinson (Springer, 2000), ISBN: 9781852332990 .
http://www.garykessler.net/library/file_sigs.html
© D.J. Foreman 2009 8
Considerations
Tradeoffs compression speedcompressed data size quality (data loss)
Areas of studyinformation theoryrate-distortion theory
Popular algorithms (all lossless)Lempel-Ziv (LZ)LZW (fast decompression)LZR (LZ-Renau) (ZIP files)LZW (Lempel-Ziv-Welch) (for GIF files)
© D.J. Foreman 2009 9
Lossless Encoding
Huffman compressionBit string representing any symbol is never a
prefix of the bit string representing any other symboli.e.; if 010 is the code for a symbol, then no other symbol starts with 010.
Frequency table must be known, computable or included
Lossless – every character gets encoded Arithmetic encoding
Probabilistic algorithmSlightly superior to Huffman, but often
patented!
© D.J. Foreman 2009 10
Methodology
Compute a model of the datae.g.; a Huffman tree based on probabilityProbability determined from input file
Map the data according to the modelRead 1 symbolSearch the model (tree) for that symbol
If it’s a Huffman tree, going left=0, right=1 Append each 0 or 1 to the code string When symbol is found, you are done Note: all symbols will have < 8 bits (thus
compression)
© D.J. Foreman 2009 11
12
Sample of prefix coding & compression
© D.J. Foreman 2009
B e a r c a t s (8 bytes of text):010101100110001110001111 encoded as123456781234567812345678 3 bytes
A simple Huffman-like code(w/o probability basis) 0 1
A 110
B 010
C 0001
E 10
R 011
S 111
T 001
Compression-Model types
StaticRead all the dataCompute frequenciesRe-read the data & encode
DynamicBuild simple basic modelModify model as more data is processed
© D.J. Foreman 2009 13
Standard Compressed-File Content
Three parts: 24-byte header with magic # (defines codec)variable-length annotation blockcontiguous segment of audio data.
Storage methodologynetwork (big-endian) byte ordermulti-byte audio data may require byte
reversal in order to operate on it by the arithmetic unit of certain processors
© D.J. Foreman 2009 14
Example 1: Header file for .AU files
© D.J. Foreman 2009 15
typedef unsigned long u_32; // unsigned 32-bit integer
typedef struct {
u_32 magic; // the “magic number”
u_32 hdr_size; // byte offset to start of data
u_32 data_size; // length (optional)
u_32 encoding; // data encoding enumeration
u_32 sample_rate; // samples per second
u_32 channels; // # of interleaved channels
} Audio_filehdr;
Following is from: #include <multimedia/libaudio.h>
AUDIO_FILE_MAGIC ((u_32)0x2e736e64) /* “.snd” */
Audio Encoding Enumerations
AUDIO_FILE_ENCODING_MULAW_8 (1) /* 8-bit ISDN u-law */
AUDIO_FILE_ENCODING_LINEAR_8 (2) /* 8-bit linear PCM */
AUDIO_FILE_ENCODING_LINEAR_16 (3) /* 16-bit linear PCM */
AUDIO_FILE_ENCODING_LINEAR_32 (5) /* 32-bit linear PCM */
AUDIO_FILE_ENCODING_FLOAT (6) /* 32-bit IEEE floating point */
AUDIO_FILE_ENCODING_DOUBLE (7) /* 64-bit IEEE floating point */
AUDIO_FILE_ENCODING_ADPCM_G721 (23) /* 4-bit CCITT g.721 ADPCM */
AUDIO_FILE_ENCODING_ADPCM_G723_3 (25) /* CCITT g.723 3-bit ADPCM */
AUDIO_FILE_ENCODING_ALAW_8 (27) /* 8-bit ISDN A-law */
© D.J. Foreman 2009 16
“Linear” values are SIGNED int’s.Floats are signed, zero-centered, normalized to ( -1.0 <= x <= 1.0 ).
Example 2: ZIP files (PkWare)Purpose Size in bytes
Signature header 4
Required version 2
GP flags 2
Method 2
Last mod time 2
Last mod date 2
CRC-32 4
Compressed size 4
Uncompressed size 4
Filename length 2
Extra field length 2
File name variable
extra variable
© D.J. Foreman 2009 17
Ref: http://livedocs.adobe.com/flex/3/html/help.html?content=ByteArrays_3.html
More info on encoding
http://www.pkware.com/documents/casestudies/APPNOTE.TXT
http://www.fileinfo.com/filetypes/compressed
© D.J. Foreman 2009 18
Recording Bit-Rate
Constant (CBR)rate at which a codec's output data should be
consumed is constantMax bit-rate matters, not the averageUses all available bandwidthNot good for storage (lossy)
Variable (VBR)Quantity of data/time unit (for output) variesBetter quality for audio and videoSlower to encodeSupported by most portable devices post 2006
© D.J. Foreman 2009 19
Lab assignment - 1
Write a single program with 2 parameters: When parameter 1 is a 1:
Open a test file, (e.g.; “xyz.txt”) Compress it using simple Huffman compression, using the
frequency table from my FTP site Output compressed file (e.g.; “xyz.enc”) to SAME folder as
input
When parameter 1 is a 2: Open compressed file “xyz.enc” Decompress the input file Output file: “xyz.dec” so it can be compared to “xyz.txt”
Parameter 2 is the full input file path & name (i.e.; file names MUST NOT be hard-coded)
Remember: a code can be a single bit!
© D.J. Foreman 2009 20
Working with Images
Modeling
Containers
A container is a FILE format, NOT a code scheme
E.g.; AVI is a container formatFormat for storageIndependent of encoding or content
Others:Ogg, ASF, QuickTime, RealMedia, Matroska,
DivX, and MP4.
© D.J. Foreman 2009 22
Pixels A pixel is 3 color-dots (squares) in a collection of dots
(squares) making up the picture. Each color-dot is for one of the three primary colors,
RGB. Minimum of 1 byte per color (depending on bit-depth,
which varies with camera), thus at least 3 bytes per pixel.
A 10MP camera (30 million color-dots) needs 30M bytes
This gets compressed by the camera (JPEG format) Final file size = 30 million bytes divided by the amount
of compression (compression ratio). An 8:1 compression ratio would reduce that 30 million
bytes to 3.75 megabytes (as done on a 4 MP camera) A RAW file has NO compression
© D.J. Foreman 2009 23
Image creation
Basic mechanismsDrawing
Line BrushModeling
Modeling/rendering programs OpenGL
Photography Film Digital
© D.J. Foreman 2009 24
Working Definitions - 1
Graphic Model data structure nodes represent random variablesPairs of nodes connected by arcs correspond
to variables that are not independent Graphic modeling
using a Graphic Model to simulate a real-world object (this is not a formal definition)
Rendering generating a 2D image from a 3D graphic model
© D.J. Foreman 2009 25
26
Working Definitions - 2
Geometric primitives – Points, line segments and polygonsDescribed by vertices
Control points – Special points attached to a node on a Bézier
curveAlter the shape and angle of adjacent curve
segment Evaluators –
Functions that interpolate a set of control points
Produce a new set of control points
© D.J. Foreman 2009
27
Renderman Interface Bytestream
A standardized interface Created by modeling programs Input to rendering programs A plaintext file
© D.J. Foreman 2009
Contexts
openGL 3.0 (2.0 + deprecated functions) openGL 3.1 only newer graphics cards
Many <= 3.0 functions deprecatedMany actions now done via shaders1st create a context (like the chicken & the
egg) create an old (e.g.; 3.0) context activate it create the new context deactivate old context
Old context needed to create new one
© D.J. Foreman 2009 28
29
OpenGL 2.0 Fixed Function Pipeline
© D.J. Foreman 2009
Vertex Data
Frame Buffer
Vertex Processor
Fragment Processor
Pixel Data
Per-vertex operations:• Eye-space coordinates• Colors• Texture coordinates• Fog coordinates• Point size
30
OpenGL 2.0 Vertex Processor Pipeline
© D.J. Foreman 2009
Vertex Coordinates
Normal Vector
Color Values
Texture Coordinates
Fog Coordinates
Model View Matrix
Model View Matrix
Texture Matrix
Projection Matrix
Primitive Setup
Clipping
Vertex Shaderreplaces these
31
Fragment Processing
© D.J. Foreman 2009
Color Summation
Per-pixel Fogging
Fragment Tests
Frame Buffer
Texture Mapping
Bitmaps/PixelRectangles
From primitive setup
fragment shader replaces these
openGL 3.1
Using shaders & Vertex Array ObjectsglBindVertexArray(my_vao_ID[0]); //
select 1st VAOglDrawArrays(GL_TRIANGLES, 0, 3); // draw 1st
objectnote difference from glBegin (GL_POLYGON)….
Note: name of out(put) variable in vertex shader must be the same as in(put) variable in fragment shader .i.e. vertex -> fragment (as in original pipeline)
© D.J. Foreman 2009 32
Lab 2- basic modeling Write a program using the OpenGL 2.0 interface:
Create 3 figures: Triangle (2D) Rectangle (2D) Irregular pentahedron (5-sided, 3D figure) (a pyramid)• See: http://en.wikipedia.org/wiki/Pentahedron for examples• ANY one Vertex pinned at 0,0,0• Allowed to be non-equilateral
Draw all 3 axes (optional, but HIGHLY recommended) The triangle is within the bounds of the rectangle
(or vice-versa), both objects in same plane, different colors The pyramid must have different colors on all 5 sides Create movement controls for the pyramid:
Cursor keys ←→↑↓ change camera location for world view L & R keys rotate pyramid (CW/CCW) about Z-axis Pinned pyramid Vertex STAYS at 0,0,0 The pyramid rotates, the “camera” position remains fixed
© D.J. Foreman 2009 33
34
Some Examples for lab 2
© D.J. Foreman 2009
0,0,0
0,0,1
0,0,0
0,0,1
6-vertex pentahedronValid for lab
Still has 5 sides!
OpenGL
PC RequirementsOPENGL library - for manipulating the model
opengl32.lib opengl32.dll (comes with Windows/XP & 7) glu32.dll “ glu.h and gl.h “
© D.J. Foreman 2009 35
36
Window Control Glut - note the “t” after the glu
http://www.opengl.org/resources/libraries/glut/glut_downloads.php#windows glut32.dll ”glut32.lib and here tooglut.h
FreeGlut (newer – Open Source)http://freeglut.sourceforge.net/ freeglut.dllfreeglut.libglut.h
glut.h does a #include of gl.h and glu.h© D.J. Foreman 2009
Programming in OpenGL
Define window Define objects Initialize “callback” functions Initialize window Run the gl main loop
© D.J. Foreman 2009 37
OpenGL Program Structure
Two parts1. Modeling, coloring, etc.
Use standard C code, with calls to OpenGL functions
Your program runs as subroutines WITHIN the OpenGL MainLoop
2. Callback functions Called by OpenGL, from OUTSIDE your program Display Reshape
Purpose of “registering” callbacks passes a pointer to your functions allows OpenGL to call them.
© D.J. Foreman 2009 38
Views
OrthographicAll views are 2-dimensionalNot good for games ignores the z-axis
PerspectiveSee model from “camera position” via
“viewport”
© D.J. Foreman 2009 39
cameraviewport
Φ
Program structure
Movement functions Shape defining functions (lines, polygons,
etc.) Reshape function (for window re-sizing) Init function Display function Main (includes call to glutMainLoop)
NOTE: gl, glu & glut prefixes on functions.Be careful!
© D.J. Foreman 2009 40
41
Skeleton of program -1#include <GL/glut.h>#include <stdio.h>#include <math.h>#include <stdlib.h>float ex,ey,ez,theta;void leftMotion (int x, int y);void rightMotion (int x, int y); GLfloat pvertices [][3]= { }; // and then their colors GLfloat pcolors[][3]={ };
// now define the polygon that USES those vertices //define any ONE face of the pyramid at a time
void pface(int a, int b, int c) // pface is a name I made up for “pyramid face”
{// 3 vertices define a faceglBegin(GL_POLYGON);glEnd();
}
© D.J. Foreman 2009
42
Skeleton of program 2void draw_pyramid() // implement the faces of the figure
{ glPolygonMode (GL_FRONT_AND_BACK,GL_FILL);}
void draw_square(){}void draw_triangle(){} void draw_axes(){
glPushMatrix ();glBegin(GL_LINES); // as pairs of points (default action of GL_LINES)glEnd();glPopMatrix ();
}
© D.J. Foreman 2009
43
Skeleton of program 3void dj_display(){ /* this is the function that draws the graphic object in the
pre-created window */glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT); /* clear the window */glLoadIdentity();// eye, at upgluLookAt(ex, ey, ez, 0,0,0, 0,1,0); draw_square();draw_triangle();draw_axes(); //last in code, means display on top.draw_pyramid();// now draw the pyramidglFlush();glutSwapBuffers ();
}
© D.J. Foreman 2009
44
Skeleton of program 4int main(int argc, char*argv[]){
glutInit(&argc, argv);glutInitWindowSize(wd,ht);glutInitWindowPosition(wx,wy);glutCreateWindow("DJ's 1st"); // define name on windowglutSpecialFunc(myspecialkeys);
/* register the call-back functions with Glut*/glutKeyboardFunc(mykey);glutDisplayFunc(dj_display); // more setupglutReshapeFunc(dj_reshaper);dj_init();
// Some texts show init being called AFTER the glutDisplayFunc call,// but it doesn't actually CAUSE display action, it just sets up the
environment.glutMainLoop(); // last (& the only real) executable stmt in programreturn 0;
}
© D.J. Foreman 2009
Basic Functions in “main”
Register your “callback” functionsglutSpecialFunc (your function name);glutKeyboardFunc (your function name);display, reshape
Running the programdj_init(); // your init routine (if any)glutMainLoop();
© D.J. Foreman 2009 45
Basic Display Function
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT);
glLoadIdentity();// eye_point, at(x,y,z) up
(x,y,z)gluLookAt (ex, ey, ez, 0,0,0, 0,1,0); draw_triangle/rectangle/pyramid, etcdraw_axes;glutSwapBuffers ();glFlush();
© D.J. Foreman 2009 46
Basic Reshape Function
void reshape (int w, int h) {
glViewport (0, 0, w, h);glMatrixMode (GL_PROJECTION);glLoadIdentity ( );glOrtho(….);glMatrixMode (GL_MODELVIEW);
}
© D.J. Foreman 2009 47
Drawing
Last drawn object is in “FRONT”MUST set bit depth AND enable bit depth test
RotationglRotate(Ѳ, x,y,z)
operates along a vector starting at 0,0,0Other rotations
Requires pre-multiplication of matrices with the identity matrix
© D.J. Foreman 2009 48
Photography (film) Silver halide crystals (AgCl, AgBr, AgI)
Exposure to light turns them blackDeveloper removes the loose (exposed) halide leaving pure
silverFixing bath flushes out unexposed AgHalide
Greatest desire of photographers: a medium withHigh speed (gathers light quickly)Low noise (no undesired artifacts)High resolution (very detailed images)
Visible lines per mm (lpm) at a specified contrast level
Notes:Larger “clumps” are faster, but limit resolutionSmaller “clumps” are hard to coat evenlyRandomness of coating prevents Moiré patternshttp://en.wikipedia.org/wiki/Moir%C3%A9_pattern
© D.J. Foreman 2009 49
FYI – a little chemistry
Ag+Br- + hv → Ag+ + Br + e-
1. Radiation frees the Br2. 1 Ag ion + 1 electron (Ag+ + e- → Ag0) -> 1 Ag
atom.3. Neighbors are now “sensitized” 4. Developing converts unexposed AgBr to AgS5. AgS & Br are washed away
This is one Ag “grain” in the “latent image”.
© D.J. Foreman 2009 50
crystal
Radiation (light)
Resolution 35 mm film for comparison
A 36 x 24 mm frame of ISO 100-speed film contains (≈) the equivalent of 20 million pixels. 1 There are NO pixels in film!
Full-frame digital (36 x 24mm) Canon EOS 5D, Nikon D3 (21MP) - $5K2
Medium format digital (6 x 4.5cm) Phase One P40+ (39MP) ($22K)
APS-C sized (≈24x16mm) Nikon D90 (12.3MP) - $900, Canon Rebel (15MP) $700 Used in most DSLR’s
Other SONY Alpha (note: image ratio 4/3)
1 Langford, Michael. Basic Photography (7th Ed. 2000). Oxford: Focal Press. ISBN 0 240 51592 7.
2 Prices as of 2009, given for comparison purposes only
© D.J. Foreman 2009 51
Photography (digital)
Image capture formatsRAW – 3 separate images (RGB)
[12 bits/pixel * (4288 * 2848)]/8=18,318,336 bytes per picture
JPEG – compressed (16x, 8x, 4x)Some cameras do BOTH (RAW + JPEG) for each
image Pixel notes:
Sensor size 35mm (or larger) vs. APS-CPixel count – any increase →decrease in pixel size Pixel size – smaller pixels can provide
Greater resolution (finer lines) Lower light sensitivity More “noise” (incorrect values)
© D.J. Foreman 2009 52
Image Manipulation
Photographic imagesGimp, Photoshop, etc
ModelingOpenGL – for creating and lighting a modelmodeling programs
Rendering converting a 3D (possibly wireframe) model
to a 2D image, then lighting and (possibly) shading it
Ray tracing another way of lighting & shading an imagevery CPU intensive
© D.J. Foreman 2009 53
54
GimpGnu Image Manipulation
Program
© D.J. Foreman 2009
Operational concepts
3 windows:Image – contains the actual resultsTools – operators, such as clone, paint, etcLayers – portions of the final image
Avoids need for changes to original Stackable Allows separation of collections of changes Allows complex changes to be applied
independently of other changes
© D.J. Foreman 2009 55
Gimp - lab
Get a digital landscape or cityscape image
Get a picture of yourself Insert the image of yourself into the
‘scape as a layer Save the new image as a new JPG file
© D.J. Foreman 2009 56
Video games vs. Movies
Elements to manipulateContent PerspectiveLighting/shadingSequence Coloring
Video game imagesDynamic
Movie imagesPre-determined
© D.J. Foreman 2009 57
Alice
Developed at CMU
Language-free programming environment
Point & click usage
Rapid prototyping
© D.J. Foreman 2009 58
59
Alice environment List of “world” objects
Insert into methodsModify characteristics (length, width, etc.)
List of world details Properties Methods (user created)Functions (built-in methods)
World window (results) Method builder
© D.J. Foreman 2009
60
Alice - Programming Programmer
Selects “verbs” from list For While Etc.
Applies values for properties Statement structure pre-defined
Blocks pre-defined Function calls for if… then… else
No need to learn a “language”
© D.J. Foreman 2009
61
Alice Lab
Open any of the Alice worlds Apply the following rules
1. Display 3 articulated characters (A, B and C)2. A flips onto its head and rotates slowly (5
times)3. B waves both of its arms slowly (5 times)4. C walks around A and B while (2 and 3
happen)5. C changes direction and walks around A and
B (while 2 and 3 repeat)6. Stop
Timing, size, shape, position are your choice
© D.J. Foreman 2009
STEGANOGRAPHY and LSB analysis
63
What is it? Hiding a message in plain sight
Inside another messageOriginal is called the “cover”
Presence of message is not obvious Can be done with text or graphics
© D.J. Foreman 2009
64
How is it done? Many algorithms for hiding a message Embedding is easier than detection Extraction is straight-forward if:
you know there is a hidden messageyou know the algorithm used
Can be very complex to detect & extract
© D.J. Foreman 2009
65
LSB embedding Concept:
Change the least significant bit of a set of bytes 1→0 and 0→1 For all bytes
Repeat for multiple sets of bytes Example with grayscale images
One pixel is 8 bits Change the last bitChanges that pixel’s shade VERY slightly.Repeat for every pixel
© D.J. Foreman 2009
LSB embedding - algorithmEmbedding function Emb (using Matlab or Freelab syntax)
c = imread(‘my_decoy_image.bmp’); % Grayscale cover image% ‘b’ is a vector of m bits (secret message)
k = 1; % Counterfor i = 1 : height for j = 1 : width LSB = mod(c[i, j], 2); if LSB = b[k] | k > m s[i, j] = c[i, j]; else
s[i, j] = c[i, j] + b[k] – LSB; end k = k + 1; endendimwrite(s, ‘stego_image.bmp’, ‘bmp’); % Stego image “s” saved to disk
67
imread The return value “c” is an array
containing the image data. If the file contains a grayscale image, “c” is an
M-by-N array. If the file contains a truecolor image, “c” is an
M-by-N-by-3 array. The class of “c” depends on the bits-per-
sample of the image data, rounded to the next byte boundary. E.g.; imread returns 24-bit color data as an array
of uint8 data because the sample size for each color component is 8 bits.
© D.J. Foreman 2009
LSB extraction - algorithmExtraction function Ext (Matlab syntax)
s = imread(‘stego_image.bmp’); % Grayscale stego imagek = 1;for i = 1 : height for j = 1 : width if k m b[k] = mod(s[i, j], 2); k = k + 1; end endend
% b is the extracted secret message as a bit string
Properties of LSB flipping
LSBflip(x) = x + 1 – 2(x mod 2)
FlipLSB(x) is idempotent, e.g., LSBflip(LSBflip(x)) = x for all x
LSB flipping induces a permutation on {0, …, 255}
0 1, 2 3, 4 5, …, 254 255
LSB flipping is “asymmetrical” (e.g., 3 may change to 2 but never
to 4)
| LSB(x) – x | = 1 for all x (embedding distortion is 1 per pixel)
top related