1

Constraints for Multimedia Presentation Generation

Joost Geurts, Multimedia and Human-Computer Interaction

CWI Amsterdam email: Joost.Geurts@cwi.nl

2

Talk overview

•Generating multimedia automatically

•Cuypers multimedia generation engine

•Multimedia and constraints–Quantitative constraints

–Qualitative constraints

•Cuypers demo

•Conclusion, future directions

3

Multimedia Presentation

•Multimedia Presentation– Image, Text, Video, Audio

–Based on Temporal and

Spatial Synchronization

•Multimedia Document–SMIL, SVG, HTML

–WYSIWYG

–Static Content

•Problem: Dynamic Content

4

Generating adaptive multimedia

•Content–Large multimedia database

•System profile–PC, PDA, WAP

•Network profile–Modem, Gigabit

•User profile–Language, Interests, Abilities, Preferences

Too costly to author manually

5

Cuypers multimedia generation engine

6

Cuypers multimedia generation engine

•Cuypers is based on–media independent presentation abstractions– transformation rules with built-in backtracking andconstraint solving

7

Semantic structure

Author does not specify complete presentation……but only rhetoric relations

8

Communicative Devices

…rhetoric relations are than transformed into presentation independent communicative devices…

9

Automatic multimedia generation

•Designer does not specify complete presentation……but only specifies requirements

•System automatically finds a solution which meets requirements

•How should the requirements be specified?–Declarative constraints

10

Constraint satisfaction

•Constraints occur often in our daily lives–Agenda, Travelling, Shopping

•Constraint paradigm for Problem Solving–DeclarativeUsed for problems with:–Many variables–Large domains–Based on domain reduction paradigm

11

Intelligent reduction of possible values

X {1,2,3,4,5},

Y {1,2} ; X Y

X {1,2}, Y {1,2} ; X Y

12

Traditional use of constraints

Quantitative constraints–Integer domain

–Reduction by arithmetic relations•Greater than (>)•Less than (<)•Equals (=)

–Example(x < y ; x [0..10], y [5..10] )

(x + y = z 3 , x = u + 1 ; x , y , z , u )

13

Solving a Constraint Satisfaction Problem

•Problem SEND

+ MORE = MONEY

•Modeling 1000 x S + 100 x E + 10 x N + D

+ 1000 x M + 100 x O + 10 x R + E= 10000 x M + 1000 x O + 100 x N + 10 x E + Y

•Domain reduction / Search•Solution

S=9, E=5, N=6, D=7, M=1, O=0, R=8, Y=2

14

Quantitative Constraints in Multimedia

…Communicative devices generate constraint-graph which the system tries to satisfy…

15

Drawbacks of quantitative constraints

•Too many (trivial) solutions that differ by:–1 pixel position, or–1 milliseconds in timing

•Not sufficiently expressive•cannot specify “no overlap” constraint

•Too low level•A.X2 B.X1

16

Allen’s 13 temporal relations

Allen’s relations are used for both spatial and temporal lay-out

17

Solution: qualitative constraints

•For non-typical domains–Boolean, –Three valued logics, –Allen’s relation

•Advantages for Multimedia generation:–More intuitive–More expressive–Smaller domains

18

Domain Reduction Rules

•InverseA before B B after AA equal B B equal A

•TransitiveA before B , B before C A before CA overlaps B, B during C A overlap C or

A during C or A starts C

•EqualsA overlap C, A [o,d,s] C A overlap C

19

Qualitative Constraints

…Qualitative solutions translate automatically to lower level quantitative constraints…

20

New problem: What if constraints are insoluble?

•Combine Prolog unification and backtracking with constraint solving

•Use Prolog rules to generate constraints•Backtrack when constraints are insoluble

Solution: Constraint Logic Programming

21

Cuypers generation engine

•Multiple layers:

–Communicative devices

generate constraints

–Qualitative constraints

translate to quantitative

constraints

–Solution of both constraints

provides sufficient

information for final

presentation

22

Cuypers demo: scenario

•Client

•Server•Server

•Server

•Server•Client

User is interested in Rembrandt and wants to know about about the “chiaroscuro” technique

Query database

Generate constraints according to:–System profile–User profile–Network profile

Solve constraints / revise constraints

Generate SMIL presentationPlay presentation

23

Conclusions

•Quantitative constraintsare insufficient for automatic multimediapresentation generation. Also need

•Qualitative constraintsto allow intuitive and effectivehigh level specification, and

•Backtrackingfor revising specific constraintswhich otherwise cause the entire set to fail

24

Discussion

•Labeling– Choice of candidate variable– Choice of candidate value

•Transitive Reasoning Rule– Infer implicit relations– Redundant

•Allen’s Relations– Not very well suited for generating MM – Non interactive

25

Future directions

•Best-first instead of depth-first–Choose “best” among possible solutions–Needs evaluation criteria

•Improve knowledge management–Make design knowledge declarative and explicit

–Preserve metadata in final presentation–Use standardized and reusable profiles

26

Thank you