mediahub: an intelligent multimedia distributed platform hub

Download MediaHub: An Intelligent MultiMedia Distributed Platform Hub

Post on 16-Mar-2016




0 download

Embed Size (px)


MediaHub: An Intelligent MultiMedia Distributed Platform Hub. MediaHub. Glenn Campbell, Tom Lunney, Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee Campus Northland Road, Derry { Campbell-g8, TF.Lunney, P.McKevitt} - PowerPoint PPT Presentation


  • MediaHub: An Intelligent MultiMedia Distributed Platform Hub Glenn Campbell, Tom Lunney, Paul Mc Kevitt

    School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee CampusNorthland Road, Derry

    {Campbell-g8, TF.Lunney, P.McKevitt}

  • OutlineGoals and objectivesKey research problemsDistributed ProcessingDistributed PlatformsArchitecture of MediaHubTools and future development

  • Project goals

    The primary objectives of this research are to:

    Interpret/generate semantic representations of multimodal input/outputPerform decision-making (fusion and synchronisation) over multimodal dataImplement MediaHub, a multimodal platform hub

  • Project objectivesFocus on following research questions:

    Will MediaHub use frames for semantic representation or XML or one of its derivatives?How will MediaHub communicate with various elements of a platform?Will MediaHub constitute a blackboard or non-blackboard model?What mechanism will be implemented for decision-making within MediaHub?

  • Key research problemsSemantic RepresentationRepresent language and visionFrames or XML?Semantic StorageBlackboard model?Non-blackboard model?Decision-makingFusion and synchronisationAI technique

  • Frames (CHAMELEON)

    [MODULEINPUT: inputINTENTION: intention-typeTIME: timestamp][SPEECH-RECOGNISERUTTERANCE:(Point to Hannes office)INTENTION: instruction!TIME: timestamp][GESTUREGESTURE: coordinates (3, 2)INTENTION: pointingTIME: timestamp] XML (M3L, SmartKom)

    list epg_browse

    now 2003-03-20T19:42:32 2003-03-20T22:00:00

    2003-03-20T19:50:00 2003-03-20T19:55:00 Todays Stock News ARD.. Semantic representation

  • Semantic storageBlackboard or Non-blackboard?

    High coupling Blackboard?Low coupling - distributed architecture?

    CommunicationVia central blackboard?Message passing between modules?

  • Decision-making (fusion & synchronisation)


    Potential for Other AI techniques

    Fuzzy LogicNeural NetworksGenetic AlgorithmsBayesian Networks (CPNs)

  • Distributed processing PVM (Parallel Virtual Machine) (Sunderam 1990, Fink et al. 1995) ICE (Amtrup 1995) DACS (Fink et al. 1995, 1996) Open Agent Architecture (OAA) (Cheyer et al. 1998, OAA 2004) JATLite (Kristensen 2001, Jeon et al. 2000) JavaSpaces (Freeman 2004)CORBA (Vinoski 1993)

  • Intelligent Multimedia Distributed Platforms Blackboard Model:

    Ymir (Thrisson 1999)

    CHAMELEON (Brndsted et al. 1998, 2001)

    Smartkom (Bhler et al. 2002, Wahlster et al. 2001, SmartKom 2004)

    DARBS (Nolle et al. 2001)

    DARPA Galaxy Communicator (Bayer et al. 2001)

    Psyclone (Psyclone 2004)

    Spoken Image/SONAS ( Nuallin et al. 1994, Nuallin & Smith 1994, Kelleher et al. 2000)

  • Intelligent Multimedia Distributed PlatformsNon-blackboard Model:

    WAXHOLM (Carlson et al. 1996)

    AESOPWORLD (Okada 1996)

    COLLAGEN (Rich et al. 1997)

    INTERACT (Waibel et al. 1996)

    Oxygen (Oxygen 2004)

    EMBASSI (Kirste 2001, EMBASSI 2004)

    MIAMM (MIAMM 2004)

  • CHAMELEONLanguage & vision integration systemconsists of ten modules, mostly programmed in C and C++DACS communication system used for communicationBlackboard stores semantic representations produced by other modules Communication between modules achieved by exchanging semantic representations between themselves or blackboard Semantic representation in form of input, output and integration frames

  • Architecture of CHAMELEON

  • SmartKom

    User adaptive interface for human-computer interaction


    Facilitates speech, gestures and facial expression input

    XML-based mark-up language, M3L, used for semantic representation

    Distributed multiple blackboard model

  • Architecture of SmartKom

  • Dialogue ManagerActs as a blackboard moduleFacilitates communication between other modulesSynchronisation

    Semantic Representation DatabaseProvides semantic representation of language and vision data

    Decision Making ModuleAI technique for a unique form of decision-makingBayesian Networks (CPNs)Neural Networks, Genetic Algorithms, Fuzzy Logic

    Project proposal

  • Architecture of MediaHub

  • Comparison of Intelligent MultiMedia Platforms

  • Software AnalysisMain Programming LanguageJavaC++

    Semantic RepresentationXMLXHTML + VoiceSMILRDF SchemaMPEG-7

    Decision MakingHUGIN (Bayesian Networks) (Hugin 2004) FuzzyJ Toolkit (Fuzzy Logic) (NRC 2004)

  • Project Schedule

  • ConclusionAn intelligent multimodal distributed platform hub called MediaHub will be developed

    MediaHub will interpret and generate semantic representations of multimodal input and output

    MediaHub will perform fusion and synchronisation of language and vision data

    Unique contribution of MediaHub is to provide a new method of decision making

    MediaHub will be tested within an existing multimodal platform (e.g. CONFUCIUS)

    - Software tools for developing the server and client sides of TeleMorph (&TeleTuras)OUTPUT:(1) SMIL Media Player - multimodal presentation (animations, graphics, text, audio)(2) TTS engine (synthetic speech output) Insufficient bandwidth - audio/speech replaced by ordinary text (->TTS). (JSML = JSAPI Markup Language)(3) Agent - modalities that are the natural for face-to-face human communication TeleTuras - very useful on an information aid for tourists (about sites, points of interest, and route planning)INPUT:(1) speech recognition - remotely by ViaVoice(2) Capture text & haptic deixis input (Lo / Hi-level UI API)(3) Communicate user input data -> Media Analysis Communicate user prefs -> Constraint Info Retrieval Communicate device status -> Constraint Info RetrievalSERVER:(1) XML based combination - semantic representation(2) reasoning/decision making - Constraints Processor(3) middleware - integrate modules