alma mater studiorum -universitÀ di bologna … · patella, t. salmoncinotti, d. tarchi, f....

1

ALMA MATER STUDIORUM - UNIVERSITÀ DI BOLOGNA

Multimedia Data Management M

Second cycle degree programme (LM) in Computer EngineeringUniversity of Bologna

Motivations

Home page: http://www-db.disi.unibo.it/courses/MDM/Electronic version: 0.02.Motivations.pdfElectronic version: 0.02.Motivations-2p.pdf

I. Bartolini

1

§Starting from the 90's, the diffusion of inexpensive tools for capturing still images, videos, sound, and other Multimedia (MM) data allowed the creation of massive collections of digital MM content

§Presence of MM data and high dimensionality were both key aspects for§ extending DB technology to “non-conventional” data types,

notably MM data (images, videos, speech, time series, etc.), basically because their digitized representation does not immediately convey their intrinsic reality

§ novel query paradigms, remarkably similarity queries§ This was also the case for text retrieval, starting as early

as the 50's, leading to the “Information Retrieval” technology

Motivations (1)

2I. Bartolini Multimedia Data Management M

2

2

§ Two paradigms evolved for the retrieval of MM content:§Concept-Based Retrieval: uses annotation that have

been manually attached to the datum § Later on in the years (late 00’s) semi-automatically

§Content-Based Retrieval: uses characteristics automatically extracted from each object (features) to characterize its content

Motivations (2)


3

§Concept-based Retrieval (also known as “Text-based” because it borrows from Information Retrieval)

§ Too many MM objects to annotate manually§ High cost of human interpretation§ Subjectivity of visual content

§ e.g., “A picture is worth a thousand words”!

§Content-based Retrieval§automatically retrieves images, video, graphics, animations,

and sound based on the visual and audio content

Motivations (3)


4

3

§Both approaches have their pros and cons§ the former depending on human annotation (which brings

issues of cost, ambiguity, and annotation quality), while § the latter introduces the “semantic gap”, i.e., the difference

between the internal system representation and the user-perceived nature of an object

§ The success of Content-Based Retrieval is due to its dramatic reduction in cost and effort for satisfying user needs and to the development of a number oftechniques able to fill the semantic gap

Motivations (4)


5

History§ First conference on Database Applications of Pictorial

Applications in 1979§ Followed by the final Report to National Science Foundation

(NSF) of the Planning Workshop on Facial Expression Understanding in 1992§ A gap of more than 10 years!

§ The Multimedia domain became a more active field of research since 1997, when Internet technologies and Web search engines became popular and with the beginning of the diffusion of inexpensive tools for capturing MM data§ With a maximum peak in the late 00’s with ubiquitous computing

smart devices, smart environments and smart interaction

Motivations (5)


6

4

§MM data are a much more powerful communication tool than traditional data in our daily life§ TV on demand, TV news, radio news, Web pages, slideshow

presentations, speech, MM messaging mobile applications (e.g., WhatsApp, Telegram, Instagram, etc.), graphic design, 3D movies, etc.

§On the other hand, MM data are much more complex than traditional data

§ There is an urgent need for advanced MM systems organizing and managing such massive, complex, and ubiquitous data types§ Traditional relational databases are ‘no longer’ suitable for

complex MM data management§ Automatic and robust systems which produce, transmit, analyze,

manage, and search MM data in an efficient and effective way are strongly required! J

Motivations (6)


7

§A very diversified field!!§Data Types

§ Textual documents, hypertext, image, audio, graphics, animation, paintings, video/movie, enriched text, spread sheet, slides, combinations of these and user interaction

§Research Problems§ Systems, content, services, user, evaluation, implementation,

social/business, applications§Methodologies

§ Database technologies, information retrieval, signal and image processing, graphics, vision, human-computer interaction, machine learning, statistical modeling, data mining, pattern analysis, data fusion, social sciences, and domain knowledge for applications

Motivations (7)


8

5

Example: advanced (Images) search engines § How do they work?

§ Different types of data/queries w.r.t. traditional (documents) search engine


9

Example: advanced (Maps) search engines § How do they work?

§ Different types of data/queries w.r.t. traditional (documents) search engine


“spatial” query in order to retrieve “nearby” points of interest

10

6

Example: advanced (Maps) search engines § “Nearby” results (i.e., “Restaurants”) to the query “Bologna towers”


11

Example: advanced (Video) search engines § How do they work? Different types of data/queries w.r.t. traditional

(documents) search engine?!?

12I. Bartolini

12

7

Example: social networking platforms§ Users personal MM data sharing


13

Example: mobile social networks§ Users personal MM data sharing

§ E.g., Instagram mobile photo-sharing application and service that allows users to share pictures and videos


14

8

Bibliography (1) the growing list… at 19th February 2020

Books:§ Candan, Sapino. Data Management for Multimedia Retrieval. Cambridge§ Zhang, Zhang. "Multimedia Data Mining: A Systematic Introduction to Concepts and Theory, Chapman

and Hall/CRC§ Chapman & Chapman. Digital Multimedia, Wiley & Sons Ltd§ Colace, De Santo, Moscato, Picariello, Schreiber, Tanca. Data Management for Pervasive Systems,

Springer§ Baeza-Yates, Ribeiro-Neto. Modern Information Retrieval. Addison Wesley § Faloutsos. Searching Multimedia Databases by Content. Kluwer Academic Publishers§ Furht, Smoliar, Zhang. Video and Image Processing in Multimedia Systems. Kluwer Academic Publishers § Rubner, Tomasi. Perceptual Metrics for Image Database Navigation. Kluwer Academic Publishers

Links:§ Google: http://www.google.it/§ Google Images: http://www.google.it/imghp§ YouTube: https://www.youtube.com/§ Vimeo: https://vimeo.com/§ Instagram: https://www.instagram.com/§ Fickr: https://www.flickr.com/

§ Wikipedia: http://en.wikipedia.org§ Wikipedia list of CBIR engines: http://en.wikipedia.org/wiki/List_of_CBIR_engines

§ Windusrf: http://www-db.disi.unibo.it/Windsurf/§ Shiatsu: http://www-db.disi.unibo.it/Shiatsu/§ Pibe: http://www-db.disi.unibo.it/PIBE/§ FeedbackbyPass: http://www-db.disi.unibo.it/FeedbackBypass/


15


Scientific papers:

[BK11] C.A. Bhatt and M.S. Kankanhalli. Multimedia data mining: State of the art and challenges. Multimedia Tools and Applications Journal (MTAP), 2011

[BBP+99] I. Bomze, M. Budinich, P. Pardalos, and M. Pelillo. The Maximum Clique Problem, Vol. 4. Kluwer Academic Publishers, 1999

[BF+10] M. Batko, F. Falchi, et al. Building a Web-Scale Image Similarity Search System. In MTAP, 47(3), 2010[BET08] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-up robust features (SURF). Computer Vision Image

Understanding, 2008[CBD99] J. Chen, C. Bouman, and J. Dalton. Active Browsing Using Similarity Pyramids. In SPIE, 1999[CPZ97] P. Ciaccia, M. Patella, and P. Zezula. M-tree: An Efficient Access Method for Similarity Search in Metric Spaces.

VLDB’97, 1997[CTB+99] C. Carson, M. Thomas, S. Belongie, J. M. Hellerstein, and J. Malik. Blobworld: A System for Region-based

Image Indexing and Retrieval. VISUAL’99, 999[DJL+08] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Image Retrieval: Ideas, Influences, and Trends of the New Age. ACM

Computing Surveys, 2008[Fag96] R. Fagin. Combining Fuzzy Information from Multiple Systems. PODS’96, 1996 [FB+94] C. Faloutsos, R. Barber, et al. Efficient and Effective Querying by Image Content.In JIIS, 3(3/4), 1994[FKK+05] R. Fagin, R. Guha, R. Kumar, J. Novak, D. Sivakumar, and A. Tomkins. Multi-structural Databases. PODS’05,

2005[FR05] D. Fogaras and B. Racz. Scaling Link-based Similarity Search. WWW’05, 2005[FSA+95] M. Flickner and H. S. Sawhney and J. Ashley and Q. Huang and B. Dom and M. Gorkani and J. Hafner and D.

Lee and D. Petkovic and D. Steele and P. Yanker. Query by image and video content: The QBIC system. IEEEComputer, 1995

[GGW03] P. Ganesan, H. Garcia-Molina, J. Widom. Exploiting Hierarchical Domain Structure to Compute Similarity. ACM TODS, 2003

[Gut84] A. Guttman. R-tree: A Dynamic Index Structure for Spatial Searching. ACM SIGMOD’84, 1984[HL11] M. Hilbert and P. Lopez. The World's Technological Capacity to Store, Communicate, and Compute

Information. In Science, 332(6025), 2011


16

9


[HX+11] W. Hu, N. Xie, et al. A Survey on Visual Content-Based Video Indexing and Retrieval. In IEEE TSMC-C, 41(6), 2011

[IBS08] I.F. Ilyas, G. Beskales and M.A. Soliman. A survey of top-k query processing techniques in relational database systems. ACM Computing Surveys, 2008

[KFS+96] F. Korn, N. Sidiropoulos, C. Faloutsos, E- Siegel, and Z. Protopapas. Fast Nearest Neighbor Search in Medical Image Databases. VLDB’96, 1996

[KMX+09] J. Kleban, E. Moxley, J. Xu, and B.S. Manjunath. Global Annotation of Georeferenced Photographs. In CIVR, Santorini, Greece, 2009

[Lin97] D. Lin. Using Syntactic Dependency as Local Context to Resolve Word Sense Ambiguity. ACL/EACL’97,1997[Low99] D. Lowe. Object recognition from local scale-invariant features. IEEE ICCV, 1999[Low04] D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints. In IJCV, 60(2), 2004[LW06] J. Li and J. Z. Wang. Real-time Computerized Annotation of Pictures. In MM, 2006[LSD+06] M. S. Lew, N. Sebe, C. Djeraba, and R. Jain. Content-based Multimedia Information Retrieval: State of the Art

and Challenges. ACM Trans. Multimed. Comput. Commun. Appl., 2006[LZL+07] Y. Liu, D. Zhang, G. Lu, and W. Y. Ma. A Survey of Content-based Image Retrieval with High-level Semantics.

Pattern Recognition, 2007[Mil95] WordNet: A.G. Miller. Wordnet: A Lexical Database for English. Communications of the ACM, 1995[MRS08] C.D. Manning, P. Raghavan, and H. Schutze. Introduction to Information Retrieval. Cambridge University Press,

2008[Nav09] R. Navigli. Word Sense Disambiguation: A Survey. In ACM CSUR, 41(2), 2009[Pos09] S. Poslad. Ubiquitous Computing Smart Devices, Smart Environments and Smart Interaction. Wiley, 2009[PYF+04] J.-Y. Pan, H.-J. Yang, C. Faloutsos, and P. Duygulu. Automatic Multimedia Cross-modal Correlation Discovery.

KDDM’04, 2004[RGL+07] R. Datta, W. Ge, J. Li, and J.Z. Wang. Toward Bridging the Annotation-Retrieval Gap in Image Search. In IEEE

MultiMedia, 14(3), 2007[RHO+98] Y. Rui, T.S. Huang, M. Ortega, and S. Mehrotra: Relevance feedback: A Power Tool for Interactive Content-

based Image Retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 1998[SWS+00] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain: Content-Based Image Retrieval at the End

of the Early Years. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000[TNM07] R. Tye, G. Nathaniel, and N. Mor. Towards Automatic Extraction of Event and Place Semantics from Flickr

Tags. In SIGIR, 2007


17


[ABB+18] O. Andrisano, I. Bartolini, P. Bellavista, A. Boeri, L. Bononi, A. Borghetti, A. Brath, G.E. Corazza, A. Corradi, S. De Miranda, F. Fava, L. Foschini, G. Leoni, D. Longo, M. Milano, F. Napolitano, C.A. Nucci, G. Pasolini, M. Patella, T. Salmon Cinotti, D. Tarchi, F. Ubertini, D. Vigo. The Need of Multidisciplinary Approaches and Engineering Tools for the Development and Implementation of the Smart City Paradigm. Proceedings of the IEEE, 2018

[ABP99] S. Ardizzoni, I. Bartolini and M. Patella. Windsurf: Region-Based Image Retrieval Using Wavelets. IWOSS’99, 1999

[Bar05] I. Bartolini. Context-based Image Similarity Queries. In Adaptive Multimedia Retrieval: User, Context, and Feedback, Vol. 3877, 2006

[Bar09a] I. Bartolini. Image Querying. In Encyclopedia of Database Systems, Springer, 2009[Bar09b] I. Bartolini. Multi-faceted Browsing Interface for Digital Photo Collections. CBMI’09, 2009 [Bar12] I. Bartolini. Content Meets Semantics: Smarter Exploration of Image Collections - Presentation of Relevant Use

Cases. SIGMAP’12, 2012 [Bar15] I. Bartolini. Similarity-based Image Retrieval for Revealing Forgery of Handwritten Corpora. SIGMAP’15, 2015[BC03] I. Bartolini and P. Ciaccia. MuSIQUE: A Multi-System Image Querying User Interface. SEBD’03, 2003[BC08a] I. Bartolini and P. Ciaccia. Imagination: Exploiting Link Analysis for Accurate Image Annotation. In Adaptive

Multimedia Retrieval: Retrieval, User, and Semantics, Vol. 4918, 2008[BC08b] I. Bartolini and P. Ciaccia. Scenique: A Multimodal Image Retrieval Interface. AVI’08, 2008 [BC10a] I. Bartolini and P. Ciaccia. Automatically Joining Pictures to Multiple Taxonomies. In SEBD’10, 2010[BC10b] I. Bartolini and P. Ciaccia. Multi-dimensional Keyword-based Image Annotation and Search. In KEYS’10, 2010[BCC+06] I. Bartolini, P. Ciaccia, L. Chen and V. Oria. A Meta-Index to Integrate Specific Indexes: Application to

Multimedia. DMS’06, 2006[BCO+07] I. Bartolini, P. Ciaccia, V. Oria and T. Özsu. Flexible Integration of Multimedia Sub-queries with Qualitative

Preferences. Multimedia Tools and Applications Journal (MTAP), 2007[BCP00] I. Bartolini, P. Ciaccia and M. Patella. A Sound Algorithm for Region-Based Image Retrieval Using an Index.

QPMIDS’00, 2000[BCP05] I. Bartolini, P. Ciaccia, and M. Patella. WARP: Accurate Retrieval of Shapes Using Phase and Time Warping

Distance. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2005[BCP06] I. Bartolini, P. Ciaccia and M. Patella. Adaptively Browsing Image Databases with PIBE. Multimedia Tools and

Applications Journal (MTAP), 2006


18

10


[BCP07] I. Bartolini, P. Ciaccia and M. Patella. PIBE: Manage Your Images the Way You Want! ICDE’07, 2007

[BCP+09] I. Bartolini, P. Ciaccia, M. Patella, and G. Stromei. The Windsurf Java Library. Copyright (c) 2009 DEIS - Alma Mater Studiorum, Università di Bologna, Italy, 2009. Web: http://www-db.deis.unibo.it/Windsurf/

[BCP10] I. Bartolini, P. Ciaccia, and M. Patella. Query Processing Issues in Region-Based Image Databases. Knowledge and Information Systems (KAIS), 2010

[BCP18] I. Bartolini, P. Ciaccia, and M. Patella. Multimedia, Similarity, and Preferences: Adding Flexibility to Your Information Needs. A Comprehensive Guide Through the Italian Database Research, 2018 (invited book chapter)

[BCW00] I. Bartolini, P. Ciaccia and F. Waas. Using the Wavelet Transform to Learn from User Feedback. DELOS'00, 2000

[BCW01] I. Bartolini, P. Ciaccia and F. Waas. FeedbackBypass: A New Approach to Interactive Similarity Query Processing. VLDB’01, 2001

[BDL17] I. Bartolini, A. Di Luzio. Towards Automatic Recognition of Narcolepsy with Cataplexy. In MoMM’17, 2017[BDL19] I. Bartolini, A. Di Luzio. ALDO: An Innovative Digital Framework for Active E-Learning. In ACM ICETC’19, 2019.

Best Presentation Award[BLP+12] I. Bartolini, C. Luzzi, M. Patella, A. Pompilio, C. Ruini. MIMESIS: A Semantic-based Approach for Poetry in

Music Digital Libraries. ECLAP'12, 2012.[BMP+13] I. Bartolini, V. Moscato, R. G. Pensa, A. Penta, A. Picariello, C. Sansone, and M. L. Sapino. Recommending

Multimedia Objects in Cultural Heritage Applications. MM4CH’13, 2013[BMP+14] I. Bartolini, V. Moscato, R.G. Pensa, A. Penta, A. Picariello, C. Sansone, and M.L. Sapino. Recommending

Multimedia Visiting Paths in Cultural Heritage Applications. Multimedia Tools and Applications Journal (MTAP), 2014

[BP00] I. Bartolini and M. Patella. Correct and Efficient Evaluation of Region-Based Image Search. SEBD’00, 2000[BP15] I. Bartolini and M. Patella. Multimedia Queries in Digital Libraries. Data Management in Pervasive Systems,

Series: Data-Centric Systems and Applications, Springer, 2015[BP18a] I. Bartolini and M. Patella. A General Framework for Real-time Analysis of Massive Multimedia Streams. In

Multimedia Systems, 2018[BP18b] I. Bartolini and M. Patella. Windsurf: the best way to SURF - (and SIFT/BRISK/ORB/FREAK, too). In Multimedia

Systems, 2018. [BP19] I. Bartolini, M. Patella. Real-Time Stream Processing in Social Networks with RAM3S. In Future Internet, 2019.

19I. Bartolini Multimedia Dta Management M

19


[BPL+18] I. Bartolini, F. Pizza, A. Di Luzio, E. Antelmi, S. Vandi, and G. Plazzi. Automatic Detection of Cataplexy. In SleepMedicine, 2018.

[BPR10] I. Bartolini, M. Patella, and C. Romani. SHIATSU: Semantic-Based Hierarchical Automatic Tagging of Videos by Segmentation using Cuts. AIEMPro’10, 2010

[BPR13] I. Bartolini, M. Patella, and C. Romani. SHIATSU: Tagging and Retrieving Videos without Worries. Multimedia Tools and Applications Journal (MTAP), 2013.

[BPS11] I. Bartolini, M. Patella, and G. Stromei. The Windsurf Library for the Efficient Retrieval of Multimedia Hierarchical Data. SIGMAP’11, 2011. Best Paper Award

[BR10a] I. Bartolini and C. Romani. SHIATSU: Annotating Your Videos the Easy Way!. SISAP ‘10, 2010[BR10b] I. Bartolini and C. Romani. Efficient and Effective Similarity-based Video Retrieval. SISAP ‘10, 2010[BZP11] I. Bartolini, Z. Zhang, and D. Papadias. Collaborative Filtering with Personalized Skylines. In TKDE, 23(2), 2011[MB+16] P. Montanari, I. Bartolini, et al. Pattern Similarity Search in Genomic Sequences. IEEE TKDE, 28(11), 2016[RCB15] W. Rao, L. Chen, and I. Bartolini. Ranked Content Advertising in Online Social Networks. World Wide Web:

Internet and Web Information Systems Journal (WWWJ), 18(3): 661-679, 2015[TBC+13] F. Tomasi, I. Bartolini, F. Condello, M. Degli Esposti, V. Garulli, and M. Viale. Towards a taxonomy of suspected

forgery in authorship attribution field. A case: Montale’s Diario Postumo. DH-CASE’13, 2013


20

alma mater studiorum -universitÀ di bologna … · patella, t. salmoncinotti, d. tarchi, f....

Documents