mmir challenges

50 %
50 %
Information about mmir challenges
Entertainment

Published on November 15, 2007

Author: Elliott

Source: authorstream.com

Content-based Multimedia Information Retrieval: Challenges & Opportunities:  Content-based Multimedia Information Retrieval: Challenges & Opportunities Stefan Rüger et al http://km.doc.ic.ac.uk Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation Need for Information Retrieval :  Need for Information Retrieval Information is of no use unless you can actually access it. Multimedia Information Retrieval:  Multimedia Information Retrieval archive text, video, images, speech, music, combinations query text, stills, sketch, speech, humming, examples content-based present results browsing, summaries, story boards document clustering, cluster summaries utilise relevance feedback Query-retrieval matrix:  Query-retrieval matrix text video images speech music sketches multimedia text stills sketch speech sound humming examples query doc Example Some applications:  Some applications medicine get diagnosis of cases with similar scans law enforcement child pornography prosecution copyright infringement (music, videos, images) CCTV video retrieval (car park, public spaces) digital libraries searching, visualisation, summaries, browsing Example: get me similar images!:  Example: get me similar images! extract, eg, 50,000 primitive features provide positive image examples, generate negative examples at random Feature selection & learning ADA-Boost, K-NN, SVM, ... eg, compute separating hyper-plane and rank all images in database accordingly Example: Jupiter video search:  Example: Jupiter video search video segmentation: generate paragraphs identify key frame of video paragraph get Jupiter example images, eg, from web Google image search: treat video search as image search [with Marcus Pickering and David Sinclair, CVIR 2002] Result list of video key frames:  Result list of video key frames Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation The semantic gap:  The semantic gap Bridging the semantic gap:  Bridging the semantic gap region segmentation + region classification (grass, water, ...) using simple models for complex concepts (grass+plates+people = barbeque) Region segmentation:  Region segmentation collaboration with AT&T Research, Cambridge Region classifiers:  Region classifiers visual categories grass, sky (blue), sky (cloudy), skin, trees, wood, water, sand, brick, snow, tarmac give regions a probability of membership Positive Examples Negative Examples Cluster Prune Cluster Nearest Neighbours Test region Probability Cluster Example: grass classifier:  Example: grass classifier Modelling semantic concepts:  Modelling semantic concepts outdoor town crowd sky grass skin tarmac Bayesian networks Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation Polysemy:  Polysemy old Volkswagen colour contrast road signs outback Relevance feedback:  Relevance feedback system needs plasticity (parameters) images are quickly assessed and user can inform system explicitly or implicitly system needs to learn from user = change the parameters Relevance feedback mechanism :  Relevance feedback mechanism centre = query = ideal result results are displayed such that distance to centre is the dissimilarity to the query user indicates her/his idea of similarity by rearranging the displayed results system recomputes optimal parameters for this specific query automatically Example: relevance feedback:  Example: relevance feedback query initial result User action:  User action After relevance feedback:  After relevance feedback number of relevant images has doubled GUI:  GUI User modelling:  User modelling simulate users who click at most three images mean average precision increase - weight space movement: 15% - query change and weight change: 58% [with Daniel Heesch, ECIR 2003] Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation The “multi” of multimedia:  The “multi” of multimedia high-level features words and phrases from text, speech recognition medium-level features face detector, regions classifiers, outdoor etc low-level features Fourier transforms, wavelet decomposition, texture histograms, colour histograms, shape primitives, filter primitives Unified theoretical framework:  Unified theoretical framework document network index time run time query network Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation System overview:  System overview [with M Pickering, D Heesch, R O’Callaghan and D Bull, TREC 2002] TREC 2002 evaluation: 10 best manual runs:  TREC 2002 evaluation: 10 best manual runs [with M Pickering, D Heesch, R O’Callaghan and D Bull, TREC 2002] Video Summary:  Video Summary story-level segmentation keyframe summary videotext summary full-text search named entities [with L Wong and M Pickering] Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation Polyphonic Music Indexing Technique:  Polyphonic Music Indexing Technique n-grams encode music as text strings using pitch and onsets index text words with text search engine process query in the same way application: eg, Query by Humming [with Shyamala Doraisamy, ISMIR 2000, ISMIR 2001, ISMIR 2002] Monophonic pitch n-gramming :  Monophonic pitch n-gramming 0 +7 0 +2 0 -2 0 -2 0 Interval: Example: musical strings with interval-only representation [0 +7 0 +2] ZGZB [+7 0 +2 0] GZBZ [0 +2 0 -2] ZBZb N-grams and polyphony:  N-grams and polyphony Polyphony: index all monophonic combinations Encoded rhythm in similar way Performed well with known-item search Studied fault-tolerance Content-based MM IR:  Content-based MM IR Multimedia Information Retrieval aims, applications and a retrieval example Challenges semantic gap polysemy the “multi” in multimedia Video Search and Summarisation Music Retrieval Information Navigation Presentation of search results:  Presentation of search results ranked list adequate? [funded by NSF-EU: Cultural Heritage Language Technologies] [with D Heesch et al] Vision: labelled clusters:  Vision: labelled clusters suggest keywords refine query drill down/up Keyword computation:  Keyword computation example: search for “computer” related keywords: “hardware”, “software”, “IBM”, “Linux”, etc Document representation:  Document representation word histogram vectors (“bag of words”) cost dog drug hospital hunt impact mafia reform … vocabulary doc1 doc2 … New document representation:  New document representation use keywords only for returned documents low-dimensional vector (10-30 dim) efficient clustering no curse of dimensionality Slide43:  Sammon Tree-Map:  Tree-Map Slide45:  DendroVis Slide46:  Radial Slide47:  Radial Conclusions:  Conclusions Multimedia Information Retrieval Challenging research questions Draws on computer vision, audio processing, natural language analysis, unstructured document analysis, information retrieval, information visualisation, computer human interaction, artificial intelligence Collaborations:  Collaborations part of the High Performance Informatics area existing collaborations with Tufts’s Perseus Digital Library Imperial’s Newton Project AT&T Research, Cambridge ISE Dept of the Ben Gurion University, Israel EE Dept of Bristol University the Greenstone Digital Library, U of Waikato, NZ intended collaborations with Center for Intelligent Information Retrieval, Umass EIE Dept of Hong Kong Polytechnic University Content-based Multimedia Information Retrieval: Challenges & Opportunities:  Content-based Multimedia Information Retrieval: Challenges & Opportunities Stefan Rüger et al http://km.doc.ic.ac.uk The semantic gap:  The semantic gap Rhythm encoding:  Rhythm encoding we use ratios, not absolute values and onset time differences, not durations ri = (oi+2 - oi+1)/(oi+1 - oi) we quantise this number (use 21 letters) this is already invariant to tempo change Keyword computation:  Keyword computation potentially interesting for the user related to the returned documents able to discriminate the returned documents candidate keywords: medium document freq rank words with (h/d)  h log(|H|/h) h returned-document frequency d document frequency H returned-document set keywords: highly ranked candidates Hierarchical clustering:  Hierarchical clustering Slide55:  drill down DendroVis

Add a comment

Related presentations

Related pages

Wer kennt gute Challenges? (Hilfe, Challenge)

Hi, ich will mit meiner Schwester lustige Challenges machen. Solche wie "Baby Food Challenge" oder die "Chubby Bunny" Challenge. Würde mich ...
Read more

SC:Community

Über die Ski Challenge Willkommen beim größten virtuellen Skirennen der Welt! Ski Challenge begeistert Jahr für Jahr tausende Spieler aus einer ...
Read more

Home | ERP-Challenge

Willkommen auf erp-challenge.de. Um eine Liste mit den für Sie verfügbaren Kursen zu erhalten, loggen Sie sich bitte ein und wählen Sie im Menü den ...
Read more

ALS Ice Bucket Challenge – Wikipedia

Die ALS Ice Bucket Challenge (deutsch Eiskübelherausforderung) war eine als Spendenkampagne gedachte Aktion im Sommer 2014. Sie sollte auf die ...
Read more

dict.cc Wörterbuch :: challenges :: Deutsch-Englisch ...

Englisch-Deutsch-Übersetzung für challenges im Online-Wörterbuch dict.cc (Deutschwörterbuch).
Read more

Lustige Challenges für Youtube (Video) - Gute Fragen ...

Ich möchte gerne viedios über meine Tiere drehen bzw. Haustiere und challenges möchte ich auch machen nun ihr?
Read more

Home - The Big Challenge

Besuchen Sie uns auf. Facebook Facebook Facebook Der Wettbewerb. ... The Big Challenge Bockenheimer Landstraße 17/19, 60325 Frankfurt am Main 069-710 423 240
Read more

MAD MONDAY CHALLENGE

Eine tolle Saison 2015 liegt hinter uns, die Meister sind gekürt, der letzte Pokal und alle Preise vergeben. Hiermit wollen wir uns nochmal bei allen ...
Read more

Vertraust Du mir? - CHALLENGE (NEU) Mit Paola Maria - YouTube

Vertraust du mir? - CHALLENGE - Falls ihr Inspirationen für neue Challenges habt, einfach in die Kommentare schreiben. Musik: 5xL Beats **MEINE ...
Read more

The Big Challenge – Wikipedia

The Big Challenge (engl.: Die große Herausforderung) ist ein europaweiter Englischwettbewerb für Schüler der 5. bis 9. Klasse. Bei dem jährlich im Mai ...
Read more