advertisement

qa overview cmu

50 %
50 %
advertisement
Information about qa overview cmu
Entertainment

Published on October 16, 2007

Author: Abbott

Source: authorstream.com

advertisement

Overview of Question Answering :  Overview of Question Answering Svetlana Stenchikova “QA with Semantic Roles” - Internship project at AT&T with the mentors: Dilek Hakkani-Tür and Gokhan Tur Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition QA is different from Search:  QA is different from Search Example: Where did Maya people live? :  Example: Where did Maya people live? the Maya people lived in: a large cultural area that archaeologists call Mesoamerica south-central Mexico the region that is now Eastern and Southern Mexico, Guatemala, Belize, El Salvador and Western Honduras small scattered villages of pole and thatch houses QA System: QA Scenarios:  QA Scenarios Limited Domain – Fixed set of documents Information is more reliable but sparse Data preprocessing is feasible Unlimited domain – Web Greater redundancy, but lower reliability On-demand data processing TREC competition Slide6:  QA System Architecture Related Work:  Related Work Insight(Russia): extensive use of patterns START (MIT): first QA system on the web. Uses Knowledge Annotation Technique: annotating resources with metadata Use specific data source for different queries Slide8:  Related Work AskMSR: count 1-, 2-, 3-grams on the snippet outputs, finds the most frequent item. Use answer tiling to generate longer answers Insight(Russia) START (MIT) Slide9:  Related Work AskMSR Insight(Russia) LCC: Extract Logical Form from questions and answers use ontology to “paraphrase” use logic provers for answer verification Search for answer in a paragraph instead of a sentence START (MIT). Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition Slide11:  QA System Architecture Who won the Nobel prize in literature in 1988? “won the Nobel prize in literature in 1988” http://www.nytimes.com/books/99/04/18/specials/rushdie-rally.html http://www.newyorker.com/printables/fact/040712fa_fact1 … Naguib Mahfouz won the nobel prize in literature in 1988 Egyptian writer won the nobel prize in literature in 1988 A distinguished Egyptian writer Naguib Mahfouz won the nobel prize in literature in 1988 Naguib Mahfouz Egyptian writer 2/3 1/3 Candidate Sentences – Actual Scenario:  Candidate Sentences – Actual Scenario The veritable terrorism of which he is a target is unjustifiable , indefensible , " wrote Naguib Mahfouz , the Egyptian who won the Nobel Prize in Literature in 1988 Mahfouz , who won the Nobel Prize in Literature in 1988 , is known for his intimate stories of Egyptian life … Slide13:  Phrase Extraction and Search Methods Search: find candidate documents Exact Phrase Conjunction of sub-phrases (inexact phrase) Slide14:  Phrase Extraction and Search: Exact Phrase If a question contains an auxiliary verb (is/was/were/are/did/do/does): drop all words between the 'wh' word and the auxiliary match the tense of the predicate with the auxiliary verb. When did Vesuvius last erupt? ~ “Vesuvius last erupted” What state does Martha Stewart live in ~ “Martha Stewart lives in” Slide15:  Phrase Extraction and Search : Inexact Phrase SRL-parse the question, identifying the argument phrases: Who invented the electric guitar? [ARG0 Who] [TARGET invented] [ARG1 the electric guitar] Search for “invented” AND “the electric guitar” Searching: Use Google search API Slide16:  Sentence Extraction Methods Sentence Extraction: finding candidate sentences Exact phrase Conjunction of phrases Predicate Evaluation of Search and Sentence Extraction:  Evaluation of Search and Sentence Extraction Accuracy: % of candidate sentences containing a correct answer Coverage: % of the queries containing at least one correct answer in the set of candidate sentences Slide18:  SRL can improve the result ONLY when a correct answer is present in candidate sentences. Future work: Improve Search/extraction Using synonyms for search and extractions Back-off on prepositional clauses Future Work on Sentence Extraction Baseline Answer Extraction:  Baseline Answer Extraction Single template: Answer appears on the left of the right side of the search phrase Question: Who invented electric guitar? Phrase: “invented the electric guitar” Pattern: […] invented the electric guitar Question: When was Nostradamus born? Phrase: “Nostradamus was born” Pattern : Nostradamus was born in […] SRL Architecture:  SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Parse: [ARG0 Who] [TARGET created ] [ARG1 the comic strip Garfield] Task: Find ARG0 of the TARGET created Semantic Role module Semantic Role module Slide21:  SRL Architecture Q: Who created the comic strip Garfield? Predicate: created Argument type: ARG0 SRL Candidate Sentence: Garfield is [ARG1 a popular comic strip] [TARGET created ] [ARG0 by Jim Davis] featuring the cat Garfield ... Extracted Answers: by Jim Davis Semantic Role module Semantic Role module Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer:  Automatic Evaluation: Mean Reciprocal Ranking (MRR) and precision of correct answer 0.24 => 0.35 is a statistically significant improvement MRR: question gets score = 1/N where N is the index of the first correct answer Use 5 top answers Precision(% questions with first answer being correct) Measure of Conciseness and Irrelevant Information in Correct Answers:  Measure of Conciseness and Irrelevant Information in Correct Answers * This experiment was done manually Presentation Plan:  Presentation Plan Overview of Question Answering QA Baseline Approach TREC competition TREC competition:  TREC competition TREC competitions QA track ~ Since 1999 Set of factoid/definition/context questions. Data Set 3 GB Slide26:  TREC history TREC 2004 - 2006 main task:  TREC 2004 - 2006 main task Define a topic target for a set of questions: Person George Bush Thing Shiite Organization Amway Event return of Hong Kong to Chinese sovereignty (Event was introduced in 2005) Question Types: List Factoid Other The answer to the "Other" question is to be interesting information about the target that is not covered by the preceding questions in the series. TREC Main Task Question Example:  TREC Main Task Question Example target text="Shiite“ FACTOID    Who was the first Imam of the Shiite sect of Islam?  FACTOID  Where is his tomb? FACTOID    What was this person's relationship to the Prophet Mohammad? FACTOID    Who was the third Imam of Shiite Muslims? FACTOID  When did he die? FACTOID  What portion of Muslims are Shiite? LIST What Shiite leaders were killed in Pakistan? OTHER    Other Scoring :  Scoring 1/3*factoid-score + 1/3*list-score + 1/3*other-score. Factoid answers are judged as: "incorrect", "unsupported", "non-exact", "locally correct", or "globally correct" factoid-score= % "globally correct" Factoid and List Answer Scoring:  Factoid and List Answer Scoring Factoid: count an answer as correct if the answer string consists of exactly the right answer (and supported by the document). Report accuracy - % of correct answers List: An individual instance is interpreted as for factoid questions IR = # instances judged correct & distinct/|final answer set| IP = # instances judged correct & distinct/# instances returned F = (2*IP*IR)/(IP+IR) Other Question Scoring:  Other Question Scoring Inspired by: Jimmy Lin and Dina Demner-Fushman. Will Pyramids Built of Nuggets Topple Over? the assessor creates a list of acceptable information nuggets from the union of the returned responses and the information discovered during question development Assessor marks “vital” and “Okay” nuggets present in the system’s answer Precision and recall are computed as a function of # VITAL and # OKAY nuggets in response TREC 2005 “Factoid” Questions Evaluation:  TREC 2005 “Factoid” Questions Evaluation TREC 2005 “List” Questions Results:  TREC 2005 “List” Questions Results TREC 2005 “other” Questions Results:  TREC 2005 “other” Questions Results Complex Interactive QA Track:  Complex Interactive QA Track Questions were created by 4 military analysts Evaluated the same way as “other” in Main QA task In 2005 competition scores range from .061 - .276 Complex Interactive QA Track:  Complex Interactive QA Track What evidence is there for transport of [goods] from [entity] to [entity]? What [relationship] exist between [entity] and [entity]? where [relationship] is a element of {"financial relationships", "organizational ties", "familial ties", "common interests"} What influence/effect do(es) [entity] have on/in [entity]? What is the position of [entity] with respect to [issue]? Is there evidence to support the involvement of [entity] in [event/entity]? Question Example:  Question Example   <topic num="1"> <template id="1"> What evidence is there for transport of [drugs] from [Bonaire] to [the United States]? </template> <narrative> The analyst would like to know of efforts made to discourage narco traffickers from using Bonaire as a transit point for drugs to the United States. Specifically, the analyst would like to know of any efforts by local authorities as well as the international community. </narrative>   </topic> References:  References http://www.umiacs.umd.edu/~jimmylin/ciqa/guidelines.html Overview of the TREC 2005 Question Answering Track E.M. Voorhees, H.T. Dang, NIST http://trec.nist.gov/act_part/tracks/qa/qa.06.guidelines.html THANK YOU:  THANK YOU

Add a comment

Related presentations

Related pages

15-112 Questions and Answers - www.andrew.cmu.edu

Overview. The following questions and answers are edited versions of questions we received and the answers we gave. We don't include every question ...
Read more

Questions and Answers - andrew.cmu.edu

Overview The following questions and answers are edited versions of questions we received via email from students and the answers we gave.
Read more

Capability Maturity Model Integration - Wikipedia, the ...

Capability Maturity Model Integration ... CMU claims CMMI can be used to guide process improvement across a project, ... CMMI Overview Characteristics of ...
Read more

SCS Undergraduate Admissions: Overview / Q&A / Tour ...

Overview of Programs; Course Listings; Undergraduate Programs; Undergraduate Admissions; Masters Programs; Masters Admissions; Doctoral Programs; Doctoral ...
Read more

Risk Management | Overview

CMU; SEI; CERT Division; Digital Library; ... Other applications of risk management principles have been ... A Technical Overview of Risk and Opportunity ...
Read more

Department of Media Relations - Carnegie Mellon University ...

The Marketing and Public Relations department ... Function Room reservation through fmhotline@qatar.cmu ... Qatar Foundation events calendar www.qf.org.qa .
Read more

info.ephyra - cs.cmu.edu

Class Summary; OpenEphyra: OpenEphyra is an open framework for question answering (QA). Overview Package : Class : Use : Tree : Deprecated : Index : Help
Read more

HP Moonshot Administration | QA

HP Moonshot Administration. Page Sections Page Sections. ... Module 1: Course Overview. ... QA Authored We offer over 500 QA ...
Read more

Masonry Contractor QA/QC Plan Sample - cdn2.hubspot.net

PART 2 . QA/QC Plan Sample. 15 Selected pages out of 200 (not the complete plan) Project-Specific Quality Plan QA/QC Plan Cover Page Table of Contents
Read more