RapidKnowledge Formation

Information Extraction and Transport, Inc.with

Veridian PSR,GMU / IB3,

and Paul Cohenunder the direction of

Murray Burke, DARPA

RKF Evaluation Issues

August 2, 2000

RapidKnowledge FormationTKCP Spec

• TKCP spec V1.1 issued

• Y1 content = ECB Chapter 7 (most)

• SQs issued– ECB– Komachi– IET

• Chapters 5 & 6 (portions)• Chapter 7 (reserved)

RapidKnowledge FormationQuestion Ground Rules

• IET’s V1.1 spec is “liberal…”– …w.r.t. KR&R issues– …w.r.t. source “difficulty” ratings

• Teams to identify KR&R “red flags” / issues– Textbook content– SQ-introduced issues & SQ “version space”

• Accepted SQs• Rejected SQs

– “Consensus” ontology of question KR&R issues• Spec includes initial issues assessment for SQs

– Initial feedback milestones: Nov 3, Dec 1

RapidKnowledge FormationPump Priming

• No viable alternatives

• Knowledge for its own sake– Prerequisite, target, fundamental primitives,

common sense– Concepts, relations, axioms

• Knowledge for tools’ sake– Lexical, grammatical…

• Ground rules– “Right” division of labor over KEs, SMEs– Anticipated RKF solutions not ready yet

RapidKnowledge Formation

IET’s SME LabIET’s SME Lab

Process ModelingProcess ModelingProcess ModelingProcess Modeling

E2E Evaluation Strategy

Test Taker(SME)

CPSample Test

CheckpointKB

CheckpointKB

Jun 2000Evaluation

Begins

Aug 2000Evaluation

Ends

Integration Teams

KBPrior

Tools

KB

KBDelta

Integrator KEs

Gatekeeper KE

Sep 2000SpecsReady

RapidKnowledge FormationExperimental Conditions

• User type: KE vs. SME

• KR tool: analogy, sketch, dialog…

• TQ: descriptive, model-based, what-if…

• Pump priming: primitives, prerequisite…