Visual Reasoning With Graphs

Yusuf PisanQualitativeReasoningGroup,The Institutefor theLearningSciences

NorthwesternUniversity, 1890 MapleAvenue,Evanston,IL 60201,USAe-mail: y-pisan@nwu.edu

A bstractUnderstandingdiagramsis an important part of human cognition. Computerprogramsneedto understandand reasonusing diagramsto communicateeffectivelywith people. This paperexplainshow line graphscan be interpretedin a domainindependentmanner. We presentacomputerprogramcalled SKETCHY that reasonsabout physicalphenomenavisually by using line graphs. SKETCHY can interpretgraphs to recover functional relationships,answercomparativeanalysis questionsand generatequalitativedescriptionsusing geometricmodels.

1 Introduction

Peopleusediagramsto solve problems,to giveexplanations,to summarizeinformation and torepresentspatial relations. Computerprogramsthat can represent,interpret and reasonwithdiagramswill haveagreat impacton education,cognitive scienceandartificial intelligence. Bymaking spatial relationshipsexplicit, diagramscan reducethe amountof searchand inferencerequired[I21. Diagramsserveboth asdevicestoaid in visualizationof the situation[I4] and asshort-term fast accessmemory devices forholding information[7]. Althoughwe do not yethave computerprogramsthat can do generaldiagrammaticreasoning,diagramshave beensuccessfullyintegratedwith computerprogramsin a number of areas: to explain complexmechanicaland dynamic sysiems[5,2,6,8),tosolve geometryand physicsproblems[7,131,toconstrainthe searchspacein problemsolving[9)and to understandtherole of visual reasoninginproblemsolving[151.

Diagram understandingrequiresbeing able toidentify objects,determinethe relevantfeaturesfor a particularproblemand map the graphicalfeaturesto the domain. A graph is aspecializedform ofdiagrammaticrepresentationthat doesnotinvolve object recognition. Different graphformatsemphasizedifferentrelationshipsbetweenvariablesf101. For instance,pie graphsare usedto show relativepercentages,bar graphsand stepgraphsto show relativeamounts,scatterplots toshow trendsin data and line graphsto showcontinuouschanges. In this paper, we onlyconsiderline graphs.

We are interestedin interpreting line graphsbecause of their extensive use in

thermodynamics,physics,economicsandotherfields. For example,an intelligent tutoringsystemthat reasonswith line graphscan use thegraph as a sharedmedium for communication.The systemcanexplainconceptsby constructinggraphical explanations. The user can alsorepresenther understandingof the conceptsgraphicallyandthecomputerprogramcan checkthe correctnessof her understanding.The userandthe computercan bothmakemodificationstothe line graph,therefore,providing an additionalcommunicationchannelbetweenthe userandtheprogram. Similar advantageswould occur forengineeringanalysisandknowledgeacquisition.

We present a computer program calledSKETCHY that reasons about physicalphenomena visually by using graphs.SKETCHY provides an interactive drawir~genvironment,answersquestionsaboutgraphsandinterprets user modifications to the graph.SKETCHY hasgeneratedinterpretationsfor allthe graphsin a college level thermodynamicstextbook[17], as well as a number ofthermodynamicgraphsfrom [181 andeconomicsgraphsfrom [I]. SKETCHY’s interpretationsaresimilar to graphdescriptionsfound in textbooks.To our knowledge, SKETCHY is the firstcomputer program that interprets graphs in a(lOmain independentmanner.

The next section (Section 2) gives examplesfrom SKETCHY. Section3 discussesthe theoryunderlying SKETCHY. Section 4 explainsthealgorithmsanddesign choicesmadein buildingSKETCHY. Finally, Section 5 discussespossibleextensionsto SKETCHY.

200 350 Quantity

I: Supply andDemandgraph for cassette tapes (Ekeleund & Tollison, p. 84)

2 Examples from SKETCHY

The input to SKETCHY is a graph drawnandlabeledby the user. The supply-demandgraph,shownin figure 1, is a typical graph found inmany introductory economics textbooks.Supply-demandgraphsareused to show threeimportant relationships:how price effects thesupply for the product, how price effects thedemand,andthemarketpriceof theproduct.

To draw the graph shown in figure 1 usingSKETCHY, theuserstartswith an emptygraphconsistingonly of two axes. The userlabelstheaxesaspriceandquantity andchoosesscalesforthe axes. Thesupplyanddemandlines aredrawnby choosing two points on the screen.SKETCHY requirestheuserto label thelines,sothey can be referredin questionandanswersaswell as in thegraph interpretation. SKETCHYdoes not have any other knowledgeabout thelabelsprice,quantity,supplyanddemand. Thelabelsare only usedto indicatethe graphobjectthat is beingreferredto. SKETCHY producesthe following interpretation for the supply-demandgraph.

For edge SUPPLY:QUANTITY and PRICE are directly

proportional.For edge DEMAND:

QUANTITY and PRICE are inverselyproportional.

Because the intersection point is not labeled,SKETCHY doesnot include that information in

the graph description. This informationcan beobtainedby askingquestionsto SKETCHY.

Q: At whatpoint is SUPPLYequal to DEMAND?The SUPPLY equals DEMANDat the pointwhen QUANTITY equals 200 and PRICEequals 5.

Q: What is the PRICE for the SUPPLY line whentheQUAN11TY is 350?For SUPPLY line, when QUANTITY is 350the PRICE is 8.

Because supply-demand graph is relativelysimple, SKETCHY’s description is concise. Amore complicated graph taken from athermodynamicstextbookis shownin figure 2.Sinceall substancesexhibit thesamequalitativebehaviorshown in the pressure-volumegraph,understanding this graphis essentialfor solvingmany thermodynamics problems. The curvedtemperaturelines (3 1°C, 40°C and 50°C) aredrawn by choosingcontrol pointsfor the curve.Lines with discontinuities,such as 10°C and20°C, are drawn by combining curved andstraight lines. Lines 10°C to 50°C aregroupedtogetherandidentified as a temperaturecontourby the user since their labels indicatea thirddimensionto the graph. The critical point isdrawn by choosinga single point andlabeling it.SKETCHY infers that the critical point is online 31°C from its location. The pressure-volume graph also has threeregions—liquid,liquid-and-vapourandvapour—correspondingtothe state(s)the substanceis in. The regionsarelabeled by points at their boundaries. Thecomplete graph interpretation SKETCHYgeneratesis:

Price

8

5

Supply

0

Figure

Demand

Pressure

Figure2: Compressionof carbondioxide (Whalley,p. 22)

PRESSURE are inversely

PRESSURE are inversely

PRESSURE are inversely

20-C has discontinuities

For line 50-C:VOLUMEand

proportional.For line 40-C:

VOLUMEandproportional.For line 31-C:

VOLUMEandproportional.For line 20-C:

The slope of

;associatingdiscontinuitieswith regionsInside region LIQUID:

VOLUME INCREASE and PRESSUREDECREASE.

Inside region LIQUID-AND-VAPOUR:VOLUME INCREASE and PRESSURE

CONSTANT.Inside region VAPOUR:

VOLUME INCREASE and PRESSUREDECREASE.

For line 10-C:The slope of 10-C has

discontinuities.

Inside region LIQUID:VOLUME INCREASE and PRESSURE

DECREASE.Inside region LIQUID-AND-VAPOUR

VOLUME INCREASE and PRESSURE

CONSTANT.Inside region VAPOUR:

VOLUME INCREASE and PRESSUREDECREASE.CRITICAL-POINT is on lines (31-C)CRITICAL-POINT is on regions (LIQUIDLIQUID-AND-VAPOUR VAPOUR)For TEMPERATURE contour:

As TEMPERATURE increases

the slopes of TEMPERATURElinesbecome more LINEAR.

;basisfor Boyle’sLawFor a constant PRESSURE:

As VOLUME increases TEMPERATURE

INCREASE.VOLUMEand TEMPERATUREare directly

proportional.For a constant VOLUME:

As PRESSURE increases TEMPERATURE

INCREASE.PRESSURE and TEMPERATURE are

directly proportional.

Theexamplesdescribedaboveareinterpretationsof static graphs,but graphsdo not have to bestatic. Graph designerstypically superimposegraphsor show sequenceof graphsto describechanges in the situation. SKETCHYdemonstrates that this natural form ofcomparativeanalysis[161can bedonevia visualprocesseson agraph.

Analyzing engineeringcycles is an importanttask in thermodynamics.The basiccycle for asteampower plant is theRankinecycle, shownin figure 3. A common modification to theRankinecycle is superheatingthe steamin theboiler to increasetheefficiencyof thecycle. Thenet work of the cycle before modification isrepresented by area 1-2-3-4-1 and aftermodificationby area1-2-3’-4’-1.

20°C

10°C

Volume

Temperature

Figure 3: Effect of superheatingon Rankinecycle

SKETCHY producesthe following interpretationwhen3 is movedto 3’ and4 to 4’:

As a result of moving 3, 4:

For point 3:The ENTROPY of 3 INCREASE.The TEMPERATURE of 3 INCREASE.

For point 4:The ENTROPY of 4 INCREASE.The TEMPERATURE of 4 CONSTANT.

For region WORK:The area covered by WORK INCREASE.

3 The theory underlying SKETCHY

SKETCHY demonstratesthat visual reasoningvia graphs is a domain independentprocess.Mostgraphsaremadeup of threebasicelements:points, lines and regions. Although themeaningsof theseelementscould changefromone domain to another,the interactionsamonggraph elementsdoes not change. Just as ourperceptualmechanismcan easily compute thegraph properties, SKETCHY derivesthem usinggeometricmodelsto perform visual reasoning.

Qualitativeresults provide valuableinsights tounderstandingof a domain. Graphsrepresentqualitative informationeffectively by presentingit in a spatial format that our perceptualmechanismcan process easily. SKETCHYqualitativelydescribesline slopesandcurvatures.A straight line represents a numericalproportionality,andaverticalora horizontal linerepresentsthat oneof the propertiesis constant

while theother is increasing. Smoothcurvesaredescribed in terms of qualitativeproportionalities[3,1II. Lines withdiscontinuiticsaredescribedby dividing them upinto qualitatively distinct regions. If the changein line slopecorrespondsto an intersectionwithanotherobject,a relationbetweenthe intersectionand the changeof slope canhe inferred (e.g.theexistenceof phasetransitionpoints).

Most of the information SKETCHY extractsfrom thegraph can be easily representedusingconstructsof qualitative physicsl3,l11. Directlyproportional lines can be representedviaqualitativeproportionalities,pointson the graphcan be representedvia correspondences.As aresult, theoutput of SKETCHY can feed easilyinto qualitativereasoners.

Even when a verbal explanationis sufficient,textbooks usegraphs to emphasizethe verbalexplanations.In thesecases,althoughthegraphsdo not containadditional information,they createan additional representationin the form of animage. The image provides additional cueswhich makes remembering the informationeasier.

Our perceptual mechanisms are good atcomparing the size and orientation of objects.Graphs exploit this ability to help performcomparativeanalysis. Themodificationmadetoincreasethe work output of the Rankinecycle(see Fig. 3), is an essential concept inunderstandingpower plants. The graphical

3,

2

Entropy

demonstrationof the modification makesacomparativeargumentwhich is a lot more lucidthan a numericalargumentcould have been.Graphical representationstake advantage ofpeoples ability to follow qualitative andcomparativeargumentsby usingthe graph as asharedmedium for communication.

Similar lines in graphsaregroupedascontoursby ourperceptualmechanism.SKETCHY treatscontourlines, identified by the user,as a singleelement. Any changesof slope(or curvature)amongcontourlines is detectedin the samewaychangesin line slopesare detected. Becausecontoursaregenerallyusedto representa thirddimension,SKETCHY derivesthe relationshipbetweenthe contour and each of the variablesrepresented on theaxes.

Graphs do not have to be drawn to scale when therepresenting only qualitative information. Acommongraph convention used in the absence ofscales is to assumethat moving from left toright on the horizontalaxisand bottom to top onthe vertical axis implies an increase in thevariablerepresentedby that axis. SKETCHYuses this assumption in making qualitativeinterpretationswhen numerical scalesare notspecified.

To determinewhat peoplenotice in graph, wehave examinedgraph descriptionsin varioustextbooksandidentified graph properties that theauthorsconsiderimportant. We claim that anycomputerprogramwill needto detectall of theseproperties to effectively understand and reasonusing graphs. The commongraph properties are:

1. Relative orientation of points, lines andregions

2. Intersectionpoint of lines3. Slopesof lines4. Changes in line slopes5. Minimum, maximum and inflection points

of lines6. Relative size of regions

SKETCHYimplements these operations andbasedon our sample of 65 graphs (from [171 ~l181and [11), we believe SKETCHYprovides strongsupport that these operators are sufficient andnecessaryforgeneralgraphinterpretation.

4 Algorithmsanddesignchoices

The graph components of SKETCHYare labels,axes, points, lines, contoursand regions. The

labels on the graphs provide the vocabulary forSKETCHY’s interpretations. The distinctionbetween spatial relations represented in a diagramand the interpretation of the diagram has beenmade in earlier work in spatial reasoning[41.Except when there is a single componentof aparticular type in the graph, unlabeledcomponents can not be referred to either by theuser or by SKETCHY. The labels on the axesare required to makemeaningful interpretationssinceaxis labelsrepresentthephysicalpropertieswhich provide the framework for the graph.When the axes have scales, SKETCHYincludesnumericalinformation in additionto qualitativeinterpretation of the graph.

Points representdiscrete state information.Points can be connected to lines (whether theyareon the line or not), thus movethe lines whenthey are moved, preserving the orientationrelationshipbetweenthem. They also serveasboundarymarkers for regions and modify theshapeof the region when moved. Moving apoint changesthe valuesof physicalpropertiesrepresented. SKETCHY interprets themodification by comparing the property valuesfor the old and thenew location.

Internally,linesare representedby anorderedlistof segmenLs(for convenience,the usercanenterequationswhich areconvertedto line segments).Since segmentscan he of any length, usingsegmentsprovides arbitrary precision for linecurvatures without needing to manipulatecomplexequations.Like points,lines alsoserveas boundary markers for regions. Moving orextending the line changes the shapeandthesizeof the region.

When the shape of the region changes,SKETCHYcompares the old areato the new areain the interpretation. If the change in theregion’sshaperesultsin including (or excluding)anobject,this information is also includedin theregion. Any changesin the line slope whiletraversing a region is included in the graphinterpretationas significant. In the pressure-volume graph, noticing that the pressurestaysconstantinside the liquid-vapour region anddecreases outside the region is an essentialobservation for understanding phase changes.

SKETCHYcompares theslopesof the contourlines to find any trendsof changes. Lines withdiscontinuities,such as temperaturelines 10°Cand 20°C in pressure-volume graph, areapproximated by curves to be able to make

qualitativestatementsaboutall the contour lines.

A common intersectionpoint of contour lineswhich indicatesa convergenceto a specific valueis detectedand includedin SKETCHY’s graphinterpretation. When all contour lines exhibitsimilar qualitativechangesin their slopes,thecontour is describedin termsof thesequalitativechanges.

Understandinga graphnecessitatesbeingable toanswer questions about it. By using built intemplates,SKETCHY answersquestionssimilarto the ones in the GRE1 and the SAT2.SKETCHY answersquestions that involvereadingvaluesof the graph (e.g. What is thepressurewhen the volume is 3 and thetemperatureis 21?), comparingslopes,areasorlocations of objects with respect to each otherand describing relations of objects to each other(e.g. Is the critical point on line 20°C?).

It is especially interesting that SKETCHY’sgraph interpretations are so similar toexplanationsfoundin textbooksgiven its lackofany other understandingof the domain. Theinterpretationsproducedusing object labelsaremeaningfuland insightful for the domain. Themajor difference is that SKETCHY’sinterpretationslack the briefnessof experts’explanationsbecauseSKETCHY does notdifferentiate between relevant and irrelevantinformation.

SDiscussion

Understandinggraphsis an important part ofhumancognition. We haveshown how visualreasoningcan be usedto interpret graphs in adomain independentway using geometricproperties. The labels in graphsprovide thevocabulary,the points and lines provide therelationshipsfor interpretinggraphs.SKETCHYproducesoutput that can be usedby qualitativereasoners and other problem solvers, The abilityof SKETCHYto interpret graphs suggests thatthese ideas arerobust.

There are a number of avenues we are exploringfor extending SKETCHY. One possibleextension is to extend SKETCHY’s drawingenvironment and interpretation mechanism tootherkinds of graphs,such as scatterplots bargraphs,and.pie graphs. By understandingthegraphpropertiesexploitedby theseothergraphic

representations,we hope to build a model forgeneralgraphunderstanding.

Anotheravenuewearepursuingis incorporatingSKETCHY in an intelligent tutoring systemwhere SKETCHY provides an additionalcommunication medium betweentheuserandthecomputer. The user and the computer programboth can make modificationsto the graph anddiscuss the graph relating the conceptsrepresentedwith other ideasin the domain. Theideasimplementedin SKETCHY can beused toconstructgraphs that the user will be able tointerpretandreachthe intendedconclusions.

6 Acknowledgments

The research reported was supported by Office ofNaval Research. I owe many thanks to myadvisor, Ken Forbus, for valuablediscussions,comments and encouragement. My gratitude alsogoesto Meryl McQueenwho patiently proofreadthe original draft and to John Everett, RonFergusonand Keith Law who read subsequentdrafts.

References

1. R. B. Ekelund and R. D. Tollison.Economics.Little, Brown, Boston,1986.2. B. Faltings. Qualitative models in conceptualdesign: a case study. In 1st InternationalConferenceon Artificial Intelligencein Design,1991.3. K. D. Forbus. Qualitativeprocesstheory.Artificial Intelligence,24:85-168,1984.4. K. D. Forbus. Qualitativereasoningaboutspace and motion. In D. Gentner and A. L.Stevens,editors, Mental models, Eribaum,Hillsdale, NJ, 1983.5. K. D. Forbus, P. Nielsen, and B. Faltings.Qualitativespatial reasoning:the clock project.Artificial Intelligence,51:417-471,1991.6. L. Joskowicz and E. P. Sacks.Computational kinematics. ArtificialIntelligence,51:381-416,1991.7. G. S. Novak Jr. and W. C. Bulko. Usesofdiagrams in solving physicsproblems.In AAAISymposiumon Reasoningwith DiagramaticRepresentations,Stanford,CA, March 1992.8. H. Kim. Qualitative reasoning aboutfluidsand mechanics.PhDthesis,1993.9. K. R. KoedingerandJ.R. Anderson.Abstractplanning and perceptual chunks: elements ofexpertise in geometry. Cognitive Science,14:511-550,1990.1GraduateRecordExamination

2ScholasticAptitude Test

10. S. M. Kossylyn. Elements of GraphDesign.FreemanandCompany,New York, NY,1994.11. B. J. Kuipers. Qualitative simulation.Artjfl cia! Intelligence,29:289-388, 1986.12. J. Larkin and H. Simon. Why a diagram is(sometimes) worth ten thousand words.CognitiveScience,11:65-69, 1987.13. T. F. McDougal and K. J. Hammond.Representingand usingproceduralknowledgetobuild geometry proofs. In Proceedingsof theEleventhNational Conference on ArtificialIntelligence,1993.14. N. H. NarayananandB. Chandrasekaran.Acomputermodel of diagrammaticreasoning.InAAAI Symposium on Reasoning withDiagrammatic Representations,Stanford,CA,1992.15. S. Tessler, Y. Iwasaki, and K. Law.Qualitative structural analysis usingdiagrammatic reasoning. In The SeventhInternational Workshop on QualitativeReasoningaboutPhysicalSystems,1993.16. D. S. Weld. Theories of ComparativeAnalysis. MIT Press,Cambridge,MA, 1990.17. P. B. Whalley. Basic EngineeringThermodynamics.Oxford University Press,NewYork, 1992.18. G. J. Van Wylen and R. E. Sonntag.FundamentalsofClassicalThermodynamics.Wiley, New York, 3rd edition, 1985.