A Theoretical Analysis of Thought ExperimentsDavid L. Hibler

Departmentof Computer Science

ChristopherNewport University

Newport News, VA 23606.

tele: (804)—594—7065

e-mail: dhiblerc~pcs.Gnu. edu

Abstract

The thought experiment methodology is a re-cently developedtechniquefor qualitativereason-ing. Thoughtexperimentsinvolve four main steps:(i) simplification of the original problem, (ii) so-lution of the simplified problem, (iii) conjecturethe result of the original problem basedon thesolution of the simplified version,arid finally (iv)verification that the conjectureis correct. Thismethodologyhasbeenimplementedand hasbeenapplied successfullyto problems involving staticelectricity, themotion of chargedpendulums,fluidflow, and thermodynamiccycles. The purposeofthis paperis to analyzethis techniqueusingprob-abilistic models

IntroductionImaginary, simplified situations are often analyzedbyhuman problem solvers in order to understandtheprinciplesbehind more realistic situations. In physics,this techniqueis sometimesreferred to as a thoughtexperiment{Prigogine~ Stengers1984].

As art example of a thought experiment, considerhow to solvethe following problem.

A certain amount of charge is placed on a con-ductor. What ultimately happensto the charge?Where does it go?

There are sophisticatedways to solve this prob-lem, but a beginning physics student can oftendeterminethe solution in the following way.

1. We know that evena small amount of chargein the macroscopicsenseinvolves a huge nuni-her of particles. Despite this, we simplify theproblemto two chargedparticles.

2. Using the fact that like chargesrepel andotherelementaryknowledge, we solve the problemfor two chargedparticles. We concludethat theparticlesseparateuntil they reachthesurfaceofthe conductor. If chargecannot leave the con-ductor (no sparks)the answerto the simplifiedproblemis that the chargemovesto the surfaceandstaysthere.

3. We conjecture that the solution to the sim-plified problem is essentiallythe same as thesolution to the original problem.

4. The previous step produces a result but we are,perhaps,uncomfortablewith it. Did we over-simplify’? Just to check, we askourselvesif itwould makeanydifferenceif we hadthreeparti-cles or four. Solving theserelatedproblems wediscoverthat the resultwasthesame.With thisamount, of verification we feel that the resultvery probablyapplieseven to a large numberof

particles.

The above exampleis an informal thought experi-mnent. It involved the following four steps: siinplifica-lion, solution, conJecture,and verification. Since theverification methodemployedin this casewasnot rig-orousit is termed heuristicverification.

Wehaveformalizedthis method arid useit for qual-itative physics problem solving. It has been imple-merited in Prolog in asystemcalledTEPS[Hibler 1992;Hibler & Biswas 1992b; Flibler & Biswas 1989]. TEPSstandsfor Thougut ExperimentProblemSolver.

In this paperwe analyzethe operationof a thoughtexperiment problem solver. We will construct someelementary,probabilistic modelsof the thought exper-iment problemsolving processanddeterminesomeofits characteristics.

First, we describethe thought experiment method-ology in more detail.

Next, we consider a thought experiment for whichthere is oneright answerandri possiblewrong answers.The thought experimentis verifiedusing heuristicver-ification. which meansthat a total of s different sim-ulations agree. Weanalyze the probability, P[C A],that the answeris correctgiven that all ssimulationsagree.This might be termedthe validity of the heuris-tic verification. This analysis allows us to provide acondition for heuristic verification to work, to discussthe expectedbehaviorof heuristicverification, and toconsiderthe effect of fine versuscoarsedescriptions.

Finally, we analyze thought experimentsin termsofthe expected time, ~ for achievinga verified result.The important quantity in this analysisis the dice-

tiveness,E~,of eachindividual thought experiment.Atheorem is proved which indicatesthe order in whichthoughtexperimnentsshould be performedbasedon ef.fectiveness.Estimation of effectivenessis discussed.

Thought ExperimentsThe purposeof this section is to describethe stepsin the thought experiment methodology in a brief,straightforwardway. The thoughtexperimnientmethod-ology hasbeendescribedin a more formal, mathemat-ical mannerelsewhere[Hibler& Biswas 1989].

The thought experiment problem solver (TEPS)which we haveimplementedusesa qualitative reason-ing systembasedon ForbusQualitative ProcessThe-ory [Forbus 1984; Hibler & Biswa.s l992a~however,thought experimentmethodologyis very flexible. Thepurposeof this paper is to analyzethis methodologyandnot any particular implementation.We will men-tion someother possibilities for implementationaswediscussthe stepsof the thought experimentprocess.

SimplificationThe first stepin athought experimentinvolves simpli-fication. Simplifications are problem transformationrules provided to theproblemsolverby thesystemde-signer. Somesimplifications rules are domain specific;others are domain independent. Practical simplifica-tion strategiesand a partial catalogof useful simplifi-

cationsarepresentedin [Hibler 1992].As anexampleof asimplification, the thoughtexper-

imenit mentionedin the introduction used PopulationReduction. The Population Reductionsimplificationmapsany problem specificationwith multiple identi-cal objects to a problem specification with only twoobjects. This state is simpler becausethere are fewervariablesinvolved, amid it is computationally mucheas-ier to determinethe time evolution of the behaviorofthe system.

Another example of a simplification is the SimpleStereotypemethod. Simple stereotypesareeasily builtinto modules describingobjects or processes. Theyrepresentsimpler versions of the object or process.Other examplesof simplifications which havebeeninn-plementedin TEPS include Monte Carlo, CombinedChange, Variable Blockingand Superposition..MonteCarlo simplification is based on randomly samplingthe stategraph. CombinedChangesamplesthe stategraph basedon the way variables change. Variableblocking is a method for ignoring variables thoughtto be irrelevant. Superpositioncombinesresults fromseparatesubproblemsbut hasbeenimplementedonlyfor special cases. Someother possiblesimplificationswhich havenot beenimplementedarediscussedin [Hi-bIer 1992].

Many abstraction techniques developedby otherscould be consideredas simplifications in our sense.If heuristic verification is usedwith these techniquesthey might lie applied in caseswhere the validity

of the abstraction is questionable.ExamplesincludeOnitological Perspective[F’alkenhainer& Forbus 1990],Structural Consolidation[Weld & Addanki; Falken-hairier & Forbus 1990], TemporalAbstraction[Kuipers1987], arid Aggregation[Weld1986]. The Exaggerationmethodof Weld[Weld 1988] can definitely be consid-ereda simnphificationin our sense.

Solution

The nextstep in a thought experimentinvolves “solv-ing” the simplified model. This mneansthat the prob-1cm solver must containareasoningenginewhich takesa problem specificationamid reasons aboutit to producesome“results”.

The TEPS ReasoningEngine TEPS takes as in-put a specification of a qualitativestate of a physicalsystem. A qualitativestate is a collection of qualita-tive values,onefor eachof the variablespertaining tothesystenn.Qualitativevaluesconsisteither of special,qualitatively significant landmarkvaluesor of intervalsbetweentwo adjacentlandmark values[Forbus1984],

The dynamical behavior of the system is describedby director indirect influenceswhichcancausechangesof qualitativestate.Theycorrespondroughly to quali-tative versionsof differentialequationsandqualitativeversionsof functional relationships.

Given a problem specification in TEPS we can sim-ulate the time evolution of the systemby generatingagraph of qualitative stateswhich can be reachedfromthe original state. This is known as a reachableenvi-sionment.

Other Reasoning Engines Other reasoningen-ginescould be usedas the basisfor a thought experi-ment problem solver. Examplesinclude riot only otherqualitative reasoningsystemssuch as the of Kuipers[Nuipers, 1986] or de Kleer and Brown[de Kleer &Brown 1984], bitt also numericalsimulation systemns.

For a nummierical reasoningsystem the input wouldcomisist of the initial state arid the dymiamical equa—tiomis. The resultwould consist of theentire numericalsimulation of the system’strajectory in phasespace.

Description of Results The thought experinnentproblem solver is designedto answerspecific equationsabout a physicalsystemngiven an initial statefor thatsystem. We arethusnot concernedaboutthe output ofthe reasoning system directly because it usually doesnot constitute an answer to a question. What doesconstitute a possible answeris specified by somede-scriptioii function, I). D is sometimescalled a descrip-tioni basis. We will assume that any query only hasafinite number of possibleanswers. D can be thoughtof as a function which classifiesthe results producedby the reasoningengineinto oneof a finite number ofcategories,one for each possible answer. Thus evenif our reasoningengineusesnumericalsimulation thedescriptionof the results is qualitative.

In the exampleinvolving chargeplaced in a conduc-tor the description function takes the output of theproblem solver and categorizes it by giving a list of re-gionswhich contain a nonzeroamountof chargein thefinal state.

TEPS contains a library of descriptionfunctions;however, a description function can also be input for aparticular problem.

ConjectureA conjecture is a guessabout the description of theresult of the original problem based on the solutionobtained on the simplified version of the problem. Wewill assume in this paper that the conjecture is alwaysthat the description function classifies the result of thesimplified problem the same way that it would haveclassified the result of the original problem. In otherwords, the same description is produced.

VerificationVerification can be rigorous or heuristic. It could evenbe empirical. Rigorous verification requires establish-ing a formal proof that the conjecture is true. Thisis usually difficult. Empirical verification consists ofcomparing the predictionswith what actually occursin the real world. Often, this approachis not practi-cal. Themost commontype of verification is heuristic.With this type of verification other simplifications aretried, and the resulting conjectures are compared withthe original conjecture. If they agree, we accepttheconjecture as a reasonable belief.

The Validity of Thought ExperimentsThe first concern of our analysis is the validity ofthoughtexperiments.Do they, in fact, give the correctanswer? Our certainty in any particular casewill de-pendon theverification methodused.If exact verifica-tion is availablethen we can be surewhetherthe resultis correct. More has been said about exact verificationelsewhere[Ilibler1992]. Many usesof thought exper-iments involve heuristic verification. Heuristic verifi-cation seemsintuitively reasonable;however,we needto clarify some of the ideas behind it. In order to dothis, we explorea probabilistic model of heuristicveri-fication, and determine the probability that a thoughtexperimentis correct given that it is heuristically ver-ified. From this we determinethe actual requirementfor heuristic verification to be useful. Wealso examinethe effect of independent versus depemident confirma-tion and of fine versus coarse descriptions on validity.

Probabilistic ModelsTo create a simple model of the thought experimentprocess we make several assumptions.

A probabilistic model requires the definition of aproblemspace.Let S be the set of all problemswhichthe problemsolver with its particular library of pro-cesseswill accept. We will assumethat we sample

the problems using some fixed probability distribution.The probability distribution we use is left implicit andis not specified in the notation. The key assumption isthat the probability distribution is fixed. Based on thissampling, we can discuss the relative frequency withwhich events which are functions of specific problems

occur and associate probabilities with these events. Ina like manner we cant define probabilities on any subsetof S.

Consider any simplification method together with itsassociated conjecture and verification methods. Whenthese methods are applied to a randomly selected prob-lem from the problemnspacewe obtain a thought ex-periment which we can characterize probabilistically.

A key assumption states that solving a problemconsistsof performing separate,independentthoughtexperimentsuntil one is verified. This independentthought experimentassumptionimplies that we are ig-noring the useof complementarymodels and inheri-tancein this analysis.Since theuseof additional infor-mation by inheritancein generalhelpstheperformanceof the problenn solver it is safe to say that the simpleanalysisprovidesa conservativeestimateof complex-ity of the problemsolving task,and,in mnost cases,theactual resultswill be better.

Correctness ProbabilityAssume that the thought experiment processuses.ssimulations whose results are compared. Our descrip-tion of the results classifies those results into n + Ipossible categories. One of those categories is correct.In other words, if we hadappliedtheclassificationto afull simulationof theoriginalproblemthe resultwouldhave been in that category. The other mm categories arewrong. Weusesmall letters c and w

1. . . w,~to denote

the correct answer and the mm wrong answers. Let C de-note the event that all s simulations have the correctanswer. Let A denote the event that all s simulationshave the same answer. Let W denotethe event that

all s simulations have a wrong answer. We indicateintersectionsof the aboveevemitsby writing the letterstogether.

The thoughtexperimentis heuristicallyverified if alls simulations have the same answer. Thus the proba-

bility of interest is P[C I A], the probability that theanswer is correct given that all simulations agree. Weknow that

P[C A] = P[C]/P[A]

if the simulations agree they mmiust obviously agreeandbe corrector agreeand be wrong. These possibil-ities are disjoint. Thus P[A] = P[AC] + P[AW]; butP[AC] = P[C] so we can rewrite P[C IA] as

where

P[CIA] = 1/(1 + B) (1)

P = P[AW]/P[C]. (2)

The smaller R is, the better the accuracyof ourthoughtexperiment;thelarger B is, the worsetheac-curacy.

To obtain more insight we analyze the componentsof R. Let a be any answer,either thecorrect answercor anyof the w

1. . , w,~wrong answers. The probability

of getting answer a in all s simulations is

Pi[a I (0)a] P2

[a I (1)a] P3

[a I (2)a}. . . P8

[a I (s - 1)a]

The notation P~[a I (j)a] denotesthe probabilitythat we obtain result a on simulation i, having had (j)a’s on all the previous j simulations. Wecall this theprobability for a confirmation of answer a. P[a I (0)a]isjust P[a]. Using this formula we obtain

P[C]=Pi[cI(0)c] ‘~[cI(s—1)c]

P[AW] = Pi[wi I (0)wi]. . . P~[wnI (s — l)wi] (4)

+ Pi[w2

I (0)w2

]. . . P3

[w2

I (s — 1)w2

]

+ Pm[wr, I(0)w~]. . .P3

[w~I (s i)w~].

Using equations3 and 4 we cantrewrite equation 2as a sum of products of ratios of probabilities. It isconvenient to write 2 as

R=r~+r~+...+r~

where rk is the geometric mean of P~[wk I (i —

1)wk]/Pj[c I (i — 1)c] over the different simulations.i.e.

= ((Pl[wk I (0)w~]/Pi[c I (0)c])...

(P3

[wk I (s - 1)wk]/P3

[cI(s - 1)cD)’~

We must be careful to note that rk dependson s.Next, let us assume r > rj,, so there exists an upper

bound on the r~,ratios for all s. In that caseR < nr~so equation 1 becomes

P[CIA]> 1/(1+nrs)

(6)

A sufficient requiremnentfor the heuristicverifica-tion method to work is that on the averageoverall simulationsthe probability for a confirmationof thecorrectanswerbegreaterthanthe probabil-ity for a confirmation of any single wrong answerby somefixed amount.

(Theaveragementionedis the geometricmean.)

If we assumethat the heuristic verification require-ment is satisfiedthenequation 7 holdsandr < 1. Thisimplies that the probability that the thought exper-innent is correct is boundedfrom below by a nnono-tonically increasing‘function of the numberof success-ful verifications. In fact, given enough verificationsP[C I A] will be arbitrarily closeto 1.

(3) On the otherhandif evenoneof thegeometricmeansin equation 5 is boundedfrom below by a quantitygreaterthan one, then for largeenough s, additionalverificationsmake P[C I A] worse and not better. Ifsomeof the rk are one and any othershavean upperbound less than one then P[C I A] eventuallystabilizesat a value beyond which no improvementis possiblewith additional verifications.

DependentVersus IndependentConfirmation

(5) The confirmation probabilities P~[a (i — 1)a] will re-duce to P~[a]if eachsimulation, i, used is indepen-dent of the previous (i 1) simulations. This is of-ten not the case, however. The simulations used inheuristic verification are often simulationsfor modelsproducedby lessextremeversionsof the saniesimplifi-cation method. The productionof an answer,a, usingoneversionmight be correlatedwith the productionofa by another version. This is why the confirming P~must be expressed in terms of conditional probabilities.

If heuristicverification useslessextremeversionsofthesamesimplification mnethodto verify answersthenhow might the conditional probabilities for the right

and wrong answersbehave? First, it is very plausi-ble to believethat P~[cI (i —. 1)c] is very closeto 1 ifi > 1. In this casethe preceding(i — 1) simulationshaveproducedcorrectanswers.The ith simulation in-volvesa more realistic (lesssimplified) model than tImepreceding ones amid it is basedon the same type ofsimplificatiomi which producedcorrect results in thesecases.Thus it would be expectedto producea correctanswer. Second,P~[wkI (i — 1)wk] should eventuallydecline as i increasessimply becausethe models be-come more realistic as i increases. Unifortumiately thecorrelationproducedby usingthe samemethodmightmakethis decline slow.

Fine Versus Coarse Descriptions

A last considerationconcernsthe questionof whethera coarse or a fine description is more reliable. Theexplicit factor of mm in equation 7 suggests that the

smaller the number of alternatives in our description

(7)

This equationis sufficiently importantthat we reit-eratewhat the quantitiesmeanin words. A thoughtexperiment is performed for which there is 1 right an-swer and n possible wrong answers. The thought ex-

perimentis verified using heuristic verification, whichmeansthat a total of s different simulations agree.P[C I A] is the probability that the answeris correctgiven that all s simulations agree. r is a type of bound

on the probability of obtaining anysinglewronganswerversus the probability of obtaining the right answer.

Thismodel demonstratescertainbasicpointsaboutthought experiments which we discuss below.

Heuristic Verification RequirementThe conditions for equation 7 to hold are so impor-tant that we expressthem asthe heuristicverificationrequirement:Heuristic Verification Requirement:

of results the larger P[C I A] is. This assumptionis soniewhat dangerous because r will depend on nalso. For example, assumen = 3 and eachsimula-tion has the sameprobabilities: P[c] = 3/9, PNvi] =I’[wi] = P[w

3] = 2/9. If we halve the number of cat-

egoriesby combining c with w1

and w2

with w3

weobtain P[c’] = 5/9, P[w~]= 4/9, and n = 1. In thefirst case,r = P[w~]/P[c] = 2/3; in the secondcaser = P[w~]/P[c’] = 4/5. Any raising of r dominatesthelowering of n if s is high enough. On the other hand,ifs is small enoughthen coarsercategoriesare better.For example, if s = 1 then P[C I A] is just P[c] andwhen categoriesare combined it is always trite thatP[c’] > P[e] so coarsercategoriesare better.

Efficiency of Thought ExperimentsThenext issueafter correctnessof thought experimentsis their efficiency comparedto a direct solution of theproblem. We analyze thought experimentsin termsof the expectedtime, TE, for achievinga verified re-stilt. From this we prove a theoremshowinghow toachievemaximum efficiency. We then derive upperboundson the probability that a thought experimentfails to be verified and on time averagetime requiredfor athought experiment.We nextdescribehow to es-timate efficiency parametersand give a heuristic rulefor maximizing efficiency. Finally, we discusspracticalefficiency issues.

Time RequirementsLet 7’~ be the total time required for the nth thoughtexperiment. This time includesthe time requiredforsimplification, conjecture,and most importantly, thetime required for verification. Verification may requirea considerableamountof time sinceit usually involvessolving at leastone other simplified model. Let P~be

the probability that the nth thought experimentfails.We assumetheseprobabilities are independent.Thuswe assumnethought experimentsare not only functionally, but alsostatistically independent.With theseas-sumptions,the expectedtime, TE, requiredfor a suc-cessfulthought experimentis

TE=TI +T2

P1

+T3

P1

P2

+T4

P1

P2

P3

+... (8)

Thus, TE dependson the whole sequenceof thoughtexperimentswhich might be performed.

The probability, ~ that the problem solver fails toobtain amiy verified solution is just

P~=P1

P2

...P,~, (9)

where mm is the number of possible thought expeninnentswhich may be performed.

Let us briefly consider the mneamminlg of equations8and 9. If there were an infinite series of possiblethoughtexperimentsn would approachinfimuty. If eachof the P~were bounded from above by a number lessthan 1 then ~ would approach zero. In this case,we would expect the problem solver to always obtain

a verified solution. Sincewe do not actually have aninfinite seriestime problemnisolver may fail. TE repre-sents the average time takento either obtain a verifiedsolution or to fail.

Thought Experiment OrderingOur first application of equation 8 for ?~jis to provethe thought experimentordering theorem.

Let the time required to solve the problem by di-rect qualitative simulation be S~we define the ef-fectiveness,E~,of a thought experiment to be E~=(S

0/Tj)(l — Pt). Thesignificanceof thus definition will

be seen later.

Thought Experiment Ordering Theorem

A sequenceof independentthought experimentswill havea minimumexpectedtimefor achievingaverified result if the sequenceis performedin orderof effectivenessfrom most effective to least efiec-live.

Proof:Considerthought experiment i and i + I which are

adjacentin the sequencefor ‘EE used in equation 8.if we interchangetheorder in which theseexperimentsareperformedthe only terms in the series which changeare the terms involving Tl~and ~+i.

‘I~(originai) = 2~(P1

.. . P~1

)+7i+1

(P1

..

Tn(e.rchanged) = ‘1~+n(Pm. . . Pj1

) +

7i(Em .. .P~1

)I~~1

+ R

By algebraic mamiipulation we discover that‘I~j(exchanged) < TR(original) if and only if

— P~÷1

)> (I/T)(l — Pt). Multiplying bywe obtain time following exchangelennma:

Exchangelemma: TE(exchanged)< Tj~(original) ifamid only if E

1+j > E~.

We prove the ordering theorennby contradiction. Ifan optimalsequencewerenot in the ordergiven by thetheoremthemi by theexchangelemnmawe couldimprovethe sequence. This would contradict the assumptionthat the sequencewasoptinrial. Q.E.D.

Upper BoundsProbability of Failure First, let us a.ssume that weemploy only siniplificatiomi methodswhich are useful.If we have a finite number of simnphifications, we have afinite nunmiber of P~representingprobability of faihmre.Oneof theseP~hasa maximnumvalue; call it I’. Anysiniphification methodwhich producesthought experi-mimentswhich alwaystend to fail verification (P~= 1) isdropped from a problemsolver as useless.Therefore,P is an upperboundon theprobability of failure of anysimplification amid this tipper bound is less than one.Wecall this the usefulnssa.5silmption, equations 10,II.

P

P <1

The usefulness assumption provides an upper boundon the probability of failure, Pp, of a thought experi-ment given in equation9.

Pp ~

As pointedout earlier,assumptionof 10 and11 guar-anteesthat as the number of thought experimentsin-creases,the probability of failure, Pp, may be madeassmall aswe please.

Average Time Next, let us determine an upperbound for Tp. This will dependon a bound for timeT~aswell as a boimnd for the P~.We assumnethat forall i,

There might be some thought experimentswhichcouldgo on foreverwithout reachinga conclusion.Thiswould bedueto thefact that thequalitativesimulationdid not terminate. In practiceany thoughtexperimentwhich goeson too long can be terminated and verifi-cationconsideredto havefailed. Thus assumption13is, in fact, acceptable.

Given equations10, 13 and the fact that all quan-tities are positive, an upper bound for Tp is T~~T+ TP+ TP

2+.... This is not an infinite series, but

it. is approximatedby an infinite seriesif manydiffer-ent simnplificationsarepossible. Furthermore,since alltermsarepositive the infinite seriesis certainly an up-per bound. We havean ordinary geomnetricseriesamidsimmceP < 1 it converges.Thus

Tp<T/(1—P). (14)

The improvementratio over straight simulation is5~/77~. Using 14 we have a lower bound on this of

> (S0

/T)([ — P). This bound is just an effec-tivenesscalculatedusingT, andP; thus, it providesUS

with an interpretationi of the effectiveness of a thoughtexperiment. The significanceof the effectiveness,E~,of thought experiment i is that if every thought ex-periment,k, is no worsethami the given experiment,i,(S

0/Tk > SJTj, amid Pk < P~)therm the thought exper-

imentproblemsolveris fasterthan straight simulationby a factor of at leastE~.

Estimation of ParametersCanwe obtain estimatesfor parameterssuch asP~andTj which characterizethought experiments? This isoneof the most importanit issuesfor practicalapplica-tions. Oneapproachis to simply assumevariousvaluesfor theseparametersfor the sakeof theoreticalanaly-sis. Another approachis to attempt to makeempiricalestimatesfor them evenif they are crude.

In order to makeempirical estimates,we must dis-tinguish betweenspecific thought experimnentswhich

are attemptsto solveuniqueproblemsamid the simphi-(10) fication methodswhich are used in the thoughtexper-(1I~ iments. Any specific thought experimenteither sue-

ceedsor it doesn’t. If we repeatit we always get thesame result. A simimphification metliod on time otherhand, may produce a verifiable result if used in onethought experiment but not in anothen. If we have

~12~ enoughexperiencewith a thoughtexperimentproblemsolver we can collect rough statistics to indicate timefrequencywith which a givensimplification method isusefuliii giving verifiable results.We canalsoestimatethe time improvementratio S0/T1for thought experi-ments usinga given simnpliflcation. The estimatesarecrude becausethe successof a simplification methodmay dependon the type of problem, i.e., the charac-teristicsof the individual problem space. If the prob-lem solver hasnot encounteredasimilar type of prob-lem before, the frequencybasedestimatemay not bevery reliable. Another reasonfor the estimatesheimigcrudeis that previouslyencounteredproblemsmay riotconstitutea randonnior sufficiently largesamnpleof the

problemspace.Next, let us makesonic estimatesof ~ in termsof

more basicquantities. A thought experimentinvolvesa simplification and a generalizationstep. The sim-plification step involves finding a simplified version ofthe problemand performing a qualitative simulationon that simplified version. Finding a simplified ver-sion takes a constantannmount of timne which is smallconnparedto the time required to perfornn the qua!—itative simulation. The generalizationstep involvesmaking a conjecture amid verifying that conjecture.Making a conjecturetakesa constantamountof timewhich is small comparedto the timrme required for aqualitativesi mnulatiomi. lgnoring thesesmall quantities

= Si + V~.S~is the time required for qualitative

simulation, and %‘~ is the time requiredfor verification.To nnakeour estimate for T~more useful we must

make some rough estimatesconcerning verification.Verification usually involves performing at least oneadditional qualitative simulation am! connparinmgtheresultsto time previoussimulation. Making the simnipli—lication and comparingthe results would take asmallamount of time comparedwith S~.Our estimatefor~ becomes‘I’~ = nS~where mm is the number of qual-itative simulations performed and S~is time averagetime taken for each. A reasommableestimnatefor n is2. The results of the original sinnulation are checkedby comparingwith oneadditional simulation. In somecasesmm could be 1. This would mean that the re-sult would be comparedwith the resultsfrom previousfailed thonmght experimentsfor the sannie problem. Itwould alsomakeanalysismore difficult asthe thoughtexperimentswould no longerbe independent.Whethera thoughtexperimentsucceededor failed would dependon theorderingof thought experiments.In some casesthe simplification method might specify an mm greaterthan 2 but theseare probably rare. Consideringboth

(13)

theseeffects an estimateof 2 seemsreasonable. Wewill assumeTi = S,~the analysiswould not begreatlydifferent if 3 or someother small numberwerechosen.

The effectivenessof a thought experimentbecomesE~= (S~/S~)(1— P~)/2.S

0/S~can be coimsideredthe

averageamount of simplification achievedby the sim-plification methodused in the ith thoughtexperiment.It representsan averagetime improvementfactor in us-ing thetwo simplified simulationsin thethought exper-iment versussimulatingthe original problem. This pa-rameteris importantbecausewe canoftenranksimpli-fication methodsby (S~/S~).Examinationof two sim-plification methodswill often indicate which is moreextremeand should thereforeyield a larger value of(S

0/S~).0mm the otherhand, knowledgerequiredto es-

timate P~may be more difficult to obtain. In this sit-uation, the thought experimentordering theoremsug-geststhe following heuristicordering rule:

Heuristic Ordering Rule:If independent simplification methods can beranked by degree of simplification, but no infor-mation is available about verification probabilitiesthen a thought experimentproblem solvershouldtry the simplification methodsin order of degreeof simplification from strongestto weakest.

If S is an upperboundon the time any of the qual-itative simulations might take then we can estimateequation14 by:

TB <2S/(1 — P).

We want the thought experiment method to takelesstime than solving the original problemdirectly byqualitative simulation. If qualitative simulation of theoriginal problem takes time S

0, we requireSO/TB > 1.

This means that we would like our upper bound to besuchthat (S

0/S)(1— P)/2 > 1; perhaps much greater

than 1. Since S is an upper boundon the S~,Se/S~5alower boundon the amount of simplification used inany individual thought experiment.

PracticalityAre thought experimentspractical? Unless absoluteverification is available even a successfulthought ex-periment provides only a reasonableconclusion andnot a certainone. Thus, this methodwill be usedonlywhen direct simulation is not feasible, usually becauseit would taketoo long. This is, in fact, the casewithmost real world problems. In order for the thoughtexperiment problem solver to be worthwhile we mustbe able to find simplifications which havean effective-nesswhich is greaterthan one,preferablymuchgreaterthan one. In order to achievethis we needan extremedegree of simplification with reasonable probability ofverification. For example, if probability of verificationis at least 1/5 then P is 4/5 and we need a degree ofsimplification (Se/S) > 10. Experiencewith TEPSis too limited to take any estimatesvery seriously;

this is a possibility for future research. With TEPSso far, thevaluesfor P arenormally lessthan onehalf.It seems likely that sinnplifications would be droppedfrom a problem solver if they fail verification in thegreatmajority of cases.

It might be argued that we need exhaustivestatis-tical testimig to determinethat the effectivenessfor agivem1 simplification is adequatelyhigh to be of use.This is not true if the degreeof simplification providedby the simplification method is high enough. In thatcase,demonstrationof evena few successfulverifica-tions makesit reasonableto believe that E~is largeenough. For example,if (S~S~)> 100 then E~> 1if (1 — P~)> 0.02; if (S~/S~)> 1000 then E~> 1 if(1 — P~)> .002; if (S

0/S~)> 1,000,000then E~> 1 if

(I — P~)> .000002. Since(1 — P~)representsprobabil-ity of successfulverification evena very few successfulexamplesindicate that theeffectivenessis high enoughif the degreeof simplification is really large.

ConclusionsWe haveprovidedapreliminary theoreticalframeworkfor the thought experimentmethodology. This frame-work combined with the empirical experience withTEPS suggests that this is a viable technique. It doesnot replace conventional methods of qualitative reason-ing but rather augments them. Further developmentof this techniqueseemsdesirable.

Therearemanypossibilities for future development.Analysis of known simplification techniquesand devel-

(15) opment of new onesis an important areaof research.Theoretical analysesof simplifications should be at-temptedif possibleand statistics on practical effectsshould be obtained. On a practicalside, it would beuseful to explore thought experimentproblemsolversas a basis for tutoring systems.The simplified modelautomatically generatedby the problem solver couldbea basisfor helpingstudentsunderstandthe originalproblem.

ReferencesFalkenhainer B. and Forbus K. D. 1990. Composi-tional modelingof physical systems.In ~4tlmInterna-tional Workshopon Qualitative Physics.

Forbus K. D. 1984. Qualitativeprocesstheory. Arti-ficial Intelligence,24:85~-168.

Hibler D. L. 1992. The Thought ExperimentMethod:A New Approachto QualitativeReasoning.PhD the-sis, The University of South Carolina.

Hibler D. L. and Biswas C. 1992a. Teps: applyingthe thought experimentmethodology to qualitativeproblemsolving. Iii B. Faltings andP. Struss,editors,RecentAdvancesin Qualitative Physics, pages345—360, MIT Press.

Hibler D. L. and Biswas G. 1992b. Thought exper-iments as a framework for multi-level reasoning. InWorkingPapersof the Sixth International Workshop

on QualitativeReasoningabout PhysicalSystems,Au-gust.

Hibler D. L. andBiswas G. 1989.The thoughtexper-iment approachto qualitative physics. In IJCAI-89,pages1279—1284.

de Kleer J. and Brown J. S. 1984. A qualitativephysics basedon confluences. Artificial Intelligence,24:7—83.

Kuipers B. 1987. Abstraction by time-scale in qual-hitative simulation. In Proceedings of AAAJ-87,pages 621—625.

Kuipers B. 1986. Qualitative simulation. ArtificialIntelligence,29:289—388,1986.

Prigogine I. and Stengers I. 1984. OrderOut of Chaos.Bantam Books.

Weld D. S. 1988. Exaggeration. In ProceedingsoftheSeventhNational Conferenceon Artificial Intelli-gence,pages291—295, Minneapolis,MN, August.

Weld D. S. 1986. The use of aggregation in causalsimulation. Artificial Intelligence.

Weld D. S. and Addanki S. 1990. Task-driven modelabstraction. In Fourth International Workshop onQualitative Physics.