:-0 appear, AAAI-94

The First Law of Robotics( a call to arms )

Daniel Weld Oren Etzioni*Department of Computer Science and Engineering

University of WashingtonSeattle, WA 98195

{weld, etzioni}~cs.washington.edu

Abstract

Even before the advent of Artificial Intelligence, sci-ence fiction writer Isaac Asimov recognized that anagent must place the protection of humans from harmat a higher priority than obeying human orders. In-spired by Asimov, we pose the following fundamentalquestions: (I) How should one formalize the rich, butinformal, notion of "harm"? (2) How can an agentavoid performing harmful actions, and do so in a com-putationally tractable manner? (3) How should anagent resolve conflict between its goals and the needto avoid harm? (4) When should an agent prevent ahuman from harming herself? While we address someof these questions in technical detail, the primary goalof this paper is to focus attention on Asimov's concern:society will reject autonomous agents unless we havesome credible means of making them safe!

The Three Laws of Robotics:1. A robot may not injure a human being, or,

through inaction, allow a human being to cometo harm.

2. A robot must obey orders given it by humanbeings except where such orders would conflictwith the First Law.

3. A robot must protect its own existence as longas such protection does not conflict with theFirst or Second Law.

Isaac Asimov ( Asimov 1942):

MotivationIn 1940, Isaac Asimov stated the First LawofRobotics,capturing an essential insight: an intelligent agentl

.We thank Steve Hanks, Nick Kushmerick, Neal Lesh,Kevin Sullivan, and Mike Williamson for helpful discus-sions. This research was funded in part by the Universityof Washington Royalty Research Fund, by Office of NavalResearch Grants 90-J-1904 and 92-J-1946, and by NationalScience Foundation Grants IRI-8957302, IRI-9211045, andIRI-9357772.

1 Since the field of robotics now concerns itself primarilywith kinematics, dynamics, path planning, and low levelcontrol issues, this paper might be better titled "The FirstLaw of Agenthood." However, we keep the reference to"Robotics" as a historical tribute to Asimov.

should not slavishly obey human commands -its fore-most goal should be to avoid harming humans. Con-sider the following scenarios:

.A construction robot is instructed to fill a potholein the road. Although the robot repairs the cavity,it leaves the steam roller, chunks of tar, and an oilslick in the middle of a busy highway.

.A softbot (software robot) is instructed to reducedisk utilization below 90%. It succeeds, but in-spection reveals that the agent deleted irreplaceableIjcTEX files without backing them up to tape.

While less dramatic than Asimov's stories, the sce-narios illustrate his point: not all ways of satisfying ahuman order are equally good; in fact, sometimes it isbetter not to satisfy the order at all. As we begin todeploy agents in environments where they can do somereal damage, the time has come to revisit Asimov'sLaws. This paper explores the following fundamentalquestions:

.How should one formalize the notion of"harm"? We define dont-disturb and restore-two domain-independent primitives that capture as-pects of Asimov's rich but informal notion of harmwithin the classical planning framework.

.How can an agent avoid performing harm-ful actions, and do so in a computationallytractable manner? We leverage and extend thefamiliar mechanisms of planning with subgoal inter-actions (Tate 1977; Chapman 1987; McAllester &Rosenblitt 1991; Penberthy & Weld 1992) to detectpotential harm in polynomial time. In addition, weexplain how the agent can avoid harm using tacticssuch as confrontation and evasion (executing sub-plans to defuse the threat of harm).

.How should an agent resolve conflict betweenits goals and the need to avoid harm ? Weimpose a strict hierarchy where dont-disturb con-straints override planners goals, but restore con-straints do not.

.When should an agent prevent a human fromharming herself? At the end of the paper, weshow how our framework could be extended to par-tially address this question.

The paper's main contribution is a "call to arms:"before we release autonomous agents into real-worldenvironments, we need some credible and computation-ally tractable means of making them obey Asimov'sFirst Law.

dard assumptions of classical planning: the agent hascomplete and correct information of the initial stateof the world, the agent is the sole cause of change,and action execution is atomic, indivisible, and resultsin effects which are deterministic and completely pre-dictable. (The end of the paper discusses relaxing theseassumptions. ) On a more syntactic level, we make theadditional assumption that the agent's world model iscomposed of ground atomic formuli. This sidesteps theramification problem, since domain axioms are banned.Instead, we demand that individual action descriptionsexplicitly enumerate changes to every predicate that isaffected.4 Note, however, that we are not assuming theSTRIPS representation; Instead we adopt an action lan-guage (based on ADL (Pednault 1989)) which includesuniversally quantified and disjunctive preconditions aswell as conditional effects (Penberthy & Weld 1992).

Given the above assumptions, the next two sectionsdefine the primitives dont-disturb and restore, andexplain how they should be treated by a generativeplanning algorithm. We are not claiming that the ap-proach sketched below is the "right" way to designagents or to formalize Asimov's First Law. Rather ,our formalization is meant to illustrate the kinds oftechnical issues to which Asimov's Law gives rise andhow they might be solved. With this in mind, the pa-per concludes with a critique of our approach and a(long) list of open questions.

SafetySome conditions are so hazardous that our agentshould never cause them. For example, we might de-mand that the agent never delete UTEX files, or neverhandle a gun. Since these instructIons hold for alltimes, we refer to them as dont-disturb constraints,and say that an agent is safe when it guarantees toabide by them. As in Asimov's Law, dont-disturbconstraints override direct human orders. Thus, if weask a softbot to reduce disk utilization and it can onlydo so by deleting valuable UTEX files, the agent shouldrefuse to satisfy this request.

We adopt a simple syntax: dont-disturb takes asingle, function-free, logical sentence as argument. Forexample, one could command the agent avoid deletingfiles that are not backed up on tape with the followingconstraint:

dont-disturb( wri tten. to. tape(f) V isa(f, f ile ) )

Free variables, such as f above, are interpreted asuniversally quantified. In general, a sequence of ac-tions satisfies dont-disturb(C) if none of the actionsmake C false. Formally, we say that a plan satisfiesan dont-disturb constraint when every consistent,totally-ordered, sequence of plan actions satisfies theconstraint as defined below.

ference tractable by formulating restricted representationlanguages (Levesque & Brachman 1985).

4 Although unpalatable, this is standard in the planning

literature. For example, a STRIPS operator that moves blockA from B to C must delete on(A,B) and also add clear(B)even though clear(x) could be defined as Vy --'on(y, x).

Survey of Possible SolutionsTo make intelligent decisions regarding which actionsare harmful, and under what circumstances, an agentmight use an explicit model of harm. For example,we could provide the agent with a partial order overworld states (i.e., a utility function). This framework iswidely adopted and numerous researchers are attempt-ing to render it computationally tractable (Russell &Wefald 1991; Etzioni 1991; Wellman & Doyle 1992;Haddawy & Hanks 1992; Williamson & Hanks 1994),but many problems remain to be solved. In manycases, the introduction of utility models transformsplanning into an optimization problem -instead ofsearching for some plan that satisfies the goal, theagent is seeking the best such plan. In the worst case,the agent may be forced to examine all plans to deter-mine which one is best. In contrast, we have explored asatisficing approach -our agent will be satisfied withany plan that meets its constraints and achieves itsgoals. The expressive power of our constraint languageis weaker than that of utility functions, but our con-straints are easier to incorporate into standard plan-ning algorithms.

By using a general, temporal logic such as that of(Shoham 1988) or (Davis 1990, Ch. 5) we could spec-ify constraints that would ensure the agent would notcause harm. Before executing an action, we could askan agent to prove that the action is not harmful. Whileelegant, this approach is computationally intractableas well. Another alternative would be to use a plannersuch as ILP (Allen 1991) or ZENO (Penberthy & Weld1994) which supports temporally quantified goals. un-fortunately, at present these planners seem too ineffi-cient for our needs.2

Situated action researchers might suggest that non-deliberative, reactive agents could be made "safe" bycarefully engineering their interactions with the envi-ronment. Two problems confound this approach: 1)the interactions need to be engineered with respect toeach goal that the agent might perform, and a generalpurpose agent should handle many such goals, and 2)if different human users had different notions of harm,then the agent would need to be reengineered for eachuser.

Instead, we aim to make the agent's reasoning aboutharm more tractable, by restricting the content andform of its theory of injury.3 We adopt the staD-

2We have also examined previous work on "plan quality"for ideas, but the bulk of that work has focused on theproblem of leveraging a single action to accomplish multiplegoals thereby reducing the number of actions in, and thecost of, the plan (Pollack 1992; Wilkins 1988). While thisclass of optimizations is critical in domains such as databasequery optimization, logistics planning, and others, it doesnot address our concerns here.

3 Loosely speaking, our approach is reminiscent of clas-

sical work on knowledge representation, which renders in-

Definition: Satisfaction of dont-disturb: Let Wobe the logical theory describing the initial state of theworld, let A1, ...,An be a totally-ordered sequence ofactions that is executable in Wo, let Wj be the theorydescribing the world after executing Aj in Wj-l, and

let C be a function-free, logical sentence. We say thatA1, ...,An satisfies the constraint dont-disturb(C)if for all j E [1, n], for all sentences C, and for allsubstitutions (J ,

if Wo F (0 then Wj F (0 (1)

u nlike the behavioral constraints of (Drummond1989) and others, dont-disturb does not require theagent to make C true over a particular time interval;rather, the agent must avoid creating any additionalviolations of C. For example, if C specifies that allofGore's files be read protected, then dont-disturb(C)commands the agent to avoid making any of Gore'sfiles readable, but if Gore's. plan file is already read-able in the initial state, the agent need not protectthat file. This subtle distinction is critical if we wantto make sure that the behavioral constraints providedto an agent are mutually consistent. This consistencyproblem is undecidable for standard behavioral con-straints (by reduction of first-order satisfiability) butis side-stepped by our formulation, because any set ofdont-disturb constraints is mutually consistent. Inparticular, dont-disturb(P(x) A -,P(x)) is perfectlylegal and demands that the agent not change the truthvalue of any instance of P .

Synthesizing Safe PlansTo ensure that an agent acts safely, its planner mustgenerate plans that satisfy every dont-disturb con-straint. This can be accomplished by requiring thatthe planner make a simple test before it adds new ac-tions into the plan. Suppose that the planner is con-sidering adding the new action Ap to achieve the sub-goal G of action Ac. Before it can do this, it must it-erate through every constraint dont-disturb(C) andevery effect E of Ap, determining the conditions (ifany) under which E violates C, as defined in figure 1.For example, suppose that an effect asserts -,P andthe constraint is dont-disturb(P V Q), then the ef-fect will violate the constraint if -,Q is true. Hence,violation(-'P, PVQ) = -,Q. In general, if violationreturns true then the effect necessarily denies the con-straint, if :false is returned, then there is no possibleconflict, otherwise violation calculates a logical ex-pression specifying when a conflict is unavoidable.5

Before adding Ap, the planner iterates through ev-ery constraint dont-disturb(C) and every effect con-sequent E of Ap, calculating violation(E, C). Ifviolation ever returns something other than False,

then the planner must perform one of the followingfour repairs:

1. Disavow: If E is true in the initial state, then thereis no problem and Ap may be added to the plan.

2. Confront: If Ap's effect is conditional of the formwhen 5 then E then Ap may be added to the planas long as the planner commits to ensuring that ex-ecution will not result in E. This is achieved byadding -,5 as a new subgoal to be made true at thetime when Ap is executed.6

3. Evade: Alternatively, by definition of violation itis legal to execute A1' as long as R = violation(E, C)will not be true after execution. The planner canachieve this via goal regression, i. e. by computingthe causation preconditions (Pednault 1988) for -,Rand Ap, to be made true at the time when Ap isexecuted. 7

4. Refuse: Otherwise, the planner must refuse to addAp and backtrack to find another way to to supportG for Ac.For example, suppose that the agent is operating

under the written. to. tape constraint mentioned ear-lier, and is given the goal of reducing disk utilization.Suppose the agent considers adding a rm paper. texaction to the plan, which has an effect of the form-,isa(paper.tex, file). Since violation returns-,written. to. tape (paper .tex) , the rm action threat-ens safety. To disarm the threat, the planner mustperform one of the options above. Unfortunately, dis-avowal ( option one) isn't viable since paper. t ex exists

6Note that -,5 is strictly weaker than Pednault's preser-vation preconditions (Pednault 1988) for Ap and C; it ismore akin to preservation preconditions to a single effect ofthe action.

7While confrontation and evasion are similar in the sensethat they negate a disjunct (5 and R, respectively ), they dif-fer in two ways. First, confrontation's subgoal -,5 is derivedfrom the antecedent of a conditional effect while evasion's-,R comes from a disjunctive dont-disturb constraint viaviolation. Second, the subgoals are introduced at differ-ent times. Confrontation demands that -,5 be made truebefore Ap is executed, while evasion requires that -,R betrue after execution of Ap. This is why evasion regresses Rthrough A".

sIf E contains "lifted variables" (McAilester & Rosen-blitt 1991) (as opposed to universally quantified variableswhich pose no problem) then violation may return anoverly conservative R. Soundness and safety are main-tained, but completeness could be lost. We believe thatrestoring completeness would make violation take expo-nential time in the worst case.

Definition 2 differs from definition 1 in two ways: (1)restore constraints need only be satisfied in Wn afterthe complete plan is executed, and (2) the goal takesprecedence over restore constraints. Our constraintsobey a strict hierarchy: dont-disturb takes priorityover restore. Note that whenever the initial state isconsistent, restore constraints are guaranteed to bemutually consistent; the rationale is similar to that fordont-disturb.

Synthesizing Tidy PlansThe most straightforward way to synthesize a tidy planis to elaborate the agent's goal with a set of "cleanup"goals based on its restore constraints and the initialstate. If the agent's control comes from a subgoal in-terleaving, partial order planner such as ucPOP (Pen-berthy & Weld 1992), then the modification necessaryto ensure tidiness is straightforward. The agent dividesthe planning process into two phases: first, it plans toachieve the top level goal, then it plans to clean upas much as possible. In the first phase, the plannerdoesn't consider tidiness at all. Once a safe plan isgenerated, the agent performs phase two by iteratingthrough the actions and using the violation function(figure 1) to test each relevant effect against each con-straint. For each non-false result, the planner gener-ates new goals as follows. ( 1) If the effect is groundand the corresponding ground instance of the restoreconstraint, ((} , is not true in the initial state, then nonew goals are necessary. (2) If the effect is ground and((} is true in the initial state, then ((} is posted asa new goal. (3) if the effect is universally quantified,then a conjunction of ground goals ( corresponding toall possible unifications as in case 2) is posted.8 Afterthese cleanup goals have been posted, the planner at-tempts to refine the previous solution into one that istidy. If the planner ever exhausts the ways of satisfyinga cleanup goal, then instead of quitting altogether itsimply abandons that particular cleanup goal and triesthe next.

Note that in some cases, newly added cleanup ac-tions could threaten tidiness. For example, cleaningthe countertop might tend to dirty the previously cleanfloor. To handle these cases, the planner must continueto perform the violation test and cleanup-goal gener-ation process on each action added during phase two.Subsequent refinements will plan to either sweep thefloor (white knight) or preserve the original cleanlinessby catching the crumbs as they fall from the counter( confrontation).

AnalysisUnfortunately, this algorithm is not guaranteed toeliminate mess as specified by constraint 2. For ex-ample, suppose that a top level goal could be safelyachieved with Ax or Ay and in phase one, the plannerchose to use Ax. If Ax violates a restore constraint,Ay does not, and no other actions can cleanup themess, then phase two will fail to achieve tidiness. One

8Case 3 is similar to the expansion of a universally quan-tified goal into the universal base (Penberthy & Weld 1992),but case 3 removes ground literals that aren't true in theinitial state.

in the initial state (i.e., it is of type file). Option two( confrontation) is also impossible since the threateningeffect is not conditional. Thus the agent must choosebetween either refusing to add the action or evadingits undesired consequences by archiving the file.

AnalYsisTwo factors determine the performance of a planningsystem: the time to refine a plan and the number ofplans refined on the path to a solution. The time perrefinement is affected only when new actions are addedto plan: each call to violation takes 0( ec ) time wheree is the number of consequent literals in the action'seffects and c is the number of literals in the CNF en-coding of the constraint. When a threat to safety isdetected, the cost depends on the planner's response:disavowal takes time linear in the size of the initialstate, refusal is constant time, confrontation is linearin the size of 5, and the cost of evasion is simply thetime to regress R through Ap.

It is more difficult to estimate the effect ofdont-disturb constraints on the number of plans ex-plored. Refusal reduces the branching factor while theother options leave it unchanged (but confrontationand evasion can add new subgoals). In some cases, thereduced branching factor may speed planning; however,in other cases, the pruned search space may cause theplanner to search much deeper to find a safe solution(or even fail to halt). The essence of the task, how-ever, is unchanged. Safe planning can be formulatedas a standard planning problem.

TidinessSometimes dont-disturb constraints are too strong.Instead, one would be content if the constraint weresatisfied when the agent finished its plan. We de-note this weaker restriction with restore; essentially,it ensures that the agent will clean up after itself-by hanging up phones, closing drawers, returningutensils to their place, etc. An agent that is guar-anteed to respect all restore constraints is said tobe tidy. For instance, to guarantee that the agentwill re-compress all files that have been uncompressedin the process of achieving its goals, we could sayrestore(compressed(f».

As with dont-disturb constraints, we don't requirethat the agent clean up after other agents -the stateof the world, when the agent is given a command, formsa reference point. However, what should the agent dowhen there is a conflict between restore constraintsand top level goals? For example, if the only way tosatisfy a user command would leave one file uncom-pressed, should the agent refuse the user's command orassume that it overrides the user's background desirefor tidiness? We propose the latter -unlike mattersof safety, the agent's drive for tidiness should be sec-ondary to direct orders. The following definition makesthese intuitions precise.

Definition: Satisfaction of restore: Building onthe definition of dont-disturb, we say that Al , ..., Ansatisfies the constraint restore( () with respect togoal G if for all substitutions 0

if Wo I= (0 then ( Wn I= (0 or G I= -,(0) (2)

could fix this problem by making phase two failuresspawn backtracking over phase one decisions, but thiscould engender exhaustive search over all possible waysof satisfying top level goals.

Remarkably, this problem does not arise in the caseswe have investigated. For instance, a software agenthas no difficulty grepping through old mail files fora particular message and subsequently re-compressingthe appropriate files. There are two reasons why tidi-ness is often easy to achieve ( e.g., in software domainsand kitchens):

.Most actions are reversible. The compress actionhas uncompress as an inverse. Similarly, a shortsequence of actions will clean up most messes ina kitchen. Many environments have been stabi-lized (Hammond, Converse, & Grass 1992) ( e.g., byimplementing reversible commands or adding dish-washers) in a way that makes them easy to keeptidy.

.We conjecture that, for a partial-order planner, mostcleanup goals are trivially serializable (Barrett &Weld 1993) with respect to each other.9

When these properties are true of restore con-straints in a domain, our tidiness algorithm does sat-isfy constraint 2. Trivial serializability ensures thatbacktracking over phase one decisions ( or previouslyachieved cleanup goals) is unnecessary. Tractability isanother issue. Since demanding that plans be tidy istantamount to specifying additional ( cleanup) goals,requiring tidiness can clearly slow a planner. Further-more if a cleanup goal is unachievable, the plannermight not halt. However, as long as the mess-inducingactions in the world are easily reversible, it is straightforward to clean up for each one. Hence, trivial serial-izability assures that the overhead caused by tidinessis only linear in the number of cleanup goals posted,that is linear in the length of the plan for the top levelgoals.

possible when achieving its goals. Unfortunately, sat-isfying constraints of this form may require that theagent examine every plan to achieve the goal in orderto find the thriftiest one. We plan to seek insightsinto this problem in the extensive work on resourcemanagement in planning (Dean, Firby, & Miller 1988;Fox & Smith 1984; Wilkins 1988).

So far the discussion has focused on preventing anagent from actively harming a human, but as Asimovnoted -inaction can be just as dangerous. We saythat an agent is vigi/ant when it prevents a humanfrom harming herself. Primitive forms of vigilance arealready present in many computer systems, as the "Doyou rea//y want to delete all your files?" message at-tests.

Alternatively, one could extend dont-disturb andrestore primitives with an additional argument thatspecifies the class of agents being restricted. By writ-ing self as the first argument, one encodes the no-tions of agent safety and tidiness, and by writing Samas the argument, the agent will clean up after, and at-tempt to prevent safety violations by Sam. Finally, byproviding everyone as the first argument, one coulddemand that the agent attempt to clean up after a//other agents and attempt to prevent all safety viola-tions. For more refined behavior, other classes (besidesself and everyone) could be defined.

Our suggestion is problematic for several reasons.(1) Since the agent has no representation of the goalsthat other users are trying to accomplish, it might tryto enforce a generalized restore constraint with tidy-ing actions that directly conflict with desired goals. Inaddition, there is the question of when the agent shouldconsider the human "finished" -without an adequatemethod, the agent could tidy up while the human isstill actively working. (2) More generally, the humaninterface issues are complex -we conjecture that userswould find vigilance extremely annoying. (3) Given acomplex world where the agent does not have completeinformation, any any attempt to formalize the secondhalf of Asimov's First Law is fraught with difficulties.The agent might reject direct requests to perform use-ful work in favor of spending all of its time sensing tosee if some dangerous activity might be happening thatit might be able to prevent.

Remaining ChallengesSome changes cannot be restored, and some resourcesare legitimately consumed in the service of a goal. Tomake an omelet, you have to break some eggs. Thequestion is, "How many?" Since squandering resourcesclearly constitutes harm, we could tag a valuable re-sources with a min-consume constraint and demandthat the agent be thrifty --i. e. , that it use as little as

ConclusionThis paper explores the fundamental question origi-nally posed by Asimov: how do we stop our artifactsfrom causing us harm in the process of obeying our or-ders? This question becomes increasingly pressing aswe develop more powerful, complex, and autonomousartifacts such as robots and softbots (Etzioni, Lesh,& Segal 1993; Etzioni 1993). Since the positronicbrain envisioned by Asimov is not yet within ourgrasp, we adopt the familiar classical planning frame-work. To facilitate progress, we focused on two well-defined primitives that capture aspects of the problem:dont-disturb and restore. We showed that the well-understood, and computational tractable, mechanismof threat detection can be extended to avoid harm.

Other researchers have considered related questions.A precursor of dont-disturb is discussed in the work

9Formally, serializability (Korf 1987) means that thereexists a ordering among the subgoals which allows each tobe solved in turn without backtracking over past progress.Trivial serializability means that every subgoal ordering al-lows monotonic progress (Barrett & Weld 1993). Whilegoal ordering is often important among the top level goals,we observe that cleanup goals are usually trivially serial-izable once the block of top level goals has been solved.For example, the goal of printing a file and the constraintof restoring files to their compressed state are serializable.And the serialization ordering places the printing goal firstand the cleanup goal last. As long as the planner considersthe goals in this order, it is guaranteed to find the obvious

uncompress-print-compress plan.

Hammond, K., Converse, T ., and Grass, J. 1992. Thestabilization of environments. Artificial Intelligence.To appear .

Korf, R. 1987. Planning as search: A quantitativeapproach. Artificial Intelligence 33(1):65-88.Leveson, N. G. 1986. Software safety: Why, what,and how. ACM Computing Surveys 18(2):125-163.

Levesque, H., and Brachman, R. 1985. A fundamentaltradeoff in knowledge representation. In Brachman,R., and Levesque, H., eds., Readings in KnowledgeRepresentation. San Mateo, CA: Morgan Kaufmann.42-70.Luria, M. 1988. Knowledge Intensive Planning. Ph.D.Dissertation, UC Berkeley. Available as technical re-port UCB/CSD 88/433.McAllester, D., and Rosenblitt, D. 1991. Systematicnonlinear planning. In Proc. 9th Nat. Conf. on A.I.,634-639.Pednault, E. 1988. Synthesizing plans that con-tain actions with context-dependent effects. Compu-tational Intelligence 4(4):356-372.Pednault, E. 1989. ADL: Exploring the middleground between STRIPS and the situation calculus.In Proc. lst Int. Conf. on Principles of KnowledgeRepresentation and Reasoning, 324-332.Penberthy, J ., and Weld, D. 1992. UCPOP: A sound,complete, partial order planner for ADL. In Proc.3rd Int. Conf. on Principles of Knowledge Represen-tation and Reasoning, 103-114. Available via FTPfrom publ ail at cs .washington. edu.

Penberthy, J ., and Weld, D. 1994. Temporal planningwith continuous change. In Proc. 12th Nat. Conf. onA.I.Pollack, M. 1992. The uses of plans. Artificial Intel-ligence 57(1).Russell, S., and Wefald, E. 1991. Do the Right Thing.Cambridge, MA: MIT Press.Shoham, Y. 1988. Reasoning about Change: Timeand Causation from the Standpoint of Artificial In-telligence. Cambridge, MA: MIT Press.Tate, A. 1977. Generating project networks. In Proc.Sth Int. Joint Conf. on A.I., 888-893.

Wellman, M., and Doyle, J. 1992. Modular utility rep-resentation for decision theoretic planning. In Proc.lst Int. Conf. on A.I. Planning Systems, 236-242.Wilkins, D. E. 1988. Practical Planning. San Mateo,CA: Morgan Kaufmann.Williamson, M., and Hanks, S. 1994. Optimal plan-ning with a goal-directed utility model. In Proc. 2ndInt. Conf. on A.I. Planning Systems.

of Wilensky and more extensively by Luria (Luria1988) under the heading of "goal conflict." Similarly, aprecursor of restore is mentioned briefly in Hammondet. al's analysis of "stabilization" under the headingof "clean up plans" (Hammond, Converse, & Grass1992). Our advances include precise and unified se-mantics for the notions, a mechanism for incorporat-ing dont-disturb and restore into standard planningalgorithms, and an analysis of the computational com-plexity of enforcing safety and tidiness.

Even so, our work raises more questions than it an-swers. Are constraints like dont-disturb and restorethe "right" way to represent harm to an agent? Howdoes a~ent safety relate to the more general softwaresafety (Leveson 1986)? Can we handle tradeoffs shortof using expensive decision theoretic techniques? Whatguarantees can one provide on resource usage? Mostimportantly, how do we weaken the assumptions of astatic world and complete information?

ReferencesAllen, J. 1991. Planning as temporal reasoning. InProceedings of the Second International Conferenceon Principles of Knowledge Representation and Rea-soning, 3-14.Asimov, I. 1942. Runaround. Astounding ScienceFiction.Barrett, A., and Weld, D. 1993. Characterizing sub-goal interactions for planning. In Proc. 13th Int. JointConi on A.I., 1388-1393.Chapman, D. 1987. Planning for conjunctive goals.Artificial Intelligence 32(3):333-377.Davis, E. 1990. Representations of CommonsenseKnowledge. San Mateo, CA: Morgan Kaufmann Pub-lishers, Inc.Dean, T., Firby, J., and Miller, D. 1988. Hierar-chical planning involving deadlines, travel times, andresources. Computational Intelligence 4( 4):381-398.Drummond, M. 1989. Situated control rules. InProceedings of the First International Conference onKnowledge Representation and Reasoning.Etzioni, 0., Lesh, N ., and Segal, R. 1993. Buildingsoftbots for UNIX (preliminary report). Technical Re-port 93-09-01, University of Washington. Availablevia anonymous FTP from pub/ etzioni/ softbots/at cs .washington. edu.Etzioni, 0. 1991. Embedding decision-analytic con-trol in a learning archite<:ture. Artificial Intelligence49(1-3):129-160.Etzioni, 0. 1993. Intelligence without robots (a re-ply to brooks). AI Magazine 14(4). Available viaanonymous FTP from pub/etzioni/softbots/ atcs.washington.edu.Fox, M., and Smith, S. 1984. ISIS -a knowldges-based system for factory scheduling. Expert Systems1(1):25-49.Haddawy, P., and Hanks, S. 1992. Representations forDecision- Theoretic Planning: Utility Functions forDealine Goals. In Proc. :~rd Int. Coni on Principlesof Knowledge Representation and Reasoning.