1

Leigh Tesfatsion Professor of Econ, Math, and Electrical and Computer Engineering

Iowa State University, Ames, Iowa http://www.econ.iastate.edu/tesfatsi/

tesfatsi@iastate.edu

Presentation Slides: Last Revised 5/14/2010Presentation Slides: Last Revised 5/14/2010

www.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010www.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010.pdf.pdf

Testing Institutional Arrangements via Agent-Based Modeling

Illustrative Findings for Electric Power Illustrative Findings for Electric Power MarketsMarkets

2

Presentation OutlinePresentation Outline

Complexity of large-scale institutionsComplexity of large-scale institutions

Agent-based test beds for institutional designAgent-based test beds for institutional design

Illustration: Illustration: An ABM test bed for studying An ABM test bed for studying efficiency and welfare implications of North efficiency and welfare implications of North American electric power markets operating American electric power markets operating under new market designsunder new market designs

Sample findings (incentive misalignments)Sample findings (incentive misalignments) Incentives for price manipulationIncentives for price manipulation Incentives for congestion inducementIncentives for congestion inducement

3

Modern societies depend strongly on large-scale institutions for production & distribution of critical goods and services (e.g., energy, finance, health care, …)

Institutional outcomes depend in complicated ways on• Physical constraints Physical constraints restrictingrestricting feasible actions• RulesRules governing participation, operation & oversight• Behavioral dispositionsBehavioral dispositions of participants• Interaction patternsInteraction patterns of participants

To be useful and informative, institutional studies need to take proper account of all four elements.

Complexity of Large-Scale Institutions

4

Mathematical Modeling of Institutional Systems:

Classical vs. Agent-Based Modeling Approaches

Classical Approach (Top Down): Model the system by means of parameterized differential equations Example: Archimedes, a large-scale system of ODEs modeling

pathways of disease spread under alternative possible health care response systems

ABM Approach (Bottom Up): Model the system as a collection of interacting “agents”

Each agent is an autonomous software program encapsulating data (attributes) and methods

Agents can contain other agents as member data (permits hierarchical constructions)

5

Meaning of “Agent” in ABMMeaning of “Agent” in ABM Agent = Encapsulated bundle of data and

methods acting within a computationally constructed

world.

Agents can represent:Agents can represent: Individuals (consumers, traders, entrepreneurs,…) Social groupings (households, communities,…) Institutions (markets, corporations, gov’t agencies,…) Biological entities (crops, livestock, forests,…) Physical entities (weather, landscape, electric grids,…)

6

Meaning of “Agent” in ABM …Meaning of “Agent” in ABM …

Cognitive agentsCognitive agents are capable (in various degrees) of

Behavioral adaptationBehavioral adaptation Goal-directed learningGoal-directed learning Social communication (talking with each other!)Social communication (talking with each other!)

Endogenous formation of interaction networksEndogenous formation of interaction networks

Autonomy:Autonomy: SSelf-activation and self-determination based on private internal data and methods as well as on external data streams (including from real world)

7

Illustration: UML diagram with “is a” ↑and “has a” ↓agent relations for an

economic ABM

8

ABM vs. Object-Oriented Programming

Key distinction is autonomy of ABM CogAgents

ABMs characterized by distributed control, not simply by distributed action.

Conventional OOP objects encapsulate data and methods but do not permit self-activation and local action choice.

9

Importance of Agent Encapsulation

•In the real world, all calculations must be done by entities actually residing in the world.

•ABM forces modelers to respect this constraint.

•Procedures encapsulated into the methods of a particular agent can only be implemented using the particular resources available to that agent.

•This encapsulation achieves a more transparent and realistic representation of real-world systems composed of interacting distributed entities with limited information and computational capabilities.

10

Constructive Replacement Constructive Replacement

In principle, as a result of agent encapsulation:

Any cognitive agent interacting with an ABM through a particular input-output public interface can be replaced by a person that interacts with the ABM through this same public interface.

• Since method implementations by the cognitive agent and its human replacement need not be the same, the resulting outcomes under replacement could differ.

• The only claim here is the feasibility of replacement due to the imposition of agent boundaries in ABMs.  

11

Role of EquationsRole of Equations Any agent in an ABM can have data and/or

methods involving equations.

These equations can be the basis in part or in whole for the agent’s actions.

ABM world events are driven solely by the actionsactions undertaken by the ABM agents within their world.

ABM world events are notnot driven by equations existing outsideoutside of the data and methods of agents. For example, “sky hook” equilibrium conditions are not permitted.

12

ABM and Institutional DesignABM and Institutional Design

Key Issues:Key Issues: Will a proposed or actual design promote efficient,

fair, and orderly social outcomes over time? Will the design give rise to unintended

consequences?

ABM Culture-Dish Approach:ABM Culture-Dish Approach: Develop a computational world embodying the

design, physical constraints, strategic participants, …

Set initial world conditions (agent states).

Let the world evolve with no further intervention, and observe and evaluate the resulting outcomes.

13

Agent-Based Test Bed Agent-Based Test Bed Development viaDevelopment via

Iterative Participatory ModelingIterative Participatory Modeling

Stakeholders and researchers from multiple disciplines join together in a repeated loopingrepeated looping through four stages of analysisthrough four stages of analysis:

1) Field work and data collection2) Role-playing games/human-subject

experiments3) Incorporate findings into agent-based test

bed 4) Generate hypotheses through intensive computational experiments.

14

Project Directors: Leigh Tesfatsion (Prof. of Econ, Math, & ECpE,

ISU) Dionysios Aliprantis (Ass’t Prof. of ECpE, ISU) David Chassin (Staff Scientist, PNNL/DOE) Research Assoc’s: Dr. Junjie Sun (Fin. Econ, OCC, U.S. Treasury,

Wash, D.C.) Dr. Hongyan Li (Consulting Eng., ABB Inc., Raleigh, NC) Research Assistants:Research Assistants: Huan Zhao (ISU Econ PhD Candidate); Chengrui Cai (ISU ECpE PhD Candidate); Pedram Jahangiri & Auswin Thomas (ISU ECpE grad students) ** Supported by grants from DOE/PNNL (Pacific Northwest Supported by grants from DOE/PNNL (Pacific Northwest

National National Laboratory, and the ISU EPRC (Electric Power Research Laboratory, and the ISU EPRC (Electric Power Research

Center)Center)

Project:Project: Integrated Wholesale/Retail Power Integrated Wholesale/Retail Power System Operation with Smart-Grid System Operation with Smart-Grid

FunctionalityFunctionality

15

Retail & Wholesale Power System Operations

Source: http://www.nerc.com/page.php?cid=1|15

16

Our Retail/Wholesale Test Bed Platform Based on Texas (ERCOT) Retail/Wholesale

Structure

Wholesale Test bed (AMESAMES)developed by ISU

Team

Retail Test bed (GridLAB-DGridLAB-D) developed by

DOE/PNNL

Seaming in Seaming in ProgressProgress

x

x

Bilateral Contracts

17

AMES/GridLAB-D Seaming: Timing DetailsAMES/GridLAB-D Seaming: Timing Details

18

Meaning of “Smart Grid Functionality”?

For our project purposes:

Smart-grid functionalitySmart-grid functionality = Service-oriented grid enhancements permitting more responsiveness to needs and preferences of retail customers.

Examples:Examples: Introduction of advanced metering and other technologies to support flexible retail contracting between suppliers and

retail consumers embedding and use of distributed energy

resources

19

Project Context:Project Context: North American North American restructuring of wholesale power marketsrestructuring of wholesale power markets

In April 2003 the U.S. Federal Energy Regulatory Commission (FERC) proposed adoption of a wholesale power market design with particular core features.

Over 50% of North American generation now operates under some variant of the FERC design.

Adopters to Date: New York (NY-ISO), mid-Atlantic states (PJM), New England (ISO-NE), Midwest/Manitoba (MISO), Texas (ERCOT), Southwest (SPP), and California (CAISO)

Note:Note: Ontario (IESO), Alberta (AESO) , and other Canadian provinces have not fully adopted FERC design

20

Regions Operating Under Some Version of FERC Design

http://www.ferc.gov/industries/electric/indus-act/rto/rto-map.asp

21

Core Features of FERC’s Market Core Features of FERC’s Market DesignDesign

• Market to be managed by an independent system operator (ISO) having no ownership stake

• Two-settlement system: Concurrent operation of day-ahead (forward) & real-time (spot) markets

• Transmission grid congestion managed via Locational Marginal Prices (LMPs), where LMP at bus k = least cost of servicing 1 additional MW of power at bus k

• Oversight & market power mitigation by outside agency

Has led in practice to complicated systems difficult to analyze by standard analytical & statistical tools !

22

Example: Complex MISO Market Organization

Business Practices Manual 001-r1 (1/6/09)

Two-Settlement Power Market System under LMP

Core ofCore of FERC FERC

designdesign

Ss

AMES project to date

Xx

Xx

23

Actual Electricity Prices in Midwest ISO Actual Electricity Prices in Midwest ISO (MISO)(MISO)

April 25, 2006, at 19:55Note this price,$156.35

24

Five Minutes Later…73% drop in price in 5 minutes!

25

Actual Electricity Prices in Midwest ISO Actual Electricity Prices in Midwest ISO (MISO)(MISO)

September 5, 2006, 14:30

Note this price, $226.25

26

Five Minutes Later…79% drop in price in 5 minutes!

27

Project Work to Date: Wholesale Project Work to Date: Wholesale LevelLevel

•Development and open-source release of AMES (Agent-based Modeling of Electricity Systems)

•AMES = ABM test bed with core FERC design features

•Used to test performance under FERC design

•Used to test performance under modifications of design

AMES Homepage (code/manual/publications):AMES Homepage (code/manual/publications): http://www.econ.iastate.edu/tesfatsi/AMESMarketHome.htm

28

AMES (V2.05) Architecture AMES (V2.05) Architecture (based on business practices manuals for (based on business practices manuals for

MISO/ISO-NE)MISO/ISO-NE)

Two-settlement system Day-ahead market (double auction, financial contracts)

Real-time market (settlement of differences)

AC transmission grid Generation Companies (GenCos)Generation Companies (GenCos) & Load-Serving Load-Serving

Entities (LSEs)Entities (LSEs) located at user-specified transmission buses Grid congestion managed via Locational Marginal Prices Locational Marginal Prices

(LMPs)(LMPs) LMP at bus k LMP at bus k = Least cost of servicing one additional MW = Least cost of servicing one additional MW

of power at bus kof power at bus k.

Independent System Operator (ISO) System reliability assessments Day-ahead scheduling via

bid/offer based optimal power bid/offer based optimal power flow (OPF)flow (OPF)

Real-time dispatch

Traders GenCos (sellers) LSEs (buyers) LearningLearning capabilities

29

AMES Modular & Extensible Architecture AMES Modular & Extensible Architecture (Java)(Java)

Market protocols & AC transmission grid structure― Graphical user interface (GUI) & modularized class structure

permit easy experimentation with alternative parameter settings and alternative institutional/grid constraints

Learning representations for traders― Java Reinforcement Learning Module (JReLM)

― “Tool box” permitting experimentation with a wide variety of learning methods (Roth-Erev, Temp Diff/Q-learning,…)

Bid/offer-based optimal power flow formulation― Java DC Optimal Power Flow Module (DCOPFJ)― Permits experimentation with various DC OPF formulations

Output displays and dynamic test cases― Customizable chart/table displays & 5-bus/30-bus test cases

30

Illustrative AMES Experimental Illustrative AMES Experimental Findings: A 5-Bus Test CaseFindings: A 5-Bus Test Case

Definition: Definition: Incentive misalignmentIncentive misalignment →→ Institutional design fails to align incentives of market participants with efficiency objectives (non-wastage of resources) and/or welfare objectives (socially desirable distribution of net benefits)

Experiments Reported Below:Experiments Reported Below: Incentive misalignment problems under FERC wholesale power market design for a range of experimental treatments:• Generator learning [intensive parameter sweep]

• Sensitivity of wholesale demand to price [0 to 100%]

31

G4

G2G1 G3

G5

Bus 1 Bus 2 Bus 3

LSE 3LSE 3Bus 4Bus 5

LSELSE 1 1 LSE 2LSE 2

Five GenCo sellers G1,…,G5 and three LSE buyers LSE 1, LSE 2, LSE 3

5-Bus Transmission Grid (Used in many ISO business practice/training

manuals)

$$$250 MW Capacity$

$ $ $$

32

GenCo True Cost & Capacity Attributes

32

33

Activities of AMES ISO During Each Operating Activities of AMES ISO During Each Operating Day D:Day D:

Timing Adopted from Midwest ISO (MISO)Timing Adopted from Midwest ISO (MISO)

33

00:00

11:00

16:00

23:00

Real-time

(spot)market

forday D

Day-ahead marketfor day D+1

ISO collects bids/offers from LSEs and GenCos

ISO evaluates LSE demand bids and GenCo supply offers

ISO solves D+1 DC OPF and posts D+1 dispatch

and LMP schedule

Day-ahead settlementReal-time settlement

34

AMES ISO (Market Operator)Public Access:Public Access: //// Public MethodsPublic Methods getWorldEventSchedule( clock time,… ); getMarketProtocols( bid/offer reporting, settlement,… ); Methods for receiving data; Methods for retrieving stored ISO data;

Private Access:Private Access: //// Private MethodsPrivate Methods Methods for gathering, storing, posting, & sending data; Method for solving hourly DC optimal power flow; Methods for posting schedules and carrying out settlements; Methods for implementing market power mitigation; //// Private DataPrivate Data Historical data (e.g., cleared bids/offers, market prices,…); Address book (communication links);

35

AMES ISO Solves Hourly DC Optimal Power Flow AMES ISO Solves Hourly DC Optimal Power Flow (OPF)(OPF)

GenCos report hourly supply offers and LSEs report fixed & price-sensitive hourly demand bids to ISO for day-ahead market

Subject to

Fixed and price-sensitive demand bids for LSE j

LSE gross buyer surplus

GenCo-reported total avoidable costs

R R

RUi

Max Max0 SLMaxj Purchase

capacityinterval for LSE j

Dual variable for

this bus-k balance

constraint givesLMP for bus k

Operating capacity

interval for GenCo i

36

AMES ISO Solves DC-OPF via DCOPFJ AMES ISO Solves DC-OPF via DCOPFJ ModuleModule

36

DC-OPF raw data (SI)

DCOPFJ Shell

Per Unit conversion

Form SCQP matrices

QuadProgJ: An SCQP solver

Per Unit SCQP output

Solution output (SI)

37

AMES Generation Company (Seller)AMES Generation Company (Seller)Public Access:Public Access: // Public Methods// Public Methods getWorldEventSchedule( clock time,… ); getMarketProtocols( ISO market power mitigation,… ); Methods for receiving data; Methods for retrieving GenCo data;

Private Access:Private Access: //// Private MethodsPrivate Methods Methods for gathering, storing, and sending data; Methods for calculating own expected & actual net earnings; Method for updating own supply offers (LEARNING); //// Private DataPrivate Data Own capacity, grid location, cost function, current wealth… ; Data recorded about external world (prices, dispatch,…); Address book (communication links);

38

AMES GenCos are learners who report strategic hourly supply offers to ISO

39

In 5-bus study, AMES GenCos use VRE learning (version of Roth-Erev stochastic

reinforcement learning)

Each GenCo maintains action choice propensities q, normalized to choice probabilities Prob, to choose actions (supply offers). A good (bad) reward rk resulting from an action ak results in an increase (decrease) in both qk and Probk.

Action Choice a1

Action Choice a2

Action Choice a3

Choice Propensity q1 Choice Probability Prob1

Choice Propensity q2

Choice Propensity q3

Choice Probability Prob2

Choice Probability Prob3

rupdatechoose normaliz

e

42

AMES GenCo learning implemented via JReLM AMES GenCo learning implemented via JReLM module module

(Java Reinforcement Learning Module developed by(Java Reinforcement Learning Module developed by Charles J. Gieseler, Comp Sci M.S. Thesis, 2005)

Market Simulation

Learning Agent

43

AMES Load-Serving Entity (Buyer)Public Access:Public Access: // Public MethodsPublic Methods getMarketProtocols(posting, trade, settlement); getMarketProtocols(ISO market power mitigation); Methods for receiving data; Methods for retrieving LSE data;

Private Access:Private Access: // Private MethodsPrivate Methods Methods for gathering, storing, and sending data; Methods for calculating own expected & actual net earnings; // Private DataPrivate Data Own downstream demand, grid location, current wealth…; Data recorded about external world (prices, dispatch,…); Address book (communication links);

44

AMES LSE Hourly Demand-Bid Formulation

Hourly demand bid for each LSE j

Fixed + Price-Sensitive Demand Bid

FixedFixed demand bid = pFLj (MWs)

Price-sensitivePrice-sensitive demand bid = Inverse demand function for

real power pSLj (MWs) over

a purchase capacity interval:

Fj(pSLj) = cj - 2dj pS

Lj

0 ≤ pSLj ≤ SLMaxj

45

R Measure for Systematically Varying the Amount of Demand-Bid Price Sensitivity

45

R = SLMaxSLMaxjj /[ ppFFLjLj + SLMaxSLMaxjj ]

For LSE j in Hour H: pFpFLjLj = Fixed demand for real power (MWs) SLMaxSLMaxjj = Maximum potential price-sensitive demand (MWs)

$/MWh

R=0.0 R=0.5

R=1.0

$/MWh $/MWh

SLMaxjSLMaxjpFLj pF

Lj pLj pLj pLj

D D

D

(100% Fixed Demand)

(100% Price-Sensitive Demand)

46

LSE Hourly Fixed Demands for LSE Hourly Fixed Demands for R=0.0R=0.0

17 = Peak demand hour

47

Mean total GenCo net earningsotal GenCo net earnings (Day 1000) for R=0.0 under varied VRE learning parameter

settings

Small beta ≅ “zero-intelligence” budget-constrainedtrading.

Learning matters !

= Sweet spot

region =

Selected

48

Price outcomes with/without GenCo learning as demand varies from R=0.0 (100% fixed) to R=1.0 (100% price

sensitive)

With GenCo Learning

(Day 1000)

R R

Avg LMP (locational marginal price)

Avg LI (Lerner Market Power Index)

49

Single-Run Illustration of Findings for R=0.0 (100% Fixed Demand)W/O Gen Learning (Day 1000)

With Gen Learning (Day 1000)

50

Summary of price findings under learning Summary of price findings under learning and and

price-sensitive demand treatmentsprice-sensitive demand treatments

BOTTOM LINE:

Even with 100% price-sensitive demand bids (R=1.0),Even with 100% price-sensitive demand bids (R=1.0), prices are much higher under GenCo learning prices are much higher under GenCo learning

due todue to strategic supply offers (econ capacity strategic supply offers (econ capacity

withholding)! withholding)! NEEDED: Countervailing powerCountervailing power == AActive LSE demand

bidding as well as active GenCo supply offers at wholesale level to support more competitive pricing to support more competitive pricing at wholesale.at wholesale.

POSSIBLE MEANS: Integrated wholesale/retail restructuringIntegrated wholesale/retail restructuring

providing array of price-sensitive retail contracts providing array of price-sensitive retail contracts and permitting retail consumers to select their LSE and permitting retail consumers to select their LSE suppliers suppliers

51

Unfortunately, since 2000 Enron scandal,

retail restructuring has been slowed

Source: www.eia.doe.gov/cneaf/electricity/page/restructuring/restructure_elect.html

(Jan 2010) Retail

52

Extraction of net surplus by ISOs in day-ahead energy markets under Locational Marginal Pricing (LMP).H. Li & L. Tesfatsion, IEEE Transactions on Power Systems, 2011, to appear.Day-ahead market activitieson a typical operating day D:

53

ISO Net Surplus Extraction ExampleISO Net Surplus Extraction Example

Cleared load = pFL. LSE

at bus 2 pays LMP2 > LMP1 for each unit of pF

L. M units of pF

L are supplied by cheaper G1 at bus 1 who receives only LMP1 per unit. ISO collects difference:ISO Net SurplusISO Net Surplus = [ LSE Payments – GenCo Revenues ] = M x [LMP2 – LMP1]

54

ISO Net Surplus Extraction Example … ISO Net Surplus Extraction Example … Cont.Cont.

ISO Net Surplus:ISO Net Surplus: INS=M x [LMP2 – LMP1]GenCo Net Surplus:GenCo Net Surplus: Area S1 + Area S2LSE Net Surplus:LSE Net Surplus: Area BTotal Net Surplus:Total Net Surplus: TNSTNS = = [INS+S1+S2+B]ISO Objective (OPF):ISO Objective (OPF):

maximize TNS subject to trans/gen constraints.

55

5-Bus Test Case Results Without GenCo Learning5-Bus Test Case Results Without GenCo Learning:: Daily ISO net surplus as LSE demand varies from

R=0.0 (100% fixed) to R=1.0 (100% price sensitive)

55

R Value

$

56

5-Bus Test Case Results with GenCo VRE Learning:

Mean ISO net surplus on Day 1000 as LSE demand varies from R=0.0 (100% fixed) to R=1.0 (100% price sensitive)

56

R Value

$

57

Empirical Comparisons

From PJM 2008 report:ISO net surplus from day-ahead market: $2.66 billion$2.66 billion

From MISO 2008 report:ISO net surplus from day-ahead market: $500$500 millionmillion

From CAISO 2008 report:ISO net surplus from day-ahead inter-zonal congestion charges: $176 million$176 million.

From ISO-NE 2008 report: Combined ISO net surplus for real-time and day-ahead

markets: $121 million$121 million.57

58

Implications of ISO Net Surplus Findings

ISO net surplus extractions not well aligned with market efficiency

Treatments resulting in greater GenCo economic capacity withholding (hence higher & more volatile LMPs) also result in greater ISO & GenCo net surplus

ISO net surplus collections should be allocated for ex ante remedy of structural/behavioral problems that encourage GenCo economic capacity withholding.

Should not be used ex post for LMP payment offsets and LMP risk hedge support (current norm)

59

ConclusionsConclusions Restructured wholesale power markets are complex large-

scale institutions encompassing physical constraints, administered rules of operation, and strategic human participants.

Agent-based test beds permit the systematic dynamic study of such institutions through intensive computational experiments.

For increased empirical validity, test beds should be iteratively developed with ongoing input from actual market participants.

To increase usefulness for research/teaching/training and to aid knowledge accumulation, test beds should be open source.

60

On-Line ResourcesOn-Line Resources Presentation SlidesPresentation Slides www.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010.pdfwww.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010.pdf AMES Test Bed Homepage AMES Test Bed Homepage (Code/Manual/Publications)(Code/Manual/Publications) www.econ.iastate.edu/tesfatsi/AMESMarketHome.htm Agent-Based Electricity Market ResearchAgent-Based Electricity Market Research www.econ.iastate.edu/tesfatsi/aelect.htm Agent-Based Computational Economics HomepageAgent-Based Computational Economics Homepage www.econ.iastate.edu/tesfatsi/ace.htmwww.econ.iastate.edu/tesfatsi/ace.htm ISU Electric Energy Economics (E3) GroupISU Electric Energy Economics (E3) Group www.econ.iastate.edu/tesfatsi/E3GroupISU.htmwww.econ.iastate.edu/tesfatsi/E3GroupISU.htm