Testing Institutional Arrangements by Waterloo Institute for Complexity and Innovation

Testing Institutional Arrangements via Agent-Based Modeling Illustrative Findings for Electric Power Markets Leigh Tesfatsion Professor of Econ, Math, and Electrical and Computer Engineering Iowa State University, Ames, Iowa http://www.econ.iastate.edu/tesfatsi/ tesfatsi@iastate.edu

Presentation Slides: Last Revised 5/14/2010 www.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010.pdf 1

Presentation Outline 

Complexity of large-scale institutions



Agent-based test beds for institutional design



Illustration: An ABM test bed for studying efficiency and welfare implications of North American electric power markets operating under new market designs



Sample findings (incentive misalignments)

Complexity of Large-Scale Institutions  Modern societies depend strongly on large-scale institutions for production & distribution of critical goods and services (e.g., energy, finance, health care, …)

 Institutional outcomes depend in complicated ways on Physical constraints restricting feasible actions Rules governing participation, operation & oversight Behavioral dispositions of participants Interaction patterns of participants

• • • •

Mathematical Modeling of Institutional Systems: Classical vs. Agent-Based Modeling Approaches  Classical

Approach (Top Down): Model the system

by means of parameterized differential equations

− Example:

Archimedes, a large-scale system of ODEs modeling pathways of disease spread under alternative possible health care response systems

 ABM

Approach (Bottom Up): Model the system as

a collection of interacting “agents”

− Each agent is an autonomous software program

Meaning of “Agent” in ABM Agent = Encapsulated bundle of data and methods

acting within a computationally constructed world.

 Agents can represent: Individuals (consumers, traders, entrepreneurs,…) Social groupings (households, communities,…) Institutions (markets, corporations, gov’t agencies,…) Biological entities (crops, livestock, forests,…) Physical entities (weather, landscape, electric grids,…) 5

Meaning of “Agent” in ABM … Cognitive agents are capable (in various degrees) of  Behavioral adaptation  Goal-directed learning  Social communication (talking with each other!)  Endogenous formation of interaction networks 6

Illustration: UML diagram with “is a” “has a”

↓

↑

and

agent relations for an economic ABM

ABM vs. Object-Oriented Programming

−

Key distinction is autonomy of ABM CogAgents

−

ABMs characterized by distributed control, not simply by distributed action.

−

Conventional OOP objects encapsulate data and methods but do not permit self-activation and local action choice.

Importance of Agent Encapsulation the real world, all calculations must be done by entities • In actually residing in the world.

• ABM forces modelers to respect this constraint. encapsulated into the methods of a particular • Procedures agent can only be implemented using the particular resources available to that agent.

Constructive Replacement In principle, as a result of agent encapsulation:

Any cognitive agent interacting with an ABM

through a particular input-output public interface can be replaced by a person that interacts with the ABM through this same public interface.

•

Since method implementations by the cognitive agent and its human replacement need not be the same, the resulting outcomes under replacement could differ. 10

Role of Equations 

Any agent in an ABM can have data and/or methods involving equations.



These equations can be the basis in part or in whole for the agent’s actions.



ABM world events are driven solely by the actions undertaken by the ABM agents within their world.



ABM world events are not driven by equations existing outside of the data and methods of agents. For example, “sky hook” equilibrium conditions are not 11 permitted.

ABM and Institutional Design Key Issues: 



Will a proposed or actual design promote efficient, fair, and orderly social outcomes over time? Will the design give rise to unintended consequences?

ABM Culture-Dish Approach: 

Develop a computational world embodying the design, physical constraints, strategic participants, …



Set initial world conditions (agent states).



Let the world evolve with no further intervention, and 12 observe and evaluate the resulting outcomes.

Agent-Based Test Bed Development via Iterative Participatory Modeling  Stakeholders and researchers from multiple disciplines join together in a repeated looping through four stages of analysis: analysis 1) Field work and data collection 2) Role-playing games/human-subject experiments 3) Incorporate findings into agent-based test bed 4) Generate hypotheses through intensive computational experiments. 13

Project: Integrated Wholesale/Retail Power System Operation with Smart-Grid Functionality Project Directors:

Leigh Tesfatsion (Prof. of Econ, Math, & ECpE, ISU) Dionysios Aliprantis (Ass’t Prof. of ECpE, ISU) David Chassin (Staff Scientist, PNNL/DOE)

Research Assoc’s: Dr. Junjie Sun (Fin. Econ, OCC, U.S. Treasury, Wash, D.C.) Dr. Hongyan Li (Consulting Eng., ABB Inc., Raleigh, NC) Research Assistants: Huan Zhao (ISU Econ PhD Candidate); Chengrui Cai (ISU ECpE PhD Candidate); Pedram Jahangiri & Auswin Thomas (ISU ECpE grad students)

* Supported by grants from DOE/PNNL (Pacific Northwest National

Laboratory, and the ISU EPRC (Electric Power Research Center)14

Retail & Wholesale Power System Operations Source: http://www.nerc.com/page.php?cid=1|15

Our Retail/Wholesale Test Bed Platform

Based on Texas (ERCOT) Retail/Wholesale Structure

Bilateral Contracts x

Wholesale

Test bed (AMES) AMES developed by ISU Team

Seaming in Progress

Retail

Test bed (GridLAB-D) GridLAB-D developed by DOE/PNNL

AMES/GridLAB-D Seaming: Timing Details

Meaning of “Smart Grid Functionality”?  For our project purposes: Smart-grid functionality =

Service-oriented grid enhancements permitting more responsiveness to needs and preferences of retail customers.

Examples: Introduction of advanced metering and

other technologies to support flexible retail contracting between suppliers and retail consumers embedding and use of distributed energy resources 18

− −

Project Context: North American

restructuring of wholesale power markets  In April 2003 the U.S. Federal Energy Regulatory

Commission (FERC) proposed adoption of a wholesale power market design with particular core features.

 Over 50% of North American generation now

operates under some variant of the FERC design.



Adopters to Date: New York (NY-ISO), midAtlantic states (PJM), New England (ISO-NE), Midwest/Manitoba (MISO), Texas (ERCOT), Southwest (SPP), and California (CAISO)

Regions Operating Under Some Version of FERC Design http://www.ferc.gov/industries/electric/indus-act/rto/rto-map.asp

Core Features of FERC’s Market Design •

Market to be managed by an independent system operator (ISO) having no ownership stake

•

Two-settlement system: Concurrent operation of day-ahead (forward) & real-time (spot) markets

•

Transmission grid congestion managed via Locational Marginal Prices (LMPs), where LMP at bus k = least

Example: Complex MISO Market Organization Business Practices Manual 001-r1 (1/6/09) X x S s

X x

Core of FERC design Two-Settlement Power Market System under LMP AMES project to date

Actual Electricity Prices in Midwest ISO (MISO) April 25, 2006, at 19:55 Note this price,$156.35

Five Minutes Later… 73% drop in price in 5 minutes!

Actual Electricity Prices in Midwest ISO (MISO) September 5, 2006, 14:30 Note this price, $226.25

Five Minutes Later… 79% drop in price in 5 minutes!

Project Work to Date: Wholesale Level

•

Development and open-source release of AMES (Agent-based Modeling of Electricity Systems)

•

AMES = ABM test bed with core FERC design features

•

Used to test performance under FERC design 27

Used to test performance under modifications of design

AMES (V2.05) Architecture

(based on business practices manuals for MISO/ISO-NE) 

Traders 

GenCos (sellers)



LSEs (buyers) Learning capabilities

 



Independent System Operator (ISO)  System reliability assessments  Day-ahead scheduling via bid/offer based optimal power flow (OPF)  Real-time dispatch

Two-settlement system  Day-ahead market (double auction, financial contracts)  Real-time market (settlement of differences)

AC transmission grid  Generation Companies (GenCos) & Load-Serving Entities (LSEs) located at user-specified transmission buses  Grid congestion managed via Locational Marginal Prices (LMPs)  LMP at bus k = Least cost of servicing one additional MW of 28 power at bus k. k

AMES Modular & Extensible Architecture (Java) 

Market protocols & AC transmission grid structure ―



Graphical user interface (GUI) & modularized class structure permit easy experimentation with alternative parameter settings and alternative institutional/grid constraints

Learning representations for traders ―

Java Reinforcement Learning Module (JReLM)

―

“Tool box” permitting experimentation with a wide variety of learning methods (Roth-Erev, Temp Diff/Q-learning,…)

Bid/offer-based optimal power flow formulation ― ―

Java DC Optimal Power Flow Module (DCOPFJ) Permits experimentation with various DC OPF formulations

Illustrative AMES Experimental Findings: A 5-Bus Test Case Definition: Incentive misalignment

→

Institutional design fails to align incentives of market participants with efficiency objectives (non-wastage of resources) and/or welfare objectives (socially desirable distribution of net benefits)

Experiments Reported Below:

Incentive misalignment problems under FERC wholesale power market design for a range of experimental treatments:

• •

Generator learning [intensive parameter sweep] Sensitivity of wholesale demand to price [0 to

5-Bus Transmission Grid

(Used in many ISO business practice/training manuals) Five GenCo sellers G1,…,G5 and three LSE buyers LSE 1, LSE 2, LSE 3

Bus 5

LSE 3

Bus 4

$$$

250 MW Capacity

Bus 1

Bus 3

Bus 2 G1 G2 $

LSE 1

G3 $$

LSE 2 31

GenCo True Cost & Capacity Attributes

32 32

Activities of AMES ISO During Each Operating Day D: Timing Adopted from Midwest ISO (MISO) 00:00

Day-ahead market for day D+1

Realtime (spot) market for day D

ISO collects bids/offers from LSEs and GenCos 11:00

ISO evaluates LSE demand bids and GenCo supply offers 16:00

Real-time settlement

23:00

ISO solves D+1 DC OPF and posts D+1 dispatch and LMP schedule Day-ahead settlement

33 33

AMES ISO (Market Operator) Public Access: // Public Methods getWorldEventSchedule( clock time,… ); getMarketProtocols( bid/offer reporting, settlement,… ); Methods for receiving data; Methods for retrieving stored ISO data;

Private Access: // Private Methods Methods for gathering, storing, posting, & sending data; Method for solving hourly DC optimal power flow; Methods for posting schedules and carrying out settlements; Methods for implementing market power mitigation; // Private Data Historical data (e.g., cleared bids/offers, market prices,…); Address book (communication links); 34

AMES ISO Solves Hourly DC Optimal Power Flow (OPF) GenCos report hourly supply offers and LSEs report fixed & price-sensitive hourly demand bids to ISO for day-ahead market LSE gross buyer surplus GenCo-reported total avoidable costs

Subject to

Fixed and pricesensitive demand bids for LSE j Operating capacity interval for GenCo i

RU i

Max0

SLMax Max j

Dual variable for this bus-k balance constraint gives LMP for bus k Purchase capacity interval for LSE j 35

AMES ISO Solves DC-OPF via DCOPFJ Module DC-OPF raw data (SI)

Solution output (SI)

DCOPFJ Shell Per Unit conversion Per Unit SCQP output Form SCQP matrices QuadProgJ: An SCQP solver 36 36

AMES Generation Company (Seller) Public Access: // Public Methods getWorldEventSchedule( clock time,… ); getMarketProtocols( ISO market power mitigation,… ); Methods for receiving data; Methods for retrieving GenCo data;

Private Access: // Private Methods Methods for gathering, storing, and sending data; Methods for calculating own expected & actual net earnings; Method for updating own supply offers (LEARNING); // Private Data Own capacity, grid location, cost function, current wealth… ; Data recorded about external world (prices, dispatch,…); Address book (communication links); 37

AMES GenCos are learners who report strategic hourly supply offers to ISO

In 5-bus study, AMES GenCos use VRE learning (version of Roth-Erev stochastic reinforcement learning)

choose Action Choice a1

update

normalize

Choice Propensity q1

Choice Probability Prob1

Action Choice a2

Choice Propensity q2

Choice Probability Prob2

Action Choice a3

Choice Propensity q3

Choice Probability Prob3



Each GenCo maintains action choice propensities q, normalized to choice probabilities Prob, to choose actions (supply offers). A good (bad) reward rk resulting from an action ak results in an increase (decrease) in both qk and Probk. 39

AMES GenCo learning implemented via JReLM module (Java Reinforcement Learning Module developed by Charles J. Gieseler, Comp Sci M.S. Thesis, 2005)

Market Simulation Learning Agent

AMES Load-Serving Entity (Buyer) Public Access: // Public Methods getMarketProtocols(posting, trade, settlement); getMarketProtocols(ISO market power mitigation); Methods for receiving data; Methods for retrieving LSE data;

Private Access: // Private Methods

Methods for gathering, storing, and sending data; Methods for calculating own expected & actual net earnings;

// Private Data Own downstream demand, grid location, current wealth…; Data recorded about external world (prices, dispatch,…); Address book (communication links); 43

AMES LSE Hourly Demand-Bid Formulation  Hourly demand bid for each LSE j Fixed + Price-Sensitive Demand Bid   Price-sensitive demand bid = Inverse demand function for a purchase capacity interval:

Fj(pSLj)

= cj - 2dj pSLj

0 ≤ pSLj ≤ SLMaxj

R Measure for Systematically Varying the Amount of Demand-Bid Price Sensitivity For LSE j in Hour H: pFLj = Fixed demand for real power (MWs) SLMaxj = Maximum potential price-sensitive demand (MWs) R = SLMaxj /[ pFLj + SLMaxj ] $/MWh

$/MWh

D D

pFLj

pLj

R=0.0 (100% Fixed Demand)

pFLj SLMaxj

R=0.5

pLj

SLMaxj

pLj

R=1.0 45 (100% Price-Sensitive Demand) 45

LSE Hourly Fixed Demands for R=0.0 17 = Peak demand hour

Mean total GenCo net earnings (Day 1000) for R=0.0 under varied VRE learning parameter settings



Small beta ≅ “zero-intelligence” budget-constrained trading. Learning matters ! = Sweet spot

region



= Selected 47

Price outcomes with/without GenCo learning as demand varies from R=0.0 (100% fixed) to R=1.0 (100% price sensitive) Avg LMP (locational marginal price)

Avg LI (Lerner Market Power Index)

With GenCo Learning (Day 1000)

Single-Run Illustration of Findings for R=0.0 (100% Fixed Demand)

W/O Gen Learning

(Day 1000)

With Gen Learning

(Day 1000)

Summary of price findings under learning and price-sensitive demand treatments

 BOTTOM

LINE:

Even with 100% price-sensitive demand bids (R=1.0), prices are much higher under GenCo learning due to strategic supply offers (econ capacity withholding)!



NEEDED:

Countervailing power = Active LSE demand bidding as well as active GenCo supply offers at wholesale level to support more competitive pricing at wholesale. POSSIBLE MEANS: Integrated wholesale/retail restructuring providing 50 array of price-sensitive retail contracts and permitting



Unfortunately, since 2000 Enron scandal, retail restructuring has been slowed Retail

(Jan 2010)

Source: www.eia.doe.gov/cneaf/electricity/page/restructuring/restructure_elect.html

Extraction of net surplus by ISOs in day-ahead energy markets under Locational Marginal Pricing (LMP). H. Li & L. Tesfatsion, IEEE Transactions on Power Systems, 2011, to appear. Day-ahead market activities on a typical operating day D: 52

ISO Net Surplus Extraction Example Cleared load = pFL. LSE at bus 2 pays LMP2 > LMP1 for each unit of pFL. M units of pFL are supplied by cheaper G1 at bus 1 who receives only LMP1 per unit. ISO collects difference:

ISO Net Surplus = [ LSE Payments – GenCo Revenues ] = M x [LMP2 – LMP1]

ISO Net Surplus Extraction Example … Cont. ISO Net Surplus: INS=M x [LMP2 – LMP1] GenCo Net Surplus: Area S1 + Area S2 LSE Net Surplus: Area B Total Net Surplus: TNS = [INS+S1+S2+B] ISO Objective (OPF): maximize TNS subject to trans/gen constraints. 54

5-Bus Test Case Results Without GenCo Learning: Daily ISO net surplus as LSE demand varies from R=0.0 (100% fixed) to R=1.0 (100% price sensitive) $

R Value 55 55

5-Bus Test Case Results with GenCo VRE Learning: Mean ISO net surplus on Day 1000 as LSE demand varies from R=0.0 (100% fixed) to R=1.0 (100% price sensitive) $

R Value 56 56

Empirical Comparisons 

From PJM 2008 report: ISO net surplus from day-ahead market: $2.66 billion



From MISO 2008 report: ISO net surplus from day-ahead market: $500 million



From CAISO 2008 report: ISO net surplus from day-ahead inter-zonal congestion charges: $176 million. million



From ISO-NE 2008 report: Combined ISO net surplus for real-time and day-ahead markets: $121 million. million 57 57

Implications of ISO Net Surplus Findings 

ISO net surplus extractions not well aligned with market efficiency



Treatments resulting in greater GenCo economic capacity withholding (hence higher & more volatile LMPs) also result in greater ISO & GenCo net surplus



ISO net surplus collections should be allocated for ex ante remedy of structural/behavioral problems that encourage GenCo economic capacity withholding.



Should not be used ex post for LMP payment offsets 58 and LMP risk hedge support (current norm)

Conclusions 

Restructured wholesale power markets are complex large-scale institutions encompassing physical constraints, administered rules of operation, and strategic human participants.



Agent-based test beds permit the systematic dynamic study of such institutions through intensive computational experiments.



For increased empirical validity, test beds should be iteratively developed with ongoing input from actual market participants.



To increase usefulness for research/teaching/training and to aid knowledge accumulation, test beds should be open source.

On-Line Resources 

Presentation Slides

www.econ.iastate.edu/tesfatsi/TestInstViaABM.Waterloo2010.pdf 

AMES Test Bed Homepage (Code/Manual/Publications) www.econ.iastate.edu/tesfatsi/AMESMarketHome.htm



Agent-Based Electricity Market Research www.econ.iastate.edu/tesfatsi/aelect.htm



Agent-Based Computational Economics Homepage www.econ.iastate.edu/tesfatsi/ace.htm



ISU Electric Energy Economics (E3) Group www.econ.iastate.edu/tesfatsi/E3GroupISU.htm