context-dependent network agents specific technology goals of funded effort: distributed computation...

Context-Dependent Network Agents

Specific technology goals of funded effort:• distributed computation and control • applications of synchronized sampling• collaboration techniques

Accomplishments• distributed rolling horizon strategies• protocols for dynamic collaboration• Markov modeling and multi-mode learning • context-dependent FACTS and PSS controls• dependable & secure protective relaying strategies• distributed power flow • evaluation of reactive power control in deregulated markets• remote-access real-time control emulator

Next steps• context detection and identification• integrated CDNA strategies• CDNA modeling, simulation and validation• extensions to computer, traffic, biological, and C2 networks

Goals/Progress/Directions

Concept

Programmatic Information

PI: Bruce H. KroghDept. of ECE, Carnegie Mellon University5000 Forbes AvenuePittsburgh, PA 15213-3890ph. +1 412 268 2472 fax -3890e-mail: [email protected]

Contract Number: WO8333-05

Start and duration of funded effort: Jan. 1, 1999 through Dec. 31, 2003

Objective: Improve agility and robustness (survivability) of large dynamic networks through agents that are:

• widely distributed• context-dependent• semi-autonomous• collaborative

• multi-modal• self-improving• local in sensing & influence• multi-objective

Physical Network

G

G

G

Agent Network

Context-Dependence

Collaboration

Sensing &Control

Learning

• multi-objective hybrid strategies• learning, diagnostics and adaptation• real-time infrastructure

SKIP

CDNA: Progress by Subtask

Task 1: Agent Templates and Modules Subtask 1.1: CDN Agent Template

Implemented and studied structures for realizing various agent behaviors and capabilities for specific network control scenarios (CMU, RPI, TAMU, UM)

Subtask 1.2: Module SpecificationsCreated initial design of control agent modules separated from power system simulation program for real-time system emulation (RPI, UIUC)

Subtask 1.3: Interface Design Designed and implemented input-output interfaces between MATLAB power system simulator and real-time system emulator (RPI, UIUC)

Subtask 1.4: Tools for CDN Agent ConstructionEvaluated various algorithmic tools for constructing agent capabilities (CMU, RPI, TAMU, UM)

Task 2: Restructured Power System ModelingSubtask 2.1: Definition of Operating Modes

Examples of operating modes for TCSC control for voltage and stability transients (RPI, TAMU)Use of Markov decision models for defining operating modes (CMU)New method for evaluating safe transient-stability operating regimes (CMU)

Subtask 2.2: Online Identification of Operating ModesAnalysis of data from phasor measurement units (PMU) to identify signatures of disturbances (RPI)Use of neural nets and synchronized sampling for improved dependability/security (TAMU)

Subtask 2.3: Decomposition and AggregationConvergence results for new distributed load flow computations (UM)

Task 3: Agent Coordination and Learning Subtask 3.1: Development of Collaboration Strategies

New system structures for distributed model predictive control (CMU)Neighbor-coordination schemes in C-Nets (collaborative networks) (CMU)

Subtask 3.2: Learning Algorithms for CoordinationGame-theoretic formulations and learning for multi-agent control (CMU)Multi-objective coordination (TAMU)

Subtask 3.3: Local Control StrategiesMultimode control of Markov decision processes (CMU)Congestion control strategies for voltage and stability transients (RPI, TAMU)Impacts of deregulation market on reactive control (TAMU)

Subtask 3.4: Robust Hybrid DynamicsConditions for certainty equivalents in switching control strategies for Markov decision processes (CMU)

CDNA: Progress by Subtask (cont'd.)

Task 4: Real Time InfrastructureSubtask 4.1: Real-time Environment

Completed demonstration system (Telelab) for remote access to the Simplex infrastructure and MATLAB Power System Toolbox (RPI, UIUC)

Subtask 4.2: Robustness FeaturesImplemented Telelab for multiple users to download and test code (UIUC)

Task 5: Tests and DemonstrationsSubtask 5.1: Demonstration Scenarios

Multi-area scenarios for TCSC voltage and transient stability (RPI, TAMU)Scenarios for distributed multi-agent model predictive control (CMU)Multi-area load flow computation scenario (UM)

Subtask 5.2: Application SimulatorMATLAB Power System Toolbox-Simplex Telelab (RPI, UIUC)

Subtask 5.3: Visualization ToolsEvaluated power system simulation tools for protective relaying scenario presentation (MATLAB and EUROSTAG) (TAMU)

Task 6: The Virtual InstituteSubtask 6.1: Customization of Lire

Upgraded LIRE for faster access and e-mail notification services (CMU)Subtask 6.2: Computer-Based Collaboration

Web-based distribution of project reports and results (All participants)

CDNA: Progress by Subtask (cont'd.)

Overview of CDNA Accomplishments

Physical Network

G

G

G

Agent Network

- collaboration techniques- distributed computation

distributed control applications of synchronized sampling

multi-objectivehybrid strategies

real-time infrastructure

- learning- diagnostics- adaptation

Specific CDNA Accomplishments

B

C

A

distributed- rolling horizon strategies- power flow

protocols for dynamic

collaboration

- Markov modeling- multi-mode learning

context-dependentFACTS controls

context-dependentPSS controls

dependable & secureprotective relaying strategies

remote-access real-timecontrol emulator

evaluation of reactive power control in deregulated markets

CDNA Universities and Principal Investigators

Current CDNA Collaborations

1. U of Minn

3. TAMU

2. CMU

4. RPI

5. UIUCagents arch.

& collaboration

ABC systemswitching controlimplementation

Telelab-MATLABreal-time emulator

ABC systemvoltage studies

synchronous data modeling

economic marketsdecentralized comp.biological networks

distributedpower flow

stabilityregion comp.

Selected Results

Transmission System Security Analysis using Network Agents

• Security analysis is done by running power flows

• We are seeking methods of solving distributed power flows using agents (computer systems) in multiple control systems

• We would like to eliminate the idea of a “security center” approach.

University of Minnesota

ISO• Trends

– Getting larger

– Standard data formats

– Less functionality in regional systems

A

C

B

D

E

LARGE AREACONTROL SYSTEM

REGION ACONTROLSYSTEM

REGION CCONTROLSYSTEM

REGION BCONTROLSYSTEM REGION D

CONTROLSYSTEM

REGION ECONTROLSYSTEM

Networked Control Systems– Region can be any size– Can extend to any

number of regions– Regions retain original

functionality– Aggregate has same

functionality as large area control system

A

REGION ACONTROLSYSTEM

CREGION CCONTROLSYSTEM

B

REGION BCONTROLSYSTEM

DREGION DCONTROLSYSTEM

E

REGION ECONTROLSYSTEM

Poor Results from Multiple Processors solving One Power Flow

• Divide power system into several areas• Solve each area on a separate processor• Communicate results of each processor with

other processors• Communication time is greater than time

saved by using multiple processors• Try to minimize data that must be sent

between processors


Security Analysis and multiple processors

• Security analysis requires solving multiple power flows, one for each contingency case

• When calculation on one case is completed, start communication

• While communication is being done, start calculation on next case


PowerFlow

Area 3

PowerFlow

Area 1

PowerFlow

Area 2

Processor 1

Processor 2

Processor 3

Time

PowerFlow

Area 4

Processor 4

Multiple processors solving multiple outage cases – calculation overlaps communications

Greatly increased speed University of Minnesota

Methods Tested

• Gauss-Seidel Method

– Filtered Solution

– Block Border Gauss Method

• Conjugate Gradient Methods

• Reduced Orthogonal Subspaces

• Diakoptics


Results of research at the University of Minnesota

• Communication is the bottleneck• Methods with only neighbor to neighbor

communications require too many iterations to solve

• Methods that exchange ‘sensitivities’ require fewer iterations Some entity must calculate the sensitivities

• We have reduced the sensitivity data that must be exchanged to a minimum without sacrificing speed

University of Minnesotareturn

COORDINATION OF DISTRIBUTED, AUTONOMOUS AGENTS

Sarosh Talukdar, Eduardo Camponogara, Haoyu Zhou

ACCOMPLISHMENTS• Extension of Model Predictive Control (the Rolling Horizon Strategy) to serve as a coordination framework for autonomous, distributed agents.

• Development of a test-bed for coordination and learning strategies in networks of stationary and mobile, autonomous, distributed agents

Carnegie Mellon University

EXTENSION OF MODEL PREDICTIVE CONTROL TO AUTONOMOUS, DISTRIBUTED AGENTS

• The communication links between agents define a set of overlapping neighborhoods.

Neighbors of an agent = adjacent agents

• For each agent, the system’s variables are divided into three sets:X: proximate variables (those variables the agent can sense or control)Y: neighborhood variables (those variables the agent’s neighbors can sense or control)Z: remote variables (all other variables)


SUFFICIENT CONDITIONS (for the successfull extension of modelpredictive control to distributed, autonomous agents)

If: • the overall-system-problem is feasible• the overall-system-problem is convex• the overall-system-problem is decomposed into sub-problems for the agents, such that each sub-problem matches its agent exactly (Z is empty for each agent)• each agent uses an iterative, interior point method to solve its sub-problem• each agent communicates the results of each iteration to its neighbors• the agents in each neighborhood work serially (one after the other)

Then: the agents’ iterations will converge to an optimal solution of the overall-system-problem

Question: are these conditions necessary?


COORDINATION HEURISTICS

There are at least two families of heuristics by which the conditionson:

• exact matchings of agents to sub-problems• problem-convexity• communication frequency within each neighborhood• serial work within each neighborhood

can be demonstrated to be unnecessary for representative networks.

These families are based on:1. tightening the resource constraints by the inclusion of “resource margins”2. learning models by which each agent can predict the actions of its neighbors

These heuristics allow the agents to work asynchronously (in parallel, each at its own speed) on realistic (non-convex) control tasks.


Collaborative Nets

Dynamic Control Problem

(DP) Min f(x,dx/dt,u,t)

s.t. h(x,dx/dt,u,t)=0 g(x,dx/dt,u,t)0

Rolling Horizon Formulation

Series of Static OptimizationProblems <(P)>

(P) Min f(X,U)

s.t. H(X,U) = 0

G(X,U) 0(P)

(P1)

C-Net

(P3)(P2)

Ag1 Solution Approach to <(P)>

1) Break (P) into {(Pm)}

2) Match Agents to {(Pm)}

3) Assemble C-Net


Experiments

A Heuristic: Constraint Margins

When is the C-Net

Context-Dependent?

The agents modify the tasks, {(Pm)},

to make up for the varying context

C-Net Penalty (%)

Number of Pendulums

APE with mutual help

Pendulums are randomly

disturbed

Agents adjust {(m)}

Asynchronous prox . exch.

(Pm): Gm(Xm,Um,Ym)-m

1) Implement Constraint Margins

2) Collaboration Protocols:

3) Context Dependency

APE

APE M-Help


Context Dependent Switching and Learning

• Contexts: different buyers and sellers (decision-makers) with the same

• Objective: to develop bidding strategies for their own profits.

• So many uncertainties for a decision-maker, G1, for example,– Unobservable infinitely many possible combinations of bidding from

G2, L1, L2.

– Transmission line capacity variations.

G1

L1

Zone 1

G2

L2

Zone 2

Transmission Line

An application in the deregulated power market


• Switching: – Using finite number of modes to describe the

infinitely many possibilities.– Designing optimal strategy for each mode and

switching between these optimal strategies.

• Learning:– Performance measurement for the switching

among the current set of strategies.– When the performance is not satisfactory, a

new mode will be identified and corresponding optimal strategy will be designed.


Multi-Mode Markov Decision Process Model

• Markov System Xk, k = 0, 1, …, state space S.

• System Mode k, k = 0, 1, …, = {1, 2, …, ||}.

• Action set U and action subset U(s) U for each sS.

• A (stationary) policy L is a mapping from S to U such that L(s) U(s) for each sS.

• At epoch k, after an action u U(Xk) is applied,

– Transition to s with probability

– Reward incurred

– Mode jumps to k+1.

• Objective: find optimal policy sequence L0, L1, … to maximize performance

).,( sXp ku

k

).,( uXf kk

. ),( kkk LXfE

k


Switching Based on Certainty Equivalence (CE)

• Let L* be the optimal policy when k is a constant

.• Suppose k is a Markov process with transition matrix

Q.• CE Switching Strategy: Apply L

* when k=.• When is the CE strategy optimal?

|| I - Q || (1 - ) B /(2 A), A and B computable

• How well does CE switching do in general?||JCE - J*|| 2 A ||I - Q|| / (1 - )2

JCE: performance under CE switchingJ*: optimal performance.

. *

kLLk


• CE switching: Use the MLE of k

• When is CE switching optimal?2(1 - max p()) (1 - ) B /(2 A)

• How well does CE switching do?||JCE - J*|| 4A(1- maxp()) / (1 - )2

JCE: performance under CE switching

J*: optimal performance.

CE Strategy for Unobservable Modes


Simple Example

• Stationary policies for each mode:– If G2 always bids $14, $19 and $25, G1 bids $10, $15 and $20.

• Case 1: – G2 bids randomly with prob. 0.2, 0.2, 0.8

– G1’s optimal bidding strategy: Always bid $20 - CE strategy!

• Case 2:– G2 bids randomly with prob. 0.3, 0.3, 0.4

– G1’s optimal bidding strategy: Always bid $15 - not a CE strategy!

G1

LG2

G2 Capacity: 1000MWG2’s possible bids:$14, $19, $25/MW

G1 Capacity: 500MWG2’s possible bids:$10, $15, $20/MW

3 possible load demand levels:500MW, 800MW, 1200MWwith probabilities 0.3, 0.4, 0.3

Interest rate: 0.1% = 1/(1+0.1%)


return

Research at TAMU: Objectives

Survivability and Protection of the system for• Transient voltage, angle, oscillation, long term voltage

stability crises.• Overflow problems.• Protective relayingResponsibility evaluations of• Loop flow Problems.Market Efficiency for• Generation Dispatch Problems.All the problems are coupled, objectives are sometimesconflicting.

Texas A&M University

CDNA Interpretations• Detection agents to detect transient angle stability, voltage

stability, oscillation, long term voltage crises using acceleration angle velocity, line flows, voltage profiles, etc. (Security Margin Monitoring)

• Stabilizing agents, congestion control agents, auction agents, protection agents, performance control agents (to compromise different objectives.)

• Need to know contexts to switch among agents for survivability, protection and market efficiency.

• CDNA activates the needed control for best performance.


Accomplishments • Transient angle stabilizing controls using TCSC, SMES,

SVC, Braking resistors.• Stabilizing controls for transient voltage stability using

TCSC.• Generation dispatch using auction agents. • Flow Decompositions of bilateral trades for responsibility

evaluations.• Demonstrate interactions between market policies and

reactive power controls: Stable financial systems imposed on a stable engineering system may cause overall instability. Bad incentives and misconceptions.


SKIP

Work to be done in 2000

• Detection Agents for Transient voltage, angle, long term voltage stability crises.

• Security Margin Monitoring for long term and short voltage problems.

• Responsibility evaluations of loop flows using Flow Decompositions.

• Congestion controls using FACTs.• Demonstrate the use of Protection Relays as

Structural Controls to avoid cascading failures..


Some Highlights• Key misconception on reactive power controls are

identified.• We demonstrate using a simple BPA system why

these concepts are wrong.• Six questions are clarified using simulations.• Pricing based on these wrong concepts may lead

to system instability. • Financial incentives should be based on solid

engineering foundations.


Q1: Is Voltage Control Effect of Generators Local by nature? What are the impacts on reducing the Var reserve?

• No. It is system-wide. And reduced Var reserve will have system-wide impact.• Reduced reserve will also cause voltage transient stability, which collapsed in seconds.

Q2: How ULTC affects Voltage Stability?

• In many cases they harm the security.


• No, definitely not.

Q3: Can Intensive Use of Smart Shunt Banks at Load Areas Replace Dynamic VAR Reserves of Generators?

Q4: What is the impact of Real Loads on Voltage Stability?

• They have substantial impacts.


Q5: How do Load Characteristics Impact on Security Margin?

• They have substantial impact.

Q6: Will a stable financial system imposed on a stable engineering system destabilize

the whole system?

• Yes, definitely. Wrong incentives and misconception of reactive power system can destabilize the whole system.

• Interactions between financial system and engineering systems need to be investigated.


Conclusions• Not all VARs are created equal.

• Misconceptions on voltage stability are demonstrated.

• New findings will enable us to accurately evaluate reactive power provisions from generators and other devices in a deregulated power market, such as power pool market, bilateral trade market or compatible market.


Identified needs:

• Reduce dependency on setting inaccuracy

• Improve selectivity between permanent and temporary faults

• Improve security/dependability

• Introduce Coordination between Control and Protection

Protective Relaying


Protective Relaying

Defined New Protective Relaying Agents: • #1 Neural Net (NN) Algorithm for Fault

detection and classification• #2 Synchronized Sampling (SS) algorithm

for fault location• #3 Coordination Between NN and SS• #4 Coordination Between NN, SS and

Control


Protective Relaying

Developed Context Dependent Approach:

• Learning (training) for NN Agents

• Line Model (on-line parameter estimation) for Synchronized Sampling Agents


Protective Relaying

Introduced New Performance Benefits:

• Better relaying (dependability/security)

• Better reclosing (recognition of permanent vs temporary faults)

• Better control (preventing cascading outages)


Protective Relaying

Introduced New Evaluation Approach:

• Definition of future use of modeling and simulation tools (local and system events)

• Use of Matlab customized software for evaluation of individual protective relaying agents

• Use of Eurostag software for evaluation of system-wide interaction among agents


return

Real-Time System Emulation

• Based on inputs from UIUC, developed a preliminary power system simulator with external control from a remote computer

• Simulator is MATLAB based; communication protocol is SOCKET

• Demonstrated with the ABC system; external control switch between several control options

Rensselaer Polytechnic Institute

Power System Dynamic Monitoring

• Worked with ISO-NE, NYISO, and NYPA to obtain monitored power system disturbance transient data from about a dozen Dynamic Recording Devices (from several vendors)

• Developed a rule-based Event Identifier for classifying system disturbances; next step will be the development of an advanced identifier using detection filters

• In the process of using data obtained from several different monitors for the same event to analyze interarea oscillations


Control Design

• Proposed alternative controller structure using remote measurements as feedback signals

• Controller structure to handle communication delay

• Further development of linear matrix inequality techniques to control systems with parametric dependence


Scenarios

• Introduce two contingency scenarios for the ABC system, in addition to the normal operating condition, and design controls for the contingencies; will continue to develop additional scenarios for the system


return

Real Time Infrastructure• How to support the deployment of control agents in real

time reliably without shutting down the normal operations is an important concern.

• Telelab integrates WWW service with a fault tolerant dynamic real time architecture, the Simplex architecture. Telelab architecture gives you the ability – to add or replace application software components on the fly

without shutting down its operation. – to protect the system operation and the integrity of equipment

from bugs that could be introduced by changes.

University of Illinois at Urbana Champaign

Telelab: Remote Lab Interface

Win98/NT

* important* important* important* important* important* important* important* important

Win98/NT


Win98/NT


Win98/NT


LynxOS

Simplex

annotated, pre-recordedpresentation (e.g. HTML) (in case of communication failures)

CORBA A/V Streams

CORBAA/V Streams

Demo available at www-drii.cs.uiuc.edu


SKIP

Next Step: A Sample Power Network

GG

G

G

G

GLoad

loadload

load

load

Circuit Breaker


“Sympathetic” Relay Tripping: A Model Problem

– Background: Short circuit and temporary overload are very different. But they are treated as if they are the same problem due to the lack of coordination. Local response could lead to cascading failures that bring down a large portion of network.

– Coordination Context: • Triggering event: a relay open followed by neighboring relay open

• Event network: SS sample locate the fault, inform overload relays to hold and related nodes to do power rerouting/load shedding

• Data network: continue monitoring and report the overload situation on the relays in the holding mode

• Control network: agents change from normal control to overload management to bring the relays from holding mode to normal mode


“Sympathetic” Relay Tripping: Model Problem - cont’d

•For each chain of events there should be a coordinated response.

•Value of information: for each fault event there can be two solutions. The CDNA solution using information or throwing resources at it. This allows us to compute the resource equivalence of CDNA.

•Research on cascaded failures (network system instability): do we have parallels in Internet or other forms of network reactions, where a coordinated response could have prevented cascading failures. Will any ideas in Internet congestion control useful for power networks?


A Sample Power Network Failure

G3G4

G6

G5

G2

G1Load

loadload

load

load

Circuit Breaker

overload

• G1 or G2 could become unstable unless controllers are switched

• The open of the overload lines could propagate the failure to the entire region

• We need to stabilize G1 and G2 controllers and re-adjust G3,4,5, 6 and normalize the overloaded lines quickly


CDNA Simulation – Contingency management and agent based control testing

cases to implement agent and context management.

• Getting RPI’s sample system implemented.

• Getting CMU’s agent sample system implemented.

G1, 2 (MATLAB)

Agents for

G1, 2 (Simplex)

G3, 4,5, 6 Network

Agents for Load

Management Shared contexts

Telnet Net Back Door

TeleLab inteface

TeleLab interface

Agents

G3,4,5,6


return

context-dependent network agents specific technology goals of funded effort: distributed computation...

Documents

learning subtask

subtask task

um subtask

agent coordination

agent capabilities cmu

modules subtask

uiuc subtask

tcsc control