Console OperationsConsole Operations(Service Desk)(Service Desk)

Console Operators are Console Operators are tasked with a wide variety of tasked with a wide variety of functions and functions and responsibilitiesresponsibilities• We are the first point of contact for We are the first point of contact for

– Qualified Scheduling EntitiesQualified Scheduling Entities– Transmission Distribution Service Transmission Distribution Service

ProvidersProviders– Market ParticipantsMarket Participants– General PublicGeneral Public– ERCOT Employees.ERCOT Employees.

• We monitor the collective health of We monitor the collective health of approximately 350 - 400 Production approximately 350 - 400 Production servers and all test and development servers and all test and development servers.servers.

• We monitor the following thresholds:We monitor the following thresholds: – Hardware thresholds - Disk, CPU, Memory Hardware thresholds - Disk, CPU, Memory

utilization and capacityutilization and capacity– Application thresholds – Deployment failure, Tibco, Application thresholds – Deployment failure, Tibco,

Paperfree transaction failure (determined by the Paperfree transaction failure (determined by the application administrators)application administrators)

– Database – internal errorsDatabase – internal errors– Network infrastructure – ports, routers, switches, Network infrastructure – ports, routers, switches,

firewall, any type of problem with network devicesfirewall, any type of problem with network devices– Telecom – Point to points and Frame Relays to Telecom – Point to points and Frame Relays to

QSE’s QSE’s

• The Node Map is one of the tools The Node Map is one of the tools used for monitoring the servers.used for monitoring the servers.

• The Alarm browser is a tool to The Alarm browser is a tool to monitor hardware, software, and monitor hardware, software, and threshold alarms.threshold alarms.

• We monitor batch processes and We monitor batch processes and automated jobs automated jobs – Commercial ApplicationsCommercial Applications

• Registration, Data Acquisition/AggregationRegistration, Data Acquisition/Aggregation• Load ProfilingLoad Profiling• Settlements and Billing SystemsSettlements and Billing Systems• Collectively about 400 - 500 jobs and a few Collectively about 400 - 500 jobs and a few

more jobs that run once a week or once a more jobs that run once a week or once a month, etc.month, etc.

– EISEIS• Data WarehouseData Warehouse• Enterprise Data WarehouseEnterprise Data Warehouse• Data Archive and other reporting and Data Archive and other reporting and

miscellaneous support functions - 180 jobs / miscellaneous support functions - 180 jobs / 18 schedules18 schedules

• Security Monitoring Security Monitoring – Login attemptsLogin attempts

• failed logins and account lock outsfailed logins and account lock outs

– Virus trapsVirus traps• port shutdown subsequent to virus detectionport shutdown subsequent to virus detection

– Port violationsPort violations• a user or computer tries to use a port not a user or computer tries to use a port not

designated for that machinedesignated for that machine

– Unauthorized wireless access attemptsUnauthorized wireless access attempts

• Dedicated EMMSDedicated EMMS monitoring monitoring – The health and heartbeat of production The health and heartbeat of production

EMS servers responsible for Energy EMS servers responsible for Energy Management and Market SystemManagement and Market System

– All the Grid Operator terminalsAll the Grid Operator terminals

• Provide level 1 support for the Provide level 1 support for the above functionsabove functions– Per specific alarm, the Operator begins Per specific alarm, the Operator begins

remedial action to resolve the problemremedial action to resolve the problem– Notify and escalate the issue to level 2 Notify and escalate the issue to level 2

support personnel when applicablesupport personnel when applicable– Establish conference bridge with the Establish conference bridge with the

Subject Matter Expert and managers to Subject Matter Expert and managers to facilitate the resolution of the issuefacilitate the resolution of the issue

• Reporting and Miscellaneous Reporting and Miscellaneous – Daily verifications of Settlement Statements Daily verifications of Settlement Statements

and Invoices – send verification report to and Invoices – send verification report to business business

– Provide daily ticket report, outstanding Provide daily ticket report, outstanding ticket report and monthly report to ticket report and monthly report to managersmanagers

– Development FOB access reportDevelopment FOB access report– Send internal IT system notificationsSend internal IT system notifications– Direct incoming calls - AP/AR, business Direct incoming calls - AP/AR, business

solicitation calls, informational, power solicitation calls, informational, power outages, switching questions, concerns and outages, switching questions, concerns and complaints, media and market participant complaints, media and market participant callscalls

All Service Desk functions - Console All Service Desk functions - Console Operator's collect information Operator's collect information needed for problem resolution and needed for problem resolution and input into the tracking system while input into the tracking system while managing the issue until completion managing the issue until completion or escalation. or escalation.

A monthly average of appx. 2100 A monthly average of appx. 2100 calls and 2300 tickets are processed calls and 2300 tickets are processed by the Console Operations by the Console Operations department.department.

Lifecycle of an Issue from Console Operations Perspective

The Operator is notified of an issue by phone, e-mail, or monitoring system

A Problem Ticket is now generated by the Operator

Begin Begin remedial remedial action to action to

resolve the resolve the problemproblem

Level 1 Support is initiated

Begin Begin remedial remedial action to action to

resolve the resolve the problemproblem

Notify and Notify and escalate escalate

the issue to the issue to level 2 level 2 support support

personnel personnel when when

applicableapplicable

Level 2 Support is contacted

Begin Begin remediremedi

al al action action

to to resolve resolve

the the probleproble

mm

Notify Notify and and

escalate escalate the the

issue to issue to level 2 level 2 support support personnpersonnel when el when applicabapplicab

lele

Establish conference bridge

with support

personnel as

requested

Level 1 and Level 2 support utilized

Begin Begin remediremedi

al al action action

to to resolve resolve

the the probleproble

mm

Notify Notify and and

escalate escalate the issue the issue to level to level

2 2 support support personnpersonnel when el when applicabapplicab

lele

Establish Establish conferenconference bridge ce bridge

with with support support

personnepersonnel when l when neededneeded

Resolution is Resolution is defined as defined as

service being service being restoredrestored

Resolution

Begin Begin remediremedi

al al action action

to to resolve resolve

the the probleproble

mm

Notify Notify and and

escalate escalate the issue the issue to level to level

2 2 support support personnpersonnel when el when applicabapplicab

lele

Establish Establish conferenconference bridge ce bridge

with with support support

personnepersonnel when l when neededneeded

Resolution is Resolution is defined as defined as

service being service being restoredrestored

Ticket Ticket is is

closed closed and and

user is user is notifienotifie

dd

Closure