Carolyn M. Engstrom

Gain a new perspective on the problem of IT Data Analytics

Leave with inspiration and information about how to apply data analytics to achieve value

The cobbler is the IT department which uses his skills and tools to

make shoes.

Shoes are metrics, output, analysis, etc.

Shoeless children are internal processes.

IT doesn’t apply tools and skills to

meet it’s own goals

Audit and Compliance are child protective

services. “Your children have no shoes!!”The broader

organization helps design them and

uses them.

Big Data centric Metrics focused Necessary evil of compliance Effectiveness dominates Efficiency lags Structured, centralized data Enterprise solutions Security Event and Incident Management Analytics are afterthoughts of implementation

Data quality worries Data efficiency worries Need to predict, forecast Historical reporting Siloed knowledge of business process “Gartner Says Power Shift in Business

Intelligence and Analytics Will Fuel Disruption”

CIO: 21 Data and analytics trends that will dominate 2016

• Statistical• Predictive Models

• Really big data!• Lots of sources!• Really important

issues to solve!

• End-user focused• Reporting• Summarize• Drill Down

• Outside Data Sources

• Unstructured Data• Extract, Transform,

Load

Source: “What Kind of Big Data Problem Do You Have?” SAS, 2014

Reac

tive

Proa

ctiv

e

Large Big Data

Dat

a Ca

pabi

lity

Data Size

Big Analytics Big Data Analytics

Business Intelligence

Big Data Business

Intelligence

They come in many sizes Big: $$$◦ Aggregated from External Sources◦ Primarily Big Data Business Intelligence

Medium: $$-$$$◦ Aggregated from Internal and External Sources◦ Operational and Security Information

Small: $◦ Internal Accumulation◦ Risk Assessment◦ Context, Calibration, Criticality

“Actionable Security Intelligence From Big, Midsize and Small Data “ by C. Warren Axelrod, Ph.D., CISM, CISSP – ISACA Journal, 2016

Achieve Insight Uncover Meaning Improve Assurance/Effectiveness Improve Efficiency Identify Trends Demonstrate Progress Prototype Requirements Improve Data Integrity Unlock Knowledge Management

“A Practical Approach to Data Analytics”, ISACA, 2011

Black box auditing◦ Frameworks◦ Methodology◦ Audit procedures◦ Standards of fieldwork

Evolving data analytics skillset Reports lack a persuasive story, meaning or

context Unique exposure to data, processes, and

risks

Source: CEB Audit Leadership: Peer Feedback- Data Analytics Vendors 2014

Define a population◦ Controls or risks

Information Provided by Client (IPE)◦ Population integrity

Non-statistical Sampling: based on frequency◦ Annual, Semi-annual, Quarterly, Automated = 1◦ Monthly = 2◦ Weekly = 5◦ Daily = 15◦ Many times daily = 25

Statistical Sampling◦ Confidence Intervals, 90% or 95%◦ Mathematical function identifies sample size◦ Not frequently used

100% population analysis◦ 1 source- IPE◦ 2 sources or more- Data Integrity◦ Removes population bias◦ Provides quantifiable measure of effectiveness

Assessment of exceptions All of these techniques support an auditor’s

conclusion

Source: CEB Audit Leadership: Peer Feedback- Data Analytics Vendors 2014

Expand your perspective on data◦ Transaction◦ Trending◦ Continuous Monitoring

If data is valuable, for the love of goodness, DO NOT USE A WORD DOCUMENT as a source of truth… EVER.

Control: For SOX in-scope apps job completion is monitored and abends are recorded in ticket software and resolved.

Batch process extracts job

fails from log

Employee selects a sample of

25

Employee searches for ticket

Employee records results in a Word doc. Embeds job

log object(s)

Monthly Quarterly 25 manual times 4Q x n apps

Audit used logs to

select their own

sample

Testing covers only failures◦ How many jobs ran successfully?

Only applied to SOX applications Manual process◦ Required about 2-3 hours per quarter per app◦ Multiple control owners

Audit coverage was minimal % of population Files maintained all over the network

Batch process extracts job completions

and fails from log

Monthly

Use data prep software to format logs

User extracts tickets

Compares

Exceptions: not timely, no ticket

Sends Exceptions

Report sent to control owner

Redesign cost about $2000 for data prep Time investment of about 40 hours Quantitative assurance◦ 100% SOX population coverage◦ 100% exception coverage

Context of success, failures, exceptions (%) Correct data quality issues Centralize file storage Increase frequency to monthly from quarterly

but decrease time

Build a table for jobs and attributes◦ Interfaces Data flow of confidential data Data flow of financial data◦ Report Integrity◦ Job number◦ Criticality

Build knowledge management Use data visualization rather than reports

Narratives about 6-20 pages long Topics◦ Access Controls◦ Change Management◦ Interfaces ◦ Job Resolution◦ Infrastructure Identification (asked to update xls) App servers Database servers and instances Servers (OS, location)

Identified Business Processes, but not financial statement accounts or disclosures

Narrative of an actual process Identify financial statement accounts and

disclosures Identify key controls May identify key reports by name Identify information on interfaces

Productionize Application Narrative◦ Change management application attributes◦ Created report out of the application◦ Improved population for change management

controls Foster Audit Knowledge Management ◦ Key Reports◦ Interface information to Chart of Accounts◦ Financial Statement Line Items◦ Custom Report for Review

Create relationships among data that was previously locked

Transform unstructured data Enforce consistency Content is more accessible Less data to maintain Improve efficiency and effectiveness of

existing tools

No previous defined vulnerability management process

Select a large-scale tool for vulnerability identification

Delays in projects due to incomplete network topography

Use Nessus to scan sample of servers (20) Collect data to baseline scores Use scripts to collect ◦ Patch levels from servers ◦ Event log entries◦ Registry settings◦ Customized reporting

Use data to clarify business requirements◦ Roles◦ Communication requirements◦ Documentation

More quantifiable data than initial business case

Established expected baselines Resourcing and timelines Calculated revised Return on Investment Defined a process Verified business requirements

1. Map regulatory/oversight requirements to internal controls

2. Inventory and leverage existing data sources 3. Use existing, free, or low cost tools4. Analyze Baseline◦ Data Flow◦ Data Integrity◦ Return on Investment

5. Re-baseline and productionize (governance)◦ Automation◦ Workflow

Don’t overlook unstructured data Unlock your small data◦ Gather and update effectively◦ Focus on context and criticality

Audit can be great sources of small data, but know the audit approach

Leverage the same data sources for different risks and insights

Data-Driven Security: Analysis, Visualization and Dashboards by Jay Jacobs and Bob Rudis (book)

Threat Modeling: Designing for security by Adam Shostack (book)

Database Debunkings Fabian Pascal (blog) Dresner Advisory Services 2016 End User

Preparation Market Study (Market Research) Storytelling with Data by Cole Nussbaumer

Knaflic (book and blog)