Protecting Information

Protecting Information

• RT6 focuses on methods to protect against indirect vulnerabilities caused by adversarial observation of computation, communication, and mission execution

• Major initiatives this year:• Protecting information through

examination of entropy and Markov decision processes• Differential privacy• Systems approaches to privacy,

access control, and machine learning

system-theoretic principles (e.g., barrier certificates) for verifying privacy

Noise-adding mechanisms to tradeoff individual privacy and aggregate performance

1

3

2 4

5

6

x1 +

w1

x 2+w 2

x3 + w3

x2 + w2

x6 +

w6

wi ⇠ N (0,�2i )

�i ��`2G2✏

�K� +

pK2� + 2✏

�

Mission-Critical Information

• Entropy maximization in Markov decision processes subject to temporal logic constraints• Synthesize constrained, entropy-

maximizing strategies• The higher the entropy, the less

predictable• Convex-optimization-based synthesis

• Entropy maximization in partially observable Markov decision processes• Extension to decision making with

limited information at runtime

Policies and Information Leakage

• Least inferable policies in partially-observable Markov decision• Accounts for both the

amount and informativeness of the adversary’s observation

• Minimizing information leakage regarding high-level task specification

Differential Privacy

• Dirchlect mechanism for differential privacy on the user simplex• Provides differential privacy to Markov decision process

properties• Ensures recipient of data cannot learn anything

meaningful from privatized simplex data• Error bounds and guidelines for privacy calibration

in differentially-private Kalman Filtering• First control-theoretic guidelines for calibrating

differential privacy• Differentially private controller synthesis with metric

temporal logic specifications• Multi-agent control policies with differential privacy

Systems and Access Control

• Privacy-preserving secure localization• Extension of privacy-preserving local optimization to

embedded device processors• Framework for secure machine learning• Interpretable security reference monitor

design with applications to autonomous agents

• Access control policy• Framework for reasoning

about disparate access policies in embedded devices

Access Control Policy Analysis for

Embedded Devices

with Grant Hernandez, Dave Tian, Anurag Yadav, and Byron Williams

Embedded Devices

• Embedded agents and the operating systems that they run are subject to numerous methods for protecting on-device information• Example: Android OS – the world’s most popular

operating system (mobile and embedded devices)• Numerous ways of protecting access to on-device

critical information• Discretionary access controls • Mandatory access controls• Linux capabilities• Middleware protections/SECCOMP

• Many attack surfaces

Reasoning about Access Control

• Challenge: there is no way to currently reason about the myriad access control mechanisms• Consequence: access policies from different

mechanisms not in concert with each other, may be mutually conflicting• What objects and processes can be accessible to

untrusted processes?

BigMAC Design

• Examine, recover, simulate, fully instantiate files, process, IPC from an extracted firmware image• Simulate boot

process to recovertypes instantiatedat runtime• Process hierarchy

and processmetadata

0 1 0 0 0 0 1 1 0 1 1 1 0 0 0 0 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 1 0 0

1 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 1 0 1 0 1 0 1 0 0 1 1 1 1 0 1 1 1 0 1 1

BigMAC

Instantiated Policy GraphQueryInput

SEPolicy Parsing

Policy Instantiation

Filesystem ExtractionDAC/CAP/Labels

Type Relations

Processes/Objects

File / Object IPC ObjectSubject / Process

FirmwareFilesystems

Real-System (Ideal)Processes/Files

Attack Graph Ground Truth Comparison

Runtime

ExtractionCtx.Ctx.

1 1

Process TreeCredentials

File DAC/MAC

Subject

DataflowFlat Dataflow

Process Tree

Fully InstantiatedAttack Graph

Backing Files Subjects

Objects

Processes

Instantiate

Extract

Parse Tag/Link

Recover Instantiate

SEPolicy

F S

Gd Gf

OverlayGs Gp

O

P

Ga

Policy Recovery

• Decompile binary SEPolicy into connected multi-edge directed graph using Access Vector rules• Subject graph includes all types and attributes used during

SELinux MAC type enforcement• Instantiate on filesystem through type enforcement rules

• Build dataflow graph from subject graph to examine paths and edges where privilege escalation may occur• Bipartite graph, worst case edges,

nodes

Cp : Si(Oj ,Cp)�����! Sj

AAACCnicbVDLTsJAFJ3iC8FH1aWbKppgYkhrNBpXRDbuxCCPBJpmOgwwMO00M1OUNKzd+CtuXGCMS/0Cd36Irh0eCwVPcpOTc+7Nvfe4ASVCmuanFpubX1hcii8nkiura+v6xmZJsJAjXESMMl5xocCU+LgoiaS4EnAMPZfistvJDf1yF3NBmH8jewG2Pdj0SYMgKJXk6Ds5JzgvOKR2x0mzJSHn7DZKXzntQ2Uc9AtO29FTZsYcwZgl1oSksntfg7du8jvv6B+1OkOhh32JKBSiapmBtCPIJUEU9xO1UOAAog5s4qqiPvSwsKPRK31jXyl1o8G4Kl8aI/X3RAQ9IXqeqzo9KFti2huK/3nVUDbO7Ij4QSixj8aLGiE1JDOGuRh1wjGStKcIRJyoWw3UghwiqdJLqBCs6ZdnSekoYx1nTq5VGhdgjDjYBrsgDSxwCrLgEuRBESBwDx7BADxrD9qT9qK9jltj2mRmC/yB9v4DZieegQ==

O(|S| ⇤ |O|)

AAACE3icbVC7TgMxEPTxJrwClDQWCImHFN0hEJQRNDQoIAggJVHkc3xgYftO9h7i5Nw/UMCv0FCAEC0NHX+DL6HgNdVoZlc7O2EiuAHf//AGBoeGR0bHxksTk1PTM+XZuVMTp5qyOo1FrM9DYpjgitWBg2DniWZEhoKdhVd7hX92zbThsTqBLGEtSS4Ujzgl4KR2ea0J7AZsMzUsihXY2sFxbqk0WW5lblVey1e6x921bq272i4v+RW/B/yXBF9kqbrYXL/7qGaH7fJ7sxPTVDIFVBBjGoGfQMsSDZwKlpeKowmhV+SCNRxVRDLTsr2fcrzslA6OYo2LXLinft+wRBqTydBNSgKX5rdXiP95jRSinZblKkmBKdo/FKUCQ4yLgnCHa0ZBZI4QqrnLiukl0YSCq7HkSgh+v/yXnG5Ugs3K1pFrYxf1MYYW0CJaQQHaRlW0jw5RHVF0ix7QE3r27r1H78V77Y8OeF878+gHvLdPQpKinQ==

O(|S|+ |O|)

AAACE3icbVC7TgMxEPTxJrwClDQWCImHFN0hEJQRNDQoIAggJVHkc3xgYftO9h7i5Nw/UMCv0FCAEC0NHX+DL6HgNdVoZlc7O2EiuAHf//AGBoeGR0bHxksTk1PTM+XZuVMTp5qyOo1FrM9DYpjgitWBg2DniWZEhoKdhVd7hX92zbThsTqBLGEtSS4Ujzgl4KR2ea0J7AZsMzUsihXY2sFxbqk0WW5lblVey1e6x931bq272i4v+RW/B/yXBF9kqbrYXL/7qGaH7fJ7sxPTVDIFVBBjGoGfQMsSDZwKlpeKowmhV+SCNRxVRDLTsr2fcrzslA6OYo2LXLinft+wRBqTydBNSgKX5rdXiP95jRSinZblKkmBKdo/FKUCQ4yLgnCHa0ZBZI4QqrnLiukl0YSCq7HkSgh+v/yXnG5Ugs3K1pFrYxf1MYYW0CJaQQHaRlW0jw5RHVF0ix7QE3r27r1H78V77Y8OeF878+gHvLdPRBqing==

Evaluation/Case Study

• Over 98% of DAC and MAC data recovered• Approx. 75% of processes (mostly missing app level)

• Prolog-based query engine filterMAC,DAC, CAP, ext. surfaces• Found additional attack paths for

existing CVE

Recovered by BigMAC Ground truth, running device

process:zygote

process:crash dump process:vold

crash dump exec:transition

*:write

vold:ptrace

A Learning Mechanism for

Learning Systems

with Washington Garcia, Scott Clouse (AFRL/ACT3)

Motivation

• Secure solutions to learning problems requires cooperation between many research disciplines.

Motivation

• Secure solutions to learning problems requires cooperation between many research disciplines.

• However, learning systems research tends to be conducted in an ad-hoc manner:• Threat modeling has only recently received attention from ML researchers. • If the threat model is known, choosing among seemingly competing

frameworks can be difficult.

Empirical risk minimization

Differentially Private Models

Certifiable Models

Adversarial Training

?

Motivation

• Secure solutions to learning problems requires cooperation between many research disciplines.

• However, learning systems research tends to be conducted in an ad-hoc manner:• Threat modeling has only recently received attention from ML researchers. • If the threat model is known, choosing among seemingly competing

frameworks can be difficult.• Lack of clarity mitigates cross-pollination between each research discipline,

causes external observers to question what progress has been made.

Motivation

• Secure solutions to learning problems requires cooperation between many research disciplines.

• However, learning systems research tends to be conducted in an ad-hoc manner:• Threat modeling has only recently received attention from ML researchers. • If the threat model is known, choosing among seemingly competing

frameworks can be difficult.• Lack of clarity mitigates cross-pollination between each research discipline,

causes external observers to question what progress has been made.

Interpretable MLVerifiable MLPrivacy ML Adversarial MLML

Potential Output:Unified proposals

Motivation

• Secure solutions to learning problems requires cooperation between many research disciplines.

• However, learning systems research tends to be conducted in an ad-hoc manner:• Threat modeling has only recently received attention from ML researchers. • If the threat model is known, choosing among seemingly competing

frameworks can be difficult.• Lack of clarity mitigates cross-pollination between each research discipline,

causes external observers to question what progress has been made.

Interpretable MLVerifiable MLPrivacy ML Adversarial MLML

Actual Output:Ad-hoc, Repeated work

External Observers

?

Motivation

Florida institute for Cybersecurity Research 24

Takeaway:• Lots of seemingly unrelated research disciplines

working on related problems• Unifying them is not immediately obvious• Has this problem been tackled before?

Anderson research model

Florida institute for Cybersecurity Research 27

Has this problem been tackled before?

In 1993, Anderson described cryptographic systems which faced similar issues:• Unclear threat models:

• what could happen vs. what was likely to happen.• Little communication due to centralized certification authority

• crypto systems mainly cared about passing certification

Unifying research in ML with systems security: • Adopt Anderson’s research model to design secure learning systems• Organize around system security strategies which manifest within ML

research

Proposed Research Model

Florida institute for Cybersecurity Research 28

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 7

Theoretical Results

(ε, δ)-DP Bounds [95]

Geometry [116]

Decoupling [86]

Empirical Tests

Complete Verification

Incomplete Verification

Ad-Hoc Methods [3]

Analysis

Learning Primitives

Privacy ML

Gradient Shaping

Poison Detection

Interpretable ML

Global Interpretations

Local Interpretations

AML-D & Verifiable ML Machine Learning

Verification

Heuristic Optimization

Space Properties

Polytope Bounding

DP-Smoothing

End-to-End ML Tooling

End-to-End Learning

Emergent Language

Bayesian Modeling

Filtering

Adversarial Detection

System Security Strategies

Protection System [98]

Enforcement

Security Goals

Protection States

Reductionist Kernelization [64]

End-to-End Argument [112]

Root of Trust [96]

Functional Restrictions [8] Sense of Self [100]

Anderson Learning Mechanism [64]

Problem Setting

Correctness

Threat Model

+

Figure 2. Interaction of Anderson’s learning mechanism as it is instantiated through system security strategies. The system security strategies are implicitlymanifested within many works related to ML and AML. Anderson’s learning mechanism informs the constant feedback loop of security strategies, newconstructions, and analysis.

application, image recognition. The Train function outputs alinear classifier which is assumed to be well regularized, andthus originally satisfies the author’s notion of correctness. Inorder to correctly attack the integrity of the classifier C, Tanayet al. establish two properties with respect to the correctness ofC: 1) the performance on natural images remains unaffected,but 2) vulnerability to adversarial examples is increased. Thisis done in practice by “tilting” the decision boundary of C byan arbitrary tilting factor k 2 R. Tanay et al. extend this furtherto fully connected layers in an MLP by tilting in the lowestd 2 R principal components of movement in the layer’s featuremapping. The model is essentially forced to attend to the ddirections which align closest with its weights, even if otherdirections have higher magnitude. The alteration is designedby the adversary to be imperceptible by manual verification.The work by Tanay et al. is similar to other poisoning attacksreviewed by Biggio and Roli [2] and Papernot et al. [9]. The

threat model allows the adversary to either control the model’salgorithms (PreProcess, Train, Predict), or the data which isfed to Train for instantiating the model. We see that as inThompson’s example, the vulnerability is patched in by a re-training scheme.

Gradient Shaping & Poison Detection. Approaches totrust for learning systems rely on either detecting a poisoningattack, or bounding the effect of individual samples. Forexample, Ma et al. propose the use of differential privacyprimitives as a defense from synthetic data poisoning attacksfor simple Logistic Regression models [41]. Under the DP-learning framework, the model is trained such that no partic-ular sample influences the model beyond a particular clippingbound. Thus lineage of trust is effectively ignored in favorof simply distrusting all future samples and limiting theirinfluence by a provable bound. Hong et al. later showed that

Instantiation of model

Florida institute for Cybersecurity Research 29

Example using:“Brittle Features of Device Authentication”

Establish likely problem setting (device fingerprints)

Conditions for correctness (true positives/negatives)

Plausible threat model (hard label adversary)

The system captures:

• Forrest et al. sense of self: encode a probabilistic definition of normal and abnormal.

The system uses sense of self to:

• Protect the integrity of network resources, by offering physical authentication channel.

• Example: Encode the sense of self using machine learning models.

Analysis:

• Ad-hoc empirical test.

• Example: Information gain, fingerprinting accuracyModify adversary

(originally none for FP systems)

Re-analyze

Pattern Recognition

Instantiation of model

Florida institute for Cybersecurity Research 37

Example using:“Brittle Features of Device Authentication”

Modify adversary

(originally none for FP systems) ZOO

Zeroth Order Optimization framework

Hard label updates based on gradient approximation

Re-analyze

Instantiation of model

Florida institute for Cybersecurity Research 39

Example using:“Brittle Features of Device Authentication”

Modify adversary

(originally none for FP systems) ZOO

Re-analyze

GTID

WDTF

USB-F

High Fals

e

Positives

Instantiation of model

Florida institute for Cybersecurity Research 42

Example using:“Brittle Features of Device Authentication”

Modify adversary

(ZOO)

Re-analyze

Pattern Recognition

DP-based Randomized Smoothing Defense (Cohen et al.)

Modifies the system’s sense of self

Instantiation of model

Florida institute for Cybersecurity Research 43

Example using:“Brittle Features of Device Authentication”

Modify adversary

(ZOO)

Re-analyze

Pattern Recognition

DP-based Randomized Smoothing Defense (Cohen et al.)

Modifies the system’s sense of self

GTID

Instantiation of model

Florida institute for Cybersecurity Research 50

Example using:”Jack” Audi automated driving system

©BT Devices

Establish likely problem setting (sensor inputs)

Conditions for correctness (crash avoidance)

Plausible threat model (in this case, innocent)

The system captures:

• Saltzer et al. end-to-end agreements (i.e., measures of agreement between inputs and outputs)

The system uses end-to-end argument to:

• Offer robustness, through Bayesian modeling: probabilistic measure of uncertainty between inputs and outputs.

• Example: Jack can perform probabilistic reasoning from perception sensor uncertainties

Analysis:

• Ad-hoc empirical test.

• Example: Drive a 550-mile automated test drive.

Modify adversary

(originally none for Jack)

Re-analyze

Instantiation of model

Florida institute for Cybersecurity Research 55

High-level feedback loop helps visualize gaps in constructions or analysis.

Analysis:

Does the analysis capture every possible failure mode in the current threat model?

©BT Devices

Constructions:

Does the existing model of uncertainty completely capture each end of the system?

Structure informs of dependencies

If we change our threat model, we must re-visit our security strategies, primitives, and analysis.

Example: Analysis of Jack in the innocent setting says little about the adversarial setting

Next Steps

• Consider framework in the context of human-machine cooperation systems

• Verification of reinforcement learning agent policies

• Apply and implement on autonomous agents

Florida institute for Cybersecurity Research 56