2020 las collaborators’ week...vulnerability detection symbolic execution without source code...
TRANSCRIPT
1
2020 LAS Collaborators’ Week
Dr. Alyson WilsonLAS Principal Investigator
Dr. Matt SchmidtLAS Director of Programs
Jamie RoseboroughLAS Director of Outreach
and Engagement
Dr. Christine BrughLAS Technical Program Manager
Dr. Jascha SwisherLAS Technical Program Manager
Lori WachterLAS Technical Program Manager
June 15-18, 2020
Contact Info● General Inquiries: [email protected]
● Specific Inquiries:○ Alyson Wilson, [email protected]○ Matt Schmidt, [email protected]○ Jamie Roseborough, [email protected]
● LAS Collaborators Week Website: https://ncsu-las.org/2020-las-collaborators-day/
2
3
LAS Collaborators Week Schedule● Monday, June 15: Plenary Session
● Tuesday, June 16: “How to Work with LAS” Sessions
● Wednesday, June 17: Technical “Office Hour” Sessions○ Analytic Rigor and Performance○ Data Triage○ Influence Campaigns
● Thursday, June 18: Technical “Office Hour” Sessions○ Machine Learning Integrity○ Human Machine Collaboration○ Selected Cybersecurity Challenges○ Additional Use Cases
Plenary SessionMonday, June 15
● Overview of LAS and how we work
● Overview of 2021 LAS interest areas
● Overview of the white paper submission process
4
“How to Work with LAS” SessionsTuesday, June 16
● Purpose○ Answer questions about logistics of working with LAS○ Provide general suggestions about how your research interests might align
with the different LAS interest areas.
● Individual sessions conducted via Zoom○ Sign up for 10-minute time slot at: https://ncsu-las.org/2020-las-collaborators-day/○ Two available blocks of time slots
■ 09:00a – 11:00a (EDT)■ 01:00p – 03:00p (EDT)
5
Technical “Office Hour” SessionsWednesday, June 17 & Thursday, June 18
● Purpose○ Provide an opportunity to speak with LAS staff who have related interests
about potential project ideas and collaborations.
● Individual sessions conducted via Zoom○ Sign up for 10-minute time slot at: https://ncsu-las.org/2020-las-collaborators-day/○ Wednesday, June 17
■ 09:00a – 11:30a (EDT) : Analytic Rigor and Performance (CFWP Section 3.1)■ 12:00p – 02:30p (EDT) : Influence Campaigns (CFWP Section 4.1)■ 02:30p – 05:00p (EDT) : Data Triage (CFWP Section 3.4)
○ Thursday, June 18■ 09:00a – 11:30a (EDT) : Machine Learning Integrity (CFWP Section 3.2)■ 12:00p – 02:30p (EDT) : Selected Cyber Security Challenges (CFWP Section 4.2)■ 12:00p – 02:30p (EDT) : Additional Use Cases (CFWP Section 4.3)■ 02:30p – 05:00p (EDT) : Human Machine Collaboration (CFWP Section 3.3) 6
Questions● If you would like to ask a question please use the Q&A feature in
Zoom
● We have multiple places in the talk where we will pause to answer questions from the Q&A
● If you are unable to ask your question through the Q&A feature today, please email [email protected] with your question, and we will get back to you.
7
What is the Laboratory for Analytic Sciences?LAS is a mission-oriented academic-industry-government research collaboration that works at the intersection of technology and tradecraft.
8https://ncsu-las.org/
9
Advance the tradecraft of intelligence analysis while leveraging novel and recent advances in research and technology
● Investigate technical approaches with the potential to address analysis challenges
● Develop analytic tradecraft that leverages research and technology to address mission needs
● Transition technology and tradecraft to partners who can operationalize and scale solutions
10
How do we work at LAS?● Mission-relevant projects
● ~90% of our work is unclassified
● Integrated, team-based approach
● Guidance is intentionally open-ended, as we are expecting you to help shape the direction of the projects
● 24 faculty (and ≈ 35 students) at 9 unique universities
● 7 industry partners and 1 national lab
● ≈ 50 government staff/IC partners
● 14 NCSU staff
11
Who is participating with LAS in 2020?
12
What are we looking for in 2021 Collaborators?
● Immersive○ Iterative approaches to solutions○ Opportunistic approaches to solutions
● Interdisciplinary○ Researchers, developers, and practitioners○ STEM, humanities, and social sciences
● Relevant Expertise○ Relevant to their own activities○ Potentially relevant to other activities
13
What are we looking for in 2021 Projects?● Impact
Will a successful outcome have a positive impact for intelligence analysts?
● InnovationIs a new approach proposed, or does it utilize new capabilities?
● EngagementAre LAS stakeholders interested in collaborating on the project?
● New Understanding○ Experimental data○ Research Papers
● New Tradecraft○ Storyboards ○ Documented Workflows
● New Capabilities○ Proofs-of-concept (e.g. Jupyter Notebooks)○ Prototypes
14
What are we looking for in 2021 Outcomes?
Questions● If you would like to ask a question please use the Q&A feature in
Zoom
15
16
Human-Machine Collaboration
Machine Learning Integrity
Influence, Cybersecurity, and Other Use Cases
Analytic Rigor and Performance
Triage
What are our areas of interest for 2021?
17
Analytic Rigor and Performance
Analytic production and
journalism
Evaluating rigor in analytic workflows
Augmenting analytic
performance
Applying rigor to language analysis
Defining and Evaluating rigor and its components
Identifying the “Fundamental Five” of analyst performance
from Amershi et al (2019)
Machine Learning Integrity
18
ML in ProductionDefine and support best practices for machine learning operations
People and MLImprove interactions between humans and algorithms
End UsersEncourage appropriate trust in automated predictions
ML human factors
Data ScientistsAccelerate development of reliable models
ML explainability
End Users as Data ScientistsEmpower individual end users to address their own challenges through ML
User-centric document classification
Label, build, deploy, monitor R&D not finished products
Human Machine Collaboration
19
Recognizing IntentUnderstand what an analyst is trying to do
Modeling intent in open-world environments
Useful interventionsEffectively support analysts in achieving their goals
Comparative studies
Microsoft Office Assistant, used with permission from Microsoft. From Wikipedia User:Norm
from Crouser et al (2020)
from Farrell and Ware (2020)from Hong and Watson (to appear)from Guo et al (2020)
20
Data Triage
Data TaggingData Retention
Information Retrieval
Data Prioritization
Data Exploration & Survey
Data Triage concerns the Classic Challenges of Big Data
Influence CampaignsIndicators, origins, &
provenance
Message content
Impact & effectiveness
Countering malign
influence
21
“The collection of tactical information about an adversary as well as the dissemination of propaganda in pursuit of a competitive advantage over an opponent” (RAND)
Influence can be:
• Online or offline
• Authentic or inauthentic activity
• Targeted or broad
• “Innocuous” or malign
Selected Cybersecurity Challenges
22
● Vulnerability Detection○ Symbolic Execution without Source Code
● Malware Evolution and Triage○ Polymorphic vs Metamorphic Obfuscation
& Detection Techniques
● Endpoint Detection and Response○ Machine Learning Research for EDR
● Cybersecurity Policy○ Effectiveness Assessment
Prioritization of Voice Data
User-Centric Document
Categorization
Processing Uniquely
Structured Forms
Handwriting Recognition in Scanned Docs
Additional Use Cases
23
Questions● If you would like to ask a question please use the Q&A feature in
Zoom
24
25
LAS White Paper and Proposal Timeline
● May 28, 2020 Call for White Papers
● June 15-18, 2020 LAS Collaborators Week
● July 17, 2020 White Papers Due● Sept 15, 2020 Preliminary Notifications
● Nov 1, 2020 Final Notifications
● Jan 1, 2021 Begin Period of Performance
● Dec 31, 2021 End Period of Performance
White Paper Submissions
● In order to propose work, you must submit a white paper ● More than one submission is fine. You should submit one white
paper for each project idea you have.
● You may submit team white papers with more than one performer.
White Papers Due July 17, 2020
26
White Paper Submissions
● All white papers must be submitted through web-based tool○ Link: https://whitepapers.ncsu-las.net
● Each white paper submission must include:○ Title○ All Funded PIs and Main POC○ Abstract○ Budget Request○ Technical Description
We ask that your abstract and white paper NOT contain classified, proprietary, or sensitive information of any kind.
27
NOTE: These will be entered separately in the submission tool and do not have to be repeated in the Technical Description
White Paper Content Guidelines● Detailed guidelines for what to include in a white paper are given in
Section 6.2 of the Call for White Papers○ Link: https://ncsu-las.org/2021-call-for-white-papers/
● Generally, the most important parts of your white paper will be the descriptions of:
○ The proposed effort (what question are you answering or problem are you solving)
○ The proposed approach (how will you address the question/problem)○ How the work aligns with LAS areas of interest○ The specific deliverables you expect from your work
28
White Paper Structure Guidelines
● Each whitepaper should be no more than 2 pages
● Optional additional page to discuss possible extensions to 2022
● Optional additional page to describe team capabilities, although a link to a website is preferred
29
White Paper Budget GuidelinesAcademic Partners● Standard award is equivalent to:
○ One month of summer salary support or academic release, plus○ One 12-month graduate student, plus ○ $3,600 in other direct costs
● Award can be used for post-docs, undergrads, etc., but must stay within total budget
● You may submit up to three additional scope options at the level of one additional graduate student each
30
White Paper Budget GuidelinesIndustry Partners● Standard award is $250k or less, which includes all direct and
indirect costs
● You may submit up to three additional scope options at the level of an additional $100k each
● If these levels of effort do not seem appropriate to the work you would like to propose, please contact Dr. Matt Schmidt, LAS Director of Programs, [email protected], to discuss other options.
31
Questions● If you would like to ask a question please use the Q&A feature in
Zoom
32
33
LAS Collaborators Week Schedule● Monday, June 15: Plenary Session
● Tuesday, June 16: “How to Work with LAS” Sessions
● Wednesday, June 17: Technical “Office Hour” Sessions○ Analytic Rigor and Performance○ Data Triage○ Influence Campaigns
● Thursday, June 18: Technical “Office Hour” Sessions○ Machine Learning Integrity○ Human Machine Collaboration○ Selected Cybersecurity Challenges○ Additional Use Cases