Git-based CTF:A Simple and Effective Approach to

Organizing In-Course Attack-and-Defense Security CompetitionSeongIl Wi, Jaeseung Choi, Sang Kil Cha

KAIST

Capture The Flag (CTF)

2https://ctftime.org/ctfs

CTF: Cybersecurity competition that involves capturing a flag

Types of CTF

3

Attack-and-Defense Style

JeopardyStyle

Real time, Realistic

One-way, Problem solving

Number of CTF Events in 2018

https://ctftime.org/event/list/past 4

Attack-and-Defense Style

JeopardyStyle

8 times 73 times

In-Course Attack and Defense CTF

5

• Class Capture-the-Flag Exercises, USENIX 3GSE ’14

• Build It, Break It, Fix It: Contesting Secure Development, ACM CCS ’16

Difficult to organize!

6

Game Server

Team1 VM Team5 VM

Attack & Defense CTF Infrastructures

Team1 Team5

TeachingAssistant

Flag

7

Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

Need interaction between teamsC1: Interactivity

Challenge

8

Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C1: Interactivity Challenge

Network, VM, DB, Scoreboard, etc.C2: Configuration

Challenge

9

Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

Need monitoring and administering

continuously

C3: MonitoringChallenge

10

Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

11

Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

• C1: Interactivity Challenge− SWPAG, USENIX ASE ’17− InCTF, USENIX ASE ’16

• C2: Configuration Challenge− SWPAG, USENIX ASE ’17− CTFd, USENIX ASE ’17

• C3: Monitoring Challenge− VM-based Framework, USENIX 3GSE ’15

• C4: Contents Creation Challenge− BIBIFI, ACM CCS ’16− SecGen, USENIX ASE ’17

Handle only a subset of the challenges

12

Recent Researches:

Previous Work: BIBIFI, ACM CCS ’16

13

Build-It Break-It Fix-It

Does not allow real-time attack and

defense exercise

Teacher should manually check

every fix

DefenseAttack

C1: Interactivity Challenge

C3: Monitoring Challenge

14

Challenges in BIBIFI

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C4: Contents Creation

Challenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Can we handle all the challenges?

15

We propose

Git-based CTF

16

GitHub as a CTF Framework

17

Local Repository

Local Repository

Developer A Developer B

Pull Pull

PushPush RemoteRepository

GitHub = DB

18

Local Repository

Local Repository

Team 1 Team 2

Pull Pull

PushPush DefenseAttack

Service

RemoteRepository

OrganizerLocal Repository

Organize

Pull

Push

19

Our Goal: Handle All the Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

20

Handle Interactivity Challenge

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

Real time attack and defense

21

Real Time Attack and Defense

Attacker Defender

Round System

Periodically award points until it is fixed by the defending team

Repository as Scoreboard

22

Local Repository

Local Repository

Team 1 Team 2

Pull Pull

Check score

Scoreboard

RemoteRepository

OrganizerLocal Repository

Evaluate

Push

Check score

23

Handle Configuration Challenge

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

Real time attack and defense

Git-based infrastructure

24

Game Server

Team1 VM Team5 VM

Git-based Infrastructure

Network, VM, DB, Scoreboard, etc.

25

Game Server

Team1 VM Team5 VM

Git-based Infrastructure

26

Game Server

Team1 VM Team5 VM

Git Repository as Service

GitHub Repository

VulnerableProgram

27

Distributed System

GitHub Repository

VulnerableProgram

Cloned GitHubRepository

VulnerableProgram

Exploit(1) Clone (2) Find

(3) Submit Exploit as a GitHub issue

Attacker

28

Handle Monitoring Challenge

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

Real time attack and defense

Git-based infrastructure

Automated verification

system

29

>_

gitctf.pyTeaching Assistant

• Verify exploit in each round• Manage the game score

Execute

Automated Verification System

30Scalable and Lightweight CTF Infrastructures Using Application Containers, USENIX ASE ’16

VulnerableProgram

Docker Container

Exploit

Docker Container

Random String

Execution Result

Copy & Execute

Exploit

Flag

Automated Exploit Verification

Public Verification

31

Local Repository

Local Repository

Team 1 Team 2

Pull Pull

PushPush

Service

RemoteRepository

OrganizerLocal Repository

Verify exploit

Pull

Push

Verify exploit Verify exploit

Publicly Accessible!

32

Handle Contents Creation Challenge

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C2: ConfigurationChallenge

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

Real time attack and defense

Git-based infrastructure

Automated verification

system

Shifting creation burden to student

Shifting Creation Burden to Student(BIBIFI, ACM CCS ’16)

33

Hands-onDevelopment

PreparedProgram

GitHub Repository

Injecting Vulnerabilities

34

PreparedProgram

VulnerableProgram

Intended Vulnerability

Unintended Vulnerability

Injection

GitHub Repository

35

Our Goal: Handle All the Challenges

Game Server

Team1 VM Team5 VM

Team1 Team5

TeachingAssistant

C1: Interactivity Challenge

C3: MonitoringChallenge

Need to invent new problems for every competition

C4: Contents Creation

Challenge

Real time attack and defense

Git-based infrastructure

Automated verification

system

Shifting creation burden to student

Evaluation Setup (Preliminary)

36

• IS521 Information Security Laboratory 2018 in KAIST, Korea

• 21 students (11 of them had no experience in CTF), 6 teams

• Preparation: Develop a simple secure messaging application (use either C or C++)

• Injection (Individual): Inject at least one vulnerability• Exercise (Individual): Report unintended vulnerabilities or

functionality bugs

Diversity of Injected Vulnerabilities

37

The students introduced 28 vulnerabilities in the 6 distinct applications

Help the instructors prepare a diverse set of CTF challenges

• 14 vulnerabilities and 18 functionality bugs were reported

• Each team had at least one unintended vulnerability

• Unintended vulnerabilities are found mostly by experienced students

Exercise with Unintended Vulnerability

38

<source/>

PreparedSource Code

010111010011101110010010Compiled

Binary

Discussion: Binary-Only CTF

39

Compile

GitHub Repository

010111010011101110010010Compiled

Binary

Discussion: Binary-Only CTF

40

GitHub Repository

Attack & DefensePlayer

Open Science

https://github.com/SoftSec-KAIST/GitCTF

41

Demo

42

Demo Scenario

43

Local Repository

Local Repository

Pull Pull

PushPush (3) Defense

Service

RemoteRepository

Local Repository

(1) Turn on evaluator

Pull

Push

Team 1(Attacker)

Team 2(Defender)

Organizer

(2) Attack

44

Open Science

https://github.com/SoftSec-KAIST/GitCTF

45

Question?

46