mesos2iam - events.static.linuxfound.org · zain malik / software engineer / schibsted media group....

39
mesos2iam Zain Malik / Software Engineer / Schibsted Media Group

Upload: others

Post on 24-Sep-2020

7 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iamZain Malik / Software Engineer / Schibsted Media Group

Page 2: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Schibsted Media Group

● 22 countries● 38 products● 1.2bn people● 30m+ daily users

Page 3: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

Fabian Selles

IvanIlves

AlanBover

SergiMansilla

JaimeJorge

ZainMalik

Page 4: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

VicentSoria

Contributor

Fabian Selles

IvanIlves

AlanBover

SergiMansilla

JaimeJorge

ZainMalik

Page 5: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

CRE (common runtime environments)

Page 6: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

>6K tasks >15k jobs >2k pods

CRE (common runtime environments)

Page 7: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

>15k jobs >2k pods>6K tasks / day>8GB / task>1.4 cpu/task~50 minutes

CRE (common runtime environments)

Page 8: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

scaling down cluster with 0 failed task

Page 9: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

scaling down cluster with 0 failed task

The creepy guys who track down all frameworks we use and ask them to implement mesos maintenance primitives

Page 10: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

scaling down cluster with 0 failed task

The creepy guys who track down all frameworks we use and ask them to implement mesos maintenance primitives

Page 11: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Team

scaling down cluster with 0 failed task

The creepy guys who track down all frameworks we use and ask them to implement mesos maintenance primitives

fyi: deathnode → https://github.com/alanbover/deathnode

Page 12: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam

Page 13: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM

• Stands for “Identity and Access

Management”

• Takes care of Who(authentication) and How(authorization)

Page 14: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM• Action-level permissions

• Resource-level permissions

• Resource-based permissions

• Tag-based permissions

• Temporary security credentials

• Service-linked roles

Page 15: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM with instances

Page 16: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

>1PB

API

Mesos clusterschedules job

>6k daily tasks

Page 17: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

>1PB

API

Mesos clusterschedules job

>6k daily tasks

Page 18: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

Page 19: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

Page 20: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

>1PB

Mesos cluster

Page 21: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

Instance profile with access to data

>1PB

assume role1

assume role2

resources allowed to role1

resources allowed to role2

Page 22: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

Instance profile with access to data

>1PB

assume role1

assume role1

resources allowed to role1

resources allowed to role1

Page 23: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Our use case

Instance profile with access to data

>1PB

assume role1

assume role1

resources allowed to role1

resources allowed to role1

Problem? Each user just need to know the other users iam role, create a job with other role and access those resources

Page 24: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

The problem

Privileged Instance Profile

>1PB

Common cluster

Page 25: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

iamlet’s dig deep

Page 26: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM

How do EC2 instances retrieve their credentials?

1. 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

2. 169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE

From version 1.11.0 in all AWS sdk clients it goes through the 1st option*

*If it's set AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, the client send the request to http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI} instead of the usual /latest/meta-data/iam/security-credentials

Page 27: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM

Actually that’s how IAM roles for ECS tasks are working

the ECS Agent populates the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable

With /credential_provider_version/credentials?id=task_UUID

in other words169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

is

169.254.170.2/credential_provider_version/credentials?id=task_UUID

Page 28: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

IAM in Action

$> curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

$> { "AccessKeyId": "ACCESS_KEY_ID",

"Expiration": "EXPIRATION_DATE",

"RoleArn": "TASK_ROLE_ARN",

"SecretAccessKey": "SECRET_ACCESS_KEY",

"Token": "SECURITY_TOKEN_STRING"

}

Page 29: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam

Page 30: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam

mesos2iam is a daemon that runs inside Mesos agents

To give us back the control of IAM policies on tasks level

Page 31: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam

● manage iptables rules● retrieve a TASK ID from container● fetch credentials for the task

Page 32: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam

mesos2iam

mesos-slave

task1 task2

Naked Instance Profile (almost no privileges)

credentials-host

Privileged Instance Profile

Page 33: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam mesos-slave

task1

credentials-host

Privileged Instance Profile

mesos2iam

169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Page 34: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam mesos-slave

task1

credentials-host

Privileged Instance Profile

mesos2iam

169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

fetch container_id

Page 35: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam mesos-slave

task1

credentials-host

Privileged Instance Profile

mesos2iam

169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

fetch container_id container_id

Page 36: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

mesos2iam mesos-slave

task1

credentials-host

Privileged Instance Profile

mesos2iam

assume role

Page 37: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

demo time

https://www.youtube.com/watch?v=ra3SOeO_yp0

Page 38: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

Opensource

CONTRIBUTIONS WELCOMEmesos2iam:https://github.com/schibsted/mesos2iam

smaug(naive credentials api):https://github.com/schibsted/smaugdeathnode: https://github.com/alanbover/deathnode

Page 39: mesos2iam - events.static.linuxfound.org · Zain Malik / Software Engineer / Schibsted Media Group. Schibsted Media Group 22 countries 38 products 1.2bn people 30m+ daily users. Team

QUESTIONS?