gitlab infrastructure 20160621
DESCRIPTION
GitLab InfrastructureTRANSCRIPT
GitLab InfrastructureStatus Report
We have HTTP queue time in monitoring, so we ran an experiment
What if we add more memory to workers?
This had a good impact across the board - less load in general
Specially on API timings (authorized_keys lookup timings)
This is why git-ssh is going faster, but there is still a long way to go.
Some things did not go well with the change
Redis leaves connections behind - GitLab max connections open -> outage
Deploys - RC3 blowed in production
On a Friday 1AM my time.
So we built staging the next Monday staging.gitlab.com is way smaller and less powerful than GitLab.com, but it has all the data.
Thanks @Jeroen
Done is better than perfect
Deploys - RC4 blew in staging that very same Monday
<
Staging Production
Postgres is still dying on us, or was it?
Query counts monitoring allowed us to corner the
problem and get it fixed In RC5
Thanks @marat!
Monitoring - improvements on how methods are measured
We are actually
showing where the time
is going now.
Thanks @Yorick!
Performance - no progress besides the API
Storage● Cephfs - dev.gitlab.org has been running on cephfs for the last month
○ Did you noticed? No? That’s good! :)○ Pushing the linux kernel to it takes 27 minutes ~1.5Gb○ Pushing the linux kernel to GitLab.com takes between 1:30hs to forever
● Our measurements were wrong, Cephfs gives 500/150 IOPS● But it scaled without a hiccup up to 98 workers nodes (clients).● We are testing behaviour when we add more nodes/ODS, etc.● We have a plan to move to Cephfs without downtime
Storage - capacity
Git data - 28TB out of 49TB
Shared data - 3TB out of 4TBwe can grow this one easily-ish
Other news● What’s coming soon
○ Multiple mount points/shards - Thanks @Alejandro!○ 2 new hires
■ Alex as a Production Engineer■ Ahmad as a Performance Specialist
● We are talking with CI to transfer knowledge into Infrastructure.● We are going to take over GitHost.io● We are starting to build infrastructure monitoring that can be shipped with
GitLab● We are hiring!
That’s all folks!