working well together: how to keep high-end game development teams productive
DESCRIPTION
During the production of PlayStation 4 launch title Killzone: Shadow Fall, Guerrilla Games struggled to finish in time as the size and scope of the game increased. Hear about the improvements they made to their build pipeline and walk away with key takeaways for making teams more productive by enabling collaboration and cooperation through good tools and processes, minimizing distance between developers, providing accurate and accessible information on the state of the project.TRANSCRIPT
#
Frank CompagnerGuerrilla Games
Working Well Together
#
• Background• Iteration• Teamwork• Stability• Releasing
Overview
Iterate
Launch
How we learned to stop worrying and love the development cycle
#
• Killzone Shadow Fall – PS4 launch title• 200 local users, 150 remote• Head revision: 1.8 TB (1.6 M files)• Depot size: 8 TB• Peak change: 180 GB / week• 60 dedicated build servers• All in Perforce since 2004
Background
#
P4 setup - nowServer
Build Farm (60)
Proxy Proxy Proxy Proxy
Users (200)
Proxy
Proxy
Proxy
Proxy
Cambridge
Guilford
San Diego
ShanghaiAmsterdam
#
P4 setup - soonCommit
Server
Build Farm (60) Users (200)
Proxy
Proxy
Proxy
Cambridge
Guilford
San Diego
ShanghaiAmsterdam
Edge
Server
Edge
Server
Edge
Server
#
P4 scale
Submits GB / week
#
• Monitors all vital stats of server and proxies• Simple Python script, produces simple html• Works for Windows and Linux servers
• Proxies cleaned by another Python script
• Both available in the Perforce Workshop
P4 monitor
#
P4 monitor
#
Iterate, Iterate, Iterate
#
Code
Compile
Link
Test
Creation means Iteration
Programmer
#
Code
Compile
Link
Test
Creation means Iteration
Programmer
IDE and Tools
Distribute
Incremental linker
CodeHotloading
#
Model
ProcessTest
Creation means Iteration
Artist
Hotloading
#
Model
ProcessTest
Creation means Iteration
Artist
Tools
DistributeHotloading
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
#
Lots of Iteration
Programmer
Rigger
Animator
Animator
Designer
Tester
Programmer
Modeler
Programmer
VFX artist
Designer
Audio Engineer
Modeler
Designer
Animator
#
Teamwork
#
• Does not look like this
Games development
#
• It’s more like this
Games development
#
• OK, like this
Games development
#
• Personal branch is fine (sandbox / streams / Git)
• Feature branches sometimes work• Give control over your environment• But create distance to others• Team work requires frequent branch switching
Can branches restore order?
#
• For us, branch switching is always expensive:– Amount of data and change– Unmergeable files– Code-Data dependencies
• Branches add complexity, we already have that• Team effort, we don’t want distance• Almost everybody works on the trunk• So why isn’t it always broken?
TBD - light
#
Stability
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation means Iteration
Team
Submit
Submit
Submit
Submit
Submit Sync
Sync
Sync
Sync
Sync
#
Programmer
Rigger
AnimatorDesigner
Tester
Creation needs Testing
Team
Submit
Submit
Submit
Submit
Submit Sync
Sync
Sync
Sync
SyncTest
Test
Test
Test
Test
#
• Test pre submit (user)• Test post submit (builder)• Test in parallel• Test as fast as possible• Still accidents do happen
StabilityUser B
Sync
< 30 minutes
User A
Submit Sync
Build Game SubmitSync
Test Game
…
Build Tools SubmitSync
Test GameSync
Cook DataSync
Sync
Upload
Cook Data Upload
Test
#
…Labels.xml
Build labels
Label 42: change 1200
write
Build machines
Label 41: change 1180
Safe-Sync Tool
read
Label 40: change 1155
Label 39: change 1150
• Each game build creates a label• Stored in Labels.xml file in depot• Builds add their result to the label• Builds can add artifacts to a label
• Nobody gets latest (not safe)• Everybody uses safe-sync tool
#
Safe Sync ToolSelecttests
Labels thatpass tests
Labels thatfail tests
• Custom safe-sync tool shows labels and test results• Use it to sync to a known good label
– This includes artifacts from other builds
Sync to get label
#
Releasing
#
• We branch for every release (often every week)• Nobody works on the release branch• Everybody works on the trunk• Changes are merged up from main to release
Release Branches are great!
main
release
= change
= release
#
Release Manager• Cherry picking rules!• Gives you a lot of control• But also lots of confusion• And mistakes are easy
• Good tools can fix this• Give flexibility• And control
(See whitepaper for details)
#
Conclusion• Iterate as fast as you can• Don’t branch too much• Test everything• Sync should be safe
• Release all the time• Control what you release
Speed
Control
#
RESOURCESPerforce Workshop (P4 monitor, cache cleaner):
https://swarm.workshop.perforce.com/files/guest/frank_compagner
ReleaseManager:
http://www.perforce.com/resources/presentations/user-conference-talks/tools-for-smooth-release-cycle
http://www.perforce.com/sites/default/files/guerrilla-games-whitepaper.pdf