handling code and assets for huge projects using state...

Handling Code and Assets for Huge Projects Using State-of-the-

Art SCM Jason Spangler

Associate Technical Director BioWare - Electronic Arts

Summary • Introduction

• How Agile and Scrum Can Help

• Asset Management and SCM

Introduction

Electronic Arts (EA) • 9000+ employees • Studios and offices worldwide

BioWare • An EA Studio with offices in Canada

and Austin, TX, and 400+ employees • BioWare's vision is to deliver the best

story-driven games in the world.

How Agile and Scrum Can Help

Why Agile and Scrum? • Projects getting larger:

• More people • More code • More assets

• Hardware and software complexity increasing • Need to react faster to changes

• Revise designs and planning faster, more often

• Need to increase quality at the same time

Agile and Scrum Can Help • Manage projects and teams (small and large)

with frequent changes, chaotic

• Maintain and improve quality

• Provide tools and processes to reduce risk due to increased size and complexity of projects

How Agile and Scrum Helps SWTOR • With such a large team, using Scrum allows

us to… • Keep scrums (sub-teams) small enough to be manageable

• Maintain communication within and between scrums

How Agile and Scrum Helps SWTOR • With such a large team, using Scrum allows

us to… • Check point development each sprint

– Make sure problems (such as a large numbers of bugs) do not build up without being fixed periodically

– Show a working version of the game each sprint

• Estimate future velocity using past velocity to increase schedule accuracy

Scrum • A popular Agile development process

• 1986: Hirotaka Takeuchi and Ikujiro Nonaka • Described a holistic development approach that values

increased speed and flexibility, compared it to rugby • The New New Product Development Game (Harvard

Business Review, Jan-Feb 1986)

• 1991-2001: Sutherland, Scwaber, and others • Apply ideas to software development, describe, called it

Scrum

Scrum • Product Backlog • Sprint Backlog • Sprint • Deliver working version of software • Repeat

Scrum Documents • Product backlog

– High level feature set of entire project

• Sprint backlog – Tasks needed to implement sprint requirements – Team members sign up for tasks

• Burn down chart – Shows number of tasks remaining for sprint

Example Backlog

From http://www.agile-software-development.com/

Sprint Diagram

Diagram from Wikimedia Commons, Crea2ve Commons license

Scrum meeting • Once per day

• Attendees stand to keep meeting shorter • Not for problem solving (handled offline)

• Each team member answers three questions • What have you done today? • What are you planning to do tomorrow? • Do you have any problems preventing you from accomplishing your goal?

Teams • Scrum teams

• Cross disciplinary, 5-10 typically • Emergent: Many form, complete initiative, dissolve, while others long-lived

• Embedded Tester in scrum team very useful

• Scrum of Scrums (SoS) • ScrumMasters attend SoS • Scrum of Scrum of Scrums possible • Used in teams of 500+

When I Started Using Scrum • Started about 4 years ago • Read Scrum tutorials and experiences online • Talked to developers using Agile

methodologies and Scrum already • Team took Scrum and ScrumMaster classes • Team and management already wanted to try

something new to improve development, so buy-in was straightforward

Learn More about Agile & Scrum • Agile Alliance

• http://www.agilealliance.org/

• Scrum Alliance • http://www.scrumalliance.org/

• Control Chaos • http://www.controlchaos.com/

Asset Management

SCM, Code, and Assets • What problems were encountered with the

development of large codebases and digital assets? • Size • Number of files • Organization • Updating on user machines

SWTOR Perforce Stats • Assets - 865 GB

• 5,582,608 files • 12,514,388 revisions

• Code + Docs + Misc - 470 GB • 5,464,529 files • 11,365,081 revisions

• Data - 95 GB • 5,023,500 files • 9,373,244 revisions

Other BioWare Server Stats • Mass Effect & Mass Effect 2

• Size: 7.5 TB • Files: 4,264,342 • Revisions: 14,271,608

• Dragon Age: Origins

• Size: 2.5 TB • Files: 11,201,821 • Revisions: 38,051,854

SCM, Code, and Assets • Why use SCM for assets in addition to code?

• Track revision history for use in identifying problems

• Retrieve past revisions of assets to compare changes

SCM, Code, and Assets • Why use SCM for assets in addition to code?

• Branch and integrate multiple development lines

– Binary assets require policy to determine which branch wins in case of changes in both branches

– Policy should avoid changing same binary assets in multiple branches

Most Common • Perforce

• Most common in US games industry

• Others • AlienBrain • Subversion • Git • CVS • Visual SourceSafe

Project Needs • Solid server with good feature set

• Can be integrated with tools

• Good client UI usable by programmers, artists, designers, managers, etc

• Windows, Linux, Mac, etc

Perforce at EA • Over 4000 users on 90+ servers

• Spread among many worldwide studios • A single team server has handled 1000+ users

• Terabytes of data on largest server • 90% of files are binary (data and assets)

• My current project has 200+ users

Using P4 Groups and Permissions • Example groups

• Engineers • Artists • Game designers • Project Management

• Example permissions • Engineers can read and write code • Non-engineers only have read-only access to code

Perforce at EA • For more information about Perforce at EA

• Scaling Source Control for the Next Generation of Game Development, by Mike Sundy & Tobias Roberts

• Perforce User's Conference, May 2007

• http://www.perforce.com/perforce/conferences/us/2007/presentations/eapresentation_paper.pdf

Reasons We Use Perforce • GUI and command line clients

• Easy to integrate into build systems and tools

• APIs for many languages, including C++, Ruby, Python, Perl

• Can handle code, assets, and data

Reasons We Use Perforce Cont’d • Atomic operations

• Delayed copying of branched files decreases disk space usage

• Proven, shown it can handle huge depot sizes with many files and many users

Perforce UI

Introducing SCM • Integrate with tools that are easy to use for

artists, designers, and other non-engineer developers

• Good server hardware – costs much less than large team being impacted by slowness or downtime

Introducing SCM • Start from beginning of project

• Show examples of how asset SCM can help, past situations that could have been avoided or fixed

• Get system running smoothly before scaling team size up

SWTOR Editor Tool

System Overview

Client Tool

Game Server

Perforce (Assets) Perforce (Data)

Database

System Overview

Client Tool Game Server

Perforce (Assets) Perforce (Data)

Database Asset

Streaming Server Asset Cache

Asset Base Install

Some Problems and Our Solutions

Problems and Solutions • Problem: One live development environment

• Led to artists and other asset creators being impacted

• Frequently caused by gameplay programmers writing scripts in same environment.

Problems and Solutions • Our Solution:

• Tools and live editing server use Perforce as persistence backend.

• Split development environments and test gameplay branch before asset creators use it.

• Use Perforce to merge data in addition to code.

Problems and Solutions • Problem: Taking extra time for asset creators

to use Perforce UI when using tools

• Extra steps needed for simple operations

• Asset creators don’t use most options in Perforce UI. – Really only use open for edit or delete, and submit


• Integrate Perforce into tools and asset-related server processes via various APIs for C++, C#, Perl, Python, etc.

• Only expose Perforce operations needed in each tool to streamline usage and increase user speed

Problems and Solutions • Problem: Large number of assets and users

leads to directory structure organization organically becoming more chaotic

• New directories created when appropriate directory already exists

• Too many files in one directory to easily understand the organization

• No one knowledgeable about the directory organization keeping an eye and correcting issues


• Someone in charge of maintaining a clean, understandable directory structure and organization for assets

• Sets best practices

• Watches activity, periodically fix/improve directory structure and communicates changes to team

Problems and Solutions • Problem: Some assets are modified in

multiple environments but are binary

• Binary files can not easily be merged (without developing specialized tools)

• Some data stored as binary could logically be merged (lists of independent entries), while some can’t (like changes to a model)


• Convert binary data that logically could be merged to XML, which can be merged much easier, then post-process it to binary data for shipping game build.

• Set best practices for team to help reduce conflicts (editing same data)

• Developed tools (scripts) to automate most merging of assets and only require manual intervention for conflicts

Conclusion

Thank you • CEDEC Secretariat • Gordon Walton, Bill Dalton, and Robert

McKenna, BioWare - EA • Mike Sundy, Tobias Roberts, and Kevin

Patterson, EA

Questions? • Q&A

• Email: [email protected]

• Facebook and LinkedIn

• I only speak and read English (sorry).