wake up and smell the metrics! metrics driven development in online games larry mellon darius kazemi...
TRANSCRIPT
Wake up and smell the metrics!
Metrics Driven Development in online games
Larry MellonDarius Kazemi
AGC ‘08
Data mining is
pure gold!
What stops
people from
reaching it?
Cultural collision is our Smaug!
Boxed goods mentality
Online service reality
Wrong assumptions
lead to painful decisions!
Larry Mellon(20+ years of engineering @ scale)
• Alberta Research Council & Jade Simulations– Distributed computing, 1982+– Optimistic computing, 1000+ CPU virtual world– Fault-tolerant cluster computing
• Synthetic Theatre of War: virtual worlds for training– DARPA: 50,000+ entities in real-time virtual worlds– ADS, ASTT, HLA & RTI 2.0, interest management
• EA (Maxis): The Sims Online, The Sims 2.0• Scalable server architecture• Automated testing to accelerate production and QA• Player, pipeline & performance metrics
• Emergent Game Technologies (CTO)• Architect for scalable, flexible MMO platform
Research era
Wife era
Boxed goods industry is
evolving online and needs to
adapt to service industry
requirements!
Why is it hard to get funding for something as clearly useful as metrics?
Nobody wants to pay for it, because no one has traditionally paid for it!
Features QA Metrics, CS, …
$$$$$$$$$$ $$
Diasporas kill off the home-rolled systems
Business Intelligence has driven the success of many other industries for years!
Las Vegas Strip
Metrics help manage complexity“When you can measure what you are speaking about and can express it in
numbers, you know something about it. But when you cannot measure it, when you cannot express it in numbers, your
knowledge is of a meager and unsatisfactory kind."
- Lord Kelvin Institution of Civil Engineers, 1883
“The general who wins the battle makes many calculations in his temple before the battle is fought. The general who loses makes but few calculations
beforehand.” -- Sun Tzu
The people who led the scientific revolution depended on metrics!
Pasteur: A science is as mature as its measurement tools. Maxwell: To measure is to know.
“The three largest factors that will influence gaming will
be […] and metrics (measuring what players do
and responding to that)”
-- Will Wright
The Secret of The Sims", PC Magazine, 2002.
http://www.pcmag.com/article2/0,1759,482309,00.asp
– GIGO –Avoid false causality by correlating
data!
3 – Final road block: Metrics are role-specific and hard to move
across boundaries
The “3P's” model of game metrics
PlayerPerformanceProcess
16
Next Gen Games Increased
Complexity Increased
Complexity of Analysis
Art from “Fun Meters for Games”,Nicole Lazzaro and Larry Mellon
17
18Next Gen GamesNext Gen Games
Tuning imbalances or exploits can throw your entire economy out of kilter, but remember to triangulate!
Tuning imbalances or exploits can throw your entire economy out of kilter, but remember to triangulate!
Know Your Players
Comparing group views is another no-brainer…
Process metrics
• Find the leaks that are slowing you down or costing you money!
• Another cultural problem– Process = evil– Tools != game feature
• Not ‘fun’ to build• No ‘status’
– Thus, worst possible programmers are assigned to most critical systems…
Why improve our process stability & iteration rates? (I’ve grown to hate this question)
1. Rapid iteration rates• Improve the player experience: soup to nuts, and
quickly, before they leave• We cannot yet write down and specify what fun is
before interacting with it!2. Team efficiency: with large and sometimes distributed
teams, a broken build or crashing client can slow down EVERYBODY!
3. We need rapid, stable content updates pushed out as fast as people can buy them!
Fixing development leaks is free money!
Process: Measurable targets & projected trends give you actionable progress metrics, early enough to react
Any metric(e.g. # clients)
Target
TimeAny Time
(e.g. Alpha)First sample Now
Risk!
Real data gives you faster focus, less debate!
• Question for the audience: if you have 20 engineers in the room, how many opinions do you have on what the team needs to do next?
• Numbers cut the debate from “what is next?” to “how do we fix the thing we found?”
• You QUICKLY go from finding the key data and fixing the problem, not searching the entire data space, just to get started!
Performance: engine metrics(Dev & Ops)
• Debugging & optimizing large-scale, complex system – With hackers, non-determinism & constant change– Tick-tock!! Users (and VPs) are waiting…
• Most engineering metrics tools are for one person, one machine, one debugger– Need system-level view to deal with MMOs
• Studio head metric: only team-wide, red-ink email– GET THIS METRICS SYSTEM IN THE LIVE SERVERS, BY
TOMORROW (tired of ‘black box debugging’ across 20 servers and 200 clients)
Lower dev & ops costs: FREE MONEY!
Profit…
NewContent
Regression
CustomerSupport
Operations
~ $10per
customer
Profit…
Regression
CustomerSupport
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Profit…
CustomerSupport
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Lower Testing CostLower Testing Cost
Profit…
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Lower Testing CostLower Testing Cost
Happy Customers Don’t CallHappy Customers Don’t Call
Process (Dev & Ops): Lower recurring costs that are worth the cost to lower
[makes you more money with the same number of players!]
Profit…
Operations
~ $10per
customer
Lower New Content CostLower New Content Cost
Lower Testing CostLower Testing Cost
Happy Customers Don’t CallHappy Customers Don’t Call
Lower bandwidth & CPULower bandwidth & CPU
Process: Rapid Iteration is how you find fun!
Project Start Launch
Implementation options(both are hard)
1. Grassroots movement & scale as required
2. DEFINE metrics as a fully funded feature needed for success in an online service!
1.a How to start your own Mini metrics project
Image by Flickr user RaeA; Creative Commons (BY-NC-SA)
1.b How to scale your own Mini metrics project
Image by Flickr user Dredrk; Creative Commons (BY-NDW)
Multiple views of data provides a deeper understanding and fewer analysis errors
Time
AIdata
Player and game actions
Minute 1
1.AI: open door2.AI: cook food
Minute 2
1.Game: fire breaks out
Screenshots
Screenshots
Minute one
Minute two
Culture Shift option:Treat metrics as a critical feature from day one!
Fund everything that helps both team and customers, not just game play!
Features QA Metrics
$$$$$$$$$$ $$$$ $$!!!
Metrics are your first-pass tool : they quickly find you a starting point that would take hours/days for you to
find by yourself
Find oddities automatically for you, then you decide what action to take
Scaling the metrics system as data scales
Train ‘bots to look for
patterns of data when the
datasets get even larger
Takeaways• Grassroots or formal feature : either way, get started with metrics
across your entire game service• Find something that will help an obviously important question to be answered
with facts rather than guesses
• To get executive and funding support, translate your day-to-day need for metrics into the metrics the executives understand to justify the funding • You will either make more cash with this change!• Or you’ll save more cash with this change!
• Presentation and aggregation are key: use the computer to do the hard parts• People should see high-level data, with easy search and correlate
functionality, and access to the raw data
The migration online is a Darwinian
moment for our industry
• Boxed goods mentalities must shift to online service realities, or die… • Complexity: people need metrics to be able to do their jobs• Player Retention: It’s not just about game features anymore!
Question:How would you rather live your life?
OR
Measure
Change
Measure
GuessChang
eHope
Slides are online at http://www.maggotranch.com/biblio.htmlwww.orbusgameworks.com/blog