jan hloušek, keen software house
TRANSCRIPT
Multiplayer challenge
Jan HloušekPetr Minařík
Sandra Lenardová
Keen Software House
• Founded 2010 by Marek Rosa• 50+ developers• Privately held• Games
- Miner Wars- Space Engineers- Medieval Engineers
• PC Steam (for now)
Space Engineers
• 3+ years on Steam Early Access• 2M+ players• 200k+ mods on Steam Workshop• VRage engine
- C# .net 4.6- DX11- Havok
Space Engineers
• Sandbox• Infinite playground• Editable / destructible environment• Voxels / Planets• Survival / Creative
Challenge
Internal challenge for delivering new multiplayer code• 2 teams joined competition• 3 months deadline• 1 million CZK reward• Released in SE beta on Christmas• Overwhelming community reaction
• Play testing• Integrating debug tools• Experimenting
- Hacky implementations of various ideas- Play testing / Honest feedback- Wrong paths abandoned without mercy
• Coding- Removing code- The least time spent on coding actually- Removed more code than added
Development cycle
Debug tools - Statistics
Debug tools - Profiler
• Server / Client• Observe remote’s profiling data• Save / load for later analysis
Debug tools - Profiler
Debug tools - Debug screens
Debug tools - Clumsy
• https://jagt.github.io/clumsy/
Debug tools - Physics Remote Debugging
Debug tools - Spectator
• https://sourceforge.net/projects/tracetool/
Debug tools - TraceTool
What we learned
• Clients were telling server what to do- Resulting in all sorts of desyncs- Ultimately resulting in physics instability and explosions
• No lag compensation- Resulting in rubber banding and jittering- Poor gameplay experience
• Time synchronization using altering physics velocities- Resulting in all sorts of desyncs- Increasing code complexity
• Universe synchronized using linear complexity- Server performance problems
• Authoritative server- Client sends player inputs- Server sends the world state changes
• Physics-based prediction- To combat network lag
• Synchronized simulation steps• Spatial optimisations
- Cluster based synchronization
• Admin tools
What we changed
Prediction - original state
Client 1 Client 2Server
Prediction - original state with lag
Client 1 Client 2Server
Lag
Prediction - removing one frame lag
Client 1 Client 2Server
Lag
Prediction - corrections
Client 1 Client 2Server
Lag
corr
ection
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
Network protocols - TCP
• Connection based
• Guaranteed reliable and ordered
• Automatic packet splitting
• Flow control
• Easy to use
Network protocols - UDP
• Connection managed by ourselves
• Possibly unreliable, unordered, duplicated
• Packet size matters
• Possible line congestion
• Requirements- Get the packets as fast as possible to end point- Old packets are no use for us- Dropped packets will soon be compensated with new data- Out of order packets are still an issue - lets drop them for now
• UDP is a better match
Network protocls - conclusion
Prediction - synchronized simulation steps
Client 1 Client 2Server
corr
ection
1
2
3
4
5
6
7
8
9
1
2
3
4
5
Prediction - server buffering / final state
Client 1 Client 2Server
corr
ection
Playout delay buffer
1
2
3
4
5
6
7
8
9
1
2
3
4
5
Spatial optimizations
• Synchronize with clients their points of interest- Saving network bandwidth and performance
• Simulate only points of interest- Rest of the world is “sleeping”
• Delta time update
- Waking up entities getting into point of interest
- Simulating them in one update using the delta time while they were sleeping
- Potential problems
Spatial optimisations
Exceptions
• Antenna systems- Spanning large distances- Have to be simulated using custom system
• Remote controls- Point of interest is switched to the new location
• Remote cameras- Multiple points of interests for each client
Admin tools
• Trash removal• Block limits• Player levels
What next?
Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing Bug fixing
And then?
Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing Optimizing
And after that?
Performance balancing• Master server monitoring slave servers’ load• Responsible for cluster management
- Adding- Removing- Moving
• Allowing MMO scale of player numbers
Continuation of...
References
• Source multiplayer guide- https://developer.valvesoftware.com/wiki/Source_Multiplayer_N
etworking
• Gaffer on games- http://gafferongames.com
• Multiplayer Game Programming- Joshua Glazer an Sanjay Madhav, Addison-Wesley Professional
Looking for a challenge?
• MMO scale multiplayer• SE XBOX port• Micro voxels• Vulkan rendering• Multiverse
Join our team!Contact:[email protected] 485511
