Part 2 of a six-part series adapted from my master’s thesis, “Creating a First-Person Action Game in Unreal Engine”. Catch up on Part 1 — Games as a Medium or browse the full series and source project on GitHub.

Have you ever wondered how multiplayer games look behind the curtains? What is at their core? Why do they sometimes behave in funny ways? In today’s topic we will look at the fundamental principles of networking physics and demystify their behaviour.

A video is available on this topic.

Introduction

Physics is the fundamental part of every simulation — it represents the state of the world. Every simulation has an input, and gives out an output. In games, inputs are for example key presses, while outputs are the player’s position. The goal of networking physics is that every client has the same world view on their screen.

There are two important aspects to consider here:

• Precision of simulation — all objects should have the same position on all clients.
• Immediate control — regardless of connection, all controls should be immediate.

The problems we run into are:

• Network bandwidth limitations
• Network response time

It is important to note that there is no ideal solution for this problem [30], but we will present two possible solutions.

Networking techniques

In this chapter we will take a look at three techniques of networking physics that are currently in use in the games industry. The fundamental choice we have to make is which part of the simulation we will network — remember, a simulation has an input and an output, so these are our two choices. Whatever choice we make will propagate throughout the entire game.

Deterministic lockstep — networking input

Deterministic lockstep is a technique in which we send the inputs of a simulation, since a simulation that has the same inputs will give the same outputs, resulting in a theoretically infinitely detailed world that is fully synchronised to all clients [31].

There are a couple of technical problems with this method:

1. The simulation must be deterministic. Meaning, given the same inputs, it always gives the same outputs. This is solved with a checksum.
2. It is very hard to guarantee that floating point numbers will be rounded up in the same way in every environment, thereby ruining determinism.
3. The simulation needs to run in discrete fixed time steps (lockstep), so that all clients can be synchronised more easily.
4. To execute one simulation step, we must receive all inputs, meaning we will have to wait for the client with the biggest network response time. As the number of clients increases, so does the chance of a networking error [32].
5. Persistent worlds need special attention. If a new client wants to join a deterministic lockstep simulation, they must receive and execute all of the inputs that happened. A persistent simulation could run a few hours, or years, depending on its design, resulting in huge packets.

These are all technically solvable, but require that the entire environment is designed around them. So we require a physics system that is completely deterministic across multiple platforms and works in lockstep. To solve the problem of networking response times (3) we use a system of rewinding and playback. Preferably the physics system runs in a separate thread from the game thread, so that we don’t need to lock fps.

This is the implementation that the new Unreal Engine “Chaos” physics system focuses on, although, at this time, there is no clear consensus on how to solve the persistence problem, which will probably be solved with a state sync method [34], more on this later.

In this technique both the client and the server execute the physics simulation. The client runs it so that it can have immediate control and the possibility of rewind and playback, while the server runs it so it can approve or discard inputs and distribute them to the clients.

An advantage of networking only inputs is that you have no practical limitations on the world size and precision, although it is important to note that some implementations specifically sacrifice precision for faster rewinding, more on this later.

Usage

This technique is used mostly for single platform games, usually RTS titles with lots of units like Age of Empires [35], but Epic Games seems to be shifting to it as well, or at least supporting it — this is a consequence of their desire to support Chaos Destruction over a network, which brings an exceptional amount of detail to a simulation, making it very difficult to network with other methods.

Theoretical player limit

Let’s now take a look at the limitations of this system.

We assume that movement is supported through six degrees of freedom and that we have 20 abilities, this results in 32 possible inputs, or 32 bits, depending on whether they are active or not in a single moment.

We assume it’s being sent at 30 Hz, this results in 960 bits per second per client. On the server side, with a 10 Gbps link, we can receive:

(10 000 000 000 bit) / (960 bit) = 10 416 666 inputs

The theoretical number of players we can support is the square root of this number, remember, we must receive all inputs to simulate one step:

√(10 416 666) = 3227 players

The client’s bandwidth is therefore:

3227 * 960 bit = 3.1 Megabits per second

Just to give you some context, with a 100 Gbps link this works out to 10206 players, at a client bandwidth of 9.8 Mbps. At 60 Hz, which is an industry standard for physics simulations [36], and a 10 Gbps link, we get 1920 bits per second for inputs, which leads to 2282 players. It should be noted that we would probably encounter both server and client side processor limitations first, rather than bandwidth, but these depend on the implementation of the rewind system and the simulation’s fidelity.

Conclusion

Let’s now conclude with some advantages and disadvantages of this technique, assuming all technical problems are solved.

Advantages:

• All clients are almost immediately aware of the entire simulation
• The simulation can theoretically be infinitely detailed
• Client packet size is very small
• Some implementations are not limited by floating point size constraints, allowing huge worlds

Disadvantages:

• Very hard to scale client numbers, since we are limited by server bandwidth
• To ensure sync we need all inputs, no matter where they happened
• It must run on the client, so we are limited by the lower end of hardware requirements
• It is very technically demanding to implement

Theoretically, if we could spatially optimize this technique it could support even more players.

Snapshot interpolation — networking outputs

Snapshot interpolation is a concept in which we send the outputs of a simulation, meaning the positions of all objects.

In this technique only the server executes the physics simulation, while the client only “lerps” (animates objects from their old position to the new one) [37]. In most cases it does not require a lockstep approach, meaning it is much easier to execute and easily supports persistent worlds.

Usage

All of the games we looked at in the last chapter use this technique, and so do nearly all FPS games.

Compression

Positional data consists of a vector for position (96 bits) and a quaternion for rotation (128 bits), which is in total 224 bits [38]. This is obviously too much for the server to handle and it must be optimised.

To find out how we can compress this data, visit the “gaffer on games” article.

In short, we round the numbers as much as we can, losing some precision. Furthermore we use delta compression, as in we only send the changes relative to the previous position — this is not always used as it can be costly since we need to send acknowledge packets.

After these optimizations we arrive at an average 83 bit packet for a single positional update. This ultimately limits this technique to a floating point number with a 32 bit size [64], meaning the simulation can only process coordinates of a 20 km squared sized world.

State sync

Most games upgrade snapshot interpolation with state sync — this combines deterministic lockstep for immediate input on some objects, while selectively updating positional data for all other objects.

To selectively update positional data we use a priority accumulator, which determines the priority of an object’s positional update, based on the time that has passed since its last update and its proximity to the client [39]. This is a somewhat CPU intensive process, since it is done on a per-client basis, and can be a limiting factor when scaling the number of clients, depending on the fidelity of the simulation. Its biggest advantage is the ability to limit client bandwidth to an acceptable amount, usually 350 kbps [40].

Thanks to this technique we can have a high fidelity world that is at least partially synchronized across all clients — this is also the biggest drawback of this technique, since the clients never know the true state of the simulation. Like we saw in the previous chapter, an acceptable refresh rate is around 20-30 Hz, and the bare minimum is 10 Hz; Apex Legends uses it at 20 Hz at 100 players and only 75 ms delay [43].

This technique is in fact used in Unreal Engine 4, which uses lockstep for its character movement and state sync for all other objects [41]. It uses a maximum distance (Net Cull Distance Squared) to determine relevancy (if something should be replicated), and then prioritises the object — it even has a tendency to update objects in front of the player more frequently (as seen in ActorReplication.cpp), but it is much weaker than the one in Battlefield 4.

If inappropriately configured, it can ruin immersion and realism — for example, in the following video [42], the game Hell Let Loose configured buildable objects to load in at a shorter distance than other players, meaning players can be seen, but not shot through a buildable sandbag.

Furthermore, if it is too aggressively optimised and under high load, it can lead to severe desynchronization events, such as those seen in Scavengers [40], during their high player count tests, where they achieved 4138 players but experienced total positional loss. It should be noted that this is the worst case scenario for this technique, where every player can see all other players.

We will explore the theoretical player limit of this technique in another topic.

Conclusion

Let’s now conclude with some advantages and disadvantages of this technique.

Advantages:

• Flexible bandwidth
• Clients don’t need to run the physics simulation
• It can be spatially optimised

Disadvantages:

• A technically limited number of physical objects
• Clients never really know the state of the simulation
• Optimisations can be costly for server performance
• Optimisations do not work in a high density environment
• The size of the world is limited because of floating point number size issues

Choosing the technique for our project

When choosing the technique for our project we need to understand our needs and limitations. The first question you need to ask yourself is what techniques are supported in your environment. With Unreal Engine 4, only state sync is really well supported — in fact the entire networking stack is focused on it, so it is an obvious choice and this project will use it as well. If you are making a large scale RTS, you would nearly always choose deterministic lockstep, since the fidelity of the world is much less of a concern for this technique, but you will have a hard time finding an engine that has it well supported.

Now that we have built up a solid understanding of networking and its limitations, we can finally start working on our multi-server architecture, which will open up many more networking possibilities. Be sure to come back next week.

Continue to Part 3 — Multi-Server architecture and Dynamic Interest Management. The full six-part series and source project are on GitHub.