Update: years after writing this chapter, I helped ship a production version of these ideas at Bohemia Interactive. See Server-Authoritative Vehicles: From Arma Reforger to DayZ.

Part 4 of a six-part series adapted from my master’s thesis, “Creating a First-Person Action Game in Unreal Engine”. Previously: Part 3 — Multi-Server Architecture and Dynamic Interest Management. Full series and source on GitHub.

Note: this chapter of the original thesis wiki was left partially translated from Croatian — I’ve smoothed over the roughest phrasing below, but some sections stay close to the source’s literal wording.

Introduction

In this chapter, we will implement networked features that are critical to a dynamic world, show the iterations we went through, and make suggestions for future iterations. These are mainly vehicles and a stable physics algorithm.

• Networked PhysX vehicles using INetworkPredictionInterface:

• Custom physics for more stable dynamic objects:

Vehicles

Vehicles are a very challenging case of networked physics, and are therefore addressed in this paper. At the heart of the challenge is the fact that any client must have immediate control, while being under server authority. Several methods for dealing with this problem will be shown through iterations.

First iteration — position replication

The first iteration is the simple movement replication that UE4 uses as a basis for all physical objects. It is activated via the replication menu via the “Replicate Movement” switch. This is the only originally available way of running a vehicle in UE4.

This solution is simply unsatisfactory, for one reason: lack of immediate control. During driving, we immediately notice that the controls are stiff — we are constantly “returned” to previous positions and the like.

The authors of the OWI Enhanced Vehicle Movement extension, which is available for free on the Epic Games Store, suggest modifying the project’s replication settings, but they only hide the problem, not treat it [82].

You can test this iteration yourself by navigating to Content/OWIContent/Shared/Maps/DesertLandscape/Maps/DesertLandscape_v1 with 2 players, without SpatialOS networking, and playing as a client.

To simulate a packet delay, as in a real environment, we need to open the console (the tilde key, `) and enter the command NetEmulation.PktLag 100 to simulate a 100 millisecond response delay.

To understand why this happens, we need to take a closer look at how the vehicles themselves function. The vehicles move because of their motion component — the WheeledVehicleMovementComponent class — because of the input it receives. The inputs are turning, throttle (forward), braking, parking brake and current speed. These inputs result in an increase in vehicle acceleration, and thus in its movement, i.e. change of position. Replication of these variables can also be seen in the ServerUpdateState function.

This is where we come to the problem — remember, we work in a networked environment, the consequence of which is that replication of client values to the server happens only 20 times per second. Based on these values, a simulation is performed on the server. On the client side, the simulation is also performed, but with current inputs, which can change at any time, not just when replicating them. Let us now look at one example of this consequence, made according to Sam Pattuzzi [83]. In the next picture you can see the result of various inputs on the client side, i.e. accelerations.

Suppose now that the server receives inputs every 5 steps. The result is a completely different acceleration calculation, because the server must assume that acceleration is constant between updates.

The result of this procedure is a different position on the server, and since the server must be authoritative, its position is taken as correct. Furthermore, if we were to solve the above problem, we still run into problems synchronising server positions. Since the server’s computed position is delayed by approximately round-trip time plus processing time, the client will always be “returned back” to some previous position, resulting in the loss of immediate control. It is important to note that this is usually not noticeable in “slow” vehicles.

In the next picture we can see why this happens.

• The blue line represents the acceleration
• The orange line represents the client position on the X axis
• The gray line represents the server position on the X axis

The following happened:

1. On the client side, acceleration increases at time 5, when the forward button is pressed.
2. At time 7, the server receives the replicated input, performs its simulation, and replicates the new position to the client.
3. At time 9, the client receives the position update and applies it locally, which results in snapping back to the old position.

Second iteration — INetworkPredictionInterface

The second iteration is a hand-crafted implementation of the INetworkPredictionInterface networking interface over PhysX vehicles. This interface is intended to solve the above problems. It works on the basis of a networked role [84]. A character can have three roles depending on who controls it and where:

Table 6: Description of networked roles [84]

So on each of these three roles a separate process takes place:

Autonomous Proxy

1. The client locally controls the autonomous proxy. The PerformMovement function initiates the physical motion logic from the motion component.
2. We compile the saved moves (FSavedMove_Character), which contain information about how we just moved, and save them in SavedMoves.
3. Similar saved moves are combined and sent to the server with the ServerMove RPC.

Authority 4. The server receives ServerMove and repeats the movement with PerformMovement. 5. The server checks whether its position matches the client’s. 6. If the positions match, a signal is sent that the movement is correct. If they do not match, the motion is corrected with the ClientAdjustPosition RPC. 7. The server sends its location, rotation and current state of the character to other clients via the ReplicatedMovement structure.

Simulated Proxy 8. If we receive a ClientAdjustPosition, we replay the server’s moves and use the SavedMoves list to step through to the final position. Resolved moves are removed from the list. 9. Replicated location data is applied, with smoothing used for the visual representation of the movement.

The result, for characters, is instant control with a server-authoritative position.

When we apply this principle to PhysX vehicles, or their motion component, we arrive at a seemingly good solution — the vehicle visually looks as if it’s moving properly, without the previous artifacts.

Implementation inspired by user MazyModz [85].

The problem we run into is a lack of current control — in its basic form this implementation is based on the fact that we send inputs to the server, it plays them and sends us the outputs, which we apply to the vehicle. The result is visible in this video [86].

You can test the delay by running the world in Content/VehicleCPP/VehicleMap_V2 with 2 players, without SpatialOS networking, and playing as a client. To simulate a packet delay, open the console (`) and enter NetEmulation.PktLag 100.

The crux of the problem lies in performing the simulation on the client, because with PhysX we cannot simulate forward or backward movement — it is a separate process performed after the game thread’s tick. This problem is also stated in the basic documentation for Unreal Engine:

“Have clients predict behavior of client owned actors based on player inputs; simulate this behavior before receiving confirmation from the server (and correcting if necessary). We use this model for Pawn movement and Weapon handling, but not for Vehicles, as the complexity of saving and replaying the physics simulation outweighs the benefit of reduced latency for vehicles handling, where typical internet response latencies aren’t that different from typical real world vehicle control response latencies.” [87]

In fact, we implemented a slightly better method than movement replication, expressed in the documentation for networked management of physical bodies:

“For Vehicles (PHYS_RigidBody Actors), there is the following network flow: 1. Press key on client 2. Send inputs (throttle, steering, rise) to server — replicated function ServerDrive called 3. Generate outputs (OutputBrake, OutputGas, etc.); pack into replicated structs that can be sent to the client — ProcessCarInput() called on server 4. Update vehicle on server and client; use outputs (OutputBrake, OutputGas, etc.) to apply forces/torques to wheels/vehicle — UpdateVehicle() called on client and server” [88]

It should be noted that with PhysX, true immediate control can be achieved using Immediate Mode to import the physics scene and include all relevant objects in it [89]. This is a favourable basis for the next iteration, but due to the complexity of execution, time constraints and the scope of the problem, it was not performed for this thesis.

Third iteration — custom physics

To demonstrate that it is possible to have both immediate control and server authority, we created our own vehicle model, inspired by the work of Sam Pattuzzi [83], which uses a simplified algorithm but is essentially identical to that in the Character Movement Component.

The fundamental difference is that we have our own hand-made physics system that handles movement, and we can call it independently of the game thread. This project is available at this repository [90]; it should be mentioned it has been minimally modified to launch on newer versions of UE4.

The result is instant control with a server-authoritative position. It should be noted that no further hydraulics simulation and the like from the PhysX model were implemented, and no rotation detection was made.

You can test the delay yourself by starting the project with 2 players, without SpatialOS networking, playing as a client, and using the NetEmulation.PktLag 100 console command as before. In this implementation, we can see that regardless of the size of the delay, current control is maintained.

There are already ready-made models that implement the same methodology in a hand-made physics system [91].

Fourth iteration — Chaos physics

Looking to the future, the Chaos physics system will use an interface at its level, simply called “Network Prediction”, which means that any physical object can at the same time be controlled and simulated/predicted, without having to wait for a server position, but with server authority [92].

It is a deterministic system that only receives input from other users. It can be tried out by downloading UE5 [93], and activating the “Network Prediction” and “Network Prediction Extensions” plugins.

Furthermore, the extension needs to be configured in the project settings — it is necessary to activate “Force Fixed Tick For Physics”, which ensures the physics system runs in discrete steps, 60 times per second.

Now we can drop the specified interface into our world — fortunately there are already prepared implementations in the extension plugin, we just need to set the default pawn to NetworkPredictionExtras_ControllablePhysicsBall. It should be mentioned that in the C++ classes there are more maps with other examples available under NetworkPredictionExtras Content/Maps.

This physical sphere is actually a very good representation of a vehicle — it can be given some input, which produces some force; it is practically only necessary to implement the interface, which sends these inputs in a predefined manner and takes care of rewinding and the like. The result is visible in this recording [94].

By turning on the packet delay simulation, we can see current control is still maintained — when we get a delayed input, the whole simulation is rewound, that input is simulated, and all the inputs that came after it locally are replayed. The result is instant control and a current picture of the world, regardless of connection delay.

This is a good basis for testing a deterministic system — namely, by running the TestMap_PhysicsControllable_Single map we can see that the green balls on both clients have the same positions. It should be noted that the positions of the green balls were never replicated — only the inputs of the users controlling the brown balls.

If you would like to observe the process of rewinding the simulation, you can use the “Network Prediction Insight” plugin, which provides an overview of the simulation frame by frame [95].

Stable physics algorithm

Sometimes we would like to simulate the precise physical movement and rotation of huge objects, which can affect the course of the game — think falling buildings that can cover a large area of a map, and the like. The goal is a stable and smoothly moving physical object in a networked environment, without any “jitter”. To prevent any possible manipulation of objects, this simulation must run on the server side.

Algorithm

This problem can be divided into two phases: the first is determining the position of the physically simulated object; the second is waking that object up to perform the simulation only when necessary.

The algorithm for running the physics simulation is as follows: on the server side the simulation is performed, and when the final location is reached and is at least 1 second identical to the previous one, it ends, because it is assumed the object has stopped.

This approach is necessary due to the limitations of state synchronization in UE4, because otherwise very small changes in the position of the object would be sent, resulting in unnecessary packets, while the simulation would not be much more accurate. It is also needed because without it there is a degradation of experience, due to constant “jitter” (small movements of the object) caused by various factors:

• A sensitive client-side prediction algorithm
• The inaccuracy of floats and transmitted quanta
• Collision checks are usually imprecise — large simulations run poorly with high precision

It is extremely important to get rid of jitter, because clients will constantly interact with these objects (walk on them). It is possible to avoid this problem by increasing the detail of the simulation and by substepping (running the simulation multiple times in one cycle), but since we are simulating thousands of physical objects, we cannot afford it.

First iteration — heartbeat

The first attempt: we perform a simulation, and turn it off; every second we perform a “heartbeat” technique that runs physics briefly so the object doesn’t float in the air. Although the result is seemingly accurate on each client, it is still different from the server, because collision calculations always change regardless of whether the object is actually at rest, and there is no need to send packets.

Second iteration — dormancy

The next attempt applied the concept of “Net Dormancy”, or a wake-up channel. In short, before we perform the simulation, we wake up the channel, which will send a notification about the result, then we put the channel back to sleep, which effectively determines the result of the simulation. The problem with this attempt was waking up — because we woke the simulation on every new collision, which again results in waking and simulating, and “jittering”.

Third iteration — on/off method

The third iteration consists of running a simulation, and once we have determined a satisfactory position, turning off the simulation, so the position can no longer change on the server. The problem with this approach is in waking the object — if it’s sitting on dynamic terrain that’s constantly moving, there may be a floating object that should actually be simulating gravity at that moment, but its simulation is turned off. The solution lies in waking the simulation: for dynamic terrain, we collect data on all objects sitting on it (that have physics enabled), and restart the simulation process for them if the terrain has changed.

With this solution, degradation of experience never happens, and in the vast majority of cases it is accurate, also because these objects are rare. One corner case that could occur is a floating object that falls but never touches the ground and so is never woken, but this can be solved with similar tracking of which objects are touching the object — though given how rare this is, and the unnecessary calculations it would add, it was not implemented.

Final implementation — position delta

After these attempts, it was decided to look at the problem from a new angle, since the previous solutions were clearly not scalable and could leak memory.

The final iteration is as follows: the physics object is set as initially network-dormant, so its changes are not sent to any clients.

When an object is constructed, its current position is remembered. When any collision occurs during the game, that body is checked to see if the difference between its position and the old position is greater than some “significant” distance (1 cm) — if so, we wake up the network channel and send the change.

The result is a perfectly synchronised object position, with no jitter, and it’s extremely cheap on the server side. The result is visible at Content/TakeoverPrototype/DestructibleEnvironment/StaticMeshPhysics/MyStaticMeshPhysicsTestV3 and in this recording [96].

Future iterations — dynamic movement replication and Z-axis locking

This algorithm does not scale well if these items move a long distance, since they replicate every time they move a significant distance. To fix this we establish a counter that increments each time we move a significant distance — we’re trying to detect if we’re continuously moving through space, which is what the last iteration failed to scale for. If we’ve triggered 3 times in a row, we assume we’re actively moving, so we activate movement replication and disable our algorithm. If the delta between the last position and current position is smaller than the acceptable distance, we disable movement replication and re-enable our algorithm. This ensures stable physics that scale well when objects are actively moved.

It is possible to further optimise some objects for which we don’t need full physics — for example, if we want objects to fall only along the Z axis, we can constrain them with clamps and a few multicasts (since clamps themselves aren’t replicated). This effect is used by games like Valheim to avoid overly chaotic physical behaviour, and to replicate less data.

It should be noted that an easier solution might be to increase the accuracy of collision detection, but this comes at a significant performance cost.

Continue to Part 5 — Creating a Fully Destructible Networked Environment. The full six-part series and source project are on GitHub.