Part 3 of a six-part series adapted from my master’s thesis, “Creating a First-Person Action Game in Unreal Engine”. Previously: Part 2 — Networked Physics Overview. Full series and source on GitHub.

In this topic we will learn how to set up our multi-server architecture using Unreal Engine 4 and SpatialOS, then work on dynamic interest management and offloading our AI to another server.

A video is available on this topic.

• Dynamic interest management:

• AI offloading to another server, while still allowing for possession:

Introduction

In order to be able to perform more interactions in the simulation, either a multithreading or a multi-server architecture needs to be implemented — this is also suggested by Dave Ratti in a discussion on networking in UE4 [44]. While multithreading would certainly affect server performance, it is not as scalable as a multi-server architecture, as it would still be limited by the number of threads, while the number of servers we can network with each other is theoretically unlimited.

When we talk about a multi-server architecture, we mean the processing of a single simulation by multiple computers — this is also called distributed simulation. This is a separate term from shard-based technologies, where we connect multiple stand-alone simulations.

There are several games that use this architecture, such as Star Citizen, which uses a version of this architecture called server meshing [45]. Furthermore, there is The Elder Scrolls Online with its “Megaserver” technology [46], which is used to scale the number of players, but across multiple worlds. Finally, Mortal Online 2 [47] plans to use it to scale the number of players in one world.

There are several publicly available technologies, one of which, called the Aether Engine [48], has several advantages such as dynamically scaling the number of available servers, but is not currently ready for production.

SpatialOS

This paper will use the (at the time) publicly available SpatialOS [5] technology from Improbable, as it already has a stable and production-ready version.

Fundamental terms

We will now explain the basic terms and concepts related to its function.

Workers [49] perform operations related to the SpatialOS system — they can be server or client workers. The server worker is equal to the classic game server, while the client worker is equal to the classic client instance of the game.

SpatialOS simulation is perceived through three concepts [50]:

• World
• Entities
• Components

The world is the source of canonical truth for the whole simulation. All the data we want to share between the workers must be stored in the entities — these are all objects in the game [51]. All entities consist of components that store data [52].

In short, we write logic to workers who use data from components on entities. This kind of architecture is needed because we want more workers to be able to access the data, without communicating with each other.

This is actually an extension of the design pattern called entity-component-system (ECS) — it is very well mapped to the UE4 system centered around actors, which is favourable for physical simulations, but it should be mentioned that it also works well with system-centered engines such as Unity, which are favourable for AI [33].

Finally, it should be mentioned that a high-bandwidth, low-latency database is used to save the state of the world [5].

Distributed simulation and connected concepts

SpatialOS provides two concepts for simulation distribution:

• Layers
• Spatial load balancing

Layering is a concept in which groups of components are organised by workers who simulate the world [53]. The following figure shows an example where at the top we have a layer that is responsible for the artificial intelligence controller component, and one layer that is in charge of the physics simulation.

Spatial load balancing is the division of one layer according to its spatial or geographical principle, so one worker cares only for components in a certain part of the world [54]. It should be noted that spatial load balancing as a feature is not yet ready for production and is therefore not used in this paper, but its application would require minimal modifications.

We will now explain some of the concepts related to simulation distribution.

Authority is a concept in which only one worker has the right to write component data at a time [55]. This is important for spatial load balancing because there authority often changes — this concept is called handover [56].

Interest is a concept in which one worker requires reading component data [57]. This concept allows clients to receive only information relevant to them.

In the following source, we see an example of an area in the world where the worker is authoritative, marked with a solid line, and the area of its interest, marked with a dotted line — they overlap due to the need for handover of authority over the entity [58].

Persistence [59] is a concept in which data from all components is stored in system images (snapshots) so that the world can be rebuilt later [60].

Limitations and the maximum player count

Operations are an extremely important limitation for the SpatialOS system — an operation is any writing or reading of data [58]. In 2019 a metric of 6 million operations was set [61], resulting in 6000 players — this was achieved with UE4, i.e. position-oriented networking. In real world applications, with the already mentioned game Scavengers, 4138 players were achieved in one place, but with very little interaction, i.e. without weapons and the like. It should be noted that in 2021 a new figure was given, of 250 million operations, resulting in tests with 10,000 players at 30 Hz [40].

Finally, it should be noted that there is currently a limit on purchased servers of 200 players, and this is a metric that will guide us in the design of game and match functionality [62].

Additional useful information

SpatialOS has solved some of the most difficult problems of distributed simulation, such as the distribution of physical simulation — their solution also covers cases where there is an overlap of authority [63].

Furthermore, the problem of large worlds has been solved. This is a problem with the data type used to store the player’s position on the server. The largest possible number with a 32-bit floating point number [64] is 2,147,483,647, so it is only possible to define a world as large as approximately 20 square kilometers. There are methods to alleviate this problem, such as shifting the origin of the world, but they complicate the game development process [65]. This problem is also addressed in Unreal Engine 5 using Large World Coordinates, which removes world size constraints [66] [67], and therefore this problem will not be addressed further in this paper.

World Inspector is a web-based tool for reviewing the state of the SpatialOS world. With it, we can see live what is happening on the running server and all connected workers [68]. It shows all entities, components, workers and their interests, as well as what load they are under.

It is started by clicking on the inspector button.

When implementing SpatialOS in the UE, there are several other terms to pay attention to, i.e. how UE concepts are translated into SpatialOS terms [69]:

Table 1: Terms of the same meaning in UE and SpatialOS [69]

It is important to be familiar with the concepts of remote procedure call (RPC) and networked multicast [70]. A remote procedure call is a function that is called locally, but performed remotely, on another computer. A networked multicast is a type of RPC that is called from the server, and runs on it and on all connected clients.

Finally, it is very important to know how to iterate when creating a game with SpatialOS [71]. As can be seen from the given source, if we have made any changes that would affect replicated properties, we need to restore the schema before placing the game on the server.

Client interest management

In this chapter, we will explain why it is necessary to manage the interest of game clients, and through which methods we can achieve this.

It is recommended that you install SpatialOS from this point forward, following the get started guide. This project uses the build from the 4.25-SpatialOSUnrealGDK-0.11.0 branch, and requires the OWI Advanced Vehicles Plugin to be installed. Take note that some changes are made at the engine level — for example those shown in the Dynamic Interest Management chapter — the project might not work without those.

As a result of positional updates, we can’t download data on all components at once, because that would obviously be too much data — after all, something happening on the completely other side of the map from the player is probably not relevant to them.

Fundamental techniques

Net Cull Distance Squared

This brings us to the first method we can use, which is the previously mentioned Net Cull Distance Squared [72]. It is a sphere of interest placed at the client’s position, expressing interest in an entity. It is defined at the entity or class level in UE4, in the replication menu. Let us now look at one example.

The following figure shows the configuration of the destructible terrain entity, which will be discussed in more detail later. Since we want this entity to replicate at 300 meters, it is necessary to convert it into centimeters (30000 cm) and enter its square, 900000000.0. As mentioned earlier, this is the core of the UE4 network driver — this query is converted into a SpatialOS interest query in the SpatialOS network driver.

A slightly better example is given in the documentation for this method. In the source, the visible distance for players is set at 200 meters, while for vehicles it is set at 400 m, with the player we are observing at the center.

Net Cull Distance Frequency

According to its original implementation in UE4, entities within a given distance are sent to the client at full frequency, which is undesirable due to the volume of networking traffic and the impact on processor performance. Furthermore, it is obvious that entities closer to the client are more important than farther ones, and therefore more positional updates should be received from them. This is usually handled by the accumulator in UE4, but it can be more granularly defined here using Net Cull Distance Frequency [73]. It should be noted that this is a method that, if enabled, applies to all queries, and cannot be configured at the level of a single query.

To enable this method, we must select in UE4: Edit > Project Settings > SpatialGDK for Unreal > Runtime Settings > Interest. Here we check the “Enable Net Cull Distance Frequency” query.

Furthermore, this method needs to be configured. First we define in which distance ratio we want full frequency (Full Frequency Net Cull Distance Ratio), i.e. at a value of 0.33, in full frequency we will get updates for entities that are 1/3 of the set distance. This can be further configured in the Interest Range Frequency Pairs menu. In this example, we set it to get 15 updates per second at 2/3 distance, and 7.5 updates per second at full distance.

Entities we’re always interested in

For some entities, we want to always receive updates, regardless of their position — this is done using a method called “AlwaysInterested”. For this, the UE4 UPROPERTY specifier AlwaysInterested [74] is used — it must be an object reference (AActor or UObject), or have a “Replicated” or “Handover” specifier. An example of this method could be performed over the team scores, for which we always want to know the status.

Actor Interest Component

This is a UE4 component that can be added to any character. It consists of:

• Query lists
• A Net Cull Distance Squared switch

It provides a way to deeply define interests. When a client has a controller with that component, the query list within it defines the data the client receives, and therefore will not use the previously mentioned methods, only this one. It should be noted that one character can have only one such component at a time. So, unlike the Unreal driver, which defines connections from object to user only once, this system allows us to define connections from user to objects multiple times [75].

Queries can be:

• A single constraint
• Multiple constraints connected with logical operators “OR” or “AND”

The following table shows a list of possible constraints:

Table 2: Description of constraints [75]

A component is added to the character by opening the character blueprint, under the component menu, selecting the specified component and pressing the add key.

This component allows us very similar modifications to those made for Battlefield 4, so we can create separate queries for cars, planes and infantry. In the following image, we have configured a component on the Player Controller to receive updates from only those characters (infantry type) that have a “TestTag” component and are within a sphere of 19500 centimeters.

This component was developed because some details of the character are not put into components, but into variables. For example, the team a character belongs to is a simple variable with a team identifier, but we cannot use component variables to identify them in constraints or in a schema. So the “Tag” component solves this problem, because components can be modified in a live environment, added to or removed from any character, and used to identify interests.

Dynamic client interest management

While the actor interest component is very useful, it is not dynamic — it cannot be modified during gameplay. Because of this, for example, it would not be possible to change interest in the enemy team if the client changed teams; furthermore, field-of-view optimisations would also not be possible.

In order to be able to dynamically change the interest of our client worker, it is necessary to first study how interest is initialised. This was a long process of studying the operation of the code itself, and the functionality of each part of the SpatialOS driver was studied in depth. The solution lies in two classes:

• USpatialNetDriver
• USpatialSender

The solution itself looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

PlayerBubbleAnd = CreateDefaultSubobject<UAndConstraint>(TEXT("PlayerBubbleAnd"));
PlayerBubbleComponent = CreateDefaultSubobject<UComponentClassConstraint>(TEXT("PlayerBubbleComponent"));
PlayerBubbleComponent->ComponentClass = UTagComponent::StaticClass();
PlayerBubbleComponent->bIncludeDerivedClasses = false;

RelativeSphere = CreateDefaultSubobject<URelativeSphereConstraint>(TEXT("Sphere1"));
RelativeSphere->Radius = 19500;

PlayerBubbleAnd->Constraints.Empty();
PlayerBubbleAnd->Constraints.Add(PlayerBubbleComponent);
PlayerBubbleAnd->Constraints.Add(RelativeSphere);

ActorInterestComponent->Queries[0].Constraint = PlayerBubbleAnd;

USpatialNetDriver* driver = (Cast<USpatialNetDriver>(GetWorld()->GetNetDriver()));
USpatialSender* sender = driver->Sender;
sender->UpdateInterestComponent(this);

So, on the client worker, we must first define the constraints (“PlayerBubbleAnd”, “PlayerBubbleComponent” and “RelativeSphere”) and add them to the interest component. Furthermore, we need to get the SpatialOS network driver from the world and get a reference to the worker itself — the “sender” variable — where we can call the function UpdateInterestComponent which allows us to update interest. It should be noted that this function is not documented anywhere, and is not used anywhere else. This all has to be done on the server — if we want to trigger the change from the client, we need to do an RPC.

Furthermore, during the execution of these tests, an error was detected in the UComponentClassConstraint type constraint, which did not actually work because the schema was not processed correctly — specifically changes over the SpatialActorChannel and SpatialClassInfoManager classes [76].

FOV-based interest

In order to perform interest queries based on the client’s field of view, we need to create a sphere of interest in front of the client, and move it accordingly — the size of the sphere should depend on the client’s resolution.

The ideal place to implement this is in the definition of the character controller itself, in the GDKPlayerController class. We use the SetControlRotation function — it is called every time a rotation update is received on the server side, which saves us an unnecessary RPC. It should be noted that with this modification we have left the driver system, and it needs to be optimised — we do this with a simple counter that runs the function to change interest only after 60 cycles, since changing interest is a demanding operation for the server’s processor.

The implementation consists of initialising the constraints in the constructor, and updating them in the “QueryTest” function.

We must first calculate the center of the sphere of interest, i.e. where the user is looking. It should be mentioned that this does not take into account the user’s resolution — something that should be updated in further iterations.

After this we have to update the constraint and add it to the interest component — it is a simple spherical and class constraint within the “AND” constraint.

Finally, we call the previously shown update of the interest component. The result is visible in the above video [77]. The image below shows the inspector, with a visible sphere of interest covering the client’s field of view, about 100 meters in front of their location.

When a client turns towards another client, it replicates — the next image shows the process of creating a character.

The following figure shows a fully replicated client.

Future iterations

The idea is to use all the presented methods to create the most efficient replication model. We will divide the interest of each client according to two aspects: field of view and proximity (a sphere around the client). The values are set with the intention of maintaining the best possible experience for the client, so that entities that are closest and in the field of view are refreshed as quickly as possible, similar to Battlefield 4.

The following table gives an overview of the frequency at which an entity will be refreshed depending on its distance in the client’s field of view.

Table 3: Update rate by field

The following table contains the average number of clients per field. We cannot influence these values — they are the result of the movement of clients through the world. This is only an estimate based on gaming experience, assuming the greater the distance the more players there are, and that there are more players in the vicinity of the client than in their field of view.

Table 4: Average number of clients per field

The result is an average of 1488 other players in the field of view and in the vicinity of the client. When we multiply these two tables, we get the total number of interactions per client.

Table 5: Number of interactions per field

We get a total of 3296 interactions per client. This number can be further reduced by updating the enemy team under full load given in the first table, while our own team is updated at half the rate. This is done with the previously implemented “Tag” system.

(Enemy Updates * 0.5 + Friend Updates * 0.5) * Interactions

(1 * 0.5 + 0.5 * 0.5) * 3296 = 2472

The result is 2472 interactions per client. If we take into account the possible 6 million operations per second, we can reach the theoretical maximum number of players given this fidelity.

(6 000 000 operations) / (2472 operations per player) = 2427 players

It should be mentioned that this prediction does not take into account projectiles, physical objects etc. On the other hand, operations are not an ideal metric because they represent the average possible number of operations from Improbable’s testing — which is not their fault, since it is generally difficult to estimate the average fidelity of environments, and it also depends on the number of servers available and their performance.

Further optimisation is possible, where we would dynamically change the update rate according to the number of players or physical objects with which an individual player interacts, but one could argue the accumulator does a good enough job on its own. It should be mentioned that this rate is expressed in integers, i.e. it is not possible to refresh less than once per second (between 0 and 1).

Creating an AI layer

Artificial intelligence is an excellent candidate for placement on its own layer, as it is very expensive to process, but latency tolerant [78].

In order to configure our world (map) with this layer, we need to create a configuration — a new class that inherits SpatialMultiWorkerSettings, in this case at the path Content/TakeoverPrototype/OffloadingTakeover. We can now configure our world.

A tutorial on this is available in the SpatialOS documentation on offloading.

We will configure it in two layers. The first layer, “DefaultLayer”, will be configured to handle all classes that could appear in the game, i.e. all Actor classes — for now, due to the unavailability of spatial load balancing, we use the single worker strategy (“SingleWorkerStrategy”). The second layer, called “AiTakeover”, will be used to process artificial intelligence, and will include the AI spawner class (“AI_Spawner”) and the AI controller class (“NPC_Controller_Takeover”), with the aforementioned strategy.

After compiling this configuration, we can add it to the world (map). Open the map you want to configure, available at Content/TakeoverPrototype/Takeover_Medium, and switch to the “World Settings” menu — if it is not visible, it can be turned on via Window > World Settings.

Here we enable layers using “Enable Multi Worker”, and select our configuration under “Multi Worker Settings Class”, where we put the previously created class “OffloadingTakeover”.

Congratulations, you are now running a multi-server architecture.

Spawning AI

In order to be able to keep track of AI, we will use the previously mentioned spawner class, “AI_Spawner”. It takes care of all the functions on the server side — creating the AI, taking control over it and giving commands.

For the purposes of this project, AI spawning is done at the request of the client, by pressing the “B” key, which is handled in the “BP_GDK_PlayerController” class. Here, via RPC on the server side, we find all the spawners (there is only one) and call its spawn function. A player ID variable is also sent so that we can uniquely identify them.

The spawner calls a function which finds the player and creates the AI in the world.

Furthermore, a controller is created for the AI, which then controls it, and finally a variable is set to control the behaviour of the AI in the decision tree, so that the AI knows who to follow.

The decision tree is very simple — the assigned character is followed.

Taking control over AI

Artificial intelligence is only in the game to improve its flow, not to directly increase interactivity, because interaction with AI has much less meaning — other players are responsible for the real interaction. The flow of the game is improved by players taking control over the AI.

In most other games, death means waiting for respawn — this can take from 30+ seconds in Squad, to a few seconds in Call of Duty. All this time is actually spent on useless waiting; people want to play the game, not stare at the respawn menu. There is no dead time in our game — we can return to the game immediately by taking control of the AI.

We achieve this through layer design — notice that we only put the AI controller on the layer, not its character. It is important that the character is on the same layer as the client controller, because this way we avoid unnecessary internal communications between layers. Although these are actually very secure and happen almost immediately [79], it is better to put this risk on the side of the AI controller than on the player, because their experience is obviously more important to us.

The process itself is again started by the user, who by pressing the “V” key sends an RPC to the server, where it again searches for the appropriate AI spawner and activates its function to take control of the AI.

In the spawner, we again find the appropriate player who requested the function.

When we find them, we find the first available AI character, un-possess it, and do the same with the player.

In the end we simply swap possession, so the player now controls the character formerly controlled by the AI, while the AI controls the character formerly controlled by the player.

The following figure shows that control of the AI has been taken, and that the client’s character (red) is now in control of the AI’s former character.

Future AI iterations

In future iterations, the AI should be upgraded with commands other than following players, and organised into squads. These would be commands for:

• Move to the marked position
• Stop
• Defend this position
• Attack this position
• Build
• Retreat
• Recruit (re-create dead squad members)

Some of these commands were prototyped using a decision tree, but this is a complex way of implementing orders, so the approach was abandoned.

Possible modes of execution are via state machines, for more stable behaviour and easier programming. Furthermore, this can be done using a Hierarchical Task Network [80], with even deeper behaviours.

Finally, depending on the game design, it is possible to open behaviour modification to the client via a system similar to the one in Dragon Age: Origins, called the Advanced Tactics System [81], where the player actually programs how AI behaviour takes place in the game.

Conclusion

We have now created a solid foundation of performance that will ensure we can add more fidelity to our simulation. Next, we will take a look at vehicles as one of the most complex networking cases and take a closer look at physics, to ensure they are ready for a fully dynamic environment. Thank you for following along.

Continue to Part 4 — Advance Networked Physics Features. The full six-part series and source project are on GitHub.