Input prediction and rollback

Overview

coherence input queues are backed by a rolling buffer of inputs transmitted between the clients. This buffer can be used to build a fully deterministic simulation with a client side-prediction, rollback, and input delay. This game networking model is often called the GGPO (Good Game Peace Out).

Input delay allows for a smooth, synchronized netplay with almost no negative effect on the user experience. The way it works is input is scheduled to be processed X frames in the future. Consider a fighting game scenario with two players. At frame 10 Player A presses a kick button that is scheduled to be executed at frame 13. This input is immediately sent to Player B. With a decent internet connection, there's a big chance that Player B will receive that input even before his frame 13. Thanks to this the simulation is always in sync and can progress steadily.

Prediction is used to run the simulation forward even in the absence of inputs from other players. Consider the scenario from the previous paragraph - what if Player B doesn't receive the input on time? The answer is very simple - we just assume that the input state hasn't changed and progress with the simulation. As it turns out this assumption is valid most of the time.

Rollback is used to correct the simulation when our predictions turn out wrong. The game keeps historical states of the game for past frames. When an input is received for a past simulation frame the system checks whether it matches the input prediction made at that frame. If it does we don't have to do anything (the simulation is correct up to that point). If it doesn't match, however, we need to restore the simulation state to the last known valid state (last frame which was processed with non-predicted inputs). After restoring the state we re-simulate all frames up to the current one, using the fresh inputs.

Determinism

In a deterministic simulation, given the same set of inputs and a state we are guaranteed to receive the same output. In other words, the simulation is always predictable. Deterministic simulation is a key part of the GGPO model, as well as a lockstep model because it lets us run exactly the same simulation on multiple clients without a need for synchronizing big and complex states.

Implementing a deterministic simulation is a non-trivial task. Even the smallest divergence in simulation can lead to a completely different game outcome. This is usually called a desync. Here's a list of common determinism pitfalls that have to be avoided:

• Using Update to run the simulation (every player might run at a different frame rate)

• Using coroutines, asynchronous code, or system time in a way that affects the simulation (anything time-sensitive is almost guaranteed to be non-deterministic)

• Using Unity physics (it is non-deterministic)

• Using random numbers generator without prior seed synchronization

• Non-symmetrical processing (e.g. processing players by their spawn order which might be different for everyone)

• Relying on a floating point across different platforms, compilations or processor types

Quickstart guide

We'll create a simple, deterministic simulation using provided utility components.

This is the recommended way of using input queues since it greatly reduces the implementation complexity and should be sufficient for most projects. If you'd prefer to have full control over the input code feel free to use theCoherenceInput and InputBuffer directly.

Our simulation will synchronize the movement of multiple clients, using the rollback and prediction in order to cover for the latency.

Player implementation

Start by creating a Player component that derives from the CoherenceClientConnection. Our Player represents a session participant and will be automatically spawned for each player that connects to the server. It will also be responsible for handling inputs using the CoherenceInput component.

When building an input-based simulation it is important to use global components like CoherenceClientConnection, that is, ones that are not subject to the live query. Objects that might disappear or change based on the client-to-client distance are likely to cause simulation divergence leading to a desync.

Our Player code looks as follows:

using Coherence.Toolkit;
using UnityEngine;

[RequireComponent(typeof(CoherenceSync))]
[RequireComponent(typeof(CoherenceInput))]
public class Player : CoherenceClientConnection
{
    // Movement speed in units/sec
    public float Speed = 5f;
    
    private CoherenceInput input;

    private void Awake()
    {
        input = GetComponent<CoherenceInput>();
    }

    // Retrieves the "movement" input state for a given frame
    public Vector2 GetMovement(long frame)
    {
        return input.GetAxisState("Mov", frame);
    }

    // Sets the "movement" state for the current frame
    public void SetMovement()
    {
        Vector2 movement = new Vector2(Input.GetAxis("Horizontal"), Input.GetAxis("Vertical")).normalized;
        input.SetAxisState("Mov", movement);
    }
}

The GetMovement and SetMovement will be called by our "central" simulation code. Now that we have our Player defined let's create a prefab for it:

A couple of things to note:

• A Mov axis has been added to the CoherenceInput which will let us sync the movement input state

• Unlike in the server-side input queues our simulation uses client-to-client communication, meaning each client is responsible for its entity and sending inputs to other clients. To ensure such behavior set the CoherenceSync > Simulation and Interpolation > Simulation Type to Client Side

• In the deterministic simulation, it is our code that is responsible for producing deterministic output on all clients. This means that the automatic transform position syncing is no longer desirable. To turn it off check the CoherenceSync > Simulation and Interpolation > Manual Position Update

• In order for inputs to be processed in a deterministic way, we need to use the fixed simulation frames. Tick the CoherenceInput > Use Fixed Simulation Frames checkbox

• Make sure to use the baked mode - inputs do not work in the reflection mode

Since our player is using the CoherenceClientConnection we must set it as the connection prefab in the CoherenceMonoBridge and enable the global query:

State implementation

Before we move on to the simulation, we need to define our simulation state which is a key part of the rollback system. The simulation state should contain all the information required to "rewind" the simulation in time. For example, in a fighting game that would be a position of all players, their health, and perhaps a combo gauge level. In a shooting game, this could be player positions, their health, ammo, and a map objective progression.

In the example we're building player position is the only state. We need to store it for every player:

using UnityEngine;

public struct SimulationState
{
    public Vector3[] PlayerPositions;
}

The state above assumes the same number and order of players in the simulation. The order is guaranteed by the CoherenceInputSimulation, however, handling a variable number of clients is up to the developer.

Simulation implementation

Simulation code is where all the logic should happen, including applying inputs and moving our Players:

using Coherence.Toolkit;
using UnityEngine;

public class Simulation : CoherenceInputSimulation<SimulationState, Player>
{
    protected override void SetInputs(Player client)
    {
        client.SetMovement();
    }

    protected override void Simulate(long simulationFrame)
    {
        foreach (Player player in AllClients)
        {
            var movement = (Vector3)player.GetMovement(simulationFrame);
            player.transform.position += movement * player.Speed * FixedTimeStep;
        }
    }

    protected override void Rollback(long toFrame, SimulationState state)
    {
        for (var i = 0; i < AllClients.Count; i++)
        {
            AllClients[i].transform.position = state.PlayerPositions[i];
        }
    }

    protected override SimulationState CreateState()
    {
        var simulationState = new SimulationState { PlayerPositions = new Vector3[AllClients.Count] };
        for (var i = 0; i < AllClients.Count; i++)
        {
            simulationState.PlayerPositions[i] = AllClients[i].transform.position;
        }

        return simulationState;
    }

    protected override void OnClientJoined(Player client)
    {
        SimulationEnabled = AllClients.Count >= 2;
        if (SimulationEnabled)
        {
            // Lets us rejoin the same simulation without restarting the app.
            StateStore.Clear();
        }
    }

    protected override void OnClientLeft(Player client)
    {
        SimulationEnabled = AllClients.Count >= 2;
    }
}

• SetInputs is called by the system when it's time for our local Player to update its input state using the CoherenceInput

• Simulate is called when it's time to simulate a given frame. It is also called during frame re-simulation after misprediction - don't worry though, the complex part is handled by the CoherenceInputSimulation internals - all you need to do in this method is apply inputs from the CoherenceInput to run the simulation

• Rollback is where we need to set the simulation state back to how it was at a given frame. The state is already provided in the state parameter, we just need to apply it

• CreateState is where we create a snapshot of our simulation so it can be used later in case of rollback

• OnClientJoined and OnClientLeft are optional callbacks. We use them here to start and stop the simulation depending on the number of clients

The SimulationEnabled is set to 'false' by default. That's because in a real-world scenario the simulation should start only after all clients have agreed for it to start, on a specific frame chosen, for example, by the host.

Starting the simulation on a different frame for each client is likely to cause a desync (as well as joining in the middle of the session, without prior simulation state synchronization). Simulation start synchronization is however out of the scope of this guide so in our simplified example we just assume that clients don't start moving immediately after joining.

Pause handling

There's a limit to how many frames can be predicted by the clients. This limit is controlled by the CoherenceInput.InputBufferSize. When clients try to predict too many frames into the future (more frames than the size of the buffer) the simulation will issue a pause. This pause affects only the local client. As soon as the client receives enough inputs to run another frame the simulation will resume.

To get notified about the pause use the OnPauseChange(bool isPaused) method from the CoherenceInputSimulation:

public class Simulation : CoherenceInputSimulation<SimulationState, Player>
{
    // ... Simulation implementation ...
    
    protected override void OnPauseChange(bool isPaused)
    {
        PauseScreen.SetActive(isPaused);
    }
}

This can be used for example to display a pause screen the informs user about a bad internet connection.

To recover from the time gap created by the pause the client will automatically speed up the simulation. The time scale change is gradual and in the case of a small frame gap, can be unnoticeable. If a manual control over the timescale is desired set the CoherenceMonoBridge.controlTimeScale flag to 'false'.

Debugging

The CoherenceInputSimulation has a built-in debugging utility that collects various information about the input simulation on each frame. This data can prove extremely helpful in finding a simulation desync point.

The CoherenceInputDebugger can be used outside the CoherenceInputSimulation. It does however require the CoherenceInputManager which can be retrieved through the CoherenceMonoBridge.InputManager property.

Setup

Since debugging might induce a non-negligible overhead it is turned off by default. To turn it on add, a COHERENCE_INPUT_DEBUG scripting define:

From that point, all the debugging information will be gathered. The debug data is dumped to a JSON file as soon as the client disconnects. The file can be located under a root directory of the executable (in case of Unity Editor the project root directory) under the following name: inputDbg_<ClientId>.json, where <ClientId> is the CoherenceClientConnection.ClientId of the local client.

Data handling behaviour can be overridden by setting the CoherenceInputDebugger.OnDump delegate, where the string parameter is a JSON dump of the data.

The debugger is available as a property in the simulation base class: CoherenceInputSimulation.Debugger. Most of the debugging data is recorded automatically, however, the user is free to append any arbitrary information to a frame debug data, as long as it is JSON serializable. This is done by using the CoherenceInputDebugger.AddEvent method:

public class Simulation : CoherenceInputSimulation<SimulationState, Player>
{
    // ... Simulation implementation ...
    
    protected override void SetInputs(Player client)
    {
        Debugger.AddEvent("myCustomEvent", new
        {
            Data = "my custom data",
            UnityFrameCount = Time.frameCount
        });
        client.SetMovement();
    }
}

Since the simulation can span an indefinite amount of frames it might be wise to limit the number of debug frames kept by the debugging tool (it's unlimited by default). To do this use the CoherenceInputDebugger.FramesToKeep property. For example, setting it to 1000 will instruct the debugger to keep only the latest 1000 frames worth of debugging information in the memory.

Serialization

Since the debugging tool uses JSON as a serialization format, any data that is part of the debug dump must be JSON-serializable. An example of this is the simulation state. The simulation state from the quickstart example is not JSON serializable by default, due to Unity's Vector3 that doesn't serialize well out of the box. To fix this we need to give JSON serializer a hint:

public struct SimulationState : IHashable
{
    [JsonProperty(ItemConverterType = typeof(UnityVector3Converter))]
    public Vector3[] PlayerPositions { get; set; }
}

With the JsonProperty attribute, we can control how a given field/property/class will be serialized. In this case, we've instructed the JSON serializer to use the custom UnityVector3Converter for serializing the vectors.

You can write your own JSON converters using the example found here. For information on the Newtonsoft JSON library that we use for serialization check here.

Comparing debug data

To find a problem in the simulation, we can compare the debug dumps from multiple clients. The easiest way to find a divergence point is to search for a frame where the hash differs for one or more of the clients. From there one can inspect the inputs and simulation states from previous frames to find the source of the problem.

Here's the debug data dump example for one frame:

"32625635555": {
    "Frame": 32625635555,
    "AckFrame": 32625635556,
    "ReceiveFrame": 32625635556,
    "AckedAt": 32625635555,
    "MispredictionFrame": -1,
    "Hash": "3fa00ad32cee971e43ee4a3206dc79f0",
    "Time": "15:48:36.977",
    "InitialState": {
      "PlayerPositions": [
        "0.2494998,3.992,0",
        "0.2494998,-2.086163E-07,0",
        "-2.086163E-07,0.7484999,0",
        "0.4989998,3.992,0"
      ]
    },
    "InitialInputs": {
      "153470363": "Mov:float2(-0.998f, 0f)",
      "322232370": "Mov: float2(0.998f, 0f)",
    },
    "InputBufferStates": {
      "153470363": {
        "LastFrame": 32625635557,
        "LastSentFrame": 32625635557,
        "LastReceivedFrame": -1,
        "LastAcknowledgedFrame": -1,
        "MispredictionFrame": null,
        "QueueCount": 0,
        "ShouldPause": false
      },
      "322232370": {
        "LastFrame": 32625635556,
        "LastSentFrame": -1,
        "LastReceivedFrame": 32625635556,
        "LastAcknowledgedFrame": 32625635556,
        "MispredictionFrame": null,
        "QueueCount": 0,
        "ShouldPause": false
      }
    },
    "Events": [
      {
        "Event": "InputSent",
        "Time": "15:48:36.977",
        "Data": {
          "ClientId": 153470363,
          "SentFrame": 32625635558,
          "Input": "Mov:float2(0f, -0.998f)"
        }
      },
      {
        "Event": "InputReceived",
        "Time": "15:48:36.978",
        "Data": {
          "ClientId": 322232370,
          "RecvFrame": 32625635557,
          "Input": "Mov: float2(0.998f, 0f)"
        }
      }
    ]
  },

Explanation of the fields:

• Frame - frame of this debug data

• AckFrame - the common acknowledged frame, i.e. the lowest frame for which inputs from all clients have been received and are known to be valid (not mispredicted)

• ReceiveFrame - the common received frame, i.e. the lowest frame for which inputs from all clients have been received

• AckedAt - a frame at which this frame has been acknowledged, i.e. set as known to be valid (not mispredicted)

• MispredictionFrame - a frame that is known to be mispredicted, or -1 if there's no misprediction

• Hash - hash of the simulation state. Available only if the simulation state implements the IHashable interface

• Initial state - the original simulation state at this frame, i.e. a one before rollback and resimulation

• Initial inputs - original inputs at this frame, i.e. ones that were used for the first simulation of this frame

• Updated state - the state of the simulation after rollback and resimulation. Available only in case of rollback and resimulation

• Updated inputs - inputs after being corrected (post misprediction). Available only in case of rollback and resimulation

• Input buffer states - dump of the input buffer states for each client. For details on the fields see the InputBuffer code documentation

• Events - all debug events registered in this frame

State hash

One of the things that are stored as part of the debugging information is the simulation state. In the case of a complex state searching for a difference across multiple clients and hundreds of frames can quickly become tedious. To simplify the problem we can use the hash calculation feature of the input debugger.

Every state simulation class/struct that implements the IHashable interface will have its hash automatically calculated and stored as part of the debugging information. The example of IHashable implementation:

public struct SimulationState : IHashable
{
    [JsonProperty(ItemConverterType = typeof(UnityVector3Converter))]
    public Vector3[] PlayerPositions { get; set; }

    public Hash128 ComputeHash()
    {
        return Hash128.Compute(PlayerPositions);
    }
}

Configuration

Input buffer

There are two main variables which affect the behaviour of the InputBuffer:

• Input buffer size - the size of the buffer determines how far into the future the input system is allowed to predict. The bigger the size, the more frames can be predicted without running into a pause. Note that the further we predict, the more unexpected the rollback can be for the player. The InitialBufferSize value can be set directly in code however it must be done before the Awake of the baked component, which might require a script execution order configuration.

• Input buffer delay - dictates how many frames must pass before applying an input. In other words, it defines how "laggy" the input is. The higher the value, the less likely clients are going to run into prediction (because a "future" input is sent to other clients), but the more unresponsive the game might feel. This value can be changed freely at runtime, even during a simulation (it is however not recommended due to inconsistent input feeling).

The other two options are:

• Disconnect on time reset - if set to 'true' the input system will automatically issue a disconnect on an attempt to resync time with the server. This happens when the client's connection was so unstable that frame-wise it drifted too far away from the server. In order to recover from that situation, the client performs an immediate "jump" to what it thinks is the actual server frame. There's no easy way to recover from such "jump" in the deterministic simulation code so the advised action is to simply disconnect.

• Use fixed simulation frames - if set to 'true' the input system will use the IClient.ClientFixedSimulationFrame frame for simulation - otherwise the IClient.ClientSimulationFrame is used. Setting this to 'true' is recommended for the deterministic simulation.

Fixed frame rate

The fixed network update rate is based on the Fixed Timestep configured through the Unity project settings:

To know the exact fixed frame number that is executing at any given moment use the IClient.ClientFixedSimulationFrame or CoherenceInputSimulation.CurrentSimulationFrame property.