Input prediction and rollback

Overview

coherence input queues are backed by a rolling buffer of inputs transmitted between the clients. This buffer can be used to build a fully deterministic simulation with a client side-prediction, rollback, and input delay. This game networking model is often called the GGPO (Good Game Peace Out).

Input delay allows for a smooth, synchronized netplay with almost no negative effect on the user experience. The way it works is input is scheduled to be processed X frames in the future. Consider a fighting game scenario with two players. At frame 10 Player A presses a kick button that is scheduled to be executed at frame 13. This input is immediately sent to Player B. With a decent internet connection, there's a big chance that Player B will receive that input even before his frame 13. Thanks to this the simulation is always in sync and can progress steadily.

Prediction is used to run the simulation forward even in the absence of inputs from other players. Consider the scenario from the previous paragraph - what if Player B doesn't receive the input on time? The answer is very simple - we just assume that the input state hasn't changed and progress with the simulation. As it turns out this assumption is valid most of the time.

Rollback is used to correct the simulation when our predictions turn out wrong. The game keeps historical states of the game for past frames. When an input is received for a past simulation frame the system checks whether it matches the input prediction made at that frame. If it does we don't have to do anything (the simulation is correct up to that point). If it doesn't match, however, we need to restore the simulation state to the last known valid state (last frame which was processed with non-predicted inputs). After restoring the state we re-simulate all frames up to the current one, using the fresh inputs.

Determinism

In a deterministic simulation, given the same set of inputs and a state we are guaranteed to receive the same output. In other words, the simulation is always predictable. Deterministic simulation is a key part of the GGPO model, as well as a lockstep model because it lets us run exactly the same simulation on multiple clients without a need for synchronizing big and complex states.

Implementing a deterministic simulation is a non-trivial task. Even the smallest divergence in simulation can lead to a completely different game outcome. This is usually called a desync. Here's a list of common determinism pitfalls that have to be avoided:

• Using Update to run the simulation (every player might run at a different frame rate)

• Using coroutines, asynchronous code, or system time in a way that affects the simulation (anything time-sensitive is almost guaranteed to be non-deterministic)

• Using Unity physics (it is non-deterministic)

• Using random numbers generator without prior seed synchronization

• Non-symmetrical processing (e.g. processing players by their spawn order which might be different for everyone)

• Relying on floating point numbers across different platforms, compilations or processor types

Quickstart guide

We'll create a simple, deterministic simulation using provided utility components.

Our simulation will synchronize the movement of multiple clients, using the rollback and prediction in order to cover for the latency.

Player implementation

Start by creating a Player component and a prefab for it. We'll use the client connection system to make our Player represent a session participant and automatically spawn the selected prefab for each player that connects to the server. The Player will also be responsible for handling inputs using the CoherenceInput component.

Our Player code looks as follows:

using Coherence.Toolkit;
using UnityEngine;

[RequireComponent(typeof(CoherenceSync))]
[RequireComponent(typeof(CoherenceInput))]
public class Player : MonoBehaviour
{
    // Movement speed in units/sec
    public float Speed = 5f;
    
    private CoherenceInput input;

    private void Awake()
    {
        input = GetComponent<CoherenceInput>();
    }

    // Retrieves the "movement" input state for a given frame
    public Vector2 GetMovement(long frame)
    {
        return input.GetAxisState("Mov", frame);
    }

    // Sets the "movement" state for the current frame
    public void SetMovement()
    {
        Vector2 movement = new Vector2(Input.GetAxis("Horizontal"), Input.GetAxis("Vertical")).normalized;
        input.SetAxisState("Mov", movement);
    }
}

The GetMovement and SetMovement will be called by our "central" simulation code. Now that we have our Player defined let's prepare a prefab for it. Create a game object and attach the Player component to it. Using the CoherenceSync inspector create a prefab. The inspector view for our prefab should look as follows:

A couple of things to note:

• A Mov axis has been added to the CoherenceInput which will let us sync the movement input state

• Unlike in the server-side input queues our simulation uses client-to-client communication, meaning each client is responsible for its entity and sending inputs to other clients. To ensure such behavior set the CoherenceSync > Simulation and Interpolation > Simulation Type to Client Side

• In the deterministic simulation, it is our code that is responsible for producing deterministic output on all clients. This means that the automatic transform position syncing is no longer desirable. To turn it off check the CoherenceSync > Manual Position Update

• In order for inputs to be processed in a deterministic way, we need to use the fixed simulation frames. Tick the CoherenceInput > Use Fixed Simulation Frames checkbox

• Make sure to use the baked mode (CoherenceInput > Use Baked Script) - inputs do not work in the reflection mode

Since our player is the base of the client connection we must set it as the connection prefab in the CoherenceMonoBridge and enable the global query:

State implementation

Before we move on to the simulation, we need to define our simulation state which is a key part of the rollback system. The simulation state should contain all the information required to "rewind" the simulation in time. For example, in a fighting game that would be a position of all players, their health, and perhaps a combo gauge level. In a shooting game, this could be player positions, their health, ammo, and a map objective progression.

In the example we're building player position is the only state. We need to store it for every player:

using UnityEngine;

public struct SimulationState
{
    public Vector3[] PlayerPositions;
}

Simulation implementation

Simulation code is where all the logic should happen, including applying inputs and moving our Players:

using Coherence.Toolkit;
using UnityEngine;

public class Simulation : CoherenceInputSimulation<SimulationState>
{
    protected override void SetInputs(CoherenceClientConnection client)
    {
        var player = client.GameObject.GetComponent<Player>();
        player.SetMovement();
    }

    protected override void Simulate(long simulationFrame)
    {
        foreach (CoherenceClientConnection client in AllClients)
        {
            var player = client.GameObject.GetComponent<Player>();
            var movement = (Vector3)player.GetMovement(simulationFrame);
            player.transform.position += movement * player.Speed * FixedTimeStep;
        }
    }

    protected override void Rollback(long toFrame, SimulationState state)
    {
        for (var i = 0; i < AllClients.Count; i++)
        {
            Transform playerTransform = AllClients[i].GameObject.transform;
            playerTransform.position = state.PlayerPositions[i];
        }
    }

    protected override SimulationState CreateState()
    {
        var simulationState = new SimulationState { PlayerPositions = new Vector3[AllClients.Count] };
        for (var i = 0; i < AllClients.Count; i++)
        {
            Transform playerTransform = AllClients[i].GameObject.transform;
            simulationState.PlayerPositions[i] = playerTransform.position;
        }

        return simulationState;
    }

    protected override void OnClientJoined(CoherenceClientConnection client)
    {
        SimulationEnabled = AllClients.Count >= 2;
        if (SimulationEnabled)
        {
            // Lets us rejoin the same simulation without restarting the app.
            StateStore.Clear();
        }
    }

    protected override void OnClientLeft(CoherenceClientConnection client)
    {
        SimulationEnabled = AllClients.Count >= 2;
    }
}

• SetInputs is called by the system when it's time for our local Player to update its input state using the CoherenceInput

• Simulate is called when it's time to simulate a given frame. It is also called during frame re-simulation after misprediction - don't worry though, the complex part is handled by the CoherenceInputSimulation internals - all you need to do in this method is apply inputs from the CoherenceInput to run the simulation

• Rollback is where we need to set the simulation state back to how it was at a given frame. The state is already provided in the state parameter, we just need to apply it

• CreateState is where we create a snapshot of our simulation so it can be used later in case of rollback

• OnClientJoined and OnClientLeft are optional callbacks. We use them here to start and stop the simulation depending on the number of clients

As a final step attach the Simulation script to the MonoBridge object on scene and link the MonoBridge back to the Simulation:

That's it! Once you build a client executable you can verify that the simulation works by connecting two clients to the replication server. Move one of the clients using arrow keys while observing the movement being synced on the other one.

Input in fixed network update

Due to the FixedNetworkUpdate running at different (usually lower) rate than Unity's Update loop, polling inputs using the functions like Input.GetKeyDown is susceptible to a input loss, i.e. keys that were pressed during the Update loop might not show up as pressed in the FixedNetworkUpdate.

To illustrate why this happens consider the following scenario: given Update running five times for each network FixedNetworkUpdate, if we polled inputs from the FixedNetworkUpdate there's a chance that an input was fully processed within the five Updates in-between FixedNetworkUpdates, i.e. a key was "down" on the first Update, "pressed" on the second, and "up" on a third one.

To prevent this issue from occurring you can use the FixedUpdateInput class:

[RequireComponent(typeof(CoherenceSync))]
[RequireComponent(typeof(CoherenceInput))]
public class Player: MonoBehaviour
{
    private FixedUpdateInput fixedUpdateInput;
    private CoherenceInput input;

    private void Awake()
    {
        var coherenceSync = GetComponent<CoherenceSync>();
        fixedUpdateInput = coherenceSync.MonoBridge.FixedUpdateInput;
        input = coherenceSync.Input;
    }

    // Retrieves the "jump" input state for a given frame
    public bool GetJump(long frame)
    {
        return input.GetButtonState("Jump", frame);
    }

    // Sets the "jump" state for the current frame
    public void SetJump()
    {
        bool hasJumped = fixedUpdateInput.GetKey(KeyCode.Space);
        input.SetButtonState("Jump", hasJumped);
    }
}

The FixedUpdateInput works by sampling inputs at Update and prolonging their lifetime to the network FixedNetworkUpdate so they can be processed correctly there. For our last example that would mean "down" & "pressed" registered in the first FixedNetworkUpdate after the initial five updates, followed by an "up" state in the subsequent FixedNetworkUpdate.

Pause handling

There's a limit to how many frames can be predicted by the clients. This limit is controlled by the CoherenceInput.InputBufferSize. When clients try to predict too many frames into the future (more frames than the size of the buffer) the simulation will issue a pause. This pause affects only the local client. As soon as the client receives enough inputs to run another frame the simulation will resume.

To get notified about the pause use the OnPauseChange(bool isPaused) method from the CoherenceInputSimulation:

public class Simulation : CoherenceInputSimulation<SimulationState>
{
    // ... Simulation implementation ...
    
    protected override void OnPauseChange(bool isPaused)
    {
        PauseScreen.SetActive(isPaused);
    }
}

This can be used for example to display a pause screen the informs user about a bad internet connection.

Debugging

The CoherenceInputSimulation has a built-in debugging utility that collects various information about the input simulation on each frame. This data can prove extremely helpful in finding a simulation desync point.

Setup

Since debugging might induce a non-negligible overhead it is turned off by default. To turn it on add, a COHERENCE_INPUT_DEBUG scripting define:

From that point, all the debugging information will be gathered. The debug data is dumped to a JSON file as soon as the client disconnects. The file can be located under a root directory of the executable (in case of Unity Editor the project root directory) under the following name: inputDbg_<ClientId>.json, where <ClientId> is the CoherenceClientConnection.ClientId of the local client.

Data handling behaviour can be overridden by setting the CoherenceInputDebugger.OnDump delegate, where the string parameter is a JSON dump of the data.

The debugger is available as a property in the simulation base class: CoherenceInputSimulation.Debugger. Most of the debugging data is recorded automatically, however, the user is free to append any arbitrary information to a frame debug data, as long as it is JSON serializable. This is done by using the CoherenceInputDebugger.AddEvent method:

public class Simulation : CoherenceInputSimulation<SimulationState>
{
    // ... Simulation implementation ...
    
    protected override void SetInputs(CoherenceClientConnection client)
    {
        Debugger.AddEvent("myCustomEvent", new
        {
            Data = "my custom data",
            UnityFrameCount = Time.frameCount
        });
        // ... movement code ...
    }
}

Since the simulation can span an indefinite amount of frames it might be wise to limit the number of debug frames kept by the debugging tool (it's unlimited by default). To do this use the CoherenceInputDebugger.FramesToKeep property. For example, setting it to 1000 will instruct the debugger to keep only the latest 1000 frames worth of debugging information in the memory.

Serialization

Since the debugging tool uses JSON as a serialization format, any data that is part of the debug dump must be JSON-serializable. An example of this is the simulation state. The simulation state from the quickstart example is not JSON serializable by default, due to Unity's Vector3 that doesn't serialize well out of the box. To fix this we need to give JSON serializer a hint:

public struct SimulationState : IHashable
{
    [JsonProperty(ItemConverterType = typeof(UnityVector3Converter))]
    public Vector3[] PlayerPositions { get; set; }
}

With the JsonProperty attribute, we can control how a given field/property/class will be serialized. In this case, we've instructed the JSON serializer to use the custom UnityVector3Converter for serializing the vectors.

Comparing debug data

To find a problem in the simulation, we can compare the debug dumps from multiple clients. The easiest way to find a divergence point is to search for a frame where the hash differs for one or more of the clients. From there one can inspect the inputs and simulation states from previous frames to find the source of the problem.

Here's the debug data dump example for one frame:

"32625635555": {
    "Frame": 32625635555,
    "AckFrame": 32625635556,
    "ReceiveFrame": 32625635556,
    "AckedAt": 32625635555,
    "MispredictionFrame": -1,
    "Hash": "3fa00ad32cee971e43ee4a3206dc79f0",
    "Time": "15:48:36.977",
    "InitialState": {
      "PlayerPositions": [
        "0.2494998,3.992,0",
        "0.2494998,-2.086163E-07,0",
        "-2.086163E-07,0.7484999,0",
        "0.4989998,3.992,0"
      ]
    },
    "InitialInputs": {
      "153470363": "Mov:float2(-0.998f, 0f)",
      "322232370": "Mov: float2(0.998f, 0f)",
    },
    "InputBufferStates": {
      "153470363": {
        "LastFrame": 32625635557,
        "LastSentFrame": 32625635557,
        "LastReceivedFrame": -1,
        "LastAcknowledgedFrame": -1,
        "MispredictionFrame": null,
        "QueueCount": 0,
        "ShouldPause": false
      },
      "322232370": {
        "LastFrame": 32625635556,
        "LastSentFrame": -1,
        "LastReceivedFrame": 32625635556,
        "LastAcknowledgedFrame": 32625635556,
        "MispredictionFrame": null,
        "QueueCount": 0,
        "ShouldPause": false
      }
    },
    "Events": [
      {
        "Event": "InputSent",
        "Time": "15:48:36.977",
        "Data": {
          "ClientId": 153470363,
          "SentFrame": 32625635558,
          "Input": "Mov:float2(0f, -0.998f)"
        }
      },
      {
        "Event": "InputReceived",
        "Time": "15:48:36.978",
        "Data": {
          "ClientId": 322232370,
          "RecvFrame": 32625635557,
          "Input": "Mov: float2(0.998f, 0f)"
        }
      }
    ]
  },

Explanation of the fields:

• Frame - frame of this debug data

• AckFrame - the common acknowledged frame, i.e. the lowest frame for which inputs from all clients have been received and are known to be valid (not mispredicted)

• ReceiveFrame - the common received frame, i.e. the lowest frame for which inputs from all clients have been received

• AckedAt - a frame at which this frame has been acknowledged, i.e. set as known to be valid (not mispredicted)

• MispredictionFrame - a frame that is known to be mispredicted, or -1 if there's no misprediction

• Hash - hash of the simulation state. Available only if the simulation state implements the IHashable interface

• Initial state - the original simulation state at this frame, i.e. a one before rollback and resimulation

• Initial inputs - original inputs at this frame, i.e. ones that were used for the first simulation of this frame

• Updated state - the state of the simulation after rollback and resimulation. Available only in case of rollback and resimulation

• Updated inputs - inputs after being corrected (post misprediction). Available only in case of rollback and resimulation

• Input buffer states - dump of the input buffer states for each client. For details on the fields see the InputBuffer code documentation

• Events - all debug events registered in this frame

State hash

One of the things that are stored as part of the debugging information is the simulation state. In the case of a complex state searching for a difference across multiple clients and hundreds of frames can quickly become tedious. To simplify the problem we can use the hash calculation feature of the input debugger.

Every state simulation class/struct that implements the IHashable interface will have its hash automatically calculated and stored as part of the debugging information. The example of IHashable implementation:

public struct SimulationState : IHashable
{
    [JsonProperty(ItemConverterType = typeof(UnityVector3Converter))]
    public Vector3[] PlayerPositions { get; set; }

    public Hash128 ComputeHash()
    {
        return Hash128.Compute(PlayerPositions);
    }
}

Configuration

Input buffer

There are two main variables which affect the behaviour of the InputBuffer:

• Input buffer size - the size of the buffer determines how far into the future the input system is allowed to predict. The bigger the size, the more frames can be predicted without running into a pause. Note that the further we predict, the more unexpected the rollback can be for the player. The InitialBufferSize value can be set directly in code however it must be done before the Awake of the baked component, which might require a script execution order configuration.

• Input buffer delay - dictates how many frames must pass before applying an input. In other words, it defines how "laggy" the input is. The higher the value, the less likely clients are going to run into prediction (because a "future" input is sent to other clients), but the more unresponsive the game might feel. This value can be changed freely at runtime, even during a simulation (it is however not recommended due to inconsistent input feeling).

The other two options are:

• Disconnect on time reset - if set to 'true' the input system will automatically issue a disconnect on an attempt to resync time with the server. This happens when the client's connection was so unstable that frame-wise it drifted too far away from the server. In order to recover from that situation, the client performs an immediate "jump" to what it thinks is the actual server frame. There's no easy way to recover from such "jump" in the deterministic simulation code so the advised action is to simply disconnect.

• Use fixed simulation frames - if set to 'true' the input system will use the IClient.ClientFixedSimulationFrame frame for simulation - otherwise the IClient.ClientSimulationFrame is used. Setting this to 'true' is recommended for the deterministic simulation.

Fixed frame rate

The fixed network update rate is based on the Fixed Timestep configured through the Unity project settings:

To know the exact fixed frame number that is executing at any given moment use the IClient.ClientFixedSimulationFrame or CoherenceInputSimulation.CurrentSimulationFrame property.