Multiplayer and client prediction in Unity

Basic multiplayer in my test app seems to be working well. It’s nearing the time to integrate everything into the main app. Still a long way to go.

The whole networking system in Scraps is a bit unusual because Unity’s physics (which I’m using extensively) can only run in real time, but I also want to retain cheat protection by having an authoritative server. I’ve never heard of anyone doing things quite the way I’m doing it, and I’ve been waiting for the moment when I realise why no-one’s done it, but it seems to be surprisingly decent so far. One day in the future, I’ll do a writeup on the hidden magic.

The thing is that really good networking has an authoritative server (so you can’t cheat) and client-side prediction (so you don’t get a delay on your commands). And lag compensation, but that’s out of scope here.

So the server controls everything that happens in the game. When you press a key to move forward, your computer tells the server that you pressed the forward button, and the server moves you forward. Then it sends back the information about where you moved. The server has the authority on everything you do, so even if you hack your game to move at double speed, the server will just snap you back when it tells you where you really are. I can do that with Unity physics, no problem. But you need to do more than that, because there’s always some latency.

If it takes 200ms to get to the server and another 200ms back, it’s going to be almost half a second after you press forward that you actually move! There are old games that work that way, but mostly these days we try to predict. When you press forward, your computer moves you forward, then when the server sends back the authoritative information about where you moved to, you can adjust if necessary. If the movement code’s exactly the same, you shouldn’t have to adjust. Bam, authoritative server, and instant response. I can still do that with Unity physics. But there’s one more thing to it.

Let’s say you’re moving along and the server sends you a message saying “you are here”. That message isn’t for where you are now, it’s for where you were 400ms ago (or whatever the latency round-trip is)! In the time between then and now, you could have done anything. You might have pressed jump. You don’t want the server to slam you back onto the ground because it hasn’t realised that you pressed jump yet.

So ideally we record all the commands that the user made since the last update from the server. Then when the server sends a new update, we can start from that moment, and replay the rest of the recorded user commands to get back to where we are now, but with any corrections from the server. Again, this is all to prevent cheating – otherwise we could just assume the client was right and tell the server where they are.

We actually need to constantly replay all the commands we’ve recorded since the last server message, every frame, until we get the next one. Now, uh oh, Unity’s physics aren’t fully deterministic, and you can’t step forward or rewind the physics anyway. Everything just runs in real time, all the time. Don’t we have to rewind time and then step forward again to do all this?

Oh yeah. And that’s the problem. You can kind of approximate Unity’s physics, and there’s a really good blog post on it here by Joe Best, but in the end he seems to come to the same conclusion as me, which is that it’d be a huge mess to try and do for vehicles (or for anything else really). A large part of that is Unity’s lack of ways to check if something will collide without, again, actually letting time pass. See the first entry in my earlier post here.

So I wrote down what I thought Scraps multiplayer must have:

  • No client control lag
  • Supports latency up to at least ~500ms
  • Cheating prevention
  • All physics can run in real time

I’m wasn’t keen for a huge job like trying to replace the entire physics system with Bullet physics or something like that. But I managed to work out a networking system different to anything I’ve seen before. I call it Vibrance. It’s semi-authoritative. So far it has a few small drawbacks and isn’t as “clean” as the above – it’s a little bit messy – but seems to work well overall. Sorry but I’m not ready to reveal the details of the implementation. That might be a mistake as someone would probably tell me immediately why it’s a terrible idea, but so far, so good.

Thanks for believing in the game so far.

Valve has some great documentation on how this stuff works in the Source engine if you’d like to know about FPS-style networking.

Bookmark the permalink. Both comments and trackbacks are currently closed.

14 Comments

  1. Joseph
    Posted September 14, 2013 at 6:49 pm | Permalink

    Thanks for the write-up of the progress so far!

    good luck with multiplayer

  2. Morten Andre
    Posted October 11, 2013 at 7:54 am | Permalink

    When will we be able 2 play multiplayer?!

    • bill
      Posted October 11, 2013 at 9:40 am | Permalink

      I’m going to play it safe and say “a few months.”

  3. Posted October 18, 2013 at 10:02 am | Permalink

    Also can you make 1 more terrian thats a city? that will make me very happy. P.S make the city huge.

  4. Posted October 18, 2013 at 10:04 am | Permalink

    Make it so that the player can decide where the wheels go

    • bill
      Posted October 18, 2013 at 1:35 pm | Permalink

      Something like this is needed eventually, yeah. Unfortunately a lot of the code assumes that the chassis format is fixed so it’s not a very easy change.

  5. Posted November 1, 2013 at 11:52 pm | Permalink

    Hi Bill,

    Cheers for including the link to my post, and I wish you the best of luck going forward with Scraps!

    Joe

    • Posted November 1, 2013 at 11:53 pm | Permalink

      Doh.. Got the address of my own website wrong *facepalm*

    • bill
      Posted November 2, 2013 at 9:25 am | Permalink

      No problem, your post is about the only Unity-related thing I could find where someone had actually tried to implement something. Everything else is “experts” on forums simply telling people they should use prediction and linking straight to the Valve posts.

  6. Marc
    Posted March 28, 2014 at 1:45 pm | Permalink

    Hi Bill,

    Thanks for that write up. I’m actually facing the same problems that you outlined here with Scraps in a new game I’m currently trying to get going. Namely, how to predict physics based movements over a network. Any plans to reveal some information about the solution that you have arrived at? Even if it’s described at a high level overview, I’d be very interested in hearing it.

    • bill
      Posted March 28, 2014 at 4:35 pm | Permalink

      I’ve actually given the information out to a few people now, so I might as well add it here. The trick is, I sort of do things backwards. I’ll give you a summary.

      In Scraps, the client does their input and it runs right away, and the input is also sent to the server (normal stuff so far). But, the server is always running at a delay which is at least as great as the worst client’s lag; that way it never has to predict the future and hence has a “true” account of events. Instead of the server telling the client where they should be, the client tells the server. If there’s a small difference, the server just corrects (we can allow bad-looking “warps” to a new position since no-one sees the server vehicles). If there’s a big discrepancy, that client gets a cheating flag against them. Too many flags and they eventually get kicked.

      That’s how it works for positioning. The way it works for weapons and damage is similar, but for major events like killing an enemy vehicle, the client isn’t allowed to even predict the death and waits for server confirmation. This is a bigger delay than a traditional system since the server is always running in the past. However, the server manipulates time on its own end to adjust its delay to always be slightly above the worst client’s latency, meaning that on a LAN or fast connection, the delay can still be very short.

      I’m not currently lag compensating. Where the server thinks everyone is will be slightly different to where each client thinks everyone is, so you have to “lead” your shots Quake-style rather than the client always being right TF2-style. It doesn’t seem like a big deal at the moment, but if it does become a big issue, it should be possible to do. It might mean having to essentially run a copy of the world for every connected client though, which would be a bit unfortunate for performance.

      • Marc
        Posted March 31, 2014 at 1:05 pm | Permalink

        I think I understand. Does that mean all the clients will effectively experience the same network delay as the laggiest player? Since I assume the clients will have to wait for a reply from the server to update their positional sate.

        As you say should work fine for LAN play but perhaps you’d have to set a maximum ping limit (~250) for online games otherwise the experience might become too sluggish.

        • bill
          Posted March 31, 2014 at 1:28 pm | Permalink

          It does. Which is why it’s important that they’re allowed to predict for most things – that way the delay is usually only perceptible when something really needs to get confirmation first, like a vehicle being destroyed. They’re not driving around with half a second on lag on the controls; that’d be unplayable.

          So the client predicts what will happen and the player sees no lag, then the server will check it a little bit later to make sure everything looks OK (wouldn’t it be great to not need cheat protection?).

          There is also a max ping to let people connect, yeah. I’ve got it on 500ms at the moment. Might be good to make it settable per game.