#42 Running game servers on Kubernetes

[English episode] 🚀🎮 In Aflevering 42 hebben we het genoegen om Mark Mandel, Developer Advocate bij Google Cloud for Games, te verwelkomen. Samen verkennen we de dynamische en complexe wereld van game-ontwikkeling, waarbij we ons verdiepen in twee opmerkelijke technologische hoogstandjes: Agones en Quilkin.

Agones is een revolutionaire serveroplossing die specifiek is ontworpen voor het beheer van game servers op Kubernetes. Met Agones kunnen ontwikkelaars in-memory stateful workloads beheren, waardoor ze de flexibiliteit hebben om servers dynamisch te schalen op basis van de vraag van spelers. Een van de meest indrukwekkende kenmerken van Agones is het vermogen om spelsessies toe te wijzen aan specifieke servers, waardoor updates van serverversies geen invloed hebben op de gameplay-ervaring van spelers die al zijn ingelogd. Dit zorgt voor een naadloze overgang en minimaliseert onderbrekingen in de gameplay.

Quilkin, aan de andere kant, biedt een geavanceerde service mesh-oplossing speciaal gericht op de behoeften van gameservers. Het stelt ontwikkelaars in staat om de netwerkcommunicatie tussen game servers te optimaliseren, waardoor ze kunnen profiteren van lagere latentie, verbeterde betrouwbaarheid, schaalbaarheid en beveiliging. Door het implementeren van Quilkin kunnen game-ontwikkelaars een naadloze en responsieve multiplayer-ervaring bieden aan hun spelers, zelfs bij de meest veeleisende workloads.

Tijdens onze boeiende discussie met Mark Mandel verkennen we de diepten van deze technologieën, begrijpen we hun impact op de gamingindustrie en ontdekken we hoe ze de toekomst van multiplayergaming vormgeven op het Google Cloud-platform. Mis deze onthullende en boeiende aflevering niet, waarin we de grenzen van game-ontwikkeling verkennen en de kracht van technologie benutten om meeslepende en boeiende game-ervaringen te creëren. 🚀🎮

https://github.com/googleforgames/agones
https://github.com/googleforgames/quilkin 

#GoogleCloud #GameDevelopment #Agones #Quilkin #Kuberetes

DEVWORLD CONFERENCE
#1 DEVELOPER CONFERENCE ON THE PLANET Ontvang 30% korting met promocode: PodcastDevWorldFriends

Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.

Transcript

Ronald: Our guest today is Mark Mandel form Google Cloud. 

Mark: Hi. I thank you so much for having me. I have no idea what you just said when your introduction was there, but I assume it was good. So well done. 

Ronald: It was very good. So for the people who don't know you, can you please introduce yourself to our listeners? 

Mark: Absolutely. So, hi, my name is Mark Mandel. Depending on whether I'm using my Australian or American accent. But yes, I'm an Australian, lives in San Francisco, I am tech lead for a developer relations for gaming on Google Cloud. I'm one of the founders Agones, which is the open source project we're talking about today, as well as a one of the founders and other one Quilkin you may end up chatting about. I have a pretty long history working in open source both in general web tech and now primarily in gaming. And yeah, that's me. I'm super passionate about just education around back ends for gaming and helping people build out backends for multiplayer games. 

Ronald: Cool. How long have you been working with Kubernetes? Do you recall the first version you utilize in your project? 

Mark: That is actually a really good question because I've had arguments with friends about this, and the reason is, is that, okay, so according to a good friend of mine, she says that I was ranting about Kubernetes pre 1.0 before I joined Google, which I don't particularly remember, but sounds like something I would actually do and my memory is not the best. So I was definitely been working with Kubernetes. I'm not even sure what version it is. I would say the first experience I had was when I first joined Google, one of my wonderful teammates, Brian Dorsey, was doing a workshop with Kubernetes for the day and he was like, Do you want to like, do you want to be teaching? It's this and you want to come help me? And I was like, Cool, I'm going to sit down and we're going to work out pods and services and deployments and get this done. And so I did that overnight. And the next day we went to a workshop together for like 15 people, but it was like, yeah, nice.

Jan: How did you do it? Just, just from for myself. How do you do the networking work in the early days?

Mark: Early days we had get on Google Cloud. So cloud privilege. I didn't have to do a lot of the heavy lifting there. So it was relatively straightforward for just general web tech kind of stuff. It was just like stole like this dodgy, get it, off you go and then let's just put stuff on it. So I believe, if I remember correctly, for that workshop we spun up, you know, Google Cloud projects for people and just let people, you know, cut loose on them for a while and turn them down at the end. 

Jan: You had it easy. 

Mark: Yes, it is. Yes. But cloud privilege to other stories. Yeah. Now, it didn't go through Kelsey's the hard way. Nothing like that. I was like, now, now that's another person's problem. I will just use the thing that makes my life easy. 

Ronald: Yeah, yeah, yeah. Great. Can you share some details about your most remarkable project you've undertaken? We're using Kubernetes. 

Mark: I mean, that's probably Agones really, I think. Yeah, is probably the most exciting thing I've done and we think we can go deep on that. But yeah, I will say the thing that got me so excited about that, if anyone's done work with the extensible nature of Kubernetes, custom resource definitions, API services, validation and mutation web hooks like the extensibility of Kubernetes is just really hugely powerful and kind of a testament to what we've been able to achieve with it, with the project that we're working on. And so like that, I think that for me is the thing that got me really excited and probably the I, I found some really weird bugs in Kubernetes, extending out Kubernetes in some interesting ways. 

Jan: Oh, okay. Because it's still very large. So you've got an I found it interesting because.

Mark: Most recently I found a panic and cube CTO in version 127. So we do API extensions, so API services that people aren't familiar with that very much like pre custom resource definitions. If you wanted to extend the API surface for Kubernetes itself, you can do that through an API service, an API extension, and it's basically a proxy layer, right? Like a message comes in to the API and gets redirected back to back to your endpoints. So you basically have to do like all the work from top to tail. We have one of those for reasons and we can get into that. But yeah, there were some code in the computer stuff that expected certain stuff to come back from certain endpoints that we had never implemented because they were new or recently new and they were the whole thing fell over. So we reported it got a fix. That's fine. But yeah, I found some some odd stuff in places. 

Jan: Awesome. And your greatest projects Agones, What is Agones? 

Mark: Okay. So let's let's set some foundation and talk a little bit about multiplayer games and how often they get built. And so the patents therein and then we can talk about a contest and how it solves that problem to the cool. Yep. Awesome. Okay. So this is and I appreciate face this with this is one way to do this type of work. There are others that have pros and cons that could be a whole other podcast episode. But when we look at Sydney, you're very fast paced, very fast paced, interactive, often competitive multiplayer games, right? We're thinking of, I'm not speaking other people's behalf, but like games like a Fortnite or an Overwatch or guys rocket league or. Yeah, yeah. The finals that came out recently. And in your part of the world, what often have is what's commonly referred to in the industry is probably like a dedicated game server, although sometimes it gets shortened to game server, that game server gets overloaded a bunch. I try and use the word dedicated game server or authoritative server. Basically what that means is say the three of us are playing a game, right? We would all connect our game clients up to this dedicated game server that in this particular instance would live on the Internet somewhere in the cloud, for example. And it's the job of that dedicated game server to be the authority of everything that's happening in the game. So say we're driving around a racetrack and I'm sort of going this way and you're going that way. Its job is to be like, okay, let's do all the physics simulation in memory and keep track of what everyone's doing and basically pass that information down from the client as it goes up. And they could do all the calculations and pass it back down with some magic and some wizardry, and we can talk about that sort of stuff if you want to. Some lies and magic and fun because time is relative and all kinds of weird stuff happens. But basically to be that authority and send that information back down to the players and there's a couple of good reasons for doing this. So one of which is latency requirements. So typically and again it varies like 50 milliseconds away from the players is about your sweet spot. Some games can go up to about 200. It again depends on the game, what you're doing. But being able to geographically place where you dedicated game servers all around the world means that you can control the latency, right. If we're if we're if I was playing with you, you're in Europe. I'm here in the U.S. That depends on where the pipes are and which networks are going over. But that's fine. You know, it might be best if we were playing on maybe somewhere like New York or depending where your data centers are, that kind of stuff, where maybe somewhere in the middle of like maybe Germany, I come across the across the pond depending on your latency metrics, but having that control of geographic location means you have control over latency. It also means that if you're putting it in the cloud, you have control of the hardware. So you know exactly where it's going to run. It was going to run on. And also people do bad things. They do bad stuff. It just they just do bad stuff. So if you hand someone, a client, a piece of software that they can run, that is the game. Ultimately, you can't trust it. So making the client authoritative is a tricky proposition. I mean, that's, again, a whole other conversations, but having basically an authoritative server in the cloud mitigates some of those issues because you control that authority. Yeah, what's going on, etc.. Okay, so that's, that's the, the long winded multiplayer game conversation,

Jan: Just clarifying for the audience. Agones is a way to run your multiplayer games. 

Mark: Yes. Yeah. So, um, so the whole point over gone is, is about running these kind of game server workloads and, and you could even extend that to what we've commonly come to kind of term and a bunch of other people that have also used it as kind of in-memory stateful workloads. Okay. Which I think is an important distinction. So if you're we're playing a game together in memory, we are playing that game together. It's in memory in that authoritative server. The whole state of the game is in there. It's super fast, super computationally heavy. So we can't like offload that to another system. We can't do like that wonderful stateless web kind of thing. Like that's not a thing which is just so convenient when you can do it, 

Ronald: react to the container and then what? 

Mark: Yeah, exactly right. You've got ten web servers running. You kill one. Not a big deal, right? If you've got all of us playing a game and you kill that container, we're all going to go on Reddit or Twitter or pass it on or whatever is what everyone's using. I mean, like, I had a bad experience. I don't like it. This game sucks, right? So we don't want that. We want to have good player experiences. So it kind of says a bunch of stuff around. Like tooling around for stuff is the primary thing that it really does and optimizations around this is that if you have players playing a game, don't interrupt that session. So it's in memory stateful and we do a bunch of optimization. Where can we do a bunch of other things that are just like super handy for how communication happens as well, which is again slightly different. But the core premise is we have we have this idea of game server and Fleet. Those are our resources. We have this kind of declarative thing called allocation, which is like, go give me a game server that matches this criteria and market is allocated. So then we know that it has players on it, but then I can scale my fleet, upscale, my fleet down. I can also get my fleet, I can roll out new versions of my game server across the fleet without any fear that I'm going to interrupt an in player session because it is marked as allocated, which means that players are on it and therefore I can kind of orchestrate my game servers at scale without that fear of interrupting player sessions. That's that's sort of the big draw. A pause there, a pause there 

Ronald: We can hear you are deep in the material. 

Mark: So I live and breathe this right like and sometimes I can't see the forest for the trees and that's fine. But like that's I think that's and if you wanted it almost think about it another way. Like we see the similar patterns in other places, in other industries. Like we're here on a zoom call. I don't know, housing does their stuff, but like we see the similar things in like VoIP and whatnot where like, yeah, everyone connects for a single point. Quite often it's running like some sort of voice over and like while this conversation is going on and we have to have a single point or one of patterns for doing that is to have a single point to be able to transfer this information, maybe to transfer video streams and other kind of stuff. So it's very similar kind of platform and very similar kind of workload. But we focused on games. We focused on games because it gives us a point of focus, of course, just to go back on the latency. 

Jan: And so you can how do you just thinking about and know where to put your server workload? 

Mark: Yeah, that is a good question. So the short answer is to start with guest is, which is like the fun part of games is you don't know how many players you're going to have at launch and you don't know where they're going to be. Yeah. So you're going to need to adjust over time. So to start you will again, you make some estimates on how many players you think you're going to show up. You've got a maximum sometimes in games. Yeah. So a usual best practice that I actually really love when scaling games is actually having a upper bounds player throttling count, right? Like once we hit mounts then we don't let out anymore in and then maybe we can expand that over time. I actually really like that is the thing is like basically a rate throttle or on an API but you should probably put like a few clusters again depending one or more clusters. And like you ask one more cluster somewhere in Europe, one or more clusters somewhere in pack again depending and then adjust over time and again as does come with some basically like their ping services latency or latency recording services, they're literally just air points so that when you are matchmaking your game, part of what you will actually do is send pings out to all the locations that you know about and the regions where you have your game service hosted. So you can track that information and send that as part of your matchmaking request. So usually your game is going to have like a low latency, slow, like a service level objective, right? Like my game can run at 100 milliseconds or less, for example. And then that'll be part of your matchmaking and how you place your players, and then you'll be like, Okay, these players need to get here in this region. So that's where I'm going to allocate from and grab a game server, and that's where my players would go. So they have a good experience. 

Jan: So you're would players in in the lobby and just just like Fortnite, Fortnite starts in the lobby and then you start a server somewhere where the ping is okay for everybody. 

Mark: Yes. So depending depends how you match, make as think that's a whole other conversation. But yes. So usually you will have multiple organized clusters around the world. And yes, depending on your latency information, you will decide which region or datacenter or depending on how specific you want to get and how many clusters you have. You will then specify, okay, this group of clusters over here, they're they're the ones, you know, in the region that I have, where I have my countries clusters, that's where I want to put my player. I'll do an allocation to one of them. To me, it has some capacity. It'll return one back to you and be like, Yep, you got one, here's your IP import. This is where you need to send your information and that's where you can get connected. 

Ronald: So I was thinking we're talking about mostly like the examples now are the multiplayer games, the short matched, let's say the shoot em ups. But there are also games that are a longer stretch like let's say I see the postcard behind you, I think it's sea of thieves, but I don't know if it's correct 

Mark: I forgot to be absolutely honest. I have a bunch of them. 

Ronald: I can see where when you have multiple servers running at one location or multiple locations, is it possible and one server is being drained or our people are leaving and you're the only one on the server. Is there a way that they those get merged with the others running your session with the other servers? 

Mark: Yeah, that's a really good question. I would say it for like the Agones Project itself, that is a decision that is up to the end user about how they want to do that. Ultimately, we are an orchestration platform, so we spin up game servers, we spin down game servers. We do have some facilities. In our system to do stuff like notify game servers. There's actually an integrated SDK that comes with the kind of system that you need to in some capacity put into either game server binary or integrate it in some way. But we do have ways of doing things like through the SDK, like, hey, a new update has come through. You're a long living game server. Either session, maybe you're doing repeats or like a lobby server or an MMO type situation. Hey, there's new version you might want to actively drain or actively scale out so that and then kick that process off. Maybe punt users out, they fall back into their matchmaking pool and then they go back and maybe there's a notification, something like that. So ultimately we provide the tools for if and how you might want to do that, but we leave that as an exercise for you because it's probably game dependent anyway. 

Ronald: Okay. Okay. If you explore the distinction coming out between software development and game development, is there something some overlap there? 

Mark: Yes. Controversial say that. I would say there's more overlap than most game developers want to admit to. Right. So that would be my bingo card. Right. You know, so but I would say there's also differences and it it really depends. So I think one of the big differences is development cycle. So you're looking at two, three years to develop a game. Quite often what you see is technical decisions get decided early on and they don't necessarily change down the path. So what we would commonly inside the wider tech world be thinking of like technical debt and like maintenance tasks in day to day through operations. Unless that game is a hit and lasts for a while. Those sort of considerations are not nearly as important. Games are a hit driven business you're not going to. Generally speaking, I would say in this industry you don't see the sort of iteration that you would have. Like I have a startup, and then we created a thing with like the least viable product and like it didn't quite work. So now we're pivoting. That doesn't happen nearly as much in games. You see it on occasion, but it's pretty rare. It's kind of we made a game. Is it going to be a hit? No, it's not. Okay. Let's move on to something else. Try it all over again and roll the roll. Yeah. So it does it does make things different from that perspective. And also there is what's the word I'm looking for? So that that's definitely a part. Sometimes I feel like the and the teams are smaller, but I may be that may just be like the perspective I see.

Jan: For software development or? 

Mark: Sorry for a game development. Yeah, but and that that could be that could be my bias. I could own that. But I definitely see I can see scenarios where small groups of people support, like large player bases. And sometimes that just also means that, like, we picked away and it works. So we're sticking with it and it's fine. Let's move on. Let's make it. Let's, let's, you know, let's focus on making the game. So that's definitely a thing as well. So it there are there are differences. 

Jan: But, a game has to as also attackers and software bugs. 

Mark: It does. Yeah. So there's some of that too. So a lot of that stuff is very similar. But the what was I going to say? The thing that makes it tricky as well is also the editor of Cycle of Finding Fund, which can also be an interesting thing because you are throwing out as much as you put in. And so there's this there is this trade off where if we look at things like continuous integration and continuous deployment, right? Let's write a ton of unit tests for this so that we know that it's rock solid, blah, blah, blah. And so you do have to kind of pick your poison a little bit and decide, you know, like, do I just hack up this thing so we can test to make sure this bit is actually fun and then do it right? Or do I do it right? And then they're like, Oh, but it's actually not fun. No one wants it. And you spend all this extra time. There are pros and cons with that that make it kind of difficult. Whereas I think I think if you're doing sort of editor development for an application, especially when you have a close relationship with your customer base, you can have that conversation where you're like, okay, we've given you a minimum viable product. Let's like sit in front of it and then like work out what the next step is. Maybe you're doing like an agile process or something that it does. It does come out a little different. There is, there is there's some art to it that I don't know if you necessarily find in other places that and that's to say that like wider tech stuff is bad. It is just different in that aspect, but also the same. I mean, you know, if you're building on an infrastructure platform, then yeah, you end up doing like continuous integration, continuous deployment kind of stuff. 

Ronald: Like Automated test pipelines for game servers. 

Mark: Absolutely. So which is actually super cool stuff. And then but then you're like, okay, we need to automate that. This player can scale down every wall that is in the entire world and how do we do that inside our game engine kind of stuff? But I will also say the one the one big thing that I think is different is launches, because you don't know how many players are going to show up. You can guess you can be horribly, horribly wrong or like a wonderful success and there are so many. Right. And they're absolutely so many. And it's a good problem to have. But it's it is also that thing that you're like. Well, if we mess up launch, then like a bunch of people are going to go away and there may never come. We may miss out on bonus money. So there are there are that complications there as well. You're you know, you can do you know, private betas you can do open betas you know, you can try and mitigate that to a degree. And it's good you should absolutely do that. 

Ronald: But still they get surprised. The Diablo 4 launch for example, you know the queue was insanely long on launch day. 

Mark: Right. Or you see stuff like I mean like among us could never have known that COVID was going to be a thing and it was just going to hit. That could just take off sometimes. Yeah, the Pokémon Go just captured that lightning was just I mean, that was that was amazing. Or even sometimes it's just, you know, some you get a popular streamer. They they just hit something and they make something that people are aware of, either an event or content or something that happens in a game and you get that spike. And that can be that can be tricky. 

Ronald: So if you want to install Agones, what are the logical steps, you can just download the repository for Agones and run it on your cluster?

Mark: Helm install, Helm repo update or Helm repo add and then helm install Agones easy really. 
 I mean we have a YAML file you can use but helm the helm chart gives you like way more configurable options. If you want to run it through something like customize you can as well. Absolutely done that. Both yeah, Helm install stores all the CRD’s, or the controller pods and their extensions and stuff that runs in that. And yeah, just take it from there. Cool, easy. 

Ronald: Jan do you have something to add to this? 

Jan: No, no. Installing it is easy.

Ronald: Aside from Agones, are there other tools in your arsenal that you'll find particularly valuable? For instance, how Quilkin contribute to your workflow? 

Mark: Yeah. Quilkin is an interesting project. Super proud of that. I mean, it's credit as well, like Agones history. We initially started at that with Ubisoft and then we've had a bunch of other game companies that have come through and helped us. I think it's over 100 now or a hundred people, I should add. But we've had a variety of games for years come through. Quilkin is a project that we started with, Embark Studios, who are also amazing and a huge history of open source work. Quilkin is an interesting one in that I refer to it as the service mesh for game servers. I think it's fairly accurate. So actually, let's talk a little bit of game server communication. 

Ronald: We are going to sit straight. 

Mark: Yeah. So traditionally speaking or I would say generally speaking, most game server communication is UDP based. There are a variety of reasons for that. Short answer is if you're running. Yeah. If it's fast, right. If you're running a TCP thing, a TCP connection and like you drop a packet, it has to go all the way back, grab the packet and send it all over again. It's going to wait. By the time that happens, usually you don't care anymore in a game, things have moved on far enough, so usually you're sending UDP packets and that's a whole other conversation. Is good. I can send you links articles and that also means that when you're talking to a game server process and this is true Agones is you generally talk directly to it, right? You're not going through a load balancer, you're not sending it somewhere else because you've got like, say what the three of us are playing game. We all need to connect to this game server. Yeah. So usually you get an IP and port and we all connect to that IP and port and we all play a game. Right. For Agones itself actually we give you a container host port. We manage that for you, that port management for you, do something special that Agonesdoes as well. But that also has some challenges primarily. Again, people aren't very nice. I don't know if you've noticed this sometimes. I mean, generally speaking, you're lovely, but some people are not very much. And if you expose information to the Internet, they may do nefarious things to it. So whether it's trying to access that port, DDOS things that they probably shouldn’t do, especially if there are games and they're like, I'm losing. I don't want to lose. So if we can find ways to information, hide, spread load, all that kind of good stuff that we've probably seen in a bunch of other places that's like awesome. And so. Proxy's right. We've talked like that. That's just a common thing. Whether it's, you know, running. I don't know if it's old school anymore, but, like, running varnish, proxies in front of stuff so that you can spread the load of web servers to like we see things like NGINX and Traefik and Envoy and like all this kind of other really cool and interesting stuff that have come in the most recent years. 

So it becomes an interesting problem in the games industry because we don't, we don't have the nice things of like HTTP or gRPC or anything like that. There's no shortcuts. I mean, there are 

Jan: Web sockets? 

Mark: That is a thing. If you're doing HTML games, that's a thing. Or like some mobile games, all these apps, I guess. But for the vast majority of like PC console games, like it's just all the only thing you can really rely on is that you get a UDP packet And if you're not as familiar with UDP as I have, that means you get an array of bytes. There's no, like, nice HTTP header. There's no, like, routing information. There's no, like, standard for this. So, like, that is all you get. And so Quilkin is a project specifically for routing and manipulating UDP packets when the only assumption you have is that it is an array of bytes, but you still want to do things like access protection. Does this player have a token? Where should that token go to? If I get rid of that token, should they be blocked? Maybe I want to do manipulations, maybe I want to compress, maybe I want to expand. Proxy things.

Jan: Is that token proxy thing in the packets? 

Mark: That token has to be in the packet. Now, where is it in the packet? You have to tell me as part of my system. Right. Because again, no standards. But it does mean that. So Quilkin is a rust based project that is a UDP proxy that can be run actually in a variety of configurations, very much like Envoy. Right. And any sort of like if you're running it as, as as as part of a service mesh, like something like Istio or just running on its own and you're managing and configuring it. So Quilkin is, is very similar, but Quilkin also has integrations with things like Agones as well to help with the management layer. But ultimately what it is doing is providing those things like access control, rate limiting, routing information. You can send packets to multiple places, you can start doing some more sophisticated stuff. So the comments are go ahead, go ahead, go ahead. 

Jan: Could you use this for streaming also? 

Mark: You know, ultimately, at the end of the day, as long as a UDP packet is going through, you could do what you like with it. There is a it has basically endpoints, where can you send stuff and it has filters. So for an application and routing and we can we can extend that. That's not too bad. So yeah, if you want to do routing of packets to a variety of places like hey mirror packets to two or three spots, you could do that. And we also have like logic filters. Like if you have different versions of the packet with different parts of the packet show, which version of it you can then set up different filters based on as your communication protocol changes over time. There's lots of fun stuff in there, but the yeah. 

Jan: I was just thinking about what could that be for? For solutions. How could I use it?

Mark: Yeah. So that the common, the very common use case for this is primarily about routing access control so that the common scenario would be something like, okay, we're all playing a game as part of the matchmaking process. We all get given a short token, I could be like a, I don't know, 12 bytes, 18 bytes depending on how you want to be cryptographically secure. In theory, we all get a short token and as part of the game client whether running Quilkin is a binary next your game client or as part of the we have some integration like we have an Unreal SDK that you can use as well or just do it yourself. It's again just appending bites to a packet. You add that token to your packet as it comes through. What instead of having an IP and port like we would do here, where like, hey, here's an individual IP and port from my particular game session, you would probably have actually have a load balancer IP or maybe even a series of proxies. There's pros and cons there in that you send the traffic to. That traffic, going to those proxies. Those proxies, if we're talking actually about the Agones integration would be backed by we have actually, which we have Quilkin. They're called agents. They basically say like introspect the information that's of the game servers that having Agones system you can write your own as well. That's certainly thing. The examples and interest affects the game servers themselves looks at their annotations is like, oh, this game server has these tokens attached to it. Cool, I'll take that information, send that back up to the proxies aggregated all on the way up. And so in the plan, those packets come into those proxies, they go, okay, what tokens. Oh, sorry, what endpoints do I have that matches off to where my game servers are, what tokens are attached to those? Okay. I have a token for this particular player. Cool. I noticed it to that endpoint because that information can now flow all the way through and it runs through a series of proxies and the player data gets routed to the game server without actually knowing any of the information about what the infrastructure looks like behind. So you get some nice information sharing. And then it does also mean that you can offload things like doing access control or maybe compression or decompression or other fun things like that off of your game server process. But that also means that if you detect that a player is doing something they shouldn't be doing, you revoke that access token. It flows through the proxies now, then gets denied access. Yeah, they get denied at the front door. And there's other fun things you can do architecturally there. But that's why I kind of call it I do call it a service mesh for game servers because you're looking at very similar patterns there of like how often can you access things and do you have access and should you? And then we can we can look at those patterns that way, too. It's a fun problem and it's super interesting. I really like it. I also feel like it's one of those and I feel like actually a very similar to service mesh. It's like it's not a problem until you felt that pain. That makes sense. Like, I feel like when I first like when I was first looking at stuff like Istio or LinkerD or anything in that kind of space, I was like, Oh, do I like, I'll just let everything talk to everything. Like, it'll be it'll be fine. And then when you start getting bigger and bigger systems and you're like, How do I keep track of what's talking to what and like, should it and how often or do I want to have like commonality, like I want to have retries or I suddenly have four clusters and like one of them falls over like short circuit and go somewhere else and like, 

Jan: Yeah Circuit breakers, sure. 

Mark: Good. Where you start, you start hitting those kinds of problems. So it's, it is I feel like one of the problems where you're like, oh, once you hit the pain, you're like, Oh, cool, I can see where this is interesting, but it's all you do. It's kind of okay. Is the complexity worth it? And so there are, you know, the tradeoffs there. 

Jan: Yeah. Awesome. 

Mark: I'll talk about this stuff forever. I will say this stuff is fascinating to me all regardless of wherever. 

Ronald: We can listen forever, but yeah, we don’t have forever. So envisioning the future. How do you foresee the role of Kubernetes evolving in the realm of game development? 

Mark: You know, I think it's just going to continue to grow. I mean, I think we've all seen it. Right. Kubernetes is the primary de facto standard for distributed systems at this point in time. Haven't seen any other challengers come up that have sort of tried to sway that. So I think. I think it will continue to grow. And the ecosystem that sits around Kubernetes, I mean, it's it's so hard to beat that. Right? There's a certain network effect that happens in there. There are some different challenges that happen in games that and like Agones that there is are quite different from probably a lot of other workloads some workloads anyway. Like for example, a game server is a more journey than say a deployment for a website. You know, they go up, they go down, we put more load on ETCD. We have a lot of conversations about ECTD. That is the thing we talk about a lot and so many interesting challenges there in versus like, you know, if you have a static deployment or something or scaled and it doesn't necessarily scale up and down nearly as fast as how much might. But again, that might be similar to what you see in sort of batch workloads. So that kind of stuff, too. So there are there are definitely stuff inside Kubernetes itself, like there's some caps that have been following for a while that have started to get into Alpha now, which are really cool and exciting and some other initiatives in that space that take some of our pain away. But yeah, I honestly, I can only see it growing unless somebody else comes along and kicks Kubernetes off the, off the perch for, for distributed systems, which, 

Ronald: You know, there will in the future maybe be a next evolution.. 

Mark: Who knows or Yes. Yes. I mean, at this point, I don't see anything better, but I'm not that smart. So, you know, there are other smarter people in the world. But yeah, Kubernetes, I it kind of does it. I mentioned at the beginning, it does blow me away just how much extensibility really is built into Kubernetes and the initial design that allowed for that. The fact that the fact that right. Like you put a CRD in and it says game server means that when I go to my kubectl client, I can go ‘kubectl get game servers’ and it'll order complete for me and all stuff. Like I don't have to do any of that work. Like that's so cool. Like it's just so cool. Like that first moment when I did that, I was like, Oh my God, I've just asked Kubernetes to show me all my game servers. Like, it's that's amazing. 

Jan: So as a Lunix guy, I like the tap completion. It's, it's awesome. 

Mark: It's fellow Linux guy. So what do you run when Linux distribution about to Debian I run Debian that's okay that's kind of that's a tradeoffs. Yeah we have to go now right well so long time Debian user X monad it's fine it's good. 

Ronald: Well, I read from the Windows, so that's what I think. 

Mark: That’s ok. So I have a Windows machine to play games on, so that's fun.

Ronald: So what are you guys currently playing? What's your favorite game? 

Mark: Oh, I don't know. Favorite. Favorite is a hard one. I just finished the PC Port of the Last of US. Part one. On the remaster after all the updates are. Oh, yeah, yeah. I waited a while. I know. It's a good experience. It'll get, I think. Yeah. No, it was actually a really good, serious, an amazing, amazing game and primarily a PC game player and like a little bit of mobile, a little bit of Switch. So just finished that and I play a ton of Marvel Snap on my phone, probably more than it’s healthy. What are you playing at the moment? 

Ronald: Playing Diablo 4 with some guys every evening and I'm a big Sonic fanboy I also play the sonic games. I got a whole room. I know you can see that. 

Mark: Oh, very cool. Yeah. See in the background. Yep. Well, yeah. 

Jan: You play a lot of Minecraft I knowh. 

Jan: Yeah, I play a lot of Minecraft. I like Minecraft. I like to automate things. So got a gold farm. I've got an iron farm. I've got all sorts of farms.

Mark: Playing, a lot of crafting stuff. Yeah, I, I struggle with it personally. I struggle with those kinds of games. They're too little, too open ended. But I do love a story. The funny thing is, is I work on multiplayer tech and I just don't really play multiplayer games, but I play with a character in a story. 

Ronald: Thank you very much. Where can people find you?

Mark: My pleasure. Thank you. I'm using social media a little sparingly these days, but if you want to find me a Linktree of Marc Mandel is probably the best, best bet across pretty much everything from whatever Twitter's calling itself these days. Mastodon. Blue Sky. LinkedIn. I think I've got a Poliworks somewhere, etc. etc. so go find me there. I may return at some point. Who knows? 

Ronald: Maybe at KubeCon?

Mark: Yeah, I won't. It's possible. It's possible. I spoke on KubCon EU last this year, this year, early this year may end up at KubCon and a next year would probably be the most likely. 

Ronald: Yeah. Paris, so good reason to go. 

Mark: Yeah. There we go. Yeah, I don't know. Yeah, I don't know if I'll end up at KubeCon EU next year. Definitely see me at Game Developers Conference and then probably some other gaming. Yeah. Throughout the year.

Ronald: Thanks again for taking your time to have a chat with us. We're going to follow Agones and Quilkin with a lot of interest in the future. So I hope to talk to you again. 

Mark: Awesome. Anytime.