Kubernetes and Complexity

Steve King: [00:13] 

Good day everyone. This is Steve King. I’m the managing director of CyberTheory. With me today is Craig Box, who is the vice president of open source and community at ARMO. Craig was formerly the Kubernetes and advocacy lead at Google Cloud. He is responsible for ARMO’s open-source Kubernetes security platform called Kubescape and for managing all relationships with the developer and open-source community. Craig joined Google in 2014, just as Kubernetes was about to be released and has acted as a vital link between Google’s Kubernetes development team and the wider community of developers who use the software. Most recently, he was deeply involved in Istio, the leading open-source service mesh, acting as Istio steering committee member and driving Istio toward its recent acceptance as a CNCF incubation project. Craig is also the founder of the popular Kubernetes podcast from Google, the largest and most popular podcast about Kubernetes and a major source for relevant news and updates in the industry. So, welcome, Craig, I’m glad you could join us today.

Craig Box: [01:32]

Thank you, Steve. One of the unfortunate things about leaving Google was that I had to leave the podcast behind. So it’s always a pleasure to be behind the microphone again.

Steve King: [01:40]

Well, good, even though if the microphones reverse, but that’s good. So let’s just jump in here. And I think my first question relates to the capacity current CISOs deal with – I think a fairly complex technology. And from where I stand over here, it looks like we’re complexities. If you follow me at all, the complexities have been my deal for a while, and we’ve rushed the board, this whole digitization movement in the process we didn’t learn much about the technologies that are required for hybrid cloud edge computing containers, etc. And I usually use the idea that if I hold 10 CISOs, I wouldn’t get any hands in terms of explaining how Kubernetes works, so the question is, as a community, have we spent enough time learning about Kubernetes as we should?

Craig Box: [02:46]

I think it’s way too complicated. And we should throw away all the digitalization that we’ve been dependent on paper, I think.

Steve King: [02:51]

There you go. You’re the first guy that agreed with me.

Craig Box: [02:58]

Kubernetes has this challenge. And it’s a word with two meanings. And the easiest analogy I can give to it is it’s a bit like Linux, Linux is a kernel. And there are a million different ways you can compile and tune the Linux kernel. But, in general, over the course of the past 20 years, we’ve moved away from having to care too much about that, we have a vendor who gives us a Linux distribution, and that made a bunch of choices for us. And now the things we need to worry about more, the interfaces between our applications, and how they relate to each other running on top of Linux, there will be some people especially running at larger scale who need to care about that layer underneath. But on the whole, some third party looks after that for us. If there’s a vulnerability in the kernel, we look to them to patch it in good time. And that, to some degree, is what we need to think of when we talk about Kubernetes. There is a wildly successful open-source project called Kubernetes. But when people talk about that, or cloud native, they are talking about running their applications in a particular way using this engine. But they have so many more things than the engine itself to worry about. If you look at the early books and deployment guidelines that talk about Kubernetes security, they say, “Here are all these parameters you can configure for when you install the agents on the nodes. And here’s what you need to do to make sure that the API server isn’t available to the internet,” and so on. A bunch of trivial things, which if you get your Kubernetes from Google, Amazon, Microsoft, whatever on-premises vendor you prefer, they are looked after for you, you don’t need to worry about that. We recommend that you install some tooling that verifies that trust, but verify model. But ultimately, the thing that you are enabling by installing something like this is the ability to load and run applications, Linux applications, to bring it back to our last deck from before on many computers in real time by API. So you now have the ability to turn on however many replicas you need for a particular thing. That means that some bad actor could come along and use that same API to turn on as many Bitcoin miners or sitting at home or whatever the modern equivalent is. So you do have to think more about the interfaces between your code and those systems. And you need to worry about the ability that these still are just applications that were running, but you now have so many knobs that you can turn in the way that they all work together. I think the explosion of edges in the graph, now you’ve got these moving parts. That is the piece we need to worry about as a community. Have we spent enough time learning it? We are still finding out what happens. And I think the right thing to do is to codify these best practices in tooling, rather than make them something that you have to worry about, that you just trust that, as we do with software, we run static analysis tools, we look at the software we’ve written with other software, we should look to do that with our cloud environments where we can.

Steve King: [05:53]

Yeah, sure. My understanding in tracking Kubernetes is that Kubernetes has some sort of native challenges, security being one of them. And I think that owes to its complexity and the native vulnerabilities. One of the best examples of a Kubernetes break-in was that Tesla cryptojacking attack. To get people to understand what is required in removing fiction from facts, how do we do that? Is that education? Is that best practices? What’s your view?

Craig Box: [06:35]

I think there’s a combination of things there. The thing that Kubernetes does is it makes it easy to do things that were more complicated before or took more steps. Cloud is a large factor of this. Beforehand, the API tools, getting another machine was writing a check and sending an email off to Dell and waiting six weeks. So cloud has made it possible to do those things a lot quicker. And then Kubernetes has automated it to the point where some action in your environment, so more traffic, hitting an application, for example, can cause your system to scale up and then can ask your Cloud Load Balancer. It is still a complicated piece of software. Now, the distributed systems are complicated. Kubernetes is a complicated example of that. It has places where things can go wrong. So, a lot of effort in the first few years of Kubernetes went to looking at the differences between the academic, single-user internal environment at Google with things came from and saying, “Now, we need to deal with multi-tenancy, we need to deal with the ideas of many groups or users wanting to access this.” It wasn’t designed to do that. And some of that was added on by teams at Red Hat. And some of that is possible in the way that the container boundaries work on top of Linux, where you can isolate workloads, some of it is still an honesty policies. If I request a certain amount of memory, it’s like, if I use more than that, the system has to kill me, there’s no real restriction and saying, “Sorry, CPU’s slacking, not memory.” So the Tesla cryptojacking thing was an application that was installed in the Kubernetes environment. There was a dashboard, which was provided early in the Kubernetes lifecycle to do visualization of running work, and then let you also deploy things and make changes. In order for that to run, it needed to have a service account from the Kubernetes environment with permission to do those things. So you say, “I want to run a thing, it needs to have permission to act and do this.” So this is a powerful piece of software, which you could just deploy on your cluster, misconfigure it to make a mistake as the operator and say this is publicly available to the internet. That is what I believe happened there. And then you’re effectively unlocking a door and saying, “Here’s a way for anyone to run any kind of things that they want to inside this environment.” And from Tesla’s perspective, they need to be happy that the people who discovered it might not necessarily have known where they were like, if a bad actor had known they were inside Tesla, they may have tried to do something more interesting than just mime cryptocurrency.

Steve King: [09:05]

Yeah, sure. As you describe that, it occurs to me that configurations, always difficult. And it’s getting more difficult with hybrid cloud and more sophisticated, if you will, container technology like Kubernetes. And as that happens, our traditional networking seems to be not very compatible with that growth pattern. And hybrid and multi-tenancy are part of that when the deployment is going across multiple clouds. It seems like with the move toward the fourth industrial revolution, we’re heading more that way every day. I know you guys talk a lot about the service mesh and as a potential solution D, could you explain how that might work in an optimal world?

Craig Box: [10:08]

The model inside the individual Kubernetes cluster is built around the idea that every workload, every pod set of containers can talk to any other pod. And that’s an academic early internet model – the idea that every node on the network has an address, and that you’re all able to speak to each other and so on. Works well when you’re inside a single company, gets harder when you start dealing with cross-cloud boundaries and almost feels like it was designed for a world where IPv6 is going to exist to be a thing. I’m not sure if that’s the fourth industrial revolution you’re alluding to there, but we still wait with bated breath. Ultimately, we want to get business value. And that may involve running a service where our users are now quite commonly worldwide. So there are a lot of people who are looking at Kubernetes to do hybrid cloud environment, running across multiple vendors, running across on-premises or on cloud or so on. So there are a couple of different things we need to deal with here. The first is the connectivity between the clusters, we can either have some overlay network of something that makes it possible to treat them all like they are one big network. Quite commonly, you’ll still have something that has gateways and these services know how to route to each other. And we do want to keep them logically independent of each other. A good pattern here is to say, “We’re going to have everything that’s deployed within one availability zone, everything close to each other, we’ll do replication within that, because we know we’re in the same data center.” It’s all very quick. And then we need to say, “In the event of a failure of that data center, we’re failing over to another one.” And the replication between those may happen asynchronously. So you need to have some idea of the topology of the network for that to work. This is all an early stage. This will happen. The concept of service mesh came along a little bit afterward and said, “We need to do a couple of different things. First of all, we would like to be able to do microsegmentation.” And what I mean by that is to say we’ve got all of these different things that previously ran on their own computers with their own IP addresses that we could do firewalling against, for example. Nowadays, we might have 10 replicas of a particular service A, the only thing that should be allowed to speak to service B. And those service B replicas could be running on any machine in the cluster, or conceivably anywhere, other clusters that were connected to around the world. So we need to be able to save this group of things by whichever parameters we define are allowed to talk to other ones. And we have those parameters because Kubernetes deployed the thing. So what we can then do is create a consistent identity source, give them all an ID and a certificate from that because we deployed them, we know who they are, and then grant them permission based on their identity, not based on where they are on the network and which machine they happen to be in or which provider they’re running on. But the identity that we gave them, then we’re able to say your identity A) you’re allowed to talk to anyone running identity, B) wherever they happen to be. The way that service mesh achieves this is normally with a thing called a Sidecar where we deploy every application. And alongside that, we deploy a little routing agent that gets those identities that intercepts all the traffic and routes between them. And it’s able to make decisions. It’s a software-defined networking agent, but it operates at the application level. We’re looking now at ways we can improve by not having to deploy an agent alongside every workload, but having something that runs just on the node and does lower-level networking. It’s hard to share that high-level networking, where you’re decoding application packet, and so on, in a multi-tenant secure way. So we’re splitting that out and saying, “We’re just going to do the layer 4 stuff at the node level.” And then you can deploy side counts for workloads where you want the extra protection. And effectively, now we’re able to validate both ends of a connection, we can say we know the identity of both ends, we know that this isn’t someone who, once upon a time, stole our certificate, and they’ve got an up-to-date certificate that solves a security, like traffic in motion problem. We know that everything on the wire is encrypted with that. If we tie that in with Kubernetes primitives to be able to do security of that data at risk, then we have some idea that we have probably meeting some criteria that our compliance team would like us to meet.

Steve King: [14:25]

Okay. Do you guys see yourselves as part of a zero trust solution because of the microsegmentation that you describe?

Craig Box: [14:39]

Yeah, I do think that that’s a common category that’s applied here. And in large part, what people think of as zero trust today is derived from some of the ways that Google runs production environments. And that obviously led to Kubernetes and Istio and some of the projects that came out of Google to allow people to run them the same way. The idea there is, it doesn’t matter what the node is you’re running on, it could be adversarial to some degree like you don’t have to be in a secure place. It’s not about where you happen to be. I’m on a trusted node, therefore, I’m allowed to access something else. It’s about the identity that you have, it’s issued to you, you can run anywhere. And you can use that identity to connect services outside the network as well. So you can have something that validates your certificates. And a public service or in a SaaS environment is something that you need to connect to. So it does help you build a zero trust system, there are some other pieces you need to build yourself. But we want to get to a point where if someone is able to exploit something in the middle of your network, we’re no longer dealing with the whole moat around the castle approach. You may be running something I know, image rendering technology or the things that resize versus images, they are a common attack vector. And you hear that, especially in mobile phones, all of the more modern iPhone hacks have come about due to a bug in processing JPEGs, encoded in PDFs, or something like that. So if you have to do some processing, where you accept user data, you end up with someone sending you a crafted file, and then they get to exploit some service, they are now five or six levels deep in your micro services stack. They’re not just breaking in through the firewall, but they’ve turned up in the middle of the network. And so you need to make sure that they can’t get access to your payment service, for example, like there’s no reason your image resizing service should be able to access your payment service. And that’s where the whole microsegmentation thing comes in. You need to deal with the fact that people can pop up in the middle far more easily than they could in the castle-and-moat approach.

Steve King: [16:41]

Yeah, sure. You correctly or appropriately describe this as a complex distributed network. What happens with 5G, in your estimation, whether it’s, let’s call it 100x factoring that speed, what’s the effect going to be on the temporal-based reception, issues that Kubernetes expects, or any container expects as a part of the architecture.

Craig Box: [17:14]

Now, there’s a couple of different ways that Kubernetes and 5G overlap. If you think of 5G as a way of ensuring everything is connected – think of it like Wi Fi in your house today. Now, it’s easy to assume that my thermometer and my TV will have IP addresses and they can all talk amongst themselves when I’m out of the house. But 5G will mean that that’s true of everything everywhere. So cars, your mobile phone, you’ve already got, but anything that’s traveling around, there are a couple of different aspects to this matter, like the mobility of things, like you mentioned, there’s my thermometer in the most part doesn’t come off my wall. Whereas 5G devices move around and they get routed to whichever base station makes sense. There will be a little bit of a challenge, which I not close enough to the wire here. Just an interesting side note – my father has worked in telephony since 50 years coming up now. And it’s always interesting, seeing how his world and mine have overlapped and then moved apart and so on. There was a period where everything started being digitized and moving into IP in the 2000s. And then it moved away again. And now we’re starting to talk about Kubernetes powering some of these 5G systems. So, I’m not 100% sure how it all works. But I can imagine the number of devices and the fact that things come and go is leading the 5G spec to require a bunch of functions that things like tracking where your phone is and how to build it and what its access permissions and sessions should be. If the system is smart enough to make sure that you are always given the same IP address as you move around, then you may end up with a situation where everything’s okay. But you need to move to that more modern identity based on something you have. And whether it be Touch ID on your phone, or whether it’s just the certificate of the phone, for example, for an IoT device. All that is doing then is making sure that there are a lot more is making, there’ll be a lot more devices out there. And Kubernetes is being used inside 5G networks to power those functions, because you need to be able to run a lot of low-impact virtual functions near these base stations. And then, you also need to deal with the fact that that will be a lot more devices on the internet. So bring on IPv6 again. So I still think that there is a bit of a disconnect and like this was going to happen anyway. And Kubernetes is a nice way of being able to manage things that are running at scale and have a lot more going on than they used to. But you could have had one without the other.

Steve King: [19:51]

Yeah, okay. I’m happy to buy that. I know that also storage has been an issue with Kubernetes. I think one of those reasons is that the entire storage infrastructure is managed without relying on any cloud resources. Is that the case? Or how does a practitioner avoid that sort of trap? Configuration thing.

Craig Box: [20:17]

It can be, you can have storage of above and below Kubernetes, which is a complicated thing to understand. So let’s look at them one at a time. You have Linux processes running inside containers, and so on. And they are used to run it on a disk. So those things need to have storage available to them and Kubernetes, as an API provider, is able to answer a request for your storage needs. So you say, “Hey, I’m going to run a database node, I need a 60 gig local disk. It doesn’t need to be high availability, because I’m going to be using this as a read replica,” for example. But you say, “Please provision me 60 gigs, wherever I happen to land.” You might land on a physical server that has some storage available, or you may have to have that mapped over the network. But the idea is that you use a consistent language to define what it is that you want. And you might also have classes so you can say I want this to be high speed or low speed, low cost, for example. And then you can have that deployed for you. It’s a lot easier if you’re running on a cloud, because you can just call an API and attach a 60 gig disk. But you can do the same with on-premises stuff. If you are trying to provide that service, if you’re on-premises, and you want to provide the facility for someone to say, “Give me 60 gigs of disk and have it automatically do network attachment.” There are a bunch of different NFS-onward solutions that allow you to do this, and a lot of them now run on top of Kubernetes. So this is where it’s confusing, because it’s storage above and below, you can run a storage system. And you can have Kubernetes handle the high availability of that system and the replication, all these things that people previously had to build into their own application, you can outsource that to the Kubernetes lab. So a lot of storage stacks from vendors these days, a lot of modern databases are using Kubernetes as their replication layer. You go to Oracle nowadays, and it’s surprised me to see the other day they have a Kubernetes operator, like this is how you go about deploying Oracle today, the assumption is that any stateful service, you’re going to deploy on top of Kubernetes. And that means they don’t need to worry too much about the different APIs that are provided by different things. It does matter a lot, the fact that things come and go in Kubernetes, you don’t want your storage nodes just to randomly disappear or be rescheduled. So you have to build a system that’s aware of this. And it’s designed for that. It’s a bit harder to do NFS out of the box. But something like MinIO, which provides something that looks a bit like Amazon S3 that’s built with replication in mind. And so you can have a Kubernetes cluster providing storage. And then you can have that storage mounted by a different Kubernetes cluster or by the same one. So if you just think of Kubernetes as a system that provides things, it’s like saying, “I have a Linux-powered NetApp system or BSD-powered NetApp system that makes storage available.” And it’s available to my other Linux ballot systems. When it comes to security of those things, it’s all the same stuff. The fact that we’ve made it easy to map and mount those storage volumes, we need to make sure that we’re correctly granting permissions to people. And that is one of the biggest areas that people have to watch out for is that if you have something like the Tesla situation where you have a service account that has permission to do anything, if you have someone get access to that account by taking control of some software that has access to it through some means, then they can pivot into doing whatever else that they want. And that’s where we’re seeing a lot of people say, “I need to look at the complexity of all of the role accounts and the access granted to those role accounts.” And where they need tools like Kubescape, they can come in and say there’s a lot going on here. Prioritize the things that matter to me, show me the vulnerabilities that might matter and how I can go about remediated.

Steve King: [24:11]

The more you talk, the more I listen and look at this stuff, it feels to me like it’s whatever you want. Let’s call them Kubernetes or super smart containers or something. It’s almost become operating system infrastructure like a standalone beast. It’s like the meta presents for running your entire IT factory.

Craig Box: [24:43] It is exactly that. It is a cloud provider that is quite commonly run on top of another cloud provider. And you will need someone in your team who understands the interfaces between the two. If you have that figured out, then most people should only have to deal with a Kubernetes part. The whole idea of this is that you can to say to your development team of people who are deploying platform applications for the people who wanted to run them, that it doesn’t matter that you’re running on Amazon, or Google or whatever, we have abstracted that away, they’ve all got storage available, we’ve just changed the class to say, “If you’re on the Google Cloud, you get given some Google flavored storage.” And you do need a different skill set, I’m not 100% sure that you need the skill set, with training people for how to set a Kubernetes environment up from scratch. Most people are going to deal with it a layer above that, they’re going to deal with how do I configure the Kubernetes knobs provided by the vendor? And then how am I going to provide a platform to my developers so that they don’t have to care about any of this?

Steve King: [25:45]

Yeah, that assumes that the person you were just describing understands Kubernetes to begin with, and again, I go back to that point where I’m not sure. Or maybe it’s. I’m sure that most of the folks leading cybersecurity today don’t. Maybe the current CIO world, maybe the IT folks really do. I don’t know what your experience has been in that regard.

Craig Box: [26:12]

It is a different set of nouns and verbs on a similar kind of problem to cloud. So you can map that thing. But another piece of the challenge is that Kubernetes and cloud before it to a large degree have to overuse the term, has shifted everything left to the idea that developers now can just deploy things, they don’t have to go through processes, they don’t have to be signed off by executives, it now makes it a thing that developers are dealing with. And the challenge is how to make that group of people care more about security, where it’s slowing them down from the thing that they can otherwise achieve. And that was true when it was, “I could deploy a VM with a credit card” and your internal secure infrastructure is going to take me six weeks to provision. And eventually those worlds got brought together, the CISOs found a way to set guardrails up and have people be able to deploy cloud infrastructure. And in the large part, that is the default these days, people are expected to use some sort of cloud infrastructure unless a pricing need moves them out of it. But even on-premises stuff has been provided with cloud like APIs. So there is a shared responsibility here and this should help you understand this. Your developers will know this, I think the choice there is to say, “How do we define guardrails and set up the environment?” The developers don’t want to end up on the front page of The New York Times any more than the CISO does for the wrong reasons. The incentives are well-aligned. It’s just a case of exposing them correctly.

Steve King: [27:43]

Well, my experience has been that the end, this is a large healthcare company that the engineering folks essentially won that battle, period, end of story. They would use a combined engineering said, “I’m over here, I’m going to do this thing, you figure out how to make your world comfortable with it.” That was the impression that was clear in person that I had in dealing with those guys and it’s great to hear your optimism for this because not understanding this at the level that you do. It’s easy to look over here and look at the history of technology, as we’ve seen it and say, “We’re designing and building this airplane that’s already taken off and on its way over the cliff.”

Craig Box: [28:44]

If you think about the airplane as an analogy, to some degree, the pilot knows how to control the airplane, but they don’t know how it’s built. They trust that there is a group of people there who understand and who do the maintenance and they don’t fly the plane until such time as the ground crew have signed off that everything’s good. And then as a passenger on the plane, I don’t have to think about it at all. I trust that process has worked well and accepting the rare cases where it doesn’t, aviation tends to be safe. And processes have made that the case and lead to the point where everyone has a skill set that works. I talked to people who touch individual parts of the cluster of 40 hours a week specifically. And they run rings around me in that one particular area, because that’s the area that they focus on. So I think through this, everyone needs to get an idea which pieces they need to care about, which pieces are relevant to them, and how we can get enough information, how the CISO can say, “My engineers have made this choice for me to some degree, how am I going to become comfortable with it? Am I going to get to the point where I trust them as the ground crew and build something?” And that might be a trust but verify thing that the engineering team provides a dashboard and says we are running these checks and we’re signed off against these things. And it is like the list that’s presented the pilot before they fly.

Steve King: [29:58]

Yeah, it’s the same engineering crew that’s using open-source APIs and that are impossible to vet in terms of what their dependencies are in the wild.

Craig Box: [30:09]

And engineers love new things.

Steve King: [30:14]

I know. Good for them, I am glad we’re able to provide new things for them. I guess the last question I have for you, Craig, we could probably talk for another hour and a half, this has been great. I appreciate you explaining it at the level of a six-year-old, which is what I am. What would you suggest that a CISO do to plan and prepare for a Kubernetes deployment?

Craig Box: [30:41]

Yeah, you need to be deliberate about it. And you need to realize that it doesn’t look like the thing you were used to in the past, even if the thing was just regular cloud stuff, the idea that things will be shared that were not previously shared is scary to some people. And that causes people to make design decisions about how things will run, you need to have a threat model that you’re comfortable with. It might be that I’m willing to have different customers who are running multiple tenants, who are tenants of my application effectively that you’re willing to let them share certain services on a machine. But the ultimate thing to think about in the generic Kubernetes environment is there is still one kernel underneath each machine that is shared between them. And then the most part, we trust that that is a safe boundary. But every now and then, there’s an exploit, which proves that that’s not the case. So what would happen in the event that some user could do something in my environment to get access to other things running on the machine, that’s the most common container escape concern we need to think about. And some people like to set up completely different nodes for completely different customers. And maybe for untrusted workloads, that makes sense. Google came along and built a Linux emulator, I think with gVisor, which was a way of being able to securely run things to give them another sandbox for generic processes, so that if we’re running more than one user stuff on a single machine, they would now have to escape two different sandboxes to get into the kernel. So that was a trade-off that they considered acceptable. That’s an open-source project that you can run. I’ve seen similar kinds of stuff coming out of Amazon, then the micro VM approach with firecracker saying, “We need to think about the fact that multiple people are going to run things in this environment, the academic ideal when Kubernetes was created, was you just have one giant cluster and you throw 10,000 machines at it and you schedule everything and it will optimize everything so much.” And you’re able to pack so much more in, the reality is that it is going to, it’s going to concern people and plan how you want things to be set up, all the observability tooling, so much stuff is exposed by modern open-source tooling that you can scrape with something like Prometheus and then you can get visibility into what things are running where, and you need a way of alerting on that, you need a way of knowing what’s going on in your environment, you need some sort of guardrails, you need policy engines, you might want to consider service mesh, you definitely want to have something that’s handling deployment to the cluster that’s not just human. So some sort of continuous integration or GetOps, if you fancy the modern buzzword of being able to deploy code that will then get processed and rolled out to the clusters, buy something, there’s just so many different pieces here that you should probably spend three or four months beforehand, understanding what you want to do before you’ve gone out and built the production release system.

Steve King: [33:46]

Yeah, that makes sense to me, for sure. Well, Craig, thank you. This has been great, as I said, and, folks, this is Craig Box, the vice president of open-source and community for ARMO and is responsible for their Kubernetes security platform called Kubescape, and for managing all of that developer relationships and the whole open-source community around us. So I appreciate you taking the time today, Craig. This was informative and instructive. And maybe we can catch up again in four or five months and see what’s been going on.

Craig Box: [34:24]

Thank you, Steve. I really appreciate it.

Steve King: [34:27]

And thank you to our audience for spending a half an hour with us and hopefully you’ve learned a lot about Kubernetes and enough to at least get you started on the education side. So we will see you again next time. And I’m Steve King, your host signing off.