Taking the time to attend KubeCon, even though I just attended virtually this year, is always a great way to keep up with what is happening in the Kubernetes & Cloud communities. There are technologies to dive into, products to explore, interesting lectures to see and a chance to meet with peers, either online or in-person, and talk about common interests.
eBPF is Everywhere
The prevalence of eBPF was very apparent at the conference this year. The release of the eBPF Documentary was a notable event, and there were also multiple sessions containing eBPF in the title or as part of the main subject. And for good reason- eBPF has proven itself to be a powerful tool in multiple areas; networking, monitoring, and of course, security.
It is said that eBPF is to the Linux kernel what JavaScript is to the browser, bringing superpowers to developers in the space. Having this very low-level access to monitor the system is truly doing that, and does not add any need to instrument existing code to be able to gain these insights.
While this ability is powerful, having this type of access also brings its own set of challenges. eBPF monitoring has the potential to bring such an abundance of data that it can become noisy, making it difficult to determine what is meaningful and what is not. For this reason, I expect eBPF knowledge and analytics are areas that we will continue to hear about consistently in the near future.
What About AI?
To no one’s surprise, this year both security and AI were in the spotlight. Everyone seems interested in exploring where AI can be helpful for their industry. For example, discussions revolved around generative AI and predictive AI, as well as using AI for online help (e.g. ChatBots) that are tailored to your product or platform by utilizing LLMs.
In a way, it feels like these days everyone has AI FOMO (fear of missing out), with conversations revolving around finding use cases for it.
It goes without saying that AI has great potential for use in multiple fields and is already applicable in multiple areas like predictive auto-scaling of pods, workloads or clusters. Generative AI is also very much on the rise with GitHub’s Copilot already having reached over a million users earlier this year.
I expect AI will still be the talk of the town in a year when we get together again for the next KubeCon and will have made progress in expanding its adoption and use. It should be noted that while AI has massive potential, there are also numerous security concerns regarding its use, including potential dangers when using LLMs for generative AI.
The Rise of Runtime Security
Another topic that came up repeatedly at KubeCon this year was runtime security and understanding the state of Kubernetes in runtime environments. This also ties back to eBPF, which is greatly improving tools for runtime security, both in terms of capability and performance.
Several of the sessions and two open-source projects at KubeCon this year discussed the need to look beyond shift-left security and do security at runtime, in live production environments. While shift-left security should always be employed for building best practices and to help minimize risk, there were numerous discussions about the need to not “only” do security at development, build time or at the time an artifact is published or stored.
Having visibility into what is actually happening at runtime is critical to determine any changes that have occurred since build time, such as whether new vulnerabilities or threats have been discovered since the code or application was deployed. Having this runtime context is critical, as this is where the threats eventually happen.
Common Attacks on Kubernetes Environments
Many discussions at KubeCon this year also focused on common Kubernetes attacks. Attackers often find Kubernetes attack paths through vulnerabilities that are unknown to organizations, using them as an easy entry point to carry out Kubernetes attacks. There are numerous common methods for attackers to exploit vulnerabilities in Kubernetes. Some of the most common Kubernetes risks continue to be:
- Insecure workload configurations.
- Supply chain vulnerabilities.
- Overly permissive RBAC configurations.
- Lack of a centralized policy enforcement.
- Inadequate logging and monitoring.
- Broken authentication mechanisms.
- Missing network segmentation controls.
- Secrets management failures.
- Misconfigured cluster components.
- Outdated and vulnerable Kubernetes components.
Internet Exposure Remains the Biggest Risk in Kubernetes
While there are numerous risks in Kubernetes, exposure to the Internet remains the biggest.
There are numerous factors that should be considered when prioritizing Kubernetes CVEs, including:
- The environment of the service (production, test or sandbox).
- Whether the vulnerable package associated with the service is actually in use.
- Whether the service is actively communicating or exposed to the Internet and in which ports.
- Whether the service has been attacked recently.
- Whether the attack environmental conditions are met to indicate if the vulnerabilities are actually exploited in the specific service.
The discussion for securing Kubernetes continues to revolve around identifying the most critical risks first, and these factors, including Internet exposure, help quickly determine which vulnerabilities are the most critical to an organization.
Final Thoughts
Going into this year’s KubeCon + CloudNative with security on top of mind made it very apparent that Cloud and Kubernetes by “default” do not apply high levels of security. For example, in most cases, if you create a managed Kubernetes cluster using all the default settings, you will end up with a cluster with public internet access to the Kubernetes API, and no network policies or admission rules.
Not having built-in security controls is not necessarily a bad thing, as it makes learning, developing and other aspects of using cloud and Kubernetes easier. However, learning how to secure these resources should also be part of the learning experience. While this varies by role, at least a general understanding of Kubernetes security is needed in order to successfully use Kubernetes and take advantage of its benefits.
Kubernetes has brought development and operations roles closer together than ever, and in some cases these roles converge, and both roles can now benefit from understanding what the other is doing. With this in mind, I anticipate that eBPF, AI and runtime security are topics we will continue to explore in the coming months following KubeCon NA 2023.