Kubernetes, commonly referred to as K8s, is an open-source container orchestration platform for automating deployment, scaling and management of containerized applications. Kubernetes was initially developed by Google for their data centers and over time evolved into a community-driven project, making it accessible and adaptable to various environments. Kubernetes offers affordability, flexibility and scalability, allowing organizations to deploy applications in on-prem, hybrid-cloud or public cloud environments.
The word Kubernetes comes from Greek origins, meaning helmsman or pilot.
The Anatomy of Kubernetes
At its core, Kubernetes operates on a cluster-level architecture consisting of two major parts:
- Control plane – Orchestrates and makes global decisions about the cluster, such as scheduling pods and responding to cluster events (launching new replicas and pods).
- Data plane – Manages the worker nodes that run the containerized applications.
The control plane acts as the brain of the operation and runs a few critical components:
kube-apiserver– exposes the Kubernetes API.etcd– Kubernetes’ backing store for all cluster data.kube-scheduler– watches for newly created pods with no assigned node and selects a node for them to run on.
K8s control plane can be self-managed or managed by a cloud provider (Amazon EKS, Azure AKS, Google GKE). On the other hand, the worker nodes are virtual machines (VMs) that host the actual application workload within containers, organized into units called pods. The pods themselves are made up of the components of the application workload.
Securing Kubernetes
To achieve optimal Kubernetes security, it is crucial to monitor and protect K8s at multiple levels: the control plane and the data plane, which includes infrastructure, applications, and containers.
A compromised Kubernetes cluster can lead to data breaches, service disruptions, and reputational damage. A single Kubernetes cluster can include dozens of applications and entry points, allowing a single breach to quickly escalate beyond the point of entry. By implementing proactive real-time and runtime security measures, organizations can avoid costly security breaches and protect their assets while maintaining compliance.
Securing Kubernetes Control Plane
Most organizations today use a managed Kubernetes service, often supplied by cloud providers such as EKS, GKE and AKS, or a common distribution like KOPS. To interact with the Kubernetes cluster, one must communicate with the Kubernetes API server which is part of the control plane. The API server exposes an HTTP API that lets end users, different parts of your cluster, and external components communicate with one another. It is critical to not only secure access to the API server but also to continuously monitor its use.
Managed Kubernetes environments have limited access to the control plane. Cloud providers allow you to export the logs for various control plane services, while an unmanaged service like KOPS requires configuring and maintaining the control plane along with all logging.
Some of the most common threats that you must protect the control plane against include:
- Unauthorized access to the API server endpoint
- Secrets dump
- Public access
- Excessive RBAC
- Kubelet manipulations
- Excessive service account permission
- Privileged pods
Recent examples of widespread K8s attacks include the RBAC Buster attack, which saw crypto miners leveraging RBAC permissions to create backdoors and run cryptocurrency miners. During the attack, crypto miners deployed DaemonSets to hijack K8s clusters’ resources after gaining access via a misconfigured API server and using RBAC to run on compromised clusters.
In another recent crypto mining attack called SCARLETEEL, attackers targeted cloud environments, bypassing security measures to execute novel intrusions. During the attack, bad actors exploited misconfigured AWS policies to escalate privileges and take control of accounts. After gaining entry, they targeted Kubernetes to deploy malware, including crypto-mining tools. Fortunately, in this instance, the attackers exploited the Kubernetes environment primarily for crypto mining, although it was determined that they had the opportunity to engage in data theft.
Securing Kubernetes Data Plane Infrastructure
As seen in the examples above, in a Kubernetes environment a compromised node can become a springboard for attackers to traverse the cluster, potentially compromising other pods and nodes. To avoid this, it is essential to implement robust security measures to protect the entire Kubernetes cluster in real-time.
How to Protect Kubernetes Data Plane Infrastructure
Protecting the data plane requires robust security measures including vulnerability scanning, CVEs contextualization and prioritization, and intelligent threat detection. Effective data plane security requires runtime insights to understand real-time environmental factors and threats.
Runtime Vulnerability Scanning in Kubernetes
It is essential to scan every running image and pod that runs an application, as vulnerable images can cause widespread damage in the cluster using privilege escalation techniques. While legacy tools have focused on scanning for misconfigurations prior to pushing into live production environments, more robust and continuous scanning is needed to ensure Kubernetes security in dynamic runtime environments.
Kubernetes Security Blindspots & Noise
For example, many legacy security tools scan images during the build phase or when they hit the repository. However, this fails to give a clear picture of which images and CVEs are actively present in live production environments, ultimately creating a noisy environment for security engineers.
The (right) modern approach to Kubernetes security is to scan images at runtime while they run on the cluster, ensuring up-to-date scanning and an accurate inventory of all running images and active CVEs, with real-time attachment to their cloud configurations.
In order to address the ever-increasing number of security vulnerabilities in applications, security teams must know where a vulnerability is (e.g., in production vs. a QA/Test environment), and whether the service has been actively communicating with the Internet or targeted by any attacks. This context makes a substantial difference in determining whether it is worth prioritizing remediation or actively mitigating potential exploits.
There are numerous factors that should be considered when prioritizing Kubernetes CVEs, including:
- The environment of the service (production, test or sandbox)
- Whether the vulnerable package associated with the service is actually in use
- Whether the service is actively communicating or exposed to the Internet and in which ports
- Whether the service has been attacked recently
- Whether the attack environmental conditions are met to indicate if the vulnerabilities are actually exploited in the specific service.
Intelligent Threat Detection
In addition to executing an effective vulnerabilities prioritization and management strategy, organizations must also conduct intelligent threat detection. Containers are dynamic and versatile, ephemeral by nature, and perform multiple actions – making them an attractive target for threat actors to exploit and perform malicious operations.
While shift-left scans for misconfigurations and vulnerabilities can help harden your cloud environment, there is no way to find and thoroughly protect against threats in the Kubernetes data plane without runtime sensors. This has led to the rise of intelligent threat detection tools to provide insights into cloud runtime environments, bringing security teams needed insight into runtime threats.
For example, malicious software, otherwise known as malware, can come in various forms ranging from ordinary processes to files and in-memory. Threat intelligence streams are needed to identify and protect from advanced malware attacks across executable files, running processes and even fileless (in-memory) malicious code. A comprehensive intelligent threat detection strategy will also include detections of malicious egress, observing Internet ingress and egress traffic in a cloud environment to ensure that no egress traffic goes to malicious IPs and domains.
The (Right) Way to Ensure Runtime Security
While we have established the importance of securing Kubernetes during runtime at the infrastructure, application and container levels, the next question is how to best monitor these environments to avoid potential security breaches.
One available architectural design to monitor pods and containers at runtime is the use of sidecars. Sidecars are additional containers deployed alongside the primary containers in a pod. They provide additional functionality such as logging, monitoring or security enforcement. While sidecars can enhance security, they also introduce increased resource load and potential risks, such as memory leaks or application failures, which must be carefully managed. They are also often difficult to deploy and manage, causing many security professionals to avoid their usage. While they may help with security monitoring, the additional weight on needed resources, difficult rollout, and potential for performance degradation make sidecars an unattractive option for many organizations.
Another option for pod and container monitoring is leveraging the Linux kernel and eBPF agents for detailed, real-time insights. eBPF is an open-source technology integrated into the Linux kernel that has become popular in recent years, offering lightweight and detailed monitoring capabilities on a level that was previously unavailable. With eBPF, organizations can monitor process execution, system calls, file access, network layer communication, and data access within Kubernetes deployments. This provides security leaders with detailed runtime insights into their Kubernetes environment, allowing them to ensure the integrity of the containerized workloads and receive visibility and deep context of Kubernetes threats and vulnerabilities at any given moment.
eBPF agents also avoid the major problems associated with sidecars since they are lightweight and use a minimal amount of CPU, allowing them to avoid placing unnecessary weight on resources or increasing the risk of application failure. eBPF agents are also significantly easier to deploy and operationalize, making them an attractive option for security leaders looking to ensure Kubernetes security without major operational overhead.
Ensuring Kubernetes security is crucial for organizations’ overall cloud security. By focusing on using runtime insights and real-time detections to secure the infrastructure, applications, and containers within Kubernetes, organizations can mitigate risks and fortify their environment against potential threats. By combining best practices such as regular patching, image validation, and security measures, with innovative runtime monitoring technologies like eBPF, organizations can confidently embrace Kubernetes while safeguarding their digital assets in an ever-evolving threat landscape.
