Cattle Not Pets: Kubernetes Fleet Management

The number of companies investing in Kubernetes has increased at a rapid rate. This has created new and complex challenges in terms of provisioning and managing Kubernetes fleets.

With this surge in adoption, it has become common for small teams to oversee fleets of clusters.

In this post, we will explore challenges companies encounter when managing Kubernetes fleets, dive into the fundamental tenets of Kubernetes fleet management, and demonstrate how Plural can assist you in your fleet management initiatives.

What is a Kubernetes fleet?

A fleet refers to any number of Kubernetes clusters where individual attention and management become impractical, and instead, you need to start controlling it in herds (you can no longer treat it like a dog, but rather need to whip it into action like cattle.)

Managing Kubernetes fleets poses a huge challenge

Maintaining Kubernetes clusters at scale is enough to drive any developer to the brink of frustration. The challenge intensifies when you consider that production-ready Kubernetes deployments involve multiple clusters across diverse environments, each running different distributions and managing various add-ons. As you expand your cluster footprint, the complexity of managing these clusters grows exponentially. While not surprising, it's an undeniable truth that demands attention.

Each element introduces additional complexity to the system, multiplied by the number of Kubernetes distributions, each with slightly different use models and capabilities that need to be understood.

Why is Fleet Management so Challenging?

Throughout the past year, we have engaged in conversations with hundreds of engineering leaders to gather insights into the common challenges faced by organizations when it comes to effectively managing large fleets.

During our conversations, we noticed five themes repeatedly pop up.

1. It’s challenging to put guardrails in place for enterprise production environments (security, compliance, access controls, etc.). From a security standpoint, it is crucial to ensure that engineers have appropriate access levels. Without implementing granular permissions, your fleet management efforts would be highly vulnerable, increasing the risk of exposing critical infrastructure to unauthorized internal stakeholders.
2. There is a lack of expertise and headcount to manage and support Kubernetes. Currently, most companies use a variety of managed cloud services instead of relying strictly on Kubernetes. Consequently, their engineering teams ultimately lack the expertise to effectively manage workloads on Kubernetes. Acquiring Kubernetes talent is costly due to the limited availability of skilled engineers in this domain.
3. Kubernetes upgrades are not predictable since you don’t know what will break before it breaks. More often than not, Kubernetes upgrades don’t happen due to the complexity of upgrading clusters as you scale your cluster footprint.
4. There are inconsistencies when deploying software between dev, staging, and production environments. This often requires hand-rolling a tedious, complex git-based release process, and is manual enough that you can’t self-serviceably expose it to other teams. Existing tooling is built primarily for simple single-cluster deployment use cases out of a unique git repository and scalability and visibility challenges often arise when working with those tools. It is extremely challenging to test and confirm that your code changes are safe for your end users.
5. Developers are currently dedicating excessive time to configuring Kubernetes clusters instead of focusing on application development. The provisioning and maintenance of these clusters can be an incredibly laborious task. Moreover, specific clusters often require certain add-ons, which necessitate installation on each cluster. When you multiply this process by the number of clusters in operation, you gain a clear understanding of the manual nature of this process and the subsequent likelihood of increased human errors.

The 5 Tenets of a Fleet Management Strategy

When developing your fleet management strategy, it's crucial to consider five key tenets. While the priority of these tenets may vary, every organization, especially those in regulated industries, will eventually need to address all five. An ideal fleet management solution would encompass these pillars within a centralized platform, allowing you to efficiently manage all your fleets through a single interface.

Fleet management rests upon five crucial tenets: Governance, Simplicity, Visibility, Automation, Security.

In the following sections, we will delve into each of these pillars, drawing insights from our conversations with numerous engineering leaders over the past year.

#1: Governance

To grow your Kubernetes fleet, it's critical to establish guardrails that ensure compliance with security policies and regulations. A Kubernetes fleet management platform should provide enterprise-ready permissions, avoiding oversharing critical infrastructure. Differentiate access levels based on roles to mitigate risks.

#2: Simplicity

Finding talented Kubernetes engineers is challenging and expensive. Kubernetes has a steep learning curve, and many developers prefer focusing on designing systems and implementing business functionality rather than handling DevOps. A fleet management platform should be easily adaptable, regardless of Kubernetes proficiency and team scale. In a recent conversation, a Head of Software Engineering referred to this scenario as a "shift-down" solution. The objective is to transfer maintenance responsibility away from staff-level engineers down the chain, allowing them to focus on core business functionality.

#3: Visibility

Managing a fleet of Kubernetes clusters requires visibility. As you expand to multiple clusters across different environments, complexity grows exponentially. Coordinating components, managing dependencies, and ensuring compatibility become intricate. Before upgrading, understanding deprecated resources and potential issues is crucial. Kubernetes has many moving pieces, making it challenging to predict what will break until it does. A single pane of glass view of clusters and services helps monitor resources and cluster health.

#4: Automation

Managing Kubernetes becomes more challenging with kubectl commands and scripts for a few clusters. But as the cluster count grows, this approach becomes impractical. Automating and standardizing routine cluster and application operations allows effortless oversight of multiple clusters, minimizing misconfigurations caused by human error. Deploying software between environments should be a fully automated self-service experience. Developers can import Git repositories and deploy services on clusters with gated promotions to ensure trustworthy code.

#5: Security

Maintaining accurate configuration and alignment across multiple clusters can be challenging. This becomes more complex with different workloads and Kubernetes distributions. Integration with existing SSO, effective authorization management through RBAC, and establishing a comprehensive audit trail are crucial. Currently, maintaining a secure environment is a manual effort, with teams managing access controls, network policies, and other security configurations. Effective auditing in Kubernetes is vital for visibility and control over cluster activity. Logging is pivotal in securing production clusters, requiring a robust audit mechanism. A complete audit trail is needed to track unauthorized events, including changes to sensitive files and their authors.

Kubernetes Fleet Management with Plural

Finding the right tools for managing infrastructure allows application teams to move quickly and focus on building their applications instead of setting up their environments. With Plural for fleet management, you gain control of your Kubernetes clusters and services. Plural gives you visibility, automation, governance, and security capabilities in an easily adaptable platform to manage the lifecycle of Kubernetes clusters across public clouds such as AWS, Azure, and GCP as well as on-prem and remote/edge locations.

Plural is a self-hosted Kubernetes fleet management platform that removes the complexity of managing Kubernetes clusters at scale. With Plural, your team can:

• Single pane of glass regardless of the cloud, on-prem, or edge environments your team uses. Plural will give your engineering organization multi-cluster visibility into your entire cluster fleet across various environments. With Plural, your engineers get self-service access to Kubernetes clusters and automated cluster lifecycle management using proven templates with guardrails included.
• Manage Kubernetes clusters and add-on upgrades in a single, intuitive interface and confidently know that upgrading a Kubernetes version won’t break anything downstream. Plural will help you with upgrading the control plane, Kubernetes add-ons, and your services. With Plural, you’ll be made aware if you have a compatible version of your add-ons for the version of Kubernetes version you are upgrading.
• Share the responsibility of managing Kubernetes tasks with a broader subset of your engineers, including those without prior Kubernetes experience. Top-tier Kubernetes talent is costly and hard to attain. Managing infrastructure shouldn’t be so challenging and pricey, and your most skilled engineers should focus on building out awesome product features to drive business value. With Plural, your team can create standard workflows to automate time-tedious and challenging tasks of configuring, and provisioning clusters across fleets in one patch rather than following the manual, error-prone process today that makes managing Kubernetes clusters challenging.

To learn more about Plural’s self-hosted fleet Kubernetes fleet management platform sign up for a custom product demo to learn more.