Learn how to install etcdctl on your Kubernetes control plane and create etcd snapshots for disaster recovery. A complete guide for DevOps engineers managing self-hosted clusters. #CentLinux #Kubernetes #etcd
Table of Contents
Introduction
If you’re running a self-managed Kubernetes cluster, etcd is the heart of your control plane. It stores every Kubernetes object—namespaces, deployments, services, RBAC configurations, and secrets—making it the single source of truth for your entire cluster . When etcd goes down without a recovery plan, you’re not just facing an outage; you’re potentially facing a catastrophic data loss scenario.
In this guide, I’ll walk you through installing etcdctl and etcdutl on a Kubernetes control plane node, then demonstrate how to take a proper etcd snapshot. These are essential skills for anyone managing production Kubernetes clusters, whether you’re preparing for the CKA exam or building enterprise-grade disaster recovery procedures.
Let’s dive in.
Read Also: Kubernetes etcd Cluster: Complete Setup Guide 2026
Prerequisites
Before we begin, ensure you have:
- SSH access to your Kubernetes control plane node (with sudo privileges)
- A working Kubernetes cluster (kubeadm-based works best)
kubectlconfigured with cluster-admin access- Access to
/etc/kubernetes/pki/etcd/directory
Watch Now to setup a Self-Managed Kubernetes Cluster
Step 1: Verify Your Cluster is Healthy
Before touching etcd, let’s confirm your cluster is operational. Run these three commands:
kubectl cluster-info
kubectl get nodes
kubectl get pods -AWhy this matters: You never want to take a snapshot of a broken cluster. The kubectl cluster-info command shows whether the API server is reachable. kubectl get nodes confirms all nodes are Ready. And kubectl get pods -A gives you a full picture of cluster workloads .
From my experience, I’ve seen engineers rush to backup etcd without verifying cluster health first—only to discover later that the snapshot captured a corrupted state. Always verify first.
Read Also: Kubernetes Basics for Sysadmins
Step 2: Set etcd Version and Download URL
ETCD_VER=v3.5.21
GOOGLE_URL=https://storage.googleapis.com/etcd
DOWNLOAD_URL=${GOOGLE_URL}Important consideration: The version you choose here must match your cluster’s etcd server version. You can check your etcd version by examining the etcd pod manifest:
sudo cat /etc/kubernetes/manifests/etcd.yaml | grep imageUsing a mismatched etcdctl version against your etcd server can lead to subtle compatibility issues. I recommend always checking the server version first and downloading the corresponding client tools.
You might wonder why we’re using storage.googleapis.com instead of GitHub. The Google Cloud Storage endpoint is often faster and more reliable for downloads, especially in cloud environments.
Step 3: Clean Up Previous Downloads
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-testThis is basic hygiene. In production environments, I’ve seen outdated binaries in /tmp cause confusion when troubleshooting. Always start clean.
Step 4: Download and Extract the Binaries
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gzThe --strip-components=1 flag is important here. It removes the top-level directory from the extracted archive, placing the binaries directly in /tmp/etcd-download-test/ rather than in a nested subdirectory .
Step 5: Install etcdctl and etcdutl
sudo cp /tmp/etcd-download-test/etcdctl /bin/etcdctl
sudo cp /tmp/etcd-download-test/etcdutl /bin/etcdutlNow here’s something many engineers overlook: etcdctl and etcdutl serve different purposes.
- etcdctl is the network client. It talks to etcd over the network and handles day-to-day operations like managing keys, checking health, and taking snapshots .
- etcdutl operates directly on etcd data files. It handles database migration, defragmentation, snapshot restoration, and status verification .
The Kubernetes documentation explicitly recommends using etcdutl for snapshot status checks because etcdctl snapshot status has been deprecated since etcd v3.5.x .
Pro tip: While installing to /bin/ works, I typically prefer /usr/local/bin/ for manually installed binaries. It keeps custom installations separate from system package-managed binaries, making updates and troubleshooting cleaner.
Step 6: Verify the Installation
etcdctl version
etcdutl versionAlways verify. I’ve encountered situations where PATH issues or permission problems silently caused commands to fail. A quick version check confirms everything is in order.
Step 7: Clean Up Download Artifacts
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-testThis cleanup step is good practice, though in production you might want to retain the binaries for future use on other control plane nodes.
Step 8: Take an etcd Snapshot
Now for the main event. This is the command that matters most:
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save ~/etcd-backup.dbLet’s break down each flag:
| Flag | Purpose |
|---|---|
ETCDCTL_API=3 | Uses the v3 API (required for modern etcd) |
--endpoints | Points to the local etcd endpoint on port 2379 |
--cacert | CA certificate for TLS verification |
--cert | Client certificate for authentication |
--key | Client key for authentication |
Critical insight: The certificate paths matter. For a kubeadm cluster, the etcd server certificate is typically at /etc/kubernetes/pki/etcd/server.crt. However, some documentation shows using peer.crt instead . The correct choice depends on how you’re connecting:
- Use
server.crt/server.keywhen connecting to the client endpoint (port 2379) - Use
peer.crt/peer.keywhen connecting to the peer endpoint (port 2380)
For snapshot operations, you’re connecting as a client to port 2379, so server.crt is correct.
Security note: Your snapshot file contains all cluster state, including secrets. Protect it accordingly—restrict permissions, encrypt at rest, and store it outside the cluster.
Read Also: Kubernetes Secrets Encryption: A Practical Guide
Step 9: Verify the Snapshot
ls -alh ~/This shows the snapshot file size and confirms it was created. For a proper verification, use etcdutl:
etcdutl snapshot status ~/etcd-backup.db --write-out=tableThis displays the snapshot hash, revision, key count, and total size . Always verify your snapshots—I’ve seen cases where snapshots were created but were incomplete due to disk space issues.
Complete Video Tutorial:
Best Practices That I follow
1. Automate Snapshot Creation
Manual snapshots are error-prone. Set up a cron job or Kubernetes CronJob that runs every 6-12 hours. The frequency depends on your RPO (Recovery Point Objective)—how much data loss can you tolerate?
2. Store Snapshots Off-Cluster
A snapshot stored on the control plane node is useless if that node dies. Copy snapshots to external storage—S3, GCS, NFS, or a dedicated backup server.
3. Test Your Restores
A backup you’ve never restored from is not a backup. Schedule quarterly restore drills in a non-production environment. This validates both your snapshot integrity and your restore procedure.
4. Monitor Snapshot Success
Set up monitoring that alerts if snapshots fail or if snapshot files are older than expected. Silent backup failures are the worst kind.
5. Document Your Recovery Procedure
Write down the exact steps to restore from a snapshot. During an actual disaster, you won’t have time to figure it out from scratch.
Conclusion
Installing etcdctl and etcdutl on your Kubernetes control plane is straightforward, but the real value lies in understanding why you’re doing it. These tools are your lifeline when disaster strikes.
The commands in this guide give you a solid foundation for etcd backup operations. Combined with regular testing and off-cluster storage, you’ll be well-prepared for whatever infrastructure failures come your way.
Remember: in self-managed Kubernetes, etcd is your cluster. Protect it accordingly.
Frequently Asked Questions
What’s the difference between etcdctl and etcdutl?
etcdctl interacts with etcd over the network for day-to-day operations. etcdutl operates directly on etcd data files for administration tasks like snapshot restoration and defragmentation .
Can I use this procedure for managed Kubernetes (EKS, GKE, AKS)?
No. Managed Kubernetes services handle etcd internally and don’t expose restore workflows to users. This guide applies to self-managed clusters (kubeadm, on-prem, VM-based) .
How often should I take etcd snapshots?
It depends on your RPO. For most production clusters, every 6-12 hours is reasonable. High-change environments may need more frequent snapshots.
Does an etcd snapshot include persistent volume data?
No. etcd snapshots only capture Kubernetes objects and state. Application data stored in PVs requires separate backup procedures .
References
- Kubernetes Documentation: https://kubernetes.io/docs/home/
- etcd Documentation: https://etcd.io/docs/
- Ubuntu Documentation: https://docs.ubuntu.com/
Recommended Courses
If you’re eager to kickstart your journey into cloud-native technologies, “Kubernetes for the Absolute Beginners – Hands-on” by Mumshad Mannambeth is the perfect course for you. Designed for complete beginners, this course breaks down complex concepts into easy-to-follow, hands-on lessons that will get you comfortable deploying, managing, and scaling applications on Kubernetes.
Whether you’re a developer, sysadmin, or IT enthusiast, this course provides the practical skills needed to confidently work with Kubernetes in real-world scenarios. By enrolling through the links in this post, you also support this website at no extra cost to you.
Disclaimer: Some of the links in this post are affiliate links. This means I may earn a small commission if you make a purchase through these links, at no additional cost to you.
Leave a Reply
You must be logged in to post a comment.