When operating Kubernetes, the kubectl version must be compatible with the Kubernetes cluster version. This is documented in both the Kubernetes official documentation under Version Skew Policy and explicitly mentioned in the Amazon EKS documentation.
kubectl
kubectl is supported within one minor version (older or newer) of kube-apiserver.
Example:
- kube-apiserver is at 1.32
- kubectl is supported at 1.33, 1.32, and 1.31
Note:
If version skew exists between kube-apiserver instances in an HA cluster, this narrows the supported kubectl versions.Example:
- kube-apiserver instances are at 1.32 and 1.31
- kubectl is supported at 1.32 and 1.31 (other versions would be more than one minor version skewed from one of the kube-apiserver components)
https://kubernetes.io/releases/version-skew-policy/#kubectl
Version Skew Policy
The maximum version skew supported between various Kubernetes components.
kubernetes.io
You must use a kubectl version that is within one minor version difference of your Amazon EKS cluster control plane. For example, a 1.31 kubectl client works with Kubernetes 1.30, 1.31, and 1.32 clusters.
https://docs.aws.amazon.com/en_us/eks/latest/userguide/install-kubectl.html#kubectl-install-update
Set up kubectl and eksctl - Amazon EKS
You must use a kubectl version that is within one minor version difference of your Amazon EKS cluster control plane. For example, a 1.31 kubectl client works with Kubernetes 1.30, 1.31, and 1.32 clusters.
docs.aws.amazon.com
When using Amazon EKS clusters, you might notice version strings like v1.29.0-eks-5e0fdde when running kubectl commands (Reference: https://aws.amazon.com/blogs/tech/how-to-upgrade-amazon-eks-worker-nodes-with-karpenter-drift/). According to the Amazon EKS User Guide - kubectl and eksctl setup documentation, this binary is identical to the upstream community version. While using the same binary, version strings might be customized for various reasons, such as cloud providers building binaries directly to establish trust, or for internal version management purposes (e.g., distinguishing specific environment builds - v1.32.0-internal, test environment - v1.32.0-test, development stages - v1.29.0-dev).
Let's customize the version string using the Kubernetes source directly.
Prerequisites
- Go development environment: Follow the minimum requirements at https://go.dev/wiki/MinimumRequirements
- Git: If git command is already available, no additional setup is needed. If not, refer to https://git-scm.com/
- Make tool: A historical tool (Reference: https://www.gnu.org/software/make/manual/make.html) that builds binaries using the Makefile in source code
Build Procedure
1. Clone source code
Clone the Kubernetes GitHub repository and navigate to the directory:
$ git clone https://github.com/kubernetes/kubernetes
$ cd kubernetes
2. Check existing version tags
Use the following command to view a list of Kubernetes versions from latest to oldest releases. When you see a colon (:) at the beginning, you can press spacebar to see more entries and press 'q' to exit:
$ git tag --sort=-taggerdate
3. Select and checkout version
Store the desired version tag in a local environment variable and use the "git checkout" command to switch to that version's source code. Here, we use an environment variable named TAG to store the value and use it in $TAG format to checkout the corresponding version:
$ TAG=v1.31.2 # Must use a version that exists in the git tag list
$ git checkout $TAG
4. Set custom tag
The Kubernetes build process is configured to primarily use existing tags stored in the Git repository. Rather than modifying this approach, let's remove the existing tags and apply a custom tag to the currently checked-out source. Note that we can't use arbitrary strings; we must follow the "v{SEMVER}-{CUSTOM}" format. SEMVER refers to the version numbering described at https://semver.org/, and the {CUSTOM} portion can be set with a simple string according to our purposes:
# Remove existing tags
git tag | xargs git tag -d
# Disable fetching remote tags
git config --local fetch.tags false
# Create new custom tag
git tag $TAG-aewstest
5. Verify tag
Use the following command to verify that the tag was properly applied:
$ git describe --tags --match='v*'
6. Build kubectl
Now let's use the make command to build. Run the following command:
$ make all WHAT=cmd/kubectl GOFLAGS=-v
7. Verify build result
The built binary is created in the _output folder. Let's run the kubectl binary:
$ _output/bin/kubectl version
Important Notes
As mentioned above, kubectl version must be within one minor version difference of the cluster version. When using in test environments, ensure thorough validation, and for production environments, it's recommended to use the officially distributed kubectl binary.
- Install Tools - kubectl:https://kubernetes.io/docs/tasks/tools/#kubectl
Install Tools
Set up Kubernetes tools on your computer.
kubernetes.io
- Amazon EKS - Set up kubectl and eksctl: https://docs.aws.amazon.com/en_us/eks/latest/userguide/install-kubectl.html
Conclusion
While this custom-built kubectl can have desired version strings and potentially be useful in CI/CD for cluster management through version checking, it's recommended to use official releases in production environments and limit custom builds to testing or development environments.
