When operating Kubernetes, the kubectl version must be compatible with the Kubernetes cluster version. This is documented in both the Kubernetes official documentation under Version Skew Policy and explicitly mentioned in the Amazon EKS documentation.
kubectl
kubectlis supported within one minor version (older or newer) ofkube-apiserver.
Example:
kube-apiserveris at1.32
kubectlis supported at1.33,1.32, and1.31
Note:
If version skew exists betweenkube-apiserverinstances in an HA cluster, this narrows the supportedkubectlversions.
Example:
kube-apiserverinstances are at1.32and1.31
kubectlis supported at1.32and1.31(other versions would be more than one minor version skewed from one of thekube-apiservercomponents)
You must use a kubectl version that is within one minor version difference of your Amazon EKS cluster control plane. For example, a 1.31 kubectl client works with Kubernetes 1.30, 1.31, and 1.32 clusters.
When using Amazon EKS clusters, you might notice version strings like v1.29.0-eks-5e0fdde when running kubectl commands (Reference: https://aws.amazon.com/blogs/tech/how-to-upgrade-amazon-eks-worker-nodes-with-karpenter-drift/). According to the Amazon EKS User Guide - kubectl and eksctl setup documentation, this binary is identical to the upstream community version. While using the same binary, version strings might be customized for various reasons, such as cloud providers building binaries directly to establish trust, or for internal version management purposes (e.g., distinguishing specific environment builds - v1.32.0-internal, test environment - v1.32.0-test, development stages - v1.29.0-dev).
Let's customize the version string using the Kubernetes source directly.
Use the following command to view a list of Kubernetes versions from latest to oldest releases. When you see a colon (:) at the beginning, you can press spacebar to see more entries and press 'q' to exit:
$ git tag --sort=-taggerdate
3.Select and checkout version
Store the desired version tag in a local environment variable and use the "git checkout" command to switch to that version's source code. Here, we use an environment variable named TAG to store the value and use it in $TAG format to checkout the corresponding version:
$ TAG=v1.31.2 # Must use a version that exists in the git tag list
$ git checkout $TAG
4. Set custom tag
The Kubernetes build process is configured to primarily use existing tags stored in the Git repository. Rather than modifying this approach, let's remove the existing tags and apply a custom tag to the currently checked-out source. Note that we can't use arbitrary strings; we must follow the "v{SEMVER}-{CUSTOM}" format. SEMVER refers to the version numbering described at https://semver.org/, and the {CUSTOM} portion can be set with a simple string according to our purposes:
Use the following command to verify that the tag was properly applied:
$ git describe --tags --match='v*'
6. Build kubectl
Now let's use the make command to build. Run the following command:
$ make all WHAT=cmd/kubectl GOFLAGS=-v
(kubectl build - start)
(kubectl build - end!)
7.Verify build result
The built binary is created in the _output folder. Let's run the kubectl binary:
$ _output/bin/kubectl version
Important Notes
As mentioned above, kubectl version must be within one minor version difference of the cluster version. When using in test environments, ensure thorough validation, and for production environments, it's recommended to use the officially distributed kubectl binary.
While this custom-built kubectl can have desired version strings and potentially be useful in CI/CD for cluster management through version checking, it's recommended to use official releases in production environments and limit custom builds to testing or development environments.
For those who have been using Windows for a long time, adding and removing programs through the Control Panel might be familiar. The "Add or Remove Programs" feature was renamed to "Programs and Features," allowing users to view, modify, and remove installed software.
0. Checking AWS CLI Version
In Windows 11, you can search for "AWS" in the "Programs and Features" menu to see the installed version of AWS CLI, which is 2.15.9.0 in this example.
(Searching with "AWS" keyword on my Windows 11 computer)
AWS CLI is a command-line tool that lets you manage AWS services. By running aws configure, you can easily set up your Access Key ID, Secret Access Key, and your target region name to control your AWS services. Initially, you might have downloaded and installed AWS CLI directly from the official website. However, checking for updates and upgrading to the latest version can be tedious. This is where winget comes in handy for managing AWS CLI upgrades.
1. Introduction to winget
winget is a package manager for Windows 10 and later, allowing you to install, upgrade, and remove software easily. You can automate software management using CLI commands. To install winget, search for "App Installer" in the Microsoft Store or visit this link and click the "Install" button.
Installing "App Installer" for winget using Microsoft Store
2. Basic winget Commands
Using winget is similar to using apt on Ubuntu. If you're familiar with the sudo apt update && sudo apt upgrade commands, you'll find winget's update and upgrade commands straightforward. Note that update is an alias for upgrade. For detailed help, visit this link or refer to the blog post (in Korean) by Microsoft MVP Soonman Kwon.
winget vs. apt (looks quite similar, doesn't it?)
3. Installing AWS CLI
While the official AWS documentation provides instructions for downloading and installing the MSI installer, you can also use winget for a hassle-free installation. Visit winget.run for the AWS CLI installation command. Note that the site may not always have the latest version information.
How to install AWS CLI from https://winget.run/pkg/Amazon/AWSCLI (Note: this this may not be the latest version on the website).
4. Upgrading and Managing AWS CLI with winget
If AWS CLI is already installed, you can upgrade to the latest version with the following command:
You can also check the installed version and the latest available version using PowerShell:
> winget list | Select-String -Pattern AWS
Freedom Scientific JAWS 2024 FreedomScientific.JAWS.2024 2024.2310.70.400 2024.240??winget
Freedom Scientific JAWS Training Table??{AE1E7553-752E-4D04-9695-EE1FB83C54AE} 25.0.2005.0
AWS Command Line Interface v2 Amazon.AWSCLI 2.15.9.0 2.16.4 winget
To update to the latest version, execute:
> winget upgrade Amazon.AWSCLI
To uninstall AWS CLI, use:
> winget uninstall Amazon.AWSCLI
5. Upgrading Multiple Packages with winget
winget allows you to upgrade multiple packages at once. To upgrade all software, use:
winget upgrade --all
or below to upgrade a specific package:
winget upgrade [package_name]
Using winget to install, upgrade, and remove AWS CLI is straightforward. Keeping your software up to date ensures you have the latest features and security patches. Regularly check for upgrades using winget to manage AWS CLI and other installed software efficiently.
This post contains the information while participating in the AEWS (Amazon EKS Workshop Study) study by CloudNet@.
1. Amazon EKS Introduction
According to the official Amazon EKS User Guide (link), Amazon Web Services (AWS) describes it as a managed service that eliminates the need to install, operate, and maintain a Kubernetes control plane. There is a diagram of the control plane in the open-source Kubernetes documentation (link) that explains the architecture of a Kubernetes cluster.
In the image regarding the control plane, there are five components: cloud-controller-manager, etcd, kube-api-server, scheduler, and Controller Manager. Instead of installing and managing these components directly, by creating a managed service like Amazon EKS, one can utilize Kubernetes nodes. For more detailed information, refer to the EKS workshop description (link).
x.y.z| x: major version, y: minor version, z: patch version
2. EKS Workshop environment and EC2 bastion VM configuration
For this study, we prepared our AWS accounts in advance following the "Start with an AWS Account" section in the EKS workshop. For setting up the practical environment, tasks ranged from setting up AWS Cloud9, installing kubectl, to installing eksctl. Thanks to the AWS CloudFormation prepared by our study leader, Mr. Kasida, we were able to participate in the study comfortably. Based on March 2024, the time of conducting this study, we chose version v1.28 for EKS, which supports add-ons and is among the most compatible and validated versions with numerous applications within the K8s ecosystem. To understand the AWS environment we are working within during the study, we referenced AWS architecture icons and schematically represented it as follows..
After downloading the CloudFormation template as described below, we tried executing it using the AWS CLI (link).
$ curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-1week.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 10373 100 10373 0 0 180k 0 --:--:-- --:--:-- --:--:-- 180k
$ aws cloudformation deploy --template-file myeks-1week.yaml --stack-name myeks --parameter-overrides KeyName=kp-ian SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 --region ap-northeast-2
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - myeks
After executing the following command, you can find out the IP address, and by using this IP address to run SSH, you can access the Shell and proceed with the subsequent tasks. The SSH ID and Password to connect can be found in the above CloudFormation template file, so please refer to it.
aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[*].OutputValue' --output text
Once you've accessed the working EC2 instance, it's necessary to configure IAM user credentials. For the convenience of the practice, enter the credentials of an IAM User with administrator privileges.
[root@myeks-host ~]# aws ec2 describe-instances
Unable to locate credentials. You can configure credentials by running "aws configure".
[root@myeks-host ~]# aws configure
AWS Access Key ID [None]: AKI..........
AWS Secret Access Key [None]: FQ.......................
Default region name [None]: ap-northeast-2
Default output format [None]: json
[root@myeks-host ~]# aws ec2 describe-instances
{
"Reservations": [
{
"Groups": [],
"Instances": [
{
"AmiLaunchIndex": 0,
"ImageId": "ami-025cebb6913219d99",...........
3. Cluster creation using eksctl
In the EKS workshop content (link), clusters are created using the eksctl command with yaml files. However, it's also possible to pass basic options directly to the eksctl command in the appropriate parameter format, and this method was explored in the study. The necessary option values were stored and utilized as environment variables.
3.1. Environment variables
The $AWS_DEFAULT_REGION and $CLUSTER_NAME environment variables are already prepared on the working (bastion) EC2 instance. We checked these and then went ahead to set up the remaining environment variables.
It will take about 15-20 minutes, so let's wait for a bit. In the meantime, opening another terminal and executing the following command will help check whether the cluster has been created.
while true; do aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output text ; echo "------------------------------" ; sleep 1; done
Once the cluster creation is complete, the terminal status will change as follows.
You can also check the deployed EKS details in the AWS console (if the console hasn't refreshed, try clicking the refresh button).
After the EKS cluster creation is complete, you can execute various commands on the EKS cluster using the kubectl command. During the study, we tried various things and confirmed many aspects, but I intend to document just one of those in the blog.
4. Check the created EKS cluster - endpoint access (Public -> Public and private)
To check the information of the EKS cluster, you can use the command "kubectl cluster-info".
(awesian@myeks:N/A) [root@myeks-host ~]# eksctl get nodegroup --cluster $CLUSTER_NAME --name $CLUSTER_NAME-nodegroup
CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE
myeks myeks-nodegroup ACTIVE 2024-03-09T18:02:34Z 2 2 2 t3.medium AL2_x86_64 eks-myeks-nodegroup-eac71230-bb27-1b00-6c14-e2c96dfc5646 managed
(awesian@myeks:N/A) [root@myeks-host ~]# kubectl cluster-info
Kubernetes control plane is running at https://088CD22A78682CF5F017CFEE329E3C1A.gr7.ap-northeast-2.eks.amazonaws.com
CoreDNS is running at https://088CD22A78682CF5F017CFEE329E3C1A.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Additionally, the "eksctl get cluster" command can also be used to check the information. One notable point was that the created endpoint was public. Being public means that the endpoint is accessible over the network. To proceed with actions like creating Pods through this endpoint, additional authentication is required. However, for simple tasks like version checking, access to the created EKS cluster was possible without any separate authentication when the endpoint is public.
Even when checked from the console, the API server endpoint access is listed as "Public".
Let's change the API server endpoint access to "Public and Private". To detect changes, we can use a total of three terminals. Two of these terminals will be used for monitoring purposes.
# Terminal A - for monitoring
APIDNS=$(aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint | cut -d '/' -f 3)
dig +short $APIDNS
while true; do dig +short $APIDNS ; echo "------------------------------" ; date; sleep 1; done
# Terminal B - for another monitoring
N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})
N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})
while true; do ssh ec2-user@$N1 sudo ss -tnp | egrep 'kubelet|kube-proxy' ; echo ; ssh ec2-user@$N2 sudo ss -tnp | egrep 'kubelet|kube-proxy' ; echo "------------------------------" ; date; sleep 1; done
# Terminal C - Public(with only one IP address)+Private. It will take 8-10 minutes.
aws eks update-cluster-config --region $AWS_DEFAULT_REGION --name $CLUSTER_NAME --resources-vpc-config endpointPublicAccess=true,publicAccessCidrs="$(curl -s ipinfo.io/ip)/32",endpointPrivateAccess=true
After waiting, you can observe in Terminal A that the section previously displaying 2 public IPs has suddenly changed to internal network subnets.
The lack of change on the right side may be because, with both Public and Private options activated, there's no need to terminate existing network connections that kube-proxy and kubelet have already established.
After the change, executing "kubectl" commands may not work. Attempting to run it could result in an error message, indicating that the visible IP address is not a Public IP. This implies that, with the cluster settings altered, the Endpoint now returns a Private IP.
(awesian@myeks:N/A) [root@myeks-host ~]# kubectl get node -v=6
I0310 03:44:52.743735 18383 loader.go:395] Config loaded from file: /root/.kube/config
I0310 03:45:23.611890 18383 round_trippers.go:553] GET https://088CD22A78682CF5F017CFEE329E3C1A.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 in 30861 milliseconds
I0310 03:45:23.612005 18383 helpers.go:264] Connection error: Get https://088CD22A78682CF5F017CFEE329E3C1A.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500: dial tcp 192.168.1.51:443: i/o timeout
Unable to connect to the server: dial tcp 192.168.1.51:443: i/o timeout
(awesian@myeks:N/A) [root@myeks-host ~]# kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: dial tcp 192.168.2.122:443: i/o timeout
Connection timeout implies that you need to make additional settings in the EKS Control plane security group to enable access to the subnet. Using the following command, an additional rule was set in the node security group to allow access to the node (pod) from myeks-host.
# EKS ControlPlane Security Group ID
aws ec2 describe-security-groups --filters Name=group-name,Values=*ControlPlaneSecurityGroup* --query "SecurityGroups[*].[GroupId]" --output text
CPSGID=$(aws ec2 describe-security-groups --filters Name=group-name,Values=*ControlPlaneSecurityGroup* --query "SecurityGroups[*].[GroupId]" --output text)
echo $CPSGID
# Add a rule to enable connection from myeks-host to nodes (pods) in the security group
aws ec2 authorize-security-group-ingress --group-id $CPSGID --protocol '-1' --cidr 192.168.1.100/32
Also, let's change the settings for kubelet and kube-proxy to connect to private IP addresses. Run the following command:
# kube-proxy rollout
kubectl rollout restart ds/kube-proxy -n kube-system
# Kubelet is applied by running systemctl restart kubelet on individual nodes. The $N1 and $N2 environment variables must be set.
for i in $N1 $N2; do echo ">> node $i <<"; ssh ec2-user@$i sudo systemctl restart kubelet; echo; done
After running the first command above, you can see that the connection to kube-proxy is made with a private IP.
If the second command runs successfully, you can see that the kubelet is also connecting to the private IP.
5. Resource deallocation
After completing the exercise, be sure to delete resources to minimize unnecessary costs.
Deleting an Amazon EKS cluster (takes about 10 minutes): eksctl delete cluster --name $CLUSTER_NAME
To delete the AWS CloudFormation stack after the above process is completed: aws cloudformation delete-stack --stack-name myeks
