"Software Defined" to Everything

This post documents the hands-on exercises I completed in my own environment, based on Chapters 4–5 of the book, as part of the 2026 Machine Learning Platform Engineering book study.

I worked through the process of connecting MLflow, Feast, and Kubeflow Pipelines, as discussed in Chapters 4–5, to see where model files and feature data are stored and how the next step reads them.

For the environment setup and execution sequence, I referred to John Kim's “04. Connecting Experiments, Features, and Pipelines” and the ml-platform-book-env repository. Since I use Ubuntu 26.04.1 LTS on WSL2, I adapted the PowerShell commands to run in Bash. This post records the preparation needed in WSL and the results I actually checked while reproducing the book's exercises with that guide.

1. Overview

Here is the overall flow:

Chapter 4: Read CSV → Train three models → Track results in MLflow → Register a model
Chapter 4: Upload feature parquet files → Register Feast definitions → Retrieve historical/online features
Chapter 5: Read target users → Retrieve Feast features → Predict with the registered model → Save parquet

“Batch inference” means reading multiple inputs together and saving their predictions to a file. This differs from an API server that responds immediately to each individual request.

Breaking down the three tools this way made their roles clearer.

Tool The question it answers in this exercise What it produces or stores

A feature is an input to a model. This example uses information such as education, occupation, and relationship status. An artifact is a file created during execution, such as a model file, a PNG chart, or a parquet file passed between steps.

2. Preparing WSL: Separating System Python from the Exercise Environment

My WSL environment already had Python 3.14.4, git, and uv installed. I kept the exercise environment separate to avoid conflicts with the existing setup. Docker, kind, and kubectl were not installed in WSL, so I prepared the environment by installing Docker Engine inside Ubuntu rather than using Docker Desktop.

Install the required packages in a WSL terminal:

sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"

Docker group membership changes take effect in a new login session. If you want to continue in your existing WSL terminal, run newgrp docker to enter a new shell with the group applied, then check docker info.

kind creates a small Kubernetes cluster inside Docker containers. It lets you practice with Kubernetes Pods and Services without preparing several separate servers. kubectl sends commands to that cluster.

mkdir -p ~/.local/bin
curl -fsSL https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64 \
  -o ~/.local/bin/kind
curl -fsSL https://dl.k8s.io/release/v1.32.2/bin/linux/amd64/kubectl \
  -o ~/.local/bin/kubectl
chmod +x ~/.local/bin/kind ~/.local/bin/kubectl
export PATH="$HOME/.local/bin:$PATH"

The versions used in this run were as follows. I left Ubuntu's system Python unchanged and used a separate Python 3.11 virtual environment for the book exercises.

Item Verified value

Checking the WSL operating system and Python environment in JupyterLab

A virtual environment is a directory containing the Python interpreter and packages used for a particular project. Keeping it separate from Ubuntu's Python makes it easier to match the versions required by the book.

3. Preparing the Repositories and Cluster

I kept the Docker images and Python packages in WSL's Linux filesystem. The working directory for this exercise was ~/project/ml-platform-lab/ml-platform-book-env. You can create the project under your own home directory when reproducing it.

mkdir -p ~/project/ml-platform-lab
cd ~/project/ml-platform-lab
git clone https://github.com/johklo/ml-platform-book-env.git
cd ml-platform-book-env

git clone https://github.com/practical-mlops/chapter-4.git upstream/chapter-4
git clone https://github.com/practical-mlops/chapter-5.git upstream/chapter-5

uv venv .venv-book --python 3.11
uv pip install --python .venv-book -r book/requirements.txt
source .venv-book/bin/activate

python bookenv.py check
python bookenv.py up

Running python bookenv.py up creates the kind cluster and prepares KFP, storage, PostgreSQL, Redis, and the MLflow server. The first run needs some time to download the container images.

This exercise uses a standalone KFP installation, rather than the full Kubeflow platform. It does not set up Kubeflow Notebook servers or an integrated authentication interface. JupyterLab runs directly in WSL.

When a Pod Restarted During Setup

During this run, metadata-grpc-deployment showed CrashLoopBackOff. This means a container is repeatedly failing and being restarted.

kubectl --context kind-book-env -n kubeflow get pods
kubectl --context kind-book-env -n kubeflow \
  logs deploy/metadata-grpc-deployment --tail=20

The logs contained this message:

MySQL database was not initialized. Please ensure your MySQL server is running.

At that point, the MySQL Pod was still in ContainerCreating. Another service had tried to connect before the image download and database preparation were complete. Before changing manifests or database authentication settings, I checked whether the dependency was ready.

Once MySQL was ready, the restarted metadata service recovered to 1/1 Running. I did not need to change the configuration or manually recreate the Pod. Finally, the up command printed The environment is ready. Next: python bookenv.py forward.

Service status checked with kubectl get pods after setup completed

READY 1/1 means the Pod's single container is ready. RESTARTS counts how many times it has restarted; it does not, by itself, mean the service is currently failing. I therefore checked both its status and readiness.

4. Opening WSL Services in a Windows Browser

In a separate WSL terminal, activate the virtual environment and start port forwarding:

cd ~/project/ml-platform-lab/ml-platform-book-env
source .venv-book/bin/activate
python bookenv.py forward

Port forwarding creates a connection between a port on your PC and a service inside the cluster. It lets the browser connect through localhost without knowing the Kubernetes service address directly.

Browser / client address Purpose

I used port 8890 for JupyterLab to avoid conflicting with the storage browser on port 8888.

jupyter lab --notebook-dir book --port 8890 --ip 127.0.0.1 --no-browser

Open the URL containing the token printed by JupyterLab in your Windows browser. In this exercise, I checked the WSL services using Chrome on Windows.

Why Create Several Buckets?

A bucket is a top-level container for files in S3 storage. Even when using the same storage service, separating training CSVs, model files, feature definitions, and final outputs makes it easier to see what each step reads and writes.

python bookenv.py buckets

Bucket What it stores in this exercise

Exercise buckets created in this cluster

5. Chapter 4 — MLflow: Recording Training Results So They Can Be Found Again

I ran ch04/01_mlflow_tracking.ipynb from top to bottom in JupyterLab.

The first part prepares data paths and storage connection settings. It then reads the income classification CSV and examines the category distributions. The dataset in this run had 30,162 rows and nine columns.

The income-classifier Experiment in MLflow contains one data exploration run and three model training runs. An Experiment is similar to a folder grouping related executions, while a Run is the record of one execution within it.

The three models trained were Decision Tree, Random Forest, and XGBoost. scikit-learn and XGBoost perform the training; MLflow records the settings, scores, and files.

Model MLflow run Test ROC AUC Test accuracy

ROC AUC measures how well a classifier distinguishes between the two classes. Higher values are preferable here. However, the original code does not fix the data split or the random seeds for some models. Running the same commands may produce scores that do not exactly match this post or the reference guide.

Also, the final XGBoost cell calculates AUC using the 0/1 predictions of a binary:hinge model. It is difficult to treat this as a comparison under the same conditions as the probability-based AUC of the other two models. This table records the results produced by the book's code.

Data exploration and training runs recorded in MLflow during this execution

Where Are the Model Files?

Run names, settings, metrics, and other records displayed in MLflow are stored in PostgreSQL. The actual model and chart files are stored in the mlflow-artifacts bucket. Being able to open the UI does not guarantee that the model files were saved correctly, so the artifacts also need to be checked.

The original notebook converts categorical inputs into numeric columns and saves their order in column_list.pkl. For example, if education and occupation are expanded into several 0/1 columns, inference must supply the same columns in the same order. The Chapter 5 inference code downloads this file later.

Registering the Model

The final cell of the original notebook finds the run with the highest test AUC and registers it as random-forest-classifier. I then ran the cell added by the guide to set its stage to Production.

The selected source run was fearless-moose-660, with a test AUC of 0.857340. The model was registered as version 1 / Production. Its source run ID was 489e1a27bc8b428c9b6f8fe9af0248e3.

The registered model's version and source run

Production is the label used by this code to find a model version. Assigning it does not automatically deploy a prediction API. MLflow 3 emits a FutureWarning about stages, but I checked that the connection still worked with the book's code.

6. Chapter 4 — Feast: Distinguishing Feature Definitions from Feature Values

Next, I ran ch04/02_feast_feature_store.ipynb.

Reading a training CSV and using Feast are separate processes. The training notebook in this book example reads CSV files directly, and Feast is prepared separately afterward. Describing this run as “using Feast for training from the start” would therefore be inaccurate.

The Feast workflow in this exercise was:

Upload source parquet files
  → Register Entity / FeatureView definitions
  → Retrieve historical features
  → Load values into Redis
  → Retrieve online features

Term A plain-language explanation In this example

apply and materialize Do Different Things

feast apply registers feature definitions in the Registry. Its logs showed messages preparing the three FeatureViews:

Deploying infrastructure for relationship
Deploying infrastructure for demographic
Deploying infrastructure for occupation

This does not mean all feature values have been loaded into Redis. feast materialize reads the source data and loads it into the online store.

I used the same period as the original README:

feast materialize 2022-09-16T00:00:00 2023-09-17T00:00:00

If you run this command directly, run it from book/ch04/feature_repo, which the notebook creates. The notebook's helper cell automatically sets that working directory.

Why Separate Historical and Online Retrieval?

Historical retrieval uses a user ID together with the reference time for the prediction to retrieve features appropriate to that time. Online retrieval quickly returns the latest values loaded into Redis.

If today's latest values are used to reproduce a prediction from the past, information that was unavailable at the time may be introduced. The Chapter 5 batch inference pipeline's use of get_historical_features() is related to this distinction.

The historical retrieval returned Sex=Male for user 9f2ac416-06e1-44a0-87bd-d4787c85bf66 as of 2023-01-30 10:59:42+00:00. The online retrieval also returned Sex=Male, but for a different user specified in the original script: 74b0638a-5742-4d1a-a03f-483794d7aa7c. These two examples do not compare changes in the same user's values. The None in the output is the return value printed by print(df.info()); it does not mean Sex is missing.

Retrieval results checked in the original Feast notebook

After completing the notebook, the local configuration files registry_local.db and feature_store.yaml are stored in feature-registry.

7. localhost in WSL and localhost in a Pod Are Different

One of the most memorable connection details in this exercise was the address.

The notebooks running in WSL connect to localhost:5000 and localhost:9000 through port forwarding. Inside a Kubernetes Pod, however, localhost refers to that Pod itself.

Execution location MLflow connection address

For the S3 API, the Pod uses minio-service.kubeflow.svc.cluster.local:9000.

This is why the local Feast configuration is not passed unchanged to the pipeline. During Chapter 5 preparation, the feature definitions are registered again in a Registry for the cluster, and a separate configuration file, feature_store_k8s.yaml, is uploaded.

Keeping the two configurations separate makes it easier to start with the execution location when investigating a problem that “works in the notebook but fails in the pipeline.”

8. Chapter 5 — KFP: Running Batch Inference in Four Steps

First, use the preparation script and Dockerfiles provided by the guide:

python book/ch05/prepare.py

docker build -t book-ch5/feast-job:0.1 \
  -f book/ch05/Dockerfile.feast-job upstream/chapter-5/scripts/feast
docker build -t book-ch5/components:0.1 \
  -f book/ch05/Dockerfile.components upstream/chapter-5

python bookenv.py load book-ch5/feast-job:0.1
python bookenv.py load book-ch5/components:0.1

prepare.py adapts the original components' image names and Redis settings to this environment. Building an image with Docker and loading it into the kind node are separate operations, so I ran load after the builds.

The guide's Dockerfiles replace an old Debian buster base image with bookworm and align the inference environment's scikit-learn version with the training environment's version, 1.5.2. These fixes were already included in the reference guide; they were not newly discovered or implemented during my exercise.

Feature Definitions and Input Files for the Cluster

kubectl --context kind-book-env apply -f book/ch05/feast-job.yaml
kubectl --context kind-book-env -n kubeflow wait \
  --for=condition=complete --timeout=300s job/feast-job
kubectl --context kind-book-env -n kubeflow logs job/feast-job

python book/ch05/upload_inputs.py

cd upstream/chapter-5
python inference_pipeline.py
cd ../..
grep 'image:' upstream/chapter-5/income_classifier_pipeline.yaml

feast-job registers the feature definitions used by the Pods. upload_inputs.py uploads the cluster configuration and the inference target file. The final Python command compiles the pipeline definition into executable YAML.

All four components referenced book-ch5/components:0.1 in this run. Creating the YAML does not execute the tasks. The pipeline must be registered with the KFP server and a run started before execution can be checked.

Registering and Running the Pipeline

The reference guide uploads the YAML and starts the run through the UI. In this exercise, I used the Python SDK in WSL to register and submit the same YAML and parameters, then checked the resulting run and graph in the KFP UI. I disabled caching so all four components actually ran.

import json
from pathlib import Path
import kfp

# Run from the repository root
client = kfp.Client(host="http://localhost:8080")
pipeline_id = client.get_pipeline_id("income-classifier-pipeline")
if not pipeline_id:
    pipeline = client.upload_pipeline(
        "upstream/chapter-5/income_classifier_pipeline.yaml",
        pipeline_name="income-classifier-pipeline",
    )
    pipeline_id = pipeline.pipeline_id

version = client.list_pipeline_versions(pipeline_id).pipeline_versions[0]
experiment = client.create_experiment("income-classifier")
params = json.loads(Path("book/ch05/run-parameters.json").read_text())
run = client.run_pipeline(
    experiment.experiment_id,
    "income-classifier-wsl-run-1",
    pipeline_id=pipeline_id,
    version_id=version.pipeline_version_id,
    params=params,
    enable_caching=False,
)
print(run.run_id)

I also encountered this error while using the SDK to run the registered pipeline:

both `pipeline_id` and `version_id` are required

My submission code had passed only the pipeline ID. After retrieving the registered pipeline's version ID and passing it to run_pipeline() as well, the run was created. This was an issue in the SDK submission code added for this exercise, rather than a failure in the book's inference components.

The main parameters were as follows. For the remaining values, I used the guide's book/ch05/run-parameters.json.

Parameter Value

KFP Experiments and MLflow Experiments are separate even when they have the same name. The former groups pipeline executions; the latter groups model training records.

The four execution steps were:

Step What it does

KFP does not share one step's Python memory with the next. Each task runs in a separate Pod, and intermediate results are passed as files. The arrows in the graph represent file input/output connections as well as task order.

The run ID was 06e5dad7-2ae6-4e46-95aa-1379f72e5a58. All four components showed green check marks in the KFP UI, and kubectl get workflows showed income-classifier-pipeline-j4b4j as Succeeded.

While the pipeline was running, I could see the first component executing.

The KFP pipeline during execution

The KFP batch inference graph executed in this WSL cluster

Checking Both the UI and kubectl

I also ran the following queries during and after execution. The Workflow name below is the actual name generated for this run. For another run, check your own name in the output of get workflows and substitute it.

kubectl --context kind-book-env -n kubeflow get workflows
kubectl --context kind-book-env -n kubeflow get pods \
  -l workflows.argoproj.io/workflow=income-classifier-pipeline-j4b4j
kubectl --context kind-book-env -n kubeflow get jobs

Workflow success and completed task Pods checked with kubectl

The task Pods showed 0/2 Completed. This was not a readiness failure: both containers had finished their work and exited. These Pods need to be interpreted differently from service Pods, such as the MLflow server, which keep running.

For the model inference task, I checked the main container's logs:

kubectl --context kind-book-env -n kubeflow logs \
  income-classifier-pipeline-j4b4j-system-container-impl-3969511708 \
  -c main --tail=18

Model file downloads and result uploads checked with kubectl logs

The logs showed artifact downloads, the inference output upload, publish success, and sub-process exited with error="<nil>". Rather than relying on a single log line, I checked the Workflow status and the final output file as well.

9. Checking the Actual File After the Success Indicator

Even when the KFP UI shows green check marks, the intended output still needs to be checked in the final bucket.

python book/ch05/show_predictions.py

Reading the file produced these results:

rows, columns: (30162, 91)
model input columns: 90
user_id in output: False
missing prediction values: 0

The file size was 282,569 bytes. Prediction values ranged from 0.0 to 1.0, with a mean of approximately 0.2494. This mean is a summary statistic of the file, not an accuracy or model quality metric.

The reference guide reports 92 columns, but this run produced 91 columns. The saved column_list.pkl contained 90 model input columns, and the output consisted of those columns plus one prediction column. I recorded the measurements from this file rather than copying the figures from the reference post.

The prediction file from this run in the final bucket

Output data and dimensions checked by reading the parquet file

Parquet is a binary format for storing tabular data. Garbled characters or PAR1 appearing when a file is opened as text in a browser do not automatically mean it is corrupt. Reading the actual table with a tool such as pandas is a more reliable check.

Successful Execution and Business Usability Are Separate Questions

The original output did not contain user_id. The inference code removes the ID from the model inputs and does not add it back to the results. Excluding an ID from model inputs and making it impossible to identify the user associated with a prediction are separate issues.

For a real service, the output should also retain the user ID, reference time, and specific model version.

Although the prediction column is named Predicted_Income_Class, the original code stores the second class value returned by predict_proba(). A value such as 0.7 is therefore an estimated probability, not a class number. The code does not include a threshold for converting the probabilities into 0/1 decisions.

The original code selects a model using test scores. For production model selection, a separate approach should be considered: choose candidates using a validation set and reserve the test set for the final assessment. Producing a prediction file in this exercise does not, by itself, validate the model's generalization performance or suitability for a business use case.

10. What I Verified in This WSL Exercise

This run focused on checking whether the book's model training, feature preparation, and batch inference workflow could also be connected in WSL.

I prepared Docker Engine and the CLI tools inside Ubuntu, then ran the book's two notebooks in a Python 3.11 virtual environment. I checked the four MLflow runs and registered model, Feast historical and online retrieval, the success of all four KFP tasks, and the final prediction file in S3. I also checked the additional preparation required for WSL and the address differences between execution locations.

These are the points I should be able to explain again:

1. MLflow tracks training settings, results, and model files; it does not train the model on your behalf.
2. Having definitions in the Feast Registry does not mean the values are already in Redis. apply and materialize need to be distinguished.
3. Historical retrieval requires a reference time. The latest online values and values from a past point in time serve different purposes.
4. WSL notebooks and Kubernetes Pods use different connection addresses. Check the execution location before copying localhost.
5. After KFP reports success, check the final file's dimensions, contents, and user identifier columns.

The central lesson from this study was checking where models and data are stored and which addresses and file names the next step uses to retrieve them, beyond installing the three tools individually.

Reconnecting or Shutting Down

I left the environment in place so the results could be checked again. To reconnect, check the cluster status and start forward and JupyterLab.

cd ~/project/ml-platform-lab/ml-platform-book-env
source .venv-book/bin/activate
python bookenv.py status
python bookenv.py forward

In another terminal:

cd ~/project/ml-platform-lab/ml-platform-book-env
source .venv-book/bin/activate
jupyter lab --notebook-dir book --port 8890 --ip 127.0.0.1 --no-browser

Pressing Ctrl+C stops the port forwarding or JupyterLab connection process. python bookenv.py down deletes the cluster and the exercise records inside it, so I did not run it while keeping the results available for review.

References

• Book: Machine Learning Platform Engineering, Chapters 4–5
• Exercise guide: John Kim's “04. Connecting Experiments, Features, and Pipelines”
• Environment repository: johklo/ml-platform-book-env, commit e9048a3cdc32da84782be3682617754db3fe8f5f
• Chapter 4 code: practical-mlops/chapter-4, commit b67a8228336781c933dd68bb7dc520bd484c3371
• Chapter 5 code: practical-mlops/chapter-5, commit 332915b2f617e7fed8646b7c096653c87eed9d30

As part of a 2026 book study on machine learning platform engineering (Korean), I have extended to summarize the operational requirements for building an MLOps platform on AWS. MLOps is a set of practices that connects data preparation, training, evaluation, deployment, monitoring, and retraining, making the entire process repeatable. This post explores the four principles and Levels 0, 1, and 2 introduced by AWS, then maps those requirements to the roles and relationships of Kubernetes-based components in an Amazon EKS environment.

MLOps combines ML, short for machine learning, with Ops, short for operations. From an operational perspective, it covers a broad range of activities, including repeated model training, inference, and the associated feature engineering. The book introduces MLOps maturity, and this post brings that discussion together with AWS’s maturity levels while considering how the book’s Kubernetes-based examples translate to Amazon EKS.

1. What Are We Operating?

From an MLOps platform perspective, two questions are central: how to support the machine learning lifecycle—from problem definition and data preparation to training, evaluation, deployment, and monitoring—and how to operate that lifecycle reliably and repeatedly.

Even after training a model that achieves the desired evaluation score, there is still much to check before using it in a service. Is the preprocessing used during training applied consistently at inference time? Which model version is deployed? Can the service handle real requests with the required performance? We also need a process for reevaluating and updating the model when its input data or operating environment changes.

The scope of an MLOps platform is therefore "the entire process through which models are created, used, and improved".

This process begins with defining the problem. The design will differ depending on whether simple rules are sufficient, whether the necessary training data is available, and which types of errors matter most. Model evaluation criteria must also be tied to the problem being solved.

For example, in fraud detection, the cost of incorrectly blocking a legitimate transaction differs from the cost of missing a fraudulent one. In a recommendation system, an improvement in offline evaluation scores does not necessarily mean that real users respond more positively. The platform should retain the data and execution results needed to make these judgments.

After deployment, we need to observe both service health and prediction quality. An API can return HTTP 200 while producing incorrect predictions. Alongside request latency and error rates, there should be a process for evaluating prediction quality once ground-truth labels become available.

The relationships a platform needs to support can be summarized through the following questions:

• Which data and code were used to build the model?
• Which configuration and execution environment were used?
• What was validated before deployment?
• Which model version is currently serving predictions?
• Which version can we return to if something goes wrong?

Being able to answer these questions from execution records makes reproducibility, incident investigation, and recovery possible.

2. Four AWS MLOps Principles for Repeatable Operations

AWS identifies four core principles: Version control, Automation, Continuous X, and Model governance. These principles apply throughout the lifecycle. As maturity increases, both the scope of what is managed and the extent of automation expand.

2.1. Version Control: Record the Conditions That Produced the Model

Knowing the Git commit of the code is not enough to expect the same result if the training data or configuration differs. Each model release should be linked to its dataset identifier, preprocessing method, training configuration, execution environment, model artifacts, and evaluation results. Connecting these records to the inference container image and deployment configuration makes it possible to trace the origins of the model currently serving predictions.

For example, S3 Versioning allows multiple versions of individual objects to be retained. If a training dataset consists of multiple files, a manifest containing the bucket, object key, and version ID of each file can identify the dataset used for a particular run.

Dataset versions and train/validation/test splits should be distinguished. A dataset version represents changes to the collected or processed data. A split determines how data within that version is assigned to different purposes. Recording the split rules and when the split was created provides a basis for comparing evaluation results.

The connections among data, code, executions, and models form their lineage. With lineage, we can rebuild a model, investigate which models were affected by incorrect data, and recover to a previous release. AWS’s ML operations guidance also emphasizes versioning code, configuration, models, and infrastructure alongside maintaining reproducible environments.

2.2. Automation: Define Inputs and Acceptance Criteria

Automation can begin by arranging commands into a sequence. For operational use, however, each stage also needs clearly defined inputs, outputs, and success or failure conditions. Training should stop if data validation fails, and a model that does not meet evaluation criteria should not be promoted to a deployment candidate. It should be possible to identify the failed stage and its cause, adjust the necessary conditions, and run the process again.

When production approval requires human review, reviewers should receive the evaluation results and details of the changes, and the approval decision should be recorded. Keeping automated work and human judgment within the same workflow reduces repetitive tasks while preserving accountable decision-making.

Managing execution environments and infrastructure configuration as code also helps reduce differences among development, validation, and production environments and makes them easier to recreate. Consistent pipelines and environments are the foundation for trusting automated results.

2.3. Continuous X: Connect Operational Results to the Next Change

AWS describes ongoing activities through CI, CD, CT, and CM: Continuous Integration, Continuous Delivery, Continuous Training, and Continuous Monitoring.

These activities form a feedback loop. If input data changes or prediction quality declines in production, the next step is to investigate the cause and consider retraining. A new candidate model proceeds to the deployment process only after passing the evaluation criteria.

Retraining and promotion to production require different acceptance criteria. Training on new data alone does not provide sufficient justification for replacing the existing model.

2.4. Model Governance: Record Responsibilities and Approval Rationale

Model governance manages model ownership, review and approval criteria, access permissions, and audit records.

It should be clear who defines the model’s business objectives, which evaluation results justify deployment approval, and how access to sensitive data and models is controlled. The people responsible for making decisions and responding to operational issues should also be identified.

Operational accountability becomes easier to explain when approval and deployment histories are linked to the following information:

• Model owners and operational maintainers
• Evaluation criteria and validation results
• Approvers and approval times
• Deployed models and their execution environments
• Issues discovered in production and the actions taken

Governance applies throughout the lifecycle, from data usage and model review to deployment, feedback, and retirement.

3. ML Workflow Automation: Levels 0, 1, and 2

The Levels 0, 1, and 2 introduced by AWS describe the degree of automation in an ML workflow. To understand them, it is more useful to examine what is being operated and how changes are handled than to look only at which tools have been installed.

3.1. Level 0: Manually Handing Over a Trained Model

At Level 0, people perform data preparation, training, validation, and the transitions between these stages. The trained model is then handed over to an engineering team.

This can also happen in an EKS environment. For example, a data scientist might train a model in a notebook and hand over the model file. An engineer then integrates it into an API, packages it as a container image, and deploys it.

Deployment to Kubernetes or EKS may already be automated, and the API delivery process may have sophisticated testing and deployment procedures. To assess the automation level of the ML lifecycle, however, we must also examine the preceding data preparation, training, and evaluation stages.

For example, when new data arrives, someone may still need to determine which code to run and in what order, check whether the model matches its preprocessing code, and associate evaluation results with the correct model. These manual connections are potential areas for improvement.

At Level 0, a useful question for reviewing current operations is:

To recreate the model currently in production under the same conditions, whom would we need to ask, and what information would we need?

If the necessary information resides on a particular person’s laptop or in their memory, a practical starting point is to move it into code, configuration, and execution records.

3.2. Level 1: Repeatedly Executing a Training Pipeline

At Level 1, the ML pipeline is automated so that training can be repeated with new data. The operational focus expands from delivering a trained model to operating a repeatable training pipeline.

A defined process runs in response to triggers such as data becoming ready or a scheduled time. The workflow uses consistent pipelines across development, validation, and production environments and delivers the resulting models to prediction services.

The key question at this stage is whether we can explain a single pipeline run. We need to connect the input dataset, validation steps, training configuration, resulting model, and evaluation results.

For an EKS-based setup, Kubeflow Pipelines is one option for defining and executing container-based ML workflows. Dividing a pipeline into components makes it possible to manage each stage’s inputs, outputs, execution order, and status.

Experiment tracking and model management have distinct roles:

• MLflow Tracking can be configured to record each run’s data, parameters, metrics, and artifacts.
• MLflow Model Registry manages registered models, versions, and aliases and links them to training runs.

Together, these records make it possible to trace a selected model back to the experiment that produced it.

Models should be evaluated against predefined criteria using data that was not used for training. When appropriate, candidate models should also be compared with the model currently in production. Managing candidate generation separately from promotion to production preserves the rationale for deployment decisions while allowing retraining to run repeatedly.

3.3. Level 2: Managing the Training Pipeline Itself Through CI/CD

At Level 2, pipeline source development, component builds, testing, deployment, prediction service operations, and monitoring are connected. AWS also describes an orchestrator for coordinating multiple ML pipelines and a model registry for tracking multiple models as components at this level.

The distinction from Level 1 becomes clear when the method of building a model changes. Providing a new dataset to an existing pipeline repeats an already defined procedure. Changing feature calculation logic, replacing a training algorithm, or adding an evaluation stage changes the pipeline itself. Those changes also need testing and release procedures.

The Level 2 workflow can be divided into three parts:

1. Develop and modify pipeline source code based on experimentation.
2. Build and test components and pipelines, then deploy them to the execution environment.
3. Validate and deploy the models produced by pipeline runs, and collect operational metrics.

In an EKS-based platform, training orchestration and serving deployment should be treated as distinct responsibilities. For example, CI can validate Kubeflow Pipelines components and pipeline definitions before registering them. For serving resources, Argo CD can synchronize the deployment configuration stored in Git with Kubernetes.

These two workflows are connected through training run identifiers, model versions, container image versions, and deployment records. This allows us to start with a prediction service running in production and trace back to how its model was created.

The three levels can be summarized by how their operational focus changes:

An orchestrator or model registry may be introduced at Level 1. Rather than determining maturity solely by the presence of tools, we should examine how changes to data and pipelines are validated and brought into operation.

4. Considering Organizational and Platform Growth

As multiple teams begin using the same platform, shared challenges extend beyond automation. We need to decide how to separate data and execution permissions among teams, how to provide environments to new teams, and how to preserve operational accountability for each model.

A useful complementary perspective is described in AWS blog post in 2022: " MLOps foundation roadmap for enterprises with Amazon SageMaker". It describes four stages of foundation building across people, operations, and technology: Initial, Repeatable, Reliable, and Scalable.

The two frameworks offer different perspectives:

• Levels 0, 1, and 2 describe the depth of ML workflow automation.
• The four-stage roadmap describes how organizational and platform operating capabilities expand.

It is therefore more appropriate to use them together than to map them one-to-one.

For an EKS-based platform, this might mean moving from manually creating environments for each team to providing shared templates and standardized deployment procedures. The platform team can provide common execution environments, deployment capabilities, and observability tools, while domain teams retain responsibility for their data, models, evaluation criteria, and operations.

Defining what should be standardized and what should remain a team-level decision allows the platform to grow while accommodating model-specific requirements.

5. What Role Should Amazon EKS Play?

Amazon EKS is a service in which AWS manages the Kubernetes control plane. With EKS Auto Mode, AWS also handles provisioning, scaling, and maintenance of data plane infrastructure, including compute, networking, and storage. Training jobs and inference services can run on this foundation, connected to tools for data storage, experiment tracking, model management, and deployment. (Reference: "What is Amazon EKS? [docs.aws.amazon.com]").

Using the operational requirements discussed above, we can divide the responsibilities as follows:

This is an example configuration intended to clarify responsibilities. In an actual implementation, the balance between managed services and components operated directly by the team should depend on requirements and operational capabilities.

In particular, observing data drift and prediction quality requires the application or a separate evaluation process to produce the relevant metrics and connect them to the monitoring and alerting system.

Decisions about which data to use for training, which evaluations a model must pass, and who approves production deployment should be defined as part of the platform’s operating procedures.

Container image versions and model versions must also be distinguished. A model may be packaged inside an inference image or retrieved from S3 or another location at runtime. Either way, deployment records should identify both the image version and the model version used.

6. What Changes When MLOps Expands to LLMOps?

Version control, automation, continuous improvement, and governance remain important for generative AI and retrieval-augmented generation (RAG) systems. Additional managed assets include prompts, retrieval data, evaluation sets, and model-provider configuration. AWS also emphasizes connecting these assets with evaluation, deployment, and monitoring in generative AI operations.

It should be possible to trace a response produced in operation back to the model, prompt, and retrieved documents that generated it. Evaluation sets are also needed to compare quality before and after a change, along with procedures for rolling back problematic configurations.

7. Closing

When designing an MLOps platform, the first step is to decide how much of the model creation and usage process should be connected.

AWS’s Levels 0, 1, and 2 help explain how the operational focus expands from a trained model to a repeatable training pipeline, and then to CI/CD and operations for the pipeline itself. The enterprise adoption roadmap adds a perspective on the operating structure needed to extend these capabilities across teams and use cases.

Kubernetes provides a foundation for running and managing training and inference workloads. Connecting that foundation to data, code, and model versions—as well as evaluation, approval, deployment, and monitoring records—can turn it into a platform that supports reproducibility and recovery. This post has mapped those operational requirements to the roles and relationships of components in an Amazon EKS-based platform.

A question worth revisiting when building an MLOps or LLMOps platform is:

Can we explain why the model currently in production was approved for deployment? And can we repeat the same process and recover when necessary?

Connecting components and retaining execution records should help us build a platform that can answer that question.

When operating Kubernetes, the kubectl version must be compatible with the Kubernetes cluster version. This is documented in both the Kubernetes official documentation under Version Skew Policy and explicitly mentioned in the Amazon EKS documentation.

kubectl

kubectl is supported within one minor version (older or newer) of kube-apiserver.

Example:

• kube-apiserver is at 1.32
• kubectl is supported at 1.33, 1.32, and 1.31

Example:

• kube-apiserver instances are at 1.32 and 1.31
• kubectl is supported at 1.32 and 1.31 (other versions would be more than one minor version skewed from one of the kube-apiserver components)

https://kubernetes.io/releases/version-skew-policy/#kubectl

You must use a kubectl version that is within one minor version difference of your Amazon EKS cluster control plane. For example, a 1.31 kubectl client works with Kubernetes 1.30, 1.31, and 1.32 clusters.

https://docs.aws.amazon.com/en_us/eks/latest/userguide/install-kubectl.html#kubectl-install-update

When using Amazon EKS clusters, you might notice version strings like v1.29.0-eks-5e0fdde when running kubectl commands (Reference: https://aws.amazon.com/blogs/tech/how-to-upgrade-amazon-eks-worker-nodes-with-karpenter-drift/). According to the Amazon EKS User Guide - kubectl and eksctl setup documentation, this binary is identical to the upstream community version. While using the same binary, version strings might be customized for various reasons, such as cloud providers building binaries directly to establish trust, or for internal version management purposes (e.g., distinguishing specific environment builds - v1.32.0-internal, test environment - v1.32.0-test, development stages - v1.29.0-dev).

Let's customize the version string using the Kubernetes source directly.

Prerequisites

• Go development environment: Follow the minimum requirements at https://go.dev/wiki/MinimumRequirements
• Git: If git command is already available, no additional setup is needed. If not, refer to https://git-scm.com/
• Make tool: A historical tool (Reference: https://www.gnu.org/software/make/manual/make.html) that builds binaries using the Makefile in source code

Build Procedure

1. Clone source code

Clone the Kubernetes GitHub repository and navigate to the directory:

$ git clone https://github.com/kubernetes/kubernetes
$ cd kubernetes

2. Check existing version tags

Use the following command to view a list of Kubernetes versions from latest to oldest releases. When you see a colon (:) at the beginning, you can press spacebar to see more entries and press 'q' to exit:

$ git tag --sort=-taggerdate

3. Select and checkout version

Store the desired version tag in a local environment variable and use the "git checkout" command to switch to that version's source code. Here, we use an environment variable named TAG to store the value and use it in $TAG format to checkout the corresponding version:

$ TAG=v1.31.2 # Must use a version that exists in the git tag list

$ git checkout $TAG

4. Set custom tag

The Kubernetes build process is configured to primarily use existing tags stored in the Git repository. Rather than modifying this approach, let's remove the existing tags and apply a custom tag to the currently checked-out source. Note that we can't use arbitrary strings; we must follow the "v{SEMVER}-{CUSTOM}" format. SEMVER refers to the version numbering described at https://semver.org/, and the {CUSTOM} portion can be set with a simple string according to our purposes:

# Remove existing tags
git tag | xargs git tag -d

# Disable fetching remote tags
git config --local fetch.tags false

# Create new custom tag
git tag $TAG-aewstest

5. Verify tag

Use the following command to verify that the tag was properly applied:

$ git describe --tags --match='v*'

6. Build kubectl

Now let's use the make command to build. Run the following command:

$ make all WHAT=cmd/kubectl GOFLAGS=-v

7. Verify build result

The built binary is created in the _output folder. Let's run the kubectl binary:

$ _output/bin/kubectl version

Important Notes

As mentioned above, kubectl version must be within one minor version difference of the cluster version. When using in test environments, ensure thorough validation, and for production environments, it's recommended to use the officially distributed kubectl binary.

• Install Tools - kubectl:https://kubernetes.io/docs/tasks/tools/#kubectl 

• Amazon EKS - Set up kubectl and eksctl: https://docs.aws.amazon.com/en_us/eks/latest/userguide/install-kubectl.html

Conclusion

While this custom-built kubectl can have desired version strings and potentially be useful in CI/CD for cluster management through version checking, it's recommended to use official releases in production environments and limit custom builds to testing or development environments.