Kubernetes and Elastic Observability Lab | Projects

Six virtual machines, two connected platforms

This SYS-255 coursework built a container-orchestration lab and connected it to a separate logging and monitoring environment. The surviving report is written as a collaborative build record, so I present the implementation as team work rather than assigning every configuration step to one person.

The environment used six Ubuntu Server 18.04 virtual machines in VMware Workstation 16. Each VM was allocated two logical processors, 2 GB of RAM, 20 GB of storage, and a network interface on an isolated VMware NAT segment.

Role	Purpose
K3s server	Kubernetes control-plane services and cluster administration
Three K3s agents	Worker capacity for scheduled workloads
Rancher host	Web management interface on a separate machine
Elastic host	Elasticsearch, Logstash, Kibana, Beats, and Nginx

The final address plan was:

Lab host	Address	Function
Master Server	`192.168.33.200/24`	K3s server
Server 2	`192.168.33.201/24`	K3s agent
Server 3	`192.168.33.202/24`	K3s agent
Server 4	`192.168.33.203/24`	K3s agent
Web Server	`192.168.33.204/24`	Rancher management
ELK Stack	`192.168.33.205/24`	Elastic services and Nginx

These are retired RFC1918 lab addresses tied to the VMware NAT configuration. Netplan was used to disable DHCP and set each static address, gateway, and name server. After applying the network configuration, the team populated /etc/hosts on the cluster machines so that each node could resolve the others predictably without depending on an external DNS record.

The project brought together two systems that are often taught separately: deploying services across a cluster and collecting enough operational data to understand what that cluster is doing.

Container and cluster model

The design work began by separating containers from virtual machines. Each VM in this lab had its own guest operating system and kernel. Containers running on a given node shared that node’s kernel while packaging the application and its user-space dependencies. That reduced the per-workload overhead and made it practical to start several service instances across machines with modest resources.

Kubernetes added a desired-state control layer over those workloads. The report examined four parts of that model:

the API server exposes the cluster objects and accepts changes to desired state
the scheduler selects a node for a pod that has not yet been assigned
controllers continually compare current state with declared state and act on differences
the cluster data store retains configuration and state needed by the control plane

kubectl was the command-line interface for interacting with the API, while Rancher later provided a web interface over many of the same resources. Namespaces grouped related objects and reduced naming collisions, although they were treated as organisation boundaries rather than assumed to be complete security boundaries.

Forming the K3s cluster

All four cluster machines were updated and had Docker installed as part of the coursework build. The first Ubuntu server then ran the K3s server role. Its node token was read from /var/lib/rancher/k3s/server/node-token, and the other three machines joined by supplying that token and the server endpoint through the K3s installation environment.

Once registered, the machines appeared as a single cluster with one server and three schedulable agents rather than four unrelated container hosts. The shared token was a practical bootstrap mechanism for the isolated build, but it also represented a credential: possession of it affected who could join a node to the cluster. It should therefore have been stored outside the build notes and rotated after exposure.

K3s was selected because the full lab had to run on six small virtual machines. It packaged the Kubernetes components into a lighter distribution and reduced the amount of manual control-plane assembly. That kept the coursework focused on cluster membership, workloads, services, storage, and observation rather than building every component independently.

The initial architecture described Docker containers, a K3s control plane, and node logs exported to Elastic. During implementation, the monitoring host was also kept on Ubuntu rather than the Windows client mentioned in the early project description. The final report therefore records the design changing as the build became more concrete.

Rancher as the management surface

Rancher was installed on the Web Server VM rather than the K3s server. The report records an RKE2 configuration beneath /etc/rancher/rke2, a TLS subject-alternative name set to the management host’s static address, and Rancher exposed over HTTPS on port 8443. The installation created a separate management cluster on that host.

The browser initially presented an untrusted-certificate warning because the lab used its own certificate rather than a publicly trusted identity. After the bootstrap credentials were retrieved and the administrator password changed, the K3s cluster was added as an “Other Cluster”. An import command was run on the server node to register the cluster with Rancher. The team also adjusted an agent image override through Rancher’s API view so that the cluster used the expected architecture-specific image.

Once imported, the web interface exposed nodes, projects, namespaces, workloads, services, ingress configuration, environment variables, logs, shell access, and deployment status. That made Rancher both an administrative convenience and a sensitive management surface. Access to it meant access to cluster configuration and, depending on permissions, workload execution.

Three types of workload were recorded:

Replicated demonstration service

The first deployment used the monachus/rancher-demo image. It was configured as a scalable deployment with three pods in the default namespace and a ClusterIP service listening on container and service port 8080.

At that stage the pods were running but had no route for a browser outside the cluster. Rancher’s load-balancing view was used to add an ingress targeting the demo service on port 8080, with an automatically generated sslip.io hostname. Repeated browser requests could then be served by different replicas. An environment variable named COW_COLOR was also changed through the workload interface, demonstrating how runtime configuration could alter the application without building another image.

Persistent game server

A Minecraft server used the itzg/minecraft-server image with one published host port, the required EULA=TRUE environment variable, and a bind-mounted data path. The report configured the service on port 25565 and mounted a node path into the container’s ./minecraft-data location.

The client connected to the address of the node on which the workload was scheduled. This showed both the convenience and weakness of direct host-port exposure: the service became reachable quickly, but the user also depended on the selected node and its address. The bind mount preserved data outside the container layer, while also tying the workload to a host path that needed its own permissions, backup, and scheduling considerations.

Apache web server

An Apache workload used httpd:latest, one pod, and a new namespace named httpd-webserver. Container port 80 was published as host port 8001; the report verified it on the node address 192.168.33.202:8001.

The namespace organised the Apache objects separately from the earlier deployment. The host-port choice, rather than the namespace alone, avoided a binding conflict on the selected node. The first browser request to the node without :8001 did not reach the service because the listener was explicitly mapped to the alternate port.

Rancher’s execute-shell option was then used to open a shell inside the running container and edit its configuration. This proved that the container could be changed interactively, but it also exposed the difference between an experiment and a repeatable deployment. Changes made in a live container can disappear on replacement; a maintained version should build a versioned image or mount controlled configuration instead.

These deployments tested more than whether a container image would start. They exercised replica counts, scheduling, namespaces, service exposure, ingress, environment configuration, and persistent data.

Building the Elastic Stack

The sixth Ubuntu server hosted Elasticsearch, Kibana, Logstash, Filebeat, Metricbeat, and Nginx.

Elasticsearch provided the indexed data store at 192.168.33.205:9200. Kibana supplied the search and visualisation interface on port 5601. Logstash accepted Beats input and forwarded events into Elasticsearch. Filebeat and Metricbeat collected log and system information, while Nginx acted as the front-end reverse proxy with HTTP basic authentication.

The VM was assigned the same static networking pattern as the cluster hosts, updated, and prepared with the package signing key and repository needed by the 2021 Elastic packages. Java and the six services were installed before individual configuration began.

In /etc/elasticsearch/elasticsearch.yml, the team named the cluster ELK, retained node-1, bound the service to 192.168.33.205, and used port 9200. After reloading and enabling the service, an HTTP request to that endpoint confirmed that the node was answering.

Kibana’s /etc/kibana/kibana.yml configuration bound the interface to the same host on port 5601 and pointed elasticsearch.hosts at the Elasticsearch endpoint. Direct browser access confirmed the interface before Nginx was placed in front of it.

For the reverse proxy, an administrator credential was written to an Nginx htpasswd file and a site configuration forwarded authenticated requests to Kibana. nginx -t validated the syntax before the service was restarted and enabled. This added an access check but did not replace Elastic’s own identity, authorisation, transport encryption, or audit controls.

Logstash used two files under /etc/logstash/conf.d: a Beats input and an Elasticsearch output. Filebeat and Metricbeat were then configured with the relevant hosts, enabled as services, and checked with their setup commands. Their Elasticsearch, Kibana, and Logstash modules were enabled so that the Elastic host could report on its own stack as well as receive external data.

The resulting flow was:

host or Kubernetes metrics
  -> Filebeat / Metricbeat
  -> Logstash Beats input
  -> Elasticsearch index
  -> Kibana search and visualisation

Metricbeat was also deployed from the K3s server using a Kubernetes manifest. Its output section was changed to address the external Elastic VM before the objects were created and their status checked. Expected Metricbeat indices appearing in Kibana confirmed more than simple network reachability: the collector, transport, index, and visualisation path were all participating.

What the combined lab exposed

The cluster and Elastic Stack made several infrastructure relationships visible:

a running container is not the same as a managed workload with replicas and desired state
a service needs an explicit route from the user to the pod
namespaces organise resources but do not automatically provide every form of isolation
persistent state needs deliberate storage rather than the container filesystem
management interfaces create their own authentication and network exposure
logs and metrics need a transport, parser, index, and visualisation path before they become useful
monitoring infrastructure must remain reachable when the system being monitored has a fault

The verification steps were layered accordingly. Service status showed whether a daemon was alive. An HTTP request tested whether it was listening. Rancher’s workload state showed whether pods had reached their desired count. A browser request tested ingress or host-port exposure. Elastic index management showed whether events had travelled through the pipeline.

The build also exposed the amount of version coupling in a platform assembled from K3s, Rancher, Docker, Elastic components, Beats, Nginx, Java, and Ubuntu packages. Installation order and compatible configuration mattered. A component could be running successfully while the overall data path was incomplete, and a green workload did not prove that its service, ingress, storage, or monitoring path worked.

Security observations

The lab prioritised getting the architecture working. Some choices were appropriate only for the contained coursework network: an import command that tolerated the local certificate, a shared cluster token written into build material, simple reverse-proxy authentication, broad same-subnet connectivity, interactive root shells, mutable running containers, bind mounts, and unpinned public images such as httpd:latest.

A stronger implementation would add:

secret storage and rotation instead of shared values in build notes
certificate management with trusted internal identities
Kubernetes role-based access control tied to individual users
network policies between workloads and management services
image pinning, provenance checks, and vulnerability scanning
restricted pod security contexts and non-root containers
protected persistent volumes and backup testing
authenticated and encrypted Beats transport
alerting based on operational baselines rather than dashboards alone

The observability platform also needs its own protection. Elasticsearch indices can contain command lines, usernames, host details, and application data. Nginx basic authentication alone does not define who may query particular indices or administer the stack. Role-based access, TLS between shippers and Logstash, retention limits, protected credentials, audit logging, and tested backups would be part of a longer-lived deployment.

This is historical team coursework based on Ubuntu 18.04 and 2021-era K3s, Rancher, and Elastic releases. Current installation methods, supported versions, default runtimes, security controls, and configuration keys should be checked against current vendor documentation. The project record is the architecture we assembled and tested: a four-node container cluster, a separate Rancher management host, replicated and stateful service patterns, and an external metrics and logging pipeline.