5 - Create complex pods

Create more complex pods with entra login, requiring entra variables and volumes
Published

June 5, 2026

In previous chapters we deployed an app with the nginx default page. Before deploying more complex images we will start by adding volumes to change that default into somthing ours.

Static websites

For static websites (or even static html served through a flask app that handles login and session through cookies) 1 replica should be enough as it avoids some issues with volumes being read twice at the same time.

We will for these work with volumes but the volumes can’t be just a routing since we don’t know which node will be running the pod (so we don’t know which server to keep the files on). For this we will use a server with nfs to store our files, in this case we’ll use the control pane for this. Just run the following in the control pane server:

apt install nfs-kernel-server
echo "/home/gitlabci/websites *(ro,no_subtree_check)" >> /etc/exports
exportfs -a

and on the nodes:

apt install -y nfs-common
which mount.nfs

For the my-app deployement we will add some information:

# /home/gitlabci/conf/websites/my-app.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-app-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadOnlyMany 
  nfs:
    server: stivm0052   # or its IP
    path: /home/gitlabci/websites/my-app
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-app-pvc
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
spec:
  replicas: 1 #changed to 1 for avoiding reading concurrency problems
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts: # add volume mounts
        - name: website
          mountPath: /usr/share/nginx/html
      volumes: # add volume declaration
      - name: website
        persistentVolumeClaim:
          claimName: my-app-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: my-app-svc
  namespace: default
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP 
kubectl apply -f /home/gitlabci/conf/websites/my-app.yaml
What changed?
  1. Added the volumes and volumemounts in the app spec.
  2. Added PersistentVolume and PersistentVolumeClaim
  • The first is the cluster wide decalration of the storage the claim sits in between the storage and the pod
  • These layers exist because k8s was designed so the volumes can be declared once and the development teams can call them without knowing the exact path

Now lets add another pod so we can work with routing in the next step:

Show YAML 
# /home/gitlabci/conf/websites/pinchsmall.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pinchsmall-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadOnlyMany
  nfs:
    server: stivm0052   # or its IP
    path: /home/gitlabci/websites/pinchsmall
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pinchsmall-pvc
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pinchsmall
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pinchsmall
  template:
    metadata:
      labels:
        app: pinchsmall
    spec:
      containers:
      - name: pinchsmall
        image: nginx:alpine
        ports:
        - containerPort: 80
        volumeMounts:
        - name: website
          mountPath: /usr/share/nginx/html
      volumes:
      - name: website
        persistentVolumeClaim:
          claimName: pinchsmall-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: pinchsmall-svc
  namespace: default
spec:
  selector:
    app: pinchsmall
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP
kubectl apply -f /home/gitlabci/conf/websites/pinchsmall.yaml

And upgrade the ingress for our domain with the first website in /test/ and the pinchsmall website at root path:

# /home/gitlabci/conf/ingress/ipese-test.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: strip-test
  namespace: default
spec:
  stripPrefix:
    prefixes:
      - /test
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ipese-test
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: default-strip-test@kubernetescrd
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - ipese-test.epfl.ch
    secretName: myapp-tls
  rules:
  - host: ipese-test.epfl.ch
    http:
      paths:
      - path: /test
        pathType: Prefix
        backend:
          service:
            name: my-app-svc
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: pinchsmall-svc
            port:
              number: 80
kubectl apply -f /home/gitlabci/conf/ingress/ipese-test.yaml

Check status of your kube

$ root@stivm0052:/home/gitlabci/conf/ingress# kubectl get all
NAME                              READY   STATUS      RESTARTS   AGE
pod/my-app-5fb5c4bb54-z9kfh       1/1     Running     0          2m55s
pod/pinchsmall-6bf6c55b9c-pl4bp   1/1     Running     0          16m
pod/test-ping                     0/1     Completed   0          49d

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP   53d
service/my-app-svc       ClusterIP   10.111.163.238   <none>        80/TCP    2m55s
service/pinchsmall-svc   ClusterIP   10.103.131.201   <none>        80/TCP    16m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-app       1/1     1            1           2m55s
deployment.apps/pinchsmall   1/1     1            1           16m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/my-app-5fb5c4bb54       1         1         1       2m55s
replicaset.apps/pinchsmall-6bf6c55b9c   1         1         1       16m
$ root@stivm0052:/home/gitlabci/conf/ingress# kubectl get ingress
NAME         CLASS     HOSTS                ADDRESS      PORTS     AGE
ipese-test   traefik   ipese-test.epfl.ch   10.95.35.4   80, 443   5m11s

Here we can see:

  • 2 pods both 1/1 Running — my-app and pinchsmall are up
  • 2 services routing internal traffic to each pod
  • 2 deployments each managing their replica as expected
  • 1 ingress ipese-test handling both /test and / on ipese-test.epfl.ch
  • test-ping is just an old completed job, harmless but can be erased with kubectl delete pod test-ping

Current configuration:

/home/gitlabci/conf
├── certs
│   ├── ipese-test.epfl.ch.crt
│   └── ipese-test.epfl.ch.key
├── ingress
│   └── ipese-test.yaml # path descriptions (traefic routes/locations)
├── traefik
│   └── values.yaml # ~traefik config - open ports and deployed in which servers as deamonset
└── websites # deployment and service definition for each app
    ├── my-app.yaml 
    └── pinchsmall.yaml

and /home/gitlabci/websites available through nfs to all the servers

Deploy image other than nginx

To start we’ll deploy a static website deployed over flask to support authentication through an auth-server. (docker image built for work purposes withihn EPFL).

For this we’ll need environment variables, pulling a new container from a private docker registry and make sure the connection to another tool works properly.

  1. Create the folder with the static files
    • mkdir /home/gitlabci/websites/secure && nano /home/gitlabci/websites/secure/index.html
  2. create your deployment and service
    • nano /home/gitlabci/conf/websites/secure.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: secure-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadOnlyMany
  nfs:
    server: 10.95.32.31
    path: /home/gitlabci/websites/secure
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: secure-pvc
  namespace: default
spec:
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secure
  template:
    metadata:
      labels:
        app: secure
    spec:
      containers:
      - name: secure
        image: ipeseserv3.epfl.ch:88/flask-entra:v0.4.9
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: website
          mountPath: /var/www
      volumes:
      - name: website
        persistentVolumeClaim:
          claimName: secure-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: secure-svc
  namespace: default
spec:
  selector:
    app: secure
  ports:
  - port: 5000
    targetPort: 5000
  type: ClusterIP
kubectl apply -f /home/gitlabci/conf/websites/secure.yaml
  1. add your new route to your ingress
# under stripPrefix.prefixes
- /secure

# under path
- path: /secure
  pathType: Prefix
  backend:
    service:
      name: secure-svc
      port:
        number: 80
kubectl apply -f /home/gitlabci/conf/ingress/ipese-test.yaml

Now…this doesn’t work because this specific app needs other infromation on the ingress like the HTTP_X_SCRIPT_NAME header but we can fix that on the ingress by adding a middleware and adding an ingress on the same file that integrates both the middleware to strip the location and the new middleware to add the header:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: strip-route
  namespace: default
spec:
  stripPrefix:
    prefixes:
      - /test
      - /secure
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: secure-forwarded-headers
  namespace: default
spec:
  headers:
    customRequestHeaders:
      X-Script-Name: /secure
      X-Forwarded-Host: ipese-test.epfl.ch
      X-Forwarded-Port: "443"
      X-Forwarded-Proto: https
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ipese-test
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: default-strip-route@kubernetescrd
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - ipese-test.epfl.ch
    secretName: myapp-tls
  rules:
  - host: ipese-test.epfl.ch
    http:
      paths:
      - path: /test
        pathType: Prefix
        backend:
          service:
            name: my-app-svc
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: pinchsmall-svc
            port:
              number: 80
---
# Separate Ingress for /secure, with the extra header middleware chained
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ipese-secure
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: websecure
    traefik.ingress.kubernetes.io/router.middlewares: default-strip-route@kubernetescrd,default-secure-forwarded-headers@kubernetescrd
spec:
  ingressClassName: traefik
  tls:
  - hosts:
    - ipese-test.epfl.ch
    secretName: myapp-tls
  rules:
  - host: ipese-test.epfl.ch
    http:
      paths:
      - path: /secure
        pathType: Prefix
        backend:
          service:
            name: secure-svc
            port:
              number: 5000
kubectl apply -f /home/gitlabci/conf/ingress/ipese-test.yaml
Important

A classic bug pattern for VXLAN/flannel on VMware environments (ens33 naming is typical VMware vmxnet3) — the NIC’s generic checksum offload miscalculates checksums on encapsulated VXLAN UDP packets, silently corrupting/dropping them, while ICMP and TCP often still work fine.

To solve it, run this command on all your nodes:

sudo ethtool -K ens33 tx-checksum-ip-generic off