Skip to content

linux

Monitoring with InfluxDB and Grafana on Kubernetes

Four years later, I still have not gotten the hang of telegraf, I'm still running my own home-made detailed system and process monitoring reporting to InfluxDB running container-lessly in lexicon and I feel the time is up for moving these services into the Kubernetes cluster. Besides keeping them updated, what I'm most looking forward is leveraging the cluster's infrastructure to expose these services (only) over HTTPS with automatically renewed SSL certs.

Kubernetes Certificate Expired

Yet Another Unexpected Hump

The regular user is unable to connect to the Kubernetes API server because the x509 certificate expired on 2024-03-21T21:37:37Z (nearly 24 hours ago):

$ kubectl get all -n minecraft-server
E0322 20:57:59.141510 3545623 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.6:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z
E0322 20:57:59.143467 3545623 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.6:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z
E0322 20:57:59.145339 3545623 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.6:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z
E0322 20:57:59.147141 3545623 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.6:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z
E0322 20:57:59.148895 3545623 memcache.go:265] couldn't get current server API group list: Get "https://10.0.0.6:6443/api?timeout=32s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z
Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-03-22T20:57:59+01:00 is after 2024-03-21T21:37:37Z

Getbukkit Expired

An Unexpected Minecraft "Update"

One day while looking at the monitoring in lexicon I noticed there was something big missing: the minecraft server that normally takes over 4GB of RAM was not running:

$ kubectl get all -n minecraft-server
NAME                                   READY   STATUS             RESTARTS        AGE
pod/minecraft-server-88f84b5fc-5kjr2   0/1     CrashLoopBackOff   152 (30s ago)   12h

NAME                       TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                           AGE
service/minecraft-server   NodePort   10.110.215.139   <none>        25565:32565/TCP,19132:32132/UDP   291d

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/minecraft-server   0/1     1            0           12h

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/minecraft-server-88f84b5fc   1         1         0       12h

$ kubectl -n minecraft-server logs minecraft-server-88f84b5fc-5kjr2
[init] Running as uid=1003 gid=1003 with /data as 'drwxrwxr-x 1 1003 1003 722 Dec 17 05:10 /data'
[init] Resolving type given SPIGOT
2024/03/15 17:44:59 Unable to find an element with attribute matcher property=og:title
[init] ERROR: failed to retrieve latest version from https://getbukkit.org/download/spigot -- site might be down

$ curl https://getbukkit.org/download/spigot
<!doctype html>
<html data-adblockkey="MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANDrp2lz7AOmADaN8tA50LsWcjLFyQFcb/P2Txc58oYOeILb3vBw7J6f4pamkAQVSQuqYsKx3YzdUHCvbVZvFUsCAwEAAQ==_UL89QGTogxdwKHwZzilx913GmK75KOL2kLgPnkgb9dD1Tc/wjgiP2tuKwPeUMm3vEXLjUWOarjD7XgGHgmalBg==" lang="en" style="background: #2B2B2B;">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <link rel="icon" href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVQI12P4//8/AAX+Av7czFnnAAAAAElFTkSuQmCC">
    <link rel="preconnect" href="https://www.google.com" crossorigin>
</head>
<body>
<div id="target" style="opacity: 0"></div>
<script>window.park = "eyJ1dWlkIjoiNDk5MDE1NDQtMTJlZi00YWQzLWI3YmQtMjA5Y2YwYzlmZjFmIiwicGFnZV90aW1lIjoxNzEwNTI2ODY5LCJwYWdlX3VybCI6Imh0dHBzOi8vZ2V0YnVra2l0Lm9yZy9kb3dubG9hZC9zcGlnb3QiLCJwYWdlX21ldGhvZCI6IkdFVCIsInBhZ2VfcmVxdWVzdCI6e30sInBhZ2VfaGVhZGVycyI6e30sImhvc3QiOiJnZXRidWtraXQub3JnIiwiaXAiOiIyMTcuMTYyLjU3LjY0In0K";</script>
<script src="/bNjGNXnzR.js"></script>
</body>
</html>

Updating Kubernetes to the Community-Owned Package Repositories

An Unexpected Update

Most days I update my little server when I log into my PC, and today it gave quite an unexpected surprise:

# apt update && apt full-upgrade -y
...
E: The repository 'https://apt.kubernetes.io kubernetes-xenial Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

This was promptly reported already yesterday as Ubuntu kubernetes-xenial package repository issue #123673 and quick triaged pointing to the announcement from August 2023: pkgs.k8s.io: Introducing Kubernetes Community-Owned Package Repositories.

Audiobookshelf on Kubernetes

Migrating a Plex Media Server to Kubernetes, was a significant improvement for the maintenance of the Plex Media Server I use to listen to podcasts and audiobooks, to keep me company while I play games, but after all these years Plex remains a very insufficient and deficient application for audiobooks.

Enter audiobookshelf (because Emby and Jellyfin are also not great)

Audiobookshelf home page

Calibrating screen color with DisplayCAL

Calibrating screen color is mostly optional these days, if you buy a good screen that comes out of factory with a good calibration profile. Nonetheless, it is recommended to re-calibrate every year or so to account for display ageing, so a few years ago I purchased a Datacolor Spyder5 Express which is known to work on Linux, with Argyll Color Management System.

What doesn't always work so well is DisplayCal. In fact, the original project is dead and was dropped from Ubuntu 20.04 but was still possible to build with python2.7 packages. That is no longer possible in Ubuntu 22.04, but there is a Python 3 fork: eoyilmaz/displaycal-py3

Migrating a Plex Media Server to Kubernetes

Running Plex Media Server on Linux is easy. Updating it is easy too. Re-using the library from an old server on a new one is also quite easy.

That said, running anything in Kubernetes is only slightly harder once, and after that updates are entirely automatic and moving from one cluster to another would be even easier.

Prologue

I’ve been using Plex Media Server for a few years, primarily to catch up with a bunch of podcasts I started listening from their beginning in the spring of 2020, and occasionally to share my Audible library with the family. The family doesn’t really use any of this, specially since they got Spotify, but this library of Podcasts has been a faithful companion of mine for the last few years, at home and abroad.

The Kubernetes cluster running on Lexicon has proven stable and convenient enough that I finally felt motivated to migrate the Plex Media Server, from the stand-alone setup into the Kubernetes cluster.

The death of Wi-Fi and Bluetooth

The Wi-Fi 6E & Bluetooth 5.2 controllers in my motherboard died today.

Until the motherboard can be replaced, the solution is to disable both in the UEFI BIOS. This is the only state in which the PC boots normally. Enabling the Bluetooth controller causes the boot process to spend about a minute trying to initialize the device, enabling the Wi-Fi controller causes the whole system to freeze with at the login screen and/or eventually reboot itself.

xHCI host controller not responding, assume dead

Today, a number of USB devices were unavailable for no apparent reason.

Having a terminal always visible running dmesg -w the following showed up:

xhci_hcd 0000:0b:00.3: Abort failed to stop command ring: -110
xhci_hcd 0000:0b:00.3: xHCI host controller not responding, assume dead
xhci_hcd 0000:0b:00.3: HC died; cleaning up

Hmm, assume dead... that doesn’t sounds good.