3.8 KiB
+++ title = "Homelab - Overview" date = 2026-05-04 description = "A quick and rough overview of my Homelab setup" draft = true
[taxonomies] categories = ["Homelab"] tags = ["Homelab", "Ceph", "Vault", "Nomad", "Consul"] +++
{% mermaid() %} block columns 3 nomad:3 ceph consul vault {% end %}
{% mermaid() %} architecture-beta group homelab(cloud)[Homelab] group cloud(cloud)[Cloud]
service db(database)[Database] in homelab
service disk1(disk)[Storage] in homelab
service disk2(disk)[Storage] in homelab
service server(server)[Server] in homelab
service proxy(server)[Proxy] in cloud
service internet(internet)[Internet]
db:L -- R:server
disk1:T -- B:server
disk2:T -- B:db
{% end %}
The goal of this post to give a quick high-level overview of my current Homelab setup, to avoid having to re-explain everything on every following post that focuses on one part of it.
Hardware
- Unifi Network setup
- Normal and Management traffic over 1GbE
- Cluster networking using 10GbE
- 3 Servers
- 4-8 Cores (8-16 Threads)
- 64GB RAM
- APC UPS
Internet
For my internet connection I only have normal residential internet with 250Mb down and 50Mb up. This means I have no public IPv4 address, but have a /48 IPv6 Prefix, which is not ideal but at least the IPv6 forces me to learn it and use "newer" technologies (if you consider IPv6 "new").
Infrastructure
Getting to the more interesting bits now.
The foundational setup consists of Consul, Nomad, Vault and Ceph.
Vault is responsible for all my secret management and runs mostly independent of anything else. For this purpose it is running as 3-node cluster, with automatic unsealing using GCP KMS and using the integrated raft storage backend. This should ensure that apart from unsealing, my vault cluster is always operational and does not depend on anything else, which might complicate setup during a cold-start / entire cluster restart.
Consul is my service discovery and service mesh of choice and also acts as a simple KV store for some minor things, to help orchestrate things.
Nomad is setup on top of consul and is my container/application orchestrator of choice. This is also integrated with vault to provide easy secret management, for any service that might need it. Basically everything that I deploy to my cluster is deployed to nomad for easier management.
The last foundational piece is my 3-node Ceph cluster, which provides the persistent storage for all of my deployed services. For this purpose it provides both S3 compatible storage using cephs' RGW and also provides block storage using RBD to my containers using the ceph-csi plugin. In the future I also plan to try out using ceph to provide storage for my desktop using RBD directly on linux or use iSCSI or NvmeoF for windows, but that is still in the planning/draft stage.
Ingress Traffic - IPv6
For HTTP/HTTPS traffic, I have a single traefik instance deployed, which handles all the routing to the different services. Traefik itself dynamically loads the configuration from consul, looking for services with specific tags.
Other TCP traffic is usually forwarded to the correct service directly, like for minecraft servers or teamspeak. I plan to improve this a bit, but currently this work fine and thus there is little urgency for me to fix this.
Ingress Traffic - IPv4
As previously mentioned, I do not have a public routable IPv4 address, which makes this setup a bit more complex. Over time I had a lot of different setups for this, but currently I settled on having a custom proxy, which dynamically loads the services that should be exposed from consul and then forwards the traffic on the ports accordingly. This setup is definetly not ideal and I want to move towards a better more scalable approach, with something like SIIT but I will have to see.