Service Foundation: DNS, Reverse Proxy, Identity, and Secrets

By the time Proxmox Backup Server was finally finished, the underlying lab was in a much better place than it had been when this rebuild started. The cluster was stable, storage was organized, backups were layered correctly, and the Dell Wyse systems had settled into useful always-on infrastructure roles.

The next problem was not really a single service, but sort of everything around the services.

Over time, my lab has accumulated several generations of naming, reverse-proxy rules, Docker assumptions, local DNS entries, authentication experiments, secret handling, and old configuration decisions. Some of those choices had been perfectly reasonable when the environment was smaller and most services lived on one host. They stopped being reasonable once the architecture spread across multiple Proxmox nodes, multiple Docker hosts, and a dedicated backup platform.

The next phase of the rebuild became a cleanup of the entire service foundation, which turned out to be a huge undertaking and took several days:

• Standardize the internal namespace
• Establish a new clean reverse-proxy boundary
• Reconfigure Docker networks
• Remove unnecessary Docker socket access
• Move credentials out of Git-managed configuration
• Centralize secrets without making applications directly dependent on the secrets server
• Keep critical infrastructure independent from the identity provider (learned from past mistakes to avoid the catch-22)
• Rebuild Authentik around the architecture that actually exists now
• Create a recovery model that still works when one of those supporting services is unavailable

This ended up touching Proxmox VE, Proxmox Backup Server, Cloudflare, UDM SE, Pi-hole, Traefik, Portainer, Authentik, PostgreSQL, OpenBao, Docker networking, certificates, firewall rules, service identities, backups, and several assumptions that had quietly survived from earlier versions of the lab.

The completed foundation now looks roughly like this:

                            wetnight.family
                                   |
                         Cloudflare DNS / ACME
                                   |
                                   v
                         Traefik - 172.16.11.203
                                   |
             +---------------------+---------------------+
             |                     |                     |
             v                     v                     v
     Management backends     Services backends       Authentik
       172.16.10.x             172.16.11.x          172.16.11.204
                                                         |
                                                         v
                                                    PostgreSQL
                                               host-local Docker only

Internal DNS
------------
Pi-hole-01 - 172.16.11.201 - wyse01
Pi-hole-02 - 172.16.11.202 - wyse02

Secrets
-------
OpenBao - 172.16.11.205 - wyse01
    |
    +--> Traefik Agent     --> /docker/traefik/secrets/*
    +--> Pi-hole-01 Agent  --> /docker/pi-hole/secrets/*
    +--> Pi-hole-02 Agent  --> /docker/pi-hole/secrets/*
    +--> Authentik Agent   --> /docker/authentik/secrets/*

traefik.lab.wetnight.family -> 172.16.11.203
friendly service names      -> CNAME -> Traefik

Choosing the Namespace

#

The lab had historically used names under shannonwetnight.com. That worked, but by 2026 I also owned wetnight.family, which had become the more appropriate household and infrastructure namespace.

I spent some time considering whether the lab should instead use one of the common internal-only naming patterns such as .local, .home, .internal, or a reserved home-network namespace.

In the end, I decided not to invent a separate pseudo-domain at all. I already own the domain, so the naming model is:

Public / externally meaningful names:
    wetnight.family

Internal lab namespace:
    lab.wetnight.family

Examples:

home.lab.wetnight.family
wyse01.lab.wetnight.family
wyse02.lab.wetnight.family
pbs.lab.wetnight.family
traefik.lab.wetnight.family
pi-hole-01.lab.wetnight.family
pi-hole-02.lab.wetnight.family
openbao.lab.wetnight.family

Authentik is slightly different because it is intended to act as a canonical identity endpoint for both internal and eventually external applications:

login.wetnight.family

Internally, that name still resolves through the lab’s split-DNS path to Traefik.

This gives the lab one coherent naming hierarchy while still keeping internal records private to Pi-hole.

Migrating Proxmox Cluster Nodes

#

The Proxmox side required more care than simply changing DNS records.

The PVE cluster already consists of home, wyse01 and wyse02. Those short node names are part of the cluster identity, so I deliberately did not attempt to rename the nodes themselves. Instead, I preserved the short names and migrated their FQDN representation:

home.lab.shannonwetnight.com   -> home.lab.wetnight.family
wyse01.lab.shannonwetnight.com -> wyse01.lab.wetnight.family
wyse02.lab.shannonwetnight.com -> wyse02.lab.wetnight.family
pbs.lab.shannonwetnight.com    -> pbs.lab.wetnight.family

The migration included several steps:

• Updating /etc/hosts
• Updating the DNS search domain
• Updating Postfix myhostname
• Correcting the Postfix generic mapping on wyse01
• Updating the affected LXC search domain
• Regenerating PVE certificates
• Regenerating the PBS certificate
• Replacing the PBS fingerprint stored by the PVE cluster

The PVE certificates were regenerated one node at a time:

pvecm updatecerts -f
systemctl restart pveproxy

After each node, cluster quorum was checked before continuing.

PBS required its own certificate regeneration:

proxmox-backup-manager cert update --force
systemctl reload proxmox-backup-proxy

Because the PBS certificate changed, the fingerprint in the PVE storage configuration also had to be updated before the cluster would trust the backup target again.

One old-domain reference was intentionally left alone, which is the notification address under shannonwetnight.com. That address still routes correctly to Exchange and remains useful for alerts, so there was no reason to change it simply for the sake of removing every old string.

Moving Edge Records

#

Cloudflare was updated alongside the internal namespace. The UDM SE DDNS configuration was moved to narrowly scoped tokens for the wetnight.family zone, and the dual-WAN DDNS records were standardized as:

ip.wetnight.family
ip2.wetnight.family

The same Cloudflare zone is used by Traefik for DNS-01 certificate validation. That allows Traefik to obtain a valid wildcard certificate without exposing the internal services themselves to the internet:

lab.wetnight.family
*.lab.wetnight.family

Authentik receives its own certificate for login.wetnight.family.

The Cloudflare token itself is no longer manually managed as a loose deployment credential. It now lives in OpenBao and is rendered onto the Traefik host by OpenBao Agent.

Standardizing Service Addressing

#

One of the larger architecture decisions was to stop mixing routable lab addresses with Docker-only addresses.

LAN / routable:
172.16.<VLAN>.<service-ID>

Docker internal:
172.31.<VLAN>.<service-ID>

For example:

Traefik
LAN:    172.16.11.203
Docker: 172.31.11.203

Authentik
LAN:    172.16.11.204
Docker: 172.31.11.204

The most important rule is that 172.31.x.x is host-local only. It is not routed, stretched between Docker hosts or used as a cross-host backend address. That means two Docker hosts can both have their own independent bridge named services-network using 172.31.11.0/24 without conflicting, because the networks never leave their hosts.

If Traefik on wyse01 needs to reach Authentik on wyse02, it uses:

172.16.11.204

instead of:

172.31.11.204

Docker bridges belong to the host that created them. Cross-host traffic uses the routable network.

Rebuilding Pi-hole DNS

#

The lab now runs two Pi-hole instances:

Pi-hole-01
LAN:    172.16.11.201
Docker: 172.31.11.201
Host:   wyse01

Pi-hole-02
LAN:    172.16.11.202
Docker: 172.31.11.202
Host:   wyse02

Both stacks use the same Git-managed DNS inventory. Instead of manually building a large collection of unrelated A records, Traefik is the single reverse-proxy ingress record:

traefik.lab.wetnight.family -> 172.16.11.203

Most friendly service names are then CNAMEs to Traefik:

home.lab.wetnight.family       -> traefik.lab.wetnight.family
pbs.lab.wetnight.family        -> traefik.lab.wetnight.family
portainer.lab.wetnight.family  -> traefik.lab.wetnight.family
pi-hole-01.lab.wetnight.family -> traefik.lab.wetnight.family
pi-hole-02.lab.wetnight.family -> traefik.lab.wetnight.family
openbao.lab.wetnight.family    -> traefik.lab.wetnight.family

The actual backend selection happens in Traefik. This keeps DNS simple and means TLS-facing service names do not need to track every backend port or physical host. The records are stored in dns-records.env, which both Pi-hole stacks load through env_file. That environment file is intentionally not a secret file. It contains only the Git-managed Pi-hole FTL DNS inventory and remains safe to commit.

Staggering GitOps Updates

#

Both Pi-hole stacks are deployed from Git through Portainer, but their polling intervals are intentionally different:

Pi-hole-01: 5 minutes
Pi-hole-02: 20 minutes

That doesm’t create a guaranteed 15-minute offset, but it reduces the chance that a bad DNS change takes out both resolvers at exactly the same time. Pi-hole-01 should normally receive and validate a change first, with Pi-hole-02 acting as the slower second deployment.

Pi-hole Secrets to OpenBao

#

The Pi-hole web/API passwords originally lived as local host secret files. The Docker consumption model was already good, so I did not redesign the Compose stacks.

Each Pi-hole still expects hard-coded crential files:

/docker/pi-hole/secrets/web_password.txt

Docker still mounts that file as:

/run/secrets/pihole_web_password

Pi-hole v6 consumes it through:

WEBPASSWORD_FILE: /run/secrets/pihole_web_password

What changed is who owns the source of truth and how it’s managed. The values now live in OpenBao KV v2:

kv/pi-hole-01
kv/pi-hole-02

Each Pi-hole LXC runs its own OpenBao Agent using a dedicated AppRole. The Agents render the correct web_password.txt file locally with mode 0600.

The result is a useful separation where OpenBao owns the secret value, OpenBao Agent owns delivery, Docker owns the container mount itself, and Pi-hole reads the ordinary file it already understands. No secret needs to appear in Git or a general-purpose environment file moving forward for any of the services being redeployed.

DNS Testing Trap

#

One subtle issue came from the Proxmox hosts themselves. The PVE nodes have entries in /etc/hosts for their own FQDNs. That means a command such as:

curl https://home.lab.wetnight.family

run directly on home may resolve through /etc/hosts instead of Pi-hole, which can make reverse-proxy testing misleading.

When I specifically want to test the Traefik path from one of those systems, I can force the request to the proxy:

curl --resolve home.lab.wetnight.family:443:172.16.11.203 \
  https://home.lab.wetnight.family

That guarantees the request is testing Traefik rather than the host’s local resolver shortcut.

Rebuilding Traefik

#

Traefik became the center of this service-foundation cleanup. The new Traefik instance is a dedicated Debian LXC on wyse01:

CT:       203
Hostname: traefik
LAN:      172.16.11.203
VNet:     srv11

The old configuration had accumulated several assumptions that no longer fit the environment:

• Docker provider discovery
• A mounted Docker socket
• A large dynamic configuration
• Docker-name backends that only worked when services shared one host
• A broad global insecureSkipVerify that was implemented as a temporary fix
  • (There is nothing more permanent than a temporary solution)
• Old Authentik ForwardAuth dependencies
• Definitions for services that did not exist anymore

The rebuild intentionally removes those assumptions.

File Provider Only

#

The static configuration uses the Traefik file provider only:

providers:
  file:
    directory: "/dynamic"
    watch: true

There is no Docker provider or /var/run/docker.sock mount. That removes a very powerful host-level interface from a container that does not need it.

The dynamic configuration is split into active modules instead of one giant file:

/docker/traefik/config/
├── traefik.yml
└── dynamic/
    ├── 00-core.yml
    ├── 10-infrastructure.yml
    ├── 20-authentik.yml
    └── inventory/
        └── future-services.yml (services to be redeployed as the lab rebuild continues)

The future-services inventory preserves names and intended routing information without loading nonexistent backends into the active control plane.

Traefik Container Hardening

#

The current Traefik Compose stack also deliberately constrains the container itself:

security_opt:
  - no-new-privileges:true

cap_drop:
  - ALL

cap_add:
  - NET_BIND_SERVICE

read_only: true

tmpfs:
  - /tmp

Only the configuration, dynamic configuration, ACME storage, logs, and Docker secret mounts are exposed to the container. The Docker socket remains absent since its no longer needed.

Traefik Secrets to OpenBao

#

Traefik still consumes ordinary Docker secrets:

/run/secrets/cf_api_token
/run/secrets/dashboard_users

The host-side files remain:

/docker/traefik/secrets/cf_api_token.txt
/docker/traefik/secrets/dashboard_users.txt

The Compose stack still defines:

environment:
  CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token

secrets:
  - cf_api_token
  - dashboard_users

The values are now authoritative in OpenBao inside:

kv/traefik

The Traefik LXC runs OpenBao Agent to render both files. The Cloudflare token and dashboard htpasswd data therefore no longer depend on manually maintained host copies as the source of truth.

Certificates

#

Traefik uses Cloudflare DNS-01 validation for the lab certificate. The wildcard ACME certificate contains:

CN:  lab.wetnight.family
SAN: *.lab.wetnight.family

A separate certificate was also issued for:

login.wetnight.family

These certificates through Let’s Encrypt renew automatically every 90 days.

Management Network Routing

#

Traefik lives on the Services VLAN, while several important management interfaces live on the Management VLAN.

The existing firewall model intentionally blocks broad Services-to-Management access. Rather than weakening that segmentation, I added a scoped UDM SE exception allowing Traefik at 172.16.11.203 to reach only the required web-management ports on the Management network.

This preserves the larger isolation policy while still allowing one controlled reverse-proxy ingress point. Cross-host Traefik backends use routable 172.16.x.x addresses.

Authentik Dependency (Legacy)

#

Before the rebuild, some infrastructure routes were placed behind Authentik ForwardAuth. That created an avoidable failure mode.

For example, a backend such as PBS could be perfectly healthy while Traefik still returned an error because the Authentik backend was unavailable. That ended up being an authentication dependency failure. The new policy is deliberately different to make Authentik an identity provider first.

The preferred integration order is:

1. Native OIDC/OAuth2
2. SAML when it is the better native fit
3. LDAP when required
4. ForwardAuth only when the application does not support a proper native identity integration

Core infrastructure does not automatically depend on Authentik. Proxmox, PBS, Pi-hole, Traefik, Portainer, OpenBao, and similar management services must retain their own recovery-capable authentication and network controls even if Authentik is unavailable. That makes identity a security layer instead of a single point of failure for every administrative interface.

Rebuilding Authentik

#

Authentik itself was rebuilt on wyse02. The LXC is sized conservatively:

Hostname: authentik
LAN:      172.16.11.204
OS:       Debian 13
vCPU:     1
RAM:      2 GB
Swap:     512 MB
Disk:     16 GB

The final design uses:

• Authentik server
• Authentik worker
• PostgreSQL 16
• The embedded proxy outpost
• No more Redis container

The old design had included Redis, a standalone proxy-outpost container, Docker socket access, and broader worker privileges. Those were removed. The server publishes only 172.16.11.204:9000.

Traefik reaches that address directly across the Services VLAN. PostgreSQL is not published outside its host-local Docker bridge. In the future, I may migrate databases over to the dedicated Database mirrored SSD pool on home.lab, but because I do not always intend to run home.lab 24/7, this is out of scope at this time.

The canonical browser URL is:

https://login.wetnight.family

Authentik Secrets to OpenBao

#

Authentik still consumes two file-backed Docker secrets:

/docker/authentik/secrets/authentik_secret_key.txt
/docker/authentik/secrets/postgres_password.txt

The Compose stack mounts those into the containers and points Authentik and PostgreSQL at the corresponding /run/secrets/... paths.

The source of truth is now:

kv/authentik

with keys:

authentik_secret_key
postgres_password

Authentik’s OpenBao Agent renders those files locally. This migration surfaced an ownership detail that is worth documenting permanently. Traefik and Pi-hole are happy with root-owned mode-0600 secret files. Authentik is not.

The Authentik server and worker need the files to remain:

uid=1000
gid=1000
mode=600

Earlier in the rebuild, leaving the files as root:root 0600 caused:

Permission denied: '/run/secrets/postgres_password'
Permission denied: '/run/secrets/authentik_secret_key'

After correcting the ownership, the services started normally. The OpenBao Agent preserves the existing ownership during renders, so the current files stay 1000:1000 0600.

Authentik Bootstrap

#

Once the secret ownership was corrected, the server and worker stopped crash-looping and began normal database migrations. During first startup, the health endpoint temporarily returned 500 while migrations were still running.

That initially looked alarming, but the logs continued progressing through migrations and eventually showed the embedded outpost creating its initial token, authenticating successfully, and connecting to its websocket.

The final state was:

authentik-server      healthy
authentik-postgresql  healthy
authentik-worker      running/healthy
embedded outpost      connected

Direct backend testing returned a redirect to the expected setup/login path:

curl -I http://172.16.11.204:9000

The same request through Traefik confirmed the reverse-proxy path was also healthy.

Adding OpenBao

#

At this point the remaining weakness was obvious. The services were no longer storing secrets in Git or ordinary environment files, but the host-backed secret files themselves were still being maintained manually.

That was better than embedding credentials in Compose, but it still left several problems

• Recovery depended on remembering which files had to exist before a stack started
• Secret rotation meant manually updating files on individual LXCs
• No central versioned source of truth
• No common machine-authentication model
• Each new service would repeat the same clunky bootstrap pattern

I wanted a proper secrets platform, but I did not want the applications themselves to become tightly coupled to it. OpenBao fit that requirement well.

OpenBao Runs Natively

#

OpenBao was deployed in its own Debian 13 LXC on wyse01:

CT:       205
Hostname: openbao
LAN:      172.16.11.205
OS:       Debian 13
vCPU:     1
RAM:      2 GB
Swap:     0
Disk:     16 GB

I deliberately installed OpenBao natively rather than putting it into Docker. The secrets system is bootstrap infrastructure. If Docker, Portainer, or a service stack is broken, I still want the secrets server to have as few dependencies as possible. The result is a small native service using integrated Raft storage.

OpenBao Storage and TLS

#

The server currently runs a single Raft voter:

node_id: openbao-01
address: 172.16.11.205:8201
state:   leader

Raft data lives under:

/opt/openbao/data

The API listens on:

https://172.16.11.205:8200

and the cluster listener uses:

https://172.16.11.205:8201

The package-default self-signed certificate was not useful because it lacked the SANs needed for the actual deployment. I generated a replacement certificate with:

CN: openbao.lab.wetnight.family

SANs:
openbao
openbao.lab.wetnight.family
secrets.lab.wetnight.family
127.0.0.1
172.16.11.205

The canonical browser name is:

openbao.lab.wetnight.family

Traefik reaches the native OpenBao HTTPS backend and uses the shared internal self-signed servers transport rather than a one-off OpenBao-specific transport.

Shamir Initialization and Unseal

#

OpenBao was initialized using Shamir key splitting:

shares:    3
threshold: 2

All three shares and the initial root token are stored in LastPass outside the OpenBao deployment. OpenBao is intentionally not configured for automatic cloud/KMS unseal.

That adds a manual step after a reboot, but it avoids introducing another external dependency into the lab’s recovery chain. When CT 205 restarts, the service starts automatically but remains sealed until two shares are supplied.

OpenBao Administrative Recovery

#

The initial root token is retained as an emergency bootstrap credential, not as a normal administrator login.

For persistent local recovery access I created breakglass-admin using OpenBao’s local userpass auth method and a broad administrative policy.

The userpass mount also has lockout protection configured. That creates an intentional three-level human access model:

Normal administration later:
Authentik / OIDC

Independent fallback:
local breakglass-admin

Emergency bootstrap:
Shamir shares + initial root token

The important part is that the second and third paths do not depend on Authentik. If identity is the thing that is broken, I still need to be able to repair identity. This avoids the catch-22 lockout I faced last year.

OpenBao KV Layout

#

Application secrets live in a KV v2 engine mounted at:

kv/

Current paths are:

kv/traefik
kv/pi-hole-01
kv/pi-hole-02
kv/authentik

The values currently managed include:

kv/traefik
  cf_api_token
  dashboard_users

kv/pi-hole-01
  web_password

kv/pi-hole-02
  web_password

kv/authentik
  authentik_secret_key
  postgres_password

Backup/recovery copies of the passwords, keys, AppRole credentials, unseal shares, and root recovery material are kept in LastPass.

Per Service AppRole

#

Machine authentication uses OpenBao AppRole. I considered several ways to distribute machine credentials, but the cleanest model for this lab was one AppRole per service rather than one role per Docker host. That keeps the permissions narrow and makes future revocation easier, and is simple enough to reproduce when adding new services.

Current roles:

traefik
pi-hole-01
pi-hole-02
authentik

Each role receives only the policy required to read its own KV path. Each SecretID is also bound to the service’s routable address:

Traefik:     172.16.11.203/32
Pi-hole-01:  172.16.11.201/32
Pi-hole-02:  172.16.11.202/32
Authentik:   172.16.11.204/32

The AppRole credentials themselves are stored in LastPass and as root-only files on the matching service LXC.

OpenBao Agent as Delivery Layer

#

The consumer applications do not receive OpenBao tokens and do not call OpenBao directly.

Instead, the bao agent process runs on each service LXC.

The Agent does roughly the following:

1. Reads that service’s local AppRole RoleID and SecretID;
2. Authenticates to OpenBao;
3. Receives a renewable service token;
4. Reads only the allowed KV path;
5. Renders the secret into the host path that Docker already expects.

For example:

OpenBao
kv/traefik
    |
    v
Traefik AppRole
    |
    v
OpenBao Agent on CT 203
    |
    v
/docker/traefik/secrets/cf_api_token.txt
    |
    v
Docker secret
    |
    v
/run/secrets/cf_api_token

This design preserves the simple Docker secret interface while giving the lab central secret storage, versioning, and access control.

AppRole Agent Trap

#

The first foreground Agent test uncovered one behavior that absolutely needed to be documented.

By default, AppRole Auto-Auth can consume and remove the SecretID file after reading it. This is intended and useful in workflows where a one-time SecretID is intentionally delivered to a workload, but not the correct approach for this lab. The service roles use reusable SecretIDs because the Agents need to survive service restarts without requiring manual credential delivery every time.

The required Agent setting is:

remove_secret_id_file_after_reading = false

Without it, the first Agent run could authenticate successfully and then the systemd service would fail later because /etc/openbao-agent/secret-id had disappeared.

Agent Runtime Files and Permissions

#

Every consumer has:

/etc/openbao-agent/role-id
/etc/openbao-agent/secret-id
/etc/openbao-agent/openbao-ca.crt
/etc/openbao-agent/agent.hcl

The RoleID and SecretID files are mode 0600. The OpenBao certificate is public trust material and is copied to each consumer so the Agent can validate the self-signed server certificate. The private TLS key never leaves CT 205. The rendered application secrets have service-specific ownership:

Traefik:     root:root 0600
Pi-hole-01:  root:root 0600
Pi-hole-02:  root:root 0600
Authentik:   1000:1000 0600

No Plaintext Backup Copies

#

OpenBao Agent template rendering can keep a .bak copy of the previous destination file. This is useful during the initial migration because it provided a simple rollback while the new delivery path was being tested.

Once OpenBao was confirmed healthy, those backup copies were removed. No .bak secret files are intentionally retained now.

All deployed OpenBao Agents have been changed to backup = false for their static secret templates. Future rotations therefore replace the rendered destination without recreating stale plaintext backup copies.

Audit Logging

#

OpenBao security audit logging is enabled, location is set here:

/var/log/openbao/audit.log

The audit log is owned by the OpenBao service account and restricted to mode 0600. The configured audit device can be verified with: bao audit list

Audit logs are managed by Debian logrotate using /etc/logrotate.d/openbao.

• Rotated daily
• Rotated early when the log reaches 50 MB
• Retains 30 rotations
• Older logs are compressed with delayed compression
• New audit logs are created as openbao:openbao with mode 0600
• OpenBao receives SIGHUP after rotation so it reopens the audit file without restarting or resealing the service

Rotation was validated by forcing a rotation and confirming that OpenBao remained on the same process, reopened the newly created audit file, and continued recording audit events.

OpenBao Backups and Recovery Order

#

OpenBao is now included in the normal Proxmox Backup Server schedule and CT 205 is configured to start automatically with its node.

The preferred disaster-recovery order is now:

1. Restore core network/PVE/PBS access
2. Restore OpenBao CT 205
3. Unseal OpenBao with two externally stored shares
4. Verify Raft, KV, policies, AppRole, and audit logging
5. Restore/start the consumer services
6. Let OpenBao Agent recreate/refresh their runtime secret files

If OpenBao is temporarily unavailable, already-rendered static secret files remain on the service LXCs. That means a running Traefik, Pi-hole, or Authentik deployment does not suddenly stop simply because the secrets server is sealed.

OpenBao is needed for new renders, rotation, and clean rebuilds rather than being part of every application request. If the complete OpenBao backup were lost, the recovery copies in LastPass are enough to rebuild a new secrets authority manually, but that is the fallback path rather than the normal one.

OpenBao Missing Authentik

#

The OpenBao UI is reachable through Traefik at:

https://openbao.lab.wetnight.family/ui/

For now, normal OpenBao authentication is still local. That is intentional while the foundation is being built.

The next step is to integrate Authentik with OpenBao using OIDC for normal human administration while preserving breakglass-admin and the Shamir/root recovery path underneath it. The intended result is convenience without circular recovery dependencies.

SMTP Can Wait

#

Authentik still uses the older Gmail SMTP path for now.

I considered creating no-reply@wetnight.family as a shared mailbox in Exchange Online and using the Microsoft tenant for Authentik mail. That eventually led to the more appropriate design of using an internal SMTP relay with Exchange Online OAuth rather than storing a mailbox password in Authentik.

That is useful work, but it is not required for the service foundation to function. It remains intentionally deferred at this time because it requires more groundwork than what it is in scope for this segment.

Current Authentication Strategy

#

Authentik is not only back up and running, but my lab now has an explicit authentication strategy.

For each future service:

Does the application support OIDC/OAuth2?
    Yes -> use native OIDC

No:
Does it have a better native SAML integration?
    Yes -> use SAML

No:
Does it require LDAP?
    Yes -> use LDAP

No:
Use Traefik ForwardAuth where appropriate

This avoids using ForwardAuth as a universal hammer, often exposing clunky double login screens. Native authentication keeps application sessions, logout behavior, claims, groups, and authorization closer to the application that actually understands them.

The same philosophy applies to OpenBao where Authentik will become the normal human login, but local recovery authentication will remain underneath it.

Foundation

#

The service foundation now has clear boundaries.

Routing

#

Reverse-proxy ingress point:

Traefik
172.16.11.203

DNS

#

Rednundant DNS:

Pi-hole-01
172.16.11.201

Pi-hole-02
172.16.11.202

Identity

#

Authentik provides idenity:

Authentik
172.16.11.204

Secrets

#

OpenBao is the authoratative application secret storage.

OpenBao
172.16.11.205

Per-service OpenBao Agents render local files, and Docker Compose continues presenting them to applications through /run/secrets.

Docker

#

Docker networks remain host-local. Cross-host service traffic uses the normal routable VLANs.

Management Segmentation

#

UDM SE remains the authoritative router, firewall, and DHCP server. Services cannot broadly reach the Management network. Traefik receives a narrowly scoped exception because it is the controlled ingress point.

Backups

#

The critical foundation LXCs, including OpenBao, are now part of the PBS backup routine.

That gives the service layer a defined restore path rather than just a deployment path.

Lessons Learned

#

• A green Traefik router or service means the configuration loaded. It does not prove that the backend exists, resolves, accepts connections, or is actually healthy.
• OpenBao can deliver the correct value and the application can still fail if the rendered file belongs to the wrong UID/GID. Authentik proved that immediately.
• Reusable Agent credentials need remove_secret_id_file_after_reading = false
  • Otherwise, Auto-Auth can consume the SecretID file that the systemd service expects to reuse.
• An unavailable identity provider should not take down access to the systems needed to repair the identity provider. The same principle applies to the secrets platform.
• Rendering static application secrets to local files means OpenBao can be temporarily sealed or offline without immediately taking down otherwise healthy services.

Preserve Future Plans Without Activating Them

#

Keeping future service definitions in an inventory file is useful. Loading dozens of nonexistent backends into the active reverse proxy is not and provides more opportunity to misconfigure something by mistake.

Up Next: High Availability

#

The original goal of moving critical services onto the low-power Dell Wyse nodes was to let the lab remain functional during long shutdowns of the larger home.lab server, which is now working.

The next goal is to make the foundation more resilient without undoing that low-power design.

The high-priority HA targets are:

OpenBao
Traefik
Authentik

Home Assistant may eventually join that list, although its stateful workload and hardware/device integrations make it a different problem.

The larger home.lab server has enough CPU and storage capacity to participate as an additional placement or failover target whenever it is online. The important requirement is that the two low-power nodes remain capable of carrying the foundation when home.lab is intentionally shut down.

Traefik will need a client-facing failover mechanism rather than simply a second container. Authentik can replicate application components, but PostgreSQL availability must be solved as its own stateful dependency.

Still No Fun Services

#

The service foundation is now much stronger, but I do not want to immediately bury it under dozens of applications before the remaining control-plane work is complete. Up next:

• Integrate Authentik with OpenBao through OIDC for normal administrative login
• Preserve and test local OpenBao break-glass recovery after that integration
• Add audit-log rotation
• Evaluate a cleaner internal certificate lifecycle
• Evaluate explicit Raft snapshots in addition to full PBS LXC backups
• Design and test OpenBao, Traefik, and Authentik high availability
• Research a clean Git-to-Traefik dynamic-configuration sync that preserves file-provider hot reload without forcing a stack redeploy

Once that is done, the larger service rebuild can finally begin. Home Assistant, Plex, media applications, dashboards, utilities, and the other application workloads will then be added on top of an infrastructure layer that has clear rules for DNS, certificates, routing, authentication, secrets, storage, backups, and recovery. That is a much better place to start building applications than the collection of one-off assumptions that existed before this cleanup.