Table of Contents

OpenBao is the authoritative secrets store for the service foundation in my home lab. It runs natively in a dedicated Debian LXC and uses integrated Raft storage.

This page is intentionally written as a recovery procedure, not as a general OpenBao tutorial. It documents the order I use to restore the secrets platform, the minimum material that must exist outside OpenBao, and the steps required to bring dependent services back after a normal reboot, an LXC restore, or a complete loss of the OpenBao datastore.

Sensitive values are intentionally omitted. Recovery copies of unseal material, root/bootstrap credentials, AppRole credentials, and application secrets are stored in an external password manager rather than in Git or this website.

The examples below reflect my lab. Replace addresses, hostnames, service names, and paths as needed for another environment.

Current Recovery Model
#

The current deployment is:

Service:        OpenBao
Version:        2.7.1
PVE CT:         205
Hostname:       openbao
LAN:            172.16.11.205
OS:             Debian 13
Storage:        Integrated Raft
Raft node ID:   openbao-01
API:            https://172.16.11.205:8200
Cluster:        https://172.16.11.205:8201
Browser URL:    https://openbao.lab.wetnight.family/ui/

The LXC starts automatically with its Proxmox node and is included in the normal Proxmox Backup Server schedule.

OpenBao still starts sealed after a restart. Automatic LXC startup does not remove the manual-unseal requirement.

Human administrative access is layered deliberately:

Normal administration
Authentik / OIDC
      |
      v
Independent local fallback
OpenBao userpass break-glass account
      |
      v
Emergency bootstrap
Shamir unseal shares + initial root token

The recovery paths underneath Authentik must remain functional even if the identity provider is unavailable.

Recovery Material That Must Exist Outside OpenBao
#

A secrets platform cannot be its own only recovery source.

The following material is stored outside OpenBao in an external password manager:

all three Shamir unseal shares;
the initial root token;
the local break-glass administrator password;
each service AppRole RoleID and SecretID pair;
application passwords and keys required to reconstruct the KV store if the Raft data and backups are both lost.

None of these values should be committed to Git, embedded in screenshots, pasted into tickets, or published in documentation.

The deployment uses three Shamir shares with a threshold of two:

Shares:    3
Threshold: 2

Any two different shares can unseal the cluster.

Recovery Order
#

The preferred order after a major lab failure is:

Recover core network access.
Recover Proxmox VE and Proxmox Backup Server access.
Restore or start the OpenBao LXC.
Unseal OpenBao.
Verify Raft, KV, policies, AppRole, and audit logging.
Restore or start dependent service LXCs.
Verify OpenBao Agent on each consumer.
Start or redeploy applications only after required secret files exist.

Core management systems must remain accessible with their own local authentication during this process. The identity provider and secrets platform should not become dependencies for reaching the systems required to repair them.

Scenario 1: Normal Reboot or Node Restart
#

This is the most common recovery case.

After the OpenBao LXC or its Proxmox node restarts, the OpenBao service starts automatically but remains sealed.

Log into the OpenBao LXC and set the local CLI environment:

export BAO_ADDR=https://127.0.0.1:8200
export BAO_CACERT=/opt/openbao/tls/tls.crt

Check the current state:

bao status

If the server is initialized but sealed, unseal it with any two different stored shares:

bao operator unseal
bao operator unseal

Enter one share at each prompt.

Verify the final state:

bao status

Expected values include:

Initialized     true
Sealed          false
Storage Type    raft
HA Enabled      true

A single-node Raft deployment can report HA capability even though it currently has only one voter.

After logging in with an authorized account or token, verify Raft:

bao operator raft list-peers

Current expected peer:

Node            Address                 State     Voter
openbao-01      172.16.11.205:8201      leader    true

Verify the audit device:

bao audit list

The expected device is the file audit backend.

Finally, verify OpenBao Agent on each consumer as needed:

systemctl is-active openbao-agent
journalctl -u openbao-agent -n 50 --no-pager

Do not restart otherwise healthy applications just because OpenBao was temporarily sealed. Already-rendered static secret files persist on the service LXCs.

Scenario 2: Restore OpenBao from Proxmox Backup Server
#

The preferred disaster-recovery path is to restore the complete OpenBao LXC from PBS.

The backup captures the important state together:

/etc/openbao/openbao.hcl;
the Raft data under /opt/openbao/data;
the TLS certificate and private key;
local authentication configuration;
KV secrets;
policies and AppRole state;
audit configuration and logs;
the installed OpenBao runtime.

Restore Procedure
#

Restore CT 205 from the appropriate PBS backup.
Confirm that the expected Services-VLAN addressing is restored.
Confirm the LXC starts successfully.
Log into the restored LXC.
Set the local CLI environment.
Check bao status.
Unseal with two externally stored Shamir shares.
Verify Raft leadership.
Verify the KV mount and current secret paths.
Verify the local break-glass authentication method.
Verify AppRole is enabled.
Verify the file audit device.
Verify each consumer Agent.

Useful checks:

export BAO_ADDR=https://127.0.0.1:8200
export BAO_CACERT=/opt/openbao/tls/tls.crt

bao status
bao operator raft list-peers
bao secrets list
bao auth list
bao audit list

The current KV paths should include the service-specific entries used by the lab:

kv/traefik
kv/pi-hole-01
kv/pi-hole-02
kv/authentik

Do not display or export secret values merely to verify that the paths exist.

Scenario 3: Recover a Consumer Service While OpenBao Is Healthy
#

A service LXC can be rebuilt independently as long as OpenBao remains healthy.

Current consumers include Traefik, both Pi-hole instances, and Authentik.

The recovery pattern is:

OpenBao KV
    |
    v
service AppRole
    |
    v
OpenBao Agent on the service LXC
    |
    v
/docker/<service>/secrets/*.txt
    |
    v
Docker Compose secret
    |
    v
/run/secrets/* inside the application

Install the OpenBao Agent Package
#

Install the OpenBao package on the rebuilt consumer LXC for the bao agent binary.

Do not enable another OpenBao server on the consumer.

apt update && apt install -y curl gpg ca-certificates

install -d -m 0755 /etc/apt/keyrings

curl -fsSL https://openbao.org/assets/openbao-gpg-pub-20240618.asc \
  | gpg --dearmor --yes -o /etc/apt/keyrings/openbao.gpg

cat > /etc/apt/sources.list.d/openbao.sources <<'EOF'
Types: deb
URIs: https://pkgs.openbao.org/deb/
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/openbao.gpg
EOF

apt update
apt install -y openbao
bao version

Create the Agent credential directory:

install -d -m 0700 /etc/openbao-agent

Restore the service’s AppRole RoleID and SecretID from the external password manager without echoing them into shell history:

read -r -s -p "AppRole RoleID: " ROLE_ID
echo
printf '%s' "$ROLE_ID" > /etc/openbao-agent/role-id
unset ROLE_ID

read -r -s -p "AppRole SecretID: " SECRET_ID
echo
printf '%s' "$SECRET_ID" > /etc/openbao-agent/secret-id
unset SECRET_ID

chmod 600 /etc/openbao-agent/role-id /etc/openbao-agent/secret-id

Copy the public OpenBao certificate to:

/etc/openbao-agent/openbao-ca.crt

Then:

chmod 644 /etc/openbao-agent/openbao-ca.crt

The private TLS key must remain on the OpenBao server.

Required Agent Settings
#

Every consumer Agent uses the same basic structure:

pid_file = "/run/openbao-agent.pid"

vault {
  address = "https://172.16.11.205:8200"
  ca_cert = "/etc/openbao-agent/openbao-ca.crt"
}

auto_auth {
  method "approle" {
    mount_path = "auth/approle"

    config = {
      role_id_file_path                   = "/etc/openbao-agent/role-id"
      secret_id_file_path                 = "/etc/openbao-agent/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }
}

template_config {
  static_secret_render_interval = "5m"
}

The setting below is critical:

remove_secret_id_file_after_reading = false

Without it, AppRole Auto-Auth can remove the SecretID file after the first successful authentication. That can make a foreground test succeed and then cause the systemd service to fail later because the stored SecretID is gone.

Production templates also use:

backup = false

No plaintext .bak secret files are intentionally retained.

Agent systemd Unit
#

Each consumer uses the same basic service:

[Unit]
Description=OpenBao Agent for service secrets
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/bin/bao agent -config=/etc/openbao-agent/agent.hcl
Restart=on-failure
RestartSec=5
User=root
Group=root

[Install]
WantedBy=multi-user.target

Enable it only after a successful foreground test:

bao agent -config=/etc/openbao-agent/agent.hcl

Healthy output includes messages similar to:

authentication successful
starting renewal process
template server received new token
rendered "(dynamic)" => "..."
renewed auth token

Stop the foreground test with Ctrl+C, validate the rendered file, and then enable the service:

systemctl daemon-reload
systemctl enable --now openbao-agent
systemctl status openbao-agent --no-pager

Validate Rendered File Ownership
#

Check the numeric UID/GID instead of relying only on account names:

stat -c '%n uid=%u gid=%g mode=%a' /path/to/secret.txt

Current expectations are:

Traefik:     uid=0    gid=0    mode=600
Pi-hole-01:  uid=0    gid=0    mode=600
Pi-hole-02:  uid=0    gid=0    mode=600
Authentik:   uid=1000 gid=1000 mode=600

Authentik is intentionally different because its application processes require the secret files to remain readable by UID/GID 1000:1000.

If a service’s IP address changed during recovery, a previously issued SecretID may fail because the AppRole is source-CIDR restricted. Restore the expected address or issue a new service credential with the correct CIDR restriction.

Scenario 4: Complete OpenBao Loss Without a PBS Restore
#

This is the last-resort recovery path.

A new OpenBao deployment cannot decrypt the old Raft data merely because the old AppRole credentials or unseal shares are still available. If the LXC backup and Raft datastore are both gone, the secrets authority must be rebuilt as a new cluster.

Reinstall OpenBao
#

On a clean Debian 13 LXC:

apt update && apt install -y curl gpg ca-certificates

install -d -m 0755 /etc/apt/keyrings

curl -fsSL https://openbao.org/assets/openbao-gpg-pub-20240618.asc \
  | gpg --dearmor --yes -o /etc/apt/keyrings/openbao.gpg

cat > /etc/apt/sources.list.d/openbao.sources <<'EOF'
Types: deb
URIs: https://pkgs.openbao.org/deb/
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/openbao.gpg
EOF

apt update
apt install -y openbao

Create the persistent directories:

install -d -o openbao -g openbao -m 0700 /opt/openbao/data
install -d -o openbao -g openbao -m 0750 /opt/openbao/tls
install -d -o openbao -g openbao -m 0750 /var/log/openbao

Recreate the TLS Certificate
#

The OpenBao listener uses a lab-specific self-signed certificate containing the service DNS names and IP address.

Create an OpenSSL configuration appropriate for the restored host:

cat > /tmp/openbao-openssl.cnf <<'EOF'
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
O = Wetnight Lab
CN = openbao.lab.wetnight.family

[v3_req]
subjectAltName = @alt_names
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
DNS.1 = openbao
DNS.2 = openbao.lab.wetnight.family
IP.1 = 127.0.0.1
IP.2 = 172.16.11.205
EOF

Generate the certificate:

openssl req -x509 -nodes -newkey rsa:4096 -sha256 -days 825 \
  -keyout /opt/openbao/tls/tls.key \
  -out /opt/openbao/tls/tls.crt \
  -config /tmp/openbao-openssl.cnf

chown openbao:openbao /opt/openbao/tls/tls.crt /opt/openbao/tls/tls.key
chmod 644 /opt/openbao/tls/tls.crt
chmod 600 /opt/openbao/tls/tls.key
rm -f /tmp/openbao-openssl.cnf

Verify the certificate:

openssl x509 -in /opt/openbao/tls/tls.crt \
  -noout -subject -issuer -dates -ext subjectAltName

Any consumer that trusts the old self-signed certificate must receive the new public certificate before its Agent can reconnect.

Recreate the Server Configuration
#

The current single-node Raft configuration is:

ui = true

api_addr     = "https://172.16.11.205:8200"
cluster_addr = "https://172.16.11.205:8201"

storage "raft" {
  path    = "/opt/openbao/data"
  node_id = "openbao-01"
}

listener "tcp" {
  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"

  tls_cert_file = "/opt/openbao/tls/tls.crt"
  tls_key_file  = "/opt/openbao/tls/tls.key"
}

audit "file" "file" {
  description = "OpenBao security audit log"

  options {
    file_path = "/var/log/openbao/audit.log"
    mode      = "0600"
  }
}

Validate it:

bao operator validate-config -config=/etc/openbao/openbao.hcl

Enable the service:

systemctl enable --now openbao
systemctl status openbao --no-pager

Initialize the New Cluster
#

Only run initialization against a genuinely new, empty Raft datastore:

export BAO_ADDR=https://127.0.0.1:8200
export BAO_CACERT=/opt/openbao/tls/tls.crt

bao operator init -key-shares=3 -key-threshold=2

Immediately store the new unseal shares and root token in the external recovery vault.

Old unseal shares and root tokens from the destroyed cluster are not credentials for this new cluster.

Unseal with two new shares:

bao operator unseal
bao operator unseal

Recreate Administrative Access
#

Use the new root token only for bootstrap.

Create the broad administrative policy:

bao policy write admin - <<'EOF'
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

Enable local userpass authentication:

bao auth enable userpass

Configure lockout behavior:

bao auth tune \
  -user-lockout-threshold=10 \
  -user-lockout-duration=15m \
  -user-lockout-counter-reset-duration=15m \
  userpass/

Create the local break-glass administrator interactively:

read -r -s -p "Break-glass password: " BG_PASS
echo

bao write auth/userpass/users/breakglass-admin \
  password="$BG_PASS" \
  policies="admin"

unset BG_PASS

Recreate the KV Engine
#

Enable KV v2:

bao secrets enable -path=kv -version=2 kv

Re-enter the application secrets from the external recovery vault.

Current paths are:

kv/traefik
kv/pi-hole-01
kv/pi-hole-02
kv/authentik

Do not reuse secret values blindly if the recovery event suggests they may have been exposed. Rotate affected credentials when appropriate.

Recreate Service Policies
#

Each consumer should receive only read access to its own KV path.

Example pattern:

path "kv/data/SERVICE" {
  capabilities = ["read"]
}

path "kv/metadata/SERVICE" {
  capabilities = ["read"]
}

Current policy mapping:

Service	KV path	Policy
Traefik	`kv/traefik`	`traefik-read`
Pi-hole-01	`kv/pi-hole-01`	`pi-hole-01-read`
Pi-hole-02	`kv/pi-hole-02`	`pi-hole-02-read`
Authentik	`kv/authentik`	`authentik-read`

Recreate AppRole
#

Enable AppRole:

bao auth enable approle

Each service uses a dedicated role with a one-hour token TTL, 24-hour maximum TTL, reusable SecretID, and a source-CIDR restriction.

Generic pattern:

bao write auth/approle/role/SERVICE \
  token_policies="SERVICE-read" \
  token_ttl="1h" \
  token_max_ttl="24h" \
  token_type="service" \
  secret_id_ttl="0" \
  secret_id_num_uses="0" \
  secret_id_bound_cidrs="SERVICE_IP/32"

Generate new RoleID and SecretID values:

bao read -field=role_id auth/approle/role/SERVICE/role-id
bao write -field=secret_id -f auth/approle/role/SERVICE/secret-id

Replace the old stored credentials in the external password manager and on the consumer LXC.

Old AppRole credentials belong to the destroyed cluster and should not be treated as valid credentials for the replacement deployment.

Post-Recovery Validation
#

A recovery is not complete merely because the UI loads.

Verify the platform in layers.

OpenBao Server
#

bao status
bao operator raft list-peers
bao secrets list
bao auth list
bao audit list

Confirm that:

the server is initialized;
the server is unsealed;
Raft reports the expected leader;
the kv/ engine exists;
required authentication methods exist;
the file audit device is enabled.

Consumer Agents
#

On each service LXC:

systemctl is-active openbao-agent
journalctl -u openbao-agent -n 50 --no-pager

Healthy Agent logs should show successful authentication, token renewal, and template rendering without permission errors.

Rendered Secrets
#

Validate files without printing their contents:

stat -c '%n uid=%u gid=%g mode=%a' /path/to/secret.txt

Do not use cat simply to prove that a secret exists.

Applications
#

After the local secret files are confirmed, validate each application using its normal health checks.

Only restart containers that actually need to reread a changed secret.

Secret Rotation After Recovery
#

For a static application secret:

update the value in OpenBao KV v2;
allow the Agent to render the new version;
confirm the destination timestamp and permissions;
determine whether the application reads the file dynamically or only during startup;
restart only the affected application when required;
validate the application;
do not retain plaintext .bak copies.

All deployed Agents use:

backup = false

This prevents template rotation from leaving stale plaintext backup files beside the active secret.

Important Failure-Mode Rules
#

OpenBao Is Not in the Normal Request Path
#

Applications consume rendered local files.

A temporary OpenBao outage does not automatically stop a healthy Traefik, Pi-hole, or Authentik instance.

OpenBao is required for new renders, rotation, and clean rebuilds.

Identity Must Not Be the Only Recovery Path
#

OIDC provides convenience for normal administration, but local break-glass and emergency bootstrap paths remain underneath it.

A broken identity provider must not prevent repair of the secrets platform.

Do Not Reinitialize Restored Raft Data
#

Never run:

bao operator init

against an existing or restored initialized Raft datastore.

Initialization is only for a new, empty cluster.

Future High Availability
#

The current OpenBao deployment is a single Raft voter.

High availability is a planned foundation upgrade.

The intended direction is a three-member Raft cluster distributed across the two low-power always-on nodes plus the larger server when it is available:

low-power node 1   always on
low-power node 2   always on
primary server     additional member when online

The design goal is for the low-power pair to maintain the critical foundation during extended shutdowns of the larger server.

The HA design still needs explicit testing around:

quorum behavior;
member placement;
storage;
certificate lifecycle;
manual unseal behavior;
node loss;
node return;
backup and Raft snapshot strategy.

The objective is not merely to enable HA. It is to make OpenBao’s failure behavior predictable and recoverable.

Quick Reference
#

Normal reboot
  -> bao status
  -> unseal with two shares
  -> verify Raft
  -> verify audit
  -> verify Agents

OpenBao LXC lost
  -> restore CT 205 from PBS
  -> start
  -> unseal
  -> verify OpenBao state
  -> verify Agents

Consumer LXC lost
  -> restore/rebuild service
  -> reinstall OpenBao Agent
  -> restore RoleID/SecretID
  -> restore public OpenBao cert
  -> foreground Agent test
  -> verify file ownership
  -> enable Agent
  -> start application

OpenBao and PBS copy both lost
  -> deploy new OpenBao
  -> initialize new cluster
  -> create new recovery material
  -> recreate local admin
  -> recreate KV
  -> restore application secrets from external vault
  -> recreate policies/AppRoles
  -> issue new AppRole credentials
  -> update consumers
  -> validate everything

The most important recovery principle is simple: restore the secrets authority before rebuilding the services that depend on it, but keep those services capable of continuing to run from already-rendered local secrets when OpenBao itself is temporarily unavailable.

Reply by Email

Current Recovery Model#

Recovery Material That Must Exist Outside OpenBao#

Recovery Order#

Scenario 1: Normal Reboot or Node Restart#

Scenario 2: Restore OpenBao from Proxmox Backup Server#

Restore Procedure#

Scenario 3: Recover a Consumer Service While OpenBao Is Healthy#

Install the OpenBao Agent Package#

Required Agent Settings#

Agent systemd Unit#

Validate Rendered File Ownership#

Scenario 4: Complete OpenBao Loss Without a PBS Restore#

Reinstall OpenBao#

Recreate the TLS Certificate#

Recreate the Server Configuration#

Initialize the New Cluster#

Recreate Administrative Access#

Recreate the KV Engine#

Recreate Service Policies#

Recreate AppRole#

Post-Recovery Validation#

OpenBao Server#

Consumer Agents#

Rendered Secrets#

Applications#

Secret Rotation After Recovery#

Important Failure-Mode Rules#

OpenBao Is Not in the Normal Request Path#

Identity Must Not Be the Only Recovery Path#

Do Not Reinitialize Restored Raft Data#

Future High Availability#

Quick Reference#