From d4541950a9e3fea8d17d50458dd6d51850c00fcd Mon Sep 17 00:00:00 2001 From: Lol3rrr Date: Sun, 27 Sep 2026 02:43:33 +0200 Subject: [PATCH] Basically finish my vault-agents blog post --- content/vault-agents/index.md | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/content/vault-agents/index.md b/content/vault-agents/index.md index 76cf155..3198deb 100644 --- a/content/vault-agents/index.md +++ b/content/vault-agents/index.md @@ -35,18 +35,27 @@ Deploying the vault agents to my servers now comes down to: 3. deploying the general config for the vault agents # Deploying templated Configs -TODO: Explain how I am using an ansible role to make this easily available +Now that vault-agents are running on my servers, it is time to actually use them for something useful. +This means I need to make it easy to integrate the deployment of templated configurations for other services, which I decided to do by introducing a simple ansible-role. +This role takes care of creating the correct configuration for the vault-agent to template out a file and placing it in a shared space, where the vault-agent can pick it up. +It will then also trigger the vault-agent on the server to restart / reload its configuration to pick up the newly created config and start outputting the templated configuration. -A couple of examples how I am going to use the vault agents templating features +So all I need to do for using templated configurations for a service that I want to deploy, is give the vault agent user the permissions to write my config to where my service expects it, give it the template to use and providing a command to execute whenever the written out file changes, in case the service does not automatically detect changes to it. # Example Uses -## Nomad mTLS -We can use vault agents to template the TLS certificates for nomads mTLS configuration. -1. Get all currently valid CA certificates and store them in the trust bundle -2. Get a leaf certificate for the current node +## Nomad - mTLS +My nomad cluster uses mTLS to encrypt its communication over the network, this means every node needs the CA certificate and its own certificate to communicate with other nodes. +Previously this meant managing the certificates and their livecycles semi-manual, I automated all the steps, but needed to trigger the updates manually. -## Consul ACLs -To configure the ACL tokens for consul +However now with the vault-agents, I can configure them to automatically fetch the shared CA certificate(s) and generate the certificate for the current node dynamically, including renewing it and notifying nomad accordingly. +This workflow is a lot easier to maintain, as there is no human involvement to rotate the leaf-certificates, and allows for enforcing best practices, like short lifetimes for the leaf-certificates. + +## Nomad - Consul ACLs +My nomad cluster is very tightly integrated into consul, to make use of its service-mesh capabilties and some other smaller features. +However because I have ACLs enabled on the consul cluster, every nomad client requires an ACL token to connect to consul. +Managing this manually would either mean a lot of maintenance for me or using tokens with very long lifetimes, which are both not good options. + +Instead I can use the consul secret-engine in vault and the vault-agents, to dynamically obtain an ACL token for each server, which contains just enough permissions for nomad and has a short lifetime to follow best-practices as much as possible (for my context). # Some Issues I want to cover some of the issues that I encountered / still have, just so I am clear on the downsides of this setup. @@ -56,6 +65,10 @@ I have one server that likes to randomly freeze on the OS level, which is mostly However because I use a vault token, which expires after at most one day with my current config, it frequently happens that the token for the server expires before I get around to restarting it, meaning I need to run my vault-agent deployment again to generate a new token for the server. This is not a huge issue for me in this case, but mostly annoying as I need to keep that in mind during maintenance and adds another kind of weird thing. +## One bad config infects the entire agent +Currently if I accidentally deploy one bad vault-agent config, for example by writing a custom config for templating a config, then the entire vault-agent on the server is not working properly and thus affects other services as well. +I have no good approach to prevent this, besides enforcing the use of the ansible-role, which could be updated to make sure that after every change the agent is still running without issues, but that is not yet implemented. + # References [^vault_pki_api]: [Vault PKI API](https://developer.hashicorp.com/vault/api-docs/secret/pki) [^consul_template_secrets]: [Consul Template Secrets](https://github.com/hashicorp/consul-template/blob/main/docs/templating-language.md#secret)