Ascode - Erik Christiaans

Building an AI Landing Zone on Azure — Part 4: Metering every call

2026-10-07T00:00:00Z

This is part 4 of a five-part series on building an AI Landing Zone on Azure. Part 1: Why every enterprise needs an AI gateway · Part 2: The platform underneath · Part 3: One front door for many models · Part 4: Metering every call (this post) · Part 5: Turning tokens into euros

Parts 1 to 3 described building a private gateway that every AI call goes through. This part is about the thing that makes the gateway worth more than a fancy reverse proxy: for every call, it records exactly what was consumed, by whom, and writes that somewhere a (finance) team can use.

This is also the part of the project that took the most effort relative to how simple it looks on the diagram, so I am going to spend as much time on what went wrong as on what the design is. If you only read one section, read the one about silent failures.

The pipeline

Hop 1: APIM emits the usage record. In the outbound section of every API policy, a fragment runs after a successful response. It reads the model and usage information, builds a JSON document, and posts it to the Event Hub REST endpoint with a bearer token obtained through APIM's user-assigned managed identity. Only 2xx responses produce a record; a 404 or 429 upstream is not considered as usage.

Hop 2: Event Hub. One hub, ai-usage, with a dedicated consumer group for ingestion, public access disabled, private endpoint in the zone. Retention is a day in non-production and a week in production, which is enough to survive a Function outage without losing records.

Hop 3: the Function App. A Python Function App on the Flex Consumption plan, VNet-integrated into its own subnet, with a user-assigned identity. It has an Event Hub trigger that batches records and upserts them into Cosmos DB, and pushes the same numbers as Prometheus metrics to VictoriaMetrics on the AKS cluster so the platform team has a live view.

Hop 4: Cosmos DB. A SQL API account, public access disabled, local authentication disabled. The usage container is partitioned by product name, because product is what every report groups by.

Hop 5: the streaming side channel. For routes that stream and are not buffered, there is no usage block for APIM to read, so a second function on a five-minute timer queries a token metric from Application Insights and writes aggregate records.

The usage record

The record APIM emits is deliberately richer than "tokens in, tokens out", because everything you want to report on later has to be captured here. It carries the APIM subscription and product (for chargeback), the model as returned by the API and the deployment name as requested, the gateway name and region, the operation and backend, and then the consumption itself: prompt tokens, cached input tokens, cache-write tokens, completion tokens, reasoning tokens and total tokens for language models, or pages processed and document size for OCR. It also carries a usageSource field that says whether the numbers came from the response body (exact) or from an estimate, and an isStream flag.

Two details here saved me later. Capturing cached and cache-write tokens separately mattered as soon as a model appeared (GTP 5.6) with a separate price for cache writes. And the fragment reads both the Chat Completions shape (prompt_tokens, completion_tokens) and the Responses API shape (input_tokens, output_tokens, with nested details), so the same fragment meters both the document-extraction workload and the coding assistant.

Exact versus estimate: the streaming trade-off

In part 3 I mentioned that whether APIM buffers a response is a per-route decision. What makes that important?

When a chat completion is not streamed, the response is one JSON body with a usage block. APIM reads it in the outbound policy and the record is exact.

When a chat completion is streamed, the response is a series of server-sent events, and unless the backend has been asked to include usage in the final chunk and APIM has buffered the whole stream, there is nothing for the outbound policy to read. You can buffer, in which case the fragment parses the terminal event and you still get exact numbers, at the cost of the caller waiting for the full answer. Or you do not buffer, in which case the fragment falls back to the estimate that APIM's own token-limit policy computed on the way in, marks the record usageSource = estimate, and the caller gets live token-by-token output.

I made this a per-route toggle in the policy rather than a global choice. The coding-assistant route buffers, because those calls are large. Interactive routes stream. As a safety net, the OpenAI policy also emits a token metric to Application Insights on every call, and the timer function scrapes that metric every five minutes and writes aggregate "streaming" records, so even routes that stream without buffering produce a total-tokens number. Those streaming records have no input/output split, which has a consequence for pricing that I will cover in part 5.

The three silent failures

Now the part that took me the most time to figure out. Each of these produced the same symptom: every component reported healthy, every test call returned 200, and the Cosmos container stayed empty. None of them threw an error I could see.

1. The Event Hub logger that does nothing

APIM has a native log-to-eventhub policy backed by a logger resource, and that logger supports managed identity authentication. It is the obvious way to send events to Event Hub. I configured it, the logger resource created fine, the policy validated, calls returned 200, and nothing ever arrived in the hub.

Against a private Event Hub, with public access disabled, the managed-identity logger silently no-ops. I do not know whether that is a limitation of how the logger resolves the endpoint or something about how it acquires the token, and Microsoft's documentation is not clear on it. What I do know is that after adding diagnostic response headers to surface internal policy state, the logger path was being reached and simply had no effect.

The fix was to stop using the logger and emit the record with a send-request policy instead: acquire a token for the Event Hubs audience with authentication-managed-identity, POST the JSON to the hub's REST endpoint. That works reliably, and it has a nice feature that the logger does not have: the endpoint is a named value, so the same fragment works in every environment. Both the token acquisition and the send have ignore-error set so that a metering hiccup never fails a caller's request, and I left a note in the fragment that flipping those to false is the fastest way to see the real error when debugging.

2. The empty setting that on the Function App

The Function App uses managed identity for its host storage as well, because shared keys are disabled on the storage account. That is configured with a triple of settings: AzureWebJobsStorage__accountName, AzureWebJobsStorage__credential = managedidentity, and AzureWebJobsStorage__clientId.

At one point, trying to stop the platform from re-injecting a connection-string value, I set the bare AzureWebJobsStorage key to an empty string. The Function App started. The Event Hub trigger attached. Invocation counts stayed at zero. Event Hub metrics showed Incoming Messages climbing steadily and Outgoing Messages flat at zero.

When AzureWebJobsStorage exists, even as an empty value, the Functions host uses it and ignores the three AzureWebJobsStorage__* settings that configure managed-identity access. The host cannot checkpoint, so the trigger never consumes, and because the host considers itself healthy there is no error to find. The fix is to remove the empty setting entirely, not set it to anything. This one is now the first line in the operations handover, because the symptom (incoming up, outgoing zero) is so specific that it should be the first thing anyone checks.

3. Contributor is not a data reader

Cosmos DB has two separate permission systems. Azure RBAC (Owner, Contributor, Reader) controls the control plane: the account, the containers, the throughput. The data plane, actually reading and writing documents, is governed by Cosmos SQL role definitions, assigned with a separate command, and Azure RBAC is not used.

"But I'm Contributor, why can't I read"? The permission errors are easy to misread as a networking problem, because everything else that fails does so for networking reasons. The fix is the built-in Cosmos DB Built-in Data Contributor role for the Function App identity and Data Reader for the reporting identity, assigned at the data plane, which Terraform can do, and which I added to the IaC code obviously.

Two smaller things that also cost time

The Flex Consumption plan does not support WEBSITE_RUN_FROM_PACKAGE, so the code is deployed with a zip deploy triggered from Terraform. A terraform_data resource hashes the function source, and when the hash changes, a provisioner exports pinned requirements with uv, downloads copies of every Python dependency for the Function App's runtime and bundles them with the code, zips the result and pushes it. It is not elegant, but it means a change to the ingestion code goes through the same merge request and pipeline as everything else.

It looks something like this:

resource "terraform_data" "function_build_deploy" {
  triggers_replace = [
    local.fn_src_hash,
    azurerm_function_app_flex_consumption.ingestion.id,
  ]

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    working_dir = local.fn_src_dir
    command     = <<-EOT
      set -euo pipefail

      # 1. Pin deps from the uv lock.
      uv export --no-hashes --no-dev --format requirements-txt -o requirements.txt

      # 2. Vendor packages for the target runtime.
      rm -rf .python_packages
      uv pip install \
        --target .python_packages/lib/site-packages \
        --python-version 3.14 --only-binary=:all: \
        -r requirements.txt

      # 3. Build the package, excluding local-only stuff.
      mkdir -p ../.terraform.tmp
      rm -f ../.terraform.tmp/function_pkg.zip
      zip -r -q ../.terraform.tmp/function_pkg.zip . \
        -x '.venv/*' -x '__pycache__/*' -x '*.pyc' -x '.python_packages/**/__pycache__/*'

      # 4. Deploy without remote build.
      az functionapp deployment source config-zip \
        --resource-group "${azurerm_resource_group.ailz.name}" \
        --name "${azurerm_function_app_flex_consumption.ingestion.name}" \
        --src ../.terraform.tmp/function_pkg.zip \
        --build-remote true
    EOT
  }
}

And APIM policy expressions are C# with a type-locked variable store. The built-in token-limit policy writes its token estimate as a 64-bit integer; an early version of my fragment read it as a 32-bit one, and the collision produced a policy error that only appeared on some routes. A conversion in the fragment solved it, but it is a reminder that policy expressions are real code and deserve the same care.

The runbook that came out of it

Everything above went into a troubleshooting section at the top of the operations handover, each step tells you which side of the pipeline is broken:

Are records actually missing, or is the report stale? Count recent documents in Cosmos.
Is the gateway returning 200? Only successful calls are metered.
Is the hub flowing? Incoming flat means the policy is not emitting; incoming up and outgoing zero means the Function is not consuming.
Data present but report stale? Refresh, and check the rate card is seeded.

Next: turning it into Euros

The pipeline now produces one trustworthy record per call, with enough detail to price it. Part 5 is about the rate card, computing cost at query time, handling cached and reasoning tokens, and the report that finally gave the finance team a monthly AI bill per cost centre.

If you have hit any of these three failures yourself, I would enjoy hearing about it; and if you would rather not hit them at all, this pipeline is part of the landing zone accelerator I am packaging. Connect with me on LinkedIn or reach out via ascode.nl.

Building an AI Landing Zone on Azure — Part 3: One front door for many models

2026-09-30T00:00:00Z

This is part 3 of a five-part series on building an AI Landing Zone on Azure. Part 1: Why every enterprise needs an AI gateway · Part 2: The platform underneath · Part 3: One front door for many models (this post) · Part 4: Metering every call · Part 5: Turning tokens into euros

With the foundation from part 2 in place, this post is about the gateway itself. The job of the gateway is deceptively simple: be the only way to reach any AI model, and make every call look the same to the platform regardless of which model it goes to. Azure API Management does that job well, but only if you make a few decisions early and stick to them.

Which APIM SKU

The gateway is APIM Premium v2, and it is VNet-injected, not VNet-integrated. The distinction is a bit confusing, so let me explain the difference.

With VNet integration (available on Standard v2 and Premium v2), APIM stays publicly reachable on its own public IP; only its outbound traffic to backends goes through your VNet. It is the right choice when you want a public API gateway that can reach private backends.

With VNet injection (Premium v2 only in the v2 tiers), APIM is deployed into a subnet of your VNet and gets a private IP. There is no public IP for the gateway. Inbound and outbound both flow through the VNet, and the NSG on the APIM subnet actually controls who can reach it.

For a regulated environment the choice was obvious, which is VNet injection. The gateway is reachable only from inside the spoke and peered networks, which in practice means the workloads on the AKS cluster and developer tooling on the private network, the developer portal and management endpoints are private, and the security team can point at an NSG rule as the control. The cost is that the APIM subnet needs service endpoints and NSG rules for everything APIM's control plane talks to (Entra ID, Event Hub, Key Vault, SQL, Storage, Azure Monitor), and that took some iteration to get right. Microsoft's documentation on the differences is quite good, have a look at https://learn.microsoft.com/en-us/azure/api-management/virtual-network-concepts.

A big downside of this solution is obviously cost. Premium v2 is the most exepnsive SKU in the APIM arsenal, but in the case of a regulated financial, there is not much choice. I scaled down to 1 instance in the non-prod environment to keep costs at a "minimum".

One identity for the gateway

APIM runs with a single user-assigned managed identity. I prefer user-assigned over system-assigned, because the identity's lifecycle is independent of the resource. I can create it, assign roles, and then create the APIM instance, rather than the other way around, which makes Terraform dependency ordering much cleaner and means rebuilding APIM does not mess up role assignments.

That identity has exactly two capabilities: Cognitive Services User on the AI Services accounts (so it can call models and Content Safety), and Event Hubs Data Sender on the usage Event Hub (so it can emit usage events, the subject of part 4). It cannot read from Event Hub, cannot touch Cosmos, cannot read Key Vault secrets it does not need. If the gateway were somehow made to misbehave, that list is the full extent of the damage.

The identity's client ID is stored as an APIM named value, along with the audience for AI Services. Policy fragments reference those named values, which means the same policy XML works unchanged across environments.

Policy fragments as the unit of reuse

The single best decision in the gateway design was to build almost nothing directly into an API's policy and instead compose each API from policy fragments. A fragment is a reusable chunk of policy XML that you include by ID. The pattern is borrowed from Microsoft's AI Hub Gateway solution accelerator, which I used as the reference architecture.

The library ended up looking like this:

Fragment	What it does
`aad-auth`	Optionally validates an Entra ID bearer token from the caller, toggled by a named value
`strip-caller-credentials`	Deletes the caller's `api-key` and subscription key headers so they are never forwarded upstream
`backend-mi-auth`	Acquires a token with the gateway's managed identity and sets it as the upstream `Authorization` header
`content-safety-aoai-inbound`	Sends the user prompt to Content Safety (prompt shield + harm analysis) in audit or enforce mode
`content-safety-mistral-outbound`	Same idea for OCR output, scanning the extracted text on the way back
`throttling-events`	Emits a custom metric whenever a backend answers 429 (rate limiting), dimensioned by backend and subscriptiion
`rate-limit-logging`	Logs rate-limit outcomes for the operations team
`usage-ingestion-aoai` / `usage-ingestion-mistral`	Emit the usage record to Event Hub.

An API's policy is then mostly a list of includes:

<inbound>
    <base />
    <include-fragment fragment-id="aad-auth" />
    <include-fragment fragment-id="strip-caller-credentials" />
    <include-fragment fragment-id="content-safety-aoai-inbound" />
    <include-fragment fragment-id="backend-mi-auth" />
    <azure-openai-token-limit ... />
    <azure-openai-emit-token-metric ... />
    <set-backend-service backend-id="azure-openai-backend" />
</inbound>

Two things make this work well in practice. First, the fragments are managed by Terraform from XML files in the repository, so a change to how caller credentials are stripped is one file, one merge request, and it takes effect on every API at once. Second, behaviour that differs per environment or per rollout stage, like whether Entra authentication is required or whether Content Safety blocks or only audits, is driven by named values, not by editing policy. Switching Content Safety from audit to enforce in production is a Terraform variable change.

The credential handover

The most important sequence in the gateway is what happens to authentication between the caller and the model, and it is the reason no application in the eco-system holds a model key.

The caller authenticates to APIM. Depending on the product this is an APIM subscription key, an Entra ID token, or both. APIM validates that, and then the strip-caller-credentials fragment deletes every credential header the caller sent. The backend-mi-auth fragment then asks Entra ID for a token for the AI Services audience using APIM's own managed identity and puts it in the Authorization header. The model backend sees a request from APIM's identity and nothing else.

The consequence is that the model deployments have local key authentication effectively unused (result being they cannot be used directly), the consumer's identity is known to APIM but not to the model, and swapping the backend from one Azure OpenAI account to another, or from one model version to another, is a change to an APIM backend definition that no consumer sees. I used that twice during the build when a preview model version turned out to be unstable and had to be rolled back.

Many backends, one shape

Behind the gateway there are currently six backends. Azure OpenAI in the gpt-5 family serves three distinct use cases, each on its own AI Services account: a document-extraction workload and a summarisation workload that use Chat Completions, and a developer coding-assistant workload that uses the newer Responses API through the unified /openai/v1 endpoint. Mistral Document AI, served through an AI Foundry serverless endpoint, handles OCR, and there are two versions of it live so that consumers can be migrated one at a time. Content Safety is itself a backend, called from policy.

Each backend is exposed as its own API with its own path (/openai, /chatgpt, /mistral, and so on) and its own policy, but because the policies are built from the same fragments, every one of them has the same authentication handover, the same safety scanning where it applies, and the same usage emission.

Where the backends differ, the policies differ in the obvious places. The OpenAI APIs use the built-in azure-openai-token-limit policy for per-subscription tokens-per-minute limits and azure-openai-emit-token-metric to push token counts to Application Insights. The Mistral API is billed per page, not per token, so it uses generic rate and quota limits per subscription instead, and its usage record carries pages processed and document size rather than token counts.

Streaming or not, you must choose

There is one design decision in the gateway that I want to point out here because it comes back in a big way in part 4. Chat completions can be streamed, with the response arriving as a stream of server-sent events, or returned as a single JSON body. Whether APIM buffers the response is set per route in the backend section of the policy, and it has consequences for both the caller experience and for what the gateway can observe.

If APIM buffers the response, it can read the whole body on the way out, including the usage block with exact token counts, but the caller only receives the answer once it is complete. If APIM does not buffer, the caller gets token-by-token output, but the outbound policy cannot read the body and the exact token counts are gone.

I made it a per-route choice. The coding-assistant route, where cost accuracy matters more than a few seconds of latency, buffers and gets exact numbers. Interactive chat routes stream live and fall back to an estimate. How that fallback works is explained in part 4.

Products as the unit of chargeback

APIM products are what turn "a gateway" into "a platform with tenants". Each consuming team or use case gets a product, such as Code Assist, Summariser, or a POC product for experiments, and each product bundles the APIs that team is allowed to call. Subscriptions live under products, and every usage record the gateway emits carries both the subscription and the product name.

That is the whole basis for chargeback in part 5. Finance does not need to know which model a team used; they need to know that the Summariser product cost this much in September. The product is the cost centre. Approval-required products give the platform team a gate for new consumers, and the developer portal (private, of course) gives teams self-service for keys and documentation.

Observability at the gateway

Every API's diagnostics go to Application Insights, and the policies emit custom metrics: token consumption per subscription, product and deployment; OCR requests per subscription; throttling events per backend. Those are what the operations team alerts on. When a backend starts returning 429 because an Azure OpenAI deployment ran out of capacity, the throttling metric spikes with the backend name as a dimension, and the platform team knows which quota to raise before anyone opens a ticket.

Next: the pipeline that watches every call

The gateway now does what it needs to do on the request path. Part 4 is about what happens after the response: the usage-emission fragment, the Event Hub, the Function App and Cosmos DB, and the bugs I tackled, each of which silently produced zero records while everything looked healthy.

As always, if you are working on something similar, or want to see how this gateway pattern could land in your own Azure estate, connect with me on LinkedIn or reach out via ascode.nl.

Building an AI Landing Zone on Azure — Part 2: The platform underneath

2026-09-22T00:00:00Z

This is part 2 of a five-part series on building an AI Landing Zone on Azure. Part 1: Why every enterprise needs an AI gateway · Part 2: The platform underneath (this post) · Part 3: One front door for many models · Part 4: Metering every call · Part 5: Turning tokens into euros

In part 1 I described the shape of the AI Landing Zone I built for a regulated insurer and the principles behind it. Before I get to the interesting AI-specific parts, I need to cover the foundation, because the gateway and the metering pipeline only work the way they do because of decisions made down here. If you have built Azure landing zones before, some of this will be familiar; the AI-specific twist is mostly in how far the "no secrets, private everything" principle is pushed.

Network: a spoke with a very strict door policy

The zone is a single spoke VNet per environment, connected to the Virtual WAN hub. Internet security is enabled on the hub connection, which means all egress from the spoke is forced through the hub Azure Firewall. The firewall is also the VNet's DNS server, so private DNS resolution for all the private endpoints is handled centrally rather than per-spoke.

Inside the spoke there are seven subnets, each with a single purpose:

Subnet	Purpose
`snet-apim`	APIM VNet injection
`snet-appgw`	Application Gateway (the only subnet with a route to the internet)
`snet-app`	AKS nodes
`snet-db`	PostgreSQL Flexible Server (delegated)
`snet-fn`	Function App VNet integration (delegated)
`snet-plink`	Every private endpoint in the zone

Every subnet gets its own NSG. Each NSG has specific allow rules for what that subnet legitimately needs, such as the APIM subnet reaching Entra ID, Event Hub, Key Vault and Azure Monitor, or the database subnet accepting connections from the app and gateway subnets, and then a deny-all inbound and deny-all outbound at priority 4096. Remember by default, all traffic within a VNet is allowed with priority 65000 which breaks the zero-trust principle we want to maintain. The one exception is the Application Gateway subnet, where the default deny breaks the platform's own management traffic (default rule with priority 65001 is in place for that but never reached); I learned that one the hard way and left a comment in the code so nobody "fixes" it back.

Route tables send everything from the app, database, function and APIM subnets to the firewall as next hop. The Application Gateway is a Private Deployment (https://learn.microsoft.com/en-us/azure/application-gateway/application-gateway-private-deployment) and therefor is also routed to the firewall. You need to register the feature to make this work:

az feature register --name EnableApplicationGatewayNetworkIsolation --namespace Microsoft.Network

The practical consequence of all this is that nothing in the zone can talk to anything unless explicitly allowed. That is exactly what you want for a compliant workload.

AKS: private, Cilium, and no node pools to manage

The cluster is a private AKS cluster running Azure Linux 3 on every node. I picked Azure Linux over Ubuntu for the smaller footprint and because Microsoft maintains the image end to end, which matters for the patching story in a regulated environment.

Networking is Azure CNI in overlay mode with Cilium as both the network policy engine and the data plane, with Advanced Container Networking Services enabled for observability and security (this enables Hubble). Outbound type is user-defined routing, so node egress goes through the hub firewall like everything else; there is no NAT gateway or load balancer outbound path to worry about.

The cluster has the OIDC issuer and workload identity enabled. This is the single most important setting on the cluster for this project, and I will come back to it below.

For compute I made a deliberate choice to lean on node auto-provisioning (Karpenter) rather than static node pools. There is a small system node pool for critical add-ons, and a default pool that starts at zero nodes. Karpenter owns everything else, driven by a single NodePool that says: E-series, on-demand, amd64, at least four vCPUs and eight gigabytes, and consolidate empty nodes after thirty minutes. The workloads that run here are memory-hungry document processors that are bursty by nature, so paying for idle nodes made no sense, and Karpenter does a much better job of picking the right SKU for a given pod shape than I would with a fixed pool.

One gotcha worth mentioning: when you combine Karpenter with Cilium you must add the Cilium startup taint to the node template, otherwise pods get scheduled onto nodes before the Cilium agent is ready and they never get networking. It is documented, but it is easy to miss.

Diagnostics for the API server, audit logs and controllers go to Log Analytics, and Defender for Containers is wired to the same workspace. The Azure Policy add-on is on, though I keep policies in audit mode where Terraform owns the configuration, because a policy that mutates resources out-of-band gives you drift in your Terraform state.

GitOps: ArgoCD as the only way things get onto the cluster

Terraform builds the cluster and installs exactly two Helm charts: ArgoCD itself and the ArgoCD "apps" chart that bootstraps an app-of-apps. From that point on, Terraform never touches the cluster again. Everything that runs on it, whether that is cert-manager, external-dns, the ingress controller, VictoriaMetrics or the business applications, is defined in a separate Git repository and reconciled by ArgoCD.

The bootstrap uses ApplicationSets with a Git directory generator: every directory under charts/<namespace>/<chart> in the GitOps repository becomes an ArgoCD application, deployed into the namespace named by the path, with per-environment values files. Adding a new platform component is a directory and a merge request. There is also a separate ArgoCD project for merge-request preview environments, so developers get an ephemeral deployment of their branch without anyone touching production tooling. That is what makes ApplicationSets so powerful: it makes it very easy to deploy an entire application landscape in a short-lived branch.

ArgoCD's UI is behind Entra ID single sign-on with group claims. Admins, developers and readers are Entra groups, and the mapping to ArgoCD roles is in the Helm values. Nobody has an ArgoCD local admin password, because the admin user is disabled at install.

The reason I am strict about this split is auditability. The cluster's state is the Git history of one repository. If the risk department wants to know what changed in production last Tuesday, that question has a precise answer.

Ingress and certificates

Ingress is Application Gateway WAF v2 with the Application Gateway Ingress Controller (AGIC) running in-cluster. The WAF policy runs OWASP 3.2 plus the bot manager ruleset. Note that the Application Gateway fronts the applications on the cluster, not APIM; the AI gateway is only reachable from inside the private network.

cert-manager issues Let's Encrypt certificates with a DNS-01 challenge against the client's public DNS zone, and external-dns keeps the private DNS zone in sync with what is running on the cluster. Both need to write to DNS zones in other subscriptions, which brings me to the part that ties the whole platform together.

Workload identity: the applications never see a secret

Every controller and every application on the cluster that needs to talk to Azure does so through a user-assigned managed identity and a federated credential. The pattern is always the same and it is entirely in Terraform:

Create a user-assigned managed identity for the component.
Create a federated identity credential on it, with the cluster's OIDC issuer as the issuer and system:serviceaccount:<namespace>:<serviceaccount> as the subject.
Assign that identity exactly the roles it needs, at the narrowest scope that works. This implements least-privilege.

The Kubernetes side is just a service account annotated with the identity's client ID. When a pod using that service account asks the Azure SDK for a token, the SDK exchanges the pod's projected Kubernetes token for an Entra ID token. No client secret, no certificate, nothing to rotate, nothing to leak.

Here is what that looks like in practice for this zone:

Component	Identity gets
AGIC	Contributor on the Application Gateway, Reader on the node resource group, a custom "join VNet" role
cert-manager	DNS Zone Contributor on the public zone (in the shared subscription)
external-dns	Private DNS Zone Contributor
AKS Key Vault CSI	Key Vault Secrets reader on the workload's Key Vault
Any AI workload app (frontend + backend)	Key Vault Secrets User on its own Key Vault, Storage Blob Data Contributor on its own container

Notice that each application gets two identities, one for the frontend and one for the backend, and each application gets its own Key Vault. That is more resources than strictly necessary, but it means the blast radius of a compromised frontend pod is one Key Vault and one storage container, and it means I can show a reviewer the exact permissions of any running process in a single table.

Where an application genuinely needs a secret at runtime, such as the connection string for its PostgreSQL database or an OAuth client secret for its own SSO, the CSI driver pulls it from Key Vault into a Kubernetes secret, again authenticated with workload identity. The secret is created by Terraform, stored in Key Vault, and read by CSI. At no point does anyone gets or needs access to the secrets.

The same principle carries across to the PaaS side. Storage accounts have shared key access disabled. Cosmos DB has local authentication disabled and is accessed through its data-plane RBAC roles. Event Hub is managed identity only. Key Vaults use RBAC authorisation with purge protection and 90-day soft delete. When I say "no shared keys anywhere" in part 1, this is what it means in practice: the option is turned off at the resource, so a future engineer cannot quietly turn it back on in an app setting.

Observability

Cluster-level diagnostics go to Log Analytics and Application Insights, and Network Watcher flow logs with traffic analytics are enabled on the VNETs, which is unmissable if you want to do proper zero trust in your VNet. Without VNET flow logs, I would have never found out that even inter-cluster traffic (over the cluster CIDR) is also subject to NSGs. Inside the cluster, VictoriaMetrics handles metrics, logs and traces. I went with VictoriaMetrics over Azure Monitor managed Prometheus mainly for cost and control: the usage pipeline in part 4 pushes token metrics straight into it, and a Grafana dashboard on top gives the platform team a live view of gateway traffic without waiting for a Power BI refresh.

How it is all deployed

Everything described here is a single Terraform root module with in-repo modules for APIM, Cognitive Services and Cosmos DB, plus the client's shared modules for VNet, AKS, Key Vault, storage and PostgreSQL. One variable, environment, selects non-production or production and drives subscriptions, IP ranges, retention and DNS domains from a locals map.

The pipeline is GitLab CI using a shared multi-environment Terraform template. It authenticates to Azure with OIDC (also with managed identities, see my other blog post on https://ascode.nl/blog/gitlab-runners-aks/), so the pipeline has no service principal secret either, and it runs plan and apply per environment. Locally, pre-commit runs terraform fmt, tflint, terraform-docs, checkov and secret detection before anything can be committed. If you don't run pre-commit locally, the pipelien forces it and will fail if pre-commit fails. The README's resource inventory is generated by terraform-docs, which sounds like a small thing but means the documentation cannot drift from the code.

What this foundation buys the AI layer

Everything in the next three parts sits on top of this. The APIM gateway is injected into snet-apim and talks to model backends through private endpoints in snet-plink. The Function App that ingests usage runs in snet-fn with a user-assigned identity, exactly like the pods on the cluster. End users reach the frontend applications on AKS through the Application Gateway; the backend workloads on the cluster call the APIM gateway directly over the private network with an API subscription key, and never hold a model key.

If you take one thing from this post, let it be this: the AI-specific parts of an AI Landing Zone are maybe thirty percent of the work. The rest is a properly built, identity-first, private-by-default platform, and if you skip it, you end up with an AI gateway that has a public IP and a connection string in an app setting, which is not much better than the whatever number of OpenAI resources you started with.

Part 3 is where it gets AI-specific: the gateway itself, and how one APIM instance becomes a single front door for OpenAI and Mistral.

If you are planning a similar platform and want to compare notes, or would like to know what it takes to get this running in your own estate, connect with me on LinkedIn or reach out via ascode.nl.

Building an AI Landing Zone on Azure - Part 1: Why every enterprise needs an AI gateway

2026-09-15T00:00:00Z

This is part 1 of a five-part series on building an AI Landing Zone on Azure. Part 1: Why every enterprise needs an AI gateway (this post) · Part 2: The platform underneath · Part 3: One front door for many models · Part 4: Metering every call · Part 5: Turning tokens into euros

Over the last year I have been building and running an AI Landing Zone for a regulated financial services client operating under DORA. It took considerably more effort than I expected when I started, and most of that effort went into things that are not in any Microsoft quickstart. So I wanted to write it down, because I think the pattern is going to be needed by a lot more organisations than currently realise it.

This first post is about the why and the overall solution. The next four go deeper into the platform, the gateway, the metering pipeline and the chargeback reporting.

The problem: AI adoption is happening whether you plan for it or not

If you work in a mid-size or large organisation, you have heard the phrase "we need to do something with AI" more than you want to. Somebody knows a great company they have worked with before, that can implement a PoC in a couple weeks with pre-configured resources (in IaC if you are "lucky") expecting to integrate with your business systems with a click of a button. Nobody seems to care about security, compliance or governance; speed is the message.

Three months later there are a number of these PoCs running. Nobody knows which ones are in production. Nobody knows what they cost, because the bill is one line item per resource and the resource is shared by three teams. Nobody knows what data is being sent to which model. And when the auditor, or in our case the DORA-driven ICT risk function, asks "which third-party AI services do you depend on, how are they authenticated and what happens if one goes down?" the honest answer is "we don't know".

We all hear this story quite often. It is not a technology failure, it is a platform failure: there was no solution available, so people built their own.

What "landing zone" means here

The general term landing zone means a pre-configured, governed environment that workloads can land in. An AI Landing Zone applies that idea specifically to the consumption of AI models. The goal I set was simple to state and hard to deliver:

Every call to any AI model goes through one private front door, is authenticated with an identity rather than a key, is inspected for safety, is metered, and is attributed to the team that made it, in euros.

Everything else in the architecture follows from that sentence.

The architecture in one picture

Read it left to right. End users reach the frontend applications running on the Kubernetes cluster through an Application Gateway with a WAF in front. Those frontends talk to backend workloads in the same cluster (protected by Cilium network policies) and it is the backends, along with developer tooling on the private network, that call the AI gateway: Azure API Management in its most locked-down configuration, Premium v2, injected into the VNet, no public IP. Consumers authenticate to APIM with an API subscription key that identifies their product and team. APIM validates it, strips it, checks the request against Content Safety, acquires a token with its own managed identity and forwards the call to the right model backend over a private endpoint.

On the way back, APIM emits a usage event: which subscription, which product, which model, how many prompt tokens, cached tokens, completion tokens, reasoning tokens, or for the OCR model, how many pages. That event lands in Event Hub, gets picked up by a small Function App, and is written to Cosmos DB. A rate card, also in Cosmos, prices it. Power BI turns it into a report that the team actually uses.

The whole thing is Terraform. One codebase, two environments, deployed by a GitLab pipeline authenticating with OIDC. No click-ops.

The design principles, and why each one is there

I want to spell out the principles because they explain almost every decision in the later posts. Each one exists because of a real requirement, not because it looked good on a slide.

Private endpoints everywhere. Every PaaS service in the zone, from Cosmos DB to Event Hub to the model endpoints, has public network access disabled and is reachable only through a private endpoint in the VNet. Egress leaves through the hub firewall. For a financial company this is not optional, and it also turns out to be the source of the most interesting challenges, which I will get to in part 4.

No shared keys, anywhere. Storage accounts, Cosmos, Event Hub, Key Vault: shared key authentication is disabled. Every component authenticates with a user-assigned managed identity, and every identity has exactly the roles it needs. The APIM identity can send to Event Hub but not read from it. The Function App identity can read from Event Hub and write to Cosmos, but cannot call a model. When a security reviewer asks "what can this thing do if compromised", the answer fits on one line.

Caller credentials never reach the model. The consumer authenticates to APIM with an Entra ID token or an APIM subscription key. APIM throws that away and authenticates to the backend with its own identity. The model endpoint has no idea the consumer exists. That means model keys do not exist in any application's configuration, and rotating or replacing a backend is invisible to consumers.

Metering is a platform concern, not an application concern. Applications do not report their own usage. The gateway observes it. That is the only way to get numbers you can trust for chargeback, because nobody can forget to instrument, and nobody can under-report.

Cost is computed at query time, from data. The pipeline stores raw usage, never prices. The rate card is a table. When Microsoft changes a price, or a new model appears with a separate rate for cache writes (which happened during the build), you update one row and history re-prices itself. I have seen enough "we hard-coded the price in the ingestion function" pipelines to know how that ends.

One codebase, many environments. Non-production and production are the same Terraform with a different variable. Anything that differs per environment, like the Event Hub endpoint the APIM policy posts to, is a named value or a local, never a copy of the code.

What is actually in the AI Landing Zone

To make this concrete, here is the inventory. I will go through each of these in the coming parts, so this is just a quick summary.

The network is a spoke VNet in a hub-and-spoke topology over Azure Virtual WAN, with dedicated subnets for the APIM injection, the Function App integration, private endpoints, the application gateway, the Kubernetes nodes, and the database. Every subnet has an NSG with an explicit deny-all at the bottom.

The platform is a private AKS cluster running Azure Linux, Cilium, workload identity and Karpenter node auto-provisioning, with ArgoCD doing GitOps for everything that runs on it. Two internal applications live there today, each with their own identities, Key Vault and PostgreSQL database.

The gateway is APIM Premium v2, VNet-injected, with a library of reusable policy fragments and a product per consuming team. It fronts Azure OpenAI (the gpt-5 family, both Chat Completions and the Responses API) and Mistral Document AI for OCR through an AI Foundry serverless endpoint. Supporting services the applications need, such as Azure Maps for geocoding, live in the zone under the same rules: private endpoint, local authentication disabled, managed identity only.

The metering pipeline is APIM → Event Hub → Flex Consumption Function App (Python) → Cosmos DB, with a side channel into VictoriaMetrics for live dashboards.

The reporting is a Cosmos rate card in EUR plus a Power BI / Fabric report that computes cost per product, per subscription, per model, per day.

And the governance wrapper is Terraform with pre-commit hooks running tflint, checkov and secret detection, a GitLab pipeline with OIDC federation to Azure, and remote state shared with the rest of the client's landing zones.

Was it worth it?

Yes, and the reason is not the technology. The reason is what changed organisationally once it existed.

Teams stopped asking for their own OpenAI resource, because getting an APIM subscription in the AI product is faster and comes with a dashboard. The finance team got a daily view of AI spend by user for the first time. The risk function got a single answer to "where do we call AI models from and how", with a diagram they could put in the DORA register. And when a model version turned out to be unstable in preview, I swapped the backend and nobody outside the platform team noticed.

None of that is possible when every team has its own key.

What comes next

In part 2 I will cover the platform underneath: the network, the private AKS cluster, GitOps with ArgoCD, and how workload identity replaces every secret an application would normally need. Part 3 is the gateway itself. Part 4 is the metering pipeline. Part 5 is the chargeback model and the report.

If you are looking at a similar problem, whether it is the regulatory angle, the cost angle or just the "we have eleven OpenAI resources and no idea what they do" angle, I am packaging this landing zone as an accelerator that can be deployed into an existing Azure estate in a matter of weeks rather than months. Feel free to connect with me on LinkedIn or reach out via ascode.nl if you would like to talk it through.

Thanks for reading, see you in part 2.

Building a Cloud-Native PKI with HashiCorp Vault

2026-04-06T00:00:00Z

So here is a problem that sounds straightforward until you actually try to solve it: Microsoft Global Secure Access (GSA) needs a subordinate CA certificate to do TLS inspection, and getting that certificate signed turns out to be a rabbit hole that goes surprisingly deep.

It ends with Vault losing. I will not make you read eleven minutes to find that out: GSA needs a certificate that is both a CA and carries a Server Auth extended key usage, and Vault cannot issue one. Its role endpoint will give you the EKU but not the CA bit; its intermediate endpoints will give you the CA bit but have no ext_key_usage parameter at all. Not in the Terraform provider, and not in the API underneath it.

I finished the build anyway. Partly because by that point I wanted to know whether the rest of it would work (it does) and partly because the interesting part of this was never the GSA certificate. It was a production-grade, HA HashiCorp Vault PKI on Kubernetes, with Azure Key Vault auto-unseal, Entra ID OIDC authentication and a two-tier CA hierarchy, deployed entirely as code with no clicking around in portals.

So: the full build, then the blocker, then what I used instead. If you only came for the blocker fix, jump to signing the GSA CSR.

Update, September 2026

Microsoft has since added a managed certificate option for GSA TLS inspection, currently in preview. Microsoft generates and operates a tenant-specific root CA for you, so the entire bring-your-own-CA problem this post is about simply does not arise. If you are starting today, read the update at the end first, then decide whether you still need any of this.

The Problem

GSA TLS inspection works by acting as a man-in-the-middle for HTTPS traffic. To do this, it needs a subordinate CA certificate - specifically one with:

basicConstraints: critical, CA:TRUE, pathlen:1
keyUsage: critical, keyCertSign, cRLSign
extendedKeyUsage: serverAuth

The CA:TRUE constraint is the key detail here. It means this certificate cannot be issued by a publicly trusted CA - Let's Encrypt, for example, will flat out refuse it. cert-manager will also refuse to even submit a CSR with IsCA: true to Let's Encrypt.

See this link for more details on the actual challenge I was facing.

So you need your own CA. And if you are going to run your own CA, you might as well do it properly.

Why Not Let's Encrypt?

Let's Encrypt only issues end-entity certificates. There is no workaround, no special flag, no exception process. This is by design - a publicly trusted CA issuing subordinate CA certificates to arbitrary customers would be a significant security problem.

If you are running cert-manager and try to submit a CSR with IsCA: true, you will get:

admission webhook "webhook.cert-manager.io" denied the request:
spec.request: Invalid value: "...": encoded CSR error:
IsCA true does not match expected value false

cert-manager catches this before it even reaches Let's Encrypt. So that removes Let's Encrypt as an option. Still love cert-manager though!

Why Not EZCA or Azure Key Vault?

I had a look at EZCA (from Keytos) first. It is a managed CA service that integrates nicely with Azure and Entra ID. The problem is that it does not support the full set of extensions that GSA TLS inspection requires - specifically the pathLenConstraint and the combination of basicConstraints=critical,CA:TRUE with the required key usages. Another dead end.

Azure Key Vault's managed CA integration (via DigiCert or GlobalSign) has the same problem. You do not get full control over the certificate profile, and subordinate CA issuance with custom constraints is not supported.

ADCS would work, but it introduces a significant dependency on Windows infrastructure that I would rather not have. Adding AD services feels... like something from the previous decade. VM's to manage, managing services that are complex for even hardened veterans, and the lack of proper automation - no thanks.

Why HashiCorp Vault?

Vault looked like it gave complete control over certificate profiles. The pki/root/sign-intermediate endpoint lets you specify what goes into the signed certificate - max_path_length, key usages, basic constraints - with no managed-service limitations in the way.

That turned out to be true for everything except the one extension I needed. Hold that thought.

It also runs perfectly well on Kubernetes, has a solid Terraform provider, integrates with Entra ID via OIDC, and supports Azure Key Vault for auto-unseal. The operational model fits well into a Kubernetes-native stack - which in my case means ArgoCD and cert-manager already running.

Plus, I've always wanted to use Vault but never had the right use case for it.

Architecture

Here is what we are building:

The two-tier CA hierarchy is important. The root CA private key never leaves Vault and is only accessible via a break-glass token. Day-to-day operations - signing workload certificates, signing the GSA CSR - go through the issuing CA. If the issuing CA is ever compromised, you revoke it and re-issue from the root without touching the root key.

Prerequisites

AKS cluster with OIDC issuer and Workload Identity enabled
cert-manager already running (we use it for Vault's own TLS certificate)
ArgoCD for GitOps deployment
Helm, Terraform, vault CLI, az CLI, kubectl, jq

Azure: Key Vault and Managed Identity

The first step is creating the Azure Key Vault and the (HSM-backed) key that Vault will use for auto-unseal. We also create a User-Assigned Managed Identity that the Vault pods will use to authenticate to Azure Key Vault via Workload Identity - no client secrets involved.

resource "azurerm_user_assigned_identity" "vault_unseal" {
  name                = "id-vault-unseal"
  resource_group_name = azurerm_resource_group.vault.name
  location            = azurerm_resource_group.vault.location
}

resource "azurerm_key_vault" "vault_unseal" {
  name                       = "kv-vault-unseal"
  resource_group_name        = azurerm_resource_group.vault.name
  location                   = azurerm_resource_group.vault.location
  sku_name                   = "standard"   # HSM-backed keys require premium
  enable_purge_protection    = true
  enable_rbac_authorization  = true
}

resource "azurerm_key_vault_key" "unseal" {
  name         = "vault-unseal-key"
  key_vault_id = azurerm_key_vault.vault_unseal.id
  key_type     = "RSA" # or RSA-HSM if you want to be really secure
  key_size     = 4096
  key_opts     = ["wrapKey", "unwrapKey"]

  rotation_policy {
    automatic {
      time_before_expiry = "P30D"
    }
    expire_after         = "P90D"
    notify_before_expiry = "P29D"
  }
}

resource "azurerm_role_assignment" "vault_unseal_crypto_user" {
  scope                = azurerm_key_vault.vault_unseal.id
  role_definition_name = "Key Vault Crypto User"
  principal_id         = azurerm_user_assigned_identity.vault_unseal.principal_id
}

resource "azurerm_federated_identity_credential" "vault_unseal" {
  name                = "vault-k8s-federated"
  resource_group_name = azurerm_resource_group.vault.name
  parent_id           = azurerm_user_assigned_identity.vault_unseal.id
  audience            = ["api://AzureADTokenExchange"]
  issuer              = var.aks_oidc_issuer_url # the oidc issuer url of your AKS cluster
  subject             = "system:serviceaccount:vault:vault"
}

The federated credential is what links the AKS service account to the managed identity. When the Vault pod starts, the Workload Identity webhook injects a signed OIDC token into the pod, which Azure exchanges for an access token for Key Vault. No secret, no rotation, no risk of credential leakage.

Deploying Vault on Kubernetes

Vault is deployed via Helm, managed by ArgoCD. The key configuration elements in values.yaml:

Auto-unseal via Azure Key Vault:

extraEnvs:
  AZURE_TENANT_ID=xxxxxx
  AZURE_CLIENT_ID=xxxxxx

seal "azurekeyvault" {
  vault_name = "kv-vault-unseal"
  key_name   = "vault-unseal-key"
}

With Workload Identity, AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_FEDERATED_TOKEN_FILE are all injected automatically by the webhook. Vault's Azure SDK picks them up without any additional configuration.

HA Raft with auto-pilot:

storage "raft" {
path    = "/vault/data"
node_id = "${HOSTNAME}"

retry_join {
leader_api_addr = "https://vault-0.vault-internal:8200"
...
}

autopilot {
cleanup_dead_servers      = "true"
min_quorum                = 3
server_stabilization_time = "10s"
}
}

Pod anti-affinity to spread the three nodes across different Kubernetes nodes - standard for any HA setup.

TLS via a cert-manager Certificate resource using a self-signed bootstrap issuer. After the PKI is operational, this gets switched to the Vault issuer.

Initialising the Cluster

With auto-unseal active, initialisation looks different from the classic Vault setup. There are no unseal keys - instead, Vault generates recovery keys that are only needed for disaster recovery scenarios.

kubectl exec -n vault vault-0 -- sh -c \
  "VAULT_CACERT=/vault/userconfig/vault-tls/ca.crt \
   vault operator init \
   -recovery-shares=5 \
   -recovery-threshold=3 \
   -format=json" > vault-recovery-keys.json

Store those recovery keys somewhere safe - not in git. After init, vault-0 unseals automatically via Azure Key Vault. Then join the other nodes:

kubectl exec -n vault vault-1 -- vault operator raft join \
  https://vault-0.vault-internal:8200

kubectl exec -n vault vault-2 -- vault operator raft join \
  https://vault-0.vault-internal:8200

Both nodes will also auto-unseal after joining.

PKI Bootstrap with Terraform

Rather than running shell scripts, the entire PKI setup is done in Terraform. Here is the resource chain:

# Root CA
resource "vault_mount" "pki_root" {
  path        = "pki_root"
  type        = "pki"
  max_lease_ttl_seconds = 315360000   # 10 years
}

resource "vault_pki_secret_backend_root_cert" "root" {
  backend     = vault_mount.pki_root.path
  type        = "internal"
  common_name = "Internal Root CA"
  issuer_name = "internal-root-ca"
  ttl         = "87600h"
  key_type    = "rsa"
  key_bits    = 4096
}

# Issuing CA
resource "vault_mount" "pki_int" {
  path        = "pki_int"
  type        = "pki"
  max_lease_ttl_seconds = 157680000   # 5 years
}

resource "vault_pki_secret_backend_intermediate_cert_request" "issuing" {
  backend     = vault_mount.pki_int.path
  type        = "internal"
  common_name = "Internal Issuing CA"
}

resource "vault_pki_secret_backend_root_sign_intermediate" "issuing" {
  backend         = vault_mount.pki_root.path
  csr             = vault_pki_secret_backend_intermediate_cert_request.issuing.csr
  issuer_ref      = vault_pki_secret_backend_root_cert.root.issuer_id
  ttl             = "43800h"
  max_path_length = 1
}

resource "vault_pki_secret_backend_intermediate_set_signed" "issuing" {
  backend     = vault_mount.pki_int.path
  certificate = join("\n", [
    vault_pki_secret_backend_root_sign_intermediate.issuing.certificate,
    vault_pki_secret_backend_root_sign_intermediate.issuing.issuing_ca,
  ])
}

The key thing here is type = "internal" on the root CA. Vault generates the key pair internally and it never leaves Vault. Combined with the policy setup that restricts pki_root/* access to a break-glass token only, the root CA key is as isolated as you can get in a software PKI.

Signing the GSA CSR - where this falls apart

Line the four requirements up against sign-intermediate, the endpoint you would obviously reach for:

Requirement	`sign-intermediate`	Notes
`basicConstraints=critical,CA:TRUE`	✅ automatic	Hardcoded for this endpoint
`pathLenConstraint=1`	✅ via `max_path_length=1`	Explicit parameter
`keyCertSign, cRLSign`	✅ automatic	Hardcoded for this endpoint
`Server Auth` EKU	❌ no such parameter	And this one is not negotiable

Three out of four is not a passing grade when the fourth is mandatory.

Extended Key Usage is normally an end-entity property - standard subordinate CA certificates do not carry it - and Vault's PKI engine is built on that assumption. So the question becomes whether any other path through Vault can produce a certificate that is both a CA and carries serverAuth.

I worked through all of them.

Every route, and where each one stops

Approach	Result
`vault_pki_secret_backend_root_sign_intermediate` with `ext_key_usage`	The field does not exist on the resource
`vault_pki_secret_backend_intermediate_cert_request` with `ext_key_usage`	The field does not exist on the resource - it supports `key_usage` only, and has no `issuer_name` either
`vault_generic_endpoint` → `sign-intermediate`, passing `ext_key_usage` as raw JSON	Accepted by Terraform, then ignored by the Vault API
`vault_generic_endpoint` → `sign-verbatim`	Refuses a CSR with `isCA=true`
Role endpoint `sign/gsa-subordinate` with `ext_key_usage`	Sets the EKU, but will not set `basicConstraints CA:TRUE`

Read that last pair together, because it is the whole story: the role endpoint gives you the EKU but not the CA bit, and the intermediate endpoints give you the CA bit but not the EKU. There is no endpoint that gives you both.

The CSR route closes off for the same reason. Even if you could get ext_key_usage into the request - and the Terraform resource gives you no field for it - sign-intermediate would only carry CSR extensions through with use_csr_values=true, and that endpoint will not take isCA=true from a CSR in the first place.

This is not a provider gap

It is worth being precise about where the limitation actually lives, because "use the API directly instead of Terraform" is the first thing anyone suggests. Dropping to vault_generic_endpoint - or to curl - changes nothing. The sign-intermediate endpoint does not accept ext_key_usage as a parameter at all. The gap is in Vault's server, and no amount of cleverness in the provider or the request body works around a parameter the API does not implement.

Upload the chain without serverAuth and GSA rejects it, which is GSA behaving correctly: it published the requirement and the certificate does not meet it.

So Vault was the wrong tool for this specific certificate. Everything else in this post works. This one thing does not, and it happened to be the reason I started.

What I used instead

Nathan McNulty has published a script that does exactly this job: Initialize-GSATLSInspection.ps1. It is worth reading even if you never run it, because it shows what the certificate has to look like when nothing is abstracting it away from you.

The approach is different in a way that matters. Rather than asking a PKI product to emit a profile it does not want to emit, it constructs the certificate directly with .NET's CertificateRequest and X509SignatureGenerator (PowerShell 7.4+), builds the DER by hand, and performs the signing through the Azure Key Vault REST API. The root lives in Key Vault as a non-exportable RSA-HSM 4096 key with a ten-year validity, so the private key never leaves the HSM - which gets you most of what the Vault root gave me, without the cluster.

The extensions it sets on the subordinate are the ones GSA actually wants:

basicConstraints:     critical, CA:TRUE, pathlen:1
keyUsage:             critical, digitalSignature, keyCertSign, cRLSign
extendedKeyUsage:     serverAuth (1.3.6.1.5.5.7.3.1)   <- non-critical
subjectKeyIdentifier: derived from the public key
authorityKeyIdentifier: references the issuer SKI

Note digitalSignature in the key usage - my original requirement block above left it out, copied from the documentation page, and it belongs there. The pathlen:1 is deliberate: it permits GSA to create the one issuing CA it needs beneath this root and prevents it going deeper.

When the abstraction will not produce the profile, drop to the layer that will. Building DER by hand feels like a step backwards until you notice it is the only approach that puts you in charge of every byte.

Update: Microsoft now issues the CA for you

Some time after I finished this, Microsoft added a managed certificate option for TLS inspection. It is in preview at the time of writing.

The short version: Microsoft generates a tenant-specific root CA, operates it, holds the private key in its own infrastructure and handles the whole certificate lifecycle including rotation. Default validity is ten years, configurable shorter via the API. Everything this post fought with - producing a certificate that is simultaneously a CA and carries Server Auth EKU, and finding a PKI willing to emit that profile - is gone, because you never produce a certificate at all.

What you still do yourself is the part that was never the hard bit: download the root CA from the TLS inspection settings, push it to your endpoints with Intune or whatever MDM you run, verify it landed in the Trusted Root store on a test device, then enable it and assign your inspection policies. Rotation is the same loop: create the new certificate, deploy the new root, enable it, validate, delete the old one.

You need an Entra Internet Access licence (trial or full) and the Global Secure Access Administrator role.

So was any of this wasted?

For the GSA use case specifically, if you are starting now: use the managed certificate. It is less work, there is no key for you to lose, and the rotation story is better than anything you will hand-roll.

There are still cases where you would not:

It is in preview. Plenty of the organisations I work with cannot put a preview feature on the path of all outbound TLS, and that alone settles it until it goes GA.
Key custody. The private key of a CA that can mint a certificate for any site your staff visit now lives with Microsoft rather than with you. That is a reasonable trade for most, and a conversation with your risk function for some, particularly where a control explicitly requires the organisation to operate its own CA.
You needed a private CA anyway. If workload certificates inside your clusters were already on the roadmap, the Vault build stands on its own merits and GSA was only the thing that prompted it.

That last one is where I landed, and it is covered at the end.

Entra ID OIDC Authentication

With the PKI working, the next step is replacing static Vault tokens with proper Entra ID authentication. The setup uses OIDC with the provider = "azure" config, which tells Vault to read groups directly from the ID token rather than calling the userinfo endpoint. This matters because Microsoft Graph's userinfo endpoint is not reliably reachable from within a Kubernetes cluster, and the groups claim is in the ID token anyway.

resource "vault_jwt_auth_backend" "oidc" {
  type               = "oidc"
  path               = "oidc"
  oidc_discovery_url = "https://login.microsoftonline.com/${var.tenant_id}/v2.0"
  oidc_client_id     = azurerm_user_assigned_identity.vault_oidc.client_id
  oidc_client_secret = data.azurerm_key_vault_secret.vault_oidc_secret.value
  default_role       = "reader"

  provider_config = {
    provider = "azure"
  }
}

Note that even though we use a managed identity as the OIDC client, Vault's OIDC auth backend still requires a client_secret. The managed identity handles the unseal path - for the OIDC login flow, a client secret is unavoidable. We store it in Azure Key Vault and pull it via Terraform, so it never lives in tfvars or state in plaintext.

Group membership is enforced through Vault external groups, not through bound_claims on the role. The external group aliases map Entra group Object IDs to Vault groups, and policies are attached to the groups. Adding a new group is a one-liner in the Terraform locals:

locals {
  vault_groups = {
    "msam-role-sg-vaultadmins"   = vault_policy.superadmin.name
    "msam-role-sg-wladmins"      = vault_policy.pki_issuing_ca.name
    "msam-role-sg-pkioperators"  = vault_policy.gsa_pki_admin.name
    "msam-role-sg-readers"       = vault_policy.pki_read.name
  }
}

The for_each on the group resources picks this up automatically.

Getting Rid of the Root Token

Once OIDC is working and you have verified that your admin group can do everything it needs to, revoke the root token:

vault token revoke "${VAULT_TOKEN}"

A superadmin policy with capabilities = ["create", "read", "update", "delete", "list", "sudo", "patch"] on path "*" covers almost everything the root token can do. The exceptions are a handful of system-level operations like vault operator seal - for those, generate a new root token from recovery keys when needed. That process requires a quorum of recovery key holders, which is exactly the right amount of friction for that level of access.

The root CA mount is isolated separately. The pki-root-admin policy is never assigned to any OIDC role. The only way to touch the root CA is a break-glass token with a short TTL and use limit, generated manually by someone with appropriate access. All access is logged via Vault's audit log.

cert-manager Integration

cert-manager connects to Vault via the vault-issuer ClusterIssuer, using Kubernetes auth so there are no long-lived credentials:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.vault.svc.cluster.local:8200
    path: pki_int/sign/workloads
    caBundle: <base64-root-ca>
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: cert-manager

The cert-manager policy only grants access to pki_int/sign/workloads and pki_int/issue/workloads. It cannot touch the root CA, cannot sign intermediate certificates, and cannot access any other secrets engine. Minimum viable permissions.

Lessons Learned

A few things that cost me time that might save you some:

Vault cannot issue a CA certificate carrying an EKU. The role endpoint sets ext_key_usage but not basicConstraints CA:TRUE; sign-intermediate sets the CA bit but has no ext_key_usage parameter; sign-verbatim refuses an isCA=true CSR outright. Check that your chosen PKI can emit the exact certificate profile you need before you build a cluster around it. Ten minutes reading endpoint parameters at the start would have saved me the whole detour.

"Just call the API directly" is not always an escape hatch. My instinct when the Terraform resource lacked a field was to drop to vault_generic_endpoint and pass the parameter as raw JSON. Terraform accepted it and Vault ignored it. When a field is missing from a well-maintained provider, consider that it may be missing because the API has nothing to bind it to.

Verify the requirement list against a working implementation, not just the documentation. The Microsoft docs page I worked from omits digitalSignature from the key usage. A script that had actually been run in anger had it.

The userinfo endpoint. If you get error reading /userinfo endpoint: EOF in Vault logs, add provider_config = { provider = "azure" } to your OIDC backend config. This tells Vault to read claims from the ID token directly and skip the userinfo call entirely. The groups claim is in the ID token anyway.

External groups are not optional. OIDC auth in Vault does not automatically map group memberships to policies. You need vault_identity_group resources of type external, with vault_identity_group_alias resources mapping the Entra group Object IDs to those groups. Without this, users authenticate successfully but get no policies.

The Raft bootstrap sequence matters. Only initialise vault-0. The other nodes join after init. If you try to initialise before vault-0 is unsealed, the other nodes will loop on failed to get raft challenge. Let vault-0 unseal via Azure Key Vault first, then join the others.

Application Gateway health probes. If you are fronting Vault with an Azure Application Gateway, the health probe needs to hit /v1/sys/health and accept status codes 200-399,429. The default 200-399 range will mark standby nodes as unhealthy. Also make sure the backend hostname override matches a SAN in Vault's TLS certificate, and upload the Vault CA certificate to App Gateway as a trusted root.

Vault's OIDC backend requires a client secret. Even with a managed identity as the OIDC client identity, Vault's OIDC auth backend requires oidc_client_secret to be set. The Azure vault_auth_backend type is for VM/workload MSI auth, not for human interactive login. Keep OIDC for humans, Kubernetes auth for workloads.

Where this ended up

I tore the cluster down. It was built for one certificate it could not produce, nothing else in that environment needed a private CA at the time, and an HA Vault cluster is not something you leave running to admire. The GSA certificate came from Key Vault via the script above and TLS inspection has been running on it since.

I do not regard the build as wasted. The two-tier hierarchy, the Key Vault auto-unseal, the OIDC group mapping and the cert-manager integration all worked exactly as designed, and the Terraform for all of it is reusable the next time a client genuinely needs a private CA - which, in a regulated estate issuing its own workload certificates, happens more often than you would think. What I would not do again is pick the PKI first and check the certificate profile second.

If you are here because you are about to try the same thing, the order is now: use the managed certificate if preview is acceptable to you, fall back to Key Vault and the script if it is not, and do not reach for Vault for this particular certificate at all. What changed is not that the Vault build was wrong. It is that GSA is no longer a reason to do it.

The Terraform, Helm values and ArgoCD manifests behind this live in a private client repository, so there is no link to give you - everything load-bearing is inlined above. If you are putting something similar together and want to talk through the shape of it, get in touch.

Until the next one!

GitLab CI/CD for Cloudflare Pages: Automated Deployments with Preview Environments

2026-01-30T00:00:00Z

Remember when the previous post ended with "Automating deployments - When I find some more time..."? Well, I found the time. Let me walk you through how I set up a GitLab CI/CD pipeline that deploys this website to Cloudflare Pages, complete with preview environments on merge requests.

The Goal

Simple:

Push to a branch, create a merge request → preview deploy on a unique URL
Merge to main → production deploy to ascode.nl
No manual steps. No clicking around in dashboards.

Prerequisites

You need two things from Cloudflare:

Account ID - find this on the sidebar of any zone in your Cloudflare dashboard
API Token - create one at dash.cloudflare.com/profile/api-tokens with these permissions:

Permission	Access
Account · Cloudflare Pages	Edit
Zone · Zone	Read

Add both as CI/CD variables in your GitLab project (Settings → CI/CD → Variables):

CLOUDFLARE_API_TOKEN - masked, protected
CLOUDFLARE_ACCOUNT_ID - masked, protected

Wrangler (the Cloudflare CLI) picks these up automatically, no extra configuration needed.

The Pipeline

The full .gitlab-ci.yml is surprisingly short. Two stages: build the static site with Eleventy, then deploy it with Wrangler.

Build Stage

stages:
  - build
  - deploy

variables:
  CLOUDFLARE_PROJECT_NAME: ascode-web
  NODE_VERSION: "20"

build:
  stage: build
  image: node:${NODE_VERSION}-alpine
  script:
    - npm ci
    - npx @11ty/eleventy
  artifacts:
    paths:
      - _site/
    expire_in: 1 hour
  rules:
    - if: $CI_MERGE_REQUEST_IID
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Nothing fancy. Install dependencies, run Eleventy, and pass the _site/ output as an artifact to the deploy stage. The rules ensure it runs on both merge requests and pushes to main.

Preview Deploys

This is where it gets nice:

deploy:preview:
  stage: deploy
  image: node:${NODE_VERSION}-alpine
  needs:
    - build
  script:
    - npx wrangler pages deploy _site/
        --project-name="${CLOUDFLARE_PROJECT_NAME}"
        --branch="${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}"
        --commit-hash="${CI_COMMIT_SHORT_SHA}"
  environment:
    name: preview/$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    url: https://${CI_MERGE_REQUEST_SOURCE_BRANCH_NAME}.${CLOUDFLARE_PROJECT_NAME}.pages.dev
  rules:
    - if: $CI_MERGE_REQUEST_IID

Every merge request gets its own preview URL based on the branch name. Create a branch called new-blog-post, and you get https://new-blog-post.ascode-web.pages.dev. The GitLab environment link shows up directly in the merge request, so reviewers can click through to the preview without having to figure out the URL.

Wrangler handles everything - if the Cloudflare Pages project does not exist yet, it creates it. It uploads the files, configures the deployment, and gives you the URL. No need to pre-configure anything in the Cloudflare dashboard.

Stopping Preview Environments

When a merge request is merged or closed, the preview environment should be cleaned up in GitLab:

stop:preview:
  stage: deploy
  image: node:${NODE_VERSION}-alpine
  needs: []
  script:
    - echo "Preview environment stopped"
  environment:
    name: preview/$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    action: stop
  rules:
    - if: $CI_MERGE_REQUEST_IID
      when: manual
      allow_failure: true

Note that this only stops the GitLab environment - the Cloudflare preview deployment itself stays around until Cloudflare cleans it up (they do, eventually). If you want to actively delete it, you can add a wrangler pages deployment delete command, but I have not found the need for that.

Production Deploy

deploy:production:
  stage: deploy
  image: node:${NODE_VERSION}-alpine
  needs:
    - build
  script:
    - npx wrangler pages deploy _site/
        --project-name="${CLOUDFLARE_PROJECT_NAME}"
        --branch="main"
        --commit-hash="${CI_COMMIT_SHORT_SHA}"
  environment:
    name: production
    url: https://ascode.nl
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

Merge to main → deploys to production. That's it. The --branch="main" flag tells Cloudflare this is the production deployment, which means it goes live on your custom domain.

A Note on Wrangler and npx

You might notice I am using npx wrangler instead of installing it as a project dependency. This keeps the blog repository clean - Wrangler is a deployment tool, not a site dependency. The npx command downloads it on the fly in the CI environment, which adds a few seconds to the deploy stage but keeps package.json focused on what matters: Eleventy and its plugins.

If the download time bothers you, add wrangler to your devDependencies and it will be installed during npm ci in the build stage. Either way works.

The Result

The workflow is now:

Create a branch and write content
Push and open a merge request
Pipeline builds and deploys a preview - check the result
Merge to main
Pipeline deploys to production - live on ascode.nl

No more manual uploads to Cloudflare Pages. No more "when I find some more time." Just git push and go.

Building a Raspberry Pi5 Cluster with Talos Linux and Cilium

2025-10-23T00:00:00Z

You know when you want to have an environment to play with, test some things, and do some general tinkering around?

Of course you can run this on your laptop (Very Happy Macbook Pro M4 Max owner!) but where is the fun in that? So, I decided to build a Kubernetes cluster consisting of Raspberry Pi's. Now I am the kind of guy that does not go half way, so this cluster not only needs to be small and quiet, but also powerful. Which means I am building this with not a single Raspberry Pi board, but a full-fledged cluster with high-availability and everything.

Here is a picture of the end result:

Let me take you through how I build this!

Kit list

Click to view complete shopping list

For those of you that want to know and possibly want to build this themselves, here is the full kit list:

Item	Amount	Cost at time of writing of article	Total Cost	Where ordered?
Raspberry Pi 5 - 16GB	4	€ 131,50	€ 526	RaspberryPi Store
512GB Raspberry Pi NVMe SSD, 22x30mm	4	€ 54,95	€ 219,8	RaspberryPi Store
Sandisk 32Gb MicroSDHC Extreme PRO	1	€ 16,95	€ 16,95	RaspberryPi Store
Geekworm Cooler for Raspberry Pi 5, passive aluminium heatsink	4	€ 9	€ 36	Amazon
M.2 NVME M-Key and PoE+ HAT for RPi5	4	€ 36,29	€ 145,16	Kiwi El4ctronics
Ubiquiti UniFi Flex 2.5G PoE	1	€ 199,65	€ 199,65	KommaGo
Ubiquiti AC Adapter 210W	1	€ 105,88	€ 105,88	KommaGo
C4Labs 8-Slot Cloudlet Cluster Case	1	€ 72,74	€ 72,74	The Pi Hut

Grand total			€ 1249,44 incl. BTW

I realise this list does not fit everyone's budget, but it is what I went with. Yes, you could get a cheaper PoE switch, but I love Ubiquiti and it is my go-to choice for networking gear. You could get Raspberry Pi4 with just SD Cards in stead of SSD's, of course you can! That is the beauty of these projects, you can do it however you want! In this case, this is my blog and my project so this is what I used.

The cluster case I ordered from a company in the UK, and if you are not located in the UK then, it gets a bit more expensive - like a lot more expensive. That was a bit of an oversight from my part, there is a good alternative here: https://www.amazon.com/GeeekPi-Cabinet-Equipment-RackMate-Rackmount/dp/B0DPGZPTPP?th=1. I don't own a 3D printer but if you do, there are some very good cluster cases out there for you to print yourself. I also removed the fans from the case cause I hate noise - that is where the passive heatsink comes in.

Lastly I had to purchase some network cables to make it all look nice.

Putting it all together

Installing Talos

For those of you that are unfamiliar with Talos: Talos is a Linux distro that is optimised for Kubernetes and is secure, immutable and minimal by default, limiting your Kubernetes attack surface. All system management is done via an API - there is no shell or interactive console.

If you are interested in the philosophy behind this, have a read here.

Installing Talos was a little less easy than I expected, mainly because the Raspberry Pi5 is not officially supported yet. Here is an overview of the (non-Pi5) supported Raspberry Pi series: (https://docs.siderolabs.com/talos/v1.11/platform-specific-installations/single-board-computers/rpi_generic). After a quick search I found a Github repo here that ported the official Talos image to work on Raspberry Pi5. Give them a star to support their work!

Preparing the Raspberry Pi

I had a previous Raspberry Pi OS Lite installed with K3S so my Pi had to be cleared out first:

sudo umount /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1p?
sudo wipefs --all --force /dev/nvme0n1
sudo dd if=/dev/zero of=/dev/nvme0n1 bs=1024 count=1

Also, you want to make sure the boot order is changed so that the Pi attempts NVMe boot first. This does assume you already have an OS on your Pi5:

# Open the Raspberry Pi configuration editor
sudo raspi-config

# Navigate to Advanced Options > Boot > Boot Order
# Highlight 'NVMe/USB Boot' and press enter
# Follow the prompts

When you do not have an OS on your Pi5 yet, you can flash your SD card using the Raspberry Pi Imager. For OS, choose Misc utility images, Bootloader (Pi5 Family), NVMe/USB Boot and start your Pi5 using this card. Once the green light flashes, the Pi5 is configured to boot from NVMe first.

Flashing the OS to an SD Card

On a side note, make sure you use specific fast SD cards as the Pi5 does not like slow cards. The type from the shopping list is good. On another note, I used 1 card. You can get more cards if you are impatient and do not want to re-flash in between tasks.

Here is where it gets a little more interesting. You would think that we flash the Talos OS onto the SD card, right? Well, no, we don't. After spending quite a few hours, I found out that even when you select the NVMe card as the install disk, it just would not install to it. Here is the associated Talos config (that, to be clear, did not work for me):

machine:
  install:
    disk: /dev/nvme0n1 # The disk used for installations.
    image: ghcr.io/siderolabs/installer:v1.11.1 # Allows for supplying the image used to perform the installation.
    wipe: true # Indicates if the installation disk should be wiped at installation time.

I worked around the problem by flashing the SD Card with Raspberry Pi OS using the Raspberry Pi Imager Raspberry Pi OS (other), Raspberry Pi OS Lite (64 bit).

I then used this card to boot the first Pi into Pi OS and then downloading the Talos OS:

curl -LO https://github.com/talos-rpi5/talos-builder/releases/download/v1.11.1-pre/metal-arm64.raw.zst
unzstd metal-arm64.raw.zst

You only need to do this once. Make sure you set the hostname and then use the downloaded image to flash your NVMe disk. Repeat this on each of your nodes:

sudo hostnamectl set-hostname rpi5-1
sudo dd if=metal-arm64.raw of=/dev/nvme0n1 bs=4M status=progress

Make sure you set a unique hostname for each node! Shutdown the node after this step is complete, remove the SD card and power on the node again; it should now boot from the SSD drive.

Preparing Talos Config

To perform any kind of management on Talos, you need to communicate (securely) with the API. In this blog, I use talosctl for management.

brew install siderolabs/tap/talosctl

Once you have talosctl installed, we can create the config to deploy the cluster. We are going to install Cilium, for which we need a Talos Patch to prepare Talos for it. Specifically, no cni should be enabled and kube-proxy will be disabled as it will be replaced by Cilium. The last line will allow workloads to be scheduled on control plane nodes, which is fine for this lab environment but in a production environment maybe not such a good idea.

> cat <<EOF > ./cilium-patch.yaml
  machine:
    install:
      disk: /dev/nvme0
      image: ghcr.io/siderolabs/installer:v1.11.3
      wipe: true
    network:
      interfaces:
      - deviceSelector:
          physical: true
        dhcp: true
  # view more in the complete command sequence

Click to see the complete command sequence

cat <<EOF > ./cilium-patch.yaml
  machine:
    install:
      disk: /dev/nvme0
      image: ghcr.io/siderolabs/installer:v1.11.3
      wipe: true
    network:
      interfaces:
      - deviceSelector:
          physical: true
        dhcp: true
        vip:
          ip: 10.10.10.15
  cluster:
    network:
      cni:
        name: none
    proxy:
      disabled: true
  allowSchedulingOnControlPlanes: true
EOF

export CONTROL_PLANE_IP=10.10.10.11
mkdir _out
talosctl gen config talosrpi5 https://$CONTROL_PLANE_IP$:6443 --output-dir _out --config-patch @cilium-patch.yaml

This will create 3 files in the _out directory:

talosconfig - this is the unique context for your cluster that contains the certificates for authentication
controlplane.yaml - for configuring control plane nodes
worker.yaml - for configuring worker nodes

⚠️ Note: You want your cluster nodes to have a fixed IP address. I found out that starting with DHCP and then setting a fixed IP as part of configuration basically broke the setup. The solution was to make a DHCP reservation for each node in your cluster, making sure that the nodes will always get the same IP address from DHCP.

Install Talos on the first node

We are now ready to configure the first cluster node. Make sure the node is up and has the correct IP address; if you hooked up a display to your RPi then you should see this on screen.

export CONTROL_PLANE_IP=10.10.10.11
talosctl apply-config --insecure --nodes $CONTROL_PLANE_IP --file _out/controlplane.yaml
export TALOSCONFIG="_out/talosconfig"
talosctl config endpoint $CONTROL_PLANE_IP
talosctl config node $CONTROL_PLANE_IP
talosctl bootstrap ## <-- this command only needs to be run ONCE on a control plane node!

If all went well, you can use talosctl health to check the status of your Talos cluster. Get the Kubernetes config using the following command:

talosctl kubeconfig --nodes $CONTROL_PLANE_IP
kubectl get nodes

# you should see something like this
NAME     STATUS   ROLES           AGE   VERSION
rpi5-1   Ready    control-plane   1h    v1.34.0

Install Talos on other nodes

I would recommend to take care of Cilium first, then install the rest of the nodes.

Install Cilium

For those of you that are unfamiliar with Cilium: Cilium is an open-source, eBPF-based networking, security, and observability platform designed for cloud-native environments such as Kubernetes clusters and containerized workloads. Cilium operates as a Container Network Interface (CNI) and provides an advanced, kernel-native framework for container networking, combining eBPF-based performance, granular security, and deep observability.

Start by installing the Cilium CLI as described on https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#install-the-cilium-cli

Next I am going to use Helm to install Cilium onto the cluster using Terraform (actually, I used OpenTofu). For a complete list of values and how to further customise the Helm deployment, look here

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.38.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 3.0.2"
    }
  }
  required_version = ">= 1.10.0"
}
# view more in the complete command sequence

Click to see the complete Terraform configuration with all Helm values

terraform {
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = ">= 2.38.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = ">= 3.0.2"
    }
  }
  required_version = ">= 1.10.0"
}

provider "kubernetes" {
  config_path = "~/.kube/config"
}

provider "helm" {
  kubernetes = {
    config_path = "~/.kube/config"
  }
}

resource "helm_release" "cilium" {
  chart            = "cilium"
  name             = "cilium"
  repository       = "https://helm.cilium.io/"
  version          = "1.18.2"
  create_namespace = false
  namespace        = "kube-system"

  set = [
    {
      name  = "ipam.mode"
      value = "kubernetes"
    },
    {
      name  = "kubeProxyReplacement"
      value = "true"
    },
    {
      name  = "securityContext.capabilities.ciliumAgent"
      value = "{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}"
    },
    {
      name  = "securityContext.capabilities.cleanCiliumState"
      value = "{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}"
    },
    {
      name  = "cgroup.autoMount.enabled"
      value = false
    },
    {
      name  = "cgroup.hostRoot"
      value = "/sys/fs/cgroup"
    },
    {
      name  = "k8sServiceHost"
      value = "localhost"
    },
    {
      name  = "k8sServicePort"
      value = 7445
    },
    {
      name  = "gatewayAPI.enabled"
      value = true
    },
    {
      name  = "gatewayAPI.enableAlpn"
      value = true
    },
    {
      name  = "gatewayAPI.enableAppProtocol"
      value = true
    }
  ]
}

If you want to check what versions are available for Cilium, run the following commands:

helm repo add cilium https://helm.cilium.io/
helm search repo cilium --versions


```bash
# results are something like this
NAME            CHART VERSION   APP VERSION     DESCRIPTION
cilium/cilium   1.18.2          1.18.2          eBPF-based Networking, Security, and Observability
cilium/cilium   1.18.1          1.18.1          eBPF-based Networking, Security, and Observability
cilium/cilium   1.18.0          1.18.0          eBPF-based Networking, Security, and Observability
...

Apply the Terraform code and wait for Cilium to be fully deployed. This can take a bit, be patient.

You can check the status of deployment with either kubectl or the cilium cli.

Install Talos on other nodes

Now that Cilium is installed, we can install the other Talos nodes. When you install Cilium straight after the first Talos node is deployed, the rest of the nodes automatically have the correct Cilium config.

Because of etcd cluster quorum (a two-node etcd cluster is not high availability (HA) because if one node is lost, the cluster will have quorum loss and enter read-only mode), we are going to deploy a total of 3 control plane nodes (hence the allowSchedulingOnControlPlanes: true config, otherwise we would only have 1 worker node on which to deploy workloads).

CONTROL_PLANE_IP=("10.10.10.12" "10.10.10.13")
WORKER_IP="10.10.10.14"
for ip in "${CONTROL_PLANE_IP[@]}"; do
  echo "Applying config to worker node: $ip"
  talosctl apply-config --insecure --nodes "$ip" --file _out/controlplane.yaml
done
talosctl apply-config --insecure --nodes "${WORKER_IP}" --file _out/worker.yaml

The end result will be a 4-node Talos cluster that looks something like this:

kubectl get nodes

NAME     STATUS   ROLES           AGE   VERSION
rpi5-1   Ready    control-plane   1h    v1.34.0
rpi5-2   Ready    control-plane   1h    v1.34.0
rpi5-3   Ready    control-plane   1h    v1.34.0
rpi5-4   Ready    <none>          1h    v1.34.0

> cilium status

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

# view more in the complete command sequence

Click to see complete cluster status and pod listings

kubectl get nodes

NAME     STATUS   ROLES           AGE   VERSION
rpi5-1   Ready    control-plane   1h    v1.34.0
rpi5-2   Ready    control-plane   1h    v1.34.0
rpi5-3   Ready    control-plane   1h    v1.34.0
rpi5-4   Ready    <none>          1h    v1.34.0

cilium status

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 4, Ready: 4/4, Available: 4/4
DaemonSet              cilium-envoy             Desired: 4, Ready: 4/4, Available: 4/4
Deployment             cilium-operator          Desired: 2, Ready: 2/2, Available: 2/2
Containers:            cilium                   Running: 4
                       cilium-envoy             Running: 4
                       cilium-operator          Running: 2
                       clustermesh-apiserver
                       hubble-relay
Cluster Pods:          20/20 managed by Cilium
Helm chart version:    1.18.2
Image versions         cilium             quay.io/cilium/cilium:v1.18.2@sha256:858f807ea4e20e85e3ea3240a762e1f4b29f1cb5bbd0463b8aa77e7b097c0667: 4
                       cilium-envoy       quay.io/cilium/cilium-envoy:v1.34.7-1757592137-1a52bb680a956879722f48c591a2ca90f7791324@sha256:7932d656b63f6f866b6732099d33355184322123cfe1182e6f05175a3bc2e0e0: 4
                       cilium-operator    quay.io/cilium/operator-generic:v1.18.2@sha256:cb4e4ffc5789fd5ff6a534e3b1460623df61cba00f5ea1c7b40153b5efb81805: 2

talosctl health
discovered nodes: ["10.10.10.11" "10.10.10.12" "10.10.10.13" "10.10.10.14"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for no diagnostics: ...
waiting for no diagnostics: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: OK
waiting for all control plane static pods to be running: ...
waiting for all control plane static pods to be running: OK
waiting for all control plane components to be ready: ...
waiting for all control plane components to be ready: OK
waiting for all k8s nodes to report ready: ...
waiting for all k8s nodes to report ready: OK
waiting for kube-proxy to report ready: ...
waiting for kube-proxy to report ready: SKIP
waiting for coredns to report ready: ...
waiting for coredns to report ready: OK
waiting for all k8s nodes to report schedulable: ...
waiting for all k8s nodes to report schedulable: OK

kubectl get pods -A

NAMESPACE       NAME                                               READY   STATUS    RESTARTS      AGE
kube-system     cilium-9tff5                                       1/1     Running   0             1h
kube-system     cilium-envoy-f8z4f                                 1/1     Running   o             1h
kube-system     cilium-envoy-jnj94                                 1/1     Running   0             1h
kube-system     cilium-envoy-qdwcn                                 1/1     Running   o             1h
kube-system     cilium-envoy-ww9pq                                 1/1     Running   o             1h
kube-system     cilium-kxkxb                                       1/1     Running   0             1h
kube-system     cilium-mvggt                                       1/1     Running   0             1h
kube-system     cilium-operator-54d6f6cccb-bkn8k                   1/1     Running   0             1h
kube-system     cilium-operator-54d6f6cccb-cdvz9                   1/1     Running   0             1h
kube-system     cilium-w9fvm                                       1/1     Running   0             1h
kube-system     coredns-7bb49dc74c-nbrd2                           1/1     Running   0             1h
kube-system     coredns-7bb49dc74c-xjkw8                           1/1     Running   0             1h
kube-system     kube-apiserver-rpi5-1                              1/1     Running   0             1h
kube-system     kube-apiserver-rpi5-2                              1/1     Running   0             1h
kube-system     kube-apiserver-rpi5-3                              1/1     Running   0             1h
kube-system     kube-controller-manager-rpi5-1                     1/1     Running   0             1h
kube-system     kube-controller-manager-rpi5-2                     1/1     Running   0             1h
kube-system     kube-controller-manager-rpi5-3                     1/1     Running   0             1h
kube-system     kube-scheduler-rpi5-1                              1/1     Running   0             1h
kube-system     kube-scheduler-rpi5-2                              1/1     Running   0             1h
kube-system     kube-scheduler-rpi5-3                              1/1     Running   0             1h

What's next?

Now that Talos and Cilium are ready, start deploying other services to your cluster. Think about ArgoCD, cert-manager, external-dns or get an intro to what Cilium can actually do with the Star Wars Demo.

Have fun!

Self-hosted Gitlab Runners on AKS, with Managed Identities

2025-09-30T00:00:00Z

This post is long overdue, I've been meaning to write this for a long time but did not get around to doing it. So here goes!

The code used in this post is located in https://gitlab.com/ascodenl/gitlab-runner-aks

The great thing about using Managed Identities in Azure is that they cannot be (ab)used to run elsewhere, like Service Principals can. Yes, you can use workload identities conditional access policies nowadays, but SP's are basically just another username/password combination that can be abused for other purposes than they were meant for. Managed Identities are a special type of Service Principal (which is an Azure Enterprise Application Registration under the hood) that can only be used when attached to an Azure resource. When connected to a Kubernetes service account, the managed identity permissions can be assumed by the pod that has that service account attached. This allows for fine-grained permissions for specific purposes, for specific Gitlab runners. For example, a runner that is purpose-built for building container images that pushes to Azure Container Registry - using a Managed Identity that only has the AcrPush permission on the ACR(s) it needs to push to.

The case for self-hosted Gitlab Runners

"Why not use the Gitlab provided runners"? I hear that quite often. In summary, self-hosted GitLab runners are ideal when you need maximum control, security, and flexibility for your CI/CD jobs, want to optimise costs at scale, or have specific compliance and infrastructure requirements that (what Gitlab calls) Instance runners cannot meet. Yes, it requires maintenance but it outweighs the added benefits - especially in environments with heavy compliance requirements like the Financial industry.

Slightly more detailed, for several reasons:

1. Full Control Over the Build Environment

Self-hosted runners give complete control over the (virtual) hardware, operating system, and software installed on the machines (in our case, Kubernetes) running CI/CD jobs. This allows you to customise environments to match your production setup, install proprietary or legacy tools, and fine-tune performance for specific workloads.

2. Security and Compliance

By running jobs on your own infrastructure, you can ensure sensitive code and data never leave your network. This is especially important for organisations with strict compliance, data residency, or privacy requirements, such as those in the public sector.
Self-hosted runners can be placed behind firewalls or VPNs, further reducing exposure to potential threats.

3. Cost Efficiency and Scalability

For teams with high CI/CD usage, self-hosted runners can be more cost-effective than paying for shared or cloud-hosted runners, especially at scale. You avoid per-minute billing and can utilise existing cloud resources as needed.
You can scale your runners as your needs grow.

4. Performance and Flexibility

Self-hosted runners can be optimised for your specific workloads, providing faster builds and more reliable performance than shared runners.
You can run jobs on specialised hardware (e.g., GPUs, large-memory machines) or in specific environments (e.g., on-premises, in particular cloud regions) that aren't available with GitLab-hosted runners.

5. Advanced Customisation

You can create custom caching strategies, use local Docker registries, or integrate with internal systems and services that aren't accessible from public runners.
Self-hosted runners allow for advanced monitoring, logging, and debugging, giving greater visibility into CI/CD processes.

Types of runners

Gitlab has 3 types of runners:

Instance runners: Shared runners that are built, deployed and managed by Gitlab for generic use.
Group runners: Deployed on Gitlab group level, inherited by all repositories (projects) that are part of that group. Self-hosted and self- managed.
Project runners: Deployed on a single repository. Self-hosted and self-managed.

Types of runtime environments

When registering a GitLab runner, you must select an executor, which determines the environment in which your CI/CD jobs will run. Each executor offers different levels of isolation, scalability, and compatibility, making them suitable for various scenarios.

Executor	Isolation	Typical Use Case	Pros	Cons
Shell	Low	Simple, local jobs	Easy, minimal setup	Low isolation, less secure
Docker	High	Reproducible, isolated builds	Clean, scalable, supports services	Needs Docker, some limits
Docker Autoscaler	High	Scalable cloud builds	Auto-scales, cloud support	Complex setup
Instance	Very High	Full VM per job, high isolation	Max isolation, flexibility	Resource intensive
Kubernetes	High	Cloud-native, Kubernetes environments	Scalable, cloud integration	Needs Kubernetes
SSH	Varies	Remote, legacy, or custom environments	Remote execution	Limited support
VirtualBox/Parallels	High	VM-based isolation on local hardware	Good isolation	Slower, needs virtualization
Custom	Varies	Anything not covered above	Flexible	Requires custom scripts

Choosing the right executor depends on your project's requirements for isolation, scalability, environment, and available infrastructure.

Our default platform of choice is Kubernetes, this article covers the Azure implementation of Kubernetes called Azure Kubernetes Service (AKS)

Creating infrastructure in Azure (or any environment for that matter) is done using Infrastructure as Code (IaC). The tool of choice is Terraform or Tofu, whatever your preference. The idea is to let a CI/CD pipeline handle the creating, updating and destruction of Azure resources, using Gitlab Runners on AKS. For this, we need several resources to make that happen.

Here is a quick overview of what we are building:

Azure configuration

The runner will use a Kubernetes Service account, which is "connected" to an Azure Managed Identity, that will be assigned roles with permissions to create resources in Azure. You will need an AKS cluster with OIDC issuer enabled. Read here how to enable if not configured yet.

The Managed Identity is created as a separate resource outside of any Terraform Module. The main reason for this is, we use RBAC to assign permissions on Azure Resources. If you want to allow a Managed Identity access to Entra ID (to read groups, primarily), assigning permissions in Entra ID requires elevated privileges that we do not want to delegate to a Managed Identity. To prevent (re)creation of a Managed Identity as part of a module, we create it separately.

locals {
  runners = {
    tf = {
      environment      = "ss"
      environment_long = "shared"
      runner_type      = "group_type"
      runner_tags      = ["azure", "platform", "ss"]
      repo_path        = "ascodenl/infra"
      run_privileged   = "true"
      purpose          = "Runner for deploying Terraform resources in the Azure Landing Zone using Gitlab"
    }
# view more in the complete command sequence

Click to see complete Terraform code

locals {
  runners = {
    tf = {
      environment      = "ss"
      environment_long = "shared"
      runner_type      = "group_type"
      runner_tags      = ["azure", "platform", "ss"]
      repo_path        = "ascodenl/infra"
      run_privileged   = "true"
      purpose          = "Runner for deploying Terraform resources in the Azure Landing Zone using Gitlab"
    }
    packer = {
      environment      = "ss"
      environment_long = "shared"
      runner_type      = "project_type"
      runner_tags      = ["infra-packer-azure", "ss"]
      repo_path        = "ascodenl/infra/base-images/azure"
      run_privileged   = "true"
      purpose          = "Gitlab runner for deploying Custom images to Shared Imaged Gallery using Packer in Azure"
    }
  }
}

resource "azurerm_user_assigned_identity" "gitlab_runner" {
  for_each            = local.runners
  name                = "${each.value.region_code}-${each.value.environment}-msi-gitlab-${each.key}"
  resource_group_name = azurerm_resource_group.gitlab[each.value.region_code].name
  location            = azurerm_resource_group.gitlab[each.value.region_code].location
  tags = merge(local.tags, {
    purpose = each.value.purpose
  })
  lifecycle {
    ignore_changes = [tags["CreatedOnDateTime"]]
  }
}

##
## Create the MSGraph application
##

data "azuread_application_published_app_ids" "well_known" {}

resource "azuread_service_principal" "msgraph" {
  client_id    = data.azuread_application_published_app_ids.well_known.result.MicrosoftGraph
  use_existing = true
}

##
## Assign the Permissions in Entra ID
##

resource "azuread_app_role_assignment" "gitlab_api_permissions" {
  for_each            = toset(["Group.Read.All", "User.Read.All", "GroupMember.Read.All", "Directory.Read.All"])
  app_role_id         = azuread_service_principal.msgraph.app_role_ids[each.key]
  principal_object_id = azurerm_user_assigned_identity.gitlab_runner["tf"].principal_id
  resource_object_id  = azuread_service_principal.msgraph.object_id
}

##
## Create a federated credential
##

resource "azurerm_federated_identity_credential" "gitlab_runner" {
  for_each            = local.runners
  name                = "ascode-${each.value.environment_long}-msi-gitlab-${each.key}-cred"
  resource_group_name = azurerm_resource_group.gitlab.name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = module.aks.oidc_issuer_url
  parent_id           = azurerm_user_assigned_identity.gitlab_runner[each.key].id
  subject             = "system:serviceaccount:${module.runners[each.key].k8s_sa_namespace}:${module.runners[each.key].k8s_sa_name}"
}

##
## Assign permissions in Azure
##

# MSI is owner on all subscriptions
resource "azurerm_role_assignment" "gitlab_runner_platform_owner" {
  for_each             = local.subscription_id
  scope                = "/subscriptions/${each.value}"
  role_definition_name = "Owner"
  principal_id         = azurerm_user_assigned_identity.gitlab_runner["tf"].principal_id
}

Note that we make the MSI specific to deploying Terraform resources (azurerm_user_assigned_identity.gitlab_runner["tf"].principal_id) owner of the subscriptions. Contributor is not going to be enough as we also want to use the pipeline to do role assignments and RBAC, therefor it needs owner permissions.

⚠️ Warning: This means that this MSI has very powerful privileges! Make sure you lock down your pipelines so that not just anyone can run them and make sure you do proper Merge Requests and code reviews!

Kubernetes resources

Remember we are using Kubernetes as the Gitlab Executor. What we are deploying is what can be described as a "runner manager", which will spin off containers (pods, actually) that will run the pipeline. Once the pipeline is finished, the pod is destroyed.

The Gitlab Runner is deployed using Helm. Gitlab maintains a Helm chart that you can find on https://gitlab.com/gitlab-org/charts/gitlab-runner/.

Gitlab Runner configuration is done in a config.toml file that we deploy using a template.

gitlabUrl: "${gitlab_url}"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
serviceAccountName: ${rbac_service_account_name}
# view more in the complete configuration

Click to see GitLab Runner Helm Values Configuration

gitlabUrl: "${gitlab_url}"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
serviceAccountName: ${rbac_service_account_name}
rbac:
  create: false
serviceAccountAnnotations: {
    azure.workload.identity/client-id: ${client_id},
    azure.workload.identity/tenant-id: ${tenant_id}
}
metrics:
  enabled: true
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        pod_labels_overwrite_allowed = ".*"
        privileged = ${run_privileged}
        service_account = "${service_account_name}"
        namespace = "${runner_namespace}"
      [runners.kubernetes.pod_labels]
        "azure.workload.identity/use" = '"true"'
      [[runners.kubernetes.volumes.empty_dir]]
        name = "docker-certs"
        mount_path = "/certs/client"
        medium = "Memory"
  name: "${gitlab_name}"
  image: "${gitlab_docker_image}"
  privileged: ${run_privileged}
  serviceAccountName: ${service_account_name}
  pull_policy: always
  secret: "${runner_secret}"
securityContext:
  runAsUser: 100
deploymentLabels: {
  deployment_repository: ${deployment_repo_path}
}
podLabels: {
    azure.workload.identity/use: "true"
}

Then we use Terraform to transform the template and deploy the Helm chart:

locals {
  gitlab_runner_vars = {
    gitlab_url                = var.gitlab_url
    gitlab_docker_image       = var.gitlab_docker_image
    gitlab_name               = "${var.environment}-k8-${var.runner_name}"
# view more in the complete chart

Click to see Terraform deploy Helm Chart

locals {
  gitlab_runner_vars = {
    gitlab_url                = var.gitlab_url
    gitlab_docker_image       = var.gitlab_docker_image
    gitlab_name               = "${var.environment}-k8-${var.runner_name}"
    run_privileged            = var.run_privileged
    service_account_name      = kubernetes_service_account_v1.gitlab_runner.metadata[0].name
    rbac_service_account_name = kubernetes_service_account_v1.gitlab_runner.metadata[0].name
    runner_namespace          = kubernetes_namespace_v1.gitlab.metadata[0].name
    client_id                 = var.mi_client_id
    tenant_id                 = var.tenant_id
    runner_secret             = kubernetes_secret_v1.gitlab_runner.metadata[0].name
    deployment_repo_path      = var.deployment_repo_path
  }
}

resource "helm_release" "gitlab_runner" {
  chart            = "gitlab-runner"
  name             = "${var.customer_tla}-${var.environment}-${var.runner_name}"
  namespace        = kubernetes_namespace_v1.gitlab.metadata[0].name
  repository       = "https://charts.gitlab.io"
  version          = var.gitlab_runner_helm_release_version
  create_namespace = false

  values = [
    templatefile("${path.module}/templates/runner-values.yml.tpl", local.gitlab_runner_vars)
  ]
}

To check for the latest version(s) of the chart:

helm repo add gitlab-runner https://charts.gitlab.io
helm repo update
helm search repo-l gitlab/gitlab-runner | head -5

The last part is to create the Kubernetes Service Account that "glues" the Managed Identity to the Gitlab Runner:

resource "kubernetes_service_account_v1" "gitlab_runner" {
  metadata {
    name      = "gitlab-runner-${random_id.this.hex}"
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
    annotations = {
      "azure.workload.identity/client-id" = var.msi_client_id
      "azure.workload.identity/tenant-id" = var.tenant_id
    }
    labels = {
      "azure.workload.identity/use" = "true"
    }
  }
}

Finally some required Kubernetes resources to make this all work (I use rbac.create = false in the config.toml because I like to be in control of what is created. Setting this to true means you have to annotate the Service Account with the correct values as it gets auto-created).

resource "kubernetes_namespace_v1" "gitlab" {
  metadata {
    name = "gitlab-${random_id.this.hex}"
  }
}
# view more in the complete Kubernetes resources

Click to see Terraform Kubernetes resources

resource "kubernetes_namespace_v1" "gitlab" {
  metadata {
    name = "gitlab-${random_id.this.hex}"
  }
}

resource "kubernetes_role" "gitlab" {
  metadata {
    name      = "gitlab-runner-${random_id.this.hex}"
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
  }
  rule {
    api_groups = [""]
    resources  = ["configmaps", "pods", "pods/attach", "secrets", "services", "namespaces"]
    verbs      = ["get", "list", "watch", "create", "patch", "update", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["pods/exec"]
    verbs      = ["create", "patch", "delete"]
  }
  rule {
    api_groups = [""]
    resources  = ["serviceAccounts"]
    verbs      = ["get"]
  }
}

resource "kubernetes_role_binding_v1" "gitlab" {
  metadata {
    name      = "gitlab-runner-binding-${random_id.this.hex}"
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "Role"
    name      = kubernetes_role.gitlab.metadata[0].name
  }
  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account_v1.gitlab_runner.metadata[0].name
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
  }
}

resource "kubernetes_cluster_role_binding_v1" "runner_admin" {
  metadata {
    name = "gitlab-runner-admin-${random_id.this.hex}"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }
  subject {
    kind      = "ServiceAccount"
    name      = kubernetes_service_account_v1.gitlab_runner.metadata[0].name
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
  }
}

Runner registration

ℹ️ Note: This article describes the new way of registering runners. Please see https://docs.gitlab.com/ci/runners/new_creation_workflow/ how to migrate.

When deploying a runner, it needs to be registered against a group or repository. Each group or repository has its own unique token. You can do this from the CI/CD settings of the repository or group, create a project or group runner, fill in the details and out comes a registration token. But who wants do do manual? Let's automate this.

Terraform has a great provider for Gitlab, found on https://registry.terraform.io/providers/gitlabhq/gitlab/latest/docs. You can use it to fully automate your Gitlab environment, including repositories, groups, authorisations, integrations, etc. We'll focus on the gitlab_user_runner resource to get the registration token.

Each group or project in Gitlab has a unique id, which is hard to find and even harder to remember. We use the path to find the id, which is a lot easier to remember. If you use Terraform to also create your groups and projects, you can even reference the Terraform resource!

Terraform provider configuration is required for Gitlab and Kubernetes (the registration token is stored in a Kubernetes secret):

provider "kubernetes" {
  config_path = "~/.kube/config" # Need to create this file from the pipeline or run locally
}

provider "gitlab" {
  base_url = "https://gitlab.com/"
  token    = data.azurerm_key_vault_secret.gitlab_token.value # this can be a Group token or a PAT token with the create_runner scope
}

First, we need to determine if we are deploying a group runner or a project runner:

data "gitlab_group" "group" {
  count     = var.runner_type == "group_type" ? 1 : 0
  full_path = var.repo_path
}

data "gitlab_project" "project" {
  count               = var.runner_type == "project_type" ? 1 : 0
  path_with_namespace = var.repo_path
}

repo_path is the path to your repo, for example ascodenl/infra/tools.

Then, dependent on what type of runner you want, create a token and store it in a Kubernetes secret. Note the reference to a Kubernetes Service Account, this will become clear later on.

resource "gitlab_user_runner" "gitlab_runner_project" {
  count       = var.runner_type == "project_type" ? 1 : 0
  runner_type = var.runner_type
  description = "${kubernetes_service_account_v1.gitlab_runner.metadata[0].name}-${var.region_code}-aks"
# view more code below

Click to see Terraform Runner Token Creation

resource "gitlab_user_runner" "gitlab_runner_project" {
  count       = var.runner_type == "project_type" ? 1 : 0
  runner_type = var.runner_type
  description = "${kubernetes_service_account_v1.gitlab_runner.metadata[0].name}-${var.region_code}-aks"
  project_id  = data.gitlab_project.project[0].id
  tag_list    = flatten([[var.region_code], [var.runner_name], var.runner_tags])
  untagged    = var.run_untagged
}

resource "gitlab_user_runner" "gitlab_runner_group" {
  count       = var.runner_type == "group_type" ? 1 : 0
  runner_type = var.runner_type
  description = "${kubernetes_service_account_v1.gitlab_runner.metadata[0].name}-${var.region_code}-aks"
  group_id    = data.gitlab_group.group[0].id
  tag_list    = flatten([[var.region_code], [var.runner_name], var.runner_tags])
  untagged    = var.run_untagged
}

resource "kubernetes_secret_v1" "gitlab_runner" {
  metadata {
    name      = kubernetes_service_account_v1.gitlab_runner.metadata[0].name
    namespace = kubernetes_namespace_v1.gitlab.metadata[0].name
  }
  type = "Opaque"
  data = {
    runner-registration-token = "" # need to leave as an empty string for compatibility reasons
    runner-token              = var.runner_type == "project_type" ? gitlab_user_runner.gitlab_runner_project[0].token : gitlab_user_runner.gitlab_runner_group[0].token
  }
}

Using this in a Gitlab pipeline

Now that the runner is deployed with the proper permissions, it is time to create a pipeline to implement this in CI/CD.

Creating a full multi environment pipeline is enough for a separate blog post, so here is the most important part:

before_script:
  - |
    if ! [ -x "$(command -v az)" ]; then
      echo -e "\e[33mWarn: az is not installed.\e[0m"
      exit 1
    else
      echo "Logging in to Azure using client_id $AZURE_CLIENT_ID..."
      az login --service-principal -u $AZURE_CLIENT_ID --tenant $AZURE_TENANT_ID --federated-token $(cat $AZURE_FEDERATED_TOKEN_FILE)
      if [[ ! -z ${ARM_SUBSCRIPTION_NAME} ]]; then az account set -n ${ARM_SUBSCRIPTION_NAME}; fi
      export ARM_OIDC_TOKEN=$(cat $AZURE_FEDERATED_TOKEN_FILE)
      export ARM_CLIENT_ID=$AZURE_CLIENT_ID
      export ARM_TENANT_ID=$AZURE_TENANT_ID
    fi

If OIDC is working correctly, the Azure token is stored in a file that is referenced in $AZURE_FEDERATED_TOKEN_FILE which usually points to /var/run/secrets/azure/tokens/azure-identity-token. Enabling OIDC on AKS deploys something called a "Mutating Admission Webhook" which takes care of getting a token that has a limited lifetime, and refreshes the token on expiration. If you are interested in how this works under the hood, look here.

I hope you enjoyed this, thanks for sticking around till the end. Until the next!

New Website, refactored

2025-05-04T00:00:00Z

Having neglected our web site for a long time, I read a post on X of someone that used Eleventy to create his web site. Being a guy that always likes to try out new technology, I thought I'd give it a go. Our previous website was created with Hugo, which is also great but still required a lot of tweaking to get right. Since I am not a web developer by trade, reading up on Eleventy seemed like it could make the job easier.

Getting started

I used the blog template available on Github to get started, cloned it into my local git environment and started customising it to the Ascode look and feel. One if the nice features of Eleventy (same with Hugo) is that you can run a local web server using eleventy --serve that starts a web server on http://localhost:8081 together with a file system watcher that once a change is made to a file, immediately processes the change and makes your changes visible through the local web server. This is great for immediate feedback on the result of the changes that were made.

There are a bunch of plugins available, we wanted to get up and running quickly so I kept the first version really simple.

Hosting on Cloudflare Pages

The previous web site was a really nice project to test out the serverless features of AWS, but it involved deploying and maintaining quite a bit of resources so we decided to keep it really simple this time and host the web site with Cloudflare Pages.

You can generate the static content for upload to Cloudflare by running the command npx @11ty/eleventy which generates all required files and stores them by default in the _site folder. Then, upload the contents of the _site folder to Cloudflare (in the Compute (Workers)/Workers & Pages section). The name you give the project is the name that is used to generate the default Cloudflare Pages url, in this case the name is ascode-web which automatically generates the url https://ascode-web.pages.dev/. That's it for basics, pretty simple - love it.

The first go for me resulted in none of the pictures being visible, they contained broken links. After some troubleshooting I found the --serve option does not clean up after itself; after removing the local _site folder and re-running eleventy --serve all was looking fine again.

Here is the documentation for the image plugin that might help you out: https://www.11ty.dev/docs/plugins/image/

Setting a custom domain

We are hosting the ascode.nl at Cloudflare which makes it super-easy to add the custom domain ascode.nl to Cloudflare Pages. Just add the custom domain on the Workers & Pages page and Cloudflare will take care of creating or updating the required DNS record, and generate an SSL certificate with the same name. Keep in mind that ascode.nl and www.ascode.nl are 2 separate custom domains in the Pages sense and need to be generated separately.

Automating deployments

When I find some more time...

Shared VPC Subnet tagging

2023-11-06T00:00:00Z

This post was previously published on lionsville.nl. That site is gone, so it is re-published here.

One of the services we provide for our customers is the deployment and management of an AWS Landing Zone. Part of the Landing Zone is a centralised networking environment that is used as the centralised VPC for ingress and egress network traffic. VPC sharing is used, implementation varies, for this article we implement a shared VPC per AWS account with 3 subnets per VPC.

Each AWS Account gets its own VPC with 3 type of subnets:

Isolated
Private
TransitGateway

With 1 subnet per Availability Zone, a total of 9 subnets are created. The subnets are then shared with Resource Access Manager (RAM) with the Team account.

For the sake of simplicity, this article will only focus on the subnet sharing part of this implementation. One of the big downsides of RAM is that the shared resource in the Team account does not inherit tags. Especially for implementations like EKS or other AWS services that rely on tagging, this is a problem. As part of the VPC deployment in the Shared Network account, several tags are created of which a few of them we would like to make available in the Team account. The tags that are discussed in this post are Name, AccountID, subnetType and aws-cdk:subnet-type.

Process

EventBridge detects an AssociateResourceShare CloudWatch Event.
EventBridge triggers the Lambda function with the CloudWatch event json as input.
The Lambda function looks up the subnet-id that is in the body of the CloudWatch event and grabs the tags that are configured on that subnet.
The Lambda function assumes a role in the team account.
The Lambda function updates the tags on the subnets in the team account.

The process in the will be discussed in more detail, using code examples (Terraform and Lambda). The code can be found here.

If you want EventBridge to be able to record events with a detail-type value of AWS API Call via CloudTrail, a CloudTrail trail with logging enabled is required. The event that EventBridge is configured to look for is the AssociateResourceShare event. Check the API Documentation for the different events that are available for RAM. The target for the event is a Lambda function that was created by Markus Muhr. Two resources are created, an event rule and an event target.

resource "aws_cloudwatch_event_rule" "ram" {
  name        = "capture-ram-creation"
  description = "Capture creation of RAM shares"

  event_pattern = jsonencode({
    "source" : ["aws.ram"],
    "detail-type" : ["AWS API Call via CloudTrail"],
    "detail" : {
      "eventSource" : ["ram.amazonaws.com"],
      "eventName" : ["AssociateResourceShare"]
    }
  })
}

resource "aws_cloudwatch_event_target" "lambda" {
  target_id = "lambda-function-target"
  rule      = aws_cloudwatch_event_rule.ram.name
  arn       = aws_lambda_alias.tagger_alias.arn
}

Lambda function

The Lambda function contains several functions:

lookup the shared subnet
assume the role
tag the team subnet
the main Lambda event handler

The example shows a simplified version of the actual function that was implemented, it should however be easy to extend upon this code.

IAM role in Team account(s)

The team account requires an IAM role to be created that can be assumed by the Lambda function in order to create the tags. This is a very simple role, which is created using org-formation In short, in order to use Terraform several resources like an S3 bucket for state backend and IAM roles are required. Org-formation uses CloudFormation under the hood for creating the resources that are required for Terraform and resources that need to be created in all accounts. When AWS Control Tower is not an option, we leverage org-formation as our account vending machine in AWS. Note the OrganizationBinding parameter, this is specific to org-formation to target or filter only specific accounts or OU's. If you don't use org-formation, omit this parameter.

LambdaTaggerPolicy:
  Type: "AWS::IAM::ManagedPolicy"
  OrganizationBinding: !Ref TeamAccountBinding
  Properties:
    ManagedPolicyName: LambdaTaggerPolicy
    Description: Policy to allow access from lz-net Lambda function
    PolicyDocument:
      Version: 2012-10-17
      Statement:
        - Effect: Allow
          Action:
            - "ec2:CreateTags"
          Resource:
            - "arn:aws:ec2:*:*:subnet/*"
            - "arn:aws:ec2:*:*:vpc/*"
    Roles:
      - !Ref LambdaTaggerRole

LambdaTaggerRole:
  Type: AWS::IAM::Role
  OrganizationBinding: !Ref TeamAccountBinding
  Properties:
    RoleName: LambdaTaggerRole
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - "sts:AssumeRole"
        - Effect: Allow
          Principal:
            AWS:
              - "<central network account ID>"
          Action:
            - "sts:AssumeRole"

That's it! Even though the current Lambda is “only” used to tag resources, you can imagine this concept can be used for all sorts of implementations. Go nuts!

Ascode - Erik Christiaans

Building an AI Landing Zone on Azure — Part 4: Metering every call

The pipeline

The usage record

Exact versus estimate: the streaming trade-off

The three silent failures

1. The Event Hub logger that does nothing

2. The empty setting that on the Function App

3. Contributor is not a data reader

Two smaller things that also cost time

The runbook that came out of it

Next: turning it into Euros

Building an AI Landing Zone on Azure — Part 3: One front door for many models

Which APIM SKU

One identity for the gateway

Policy fragments as the unit of reuse

The credential handover

Many backends, one shape

Streaming or not, you must choose

Products as the unit of chargeback

Observability at the gateway

Next: the pipeline that watches every call

Building an AI Landing Zone on Azure — Part 2: The platform underneath

Network: a spoke with a very strict door policy

AKS: private, Cilium, and no node pools to manage

GitOps: ArgoCD as the only way things get onto the cluster

Ingress and certificates

Workload identity: the applications never see a secret

Observability

How it is all deployed

What this foundation buys the AI layer

Building an AI Landing Zone on Azure - Part 1: Why every enterprise needs an AI gateway

The problem: AI adoption is happening whether you plan for it or not

What "landing zone" means here

The architecture in one picture

The design principles, and why each one is there

What is actually in the AI Landing Zone

Was it worth it?

What comes next

Building a Cloud-Native PKI with HashiCorp Vault

The Problem

Why Not Let's Encrypt?

Why Not EZCA or Azure Key Vault?

Why HashiCorp Vault?

Architecture

Prerequisites

Azure: Key Vault and Managed Identity

Deploying Vault on Kubernetes

Initialising the Cluster

PKI Bootstrap with Terraform

Signing the GSA CSR - where this falls apart

Every route, and where each one stops

This is not a provider gap

What I used instead

Update: Microsoft now issues the CA for you

So was any of this wasted?

Entra ID OIDC Authentication

Getting Rid of the Root Token

cert-manager Integration

Lessons Learned

Where this ended up

GitLab CI/CD for Cloudflare Pages: Automated Deployments with Preview Environments

The Goal

Prerequisites

The Pipeline

Build Stage

Preview Deploys

Stopping Preview Environments

Production Deploy

A Note on Wrangler and npx

The Result

Building a Raspberry Pi5 Cluster with Talos Linux and Cilium

Table of Contents

Kit list

Putting it all together

Installing Talos

Preparing the Raspberry Pi

Flashing the OS to an SD Card

Preparing Talos Config

Install Talos on the first node