In this guide (47 items)
API Gateway
API Gateway Guidelines
MANDATORY FOR ALL AI AGENTS: Every public REST API in a multi-service platform sits behind an API gateway. The gateway terminates TLS, validates auth, enforces edge rate limits, propagates trace context, and aggregates responses where it makes sense. Services downstream MUST trust forwarded identity headers ONLY when set by the gateway (mTLS or private network). The gateway is part of the edge tier — distinct from per-service Spring Security, which is the defense-in-depth tier.
Project-agnostic: code samples use placeholder service names (
platform-auth-service,platform-payment-service), domains (api.example.com,auth.example.com), and base package (com.example.platform). Replace with your project’s actual values.
Reference implementation: KrakenD
2.13.xwith Flexible Configuration + Go custom plugins. KrakenD is the recommended default for declarative, code-light gateways in JVM platforms (no JVM dependency in the gateway itself, ~10ms p99 overhead, declarative JSON config).
Companion guides:
- api-design.md — endpoint contracts that the gateway routes.
- security.md — auth tier defense in depth; JWT validation downstream.
- obs-tracing.md — W3C Trace Context propagation through and beyond the gateway.
- adr.md — gateway choice + topology + auth model are ADR-worthy.
- arch-doc-spec.md §System Context — gateway is the C1 boundary.
Companion commands:
/arch-threat— threat-model the edge./arch-deps— gateway image dependency audit.
Table of Contents
- Principles
- Tier Responsibilities
- Gateway Selection Matrix
- KrakenD — Canonical Reference
- Configuration Split
- Flexible Configuration Template
- Backend Hosts + Env Overrides
- Authentication at the Edge
- Claims-to-Headers Forwarding
- Skip-Paths Pattern
- Rate Limiting at the Edge
- CORS
- W3C Trace Context Propagation
- Custom Plugins (Go)
- TLS / mTLS
- Aggregation Patterns
- Caching at the Edge
- Service Discovery
- Observability
- Deployment
- Anti-Patterns
- Pre-Merge Checklist
1. Principles
- The gateway is dumb glue. Business logic NEVER lives there. The gateway routes, authenticates, rate-limits, traces, and aggregates — nothing else.
- Defense in depth. Gateway enforcement does NOT replace per-service Spring Security. Downstream services re-validate JWT (signature, exp, aud) — gateway crashes or misconfig should not be a security breach.
- Stateless. No gateway session. Every request authenticates independently via JWT.
- Declarative > imperative. Prefer declarative config (JSON / template) over plugin code. Plugins are an escape hatch for behavior that the declarative layer cannot express.
- Configuration is data. Split per concern, commit to repo, version with the gateway. Never edit configuration in a running gateway dashboard.
- Trace context is non-negotiable. The gateway MUST propagate W3C
traceparentto downstream services or generate one if absent. See §13. - mTLS between gateway and services in production. Or at minimum, services trust forwarded identity only when on a private network with mTLS at the cluster edge.
2. Tier Responsibilities
| Layer | Owns | Examples |
|---|---|---|
| CDN / WAF | DDoS absorption, geo filtering, bot mitigation, static asset caching | CloudFlare, Akamai, AWS WAF |
| API Gateway | TLS termination, AuthN edge enforcement, edge rate limit, CORS, trace context, request routing, response aggregation, schema validation, transformation | KrakenD, Kong, Spring Cloud Gateway, AWS API Gateway, Apigee |
| Service Mesh (optional) | mTLS service-to-service, retries, circuit breakers, traffic shifting | Istio, Linkerd, AWS App Mesh |
| Service (Spring Security) | JWT re-validation, fine-grained authorization, business rules | Each :bootstrap module |
Hard rule: the gateway authenticates; it does NOT authorize per-resource. Per-resource authorization (e.g., “can user X read invoice Y?”) is a domain concern that lives in the service.
3. Gateway Selection Matrix
| Need | Choose | Why |
|---|---|---|
| Self-hosted, declarative config, plugin via Go, low resource footprint | KrakenD | No DB, no JVM, 10ms p99 overhead. Declarative JSON. Default for JVM platforms. |
| Self-hosted, mature plugin ecosystem, DB-backed config dashboard | Kong | Postgres/Cassandra backed; rich plugin marketplace; OSS + enterprise tiers. |
| JVM-native, Spring ecosystem integration, code-first | Spring Cloud Gateway | Useful when the team is Spring-only; pays JVM startup cost. |
| AWS-native, managed, IAM integration | AWS API Gateway | Pay-per-request; deep AWS integration; vendor lock-in. |
| Enterprise, governance/portal/dev-portal needs | Apigee / Mulesoft | Heavyweight; pick only if compliance demands it. |
| Edge-only minimal | Cloud-managed (CloudFront + Lambda, Cloudflare Workers) | If business logic stays at the service layer and edge is only TLS + auth. |
Decision is an ADR. Document choice + rejected options.
4. KrakenD — Canonical Reference
KrakenD is Go-based, stateless, declarative. Configuration is a single JSON file (or a template that compiles to one). Custom plugins are Go .so files dropped in a folder. No runtime DB.
4.1 Repo layout
platform-gw/├── config/│ ├── krakend.tmpl # Flexible Configuration template (Go template syntax)│ └── settings/│ ├── service.json # name, port, timeouts│ ├── hosts.json # backend service hosts│ ├── cors.json # CORS policy│ ├── jwt.json # JWT + JWKS (Keycloak/Auth0/etc.)│ ├── ip_resolver.json # IP geolocation│ ├── trace_context.json # W3C Trace Context propagation│ ├── rate_limit.json # rate limiting│ ├── logging.json # structured logging│ ├── metrics.json # metrics + collection interval│ ├── tls.json # server TLS│ └── client_tls.json # outbound TLS (to backends)├── plugins/│ ├── jwt-headers/ # custom plugin: JWT → headers + skip-paths│ ├── ip-resolver/ # custom plugin: geolocation│ ├── trace-context/ # custom plugin: W3C trace propagation│ └── build/ # compiled .so files (gitignored)├── Dockerfile # multi-stage production image├── docker-compose.yml # local dev environment└── Makefile # build + dev commands4.2 Pinning
- KrakenD
2.13.x— pin minor inDockerfile. - Plugins compiled with the same Go toolchain version as the KrakenD image; mismatch breaks loading. Use a
plugins/Dockerfile.builderaligned to KrakenD’s documented Go version.
5. Configuration Split
Each concern lives in its own JSON under config/settings/. The template (§6) composes them.
| File | Owns |
|---|---|
service.json |
service name, port, timeouts, cache TTL |
hosts.json |
backend service URLs (env-overridable) |
cors.json |
allow_origins, allow_methods, exposed_headers |
jwt.json |
JWKS URL, issuer, claims-to-headers map, skip_paths |
rate_limit.json |
service + endpoint + per-client rates, strategy |
trace_context.json |
enable/disable W3C propagation |
logging.json |
level, format (JSON in prod) |
metrics.json |
collection interval, listen address |
tls.json |
server cert/key, min/max version, mTLS |
client_tls.json |
outbound TLS to backends (mTLS recommended) |
ip_resolver.json |
geolocation provider config |
accept_language.json |
default locale + parsing |
Split benefits: review per concern, env-promote per concern (e.g., different rate_limit.json per env), reusable across services in the platform.
5.1 Examples
service.json:
{ "name": "platform-gw", "port": 8080, "timeout": "30s", "cache_ttl": "300s"}hosts.json (defaults, overridable via env — see §7):
{ "auth": "http://platform-auth-service:8080", "payment": "http://platform-payment-service:8080", "notifications": "http://platform-notifications-service:8080"}rate_limit.json:
{ "service_max_rate": 500, "service_client_max_rate": 50, "endpoint_max_rate": 100, "endpoint_client_max_rate": 20, "strategy": "ip"}cors.json:
{ "allow_origins": ["https://app.example.com"], "allow_methods": ["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"], "allow_headers": ["Content-Type", "Authorization", "Idempotency-Key", "Traceparent", "Tracestate"], "expose_headers": ["Content-Length", "Content-Type", "Traceparent", "Tracestate", "Trace-Id"], "max_age": "12h"}CORS rule: include Traceparent, Tracestate, and Trace-Id in expose_headers so browser clients can correlate traces in DevTools.
6. Flexible Configuration Template
KrakenD’s Flexible Configuration compiles a Go template into the runtime JSON. Settings are exposed as .<filename> variables.
config/krakend.tmpl (excerpt):
{ "$schema": "https://www.krakend.io/schema/krakend.json", "version": 3, "name": "{{ .service.name }}", "port": {{ .service.port }}, "timeout": "{{ .service.timeout }}", "cache_ttl": "{{ .service.cache_ttl }}", {{- if not .tls.disabled }} "tls": { "keys": {{ marshal .tls.keys }}, "min_version": "{{ .tls.min_version }}" }, {{- end }} "plugin": { "pattern": ".so", "folder": "/opt/krakend/plugins/" }, "extra_config": { "plugin/http-server": { "name": [ {{- if .trace_context.enabled }}"krakend-trace-context",{{ end }} {{- if .jwt.enabled }}"krakend-jwt-headers"{{ end }} ], "krakend-jwt-headers": {{ marshal .jwt }} }, "telemetry/logging": {{ marshal .logging }}, "telemetry/metrics": {{ marshal .metrics }}, "router": { "@comment": "expose tracing-related response headers for browser clients", "expose_headers": ["Traceparent", "Tracestate", "Trace-Id"] } }, "endpoints": [ {{- range $i, $endpoint := .endpoints.list }}{{- if $i }},{{ end }} { "endpoint": "{{ $endpoint.path }}", "method": "{{ $endpoint.method }}", "output_encoding": "{{ default "no-op" $endpoint.output_encoding }}", "backend": [ { "host": [{{ printf "%q" (env $endpoint.host_env | default $endpoint.host_default) }}], "url_pattern": "{{ $endpoint.url_pattern }}", "encoding": "{{ default "no-op" $endpoint.backend_encoding }}" } ] } {{- end }} ]}Build at startup (KrakenD reads the compiled JSON):
krakend run -c krakend.json -d -t7. Backend Hosts + Env Overrides
Backends declared in hosts.json should be defaults that env vars override. Lets the same template ship to all environments.
Template pattern:
{{ env "AUTH_HOST" | default .hosts.auth }}Resolved at template compile time: env var if set, otherwise the hosts.json value.
| Env | Default → Override |
|---|---|
| Local Docker Compose | hosts.json defaults work as-is (compose network) |
| Local micro outside Docker | AUTH_HOST=http://host.docker.internal:9095 |
| Staging | AUTH_HOST=http://platform-auth-service.staging.svc:8080 |
| Production | AUTH_HOST=https://auth.internal.example.com |
Compose forwards env vars to the gateway container:
services: gateway: image: platform/gw:latest environment: AUTH_HOST: ${AUTH_HOST:-http://platform-auth-service:8080} PAYMENT_HOST: ${PAYMENT_HOST:-http://platform-payment-service:8080}8. Authentication at the Edge
The gateway validates JWT at the edge. Downstream services re-validate (defense in depth).
jwt.json:
{ "enabled": true, "jwks_url": "https://auth.example.com/realms/platform/protocol/openid-connect/certs", "issuer": "https://auth.example.com/realms/platform", "cache_ttl_minutes": 60, "claims_to_headers": [ { "claim": "preferred_username", "header": "x-username" }, { "claim": "realm_access.roles", "header": "x-user-roles" }, { "claim": "sub", "header": "x-user-id" } ], "skip_paths": [ "/public/*", "/<service>/api/v1/login", "/<service>/api/v1/accounts/register", "/<service>/api/v1/token/refresh", "/<service>/api/v1/forgot-password", "/<service>/v1/api-docs", "/<service>/swagger-ui", "/<service>/actuator/health", "/<service>/api/v1/webhooks/*" ], "add_ip_header": true, "ip_header_name": "x-ip"}Validation contract (whether built-in KrakenD or custom plugin):
- Verify JWT signature against JWKS public keys.
- Verify
issmatches configuredissuer. - Verify
aud,exp,nbfclaims. - Cache JWKS keys with TTL (60 min default; refresh on rotation event if available).
- On failure: respond
401withWWW-Authenticate: Bearer error="invalid_token".
JWKS cache rule: fail closed if cache empty AND fetch fails — never allow tokens with un-verifiable signatures.
9. Claims-to-Headers Forwarding
After validation, the gateway extracts JWT claims into forwarded request headers. Downstream services read these instead of parsing JWT themselves (Spring Security still re-validates the original token for defense in depth).
| Claim | Header | Purpose |
|---|---|---|
sub |
x-user-id |
Stable user identifier |
preferred_username |
x-username |
Display name (logging only — never auth decision) |
realm_access.roles |
x-user-roles |
RBAC roles (CSV) |
tenant_id |
x-tenant-id |
Multi-tenant routing |
Forwarding nested claims (e.g. realm_access.roles — dot path) requires custom plugin (KrakenD built-in only forwards top-level claims).
Downstream rule: services MUST trust x-* headers only when reached via mTLS or private network. On a public network, treat them as user-controlled and re-derive from JWT.
10. Skip-Paths Pattern
Endpoints that MUST be reachable without JWT:
| Pattern | Why |
|---|---|
/<service>/api/v1/login |
Cannot obtain JWT without authenticating |
/<service>/api/v1/accounts/register |
New user has no JWT yet |
/<service>/api/v1/token/refresh |
Refresh flow uses refresh token, not access token |
/<service>/api/v1/forgot-password |
Recovery flow |
/<service>/api/v1/webhooks/<provider> |
External signature, not JWT |
/<service>/v1/api-docs + /<service>/swagger-ui |
OpenAPI doc (consider env-gated) |
/<service>/actuator/health |
K8s liveness/readiness probes |
/public/* |
Public read-only resources |
Pattern rules:
- Exact match: full path.
- Prefix match: trailing
/*—/public/*matches/public/anything/here. - Order independent (the matcher checks all).
Webhook rule: skip JWT, then enforce HMAC signature verification inside the service. Never expose a webhook endpoint with no signature check.
Swagger rule: in prod, EITHER disable Swagger entirely OR skip-path it but require an internal-network IP. Public Swagger leaks API surface.
11. Rate Limiting at the Edge
The gateway implements layer 1 rate limiting (broad). Per-tenant fine-grained limiting still belongs in the service.
rate_limit.json semantics:
| Setting | Meaning |
|---|---|
service_max_rate |
Max req/s across the entire gateway |
service_client_max_rate |
Max req/s per identified client (default: IP) |
endpoint_max_rate |
Max req/s per endpoint |
endpoint_client_max_rate |
Max req/s per endpoint per client |
strategy |
ip / header / header_value — what identifies a client |
Defaults (tune per traffic):
{ "service_max_rate": 500, "service_client_max_rate": 50, "endpoint_max_rate": 100, "endpoint_client_max_rate": 20, "strategy": "ip"}Strategy choice:
ip— public APIs (simple but bypassable via proxy chains; combine with CDN/WAF in front).header:x-user-id— authenticated APIs (post-JWT validation; cost: rate-limit applied after auth, so attacker can still cost JWT validation cycles).header:x-api-key— partner integrations.
Response on exceed:
HTTP/1.1 429 Too Many RequestsRetry-After: 47RateLimit-Limit: 100RateLimit-Remaining: 012. CORS
Edge-only. Browser preflight handled here; services never see OPTIONS.
{ "allow_origins": ["https://app.example.com", "https://admin.example.com"], "allow_methods": ["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"], "allow_headers": ["Content-Type", "Authorization", "Idempotency-Key", "Traceparent", "Tracestate"], "expose_headers": ["Content-Length", "Content-Type", "Traceparent", "Tracestate", "Trace-Id"], "max_age": "12h"}Hard rules:
allow_origins: ["*"]forbidden whenAccess-Control-Allow-Credentials: true.- Per-env allowlist (different
cors.jsonfor dev/staging/prod). - Always expose
Traceparentso browser clients can correlate.
13. W3C Trace Context Propagation
Full reference: obs-tracing.md §W3C Trace Context.
The gateway MUST propagate or generate W3C Trace Context. Without it, distributed tracing is incomplete from the user’s first hop.
13.1 W3C traceparent format
traceparent: 00-<trace-id>-<parent-id>-<flags> ^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 32 hex chars (16 bytes) — trace-id, identifies the whole trace | 16 hex chars (8 bytes) — parent-id (span-id of the caller) | 02 hex chars (1 byte) — flags, e.g. "01" = sampled 02 hex chars version, currently always "00"Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
13.2 Gateway behavior (canonical)
For every inbound request:
- Parse
Traceparentheader. If valid → propagate as-is. - Else, parse legacy
Trace-Id(bare 32-hex). If valid → construct00-<traceId>-<newParentId>-01. - Else, generate fresh trace-id (16 random bytes hex) + parent-id (8 random bytes hex), flags
01. - Set both
TraceparentandX-Traceparentheaders on the outbound request to the backend (latter as a defensive copy for legacy consumers). - Log
traceId,method,path,source(propagated|from-trace-id|generated),client_ip.
13.3 Custom plugin (Go) — minimal
KrakenD ships a built-in trace-context flag, but a custom plugin gives logging + legacy Trace-Id fallback. Reference shape:
var traceparentRegex = regexp.MustCompile(`^([0-9a-f]{2})-([0-9a-f]{32})-([0-9a-f]{16})-([0-9a-f]{2})$`)var traceIDRegex = regexp.MustCompile(`^[0-9a-f]{32}$`)
func handle(w http.ResponseWriter, req *http.Request) { traceparent := req.Header.Get("Traceparent") source := "propagated"
if !traceparentRegex.MatchString(traceparent) { traceID := req.Header.Get("Trace-Id") if traceIDRegex.MatchString(traceID) { source = "from-trace-id" } else { traceID = randomHex(16) source = "generated" } parentID := randomHex(8) traceparent = fmt.Sprintf("00-%s-%s-01", traceID, parentID) req.Header.Set("Traceparent", traceparent) }
req.Header.Set("X-Traceparent", traceparent) logger.Info("trace resolved", "traceId", traceparent[3:35], "method", req.Method, "path", req.URL.Path, "source", source, "client_ip", req.RemoteAddr, ) next.ServeHTTP(w, req)}13.4 Tracestate
Vendor-specific span context — usually opaque to the gateway. Propagate as-is. Do NOT parse or generate. Order preserved.
13.5 Downstream contract
Backend services MUST:
- Read
Traceparentfrom inbound headers. - Continue the span (Micrometer Tracing auto-handles this on Spring Boot 3 when a bridge —
micrometer-tracing-bridge-otelormicrometer-tracing-bridge-brave— plus a reporter — OTLP exporter orzipkin-reporter-brave— is on the classpath — see obs-tracing.md). - Propagate to further outbound calls (HTTP + Kafka + RabbitMQ).
- Stamp
traceIdin MDC for log correlation.
14. Custom Plugins (Go)
KrakenD plugin types:
- HTTP server plugin — wraps incoming requests (before any backend call). Pattern:
name: "krakend-<name>"registered inplugin/http-server. - HTTP client plugin — wraps outgoing backend calls.
- Modifier plugin — request/response body transformation.
14.1 When to write a plugin
- Built-in KrakenD lacks a feature (nested-claim header forwarding, legacy trace-id fallback, IP geolocation).
- A behavior must be tested independently and shipped across services (write once, deploy as
.so).
14.2 When NOT to
- Logic that belongs in the service domain.
- Anything stateful (gateway is stateless — DB lookups break this).
- Heavy CPU work — adds tail-latency to every request.
14.3 Build constraints
- Go toolchain version MUST match the KrakenD image’s documented Go version. Mismatch → plugin fails to load with cryptic error.
- Plugins are loaded via
pattern: ".so"fromfolder: "/opt/krakend/plugins/". Build .so artifacts in aplugins/Dockerfile.builderthat copies the same Go version. - Test plugins in isolation with Go unit tests before integration.
14.4 Required plugin contract (minimum)
var HandlerRegisterer = registerer("krakend-<name>")
type registerer string
func (r registerer) RegisterHandlers(f func( name string, handler func(context.Context, map[string]interface{}, http.Handler) (http.Handler, error),)) { f(string(r), r.registerHandlers)}
func (r registerer) registerHandlers(_ context.Context, extra map[string]interface{}, h http.Handler) (http.Handler, error) { // parse extra config from "extra_config.plugin/http-server.<plugin name>" return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { // logic here h.ServeHTTP(w, req) }), nil}
func main() {} // required, never called15. TLS / mTLS
15.1 Server-side TLS (gateway-facing)
{ "disabled": false, "keys": [ { "private_key_path": "/certs/tls.key", "public_key_path": "/certs/tls.crt" } ], "min_version": "TLS12", "max_version": "TLS13"}Hard rules:
- TLS 1.2 minimum; TLS 1.3 preferred.
- HSTS header on all HTTPS responses:
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload. - Disable RC4, 3DES, CBC-with-SHA1 cipher suites.
- Automate cert rotation via cert-manager (ACME) — manual rotation is a future incident.
15.2 Outbound TLS to backends (gateway → service)
client_tls.json:
{ "allow_insecure_connections": false, "ca_certs": ["/certs/ca.crt"], "client_certificate": "/certs/gw-client.crt", "client_private_key": "/certs/gw-client.key"}Production rule: mTLS to backends. Backend services authenticate the gateway via client cert.
15.3 Don’t
"allow_insecure_connections": truein production. Audit and ban.
16. Aggregation Patterns
16.1 Single backend (the default)
{ "endpoint": "/payments/{id}", "method": "GET", "backend": [ { "host": ["{{env "PAYMENT_HOST"}}"], "url_pattern": "/api/v1/payments/{id}" } ]}16.2 Aggregation (thin client, thick gateway)
Combine multiple backends into one response. Useful when the client (mobile, SPA) would otherwise make N parallel calls.
{ "endpoint": "/dashboard", "method": "GET", "backend": [ { "host": ["{{env "USER_HOST"}}"], "url_pattern": "/api/v1/users/me", "group": "user" }, { "host": ["{{env "INVOICE_HOST"}}"], "url_pattern": "/api/v1/invoices?limit=5", "group": "invoices" }, { "host": ["{{env "BALANCE_HOST"}}"], "url_pattern": "/api/v1/balance", "group": "balance" } ]}Trade-offs:
-
- One request from client.
-
- Parallel backend fan-out reduces client-perceived latency.
- − Gateway becomes coupled to backend response shapes.
- − Schema changes downstream require gateway template updates.
Rule: use aggregation for read endpoints serving a specific UI screen. NEVER aggregate for write operations (transactions are a domain concern, never a gateway concern).
16.3 Sequential calls
Use sequential: true when one backend’s response feeds another’s request. Use sparingly — couples the gateway to ordering logic.
{ "endpoint": "/order-summary/{id}", "method": "GET", "extra_config": { "proxy": { "sequential": true } }, "backend": [ { "host": ["{{env "ORDER_HOST"}}"], "url_pattern": "/api/v1/orders/{id}" }, { "host": ["{{env "INVOICE_HOST"}}"], "url_pattern": "/api/v1/invoices/{resp0_invoiceId}" } ]}17. Caching at the Edge
Gateway caches GET responses for high-traffic catalog endpoints. NEVER for authenticated user-specific data.
{ "endpoint": "/public/catalog/{id}", "method": "GET", "extra_config": { "qos/http-cache": { "max-age": 300, "shared": true } }}Rules:
- Authenticated endpoints:
no-store(no edge cache). - Public catalog:
max-age 60-300typically. - Invalidation: avoid manual invalidation; pick a TTL the data tolerates being stale.
18. Service Discovery
Three modes:
| Mode | When |
|---|---|
DNS-only (e.g., K8s Service DNS) |
Default. Gateway hits http://payment.svc.cluster.local:8080. K8s Service is the discovery layer. |
| Env-var hosts.json | Compose / non-K8s. Hosts injected at deploy time. |
| Consul / etcd / Eureka | When dynamic registration needed (Spring Cloud platforms). KrakenD supports via Subscribers. |
Recommendation: keep discovery in the platform layer (K8s, Consul), not the gateway. Gateway just hits stable DNS names or env-supplied URLs.
19. Observability
19.1 Metrics
metrics.json:
{ "collection_time": "30s", "listen_address": ":8090"}Prometheus scrape:
scrape_configs: - job_name: "platform-gw" metrics_path: /__stats/ static_configs: [{ targets: ["gateway:8090"] }]Key metrics to alert on:
| Metric | Alert |
|---|---|
krakend_http_responses_total{status_code="5xx"} rate |
> 1% over 5min → page |
krakend_http_request_duration_seconds p99 |
> 500ms over 5min → page |
krakend_router_response_size p99 |
> 1MB → investigate |
| Plugin error counters | > 0 → investigate |
19.2 Logs
JSON in production. Structured fields: traceId, method, path, status, latency_ms, client_ip. Never log full Authorization headers or request bodies.
{ "level": "INFO", "logging": { "format": "json", "level": "INFO" } }19.3 Traces
Gateway propagates W3C trace context (§13). For gateway’s own spans (TLS termination, JWT validation, backend call), enable KrakenD’s OpenTelemetry integration:
{ "telemetry/opentelemetry": { "service_name": "platform-gw", "exporters": { "otlp": [ { "name": "otel-collector", "host": "otel-collector", "port": 4317, "use_http": false } ] } }}20. Deployment
20.1 Dockerfile (multi-stage)
# Stage 1: build pluginsFROM golang:1.22 AS plugin-builderWORKDIR /pluginsCOPY plugins/ .RUN find . -name "main.go" -execdir go build -buildmode=plugin -o ../build/{}.so . \;
# Stage 2: runtimeFROM krakend:2.13.1COPY --from=plugin-builder /plugins/build/*.so /opt/krakend/plugins/COPY config/ /etc/krakend/USER krakendEXPOSE 8080 809020.2 K8s
- Deploy as
Deploymentwith HPA (CPU + custom metricreq/s). PodSecurityContext: non-root, read-only root FS, drop all capabilities.NetworkPolicy: ingress only from CDN/WAF; egress to backend services + JWKS endpoint.- Liveness probe:
GET /__health. - Readiness probe:
GET /__healthafter JWKS fetch succeeded.
20.3 Resource sizing
Baseline (tune per traffic):
requests: cpu: 200m, memory: 128Milimits: cpu: 1, memory: 256Mi- 2 replicas minimum (HA), HPA up to 10.
21. Anti-Patterns
| Anti-pattern | Why wrong | Use instead |
|---|---|---|
| Business logic in gateway plugins | Gateway becomes a monolith again | Move logic to a service |
| Stateful gateway (DB lookups, session) | Breaks horizontal scaling | Stateless; offload state to service |
| Authorizing per-resource at the gateway | Domain concern leak | Service-level @PreAuthorize + domain policy |
allow_origins: ["*"] with credentials |
Browser blocks; also a security risk | Allowlist origins per env |
allow_insecure_connections: true in prod |
MITM risk | mTLS to backends |
| JWT validated only at gateway | Defense in depth gone if gateway misconfig | Services re-validate (see security.md §2) |
| Manual gateway config edits in production | Drift; rollback impossible | Git + CI/CD pipeline |
| Plugins built with wrong Go version | Cryptic load failures | Pin Go version in Dockerfile.builder |
| Public Swagger / Actuator without IP allowlist | API surface leak | Disable in prod OR private network only |
No Traceparent propagation |
Distributed tracing breaks at edge | Built-in or custom plugin (§13) |
| Skip-paths includes write endpoints | Auth bypass | Only skip auth flows + webhooks (with HMAC) + health |
| Rate limit by IP behind CDN | All requests appear from CDN IPs | Use X-Forwarded-For strategy AND CDN-level rate limit |
| Aggregation for write endpoints | Distributed transaction nightmare | Per-service writes; orchestrate via saga |
| Caching authenticated responses | Cross-user data leak | no-store for authenticated; cache only public |
| Gateway → backend over plain HTTP in prod | mTLS missing | client_tls.json with mTLS |
| Manually editing JWT cache TTL high (24h) | Compromised key takes 24h to revoke | 60min default; bust on rotation event |
22. Pre-Merge Checklist
For PRs touching platform-gw/ or any gateway configuration:
- Change documented in PR description with rationale (decision = ADR if it affects auth model, topology, plugin set).
-
service.json/hosts.json/ per-concern files validated against the KrakenD JSON schema (krakend check -d -t -c krakend.json). - No business logic added to a plugin.
- Plugin builds against the documented Go version matching the KrakenD image.
-
jwt.jsonskip_paths reviewed — only auth flows + webhooks + health (no write endpoints). - CORS
allow_originsper env (no*with credentials). -
Traceparent,Tracestate,Trace-Idin CORSexpose_headers. - Trace context plugin (or built-in) enabled; legacy
Trace-Idfallback considered. - Rate-limit values tuned for the route (avoid one-size-fits-all
100 req/s). - Outbound mTLS configured for production (
client_tls.json). - HSTS + TLS 1.2+ enforced; TLS 1.0/1.1 disabled.
- Metrics endpoint scraped by Prometheus; alerts updated.
- Structured JSON logging in production; no PII in logs.
- Plugin .so files NOT committed (build artifacts in CI only).
- Docker image scanned (Trivy / Grype); zero CRITICAL CVEs.
- Smoke test executed against staging —
/__healthreturns 200, JWT validation works, trace context propagates end-to-end. - ADR drafted if changing gateway product (KrakenD → other), aggregation policy, or JWT validation contract.