TD5 — Infrastructure Recommendations
Decision Status Labels ✅ Decided — confirmed in a stakeholder session. Can be built against. 🔍 Under Analysis — options identified. Do not build against yet. 📅 Pending Session — requires a specific session or external input before a decision can be made.
1. What This Document Covers
Infrastructure decisions for this platform divide cleanly into two groups: what we already know must be true (requirements driven by architectural decisions made in TD1–TD4), and what cannot be confirmed until the ITS infrastructure session happens (cloud provider, orchestration, network topology, scaling model).
This document is heavier on requirements and constraints than on confirmed decisions. That is accurate — not a gap. The infrastructure session should be run with this document as the input brief, so the right questions get answered in the right order.
2. Confirmed Infrastructure Requirements
These are not technology choices. They are requirements that any chosen infrastructure must satisfy. They are derived from decisions already confirmed in TD1–TD4.
✅ Containerisation — Docker for all Core services
All Core engine services (FastAPI and Node.js) are containerised using Docker. This was confirmed in TD1. Containers give the Core team consistent environments across dev, staging, and production; make the internal multi-language architecture (FastAPI + Node.js) operationally uniform; and make the choice of orchestration platform independent of the runtime.
Module teams are not mandated to use Docker for their own services. The requirement applies only to Core's own services.
✅ Deployment independence — modules never block Core, Core never blocks modules
Infrastructure must support independent deployment pipelines for Core and for each business module. A Core deployment must not require any module to redeploy. A module deployment must not require coordination with Core or other modules. These are operational expressions of the Deployment Independence Policy.
This means: shared deployment pipelines between Core and modules are not acceptable. Core has its own CI/CD pipeline. Each module team owns their own CI/CD pipeline.
✅ Multi-environment — minimum three environments
At minimum: development, staging, and production. Each environment is fully isolated — different API keys, different bus credentials, different database instances. A mistake in staging must be physically impossible to affect production.
Core's dev environment is available to module teams for integration testing. Core's staging environment mirrors production configuration. Module teams test against Core staging before promoting to production.
✅ Audit Log durability — immutable, cannot be lost
The Audit Log Backbone is append-only and must never lose a record. Infrastructure for the audit log must provide durability guarantees beyond what a typical database offers — write-ahead logging, point-in-time recovery, and backup retention sufficient for compliance review. Events written to the audit log are considered permanent records.
✅ Event bus durability — replay must be possible
The event bus must support message replay. If a subscriber goes down during a peak operation window and misses events, it must be able to replay from a defined offset when it recovers. Infrastructure that does not support replay is not acceptable for this platform.
This is the single most important infrastructure requirement for the event bus. It is what separates a reliable event-driven platform from a fragile one.
✅ Peak load profile — predictable spikes, not constant high load
The platform's load profile is not uniform. During City Selection and Registration Open windows, Core's API Gateway and event bus will handle very high concurrent load. Outside those windows, load is low. Infrastructure must handle the peaks without requiring year-round provisioning at peak capacity.
This profile strongly favours auto-scaling approaches — infrastructure that scales up during known peak windows and scales back down between them. Fixed-capacity infrastructure sized for peak load is significantly wasteful given how infrequent the peaks are.
✅ ITS integration network boundary
Core must be able to reach ITS's systems for the data integration (C-005) and the identity validation (C-001). The network topology — whether Core and ITS share a VPC, use VPC peering, communicate over a private network, or over the public internet with mutual TLS — must be confirmed in the ITS infrastructure session. No Core infrastructure should be provisioned before this boundary is understood.
3. Core Service Infrastructure
✅ Core is one platform — one infrastructure footprint
All Core engine services (15 engines, FastAPI and Node.js) are owned and operated by the Core team under one infrastructure footprint. There is no scenario where one engine is operated by one team and another engine by a different team. One team, one platform, one set of infrastructure responsibilities.
🔍 Container orchestration
Docker containers are confirmed. What orchestrates them is under analysis. The choice is constrained by: the Core team's operational capacity (a small team should not be managing a large, complex Kubernetes cluster), the cloud provider decision (managed Kubernetes services simplify operations significantly), and the scale requirements of Core's services.
| Option | Description | Suited when |
|---|---|---|
| Managed Kubernetes (EKS, GKE, AKS) | Cloud-managed control plane, Core team manages workloads | Cloud provider is confirmed, Core team has Kubernetes experience |
| AWS ECS / Cloud Run / Azure Container Apps | Simpler managed container service, less operational overhead than Kubernetes | Core team prefers lower operational complexity |
| Self-hosted Kubernetes (K3s, kubeadm) | Full control, no cloud provider dependency | On-premise or private cloud requirement |
Decision pending the ITS infrastructure session.
🔍 API Gateway technology
Three options were documented in TD1 Section 4: managed gateway (AWS API Gateway, Kong Cloud, Azure APIM), self-hosted gateway (Kong, Traefik, Nginx, Envoy), or a Core-built lightweight Node.js gateway service. The choice depends on the cloud provider decision and the Core team's preference for operational simplicity vs control. Decision pending the ITS infrastructure session.
🔍 Internal service communication infrastructure
Within Core, FastAPI and Node.js services need to communicate with each other (e.g., Eligibility Engine calling Config Cascade). Whether this is HTTP over the internal network or a lower-latency protocol (gRPC) affects what internal infrastructure is needed. For HTTP: standard internal DNS and service discovery suffice. For gRPC: a service mesh or explicit service registry may be needed. Decision pending the Core team's internal design session.
🔍 Database infrastructure per engine
Each Core engine manages its own persistence with potentially different database technology suited to its access patterns. The databases requiring infrastructure decisions are:
| Engine | Access pattern | Candidate database type |
|---|---|---|
| RBAC Engine | High-frequency role hierarchy lookups | Fast key-value or graph |
| Audit Log Backbone | High write throughput, append-only, searchable | Time-series or event store |
| Config Cascade Engine | Hierarchical key-value reads, cached | Key-value with TTL |
| Eligibility Engine | Complex queries across ITS-synced data | Relational |
| Allocation Engine | State management, transactional | Relational |
| Miqaat Lifecycle | Workflow state, low volume | Relational |
| ITS Sync Engine | Bulk write on sync, read-heavy | Relational |
| HR Bank | Aggregation across itsIds and events | Relational or document |
Database technology selection per engine is pending the Core team's build planning session. The infrastructure session with ITS should confirm whether there is a preferred database platform or managed service constraint.
4. Event Bus Infrastructure
✅ Event bus capability requirements — confirmed
Whatever technology is chosen, the bus must satisfy all of the following. These are non-negotiable — derived from TD2:
| Requirement | Why required |
|---|---|
| Message durability — no events lost on broker restart | At-least-once delivery guarantee depends on this |
| Message replay — consumers can replay from a defined offset | Modules that go down during peak windows must recover missed events |
| Consumer groups — multiple independent subscribers per topic | Each subscriber maintains its own offset independently |
| Dead Letter Queue (DLQ) — failed events land in an inspectable queue | Platform Admin's Event Bus Health monitor depends on this |
| Topic-level consumer access control (subscribe scope) | Prevents modules subscribing to topics they are not entitled to |
| Topic-level publish access control (publish scope) | Required if Option B (direct bus publishing) is confirmed — see TD2 Section 3 |
| Schema validation at publish time (optional but preferred) | Reduces the need for a separate validation step if Option B is confirmed |
| Horizontal scaling — throughput scales with broker instances | Required to handle City Selection and Registration Open peaks |
🔍 Messaging technology
The specific technology is under analysis. The capability requirements above are the evaluation criteria. The decision will be confirmed before Core's event bus infrastructure is provisioned and documented here.
🔍 Bus hosting model — managed vs self-hosted
A managed bus service (cloud-provider-native or SaaS) reduces operational overhead significantly — no broker cluster to manage, no replication configuration, no disk capacity planning. A self-hosted broker gives more control over data residency and cost at scale, but requires the Core team to operate it.
Given the Core team's operational capacity constraint, a managed service is the working preference — but this depends on which cloud provider is confirmed and what managed options exist on that provider.
5. Module Infrastructure Principles
Core does not prescribe module infrastructure. Each module team owns its own hosting, database, CI/CD, and scaling. The platform does not require modules to use a specific cloud provider or deployment model.
However, three requirements apply to all modules from an infrastructure perspective:
✅ Module infrastructure must reach Core's API Gateway
Wherever a module is deployed, it must be able to make HTTPS calls to Core's API Gateway. This is the only network dependency between Core and any module. Whether that is over a private network, VPC peering, or the public internet (with TLS) is determined by the network topology confirmed in the ITS infrastructure session.
✅ Module infrastructure must reach the event bus
For event publishing and subscribing, modules need network access to the event bus endpoint. The connection details (endpoint URL, port, TLS requirements) will be provided by Core when a module is onboarded and its bus credentials are provisioned.
✅ Module CI/CD must be independent
A module's deployment pipeline must not depend on Core's deployment pipeline and must not depend on other modules' pipelines. Shared deployment gates between modules are prohibited. Each module promotes through its own environments on its own schedule.
6. Observability and Monitoring
✅ Core team monitors Core — Platform Admin is the operational console
Platform Admin (the Core operator console, TD1 Section 2) is the primary operational view for the Core team. It surfaces miqaat lifecycle state, RBAC assignment, Config Cascade values, ITS sync health, allocation state, event bus consumer lag, DLQ contents, and the audit log. All Core engine health is visible in one place.
🔍 Underlying observability tooling
The specific tooling for logs, metrics, and traces (e.g., CloudWatch, Datadog, Prometheus + Grafana, ELK stack) is under analysis and tied to the cloud provider decision. Requirements:
- All Core services (FastAPI and Node.js) emit structured JSON logs in the shared format established in TD1
- Metrics include: API Gateway request rates and latency per endpoint, event bus consumer lag per subscriber, database query latency per engine, sync job success/failure rates
- Distributed tracing via
correlationId— the same ID flowing from a module's API call through Core's engines and into any resulting events, enabling end-to-end trace reconstruction
🔍 Alerting
Alerting thresholds and on-call model are pending the ITS infrastructure session. Minimum alert conditions that must be covered:
- ITS sync falling behind by more than a defined threshold
- Core API Gateway error rate exceeding threshold
- Event bus consumer lag exceeding threshold per subscriber
- DLQ depth increasing (events failing repeatedly)
- Any Core engine returning 5xx errors
7. Constraints for the ITS Infrastructure Session
This is the input brief for the ITS infrastructure session. These questions must be answered before Core infrastructure can be provisioned.
| Question | Why it matters |
|---|---|
| What cloud provider does ITS currently run on? | Determines whether Core shares the provider (simplifies networking) or runs on a different provider (requires explicit peering/connectivity planning) |
| Is there an existing VPC or private network Core must connect to? | Determines network topology for C-001 (token validation) and C-005 (data integration) |
| Is there a preferred cloud provider or organisational constraint? | May narrow the infrastructure options significantly |
| What are ITS's maintenance window patterns? | Core must handle ITS downtime gracefully — informs sync resilience requirements |
| Does ITS have existing managed database services Core can reuse for its sync copy? | May simplify ITS Sync Engine infrastructure |
| Is there an existing CI/CD toolchain (GitHub Actions, Jenkins, GitLab CI)? | Informs Core's own pipeline approach for consistency |
| Are there data residency requirements (data must stay in a specific country or region)? | Could constrain cloud provider and region selection |
| What is the expected participant geography for the next miqaat? | Informs whether multi-region deployment is needed for latency |
Decision Summary
| Decision | Status | Notes |
|---|---|---|
| All Core services containerised with Docker | ✅ Decided | TD1 — applies to Core services only, not modules |
| Minimum three environments (dev / staging / prod) | ✅ Decided | Fully isolated per environment |
| Core has its own CI/CD pipeline, independent of modules | ✅ Decided | Deployment Independence Policy |
| Each module has its own CI/CD pipeline | ✅ Decided | No shared deployment gates |
| Audit Log must be durable — no record can be lost | ✅ Decided | Append-only, point-in-time recovery required |
| Event bus must support replay from a defined offset | ✅ Decided | Critical for recovery from outages during peak windows |
| Event bus must support consumer groups | ✅ Decided | Each subscriber maintains its own offset |
| Event bus must support DLQ | ✅ Decided | Platform Admin Event Bus Health depends on it |
| Event bus must support topic-level consumer access control | ✅ Decided | Subscribe-scope enforcement |
| Event bus must support horizontal scaling | ✅ Decided | Required for City Selection and Registration Open peaks |
| Auto-scaling preferred over fixed-capacity provisioning | ✅ Decided | Load profile is peak-heavy, not constant |
| All Core services emit structured JSON logs in shared format | ✅ Decided | TD1 shared logging standard |
| Distributed tracing via correlationId across all services | ✅ Decided | End-to-end trace from module call through Core to events |
| Modules must reach Core API Gateway over HTTPS | ✅ Decided | Only network dependency between Core and modules |
| Container orchestration technology | 🔍 Under Analysis | Pending ITS infrastructure session and cloud provider decision |
| API Gateway technology | 🔍 Under Analysis | Three options in TD1 Section 4 — pending infrastructure session |
| Database technology per Core engine | 🔍 Under Analysis | Per-engine decision before each engine build begins |
| Messaging / event bus technology | 🔍 Under Analysis | TD2 Section 6 — capability requirements confirmed |
| Bus hosting model (managed vs self-hosted) | 🔍 Under Analysis | Managed preferred given Core team capacity constraint |
| Internal service communication (HTTP vs gRPC) | 🔍 Under Analysis | Core team internal design decision |
| Observability tooling (logs, metrics, traces) | 🔍 Under Analysis | Tied to cloud provider decision |
| Alerting thresholds and on-call model | 🔍 Under Analysis | Pending infrastructure session |
| Cloud provider | 📅 Pending Session | ITS infrastructure session — first decision to make |
| Network topology (VPC, peering, public/private endpoints) | 📅 Pending Session | ITS infrastructure session |
| Data residency requirements | 📅 Pending Session | ITS infrastructure session |
| Multi-region requirement | 📅 Pending Session | Depends on participant geography for next miqaat |