TD2 — Event Driven Structure and Communication Patterns
Decision Status Labels ✅ Decided — confirmed in a stakeholder session. Can be built against. 🔍 Under Analysis — options identified. Do not build against yet. 📅 Pending Session — requires a specific session or external input before a decision can be made.
1. The Problem We Are Starting From
No team has event-driven architecture today. That is the problem we are solving.
Every integration across the 32 modules today is manual — Excel exports, WhatsApp forwards, copy-paste between systems. When AMS needs data from VMS, someone exports a file. When an allocation changes, someone sends a message. When a phase transition happens in one system, someone manually updates three others.
Automating that with webhooks and point-to-point polling only makes the dirty sync faster — it does not make it clean. It recreates the dependency graph that already exists. One module goes down, others fail. There is no replay if a call is missed. There are no ordering guarantees. Every new integration requires both teams to coordinate and both codebases to change.
The goal is not to add event-driven architecture on top of what exists. The goal is to replace the manual coordination layer with a structured, decoupled communication model — built into how the platform works, not bolted on afterwards.
✅ What we are explicitly not doing
Webhook-based module-to-module integration Teams expose webhooks, other teams call them. This recreates the point-to-point dependency graph. One team goes down, the caller fails. No replay, no ordering, no audit. Prohibited under the Communication Policy.
Database-level sync or shared databases Direct reads across module database boundaries destroy deployment independence. A schema change in one module breaks every other module reading that schema. Prohibited under the Data Ownership Policy.
2. The Two Communication Patterns
✅ Every interaction between systems uses one of two patterns
There is no third pattern. Every time a module needs something from another part of the platform, it either asks Core for a decision (Pattern A) or tells the platform something happened (Pattern B). The choice between them depends on whether the module needs an answer before it can proceed.
✅ Pattern A — Synchronous Decision Call
Used when a module needs an answer before it can proceed.
Module → HTTP call → Core Engine API
Core → computes answer from its own state or ITS sync data
Core → returns: result + reason
Examples:
→ Is this itsId eligible for this miqaat? (Eligibility Engine)
→ Is this itsId allowed to approve a volunteer? (RBAC Engine)
→ What is the capacity config for this city? (Config Cascade Engine)
→ Does this indent fit within the budget? (Procurement Registry)
The module waits for the response. Core computes synchronously and returns. This is a standard HTTP call — every module stack in use today (.NET 4.5, FastAPI, Node.js) can make HTTP calls. No new infrastructure is required for a module to adopt Pattern A.
What Core never does in Pattern A: Core does not call back into the module, does not read the module's database, and does not store the module's data to answer the question. Core answers from its own state (ITS sync, role assignments, config cascade values, miqaat lifecycle state).
✅ Pattern B — Asynchronous Event Publication
Used when a module has completed an action that other parts of the platform may care about.
Module → publishes event → Event Bus
Core Engines → subscribe to events they need
Other Modules → subscribe to events they need
Publisher does not know or care who is listening
Examples of events modules publish:
→ registration.confirmed (AMS publishes after registration is recorded)
→ payment.received (Payment module publishes after payment clears)
→ volunteer.assigned (VMS publishes after volunteer role is assigned)
→ pass.printed (Scanning publishes after a miqaat pass is printed)
→ khidmat.assigned (VMS publishes — Core HR Bank subscribes)
→ capacity.configured (Accommodation module publishes — Core Capacity Balance subscribes)
The publisher does not know who is listening and does not wait for a response. Subscribers react independently. Core engines subscribe only to the events they need for their own decision state. Modules subscribe to events from other modules only via the bus — never via direct calls.
What does NOT go on the event bus: synchronous decision requests. Pattern A and Pattern B are not interchangeable. If a module needs an answer before proceeding, that is always Pattern A.
3. Module Authentication to Core
Every call a module makes to Core — whether a synchronous decision call (Pattern A) or publishing an event to the bus (Pattern B) — requires the module to identify itself. There are two distinct identity layers in every request: who the module is, and who the user is. These are not the same thing and must not be conflated.
✅ Two identity layers on every Core request
Layer 1 — User identity When a user action triggers a Core call (a coordinator approves a volunteer, a user submits a registration), the module includes the user's ITS token in the request. The Identity Bridge at the API Gateway validates it, extracts the itsId, and passes it downstream as a clean header. No Core engine sees the raw token. This answers: who is the human acting?
Layer 2 — Module identity Separate from the user, Core needs to know which module is making the call. This matters for audit logging (which module made this request?), rate limiting (per-module throttle during peak load), and access control (a module should only access the Core data it is entitled to). This answers: which system is calling?
✅ API keys per module — module identity mechanism
Core issues each module a unique API key via Platform Admin. The module includes it as a header on every call to Core:
X-Core-Api-Key: <module-specific-key>
At the API Gateway, the Identity Bridge validates both the API key (module identity) and the ITS token if present (user identity), then strips both and passes only clean, verified headers downstream to the Core engine:
X-Module-Id: vms
X-Its-Id: ITS-1234567 ← only present on user-initiated calls
No Core engine downstream ever sees a raw token or API key. Every engine receives only a verified module identity and, where applicable, a verified itsId.
Why API keys and not a more sophisticated approach: Every stack in active use across the 32 modules — .NET 4.5, FastAPI, Node.js — can add an HTTP header with zero additional libraries. Approaches like mTLS or OAuth client credentials are more secure in theory but introduce complexity that legacy stacks cannot easily absorb. API keys with rotation support and revocation from Platform Admin is the right balance for this platform.
API key lifecycle — managed from Platform Admin:
- Core issues one key per module per environment (dev / staging / prod)
- Key rotation: Core supports a dual-key grace period so modules can rotate without downtime
- Revocation: immediate, from Platform Admin. A compromised key is revoked in one action — no module restart required
- Key scope: a module's key grants access only to the Core API surface that module is entitled to use — not a master key to all of Core
✅ Two call types — user-initiated and system-initiated
Not every call to Core is triggered by a human action. Core must handle both:
User-initiated call A coordinator in VMS clicks "Approve volunteer" → VMS calls Core RBAC. The request carries both the module API key and the user's itsId. Core knows: VMS made this call, on behalf of user ITS-1234567. RBAC evaluation, audit log, and any downstream decisions carry both identities.
Headers:
X-Core-Api-Key: <vms-key>
Authorization: Bearer <ITS-token>
Gateway passes downstream:
X-Module-Id: vms
X-Its-Id: ITS-1234567
System-initiated call A VMS background job fetches Config Cascade values at startup — no user triggered this. The request carries only the module API key. Core handles this as a system call with no user context.
Headers:
X-Core-Api-Key: <vms-key>
Gateway passes downstream:
X-Module-Id: vms
(no X-Its-Id)
Core engines that require a user itsId (RBAC checks, audit-attributed decisions) will reject system-initiated calls. Engines that serve configuration or reference data (Config Cascade, Vendor Registry) accept system-initiated calls. This distinction is defined per endpoint in the API contract (TD3).
🔍 Event bus authentication — mechanism under analysis
When a module publishes an event to the bus or subscribes to event topics, the bus also needs to know which module is acting. The principle is confirmed: Core-issued credentials per module, enforced at the bus level. A module can only publish to its own registered event types — it cannot publish under another module's event types, and it cannot subscribe to topics it has not been granted access to.
The exact credential mechanism for bus access (whether the same API key, a separate bus-specific credential, or a derivative) depends on the messaging technology decision (Section 6). The access control model — per-module publish scope, per-module subscribe scope, both managed from Platform Admin — is confirmed regardless of the technology choice.
✅ Core API endpoint authorization — read/write split
Once the API Gateway has validated the API key and resolved X-Module-Id, the next question is: is this module allowed to call this specific endpoint?
The enforcement model is a read/write split:
Read endpoints — open to any module with a valid API key. Any module can call any Core read endpoint without further restriction. This is safe because the response is naturally scoped to the request — a module asks about a specific itsId or miqaatId and receives only that data. There is no way to retrieve another module's operational data through a Core read endpoint.
Write endpoints — restricted to Platform Admin and designated Core processes only. No business module ever calls an endpoint that mutates Core state directly. Miqaat lifecycle transitions, role assignments, config cascade overrides, allocation engine triggers — all of these are operator or Core-internal actions only. The API Gateway rejects any write endpoint call from a module API key at the gateway layer before the request reaches the engine.
This model is simple to implement and simple to reason about. It avoids the complexity of per-module endpoint ACLs while ensuring no business module can corrupt Core's authoritative state.
✅ Event publish authorization — requirements confirmed, enforcement mechanism under analysis
Three authorization requirements on every event publish are confirmed regardless of how the mechanism is implemented:
- A module cannot publish under another module's identity —
sourceModulemust match the calling module - A module can only publish event types it has registered — VMS cannot publish
registration.confirmed, which belongs to AMS - Every event must conform to the registered schema for its
eventTypeandschemaVersion
How these are enforced is under analysis. Two realistic approaches:
Option A — Core gateway mediates all publishing
Modules POST events to a Core gateway endpoint (POST /v1/events/publish). The gateway runs all three checks before forwarding to the bus. Technology-agnostic — the bus does not need to support topic-level ACLs. Schema validation, identity check, and type authorization all happen in Core code.
The concern with this approach: during peak windows (City Selection, Registration Open), thousands of events may be generated concurrently. This endpoint would absorb all of that traffic on top of the synchronous decision calls (RBAC, eligibility, config) already hitting Core's gateway. Two different traffic types competing on the same gateway — the gateway becomes a potential bottleneck for what the bus is designed to handle natively.
Option B — Direct bus publishing with bus-level enforcement Modules publish directly to the bus using Core-issued bus credentials. The bus enforces topic-level ACLs — a module's credentials only grant publish access to its registered event type topics. Schema validation either happens at the bus (if the chosen technology supports it), via a lightweight pre-publish call to the Domain Event Schema Registry, or asynchronously after publish via a Core validation consumer.
This is how event-driven systems typically work in practice. The bus is built for high-throughput concurrent publishing. Core's gateway handles only synchronous decision calls, which are lower volume and more latency-sensitive. The Audit Log Backbone still captures all events by subscribing to every topic on the bus — Core does not need to be in the publish path to maintain the audit record.
The constraint: this approach relies on the bus technology supporting fine-grained per-module topic ACLs for publishing — which is a confirmed selection requirement for Section 6. It also means the enforcement is distributed (bus + schema registry) rather than centralised in Core's gateway.
The decision between these two options is tied to the messaging technology selection (Section 6). Until that decision is made, no module should build a hard dependency on either publish path.
✅ Event subscription scoping — declared at onboarding, provisioned by Core
A module declares its event subscriptions in its System Definition Document before build begins. Core provisions bus-level topic access based on those declared subscriptions when the module is onboarded. The module's bus credentials only grant access to the topics it declared — it cannot subscribe to event types it did not register for.
This prevents data leakage: a module cannot listen to another module's operational events just by knowing the topic name. If a module's requirements change and it needs to subscribe to a new event type, it updates its System Definition Document and requests a subscription update from the Core team. Core provisions the additional access in Platform Admin.
Unlike publish authorization (which is enforced at the Core gateway), subscription scoping relies on the bus technology supporting topic-level consumer access control. This is a confirmed capability requirement for the messaging technology decision in Section 6.
4. Event Envelope Schema
✅ Core owns the event envelope schema
Every event published to the bus — by any module, by any Core engine — must conform to Core's event envelope schema. The schema is language-agnostic (JSON Schema specification). Both FastAPI and Node.js Core services produce and validate events against the same spec. Modules receive and validate the same envelope.
✅ Standard event envelope
{
"eventId": "uuid-v4 — unique per event instance",
"eventType": "domain.action — e.g. registration.confirmed",
"schemaVersion": "1.0 — version of this event type's payload schema",
"timestamp": "ISO 8601 UTC — when the event occurred",
"idempotencyKey": "string — used by consumers to detect duplicate delivery",
"sourceModule": "string — which module or Core engine published this",
"miqaatId": "string — which miqaat this event belongs to (if applicable)",
"itsId": "string — itsId of the actor who caused this event (if applicable)",
"correlationId": "string — tracing ID linking related events across systems",
"payload": { }
}
eventId — globally unique. Assigned by the publisher. Used by Core's audit log to deduplicate across replays.
idempotencyKey — required by the Idempotency Policy. Consumers must use this to detect and safely ignore duplicate delivery. At-least-once delivery is the platform guarantee; consumers must not assume exactly-once.
schemaVersion — the version of this specific event type's payload schema. A breaking change to a payload schema (removing a field, changing a field type) requires a new schema version. Old consumers continue receiving the old version until they migrate.
correlationId — passed through from the originating request. Allows tracing a chain of events back to the original trigger — useful for debugging allocation failures, eligibility disputes, and audit investigations.
✅ Schema versioning — breaking payload change = new schema version
The Domain Event Schema Registry (Core, Node.js) owns the versioned schema for every event type. When a module wants to change an event's payload in a breaking way, it must register a new schema version and publish under the new schemaVersion. Existing subscribers continue receiving the old version. The module team is responsible for migrating subscribers before deprecating the old version.
Non-breaking changes (adding an optional field) do not require a new schema version. Subscribers that do not use the new field are unaffected.
5. What Core Publishes vs What Modules Publish
✅ Modules publish state change events
Business modules are the primary publishers. Every significant state change in a module becomes an event. The module owns the event payload definition (registered with Core's Domain Event Schema Registry) and is the sole publisher of its own events.
| Module | Example events published |
|---|---|
| AMS (Attendance) | registration.confirmed, attendance.recorded |
| VMS (Volunteers) | volunteer.assigned, volunteer.approved, khidmat.assigned |
| Accommodation | capacity.configured, accommodation.allocated |
| Scanning / Kiosk | pass.printed, checkpoint.scanned |
| Payment module | payment.received, payment.failed |
| Any module | [domain].[action] — named by the module, registered with Core |
✅ Core engines publish decision events
Core engines also publish events when they complete a decision that other parts of the platform need to react to. These are not module-to-module events — they are Core-to-platform broadcasts.
| Core Engine | Example events published |
|---|---|
| Capacity Balance Engine | capacity.imbalance.detected |
| Allocation Engine | allocation.confirmed, allocation.waitlisted |
| Miqaat Lifecycle | miqaat.phase.changed, miqaat.created, miqaat.archived |
| RBAC Engine | role.assigned, role.revoked |
| Eligibility Engine | eligibility.status.changed |
✅ Core engines subscribe to module events
Core engines subscribe only to the events they need to maintain their decision state. They do not subscribe to everything — only what their engine logic requires.
| Core Engine | Subscribes to |
|---|---|
| Capacity Balance Engine | capacity.configured |
| HR Bank | khidmat.assigned |
| Eligibility Evaluation Engine | attendance.recorded, payment.received |
| Audit Log Backbone | All events (full stream — immutable record) |
6. Messaging Technology
🔍 Event bus technology — under analysis
The specific technology powering the event bus is under analysis. The communication pattern (Pattern B) and the event envelope schema are confirmed regardless of which technology is chosen. The technology decision is about operational characteristics: durability, replay capability, consumer group support, managed vs self-hosted, and compatibility with the Core team's operational capability.
Considerations that will drive the decision:
Durability and replay — events must be replayable. If a module's consumer goes down during a peak operation period (City Selection, Registration Open), it must be able to replay missed events from a defined offset when it comes back up. A messaging system that drops undelivered events is not acceptable.
At-least-once delivery — the platform assumes at-least-once delivery (hence the Idempotency Policy). Exactly-once is not a hard requirement — it shifts complexity to the infrastructure and constrains technology choice.
Consumer groups — multiple independent modules may subscribe to the same event type. The messaging system must support consumer groups so each subscriber receives its own copy of the event and progresses its own offset independently.
Dead Letter Queue (DLQ) — events that repeatedly fail processing must land in a DLQ visible in Platform Admin's Event Bus Health monitor. The Core team needs to be able to inspect, retry, or discard failed events from a single operational view.
Managed vs self-hosted — the Core team's operational capacity is a constraint. A fully managed service reduces operational overhead but introduces a cloud vendor dependency. A self-hosted solution gives more control but requires the Core team to manage availability and scaling. This is tied to the infrastructure decision in TD5.
The technology decision will be confirmed and documented here before Core's event bus infrastructure is provisioned.
7. Tiered Adoption Path — Bringing Existing Modules On
✅ Not all modules adopt at the same time — tiered by current stack
The event bus is the long-term goal. Not every module gets there at the same time. The adoption path is tiered by the module's current technical state. The pattern and policy are decided; the specific module-by-module sequencing is confirmed as part of the existing module audit process (see BusinessModulesAndPolicies.md, Part 5).
Tier A — Greenfield modules Built on Core from day one. Publish events natively using the Core event client. Subscribe to events natively. No migration debt. These modules prove the pattern first and reduce adoption risk for others. RMS, Nikah, Rasme Saifee are in this tier.
Tier B — Modern REST modules (FastAPI, Node.js) Already have well-structured codebases. Adoption path: add the event client library, progressively replace internal decision logic with Core API calls (Pattern A), add event publication on state changes (Pattern B). Can run old and new logic in parallel during transition.
Tier C — Legacy modules (.NET C# older versions) Cannot easily adopt an event client library directly. Adoption path in two steps: first, add HTTP calls to Core Decision APIs (Pattern A only — just HTTP, any stack can do it); second, add a database outbox table that a Core-owned relay agent reads and publishes as events on the bus. The application code change for step two is minimal — just write to the outbox table inside the existing transaction.
Tier D — Stored-procedure-heavy modules The stored procedure writes to an outbox table inside the same database transaction. A separate Core-owned relay service reads the outbox and publishes events to the bus. Application code change is minimal; the relay agent handles the event publishing. This path allows even the most constrained legacy systems to participate in the event-driven platform without a full rewrite.
📅 Module-by-module adoption sequence
Which specific modules move through which tier, and in what order, is confirmed as part of the technical discovery sessions with each module team. The sessions with the ITS team for existing module integration are still pending. Specific adoption timelines will be set once those sessions are complete.
8. Event Bus Health and Operations
✅ Platform Admin monitors all event bus health
Platform Admin (the Core operator console) includes an Event Bus Health control surface. This is the single operational view for the Core team across all event consumers. It surfaces:
- Consumer lag per subscriber module — how far behind each module is from the latest event on its subscribed topics
- Failed event counts — events that exceeded retry limits across all consumers
- Dead Letter Queue (DLQ) contents — failed events available for inspection, retry, or discard
- Schema version mismatch alerts — where a consumer is receiving events in a schema version it has not declared support for
The Core team operates this. Module teams are responsible for their own consumer health (staying within acceptable lag thresholds) and for handling their own DLQ events where applicable.
Decision Summary
| Decision | Status | Notes |
|---|---|---|
| No webhook-based module-to-module integration | ✅ Decided | Prohibited — Communication Policy |
| No shared database or cross-module DB reads | ✅ Decided | Prohibited — Data Ownership Policy |
| Pattern A — synchronous HTTP call to Core API for decisions | ✅ Decided | Used when module needs answer before proceeding |
| Pattern B — asynchronous event publication for state changes | ✅ Decided | Used after module completes an action |
| No direct module-to-module calls of any kind | ✅ Decided | All communication via Core or event bus |
| Two identity layers on every Core request (module + user) | ✅ Decided | Separate concerns — see Section 3 |
| API keys per module — module identity mechanism | ✅ Decided | Issued by Core via Platform Admin; one key per module per env |
| Identity Bridge strips credentials, passes clean headers downstream | ✅ Decided | Engines receive X-Module-Id and X-Its-Id only |
| Two call types — user-initiated (with itsId) and system-initiated (without) | ✅ Decided | Endpoint contract defines which type each API accepts |
| API key lifecycle managed from Platform Admin | ✅ Decided | Dual-key rotation, immediate revocation |
| Core API read endpoints open to any module with valid API key | ✅ Decided | Response is naturally scoped to the request |
| Core API write endpoints restricted to Platform Admin and Core processes | ✅ Decided | No business module ever mutates Core state directly |
| Event publish authorization requirements (sourceModule, eventType, schema) | ✅ Decided | Three checks required — where they run is under analysis |
| Module publish scope — own event types only | ✅ Decided | Cannot publish under another module's event types |
| Event publish enforcement mechanism (Core gateway vs direct bus with ACLs) | 🔍 Under Analysis | Tied to messaging technology decision — load concern documented |
| Event subscription scope declared in System Definition Document | ✅ Decided | Core provisions bus access at onboarding |
| Bus technology must support topic-level consumer access control | ✅ Decided | Confirmed requirement for messaging technology selection |
| Core owns the event envelope schema | ✅ Decided | JSON Schema spec — language-agnostic |
| Standard event envelope fields (eventId, idempotencyKey, etc.) | ✅ Decided | See Section 4 |
| At-least-once delivery — consumers must be idempotent | ✅ Decided | Idempotency Policy |
| Breaking payload change = new schema version | ✅ Decided | Domain Event Schema Registry enforces this |
| Modules are sole publishers of their own events | ✅ Decided | Core engines are sole publishers of Core decision events |
| Core engines subscribe only to what they need | ✅ Decided | Explicit subscription list per engine |
| Audit Log Backbone subscribes to full event stream | ✅ Decided | Immutable record of all events |
| Tiered adoption path (Greenfield / REST / .NET / Stored-proc) | ✅ Decided | Pattern confirmed — sequencing per module TBD |
| DLQ + consumer lag monitoring in Platform Admin | ✅ Decided | Core team operational responsibility |
| Event bus messaging technology | 🔍 Under Analysis | Durability, replay, DLQ, managed vs self-hosted |
| Event bus authentication credential mechanism | 🔍 Under Analysis | Depends on messaging technology choice |
| Module-by-module adoption sequence and timeline | 📅 Pending Session | Technical discovery sessions with each module team |