Core Utilities
This file covers the utility layer in depth. For the definition of what a utility is vs a Core engine, the three-question test, and how utilities fit in the federation model, refer to CoreRolesRespons.md. This file does not repeat those foundations — it goes deeper on each utility and introduces the AI layer.
Two Categories of Utilities
Operational Utilities are pluggable capabilities — delivery mechanisms, form processing, authentication helpers, and shared UI infrastructure. They provide a capability. They do not encode a shared business rule. Any of them can be swapped or upgraded without touching business logic.
AI Utilities are a layer above operational utilities. They apply machine intelligence to specific, recurring platform problems — problems that have been observed directly across the 9 stakeholder discovery sessions and cannot be solved cleanly by deterministic rules alone.
Part A — Operational Utilities
1. Notifications Engine
The Notifications Engine is the platform's single delivery router for all outbound communication. No module builds its own notification sending logic. Every notification — whether triggered by a registration, an allocation, an HR assignment, or an ops broadcast — flows through this engine.
Why it has to be central: If every module sends its own WhatsApp messages and push notifications, a Mumin registering for Ashara Mubaraka could receive five messages from five different systems in thirty seconds, all saying slightly different things, with no coordination. That is already happening today.
What the engine handles:
| Channel | Delivery target | Volume context |
|---|---|---|
| WhatsApp Business API | Individual Mumineen | 70+ city groups, 1:1 confirmations, OTP |
| SMS | Fallback when WhatsApp not reachable | Particularly for international Mumineen |
| Push Notifications | Mobile app shell | In-app alerts, reminders, status updates |
| Formal communications | Passes, certificates, Raza letters | |
| In-app Banner | Via the Mobile Push Shell | Low-urgency updates visible on next open |
How a module uses it: A module publishes an event to the bus, or calls Core's Notifications API directly, with: recipient itsId, notification type, template key, and context variables. The Notifications Engine resolves the channel preference for that Mumin, populates the template, and dispatches. The module never knows which channel was used.
Template management: Notification templates (WhatsApp message text, SMS text, push body) are managed centrally in the Notifications Engine configuration — not hardcoded in modules. Ops teams can update copy without a code deployment.
Volume handling: At Ashara Mubaraka scale, outbound notifications at registration open could reach 240,000+ in a short window. The engine is backed by the Job Queue — notifications are queued and dispatched in controlled batches, not fired synchronously in a single burst.
Broadcast vs individual: Individual triggered notifications (your registration is confirmed) and broadcast announcements (tonight's Bethak timing has changed) use the same engine but different templates and dispatch modes. Broadcast mode is specifically for the 70+ WhatsApp group structure used during live events.
2. Queue Fair — Virtual Waiting Room
When registration opens for Ashara Mubaraka, tens of thousands of Mumineen attempt to register simultaneously. Without a waiting room, the backend receives a traffic spike it cannot absorb and either slows to a crawl or crashes — exactly when it needs to be most reliable.
Queue Fair is the virtual waiting room that sits in front of the API Gateway. Mumineen enter a queue. They are admitted to registration in controlled, predictable batches. The backend receives a steady stream of requests it can handle, not a wall of concurrent hits.
Key operational details:
- Queue Fair is only active during declared high-traffic windows — when registration opens, when a major allocation run happens, during Bethak registration for high-demand Miqaats. It is not always on.
- The activation window is configured per Miqaat in Core's Config Cascade — not hardcoded. Ops teams can turn it on or adjust the batch size without a deployment.
- Senior roles bypass the queue automatically. Aamil Saheb, Saadaat Kiraam, and other designated roles are configured in Config Cascade and skip Queue Fair. They reach the backend directly. This is the Queue Fair Bypass Policy.
- Queue Fair is adopted (external service at queue-fair.com). It integrates at the API Gateway layer — no module is aware of it.
Boundary note — utility behaviour, Core-controlled policy:
Queue Fair sits in front of the API Gateway. It is a utility — not a Core module. However, its behaviour is configured through Core's Config Cascade engine: when Queue Fair activates, it reads bypass_roles from Config Cascade at the Miqaat level. Senior roles (Aamil, Saadaat Kiraam) bypass automatically — no custom code per module. Queue Fair is pluggable; its policy is Core-controlled.
Future path — build and own: Queue Fair is a third-party dependency. Using it now is the right call — it is proven, fast to integrate, and handles the immediate scaling need. However, as the platform matures, there is a clear case for building a first-party virtual waiting room inside Core. Building it internally removes the third-party dependency, gives full control over bypass logic, burst shaping, and telemetry, and removes per-event licensing cost. This is not a Phase 1 concern, but it should be on the roadmap as a Phase 3+ candidate.
3. Job Queue + Workers — Core Internal
This utility is exclusively for Core's own background processing. It is not available to modules. Modules run their own independent background job systems.
What it runs:
| Job type | Engine using it |
|---|---|
| Dispatch 240K+ WhatsApp / push / SMS | Notifications Engine |
| Send approved broadcast to 70 WhatsApp groups | Broadcast Engine |
| Full Miqaat allocation run for a city or zone | Allocation Engine |
| Process one group Sharaf photo → crop individuals | Cropping AI |
| Generate and print 4–5K certificate serials | Kiosk / Misaal series |
| Relay outbox events from legacy systems to the bus | VMS Outbox Relay Agent |
| Bulk attendance sync (legacy path during migration) | Scanning Engine |
| Post-upload: thumbnail, virus scan | File processing (module-owned) |
What Core owns here entirely: the queue infrastructure, the workers, the internal database, retry policy, dead-letter queue, and the Platform Admin visibility into job health. No module sees inside this.
Note on technology: The specific queue technology (NATS JetStream work queues, SQS, or alternatives) is under analysis and will be decided separately.
4. SMS Gateway
A dedicated utility for SMS delivery — separate from the Notifications Engine dispatch logic, which uses it as one of its channels.
When SMS is used: When a Mumin's WhatsApp number is not reachable, not registered on WhatsApp, or has opted out of WhatsApp notifications. International Mumineen in regions with poor internet but good SMS coverage. OTP delivery as a fallback to WhatsApp OTP.
Implementation: Plugged into a gateway provider (e.g., AWS SNS, Twilio, or a local Indian gateway for domestic numbers). The Notifications Engine calls the SMS Gateway utility. The module that triggered the notification is not aware of which channel was used.
5. OTP / Two-Factor Authentication Utility
For sensitive operations — login from a new device, making a change to a Miqaat registration, approving an HR quota change — a second factor is required beyond the ITS token.
What the utility provides: Generate a time-bound OTP, deliver it via WhatsApp or SMS, validate the submitted OTP, and return pass/fail to the calling Core engine or module. The OTP generation and validation are stateless from the module's perspective — the utility handles the storage and expiry.
Why not in each module: If each module implements its own OTP, Mumineen get OTPs from 5 different senders with no coherent experience. More importantly, OTP delivery security is a platform-level concern.
6. Dynamic Forms Engine
Observed across multiple stakeholder sessions (Day 9 — Vajebaat Bethak, Day 1 — Misaaq, Day 4 — Qadambosi): different Miqaats and different modules need custom data collection forms that change event to event. Building new screens for every variation is not sustainable.
What the utility provides: A form schema registry and a runtime form renderer. An ops team member defines a form (field types, validation rules, conditional logic, required/optional), saves it with a form key, and the Mobile Push Shell or web UI renders it dynamically when called with that key.
What modules do with it: A module that needs a custom data collection step — an Araz submission form, a Misaaq applicant questionnaire, a Qadambosi request intake form — registers the form schema with this utility. It does not build a new UI. The utility handles the rendering and submission. The module receives the submitted data.
Who manages form schemas: Each module team owns their form schemas. The utility only provides the registration, rendering, and submission collection mechanism.
Deliberately Not in Core Utilities
These capabilities were considered and excluded. The last column notes where ownership sits or what the future path is.
| Capability | Why excluded | Owner / Future path |
|---|---|---|
| File Store | Each module owns its own data — its files are part of that data. Centralising storage creates a bottleneck and a Core dependency. Modules set up their own storage (S3 or equivalent) with their own access policies. Cross-module file needs (e.g. Cropping AI accessing Sharaf photos) are handled via event-driven file references, not a shared store. | Each module independently |
| PDF / Document Generation | Templates are entirely module-specific. If Core rendered module PDFs, module data would have to leave the module to format a document — a data ownership violation. Commodity library any module can import. The platform design system (covered separately in core_design_system.md) provides shared styling guidelines for visual consistency. | Each module; refer core_design_system.md for style |
| Print Utility (Kiosk / Label) | On-site device management is the Kiosk / Pass Engine module's concern. Modules that need to print send a request to the Kiosk module via the event bus — not to Core. | Kiosk / Pass Engine module |
| Mobile Push Shell | A shared app shell is a UX and product architecture decision, not a Core infrastructure concern. Core's job is decisions and shared business logic — not mobile app structure. In a true federation each module could ship its own app. The "one app" principle is valid but sits outside Core's boundary. Future path: can be formalised later as Shared Product Infrastructure alongside the Design System — a separate layer that Core does not own or operate, but the platform team governs. | Shared Product Infrastructure (future consideration) |
Part B — AI Utilities
The problems below were observed directly in stakeholder discovery sessions and field conversations. These are not speculative AI use cases — they are real operational pain points that AI is specifically well-suited to address.
The AI utilities described here run as background intelligence layers on top of Core data and event bus signals. They do not replace any Core engine. They augment it with intelligence.
AI-1. Sharaf Photo Cropping
The problem: Group Sharaf photos are taken at every Miqaat, capturing all Mumineen present. The value of these photos — a personal record of a Sharaf moment — is greatest when each Mumin receives their own individual crop, not just the group image. Doing this manually is impossible at scale.
What the AI utility does: Receives a group photo uploaded to the File Store. Detects every face in the image. Crops an individual image for each person, centered and padded correctly. Stores each cropped image in the File Store tagged to the detected itsId (where face recognition matches). Unmatched faces are stored for manual review.
Confidence threshold handling: Where face recognition confidence is below the threshold, the crop is marked for human verification before being associated to an itsId. The utility never silently makes a wrong association.
Data flow: Photo uploaded → sharaf.photo.uploaded event → Cropping AI job queued in Core's Job Queue → Processed → Individual crops stored → sharaf.crop.completed event published → Mumin's profile receives their personal Sharaf photo.
AI-2. Talent and HR Matching
The problem: "Hidden talent" was named explicitly in the AMS stakeholder session. When a new Miqaat's HR team starts planning, they are working from their own memory and their own contact list. Mumineen who performed exceptionally in a different event's HR team — in a different city, under a different department head — are invisible to the new planners.
What the AI utility does: Given a department, a role, a Miqaat, and optionally a city, it surfaces a ranked shortlist of Mumineen from the HR Bank who are likely matches — based on past khidmat history, the roles they have held, the departments they have worked in, attendance reliability at past events, and any positive flags in their record.
What it does NOT do: It does not make assignments. It does not bypass the department head's judgment. It gives the planning team a starting point that is better than memory.
Also useful for: Identifying Mumineen who have never been given a khidmat opportunity but whose profile suggests they could contribute (new talent pipeline), and flagging capacity gaps early in planning ("at current recruitment pace, Zone North's Construction team will be 40% short of quota").
AI-3. Registration Demand Forecasting
The problem: Platform capacity planning today is based on the previous event's attendance, adjusted by gut feel. For large-scale events like Ashara Mubaraka, this means either over-provisioning infrastructure (expensive) or under-provisioning and hitting Queue Fair limits prematurely.
What the AI utility does: Using historical registration data by Jamaat, zone, sub-grade, and prior attendance patterns, it forecasts expected registration volume by time window, city, and zone for an upcoming Miqaat. The forecast is available to the ops team and the Core team for infrastructure scaling decisions and Queue Fair activation timing.
Useful signals it uses: Registrations per Jamaat in prior events of the same type, sub-grade distribution in each Jamaat, historical conversion rate from eligibility to registration, Relay City capacity constraints that limit effective registration ceiling.
Important caveat: Forecasts are advisory — they inform human decisions on capacity and Queue Fair batch sizes. They do not auto-configure the system.
AI-4. Broadcast Content Assistant
The problem: Drafting WhatsApp broadcast messages in Lisan ud Daawat — with the correct terminology, respectful address conventions, and appropriate tone for different message types (operational update vs urgent alert vs celebratory announcement) — is time-consuming and inconsistent when done by multiple operators across cities.
What the AI utility does: Given a message intent in English or informal text, it drafts a properly formatted Lisan ud Daawat broadcast message suitable for the community's communication style. The operator reviews and approves before dispatch — the AI draft is a starting point, not an auto-send.
Practical inputs: The operator says "tell the Mumineen in Zone A that tonight's Bethak timing has changed to 9pm." The AI produces a correctly phrased WhatsApp message. The operator edits if needed, approves, and dispatches through the Broadcast Engine.
Why AI and not templates: Timing, venue, and context change every session. Templates cannot cover the variation. The AI utility handles the combination of context variables into a natural, correctly toned message.
AI-5. Anomaly Detection — Registration and Eligibility Fraud
The problem: At scale, patterns of unusual behaviour are impossible to spot manually. Examples observed or inferred: multiple registration attempts for the same itsId from different devices in quick succession, eligibility checks for a cluster of itsIds from the same IP in rapid sequence (automated scraping of eligibility status), attempts to register for a city that the Mumin's zone should not permit.
What the AI utility does: Monitors the event stream in near-real-time. Flags anomalous patterns to Platform Admin. Does not block requests automatically — flags them for human review. The Audit Log provides the evidence trail.
Signals it watches: Registration velocity per itsId and per device fingerprint, eligibility check patterns that do not match normal human usage, allocation request sequences that suggest scripted behaviour, sudden cluster of the same itsId across geographically inconsistent API gateway regions.
AI-6. Istefaada Grouping Optimisation
The problem: Istefaada runs in phases, with Mumineen grouped into batches for each Kitab session. Today, grouping is done manually, balancing gender, age, learning history, and city/zone logistics. Getting this right for thousands of participants is a significant planning burden.
What the AI utility does: Given the registered participants, their Kitab completion history from the HR Bank / Mumin Info Aggregator, their zone and city, and the session structure (number of groups, capacity per group, session schedule), it proposes an optimised grouping plan that respects the hard constraints (venue capacity, gender separation) and optimises on learning continuity (keeping related learners together where possible) and logistics (minimising unnecessary cross-city travel).
Output: A draft grouping proposal that the Istefaada team reviews, adjusts where needed, and then confirms. The AI does the combinatorial heavy lifting. The team retains final authority.
AI-7. Smart Notification Timing
The problem: Sending a critical operational message to 50,000 Mumineen at 11pm local time — or at the exact same moment as three other messages from other modules — produces poor read rates and Mumin fatigue.
What the AI utility does: When the Notifications Engine dispatches a non-urgent message, it uses learned patterns (when does this segment of Mumineen typically open WhatsApp? what is their recent message open behaviour?) to shift delivery to the optimal window — within the delivery deadline the ops team configured. Urgent messages bypass this and go immediately.
What it learns from: Open/read timestamps for past notifications (where delivery receipts are available via the WhatsApp Business API), time-of-day patterns by Jamaat and city (different cities have different active windows), and message type patterns (operational alerts are read faster than informational updates regardless of timing).
AI-8. Legacy Codebase Analysis — VMS Extraction Support
The problem: The VMS (Vaaz Management System) is a large, long-running .NET codebase. Years of business logic are embedded in stored procedures, in-application code, and undocumented rule combinations. Extracting what VMS does — as a precursor to de-bundling its Core-eligible logic and migrating the rest — is a major analysis task. Doing it manually risks missing logic that only surfaces during edge cases.
What the AI utility does: Reads the VMS codebase (with appropriate access). Documents what each stored procedure and service method does in plain language. Identifies patterns that look like eligibility rules, allocation logic, or RBAC checks — candidates to migrate to Core engines. Produces a structured extraction map: what stays in VMS, what moves to Core, what needs rewriting, and what can be retired.
This is AI-as-tool, not AI-as-product. It is a one-time (or periodic) analysis utility for the Core team's use during the VMS extraction track. It does not run in production against live Mumineen data.
AI-9. Eligibility Edge-Case Reviewer
The problem: Not every eligibility decision is clean. There are borderline cases — a Mumin who missed the eligibility threshold by one Bethak, a rule that changed mid-cycle, a family circumstance the system has no code for. Today these are handled manually, slowly, and inconsistently depending on who picks up the request.
What the AI utility does: For cases flagged as borderline by the Eligibility Engine (where the answer is not a clear pass or clear fail), the AI utility pulls the Mumin's full history — prior Miqaat attendance, Istefaada records, sub-grade standing, any prior override decisions — and produces a structured recommendation: approve, review (human discretion needed), or reject, with the reasoning laid out clearly.
What it does NOT do: It does not make the final call. The recommendation goes to a designated reviewer in Platform Admin. The human confirms or overrides. Every decision is logged with the AI recommendation and the human outcome — building an audit trail and improving future recommendations.
AI-10. Smart Slot Scheduling — Allocation Distribution
The problem: Distributing capacity slots across cities and zones is manually contested every Miqaat. Zone A feels it received fewer slots than Zone B despite similar population. Families with accessibility needs end up in venues that are difficult to reach. Prior allocation history is not factored in. The process is opaque.
What the AI utility does: Given the registered participants, their zone, family size, accessibility flags from their ITS profile, prior allocation history, and the available venue capacities per city and zone, it produces an optimised slot distribution recommendation. Balancing criteria: equity across zones, family size fit to venue capacity blocks, accessibility routing, and historical fairness (zones that received fewer preferred slots in prior events get priority weighting).
Output: A draft allocation distribution plan that the Core team reviews before activating. The Allocation Engine consumes it. The AI does not write directly to the Allocation Engine — the recommendation is a human-confirmed input.
AI-11. Accommodation Optimizer
The problem: Matching arriving Mumineen to available accommodation blocks is done manually, juggling family size, gender segregation rules, Relay City proximity, accessibility needs, and Miqaat date ranges. For large events, this is hundreds of hours of coordination effort, and re-assignments during the event are common because the initial match was suboptimal.
What the AI utility does: Given the arriving family groups (size, gender composition, accessibility needs, travel dates, city), and the accommodation inventory (block sizes, locations, accessibility ratings, dates available), it produces an optimised matching proposal. Families are matched to blocks in a way that minimises re-assignments, respects hard constraints, and reduces walk distance to venues where possible.
Output: A draft accommodation assignment list. The accommodation module team reviews and adjusts before committing. The AI absorbs the combinatorial complexity — the team focuses on exceptions.
AI-12. Natural Language Admin Query
The problem: Questions that should take thirty seconds — "How many families from Mumbai are allocated to Zone North?" or "Which Jamaats have the highest proportion of unconfirmed registrations?" — today require an IT ticket, a SQL query, or waiting for a specific report to be built. Non-technical ops team members cannot self-serve data questions.
What the AI utility does: The Platform Admin console includes a natural language query interface. An admin types a question in plain English. The AI translates it into a Core API query or a structured data read, executes it against Core's data, and returns the answer in plain language with the option to export. No SQL. No IT ticket. No waiting.
Scope boundary: Queries are read-only. The utility surfaces data — it never writes, allocates, or changes any record. All queries are logged in the Audit Log with the question text and the query it generated.
AI-13. Complaint Auto-Classification and Routing
The problem: Support requests during a live Miqaat arrive in volume and in multiple channels — WhatsApp messages to city coordinators, calls to the ops team, messages through the Mumin app. Routing them manually to the right team wastes time that does not exist during operational windows.
What the AI utility does: When a support request arrives (text message or form submission), the AI classifies it by type — allocation dispute, eligibility question, technical issue, accommodation complaint, access / entry issue — and routes it to the correct team's queue in Platform Admin. Before handoff, it pulls relevant context from Core's records: the Mumin's current allocation status, eligibility decision, pass status, and any prior support interactions. The receiving team gets a pre-loaded context view, not a blank ticket.
What it does NOT do: It does not resolve the complaint. It does not communicate back to the Mumin. It removes the classification and triage burden so the team reaches resolution faster.
AI-9. OCR and Physical Form Processing
The problem: Some Miqaat processes still involve physical paper — handwritten Araz submissions, physical Misaaq application forms in cities without reliable internet, handwritten attendance sheets at smaller Bethak venues. These need to enter the digital platform without a full manual re-entry workflow.
What the AI utility does: Receives a photographed or scanned form. Extracts field values using OCR and field recognition. Maps extracted values to the appropriate schema (Misaaq application fields, Araz submission fields, attendance sheet columns). Returns a structured data object for human review before commit. A human confirms — the AI does not write directly to any module's database.
Accuracy handling: Extracted fields with low confidence are highlighted for the reviewer. The utility does not guess silently. Every committed record carries a flag indicating it originated from OCR processing.
The AI Integration Principle
AI utilities are consumers of Core — not components of Core. They read from the event stream and Core APIs. They write recommendations back as inputs to human decisions or as hints to Core engines.
AI utilities do not make final allocation, eligibility, or RBAC decisions. Core does.
AI augments human judgment. Core enforces the final decision. This boundary must be maintained so that accountability remains clear and every consequential outcome has a human confirmation in the audit trail.
Utility Readiness — Build, Adopt, or Defer
Not all utilities are needed on day one. The table below indicates the approach and earliest phase relevance.
| Utility | Approach | When needed |
|---|---|---|
| Queue Fair | Adopt (external) | Phase 2 — before registration opens at scale |
| Notifications Engine | Build | Phase 1 — first modules need confirmation messages |
| Job Queue + Workers | Build (Core internal) | Phase 0 — Core engines need background processing |
| SMS Gateway | Adopt | Phase 1 — parallel to WhatsApp |
| OTP / 2FA | Build | Phase 1 — sensitive operations |
| Dynamic Forms Engine | Build | Phase 2 — Misaaq, Qadambosi, Vajebaat intake |
| Design System | See core_design_system.md | Covered separately |
| Mobile Push Shell | Build | Phase 2 — Mumin-facing app |
| Each module owns its own storage | ||
| Each module uses its own PDF library | ||
| Kiosk / Pass Engine module owns this | ||
| Future: Shared Product Infrastructure | ||
| AI-1 Sharaf Cropping | Build | Phase 2 — first Sharaf event |
| AI-2 Talent Matching | Build | Phase 3 — HR Bank needs sufficient history first |
| AI-3 Demand Forecasting | Build | Phase 2 — needs 1–2 events of data |
| AI-4 Broadcast Assistant | Build | Phase 2 — before first live broadcast |
| AI-5 Anomaly Detection | Build | Phase 2 — active at first high-volume registration |
| AI-6 Istefaada Grouping | Build | Phase 3 — specific to Istefaada onboarding |
| AI-7 Smart Notification Timing | Build | Phase 3 — needs notification history to learn from |
| AI-8 VMS Codebase Analysis | Build | Track B — VMS extraction work stream |
| AI-9 Eligibility Edge-Case Reviewer | Build | Phase 2 — active once Eligibility Engine is live |
| AI-10 Smart Slot Scheduling | Build | Phase 2 — before first allocation run at scale |
| AI-11 Accommodation Optimizer | Build | Phase 3 — needs accommodation module onboarded |
| AI-12 Natural Language Admin Query | Build | Phase 2 — Platform Admin surface |
| AI-13 Complaint Auto-Classification | Build | Phase 2 — before first live Miqaat |
| AI-14 OCR / Form Processing | Build | Phase 3 — physical-to-digital bridge |