Appearance
System Architecture
The Kaya Sync platform is built on an event-driven, multi-tenant, Uber-shaped dispatch architecture. It runs on DigitalOcean for all compute + managed data services (single vendor, one bill), with Cloudflare kept as the free edge and R2 as object storage (zero-egress). Services are split by workload from day 1: the two HTTP API surfaces (core + api) plus the AI sidecar on App Platform (PaaS), real-time location + dispatch on a Droplet (persistent WSS), and stateful data on Managed Postgres + Redis.
The platform exposes two distinct API surfaces from MVP: core.kayasync.com (used by our own admin app, client portal, and mobile app) and api.kayasync.com (used by third-party customers and partners). The public API is modelled as just another client of the core — same underlying business logic, different auth model, different rate-limit tiers, independent versioning. This lets us evolve the core weekly while holding the public contract stable for months at a time. See API surface below.
Kubernetes, Kafka, and multi-region replication only enter the picture when volume forces them — see Hosting & Data Plan for phasing.
The split at MVP is deliberate: it draws the language, service, and API-audience boundaries that a mini-Uber-scale platform needs so we grow into them, rather than rewriting under load.
Topology at a glance
┌──────────────────────────────────────────────────────────────────────────┐
│ Cloudflare (free edge) │
│ DNS · TLS · DDoS · Turnstile · WAF · Rate Limiting · CF Access · R2 │
│ │
│ kayasync.com wiki.kayasync.com cdn.kayasync.com (R2) │
│ admin.kayasync.com app.kayasync.com │
└─────┬─────────────┬──────────────────┬──────────────────┬────────────────┘
│ │ │ │
│ HTTPS │ HTTPS │ HTTPS │ WSS (GPS +
│ (admin, │ (external │ (internal │ offers,
│ client, │ customers / │ service auth) │ operator app)
│ mobile) │ partners) │ │
▼ ▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────────┐
│ core.kayasync.com│ │ api.kayasync.com │ │ ws.kayasync.com │
│ Hono TS │ │ Hono TS │ │ Caddy → Go services │
│ App Platform │ │ App Platform │ │ DO Droplet s-2vcpu-4gb │
│ Basic 1CPU/1GB │ │ Basic 1CPU/1GB │ │ 2 vCPU · 4 GB │
│ │ │ │ │ │
│ CORE API │ │ PUBLIC API │ │ location-svc (Go) │
│ 11 svc modules │ │ Thin translator: │ │ dispatch-svc (Go) │
│ Auth: JWT + CF │ │ API key/OAuth │ │ NATS JetStream │
│ Access (admin) │ │ → service JWT │◀────────▶│ + snapshots to R2 │
│ │ │ → core internal │ intern. │ │
│ │ │ │ HTTP │ │
└──────┬───────────┘ └──────┬───────────┘ └──────────┬───────────┘
│ │ │
│ │ ai-sidecar (Pro 1CPU/2GB) │
│ │ (Python FastAPI, App Platform) │
│ │ called by core internal HTTP │
│ │ │
│ DO VPC private network (free, no egress cost) │
▼ ▼ ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ DO Managed Postgres Production 2 vCPU / 4 GB / 38 GB + standby node │
│ + TimescaleDB extension │
│ - txn state · audit log · location-history hypertables │
│ - automatic failover, PITR (7-day), automated backups │
│ │
│ DO Managed Redis Basic 2 GB │
│ - Redis GEO index (last-known op location) · sessions · rate limit │
│ │
│ Backups nightly pg_dump → Cloudflare R2, 30-day │
│ (belt-and-braces on top of DO's managed backups) │
└──────────────────────────────────────────────────────────────────────────┘
▲ ▲
│ background sync │ FCM push
│ │
┌──────────┴────────────────────────────────┐ ┌──────────┴──────────────┐
│ React Native operator app │ │ Firebase Cloud │
│ On-device TF Lite (fingerprint / live) │ │ Messaging (free) │
│ Offline-first capture + sync queue │ │ │
│ REST: core.kayasync.com/v1/operator/* │ │ │
│ WSS: ws.kayasync.com (GPS + offers) │ │ │
└───────────────────────────────────────────┘ └─────────────────────────┘Recommended MVP tier is ~$371/mo all-in (see Cost model below for full breakdown). The user is comfortable spending $100-1000/mo at MVP; we deliberately invest the headroom in HA on the DB layer, real observability, and error tracking rather than squeezing dollars on compute. Third of the budget covers real production-grade infrastructure; two-thirds are reserve for growth.
Domain map
| Domain | Purpose | Runs on |
|---|---|---|
kayasync.com | Marketing site (Astro) | Cloudflare Workers |
wiki.kayasync.com | Internal wiki (VitePress) | Cloudflare Pages |
cdn.kayasync.com | R2 media (scan videos, SOD photos) | Cloudflare R2 |
admin.kayasync.com | Ops staff dispatch console (Next.js) | DO App Platform (behind CF Access) |
app.kayasync.com | Enterprise client portal (Next.js) | DO App Platform |
core.kayasync.com | Core API — internal clients | DO App Platform (Hono TS) |
api.kayasync.com | Public API — external customers, partners | DO App Platform (Hono TS, separate app) |
ws.kayasync.com | WSS endpoint for operator app (GPS + offers) | Caddy on dispatch droplet → Go services |
status.kayasync.com | Public status page (uptime, incident history) tied to api.kayasync.com | Better Stack / Statuspage (SaaS, ~$0-30/mo starter — Stage 2 concern) |
Cost model
MVP budget envelope: $100-1000/mo (approved). Recommended tier lands at ~$371/mo all-in — ~37% of the ceiling. The headroom explicitly buys reliability + observability, not more compute.
Recommended MVP tier
| Line | Provider | What it gets us | $/mo |
|---|---|---|---|
core app | DO App Platform Basic 1CPU/1GB | Serves internal clients (admin, client, mobile) | 12 |
api app | DO App Platform Basic 1CPU/1GB | Public API for enterprise customers + partners | 12 |
ai-sidecar app | DO App Platform Pro 1CPU/2GB | Heavier ML jobs (PaddleOCR + OpenCV) — Pro tier for memory headroom | 25 |
| Dispatch droplet | DO Droplet s-2vcpu-4gb (2 vCPU / 4 GB) | Runs location-svc + dispatch-svc + NATS + Caddy | 24 |
| Droplet backups | DO snapshot backups | 20% of droplet cost, weekly + retention | 4 |
| Managed Postgres | DO Managed Production 2 vCPU / 4 GB / 38 GB + standby | Auto-failover on Production tier — critical for the ledger + audit log | 120 |
| Managed Redis | DO Managed Basic 2 GB | GEO index + sessions headroom | 30 |
| DO subtotal | 227 | ||
| Observability | Grafana Cloud Pro (or Datadog starter) | Metrics + traces + log aggregation, alerting | 50 |
| Error tracking | Sentry Team | Real error tracking with source-map upload, release tracking | 26 |
| Edge security | Cloudflare Pro | Better WAF, advanced rate-limit rules, Bot Fight Mode | 20 |
| Secret management | 1Password Teams (6 users at MVP) | Rotatable secrets, shared vaults for the eng team | 48 |
| Object storage | Cloudflare R2 (< 10 GB) | Scan media, backups | 0 |
| Third-party subtotal | 144 | ||
| All-in MVP | ~$371/mo |
Per-transaction fees (SMS via Africa's Talking, mobile-money via Flutterwave, email via Resend) are invoiced separately by their providers — none material at MVP scale.
Why the specific upgrades over the bare-minimum Basic tier
| Upgrade | Baseline was | Cost delta | What it buys |
|---|---|---|---|
| Postgres → Production | Basic 2 GB, $30 | +$90 | Automatic failover + standby node. The DB is the platform's single point of truth. No excuse to run it without HA at $120/mo. |
| Redis → Basic 2 GB | Basic 1 GB, $15 | +$15 | GEO index headroom for Stage 2 growth; keeps us off the resize path for 12+ months |
ai-sidecar → Pro | Basic 1 GB, $12 | +$13 | ML workloads need memory. PaddleOCR + OpenCV in a 1 GB container swaps under load. |
| Dispatch droplet → s-2vcpu-4gb | s-1vcpu-2gb, $12 | +$12 | 2× CPU and 2× RAM headroom for NATS JetStream + Go WSS + Caddy on the same box |
| Grafana Cloud Pro | Free tier (10K series, 14-day retention) | +$50 | Longer retention, per-service dashboards, real alerting. Free tier hits limits fast once real traffic exists. |
| Sentry Team | Free (5K errors/mo) | +$26 | Source-map upload, release tracking, integrations — free tier isn't usable for production. |
| Cloudflare Pro | Free | +$20 | Advanced rate-limiting rules, Bot Fight Mode, image optimisation. Free is fine for the marketing site; not for the public API. |
| 1Password Teams | Ad-hoc secret sharing | +$48 | Rotate secrets on schedule, audit access. Non-negotiable for a team of 6+ working across shared infra. |
What's deferred to Stage 2 upgrade
Only add these when volume / real customers force it:
| Deferred item | When to add | Estimated cost delta |
|---|---|---|
Second core app instance (HA on App Platform) | When admin ops are 24/7 and 30-second restarts matter | +$12/mo per instance |
| Second dispatch droplet + DO Load Balancer | When >1000 concurrent WSS or 24/7 pilot ops live | +$30/mo (droplet) + $12/mo (LB) |
| Managed Postgres → Production 4 vCPU / 8 GB | When p95 query latency > 100 ms sustained | +$120/mo → ~$240/mo total |
| Redpanda Cloud (or Confluent) | When event volume > 1000/sec sustained | +$50-100/mo |
| DOKS (Kubernetes) | When we need pod-level autoscaling on 3+ services | +$40/mo control plane + node cost |
| Multi-region | When we serve a second country with data-residency needs | +100% of current bill |
Cost trajectory
| Stage | When | All-in/mo | Delta | Trigger |
|---|---|---|---|---|
| MVP recommended | Sep 2026 – Mar 2027 | ~$365 | baseline | Approved envelope |
| Stage 2 — controlled pilot | Apr – Aug 2027 | ~$550 | +$185 | Add HA on core app + dispatch droplet, upgrade Postgres tier |
| Stage 3 — regional scale | Sep 2027 – 2028 | ~$1,500 | +$950 | Add DOKS, Redpanda Cloud, second region, dedicated observability stack |
| Stage 4 — Uber-scale | 2029+ | 10K+ | large step | Multi-region active-active, custom matching engine, dedicated SRE team |
API surface
Two HTTP API surfaces from MVP, one WSS surface. The two HTTP surfaces have different audiences, different auth models, different rate-limit tiers, and independent release cadence.
Core API — core.kayasync.com
Used by: our own clients — the admin app, the client portal, the operator mobile app, and the public API service when it needs to do real work.
- Auth: JWT (issued via SMS OTP for operators; SSO for admin and client users)
- CF Access layered on all
/v1/admin/*routes — double-auth (network + application) for the internal ops surface - CF Access seat cap: free tier is capped at 50 seats (admin-only by policy). Ops-team growth past ~40 users triggers either a CF Access paid plan (~$3/user/mo) or a move to a self-hosted Zero Trust proxy. Both are Stage 2 concerns, not MVP.
- Rate limits: generous — these are our own clients, not third parties (see table below)
- CORS: locked to
admin.kayasync.comandapp.kayasync.com(no wildcards) - No public docs — the internal surface is not published
core.kayasync.com/
├── v1/
│ ├── admin/* ← CF Access + admin JWT
│ │ (dispatch console, user mgmt, settlement approvals,
│ │ fraud review, corridor config, operator directory)
│ │
│ ├── operator/* ← operator JWT (issued after SMS OTP)
│ │ (used by the React Native mobile app for REST calls;
│ │ accept-offer, submit-scan, availability toggle, etc.)
│ │
│ ├── client/* ← client JWT (SSO or password)
│ │ (used by app.kayasync.com — place order, track,
│ │ view invoices, download statements)
│ │
│ └── internal/* ← service-account JWT (short-lived, VPC-only)
│ (called by api.kayasync.com, ai-sidecar callbacks,
│ dispatch-svc state updates, cron workers)
└── (WSS is on ws.kayasync.com, not here — see below)Public API — api.kayasync.com
Used by: third parties — enterprise customers with their own systems, integration partners, external monitoring dashboards.
- Auth: API key (per-tenant, per-environment) OR OAuth 2.0 (for partner-managed apps that act on behalf of end users)
- Cloudflare Rate Limiting: aggressive, tied to plan tier (free / startup / enterprise). Configured at the Cloudflare edge, not just the app
- Cloudflare Bot Fight Mode + WAF Managed Rules: on
- CORS:
*— public API, third-party browser apps allowed - Public OpenAPI docs at
api.kayasync.com/docs(interactive Swagger UI or Redoc) - Public status page at
status.kayasync.com(uptime, incident history) - Versioned URL prefix (
/v1/) — new major versions get a new prefix, old versions supported for 12+ months
api.kayasync.com/
├── v1/
│ ├── orders POST create order
│ │ GET list orders (tenant-scoped)
│ ├── orders/:id GET order detail
│ ├── orders/:id/tracking GET current state + coarse-grained location
│ ├── orders/:id/events GET state transitions (paged)
│ ├── orders/:id/stream GET SSE stream of live updates
│ ├── webhooks/subscribe POST register callback URL for events
│ ├── webhooks/deliveries GET delivery log (debugging)
│ ├── operators/availability GET aggregate availability, not per-operator
│ └── settlement/statements GET monthly invoice / statement PDF
└── docs/ ← public OpenAPI + interactive explorerThe public API deliberately does NOT expose:
- Individual operator IDs, phone numbers, or trust tiers
- Real-time GPS coordinates (only coarse state buckets:
at_origin,in_transit,near_destination,delivered) - Internal fraud / collusion / dispute scores
- Admin-only operations (approvals, overrides, user management)
- Akroma internal signals or reason codes
- Anything from Kaya Sync's Level-1 non-disclosure list (Fast-Pass, trust tiers, fraud scoring, settlement gating, phone-number anchoring, Akroma, Capture Copilot, AI-gated settlement, automated payouts, wallet, escrow, payment processor)
WebSocket entry — ws.kayasync.com
Used by: the operator mobile app for GPS heartbeat + offer subscription.
- Auth: operator JWT presented on the WSS handshake (same JWT as REST calls)
- TLS: Caddy on the dispatch droplet (auto-renew via ACME)
- Reverse-proxied to
location-svcfor GPS ingest and todispatch-svcfor offer subscription - No CF Access — public accessible with JWT (operators install the app on their own phones)
- Direct to the droplet — deliberately NOT proxied through
core.kayasync.com, because Hono/Node is the wrong runtime for high-concurrency WSS
Auth + rate limits
| Route | Auth | Cloudflare edge limit | App-layer limit |
|---|---|---|---|
core.kayasync.com/v1/admin/* | CF Access + admin JWT | 100 req/min per IP | 60 req/min per admin JWT |
core.kayasync.com/v1/operator/* | operator JWT | 300 req/min per IP | 60 req/min per operator JWT |
core.kayasync.com/v1/client/* | client JWT | 200 req/min per IP | 120 req/min per client JWT |
core.kayasync.com/v1/internal/* | service-account JWT | (VPC-only, no edge) | 10,000 req/min per service-account |
api.kayasync.com/v1/* (free tier) | API key | 60 req/min per API key | 30 req/min hard cap |
api.kayasync.com/v1/* (startup tier) | API key | 500 req/min per API key | 300 req/min hard cap |
api.kayasync.com/v1/* (enterprise) | API key or OAuth | Custom per contract | Custom per contract |
ws.kayasync.com (GPS heartbeat) | operator JWT on handshake | 1 conn/min per IP (reconnect budget) | 1 conn per operator |
Rate limits on the public API are part of the pricing model, not a technical implementation detail. Free tier gets 60 req/min. Startup tier ($99/mo) gets 500. Enterprise gets custom SLA. Cloudflare enforces at the edge; the app enforces the hard cap as a second line of defence.
Service-to-service auth
When api.kayasync.com calls core.kayasync.com/v1/internal/* (or when ai-sidecar calls back into core with an OCR result, or when dispatch-svc calls core to update order state), the caller presents a short-lived service-account JWT:
json
{
"sub": "api-public-service", // or "ai-sidecar", "dispatch-svc", "core-cron"
"tenant_id": "acme-logistics-gh", // for api.kayasync.com callers: derived from API key
"scopes": ["orders:read", "orders:write", "operators:read"],
"exp": 1723388400 // 5 minutes from issue
}Signed with a shared HS256 key rotated quarterly via 1Password + GitHub Secrets. All service-to-service calls happen over the DO VPC — no public routing.
Wire protocol for internal RPC: HTTP+JSON at MVP (Hono routes on core.kayasync.com/v1/internal/* called from api, ai-sidecar callbacks, dispatch-svc state updates). gRPC considered and deliberately deferred — the ~5-10× payload savings and ~2-3× latency win don't matter at MVP volume, and HTTP+JSON stays curl-debuggable in production. Trigger to swap in gRPC: internal RPC p95 > 100 ms sustained, volume > 1000 req/sec sustained, or a public gRPC contract requested by an enterprise integration. Stage 2 / Stage 3.
For the public API caller flow specifically:
- External customer's system sends request with
Authorization: Bearer <api-key>toapi.kayasync.com/v1/orders api.kayasync.comvalidates API key → derives tenant ID and scopes from the key's planapi.kayasync.commints a service-account JWT scoped to that tenant + those scopes (5 min expiry)api.kayasync.comcallscore.kayasync.com/v1/internal/ordersover VPC with the service-account JWTcore.kayasync.comvalidates the JWT, enforces tenant scoping server-side, executes the operation- Response flows back →
api.kayasync.comtranslates to public response shape → returned to customer
Why not just have api.kayasync.com share the same DB as core? Because the whole point of the split is that api.kayasync.com doesn't own business logic — it authenticates, rate-limits, translates, and forwards. If it started reading Postgres directly it would duplicate logic that lives in the core, which is exactly the maintenance disaster this design avoids.
Versioning + evolution
- Core API — versioned per route family (
/v1/admin/*), no global version. Rev in place with backwards-compatible additions; breaking changes get a new major-numbered path. - Public API — single global version prefix (
/v1/,/v2/). New major version = new prefix, old version supported for 12 months minimum. Deprecation notices in response headers 6 months before removal. - Internal service-to-service — no version, changes freely with each core deploy.
The public API is the CONTRACT with the outside world. Everything else is an implementation detail we can change on our own timeline.
Core services (logical)
The platform is conceptually 12 services per Port Allocation. At MVP the physical deployment is 5 apps, not 12:
| # | Physical app | Contains (logical services) | Runs on |
|---|---|---|---|
| 1 | core (Hono TS) | Identity, Device, Vehicle, Order, Scan Ingestion, Integrity AI orchestrator (thin — calls ai-sidecar for the ML work), Settlement, Carbon, Fraud, Audit, Notification (11 modules) | DO App Platform → core.kayasync.com |
| 2 | api (Hono TS) | Public-API thin translator — auth (API key/OAuth), rate limit, request/response translation, forwards to core | DO App Platform → api.kayasync.com |
| 3 | ai-sidecar (Python FastAPI) | Integrity AI heavy lift — PaddleOCR, OpenCV, MediaPipe. Not a full logical service; a compute worker the Integrity AI module inside core invokes via internal HTTP | DO App Platform |
| 4 | location-svc (Go) | Real-time GPS ingestion, Redis GEO writes | DO Droplet → ws.kayasync.com |
| 5 | dispatch-svc (Go) | Matching + offer broadcast, race-safe accept | Same droplet as location-svc |
The 12 logical "services" below map to these 5 physical apps: 11 run as modules inside core; Orchestration is fully split into location-svc + dispatch-svc. ai-sidecar is not a logical service — it's a compute worker for the Integrity AI module. The port map governs how the Hono-hosted modules split out into separate deployables later.
| Service | Owns | Splits out when |
|---|---|---|
| Identity & Access | Operator/originator accounts, JWT issue, RBAC | Multi-region deployment needs local issuance |
| Device Governance | Device binding, attestation, trust tier | Mobile install base > ~5,000 active devices |
| Vehicle & Capacity | SOD attestation, vehicle profile, capacity inference | Capacity-AI inference becomes a bottleneck |
| Order & Container | Order lifecycle, container state machine, multi-leg routing | Order volume > 10k/day |
| Scan Ingestion | Upload pre-signing, scan record creation, dedupe | Media volume saturates a single host's bandwidth |
| Integrity AI | Server-side OCR, fingerprint match, geo / time validation | Need a GPU host for batch ML |
| Orchestration | Aggregation/disaggregation scoring, dispatcher queue | Already split at MVP — runs as location-svc + dispatch-svc in Go on the dispatch droplet (behind ws.kayasync.com); see "Real-time location + dispatch" section below |
| Settlement | Eligibility check, HOLD/RELEASE/REVERSE workflow, payout dispatch | Settlement volume warrants finance-isolated host |
| Carbon Ledger | Per-journey carbon calc, ESG export | Partner reporting volume warrants own host |
| Fraud & Collusion | Replay/duplicate/dyad/relabel detection, risk flags | Detection model needs a dedicated worker |
| Audit / Event Store | Append-only event ledger, custody chain reconstruction | Event volume saturates Postgres write capacity |
| Notification | SMS, push, USSD, in-app, email fan-out | High concurrency or per-channel scaling |
All 12 services share one Postgres schema at MVP. The split-out trigger is volume or compliance, not architectural purity.
Runtime stack
Polyglot by workload — one language would be wrong here. Real-time WebSocket concurrency (100K+ operators) and geospatial matching are fundamentally different workloads to CRUD, and pretending otherwise costs a rewrite at scale.
| Layer | Choice | Runs on (MVP recommended tier) | Why |
|---|---|---|---|
| Core API (CRUD, auth, orders, admin, invoicing, settlement, MIS) | TypeScript on Hono | core.kayasync.com on DO App Platform (Basic 1 CPU / 1 GB) | 70% of code volume, 5% of concurrency load. Fast dev, shared types with mobile + web, huge hiring pool. Fastify considered — defensible, tabled for Backend #1 Sprint 0 spike. |
| Public API (external customers + partners) | TypeScript on Hono | api.kayasync.com on DO App Platform (Basic 1 CPU / 1 GB) | Thin translator: API-key / OAuth auth, rate limit, request/response translation, forwards to core via VPC. Independent versioning + release cadence. |
| Real-time location ingest (operator GPS over WebSocket) | Go | DO Droplet s-2vcpu-4gb (2 vCPU / 4 GB) | Node breaks around 30-50K concurrent WS per node; Go handles 500K comfortably. Droplet (not App Platform) — persistent WSS + no cold starts. |
| Dispatch / matching (find N-best operators, broadcast offer, race-safe accept) | Go | Same droplet as location-svc | Geospatial radius queries + goroutine-per-broadcast + Redis SETNX. Uber's real dispatch backbone is Go/Java. |
| AI / OCR / video | Python on FastAPI | DO App Platform (Pro 1 CPU / 2 GB — memory headroom for ML) | CV/ML ecosystem (PaddleOCR, OpenCV, MediaPipe) non-negotiable in Python. Called by core internal HTTP. |
| Mobile | React Native + TensorFlow Lite | Operator devices | Shared types with backend; on-device fingerprint + replay detection aligned with offline-first principle. REST to core.kayasync.com, WSS to ws.kayasync.com. |
| Internal admin / client portal | Next.js 15 (App Router) + React 19 | DO App Platform (Basic 1 CPU / 1 GB, ~$12/mo when added) | Same codebase for admin.kayasync.com and app.kayasync.com with RBAC-scoped routing. Astro rejected — islands architecture makes shared state across live-dispatch dashboards awkward. |
| Reverse proxy / TLS | DO App Platform LB (App Platform apps) + Caddy (dispatch droplet, ws.kayasync.com) | — | App Platform auto-provisions TLS + LB; Caddy on the droplet handles WSS termination + ACME renewal. |
| DB | Postgres + TimescaleDB extension | DO Managed Postgres Production (2 vCPU / 4 GB / 38 GB + standby) | Production tier for automatic failover. TimescaleDB extension supported on DO Managed. Same engine for txn state, audit log, and location-history hypertables. |
| Cache + geospatial index | Redis | DO Managed Redis Basic 2 GB | GEOADD / GEORADIUS at sub-ms latency for driver-location lookup. Managed = no self-host burden. |
| Event bus | NATS (with JetStream) | Single instance on dispatch droplet | Handles 25 events/sec at MVP (100 ops × 5 s heartbeat) with 50 MB RAM. Chosen over MQTT because MQTT lacks streams/replay + request-reply + KV — not designed for backend microservice communication (Uber runs MQTT for mobile↔server only, Kafka for backend). Kafka / Redpanda deferred to Stage 3; MQTT considered as a third surface for mobile heartbeats at Stage 3 if cellular data cost forces it. |
| Geospatial index (in-service) | Uber's H3 — Go + Python + TS bindings | In-process | Hex-grid cells for O(1) neighbour lookup. Every scan, operator, order lives in an H3 cell from day 1. |
| Object storage | Cloudflare R2 | Cloudflare (kept, not DO Spaces) | Zero egress. Serving 2 TB/mo of scan video to enterprises: R2 = $7.50, DO Spaces = $17.50. Gap widens at Stage 3. Worth breaking the single-vendor rule for. |
| Observability | Grafana Cloud Pro (or Datadog starter, or self-hosted Grafana + Loki + Prometheus later) | External SaaS | Metrics + traces + log aggregation with real alerting. Free tier hits limits fast; Pro is $50/mo. |
| Error tracking | Sentry Team | External SaaS | Source-map upload, release tracking, integrations. Free tier isn't usable for production. $26/mo. |
| Edge security | Cloudflare Pro | Cloudflare | WAF Managed Rules, advanced rate-limiting rules, Bot Fight Mode. $20/mo. |
| Secrets | 1Password Teams + GitHub Secrets + DO env vars | External | Rotatable secrets, shared vaults, audit trail. Vault deferred to Stage 3. |
| Push | Firebase Cloud Messaging | External | Unlimited free |
| SMS / USSD | Africa's Talking | External | Best Ghana coverage, has USSD |
| Mobile money | Flutterwave | External | Best Africa-wide coverage |
| Transactional email | Resend | External | Already wired (lead form); 3K/mo free |
Rejected alternatives
Choices we considered and explicitly did not take, with reasoning so future readers don't re-open closed debates.
| Rejected | Category | Why not |
|---|---|---|
| Elixir / OTP | Runtime | Perfect fit for real-time fan-out (Discord runs 14M concurrent WS on it), but Kenya/Africa hiring pool too thin and OTP paradigm ramp too costly at team size 5. Re-evaluate at 10+ engineers. |
| Rust | Runtime | Highest perf, slowest to write — wrong for MVP velocity. Revisit for the matching engine at Stage 4. |
| Java / JVM | Runtime | Mature and battle-tested at Uber scale, but heavy Ops burden on a small team. |
| Node for everything | Runtime | Breaks around 5–10K concurrent driver WSS connections. That's why Go owns location + dispatch. |
| Full-Go on backend | Runtime | Breaks the shared-types story with mobile (React Native TS) and Next.js. TS + Go polyglot wins. |
| DO Spaces | Object storage | R2's zero-egress model wins by ~$10–500/mo depending on stage. Worth breaking the single-vendor rule for. |
| Kafka at MVP | Event bus | ~$72/mo for 3-node self-host + real ops burden for 25 events/sec of traffic. NATS handles this in its sleep. Kafka / Redpanda deferred to Stage 3. |
| Kubernetes at MVP | Orchestration | K3s / DOKS added at Stage 3 when we horizontally scale pods. Docker Compose on the dispatch droplet + App Platform is enough for 5 apps. |
| MQTT as backend event bus | Event bus | Designed for constrained-device telemetry over unreliable networks — right for driver telemetry at 10K+ operators on cellular, wrong for backend microservice communication (no request-reply, no streams/replay, no KV/object store). Uber runs MQTT/Bifröst for mobile↔server only, not for backend — we do the same, deferring MQTT to Stage 3 as a third messaging surface for mobile heartbeats only if cellular data-cost complaints force it. |
Not a rejection — category clarification
gRPC. gRPC is point-to-point RPC; NATS is an event bus. They solve different problems, and real production systems use both (Uber does: gRPC for sync service-to-service, Kafka for async events). At Kaya Sync we use HTTP+JSON for internal RPC at MVP (debuggability with curl / DevTools / Postman wins over Protobuf perf at 25 events/sec). Trigger to swap in gRPC: internal RPC p95 latency > 100 ms sustained, or internal RPC volume > 1000 req/sec sustained, or public gRPC contract requested by an enterprise integration — all Stage 2 / Stage 3 concerns. NATS remains the event bus regardless of whether internal RPC is HTTP+JSON or gRPC.
Why DigitalOcean, not Hetzner
Single-vendor billing (one primary invoice for Bashirat), managed data services from day 1 (Postgres Production with auto-failover + PITR, Redis with automated snapshots — removes real SRE burden), App Platform for git-push deploys on core + api + AI sidecar (frees the SRE contractor to focus on the dispatch droplet + observability). Cost delta vs a bare-minimum Hetzner 3-box: ~+$310/mo — accepted deliberately for HA on the DB layer, real observability + error tracking, and single-vendor operational simplicity. See Cost model for the full breakdown.
Data flow — origin scan example
- Operator captures origin scan video on the mobile app (offline-capable, encrypted at rest)
- On reconnect, mobile requests a presigned upload URL from
core.kayasync.com/v1/operator/scans/upload-url - Mobile uploads directly to R2 (presigned URL, 900 s validity)
- Mobile notifies core the upload is complete (
core.kayasync.com/v1/operator/scans/:id/submit) - Core publishes a
scan.uploadedevent to NATS - Scan Ingestion module (inside core) validates the upload, writes the scan record to Postgres
- Integrity AI module (inside core) subscribes and:
- Pulls the video from R2
- Calls
ai-sidecarover VPC internal HTTP for OCR + fingerprint - Writes the integrity result + reason codes to Postgres
- Publishes
scan.assessedevent
- Order/Container module subscribes → transitions container state per State Model
- Notification module subscribes → fans out SMS/push to originator + operator
- Every transition becomes a row in the audit/event ledger — see Acceptance Criteria §15
Total round-trip target: < 5 s under normal load, async beyond that — settlement is never blocked on upload latency.
Real-time location + dispatch (Uber-shaped hot path)
The dispatch loop is the highest-concurrency, lowest-latency path in the platform. It is intentionally isolated on the dispatch droplet (ws.kayasync.com) with its own language (Go) and its own scaling profile — separate from core.kayasync.com so Node runtime constraints never bottleneck real-time dispatch.
Location ingestion
Operator mobile app
│ WSS to ws.kayasync.com (JWT on handshake)
│ GPS heartbeat every 5-10 s
▼
Caddy (dispatch droplet) → location-svc (Go)
├── Validate heartbeat (auth, freshness, plausible-position check)
├── Redis: GEOADD op_locations lng lat op_id ← in-memory geo index
├── NATS: publish driver.location.updated ← for consumers
└── Timeout: if no GPS for 60 s → mark stale, remove from dispatch pool
Async consumer (Go):
▼
Postgres/TimescaleDB hypertable location_history ← columnar, partitioned by timeKey libraries: gorilla/websocket, go-redis/redis/v9, uber/h3-go, nats-io/nats.go.
Reliability patterns: heartbeat timeout + auto-reconnect + backpressure (drop oldest GPS on Redis pressure — location freshness beats history).
Order dispatch
Originator submits order:
- SMS/USSD via Africa's Talking → Notification module → core.kayasync.com/v1/internal/orders
- OR: enterprise via api.kayasync.com/v1/orders → service JWT → core.kayasync.com/v1/internal/orders
- OR: client portal via app.kayasync.com → core.kayasync.com/v1/client/orders
│
│ Order module writes to Postgres, publishes: NATS order.created
▼
dispatch-svc (Go, subscribes to order.created)
1. Redis GEORADIUS ← find operators within R km of pickup
2. Filter: trust tier, availability, vehicle type, corridor eligibility
3. Rank: distance (0.5) + rating (0.2) + trust tier (0.2) - recent-decline (0.1)
4. Take top-N (default N=5)
5. Parallel broadcast (goroutine per operator):
├── WSS "offer" frame via ws.kayasync.com to operator app
└── FCM push (fallback if WS not connected)
6. Race for accept:
├── First operator POSTs core.kayasync.com/v1/operator/orders/:id/accept
├── Core calls dispatch-svc /v1/internal/dispatch/lock via VPC
├── First writer wins via Redis SETNX order:{id}:winner
└── Losers receive "already assigned" WSS frame
7. If 30 s timeout, widen radius + retry
8. NATS publish: order.assigned → Order module updates state, Notification fans outRace safety: SETNX order:{id}:winner {op_id} EX 60 — first successful write is the assignment. Simpler than optimistic UI + rollback.
Geospatial sharding at scale: Every operator lives in an H3 resolution-6 hex cell (~36 km²). Dispatch queries the cell + 6 neighbours only. At Stage 3 we shard the dispatch-svc pods by H3 cell — one pod owns a cluster of cells.
Deployment isolation
Each API surface degrades independently:
core.kayasync.comdown: admin app can't dispatch overrides, client portal loses live tracking updates, but existing in-flight orders continue (dispatch-svc has state in memory + Redis, mobile operators keep receiving offers via WSS)api.kayasync.comdown: external enterprise / partner API calls fail — but internal ops are unaffected. Public API is deliberately isolated so external traffic can't take down internal ops.ws.kayasync.com/ dispatch droplet down: no new offers issued, no new GPS ingested, but core-served REST endpoints still work — CRUD, admin, invoicing, settlement queries continue. Operators fall back to FCM push for offers. Existing in-flight orders can still complete via REST.- Managed Postgres primary down: automatic failover to standby (Production tier) — typically <60s. During failover, both
coreanddispatch-svcserve cached reads only, writes queue in NATS JetStream for post-recovery replay. - Managed Redis down:
dispatch-svckeeps last-known-good GEO state in memory for its dispatch pool; new location writes are lost, but stale-check timers prevent stale operators being offered.
No single tenant of the platform (admin, operator, client, public) can take down another tenant's surface. This is the whole point of the multi-domain split.
Pages in this section
- Hosting & Data Plan — full cost rationale, capability mapping, phasing
- Port Allocation — port map for when services split out
- Service boundaries & APIs — to be added per service as designs are completed
- Data model & database schema — to be added
- Authentication & access control — to be added