Work
Platform Reliability Infrastructure
Durable, auditable infrastructure for event delivery and tenant lifecycle management, both auto-recoverable and fully auditable.
Brief
The platform needed durable, auditable infrastructure for two different problems: delivering events to external systems reliably, and safely provisioning, upgrading, and retiring tenants without manual intervention.
Approach
- Webhooks Service: Built an instant-delivery service backed by caching, automatic retries, and a dead-letter queue. The service ran behind mTLS with dedicated ingress and distributed locks.
- Partner Routing: Extended routing to external CRM and messaging systems, using a soft-delete pattern to keep audit records queryable beyond the webhook’s lifecycle.
- Tenant Workflows: Built stateful workflows for tenant provisioning and upgrades, handling distributed database schema migrations via Liquibase.
- Disaster Recovery: Managed cross-region replication (including encryption keys) across more than 30 regions and 10 environments.
- Reliability: Made both systems auto-retriable, crash-recoverable, fully auditable, and heavily alerted.
Result
- Removed manual intervention from tenant lifecycle operations across 30+ regions and 10 environments.
- Maintained a flawless record of zero failed schema migrations.
- Kept a full audit trail and stopped wasted retries against dead endpoints.
- Held P99 response times under 200 milliseconds even as the system scaled.