Chat And Messaging Service
IntermediateOverview
A chat service delivers messages between users in near real time, supporting one-to-one and group conversations with delivery and read receipts. The central challenge is maintaining persistent, low-latency connections for millions of concurrent users while guaranteeing ordered, durable message delivery.
Functional Requirements
- Send and receive one-to-one messages in near real time.
- Support group conversations with multiple participants.
- Show presence (online/offline) and delivery/read receipts.
- Persist message history and deliver messages queued while a recipient was offline.
Non-Functional Requirements
- Low latency: end-to-end message delivery under ~500 ms for online users.
- High availability and durability: an accepted message is never lost.
- Ordered delivery within a conversation.
- Scale to hundreds of millions of concurrent long-lived connections.
Capacity Estimation
Assume 50M concurrent users and 40 messages sent per user per day across 500M DAU.
- QPS (Messages): 500M x 40 = 20B/day -> 20B / 86,400 s ~ 230,000 messages/sec (peak ~3x ~ 700,000/sec).
- Connections: 50M concurrent WebSocket connections; at ~50–100k connections per gateway node, that is 500–1,000 gateway nodes.
- Storage: a message ~ 300 bytes (sender, recipient, body, timestamps, status). Per day: 20B x 300 B = 6 TB/day -> ~2.2 PB/year, so retention tiers and archival are required.
- Bandwidth: ingress 230,000/s x 300 B ~ 69 MB/s; egress is higher for group messages (fan-out multiplies per recipient).
High-Level Architecture
The core architecture uses stateful connection gateways holding WebSocket connections to clients. When User A sends a message, it hits Gateway A, is persisted to a database, and passed to a message broker. A Session Registry is queried to find out which gateway User B is connected to. The broker routes the message to Gateway B, which pushes it down User B's WebSocket.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| message | conversation_id (partition), message_id (sort, time-ordered), sender_id, body, status | Wide-column store partitioned by conversation_id, clustered by time (e.g. Cassandra or ScyllaDB) |
| conversation | conversation_id (PK), participant_ids, type (1:1/group), last_message_at | Wide-column / document store |
| user_session | user_id, gateway_id, connected_at | In-memory registry (fast lookup, TTL) like Redis |
| inbox | user_id, undelivered message_ids | Durable queue per offline user |
Detailed Design
Connection Layer
Clients hold a persistent connection to a stateless Connection Gateway. WebSockets are the primary transport because they are full-duplex, letting the server push messages without client polling; this is the core client-server communication choice for the system. A Session Registry (Redis) maps each user_id to the gateway node currently holding its connection so a message can be routed to the right node.
Message Send Flow
When user A sends to user B: the gateway persists the message (durability first), then looks up B's gateway in the session registry and forwards the message through the broker to that gateway, which pushes it down B's socket. If B is offline, the message lands in B's durable inbox and a push notification is triggered; B drains the inbox on reconnect. Delivery/read receipts flow back on the same path as small control messages.
Message Ordering
To ensure strict ordering, the system generates a monotonic, time-sortable message_id (like Snowflake or UUIDv7) at the point of origin. The client UI relies on this ID to order the conversation history regardless of network delivery delays.
Bottlenecks & Solutions
The Session Registry becomes a major bottleneck since every single message requires a read to find the recipient's gateway. Caching gateway locations locally reduces this but introduces staleness on reconnects. Group chat fan-out is another bottleneck: a message to a 10,000-person group requires 10,000 registry lookups and 10,000 pushes, necessitating a dedicated fan-out service that reads from Kafka rather than doing it synchronously in the gateway.