Designing a Chat System

Like explaining to a 5-year-old: Imagine you and your friend want to pass notes in class. But instead of walking over every time, you each hold one end of a string with a cup attached. When your friend shakes the string, you feel it immediately. That’s like a WebSocket — a permanent, two-way string between you and the server, so messages arrive the instant they’re sent.

Building a chat system means solving two fundamentally different problems: real-time delivery (messages must arrive instantly) and persistence (you must be able to scroll back through history from a new device). Most of the complexity lives in the gap between these two requirements.

Requirements

Functional

1-to-1 and group messaging (up to 1,000 members per group)
Message delivery with read receipts (sent → delivered → read)
Online/offline presence indicators
Message history with pagination
Media messages (images, files — up to 100MB)

Non-Functional

Latency: messages delivered in < 100ms on the same continent
Scale: 500M daily active users, 100B messages/day
Durability: messages stored for at least 5 years
Availability: 99.99% uptime

Capacity Estimation

Metric	Value
Messages/day	100B
Messages/second (peak)	~2M
Avg message size	200 bytes
Storage/day	~20 TB
Storage/year	~7 PB
Active WebSocket connections (peak)	~500M

500M concurrent WebSocket connections is the hardest number here. A single server holds ~65K TCP connections — you need ~8,000 chat gateway servers just for connection management.

High-Level Architecture

         Client A                Client B
            │                      │
            │ WebSocket            │ WebSocket
            ▼                      ▼
    ┌───────────────┐      ┌───────────────┐
    │  Chat Server 1│      │  Chat Server 2│  ← millions of these
    └───────┬───────┘      └───────┬───────┘
            │                      │
            └──────────┬───────────┘
                       │ pub/sub
                  ┌────▼────┐
                  │  Kafka  │  ← message bus
                  └────┬────┘
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
    ┌──────────┐ ┌──────────┐ ┌──────────┐
    │ Message  │ │ Presence │ │  Push    │
    │ Storage  │ │ Service  │ │  Service │
    └──────────┘ └──────────┘ └──────────┘

Real-Time Delivery: WebSocket Deep Dive

HTTP is request-response: the client asks, the server answers. For chat, the server needs to push messages to the client unprompted. Options:

Technique	How it works	Downside
Short polling	Client asks “any messages?” every 2s	Wastes bandwidth, high latency
Long polling	Server holds request open until a message arrives	Still half-duplex; complex timeouts
WebSocket	Full-duplex TCP connection, both sides push at will	Requires stateful connection management
Server-Sent Events	Server pushes, client can’t send	One-way only

WebSocket is the right choice for chat. The connection lifecycle:

Client → Server: HTTP Upgrade request
Server → Client: 101 Switching Protocols
--- now both ends can send frames at any time ---
Client → Server: { "type": "message", "to": "user_b", "body": "hey!" }
Server → Client: { "type": "message", "from": "user_a", "body": "hey!", "id": "msg_abc" }
Server → Client: { "type": "delivered", "msg_id": "msg_abc" }

Message Routing Problem

The post office analogy: Chat server 1 has User A connected. Chat server 2 has User B connected. When A sends to B, server 1 doesn’t have B’s connection — it’s like two post offices in different cities. They need a shared message board (Kafka) to relay messages.

User A (on Server 1) sends to User B (on Server 2). Server 1 doesn’t have B’s connection. Solution:

Server 1 receives message from A
Publishes to Kafka topic user-{B_id}
Server 2 subscribes to user-{B_id} (because B is connected there)
Server 2 delivers to B’s WebSocket

A service registry (e.g. Redis hash) maps user_id → chat_server_id so Server 1 knows which Kafka partition to target.

Message Storage

Chat history has a very specific read pattern: “give me the last 50 messages in conversation X, then the 50 before that.” This is a time-series access pattern — perfect for Cassandra.

-- Cassandra table design
CREATE TABLE messages (
  conversation_id UUID,
  created_at      TIMEUUID,  -- TimeUUID = timestamp + uniquifier
  message_id      UUID,
  sender_id       UUID,
  body            TEXT,
  media_url       TEXT,
  PRIMARY KEY (conversation_id, created_at)
) WITH CLUSTERING ORDER BY (created_at DESC);

Why Cassandra?

Partition key = conversation_id: all messages for a chat are co-located on one node
Clustering key = created_at: messages stored in time order, range queries are fast
Wide rows: a single partition can hold millions of messages efficiently
Linear write scalability: chat is write-heavy, Cassandra excels at this

Delivery Receipts

Three states: sent (stored on server), delivered (reached recipient’s device), read (user opened the conversation).

Sender                  Server                  Recipient
   │── send msg ──────▶│                            │
   │◀── ack (stored) ──│── push via WS ────────────▶│
   │                   │◀── delivered ack ───────────│
   │◀── delivered ─────│                            │
   │                   │         [user opens chat]  │
   │                   │◀── read receipt ────────────│
   │◀── read ──────────│                            │

Receipts are stored as a bitmask or separate status row — don’t update the original message row on every read (that creates hot write paths at scale).

Presence Service

The classroom attendance analogy: Every student who arrives signs a register. When they leave, they sign out. Anyone can glance at the register to see who’s in school. That’s a presence service — a shared register of who’s online.

Online/offline presence is inherently ephemeral. Store it in Redis with TTL:

SETEX presence:{user_id} 30 "online"  # expires in 30s

Each client sends a heartbeat every 20 seconds. If the server stops receiving heartbeats, the key expires and the user appears offline. On explicit disconnect, delete the key immediately.

Fan-out problem: if a popular user (1M followers) comes online, you can’t push a notification to all followers synchronously. Solution: only push presence to users with an open chat window with that person, not to all followers.

Offline Message Delivery

When User B is offline:

Message stored in Cassandra (durable)
Push notification sent via APNs (iOS) / FCM (Android)
When B comes back online, their client requests unread messages since last sync

GET /api/messages/sync?since=1718200800&limit=100
→ { messages: [...], cursor: "next_page_token" }

Media Handling

Never send media through the chat server. Use pre-signed URLs:

1. Client → Upload Service: "I want to send a 5MB image"
2. Upload Service → Client: pre-signed S3 URL (expires in 15min)
3. Client → S3: upload image directly
4. Client → Chat Server: message with media_url pointing to S3
5. Recipients download from CDN (CloudFront in front of S3)

This keeps the chat server lean and offloads bandwidth to S3/CDN.

Trade-offs & Decisions

Decision	Choice	Reason
Real-time transport	WebSocket	Full-duplex; best latency for chat
Message bus	Kafka	Durable, replayable, partitioned by user
Message storage	Cassandra	Write-optimized; partition-by-conversation is natural
Presence storage	Redis (TTL)	Ephemeral data; fast reads for online status
Media	S3 + CDN	Chat servers shouldn’t handle binary payloads
Delivery guarantees	At-least-once	Idempotent message IDs prevent duplicates on retry

The hardest part of building chat at scale isn’t the happy path — it’s the edge cases: what happens when a user is on a flaky connection? When a chat server crashes mid-delivery? When a group has 500 members and someone sends a message? Each of these forces you to think carefully about delivery guarantees and fan-out amplification.