Designing a Notification Service

Like explaining to a 5-year-old: Imagine your school wants to tell every student about a snow day. Instead of the principal personally calling 500 students, they send one message to a helper (the notification service). The helper splits the list — some students get phone calls, some get texts, some get emails — and makes sure everyone gets the message, even if some phones are off or some emails bounce. If a call fails, the helper tries again later. That’s a notification service.

Every major app needs to tell users things: “Your order shipped.” “You have a new follower.” “Your meeting starts in 10 minutes.” At scale, this becomes a distributed systems problem. Sending 100M push notifications during a Black Friday sale requires queuing, batching, retry logic, and careful ordering — or your CDN will throttle you and half your users won’t get notified.

Requirements

Functional

• Send notifications via: push (iOS/Android), email, SMS, in-app
• Support immediate and scheduled notifications
• Respect user preferences (channels, time zones, do-not-disturb)
• Track delivery status: sent → delivered → read
• Support notification templates with variable substitution

Non-Functional

• Throughput: 10M notifications/hour at peak
• Latency: push notifications delivered within 5 seconds of trigger
• Reliability: at-least-once delivery (no dropped notifications)
• Idempotency: duplicate triggers must not send duplicate notifications

High-Level Architecture

Trigger Sources                    Notification Service
──────────────                     ────────────────────
Order Service ──▶                  ┌───────────────────┐
Auth Service ───▶  Kafka topic  ──▶│  Notification     │
Marketing ──────▶  (notif-events)  │  Orchestrator     │
                                   └────────┬──────────┘
                                            │ route by channel
                     ┌──────────────────────┼──────────────────────┐
                     ▼                      ▼                      ▼
             ┌──────────────┐    ┌──────────────────┐    ┌──────────────┐
             │  Push Queue  │    │   Email Queue    │    │  SMS Queue   │
             └──────┬───────┘    └────────┬─────────┘    └──────┬───────┘
                    │                     │                      │
                    ▼                     ▼                      ▼
             ┌──────────────┐    ┌──────────────────┐    ┌──────────────┐
             │   APNs/FCM   │    │  SendGrid/SES    │    │ Twilio/SNS   │
             │   Workers    │    │  Workers         │    │ Workers      │
             └──────────────┘    └──────────────────┘    └──────────────┘

Data Model

-- Notification templates
CREATE TABLE notif_templates (
  template_id   UUID PRIMARY KEY,
  name          VARCHAR(100) UNIQUE,
  channel       VARCHAR(20),  -- push|email|sms
  subject       TEXT,         -- email subject / push title
  body_template TEXT          -- "Hello {{name}}, your order {{order_id}} shipped."
);

-- User device tokens
CREATE TABLE user_devices (
  user_id       BIGINT NOT NULL,
  device_token  TEXT NOT NULL,   -- APNs/FCM token
  platform      VARCHAR(10),     -- ios|android
  last_active   TIMESTAMPTZ,
  PRIMARY KEY (user_id, device_token)
);

-- User preferences
CREATE TABLE user_notif_prefs (
  user_id       BIGINT PRIMARY KEY,
  push_enabled  BOOLEAN DEFAULT true,
  email_enabled BOOLEAN DEFAULT true,
  sms_enabled   BOOLEAN DEFAULT false,
  quiet_start   TIME,   -- 22:00 -- don't disturb between quiet_start
  quiet_end     TIME,   -- 08:00 -- and quiet_end in user's timezone
  timezone      VARCHAR(50) DEFAULT 'UTC'
);

-- Notification log (for deduplication and tracking)
CREATE TABLE notif_log (
  notif_id      UUID PRIMARY KEY,
  user_id       BIGINT NOT NULL,
  template_id   UUID,
  channel       VARCHAR(20),
  status        VARCHAR(20),  -- queued|sent|delivered|failed
  sent_at       TIMESTAMPTZ,
  idempotency_key VARCHAR(100) UNIQUE  -- prevents duplicate sends
);

Idempotency: The Critical Detail

The postal worker analogy: If a postal worker accidentally drops the same package twice, they don’t deliver it twice — they check the tracking number and realize they already delivered it. Your notification service needs the same “tracking number” check.

Without idempotency, a Kafka consumer restart or a retry after a timeout can send the same notification twice. Users hate double-notifications.

def send_notification(event: NotifEvent):
    # Idempotency key = deterministic hash of what this notification IS
    idem_key = sha256(f"{event.trigger_type}:{event.user_id}:{event.entity_id}").hexdigest()

    # Atomic check-and-insert
    result = db.execute("""
        INSERT INTO notif_log (notif_id, user_id, idempotency_key, status)
        VALUES (gen_random_uuid(), %s, %s, 'queued')
        ON CONFLICT (idempotency_key) DO NOTHING
        RETURNING notif_id
    """, [event.user_id, idem_key])

    if result.rowcount == 0:
        # Already sent or in-progress — skip
        return

    # Proceed with actual delivery
    deliver(event, notif_id=result.fetchone()["notif_id"])

User Preference Checking

Before sending, always check:

1. Is this channel enabled for the user?
2. Is the user in do-not-disturb hours?
3. Are they rate-limited? (max N notifications per hour per user)

def should_send(user_id: int, channel: str, notif_type: str) -> bool:
    prefs = get_user_prefs(user_id)

    if channel == "push" and not prefs.push_enabled:
        return False

    if channel == "sms" and not prefs.sms_enabled:
        return False

    # Check quiet hours in user's local timezone
    user_now = datetime.now(tz=pytz.timezone(prefs.timezone))
    if is_quiet_hours(user_now.time(), prefs.quiet_start, prefs.quiet_end):
        if notif_type != "critical":  # critical alerts bypass quiet hours
            schedule_for_morning(user_id, channel, notif_type)
            return False

    # Rate limiting: max 20 push per hour
    if count_recent_notifs(user_id, channel, last_hours=1) >= 20:
        return False

    return True

Push Notification Delivery

Mobile push goes through Apple (APNs) or Google (FCM). You don’t push directly to the phone — you push to Apple/Google’s servers, which hold the delivery until the phone is online.

async def send_push(user_id: int, title: str, body: str):
    devices = get_user_devices(user_id)

    for device in devices:
        if device.platform == "ios":
            response = await apns_client.send(
                token=device.device_token,
                notification={"title": title, "body": body},
                push_type="alert"
            )
        else:  # android
            response = await fcm_client.send(
                token=device.device_token,
                notification={"title": title, "body": body}
            )

        if response.status == "InvalidToken":
            # Token expired — user uninstalled or re-installed app
            db.execute("DELETE FROM user_devices WHERE device_token = %s",
                      [device.device_token])

Token management is the hidden complexity. App tokens expire when users reinstall, change phones, or revoke permissions. A token graveyard (sending to millions of invalid tokens) wastes quota and causes APNs rate limiting.

Fan-out for Marketing Campaigns

The megaphone analogy: Instead of whispering to each person individually, you use a megaphone. But a megaphone can only shout at 1,000 people at once. So if you have 10M subscribers, you need 10,000 megaphone blasts. That’s how bulk campaigns work — batch processing through a queue.

Sending to 10M users for a marketing campaign:

def launch_campaign(campaign_id: str, segment_query: str, template_id: str):
    # Don't load all 10M user IDs into memory
    # Stream in batches and enqueue
    cursor = db.stream_query(segment_query, batch_size=10_000)

    for batch in cursor:
        # Enqueue each batch as a single Kafka message
        kafka.produce("campaign-fanout", {
            "campaign_id": campaign_id,
            "user_ids": [row["user_id"] for row in batch],
            "template_id": template_id,
            "scheduled_at": campaign.send_time
        })

# Workers process batches, check preferences, send
def process_campaign_batch(batch):
    for user_id in batch["user_ids"]:
        if should_send(user_id, channel="push", notif_type="marketing"):
            enqueue_push(user_id, batch["template_id"])

Delivery Status Tracking

APNs and FCM return delivery receipts. Update the log:

APNs response: {"apns-id": "abc", "reason": null}  → "delivered"
APNs response: {"reason": "DeviceTokenNotForTopic"}  → "failed", delete token
FCM response: {"success": 1}                        → "delivered"
FCM response: {"error": "NotRegistered"}             → "failed", delete token

For email, use webhooks from SendGrid/SES: delivered, opened, bounced, spam_complaint.

Retry Strategy

Failed notifications should retry with exponential backoff, but not infinitely:

RETRY_DELAYS = [30, 120, 600, 3600, 86400]  # 30s, 2m, 10m, 1h, 24h

def handle_failure(notif_id: str, attempt: int, error: str):
    if attempt >= len(RETRY_DELAYS):
        mark_failed(notif_id, "max_retries_exceeded")
        alert_oncall_if_critical(notif_id)
        return

    delay = RETRY_DELAYS[attempt]
    schedule_retry(notif_id, delay_seconds=delay, next_attempt=attempt + 1)

Transient failures (APNs timeout) → retry. Permanent failures (invalid token, unsubscribed) → don’t retry, clean up.

Trade-offs & Decisions

Decision	Choice	Reason
Delivery guarantee	At-least-once	Better to receive twice than never; idempotency handles duplicates
Idempotency	DB unique constraint on key	Atomic; survives service restarts
Campaign fan-out	Streaming batches via Kafka	Avoid OOM; natural backpressure
Token cleanup	On first 404/InvalidToken response	Immediate cleanup prevents quota waste
Quiet hours	Schedule for morning, not drop	Better UX than silently dropping notifications

---

A notification service is one of those systems that looks simple until you think about failure modes: what if APNs is down? What if a user has 12 devices? What if the same Kafka message is processed twice? Idempotency and at-least-once delivery are the two ideas that make everything else tractable.