Designing a News Feed (Twitter/Instagram)

Like explaining to a 5-year-old: Imagine you follow 20 friends on Instagram. When you open the app, you don’t want to go knock on each friend’s door to check what they posted today. Instead, every morning, someone collects all the new photos from everyone you follow and puts them in a pile on your doorstep, sorted by what you’d like most. That delivery person is the news feed system.

A news feed feels simple — scroll, see posts. Behind the scenes it’s one of the most complex distributed systems problems in existence. You’re personalizing content in milliseconds for hundreds of millions of users, each following hundreds of people, some of whom have millions of followers themselves.

Requirements

Functional

• See posts from people you follow, in reverse-chronological or ranked order
• Feed loads in < 300ms
• Supports text, images, videos, and reshares
• Infinite scroll with pagination
• Feeds are eventually consistent (a new post may take seconds to appear)

Non-Functional

• Scale: 300M DAU, each user follows ~200 accounts on average
• Write throughput: ~10K posts/second at peak
• Read throughput: ~1M feed loads/second
• Freshness: new posts visible within 5 seconds for most users

The Core Challenge: Fan-out

The newspaper analogy: One journalist writes an article. It gets printed in 1 million copies. If you wait until each reader asks for the newspaper, you print on demand — slow. If you print and deliver before they ask, they get it instantly — but you need warehouses everywhere.

When User A (with 5M followers) posts, the system must somehow make that post appear in 5M feeds. There are two strategies:

Fan-out on Write (Push Model)

When A posts, immediately write to the feed cache of all 5M followers.

Post created
    │
    ▼
Feed Fanout Worker
    │
    ├──▶ Write to feed:{follower_1} in Redis
    ├──▶ Write to feed:{follower_2} in Redis
    ├──▶ ...
    └──▶ Write to feed:{follower_5M} in Redis  ← takes seconds for celebrities

Pros: feed reads are instant — just read a pre-built list.
Cons: “celebrity problem” — one post by a user with 50M followers means 50M Redis writes.

Fan-out on Read (Pull Model)

When user loads feed, query the last N posts from everyone they follow.

SELECT * FROM posts
WHERE author_id IN (SELECT followee_id FROM follows WHERE follower_id = ?)
ORDER BY created_at DESC
LIMIT 20;

Pros: writes are instant; no fan-out delay.
Cons: reads are expensive — joining across potentially 1,000 accounts’ post tables.

Hybrid Approach (What Twitter/Instagram Actually Do)

User type	Strategy	Reason
Regular users (<10K followers)	Fan-out on write	Fast read; manageable write load
Celebrities (>10K followers)	Fan-out on read	A write fan-out to 50M users is too slow

On feed load:

1. Fetch pre-built feed from Redis (covers regular users you follow)
2. Merge in recent posts from celebrities you follow (pull in real-time)
3. Rank and return top 20

Data Model

-- Posts table (sharded by user_id)
CREATE TABLE posts (
  post_id    BIGINT PRIMARY KEY,   -- Snowflake ID (time-sortable)
  user_id    BIGINT NOT NULL,
  body       TEXT,
  media_urls JSONB,
  created_at TIMESTAMPTZ NOT NULL,
  like_count INT DEFAULT 0,
  repost_count INT DEFAULT 0
);

-- Follows table (sharded by follower_id)
CREATE TABLE follows (
  follower_id  BIGINT NOT NULL,
  followee_id  BIGINT NOT NULL,
  followed_at  TIMESTAMPTZ NOT NULL,
  PRIMARY KEY (follower_id, followee_id)
);

-- Feed cache in Redis (pre-built per user)
-- Key: feed:{user_id}
-- Value: sorted set of post_ids, score = timestamp
ZADD feed:{user_id} {timestamp} {post_id}
ZREMRANGEBYRANK feed:{user_id} 0 -501  -- keep top 500 only

Why Snowflake IDs? Post IDs encode time, so you can sort by ID without a created_at index. Twitter created this; Meta uses a similar system.

System Architecture

           Client
              │ HTTP/2
              ▼
        API Gateway
              │
     ┌────────▼─────────┐
     │   Feed Service   │
     └────────┬─────────┘
              │
     ┌────────▼─────────┐       ┌─────────────┐
     │   Feed Cache     │       │  Post Store  │
     │   (Redis)        │       │  (Cassandra) │
     └──────────────────┘       └─────────────┘
              │
     ┌────────▼─────────┐
     │  Fanout Service  │◀── Kafka (post_created events)
     └──────────────────┘
              │
     ┌────────▼──────────────────┐
     │  Graph Service            │
     │  (follower/following data)│
     └───────────────────────────┘

Feed Generation Pipeline

1. User A creates post
   → Saved to Post Store (Cassandra)
   → post_created event published to Kafka

2. Fanout Service consumes event
   → Fetches follower list from Graph Service (paginated, 1K at a time)
   → Filters out celebrities (>10K followers)
   → Batch-writes post_id to Redis feed lists for all followers
   → Rate-limits: use exponential backoff for large follower counts

3. Celebrity posts
   → Not written to follower feeds
   → Stored in celebrity_post_ids:{celebrity_id} sorted set

4. Feed read request
   → Fetch top-500 post IDs from Redis feed:{user_id}
   → Fetch recent posts from celebrities the user follows (from celebrity_post_ids sets)
   → Merge and sort both lists by timestamp
   → Batch-fetch post objects from Cassandra (or Redis post cache)
   → Apply ranking model
   → Return top 20

Ranking

Raw reverse-chronological feeds are easy. Ranked feeds (Instagram, Facebook) are harder.

Simple ranking score:

def score(post, viewer):
    recency    = decay(post.created_at)          # older posts score lower
    affinity   = edge_weight(viewer, post.author) # how often you interact
    content_q  = post.engagement_rate             # likes/views ratio
    return recency * 0.4 + affinity * 0.4 + content_q * 0.2

Production systems use a full ML model (Facebook’s ranking model has billions of parameters). For the interview, a simple weighted score formula is sufficient.

Pagination

Use cursor-based pagination, not offset:

# Bad: SELECT ... LIMIT 20 OFFSET 1000
# (database scans 1020 rows, discards first 1000)

# Good: cursor = last_seen_post_id
GET /api/feed?cursor=post_id_abc&limit=20
→ SELECT * FROM posts WHERE post_id < cursor ORDER BY post_id DESC LIMIT 20

Cursor-based is O(log n) vs offset’s O(n) scan.

Edge Cases

Feed gaps for offline users: If you’re offline for a week, your 500-item Redis cache is stale. On next open, hydrate with a full pull from the last week’s posts from accounts you follow.

Deleted posts: Redis feed lists contain post IDs. When a post is deleted, you can’t update every user’s cache. Instead, fetch the post object when rendering — if it returns 404, skip it in the feed.

New user / empty feed: Bootstrap by showing trending posts, mutual friends’ posts, or posts from suggested accounts.

Trade-offs & Decisions

Decision	Choice	Reason
Fan-out strategy	Hybrid push/pull	Pure push fails for celebrities; pure pull is too slow
Feed storage	Redis sorted sets	O(log n) reads; TTL for inactive users
Post IDs	Snowflake (time-sortable)	Sort without timestamp index; globally unique
Post storage	Cassandra	Write-optimized; partition by user; fast range queries
Ranking	ML model on pre-fetched candidates	Can’t rank all posts; narrow to candidates first, then rank

---

The news feed problem is where every distributed systems concept collides: fan-out, caching, eventual consistency, ranking, and pagination all interact. The hybrid push/pull architecture is the practical answer — not perfect for everyone, but correct for almost everyone almost all the time.