System design: multi-user whiteboard

As tech leaders, we're being asked to be very good at system designs, scalability. This is actually part of most interviews at companies hiring leadership positions. While this is everyday work, we are usually stuck in our own scope's problems, and system design interviews might feel challenging because they make us think about issues we are mostly not working on. I also face the same issue. I want to force myself to work on such challenges, and what better way than making it public for accountability.

For resources, I was recommended this YouTube channel and the System Design Primer.

Problem

We want to design a solution similar to r/place, which is a whiteboard that allows multiple users to connect simultaneously to that board and change the color of a pixel in that board.

Here are the restrictions:

Initial thinking

The goal is to design a pragmatic solution, not introducing unnecessary systems early. There is no account management, we only have to look into the number of users connected to the system and making changes. The scalability of the system will be a thought after the basic system design is made.

I will start with a basic user connection for the first time to the board. I won't necessarily think about the other users yet. I will just focus on 1 user making changes to a pixel, and the lock of the user. So an early system drawing would look like:

flowchart LR
    subgraph browser [Browser]
        frontend[User 1]
    end

    subgraph api [API]
        server[Server]
        db[(Database)]
    end

    frontend -->|Load whiteboard| server
    server -->|Return initial state| frontend
    server --- db

In this graph, User 1 connects to the board, it gets the initial color schema, coordinate-based, from the server.

The server gets that information from a data storage. It will have to keep a connection ID to know who is connected, a stable userID to recognise users across reloads (the connection ID changes), and a board state with a coordinate => color mapping.

Now we can already make some small assumptions on that basic design. What DB are we talking about?

So far the data structure is quite simple, write and read heavy. A simple key/value DB should be good enough for that, something like redis or dynamoDB would work perfectly for fast lookup.

Communication between Browser and server should be using websocket as it will keep alive a connection to a user, allow both direction communication - especially important for the "notified as soon as possible" part. Other solutions would have to regularly poll the backend and would always have a delay.

Here is the initial thinking basic system:

flowchart LR
    subgraph browser [Browser]
        frontend[User 1]
    end

    subgraph api [API]
        server[Server]
        userdb[(User DB)]
        boardState[(BoardState)]
        lockUser[(Lock user with TTL)]
    end

    frontend -->|Websocket| server
    server --- boardState
    server --- lockUser
    server --- userdb

Concerning the schema, to have a simple list of connected users, we would have a user management DB with userID (generated and stored in a cookie), and connectionID from the websocket.

Concerning the board state DB, here we need to get the color of each pixel of the board. So a simple coordinates {x,y} and color.

For the locking part, we will be using the Redis built-in TTL (time to live). We check the lock for a specific userID: if there is none, we allow the change; if there is an existing lock, the change is discarded and notified to the user. A simple userID and TTL works.

User table

User IDConnection ID
uuidstring

Board state table

CoordinateColor
{x, y}#123456

Lock table

User IDTTL
uuid300s

We have completely ignored the notification part and any scalability and optimization.

Here is the user flow, single user:

sequenceDiagram
    autonumber
    box Browser
        participant frontend as User 1
    end
    box API
        participant server as Server
        participant lockdb as Lock DB
        participant userdb as User DB
        participant boardState as BoardState
    end

    rect rgb(235, 245, 255)
        Note over frontend,boardState: Step 1: Initial Connection
        frontend->>server: Connect
        server->>boardState: Get board state
        server->>userdb: Add user connection
        server-->>frontend: Return board state
    end

    rect rgb(235, 255, 240)
        Note over frontend,boardState: Step 2: Pixel Update
        frontend->>server: Change pixel color
        server->>lockdb: Is user locked?
        alt user not locked
            server->>boardState: Update board color
            server->>lockdb: Lock user for 5 min
            server-->>frontend: Change accepted
        else user locked
            server-->>frontend: Change rejected (retry in N seconds)
        end
    end

One question this flow raises: what happens if the same user sends two changes at the exact same time (two tabs, or a script)? "Check the lock" then "set the lock" are two separate steps, so both requests could pass the check before either one sets the lock, and the user would place two pixels.

The fix is to check and set in one atomic operation. In Redis, that is:

SET lock:{userID} 1 NX EX 300

Redis runs this as one single operation, so only the first request gets OK back and is allowed to change the pixel. Every other request gets nothing back and is rejected, until the key expires.

Multiple users

The current design works fine on a single user. If we add multiple users, they can all connect to the webserver, they get registered and can perform pixel modification. But what is missing is the notification system. How to show changes to other users.

So for each action performed, we should trigger a notification to every other user on the board with the change, the frontend will manage the change of color.

The change needs to go into a pubsub topic, and workers can pull the changes and notify the other users of the new color of the pixel. There are two ways to send the change into the topic:

Technically these workers would live in each of the websocket server to use the opened connections to send the notification.

flowchart LR
    subgraph browser [Browser]
        frontend[User 1]
        frontend2[User 2]
        frontend3[User 3]
    end

    subgraph api [API]
        subgraph wsServer [Websocket server]
            server[Server]
            worker[Notification Worker]
        end
        userdb[(User DB)]
        boardState[(BoardState)]
        lockUser[(Lock user with TTL)]
    end

    subgraph notification [Notification]
        pubsub[PubSub Topic]
    end

    frontend <-->|Websocket| wsServer
    frontend2 <-->|Websocket| wsServer
    frontend3 <-->|Websocket| wsServer
    server --- boardState
    server --- lockUser
    server --- userdb

    boardState -->|DB trigger| pubsub
    server -.->|or publish directly| pubsub
    pubsub -->|subscribe| worker

Optimization & scalability

Now the system is working as expected, the basic requirements are all set:

Latency(per change) = Latency(lock) + Latency(change logged) + Latency(pubsub worker)

We are talking about a few milliseconds.

Websocket connections

In terms of scalability, how to handle that architecture at scale? Most of it is already scalable, we can scale the websocket via websocket load balancer to separate user connection and notification across servers.

Initial load of the board

The current concern is about the initial load. When a user connects for the first time, they need to get the current state of the board.

flowchart LR
    subgraph browser [Browser]
        frontend[User 1]
    end

    subgraph api [API]
        server[Server]
        db[(Database)]
    end

    frontend -->|Connect| server
    server -->|Get Initial state| frontend
    server --- db

With the current architecture, each user connecting to the board would require a full scan of the board as initial load. This might not be efficient, especially if it needs to happen in a very short amount of time.

A first solution to explore is a regular snapshot of the board, triggered every few minutes. The initial load would then get the last snapshot + the last few diffs that happened since the last snapshot.

There is a different solution which could be working more efficiently. Currently the design stores one coordinates => color key per pixel, which means one lookup per pixel and a lot of overhead per key. The board has a fixed size, so we don't need keys at all: a single dense array where the position gives the coordinates would do. Even with full 24-bit colors, that is only 1000 x 1000 x 3 bytes = 3 MB. And if we decide to allow only a subset of colors for each pixel, we can shrink it even more into a bitmap, specifically a Redis BITFIELD. For a 1000x1000 board, a 16-color palette would allow storing each pixel in 4 bits, which would result in a size of:

1000 x 1000 x 4 = 4 million bits / 8 = 500 KB

Then we can maintain at all times the state of the whiteboard into a single bitmap array which only stores the colors: the coordinates are implied by the position in the array (offset = (y × 1000 + x) × 4 bits). The changes would be much more efficient, the initial load and notification would also be much faster.

Frontend to Backend interaction, data transfer optimization

Finally, another potential question might be about the data to transfer to frontend. My initial thought was running with a classic JSON, sending only diffs to the frontend instead of full state for each modification.

But we can optimize the communication for low data in case of mobile devices trying to access with low data. Here probably the bitmap array combined with protobuf messages over the websocket would reduce drastically the data needed for the frontend to work correctly.

We can also batch change notifications instead of sending every single change. Batching notifications every 100–250 ms would allow multiple changes on the same pixel to trigger only one notification per batch, while staying close enough to "as soon as possible" that users won't notice the delay.

Conclusion

This challenge felt easy, but I remember that when I did it, I fell short on the details and scalability. I had the right instincts, but the details were missing.

Nowadays with AI, knowing by heart what Redis or DynamoDB can do might not matter as much. What matters is understanding what your choices are (DB trigger, key/value databases, etc.) and being able to pick the right technology for the outcome you want.

I remember suggesting a relational database for cold storage, storing every action. Keeping a history of every change isn't a bad idea in itself (replays, moderation, rebuilding the board after a crash), but nobody asked for it, and I didn't tie it to any requirement. That was not really pragmatic of me, and it played against me, as I had stated at the beginning that I am more of a pragmatic engineer. Spending the time now, digging into what the right answers would have been, helps me figure out where I fell short.