System design: multi-user whiteboard
As tech leaders, we're being asked to be very good at system designs, scalability. This is actually part of most interviews at companies hiring leadership positions. While this is everyday work, we are usually stuck in our own scope's problems, and system design interviews might feel challenging because they make us think about issues we are mostly not working on. I also face the same issue. I want to force myself to work on such challenges, and what better way than making it public for accountability.
For resources, I was recommended this YouTube channel and the System Design Primer.
Problem
We want to design a solution similar to r/place, which is a whiteboard that allows multiple users to connect simultaneously to that board and change the color of a pixel in that board.
Here are the restrictions:
- Up to 1000 users
- The challenge considers we only have access to 1 board. No multi-board system.
- Each user can only change 1 pixel every 5min
- Changes need to be carried to all connected users as soon as possible
- Board size: 1000×1000px
Initial thinking
The goal is to design a pragmatic solution, not introducing unnecessary systems early. There is no account management, we only have to look into the number of users connected to the system and making changes. The scalability of the system will be a thought after the basic system design is made.
I will start with a basic user connection for the first time to the board. I won't necessarily think about the other users yet. I will just focus on 1 user making changes to a pixel, and the lock of the user. So an early system drawing would look like:
flowchart LR
subgraph browser [Browser]
frontend[User 1]
end
subgraph api [API]
server[Server]
db[(Database)]
end
frontend -->|Load whiteboard| server
server -->|Return initial state| frontend
server --- db
In this graph, User 1 connects to the board, it gets the initial color schema, coordinate-based, from the server.
The server gets that information from a data storage. It will have to keep a connection ID to know who is connected, a stable userID to recognise users across reloads (the connection ID changes), and a board state with a coordinate => color mapping.
Now we can already make some small assumptions on that basic design. What DB are we talking about?
So far the data structure is quite simple, write and read heavy. A simple key/value DB should be good enough for that, something like redis or dynamoDB would work perfectly for fast lookup.
Communication between Browser and server should be using websocket as it will keep alive a connection to a user, allow both direction communication - especially important for the "notified as soon as possible" part. Other solutions would have to regularly poll the backend and would always have a delay.
Here is the initial thinking basic system:
flowchart LR
subgraph browser [Browser]
frontend[User 1]
end
subgraph api [API]
server[Server]
userdb[(User DB)]
boardState[(BoardState)]
lockUser[(Lock user with TTL)]
end
frontend -->|Websocket| server
server --- boardState
server --- lockUser
server --- userdb
Concerning the schema, to have a simple list of connected users, we would have a user management DB with userID (generated and stored in a cookie), and connectionID from the websocket.
Concerning the board state DB, here we need to get the color of each pixel of the board. So a simple coordinates {x,y} and color.
For the locking part, we will be using the Redis built-in TTL (time to live). We check the lock for a specific userID: if there is none, we allow the change; if there is an existing lock, the change is discarded and notified to the user. A simple userID and TTL works.
User table
| User ID | Connection ID |
|---|---|
uuid | string |
Board state table
| Coordinate | Color |
|---|---|
{x, y} | #123456 |
Lock table
| User ID | TTL |
|---|---|
uuid | 300s |
We have completely ignored the notification part and any scalability and optimization.
Here is the user flow, single user:
sequenceDiagram
autonumber
box Browser
participant frontend as User 1
end
box API
participant server as Server
participant lockdb as Lock DB
participant userdb as User DB
participant boardState as BoardState
end
rect rgb(235, 245, 255)
Note over frontend,boardState: Step 1: Initial Connection
frontend->>server: Connect
server->>boardState: Get board state
server->>userdb: Add user connection
server-->>frontend: Return board state
end
rect rgb(235, 255, 240)
Note over frontend,boardState: Step 2: Pixel Update
frontend->>server: Change pixel color
server->>lockdb: Is user locked?
alt user not locked
server->>boardState: Update board color
server->>lockdb: Lock user for 5 min
server-->>frontend: Change accepted
else user locked
server-->>frontend: Change rejected (retry in N seconds)
end
end
One question this flow raises: what happens if the same user sends two changes at the exact same time (two tabs, or a script)? "Check the lock" then "set the lock" are two separate steps, so both requests could pass the check before either one sets the lock, and the user would place two pixels.
The fix is to check and set in one atomic operation. In Redis, that is:
SET lock:{userID} 1 NX EX 300
SET lock:{userID} 1: create a key for that user (the value doesn't matter)NX: only if the key does Not eXist yetEX 300: the key EXpires after 300 seconds (5 min)
Redis runs this as one single operation, so only the first request gets OK back and is allowed to change the pixel. Every other request gets nothing back and is rejected, until the key expires.
Multiple users
The current design works fine on a single user. If we add multiple users, they can all connect to the webserver, they get registered and can perform pixel modification. But what is missing is the notification system. How to show changes to other users.
So for each action performed, we should trigger a notification to every other user on the board with the change, the frontend will manage the change of color.
The change needs to go into a pubsub topic, and workers can pull the changes and notify the other users of the new color of the pixel. There are two ways to send the change into the topic:
- DB trigger: if the DB supports it (e.g. DynamoDB Streams), every write automatically produces an event. This is the more reliable option: the event is tied to the write itself, so no change can be saved without being sent. Redis doesn't really offer this, so it would depend on the DB we pick.
- The server publishes the event itself, right after writing the change. This is simpler and works with any DB, but if the server crashes between the write and the publish, the change is saved but never sent, and other users won't see it until they reload the board.
Technically these workers would live in each of the websocket server to use the opened connections to send the notification.
flowchart LR
subgraph browser [Browser]
frontend[User 1]
frontend2[User 2]
frontend3[User 3]
end
subgraph api [API]
subgraph wsServer [Websocket server]
server[Server]
worker[Notification Worker]
end
userdb[(User DB)]
boardState[(BoardState)]
lockUser[(Lock user with TTL)]
end
subgraph notification [Notification]
pubsub[PubSub Topic]
end
frontend <-->|Websocket| wsServer
frontend2 <-->|Websocket| wsServer
frontend3 <-->|Websocket| wsServer
server --- boardState
server --- lockUser
server --- userdb
boardState -->|DB trigger| pubsub
server -.->|or publish directly| pubsub
pubsub -->|subscribe| worker
Optimization & scalability
Now the system is working as expected, the basic requirements are all set:
- lock in Redis would lock users that performed an action in the last 5min
- each color change triggers a notification with all the connected user into the board "as soon as possible". The websocket removes most of the latency on notification, the only latency visible would be:
Latency(per change) = Latency(lock) + Latency(change logged) + Latency(pubsub worker)
We are talking about a few milliseconds.
Websocket connections
In terms of scalability, how to handle that architecture at scale? Most of it is already scalable, we can scale the websocket via websocket load balancer to separate user connection and notification across servers.
Initial load of the board
The current concern is about the initial load. When a user connects for the first time, they need to get the current state of the board.
flowchart LR
subgraph browser [Browser]
frontend[User 1]
end
subgraph api [API]
server[Server]
db[(Database)]
end
frontend -->|Connect| server
server -->|Get Initial state| frontend
server --- db
With the current architecture, each user connecting to the board would require a full scan of the board as initial load. This might not be efficient, especially if it needs to happen in a very short amount of time.
A first solution to explore is a regular snapshot of the board, triggered every few minutes. The initial load would then get the last snapshot + the last few diffs that happened since the last snapshot.
There is a different solution which could be working more efficiently. Currently the design stores one coordinates => color key per pixel, which means one lookup per pixel and a lot of overhead per key. The board has a fixed size, so we don't need keys at all: a single dense array where the position gives the coordinates would do. Even with full 24-bit colors, that is only 1000 x 1000 x 3 bytes = 3 MB.
And if we decide to allow only a subset of colors for each pixel, we can shrink it even more into a bitmap, specifically a Redis BITFIELD. For a 1000x1000 board, a 16-color palette would allow storing each pixel in 4 bits, which would result in a size of:
1000 x 1000 x 4 = 4 million bits / 8 = 500 KB
Then we can maintain at all times the state of the whiteboard into a single bitmap array which only stores the colors: the coordinates are implied by the position in the array (offset = (y × 1000 + x) × 4 bits). The changes would be much more efficient, the initial load and notification would also be much faster.
Frontend to Backend interaction, data transfer optimization
Finally, another potential question might be about the data to transfer to frontend. My initial thought was running with a classic JSON, sending only diffs to the frontend instead of full state for each modification.
But we can optimize the communication for low data in case of mobile devices trying to access with low data. Here probably the bitmap array combined with protobuf messages over the websocket would reduce drastically the data needed for the frontend to work correctly.
We can also batch change notifications instead of sending every single change. Batching notifications every 100–250 ms would allow multiple changes on the same pixel to trigger only one notification per batch, while staying close enough to "as soon as possible" that users won't notice the delay.
Conclusion
This challenge felt easy, but I remember that when I did it, I fell short on the details and scalability. I had the right instincts, but the details were missing.
Nowadays with AI, knowing by heart what Redis or DynamoDB can do might not matter as much. What matters is understanding what your choices are (DB trigger, key/value databases, etc.) and being able to pick the right technology for the outcome you want.
I remember suggesting a relational database for cold storage, storing every action. Keeping a history of every change isn't a bad idea in itself (replays, moderation, rebuilding the board after a crash), but nobody asked for it, and I didn't tie it to any requirement. That was not really pragmatic of me, and it played against me, as I had stated at the beginning that I am more of a pragmatic engineer. Spending the time now, digging into what the right answers would have been, helps me figure out where I fell short.