MeetNote AI

Two systems, one repository

The web process boots in a fixed order: connect to MongoDB, start the mediasoup worker, create every SQS queue, start the consumers, then attach Socket.IO and listen. Queue creation always runs, even when the consumers are switched off, because this process still needs to publish jobs.

There is a second entry point that skips the HTTP server, the sockets and mediasoup entirely and starts only the consumer loops. Run it alongside a web process with workers disabled and the transcription load scales independently of the media server; run neither and the single process does both. Which shape is in use is one environment variable.

Routing the media

There are three ways to move video between participants. A mesh has everyone send their stream to everyone else, which is fine at two people and unusable at six, because each participant uploads one copy of themselves per peer. An MCU mixes all the streams server-side into a single composite, which is cheap for clients and expensive for the server, since every room costs a live transcode.

MeetNote uses the third option. A mediasoup worker is created at boot and each room lazily gets its own router; participants create a send transport and a receive transport, publish their tracks to the router, and subscribe to everyone else's. Nothing is decoded or re-encoded on the way through, so a room costs bandwidth, not CPU.

Media does not travel over the same connection as the application traffic. Signalling runs over Socket.IO on the ordinary HTTPS port through the reverse proxy, while RTP goes straight to the host on a dedicated UDP port range, with TCP available as a fallback for networks that block it.

When a participant leaves, their consumers, producers and both transports are closed explicitly, and once the last one is gone the router itself is closed. Media objects are not garbage collected on their own.

Getting into a room

Socket.IO authentication happens in middleware before any handler runs: the client supplies its JWT in the handshake, and a connection without a valid token is rejected outright rather than being allowed to connect and then filtered per event.

The first person to create a room becomes its host. Anyone else knocks: they land in a waiting set and the host receives their request live, approving or denying it. Approval is tracked per socket and per user, so someone who was let in and then reloaded is readmitted without having to knock a second time.

Capturing the audio

Recording happens in the browser, per participant, using MediaRecorder on an audio-only clone of the local stream. Each participant produces their own Opus-in-WebM file rather than the server mixing everyone into one, because separate tracks survive one person having a bad microphone.

The interesting part is how the stream is cut. To ship audio while the meeting is still running, the recorder is stopped, its accumulated blob is sealed, and a brand new recorder is started on the same stream. Slicing a running WebM stream produces fragments with no header that no decoder will open; restarting the recorder means every chunk is a complete, independently decodable file. Each chunk carries its offset from the start of the recording, which is what makes the transcript timeline reassemble correctly later.

Batching while the meeting is still running

Uploaded chunks are not transcribed one at a time. A per-room accumulator collects them and seals a batch when either it holds a chunk from every currently active participant, or fifteen seconds pass, whichever comes first. The participant count is read from the meeting document each time, so someone joining or leaving mid-call changes the seal condition immediately.

A sealed batch gets a monotonic index, a start offset taken from the earliest chunk in it, and a flag marking the final batch. Then it goes onto a queue and the live process forgets about it. This accumulator is the one piece that cannot move to a worker: it depends on per-room timers and the live participant count, both of which only exist in the process holding the sockets.

When the meeting ends, whatever has accumulated is sealed as the final batch. If nothing is left over but batches were already sent, the total is recorded directly instead, because otherwise a browser tab closed mid-sentence would leave the pipeline waiting for a final batch that is never coming.

Transcribing a batch

The transcription worker takes one batch, mixes its per-participant WebM files into a single 16 kHz mono WAV with FFmpeg, and sends that to Deepgram, asking for utterances rather than a flat string.

FFmpeg's mixer averages its inputs by default, which quietly halves everyone's volume in a two-person call and does worse with more. In a meeting where most tracks are silence, that attenuates the one person actually talking until the transcriber returns nothing at all. The mix is configured to sum instead of average.

Each returned utterance is stored with its timing shifted by the batch's start offset, so segments carry absolute positions in the meeting rather than positions within their own batch. The full text is rebuilt by sorting on that absolute time, which means batches can finish out of order without scrambling the transcript.

Every batch index is recorded in a set on the transcript document, and a batch already in that set is skipped. Queues deliver at least once, so a handler that is not safe to run twice will eventually corrupt something.

Finalizing, and waiting for work you do not own

When the last participant leaves, the live process seals the final batch and publishes one finalize job. Everything expensive happens from there on in a worker.

That worker has a problem a queue cannot express: it must not finish until every transcription batch has landed, and there is no primitive for waiting on other jobs. So it checks how many batches have been processed against how many were expected, and if the answer is not yet, it re-publishes itself with a delivery delay and returns. Roughly twenty of those deferred passes later it gives up waiting and proceeds on a best-effort basis rather than blocking a meeting forever.

Once drained, it re-scans the room directory on disk rather than trusting the file list in the message, because late uploads are real. The audio is uploaded to Cloudinary, the recording is marked ready with its duration, and note generation is published to its own queue. If no incremental transcript exists at all, it falls back to mixing the whole meeting into one file and transcribing that in a single pass.

The failure paths are as deliberate as the happy one. A meeting with no captured audio is marked skipped, not failed, so the dashboard shows a neutral state instead of an alarming one. A hard failure records that status before rethrowing, so the job can retry without leaving the interface stuck on a blank pending card.

Notes, and the document that comes out

Note generation is its own queue and its own worker. It builds a prompt around the transcript that also carries the meeting title, agenda, participants, date and duration, and demands a fixed markdown structure: a two-to-three sentence summary, discussion points grouped under sub-headings, numbered decisions, and action items as a checklist naming a person.

Two rules in that prompt do most of the work. Sections with no real content must be omitted entirely rather than filled with the model's best guess, and nothing may be inferred that the transcript does not state. A summary that invents an action item is worse than no summary.

There are two providers behind an interface, tried in order. Gemini runs first because its notes are better; Groq exists as a backstop for outages and rate limits. Only when every configured provider has failed does the job throw, which lets the queue retry it and eventually park it.

The markdown is stored as-is. The PDF is rendered in the browser on demand: the markdown is parsed to tokens and mapped onto a styled document, so headings, checklists, tables and code blocks all keep their meaning on the page rather than being flattened into text.

Everything around the meeting

A meeting is a document with a lifecycle, not just a room ID. Meetings can be scheduled ahead, repeat on a recurrence that is expanded into individual occurrences up to a cap, and be invited to by user. Their status is partly derived rather than stored: a scheduled meeting becomes ready inside a window before its start time and missed once a grace period has passed, so nothing has to run a cron job to move it along. Invitations are exported as calendar files.

Notifications go out over two channels from one call: into the live socket for anyone with the app open, and over Web Push for anyone who does not. Push subscriptions that come back gone or not-found are deleted on the spot instead of being retried forever. The client is an installable PWA that precaches only the static shell, with the API, the socket endpoint and room URLs explicitly excluded so a service worker can never serve a stale answer to real-time traffic.

Shipping it

The client is on Vercel. The server runs on a single small EC2 instance under pm2, behind nginx which terminates TLS and upgrades the WebSocket, with the application port never exposed. RTP is the exception and reaches the host directly on its own port range.

Deployment is a GitHub Actions workflow that assumes an AWS role through OIDC, so there are no long-lived keys in secrets, and drives the box over SSM rather than SSH, which stays closed entirely. It fires only for changes under the server directory, since the client deploys itself, and it refuses to run two deploys at once. The pull on the box is fast-forward-only so a drifted instance fails loudly instead of quietly creating a merge commit.

One more thing runs on boot: any recording still marked in-progress is republished as a finalize job. There are no live rooms immediately after a start, so anything in that state was orphaned by a crash, and re-enqueuing it is what stops a meeting hanging in a non-terminal state forever.