Voice notes with AI: how BillableMe turns audio into text with Whisper

As a freelancer you get your best ideas in the car or on site, exactly when typing is not an option. That is why BillableMe got voice notes: you speak a to-do, reminder or note, and half a minute later the text appears next to it, searchable. This is how I approached it.

The requirements

• Audio stays on my own server, without an external transcription service.
• Dutch and English, with automatic language detection.
• No GPU: it has to run on the existing Docker Swarm.
• The user never waits for the transcription.

Step 1: recording in Flutter

The app records with the record package in AAC (m4a), asks for microphone permission and lets you pick a category, a flag and a title right away. The file goes to the NestJS API as a multipart upload. From that moment the note is visible in the list, with the status “processing”.

Step 2: a queue instead of waiting

The API stores the audio file on a shared volume, creates a record in PostgreSQL and puts a job on a Bull queue in Redis. The HTTP call therefore finishes in a fraction of a second. A worker picks up the job, calls the transcription service and retries up to three times with increasing delays when something fails. The status moves from pending to processing to completed or failed.

Step 3: a self-hosted Whisper service in Python

The transcription itself happens in a small FastAPI service running OpenAI Whisper. I chose the base model: 74 MB, roughly 30 seconds per minute of audio on a CPU, and more than good enough for spoken notes in Dutch and English. The model size is an environment variable, so I can scale up to small or medium when the hardware allows it. ffmpeg ships in the container to decode m4a.

Step 4: back in the app

As soon as the text is in the database, the app shows it below the note. You can play the recording at 0.75 to 1.5 times the speed, edit the note, filter by category or period and search through the transcriptions.

What I learned

• Pick the smallest model that is good enough. The small model is three times slower than base on a CPU; for short spoken notes base was plenty.
• Asynchronous is not a luxury. Waiting thirty seconds in a mobile app feels like forever. A queue with a visible status makes it feel natural.
• Keep the audio local. One shared volume between the API and the Whisper service is all it takes. The recordings never leave the server.

Would you like a similar AI feature in your own app? Get in touch.