Some implementation details, since getting this to work well was not trivial.
My goal was “press hotkey, start talking, see text within ~1–2 seconds” on an M2 MacBook Pro, and support multiple languages.
First attempts (cloud)
– I tried Hugging Face real-time transcription. It worked but latency was all over the place and costs would not scale.
– I tried OpenAI real-time transcription. Latency was better, but when there was background noise, it'd transcribe wrong things. Saw 200ms responses. I can bring that back if I can make it stable.
– I briefly experimented with Gemini for transcribing and formatting multi-language text. Quality was not consistent enough compared to Whisper for Multi language.
Local experiments
– I used FFmpeg + Whisper CLI in a bunch of ways: batching, buffering, trying to “stream” partial results out of Whisper to make it feel live.
– I also tried a local Llama model to format the raw transcript into an email. On an M2 Pro this took ~2 seconds for short emails and got much slower for long text. It looked nice but the latency was not acceptable for everyday use.
Where I ended up (for now)
– Current version sticks to FFmpeg + Whisper CLI locally, optimized for short chunks so you usually see text within about 1–2 seconds.
– I dropped the heavy on-device LLM formatting and keep the formatting logic much simpler so it stays predictable and fast.
Next step is to re-introduce “smart” formatting and meeting notes, but only when I can do it without blowing up latency. Happy to dig deeper into any of these if people are curious.
I second this. I think playwright and cypress are the main ones you should use. We already implemented something for Cypress and looking to do the same thing for Playwright. It'll be pretty similar to this: cypress.preflight.com
Would love to get your feedback so we can add it :)
100% agree but I still think there are still people using Selenium. I think we could help people navigate to Cypress or Playwright or no-code solutions
I really like playwright's locators as well. It looks pretty cool. We actually have a Cypress POM generator tool which uses AI and let's you test emails etc: https://cypress.preflight.com/
This is actually very interesting. I like how it's done. Very great job.
We also do the same thing but we also understand the box (context) then we can say. We should be on Sign in box and a button called Sign in should be present. If we are on a different page and we cannot find something similar to it, we will give better error messages. Like: we should have been on Sign in page but we are on sign up. etc.
You are right every testing solutions out there push UIEvents to the page rather than clicking with an actual mouse. That's why puppeteer, selenium etc are scraping tools not testing tools
We have done all the ground work. Like:
- Concurrency
- Adapt to the changes. Our selectors are like this: "Click on 'Login' button in the 'Sign in' form"
- Update the tests with an HTML/Video player etc
I think the issue here is Selenium or Playwright they depend on Selectors which depends on UI. And when there is a change it breaks the tests. We are working on something to generate you an adapting code (Cypress first) and let you know when there needs to be a change in your test script.
We have an AI model that understands the page structure as humans do. So we can do this "Click on 'Sign in' on the 'Login' page".
We have a no-code tool as well which adapts to the changes. But we want to generate the code for people who want to keep things internally.
My goal was “press hotkey, start talking, see text within ~1–2 seconds” on an M2 MacBook Pro, and support multiple languages.
First attempts (cloud) – I tried Hugging Face real-time transcription. It worked but latency was all over the place and costs would not scale. – I tried OpenAI real-time transcription. Latency was better, but when there was background noise, it'd transcribe wrong things. Saw 200ms responses. I can bring that back if I can make it stable. – I briefly experimented with Gemini for transcribing and formatting multi-language text. Quality was not consistent enough compared to Whisper for Multi language.
Local experiments – I used FFmpeg + Whisper CLI in a bunch of ways: batching, buffering, trying to “stream” partial results out of Whisper to make it feel live. – I also tried a local Llama model to format the raw transcript into an email. On an M2 Pro this took ~2 seconds for short emails and got much slower for long text. It looked nice but the latency was not acceptable for everyday use.
Where I ended up (for now) – Current version sticks to FFmpeg + Whisper CLI locally, optimized for short chunks so you usually see text within about 1–2 seconds. – I dropped the heavy on-device LLM formatting and keep the formatting logic much simpler so it stays predictable and fast.
Next step is to re-introduce “smart” formatting and meeting notes, but only when I can do it without blowing up latency. Happy to dig deeper into any of these if people are curious.