Nice work on the speech-to-speech pipeline! You're absolutely right that it has to go through the text intermediate step - that's actually where a lot of the interesting processing can happen.
I've found that the speech->text->speech approach gives you much more control over the output quality. The text intermediate step lets you clean up transcription errors, adjust tone, and even restructure the content before converting back to speech.
Have you experimented with different text processing steps in between? I've been building something similar at voicevoyage.io focused on that middle text processing layer - turning raw transcriptions into polished content before the final output.
For local speech-to-text, Whisper remains the gold standard - you can run it locally with good accuracy across languages. For speech-to-speech, you'd typically chain Whisper with a local TTS model like Coqui TTS or use something like Tortoise TTS for higher quality but slower processing. The key is balancing accuracy, speed, and resource usage based on your specific use case. If you're doing content creation workflows, consider what post-processing you might need - sometimes the raw transcription needs structure and enhancement beyond just accurate words.
I've found that the speech->text->speech approach gives you much more control over the output quality. The text intermediate step lets you clean up transcription errors, adjust tone, and even restructure the content before converting back to speech.
Have you experimented with different text processing steps in between? I've been building something similar at voicevoyage.io focused on that middle text processing layer - turning raw transcriptions into polished content before the final output.