How I Replaced Cloud TTS with a Self-Hosted Neural Voice Stack on an RTX 3060
For nearly two years, my internal business workflows, Telegram bots, and automated learning summaries relied on cloud-hosted text-to-speech APIs. Initially, Microsoft Edge-TTS provided a generous, zero-cost bridge. But running production workloads over unofficial browser-endpoint scraping is inherently fragile: sudden upstream schema changes, IP rate limits, and latency spikes across international transit routes routinely caused intermittent delivery drops. Commercial APIs like Azure Speech or Google Cloud TTS offered stability, but recurring meter billing scales awkwardly when generating long-form narration.
With a dedicated RTX 3060 12GB GPU passed through to a Windows 11 VM on my Unraid server, I decided to bring the entire speech synthesis stack fully on-premises.
Here is how I architected a self-hosted neural TTS pipeline using Alibaba’s open-source CosyVoice 2, hooked it into my Synology DSM n8n automation cluster, managed GPU memory alongside my other local AI models, and achieved instant sub-second audio responses across Chinese, English, and mixed-language speech.
1. The Design Goals
Before touching code, my target operational envelope required four non-negotiable criteria:
- Fluid Bilingual Code-Switching: In Malaysian and Southeast Asian business communication, text is almost never purely monolingual. Sentences naturally mix Chinese syntax with English technical terms (e.g., “请检查这个 pull request 并确认 webhook 状态”). The engine must articulate both languages without sudden cadence jumps or unnatural accent shifts.
- Zero-Shot Voice Variety: I needed distinct voice profiles—cheerful child voices for storybook projects, authoritative male anchors for news roundups, and clear conversational female tones for support bots—cloned from brief reference clips without days of fine-tuning.
- Constrained VRAM Footprint: The RTX 3060 has 12GB of VRAM. The VM regularly runs local LLMs (Qwen GGUF on LM Studio) and ComfyUI image/video pipelines. The TTS engine had to either idle under ~3.5GB of VRAM or provide frictionless cold-start toggles.
- Drop-in Webhook Compatibility: My central automation runs on Synology DSM via n8n (
n8n.internal:5678). Replacing cloud nodes could not require rewriting dozens of downstream workflow schemas.
2. Model Selection: Why CosyVoice 2 Won
I evaluated the primary open-source contenders against my bilingual requirement:
| Model Architecture | Strengths | Drawbacks | Verdict |
|---|---|---|---|
| Edge-TTS / Piper | Ultra-lightweight, CPU-only | Robotic cadence, rigid voices, cloud dependency | Deprecated |
| ChatTTS | Conversational laughter, pause tags | Unpredictable halluncinations, weak English | Not suitable |
| GPT-SoVITS v2 | Unrivaled character cloning accuracy | Phoneme boundaries stumble on mixed Chinese-English | Secondary / On-demand |
| CosyVoice 2 (0.5B) | Superb bilingual flow, 300ms first-packet latency | 3.2GB VRAM baseline | Primary Production Engine |
CosyVoice 2 (the 0.5B parameter variant from Alibaba’s FunAudioLLM team) emerged as the definitive winner. By combining an autoregressive language model with flow-matching diffusion decoders, it treats cross-lingual text-to-speech as a unified sequence. English terms embedded inside Chinese phrases flow with native pronunciation and natural human breath pauses.
3. Infrastructure & Network Topology
My self-managed infrastructure runs on a tiered hardware split:
[ Internet / Clients ]
│
▼
[ Synology DSM Storage Node ]
├── Portainer / Traefik Reverse Proxy
└── n8n Automation Engine
│ HTTP POST / GET (/webhook/ltts)
▼ (Local 1Gbps LAN)
[ Unraid Host ]
└── Windows 11 VM (GPU Passthrough)
├── NVIDIA GeForce RTX 3060 12GB
└── CosyVoice 2 FastAPI Service (127.0.0.1:50000)The Windows VM hosts the CUDA 12.x PyTorch runtime. I exposed a lightweight FastAPI wrapper inside CosyVoice’s runtime script (webui.py), allowing my Synology n8n instance to trigger generation over LAN via http://gpu-node.internal:50000/tts.
4. Managing GPU Memory: Resident vs. On-Demand
Leaving every AI model loaded simultaneously will exhaust 12GB of VRAM instantly. I designed a dual-tier lifecycle:
- CosyVoice 2 as the Resident Daemon: Consuming only 3.2 GB of VRAM, CosyVoice 2 runs as an auto-starting background service when Windows boots (
shell:startup\CosyVoice 2.lnk). This leaves 8.8 GB free for general desktop and lightweight inference. - One-Click State Toggle Batches: For heavy tasks (such as running 1080p video generation in ComfyUI or 14B LLMs), I created an idempotent batch script on the desktop:
@echo off
title CosyVoice 2 Toggle
for /f "tokens=5" %%a in ('netstat -aon ^| findstr :50000') do (
taskkill /F /PID %%a >nul 2>&1
echo STOPPED. GPU memory freed.
exit /b
)
:: If stopped, launch background worker
start /b python webui.py --port 50000 --model_dir pretrained_models/CosyVoice2-0.5B
echo ONLINE - CosyVoice 2 loaded to GPU.A single double-click halts the process and reclaims all 3.2GB within 500ms; clicking it again restores the service in under 25 seconds.
5. Integrating with Synology N8N
In n8n, I configured a dedicated workflow: Local TTS API (CosyVoice) listening on /webhook/ltts.
To ensure total interoperability with existing scripts, the workflow accepts both:
GET /webhook/ltts?text=Hello&voice=child: Ideal for browser testing and simple URL triggers.POST /webhook/lttswith JSON body{"text": "...", "voice": "..."}: Used by bots and backend services.
The n8n pipeline consists of three streamlined nodes:
- Webhook Node: Receives incoming requests.
- Parse & Clean Text (Code Node): Normalizes character encodings and defaults voice selections.
- HTTP Request Node: Forwards the JSON payload to
http://gpu-node.internal:50000/ttsand streams the binaryaudio/wavresponse directly back to the requester.
6. Curating Zero-Shot Reference Voices
CosyVoice 2 synthesizes speech using zero-shot voice cloning: feeding a 3- to 5-second clean audio sample allows the model to clone the acoustic timbre instantly.
I packaged a library of curated reference samples into asset/voices/ and mapped them to intuitive alias parameters:
| Voice Key | Style Profile | Sample Source |
|---|---|---|
child / girl | Cheerful female child voice | Clean 4s preschool dialogue clip |
boy | Energetic young boy (anime style) | 5s youth animation dialogue |
male / yunxi | Engaging, modern tech male anchor | 5s professional narration |
news | Deep, formal broadcast presenter | 5s documentary voiceover |
female | Warm, conversational support agent | 5s acoustic reading sample |
default | Original CosyVoice high-fidelity voice | Model’s baseline reference |
Switching voices is as simple as passing &voice=child in the query string or {"voice": "male"} in the POST payload.
7. Real-World Performance
Running on the RTX 3060:
- Generation Speed: Real-time factor (RTF) averages 0.18. A 10-second sentence renders in under 1.8 seconds.
- Latency: First audio packet arrives at n8n in approximately 320ms, enabling near-instant voice replies on Telegram.
- Cost & Privacy: Zero monthly cloud API charges, zero audio data leaving my local subnet, and complete immunity to external provider outages.
Self-hosting modern neural TTS is no longer an exercise in brittle compile scripts and robotic phoneme tables. With open-source diffusion models and a modest 12GB workstation GPU, you can run a private, high-fidelity voice engine that surpasses commercial cloud offerings in customization and control.
Want a private AI voice pipeline for your business?
If your support bots, content workflows, or internal tools are paying recurring cloud TTS bills — or struggling with bilingual Chinese-English code-switching — I design and deploy self-hosted speech and n8n automation stacks on hardware you control.
- 📱 WhatsApp: +60 12-797 2969
- 📧 Email: [email protected]
- 🌐 Website: hoelee.com
