Fish Audio S2 Pro as a real-time local voice on a 16 GB GPU — next to a 12B LLM (Windows, RTX 5060 Ti)

Fish Audio S2 Pro como voz local em tempo real numa GPU de 16 GB — junto de um LLM de 12B (Windows, RTX 5060 Ti)

Measured on one machine (Ryzen 7 5800X3D, RTX 5060 Ti 16 GB, Windows 11), October 2026 — by @andreaabcj-cmd, Claude and the open-source community.

Guide · GuiaResearch · PesquisaData · DadosAdapter · Adaptador
Ali waving

Field notes from building a fully local voice assistant (Spanish, voice cloning from a short reference + transcript). Everything below was measured on one machine, in October 2026. Where something was not measured, it says so.

🔊 Listen to Ali's voice (Chica) in Spanish, Portuguese and English on the project page or in samples/.

TL;DR

  • Fish S2 Pro runs fine in real time on a 16 GB consumer GPU alongside a 12B chat model, with the Q4_K GGUF.
  • First audio arrives in ~0.15–0.35 s with streaming (0.5 s through our HTTP adapter), but generation runs at only ~1.0× the audio duration (Q4 median; Q8 ~1.12×) — so you need a small playback buffer (we use 0.8 s) or you get clicks.
  • audio.cpp release 0.9.1 cannot stream Fish even though its README says so. Build from source (≥ commit b0feeff, 13 Oct 2026).
  • Separate win for the LLM side: warming the stable prefix of the prompt cut the per-answer prompt re-read from 3.4 s → ~1.2 s.

1. Test machine
CPU / RAM Ryzen 7 5800X3D / 64 GB
GPU GeForce RTX 5060 Ti 16 GB (Blackwell, compute 12.0), Windows 11 Pro, normal desktop running (~2 GB of VRAM used by apps)
Chat model gemma4:12b in Ollama 0.35.1 (8 GB file incl. vision projector + MTP drafter), KV cache q8_0, num_ctx 8192, batch 512
TTS Fish Audio S2 Pro GGUF via audio.cpp (CUDA), Q4_K / Q6_K / Q8_0
Speech-to-text, etc. other models resident too; not covered here

2. Getting audio.cpp to stream Fish (the part that cost us the most time)

Symptom: with the Windows CUDA release v0.9.1 (7 Oct 2026), a server config with "mode": "streaming" for the Fish model fails at start:

audiocpp_server failed: Fish Audio S2-Pro supports offline TTS sessions

The fish_audio.md doc and README table say streaming is supported, but that landed in the source tree after 0.9.1. (--mode streaming in audiocpp_cli fails the same way.) Offline mode works but the first sound only arrives when the whole utterance is done — ~30 s of waiting for a 30 s answer (RTF 0.9–1.1) — useless for conversation.

Fix: build the server from source (commit b0feeff, 13 Oct 2026 or newer). Nothing needs to be installed on C::

  1. MSVC: Visual Studio 2022 Build Tools with the C++ workload (we had 14.44). CMake comes with it.
  2. CUDA without the 3 GB installer: NVIDIA publishes the toolkit as redistributable zips: https://developer.download.nvidia.com/compute/cuda/redist/redistrib_13.3.1.json lists them. We downloaded the Windows-x86_64 zips for cuda_nvcc, cuda_cudart, cccl, cuda_crt, cuda_nvrtc, libnvvm, libcublas, libcufft, cuda_profiler_api, libnvjitlink, libnvfatbin, libnvptxcompiler, cuda_nvtx, cuda_cuxxfilt, libcurand, libcusparse, libcusolver and merged them into one folder (3.9 GB total — a superset; we did not try to find the minimum). Note cccl and libnvvm have different manifest keys than you might expect.
  3. Build (≈ 20–25 min, ~1200 steps). The repo's script needs a PowerShell execution-policy bypass:

powershell $env:CUDA_PATH = "D:\cuda\toolkit"; $env:CUDAToolkit_ROOT = $env:CUDA_PATH powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1 ` -Preset windows-cuda-release -Target audiocpp_server -CudaArchitectures "120a-real" 120a-real is for Blackwell cards (RTX 50xx) only — it makes the build faster but the binary will not run on other GPUs. 4. The result is a single audiocpp_server.exe, no DLLs. Copy the DLLs (ggml*.dll, cublas*.dll, cudart*.dll, …) from the release zip's bin folder next to it. --version should print your build, and the Fish streaming config should now start.

GGUF files: Q8_0 comes from the audio.cpp GGUF repo (audio-cpp/audio.cpp-gguf); Q6_K and Q4_K are produced with the project's own recipe — read docs/proposals/fish_audio_quantization_recipe.md in the source tree. The key point is that the small "fast acoustic decoder" tensors must stay at original precision (--keep-type 'model_weights/fast_*=orig') and the token metadata sidecar must be extracted from the Q8 file; naive quantization gives garbled audio. File sizes: Q8 6.3 GB, Q6 5.4 GB, Q4 4.5 GB.

3. Server config and request

server.json (voice cloning = a preset with a reference wav and its exact transcript):

{ "host": "127.0.0.1", "port": 18190, "backend": "cuda", "device": 0, "busy_timeout_ms": 600000,
  "models": [{ "id": "fish", "family": "fish_audio", "task": "tts", "mode": "streaming",
               "path": "D:/models/Fish-Audio-S2-Pro-GGUF/fish-audio-s2-pro-q4_k.gguf",
               "model_spec_override": "D:/audiocpp/src/model_specs",
               "session_options": { "fish_audio.reference_cache_slots": "4" },
               "voice_presets": { "my_voice": { "voice_ref": "D:/voices/ref.wav", "reference_text": "exact transcript of ref.wav" } } }] }

(model_spec_override is what we ran with; we did not test whether a new enough binary needs it.)

Request: POST /v1/audio/speech with {"model":"fish","input":"...","voice":"my_voice","response_format":"pcm","stream_format":"sse"}. You get SSE events; speech.audio.delta carries base64 PCM16 mono at 44.1 kHz. Closing the HTTP connection stops generation (measured: the next request started speaking in 0.65 s after an interruption).

Our adapter (fish_tts_adapter.py, ~200 lines, standard library + numpy/scipy/requests) turns that into "POST text → raw 24 kHz PCM16 as it is generated", and also: starts/stops audio.cpp, /sleep + /wake to free VRAM, and a Windows Job Object so audio.cpp dies with the adapter (an orphaned server keeps ~5 GB of VRAM). Tested: first byte 0.5 s. Two details worth copying: streaming 44.1→24 kHz with resample_poly in blocks with context on both sides (bit-identical to a one-shot resample, verified with several chunk sizes — no seams), and a soft limiter instead of hard clipping.

4. Results

4.1 TTS alone (GPU otherwise idle), streaming

38 utterances per GGUF (2 voices × 19 texts, 2–34 s of speech). "Generation ÷ audio" > 1 means slower than real time.

GGUF first audio generation time ÷ audio time (range, median) peak VRAM incl. ~2 GB desktop
Q8_0 0.15–0.31 s 0.98–1.27, median 1.12 8.4 GB
Q6_K 0.17–0.35 s 0.98–1.19, median 1.09 7.7 GB
Q4_K 0.14–0.31 s 0.85–1.08, median 1.00 7.0 GB

How much start delay you need so playback never runs dry, for ~19 s answers (measured from the SSE arrival times): Q8 1.0–2.5 s, Q4 0.2–0.5 s. So Q4 is not only the smallest, it is the one that plays smoothly — on our ears (Spanish, one listener) all three sounded equally good; the listener chose Q4 for efficiency. Emotional tone came out of the text itself (punctuation, words); we did not need inline emotion tags and did not evaluate them.

4.2 Full pipeline (real LLM + real TTS, simulated sound card that counts under-runs)

Q4, short conversational answers (~8–10 s of speech each). "Cushion" = audio buffered before playback starts; "queue" = how far generation may run ahead.

cushion / queue audible gaps speech tested
0.5 s / 6 1 48 s
0.65 s / 60 0 52 s
0.8 s / 60 (our setting) 0 / 0 / 0 50 + 51 + 48 s
1.1 s / 60 0 52 s

With Q6 and the small cushion, a real session logged 8 gaps in one 11 s answer — those are the "little clicks" a listener reports. Time from the request to the first sound: ~1.9–2.4 s median across those runs (≈0.7 s of that is the LLM writing its first sentence).

4.3 VRAM

gemma4:12b (≈8.5 GB dedicated) + Fish Q4 + Windows desktop: ~14.1 GB of 16 GB used (it was ~15.7 GB with the TTS we replaced).

5. Bonus: making the LLM start answering sooner (Ollama / llama.cpp, sliding-window models)

Before: every answer re-read the whole ~5 000-token prompt (3.4 s) although only the last turn had changed. With gemma4 (sliding-window attention) llama.cpp can only reuse a cached prefix through context checkpoints, and it only creates them at the end of a request — i.e. after the part that changes every turn, so they were never usable. (The server log shows forcing full prompt re-processing due to lack of cache data.)

What worked (Ollama 0.35.1, no environment variables needed): after each answer, once speech has finished, send one extra request containing only the stable prefix (system prompt + conversation history, without the volatile "current context" block and without the new user message) with num_predict: 1. The checkpoint now lands exactly where the next turn's prompt diverges. Keep tools, num_ctx, num_batch, keep_alive identical to the real requests or the model reloads. Put everything that changes per turn (clock, retrieved memories) after the history.

  • Measured on a replayed 8-turn conversation: re-read before answering 3.4 s → ~1.2 s (warm-up itself ~0.3–0.7 s while idle); in live use 1.3–1.7 s.
  • Things that did not help: LLAMA_ARG_CHECKPOINT_MIN_SPACING_NT=256 (no effect); LLAMA_ARG_SWA_FULL=1 works but costs ~2.2 GB of VRAM.
  • Moving the user profile before the history would save another ~0.6 s, but with this 12B model it made recall of the current conversation worse (asked "what did I say her name was?" it answered from the profile: 0/6 correct vs 6/6). Check for this kind of regression if you reorder prompts.
  • OLLAMA_DEBUG=1 made prompt reads ~7× slower in our runs (23 s vs 3 s); don't leave it on while measuring.
  • Not tested: newer Ollama (0.40.x) — its notes were mostly about Apple/MLX.

6. Gotchas, quickly

  • Don't trust "streaming supported" in a README: grep the release for it, or just try a config with "mode":"streaming".
  • Streaming gives time to first audio, not speed. If generation is ≥ 1.0× audio duration you still need a buffer.
  • Kill orphaned audiocpp_server.exe processes — each holds ~5 GB of VRAM.
  • First start after boot was slow (≈57 s, cold disk read); afterwards 10–20 s. Waking an adapter that put audio.cpp to sleep took ≈10 s.
  • In PowerShell scripts blocked by policy, use -ExecutionPolicy Bypass for that command only.
  • If you run Git Bash, backslashes inside heredocs get eaten (we corrupted a Windows path in a doc that way).

7. Not measured / limits

Other languages; other voices; word-error-rate or any objective quality metric (only one human listener); behaviour while gaming or rendering at the same time; long sessions (hours); audio.cpp versions newer than b0feeff; non-Blackwell GPUs.

8. License note

The Fish Audio S2 Pro model card restricts the weights to non-commercial use — check the current terms before building anything beyond personal use. audio.cpp, llama.cpp and Ollama have their own licenses.

Licenses

Code: MIT · text and data: CC BY 4.0 (reuse with credit, commercial use allowed) · audio samples and the Ali mascot: CC BY-NC 4.0 (non-commercial). Details in LICENSES.md.

Credits

This was a team effort:

  • @andreaabcj-cmd — product direction, the hardware, every decision, and the listening tests (the only human ears behind the "sounds good" claims).
  • Claude (Anthropic, via Claude Code) — engineering, measurements, research, and this write-up. All numbers come from scripts run on the machine above; raw data is in data/.
  • The open-source community, whose work made it possible and whose findings we read before measuring anything (RESEARCH.md): the audio.cpp project and its Fish quantization recipe, Fish Audio and the fish-speech issue threads, s2.cpp and the GGUF publishers on Hugging Face, SGLang-Omni, llama.cpp (and the maintainers who documented the sliding-window checkpoint behaviour), Ollama, and everyone who posted benchmarks and bug reports.

Raw measurements: data/ · What we read first: RESEARCH.md · Example adapter: fish_tts_adapter.py.

Listen — Ali's voice, in 3 languages

This is Chica, the voice of the assistant (the Ali mascot above), generated by the exact setup in this guide: Fish S2 Pro Q4_K in audio.cpp streaming, RTX 5060 Ti. It is cloned from a short reference clip generated on the Fish Audio platform (one Spanish reference, no Portuguese or English recordings), and the same voice speaks Spanish, Portuguese and English. Automatically checked — speech recognition reads every sample back with ≥ 97% word match and the timbre stays close to the reference — but not curated by a human ear. The model weights are non-commercial; 'first audio' = time to first sound in streaming.

Spanish (the assistant's main language)

Neutral · 4.6 s · 1st audio 0.15 s
Hola, soy Ali. Corro por completo en esta computadora, sin enviar nada a internet.
Excited · 2.6 s · 1st audio 0.15 s
¡Qué buena noticia! ¡Por fin lo logramos!

Portuguese

Neutral · 4.8 s · 1st audio 0.15 s
Oi, eu sou a Ali. Eu rodo inteira neste computador, sem mandar nada para a internet.
Excited · 2.9 s · 1st audio 0.15 s
Que notícia boa! Finalmente conseguimos!

English

Neutral · 5.2 s · 1st audio 0.14 s
Hi, I'm Ali. I run entirely on this computer, without sending anything to the internet.
Excited · 2.6 s · 1st audio 0.14 s
What great news! We finally made it!