← Updates / Benchmark · FennecBaarcha ·
A different spelling.
The same word?
What matters is getting the message from a Tunisian conversation. Derja, French and English can move between spellings and scripts. Our audit looks beyond those written differences.
Fennec’s lowest audited error rates With spelling, script & colloquial allowances
- Word error rate
- 14.19%
- Character error rate
- 6.44%
644 recordings · 50m 11s · All clips included
Before variant allowances: 24.09% WER / 8.75% CER. See what changes ↓
How to read this A post-hoc text audit on public development sets, with the broadest allowances tested. These rates measure transcript agreement; conversational understanding has not been measured directly. This is not a blind holdout.
01 / Reading the errors
Follow the score, step by step.
692of 1,684 word-edit penalties removed under the broadest variant policy
A transcript can use another written form of a Derja word, join an article to a French word, or write a recognizable English phrase in Arabic script. Standard WER counts those differences too.
Fennec-only, post-hoc text audit. These adjusted scores are not used in the model comparison. The alternatives were reviewed from text, without audio adjudication.
More interpretive
- Word error rate
- 14.19%
- Character error rate
- 6.44%
992 word edits remain · 692 penalties removed
Also allow broader colloquial forms and contractions, including ناخذ / ناخو and على البحر / عالبحر. This policy needs more linguistic judgment and is reported separately.
A broader colloquial allowance, beyond spelling alone.
View every scoring step
| Policy | Word edits | WER | CER |
|---|---|---|---|
| Standard scoring | 1,684 | 24.09% | 8.75% |
| Allow spacing differences | 1,446 | 20.69% | 8.75% |
| Allow spelling variants | 1,197 | 17.12% | 7.92% |
| Allow another script | 1,116 | 15.97% | 6.82% |
| Allow colloquial forms | 992 | 14.19% | 6.44% |
Small edits can still change meaning.
تنجم → تنجمش changes negation. عنده → عندها changes the pronoun. Neither is forgiven just because it is one character away.
Remaining edits are disagreements.
The broadest policy leaves 992 edits, including 249 one-character substitutions and 68 possible filler insertions. Listening is needed to distinguish recognition errors from omitted fillers or reference mistakes.
02 / The comparison
Same audio. Same scoring.
Fennec has the lowest word error rate in this run. Every model below uses the same light text normalization. The Fennec-only variant allowances highlighted above are excluded so the comparison uses one scoring policy for everyone.
644 clips · 6,990 reference words Lower is better
| Model / configuration | Word error rate |
|---|---|
| FennecOurs20261008-39663 · frozen CTC decoder | |
| LinTO TunisianVosk · CPU | |
| Whisper Tunisian CodeSwitchWhisper v3 · greedy / Arabic | |
| MAI-Transcribe 2Verbatim · automatic language | |
| GPT-TranscribeAutomatic language · no prompt | |
| Gemini 3.5 TranscribeAutomatic language · no prompt | |
| Qwen3-ASR 1.7BBF16 · batch 12 | |
| Whisper large-v3Greedy decoding / Arabic |
All models completed the same clips. Empty transcripts and outputs that reached the generation limit remain in the scores. These results describe the configurations tested.
03 / Transcription speed
A short wait for the words.
Recorded local transcription time for Fennec on an NVIDIA B200, with the model already loaded. One clip at a time, across all 644 recordings.
- Median per clip
- 34msHalf the clips finished within this time.
- 95th percentile
- 80ms95% of clips finished within this time.
- Audio / processing time
- 115×Audio duration divided by total call time.
50m 11s of audio in 26.1 seconds of summed transcription calls. Real-time factor: 0.0087. This measures short clips, not continuous or streaming audio.
What the timer includes
Local per-clip transcription with the model already loaded, including audio handling and decoding. All clips included; no warm-up clips excluded. Batch size: 1. Excluded: Model loading, Upload and network time, Gateway overhead.
The GPU was shared with other jobs. These are observed model-call timings, not a service latency promise. API runs include network and provider time, and other local models used different batching, so we do not rank their speed against these numbers.
04 / Behind the numbers
The method, in the open.
One frozen set of recordings, retained outputs, and corpus-level error counts. Independent implementations agree on WER. Fennec’s direct output also matched the live gateway on three development clips.
- Code-switched speech
- 300 clips
- Tunisian-only speech
- 344 clips
- Fennec release
- 20261008-39663
How WER and CER are calculated
WER is the sum of substituted, deleted and inserted words divided by the reference word count. CER uses character edits divided by reference characters, excluding whitespace. We aggregate counts across the corpus rather than averaging clip percentages. Insertions can take an error rate above 100%.
Light normalization uses NFKC, lowercase, Arabic vowel-mark and tatweel removal, and unified alef forms. Punctuation is spaced except internal apostrophes and numeric separators. French accents, negation, numbers and word boundaries are retained.
The variant audit permits explicit alternatives, rewrites only accepted hypothesis spans to the reference form, then recomputes WER and CER. The denominators remain 6,990 words and 30,060 characters. These allowances were chosen after inspecting Fennec’s outputs; apply the same frozen policy to all models before making an adjusted ranking.
Model configurations & output limits
Fennec
FP32, TF32 off; linear_union_prefix decoder, beam 100, alpha 0.3, beta 0.5, old/new LM weights 0.25/0.75.
0 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures
LinTO Tunisian
Local Vosk Tunisian model; standard 4,000-byte PCM chunks.
12 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures
Whisper Tunisian CodeSwitch
FP32, greedy decoding, language=ar, maximum 440 new tokens.
0 empty transcripts · 2 possible token-limit truncations · 0 unresolved service failures
MAI-Transcribe 2
microsoft/mai-transcribe-2 via OpenRouter; Azure enhanced mode, verbatim transcription, automatic language.
11 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures
GPT-Transcribe
gpt-transcribe through the OpenAI transcription API; no language hint or prompt.
11 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures
Gemini 3.5 Transcribe
google/gemini-3.5-transcribe via OpenRouter; automatic language, no prompt.
17 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures
Qwen3-ASR 1.7B
Stock Qwen3-ASR-1.7B, BF16, SDPA, left padding, batch 12, greedy decoding, maximum 512 new tokens. Automatic language.
11 empty transcripts · 3 possible token-limit truncations · 0 unresolved service failures
Whisper large-v3
Stock Whisper large-v3, FP32, greedy decoding, language=ar, maximum 440 new tokens.
0 empty transcripts · 7 possible token-limit truncations · 0 unresolved service failures
Language-hint follow-ups
Exploratory runs on the same audio. OpenAI received generic Tunisian context and Arabic, French and English hints; Gemini received an Arabic hint. No reference transcripts were supplied.
| Configuration | WER | CER |
|---|---|---|
| GPT-Transcribe · automatic | 48.73% | 25.08% |
| Gemini 3.5 Transcribe · automatic | 63.13% | 41.78% |
| GPT-Transcribe with hints | 43.93% | 20.20% |
| Gemini with Arabic hint | 93.16% | 47.02% |
Gemini’s Arabic-hinted result includes one runaway transcript with 2,369 inserted words against a 10-word reference. It remains counted and substantially worsens the corpus score. Two initially failed requests succeeded on retry.
What this benchmark cannot establish
- Blind generalization: these public sets were repeatedly inspected during Fennec development. The overlap check covered the original TunSwitch training manifest, not all training and decoder data.
- Semantic correctness: the variant audit used text, not human adjudication of the recordings. Names, numbers and negation were not separately annotated.
- Isolated speed: local runs shared hardware, while API calls included network time. Qwen used batch 12; its smoke outputs differed from single-clip inference on two of three clips. The final run uses the fixed batch configuration throughout.
- Every possible model setup: Whisper used greedy decoding with Arabic specified. Provider aliases can change, and the commercial APIs did not expose immutable snapshots. No Claude transcription run is included.
Take the numbers with you.
Versioned aggregate results, error counts and the explicit variant rules.
Next questions
Did the meaning come through?
That deserves a test of its own. A useful next benchmark would have Tunisian listeners annotate held-out conversations, then check whether each transcript preserves the intent and the facts.
Meaning and key details
Intent, names, dates, amounts and negation, plus whether a summary captures what was said. Report these separately from spelling agreement.
Real conversation conditions
Noise, overlapping speakers and longer recordings. Measure upload-to-transcript latency, concurrent requests and cost per audio hour as separate service tests.
These are proposed measurements; this run does not establish them. Follow future results in Updates.