← Updates / Benchmark · Fennec ·

A different spelling.
The same word?

What matters is getting the message from a Tunisian conversation. Derja, French and English can move between spellings and scripts. Our audit looks beyond those written differences.

Fennec’s lowest audited error rates With spelling, script & colloquial allowances

Word error rate
14.19%
Character error rate
6.44%

644 recordings · 50m 11s · All clips included

Before variant allowances: 24.09% WER / 8.75% CER. See what changes ↓

How to read this A post-hoc text audit on public development sets, with the broadest allowances tested. These rates measure transcript agreement; conversational understanding has not been measured directly. This is not a blind holdout.

01 / Reading the errors

Follow the score, step by step.

692of 1,684 word-edit penalties removed under the broadest variant policy

A transcript can use another written form of a Derja word, join an article to a French word, or write a recognizable English phrase in Arabic script. Standard WER counts those differences too.

Fennec-only, post-hoc text audit. These adjusted scores are not used in the model comparison. The alternatives were reviewed from text, without audio adjudication.

Scoring policy Each step includes the previous allowances

More interpretive

Word error rate
14.19%
Character error rate
6.44%

992 word edits remain · 692 penalties removed

Also allow broader colloquial forms and contractions, including ناخذ / ناخو and على البحر / عالبحر. This policy needs more linguistic judgment and is reported separately.

Referenceناخذ
Fennecناخو

A broader colloquial allowance, beyond spelling alone.

View every scoring step
All 644 clips · unchanged reference denominators
PolicyWord editsWERCER
Standard scoring1,68424.09%8.75%
Allow spacing differences1,44620.69%8.75%
Allow spelling variants1,19717.12%7.92%
Allow another script1,11615.97%6.82%
Allow colloquial forms99214.19%6.44%

Small edits can still change meaning.

تنجم → تنجمش changes negation. عنده → عندها changes the pronoun. Neither is forgiven just because it is one character away.

Remaining edits are disagreements.

The broadest policy leaves 992 edits, including 249 one-character substitutions and 68 possible filler insertions. Listening is needed to distinguish recognition errors from omitted fillers or reference mistakes.

02 / The comparison

Same audio. Same scoring.

Download scores

Fennec has the lowest word error rate in this run. Every model below uses the same light text normalization. The Fennec-only variant allowances highlighted above are excluded so the comparison uses one scoring policy for everyone.

Recording subset

644 clips · 6,990 reference words Lower is better

Word error rate · All recordings. Same normalization for every model.
Model / configurationWord error rate
FennecOurs20261008-39663 · frozen CTC decoder
24.09%
LinTO TunisianVosk · CPU
31.57%
Whisper Tunisian CodeSwitchWhisper v3 · greedy / Arabic
34.06%
MAI-Transcribe 2Verbatim · automatic language
42.23%
GPT-TranscribeAutomatic language · no prompt
48.73%
Gemini 3.5 TranscribeAutomatic language · no prompt
63.13%
Qwen3-ASR 1.7BBF16 · batch 12
68.60%
Whisper large-v3Greedy decoding / Arabic
84.54%

All models completed the same clips. Empty transcripts and outputs that reached the generation limit remain in the scores. These results describe the configurations tested.

03 / Transcription speed

A short wait for the words.

Recorded local transcription time for Fennec on an NVIDIA B200, with the model already loaded. One clip at a time, across all 644 recordings.

Median per clip
34msHalf the clips finished within this time.
95th percentile
80ms95% of clips finished within this time.
Audio / processing time
115×Audio duration divided by total call time.

50m 11s of audio in 26.1 seconds of summed transcription calls. Real-time factor: 0.0087. This measures short clips, not continuous or streaming audio.

What the timer includes

Local per-clip transcription with the model already loaded, including audio handling and decoding. All clips included; no warm-up clips excluded. Batch size: 1. Excluded: Model loading, Upload and network time, Gateway overhead.

The GPU was shared with other jobs. These are observed model-call timings, not a service latency promise. API runs include network and provider time, and other local models used different batching, so we do not rank their speed against these numbers.

04 / Behind the numbers

The method, in the open.

One frozen set of recordings, retained outputs, and corpus-level error counts. Independent implementations agree on WER. Fennec’s direct output also matched the live gateway on three development clips.

Code-switched speech
300 clips
Tunisian-only speech
344 clips
Fennec release
20261008-39663
How WER and CER are calculated

WER is the sum of substituted, deleted and inserted words divided by the reference word count. CER uses character edits divided by reference characters, excluding whitespace. We aggregate counts across the corpus rather than averaging clip percentages. Insertions can take an error rate above 100%.

Light normalization uses NFKC, lowercase, Arabic vowel-mark and tatweel removal, and unified alef forms. Punctuation is spaced except internal apostrophes and numeric separators. French accents, negation, numbers and word boundaries are retained.

The variant audit permits explicit alternatives, rewrites only accepted hypothesis spans to the reference form, then recomputes WER and CER. The denominators remain 6,990 words and 30,060 characters. These allowances were chosen after inspecting Fennec’s outputs; apply the same frozen policy to all models before making an adjusted ranking.

Model configurations & output limits

Fennec

FP32, TF32 off; linear_union_prefix decoder, beam 100, alpha 0.3, beta 0.5, old/new LM weights 0.25/0.75.

0 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures

LinTO Tunisian

Local Vosk Tunisian model; standard 4,000-byte PCM chunks.

12 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures

Whisper Tunisian CodeSwitch

FP32, greedy decoding, language=ar, maximum 440 new tokens.

0 empty transcripts · 2 possible token-limit truncations · 0 unresolved service failures

MAI-Transcribe 2

microsoft/mai-transcribe-2 via OpenRouter; Azure enhanced mode, verbatim transcription, automatic language.

11 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures

GPT-Transcribe

gpt-transcribe through the OpenAI transcription API; no language hint or prompt.

11 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures

Gemini 3.5 Transcribe

google/gemini-3.5-transcribe via OpenRouter; automatic language, no prompt.

17 empty transcripts · 0 possible token-limit truncations · 0 unresolved service failures

Qwen3-ASR 1.7B

Stock Qwen3-ASR-1.7B, BF16, SDPA, left padding, batch 12, greedy decoding, maximum 512 new tokens. Automatic language.

11 empty transcripts · 3 possible token-limit truncations · 0 unresolved service failures

Whisper large-v3

Stock Whisper large-v3, FP32, greedy decoding, language=ar, maximum 440 new tokens.

0 empty transcripts · 7 possible token-limit truncations · 0 unresolved service failures

Language-hint follow-ups

Exploratory runs on the same audio. OpenAI received generic Tunisian context and Arabic, French and English hints; Gemini received an Arabic hint. No reference transcripts were supplied.

All 644 clips · standard normalization
ConfigurationWERCER
GPT-Transcribe · automatic48.73%25.08%
Gemini 3.5 Transcribe · automatic63.13%41.78%
GPT-Transcribe with hints43.93%20.20%
Gemini with Arabic hint93.16%47.02%

Gemini’s Arabic-hinted result includes one runaway transcript with 2,369 inserted words against a 10-word reference. It remains counted and substantially worsens the corpus score. Two initially failed requests succeeded on retry.

What this benchmark cannot establish
  • Blind generalization: these public sets were repeatedly inspected during Fennec development. The overlap check covered the original TunSwitch training manifest, not all training and decoder data.
  • Semantic correctness: the variant audit used text, not human adjudication of the recordings. Names, numbers and negation were not separately annotated.
  • Isolated speed: local runs shared hardware, while API calls included network time. Qwen used batch 12; its smoke outputs differed from single-clip inference on two of three clips. The final run uses the fixed batch configuration throughout.
  • Every possible model setup: Whisper used greedy decoding with Arabic specified. Provider aliases can change, and the commercial APIs did not expose immutable snapshots. No Claude transcription run is included.

Next questions

Did the meaning come through?

That deserves a test of its own. A useful next benchmark would have Tunisian listeners annotate held-out conversations, then check whether each transcript preserves the intent and the facts.

Meaning and key details

Intent, names, dates, amounts and negation, plus whether a summary captures what was said. Report these separately from spelling agreement.

Real conversation conditions

Noise, overlapping speakers and longer recordings. Measure upload-to-transcript latency, concurrent requests and cost per audio hour as separate service tests.

These are proposed measurements; this run does not establish them. Follow future results in Updates.