research · benchmark report · october 2026
the togi support benchmark.
753 customer-support messages in hindi, roman-script hindi, tamil and telugu, written the way people actually type. each message is scored three ways: which of eight intents it carries, how urgent it is on a scale of one to five, and which language the reply should be in. twelve models are scored on it, then on two datasets none of them were trained on. every number from the report is on this page, including the test we lose.
01 · the benchmark
hindi is nearly solved. tamil and telugu are not. urgency is unsolved.
choose a metric. all twelve models answer the same 753 messages with the same eight options. the togi models are the solid bars.
share of the 753 messages assigned the correct intent out of eight.
- decide-v1 · 4B0.98
- decide-v0 · 4B0.96
- clef-8bit · 27B0.88
- pplx-decider-flash0.86
- clef-flash0.83
- kev-9b · 9B0.80
- yama · 586M0.78
- kev-4b · 4B0.72
- laya-indic0.71
- strands-decider-2b · 2B0.69
- kev-0.8b · 0.8B0.53
- laya-multilingual0.45
three checks the chart cannot show
| decide-v1 | decide-v0 | pplx-decider-flash | clef-8bit | yama | |
|---|---|---|---|---|---|
| intent accuracy, 314 messages with no near-duplicate in training | 0.97 | 0.94 | 0.86 | 0.84 | 0.76 |
| tamil intent accuracy, same 314 | 0.98 | 0.94 | 0.86 | 0.81 | 0.84 |
| hard set: 24 messages every released model got wrong, solved | 20 | 10 | 0 | 0 | 6 |
02 · the script penalty
one situation, four ways of writing it.
intent accuracy by language bucket for every released model. read across a row to see what a model loses when hindi moves from devanagari to roman script, and again when the language is tamil or telugu.
03 · out of domain
massive indic: eighteen scenarios, human-localised.
500 voice-assistant requests per language from the massive test split, localised by people rather than translated by machine. yama was trained on this dataset. pplx-decider-flash and decide-v0 never saw it. decide-v1 did.
04 · unseen data
where it holds, and where we lose.
cause-and-effect reasoning, a retail intent schema no model had seen, and the benchmark re-scored with the answer order shuffled. the mint mark flags the test pplx-decider-flash wins.
- decide-v10.848
- pplx-decider-flash0.737
- decide-v00.642
- yama0.635
- decide-v10.842
- yama0.644
- pplx-decider-flash0.622
- decide-v00.540
- pplx-decider-flash0.955
- decide-v10.940
- decide-v00.930
- yama0.900
pplx-decider-flash leads by 1.5 points on the unseen retail schema. on cause and effect, decide-v1 leads by 11 points overall and by 22 on tamil, though its probabilities there are over-confident. shuffling the answer order changes 1.2% of decide-v1's answers and none of its accuracy. pplx-decider-flash loses 3 points when the order changes.
| model | calibration error, cause and effect ↓ | calibration error, retail ↓ | answers unchanged when options are shuffled | accuracy, original → shuffled |
|---|---|---|---|---|
| decide-v1 | 0.211 | 0.023 | 98.8% | 0.979 → 0.979 |
| decide-v0 | 0.207 | 0.042 | 99.2% | 0.963 → 0.961 |
| pplx-decider-flash | 0.116 | 0.079 | 91.2% | 0.858 → 0.827 |
| yama | 0.186 | 0.142 | 91.4% | 0.776 → 0.776 |
05 · the recipe
how decide-v1 was trained.
- torso
- gemma-4-E4B-it
- adapter
- low-rank adapter, rank 16, on attention and mlp · 36.2M trainable parameters
- head
- a pointer head that scores each option in one forward pass
- data
- 272,577 training examples, one pass
- compute
- 8,518 steps · 10.3 hours on one nvidia L40S
- validation
- final validation accuracy 0.9725
- why gemma
- 2.1 tokens per tamil word, against 3.8 for qwen3.5
06 · caveats
what this report does not show.
- 01the benchmark is in-distribution for decide: the same intents, the same writing style and the same urgency scale it was trained on. on the 314 messages with no near-duplicate in training, intent accuracy is 0.97.
- 02urgency labels follow our own one-to-five scale. there is no public standard to compare against.
- 03the massive and sib-200 training sets are translations, not native text. the retail test set was written by a language model.
- 04on cause-and-effect questions decide-v1 is over-confident: expected calibration error 0.21 against 0.12 for pplx-decider-flash. calibrating on an unseen task is the first item for v2.
- 05none of this yet measures real support tickets or text written by native speakers in the wild.
the full table, every model, every metric
| model | intent | urgency, exact | reply language | confident and wrong ↓ | p50 ms |
|---|---|---|---|---|---|
| decide-v1 | 0.98 | 0.80 | 0.98 | 0.01 | — |
| decide-v0 | 0.96 | 0.77 | 0.98 | 0.03 | 168 |
| clef-8bit | 0.88 | 0.34 | 0.57 | 0.04 | 483 |
| pplx-decider-flash | 0.86 | 0.28 | 0.76 | 0.03 | 72 |
| clef-flash | 0.83 | 0.29 | 0.79 | 0.04 | 100 |
| kev-9b | 0.80 | 0.32 | 0.77 | 0.01 | 134 |
| yama | 0.78 | 0.21 | 0.37 | 0.06 | 125 |
| kev-4b | 0.72 | 0.24 | 0.67 | 0.00 | 136 |
| laya-indic | 0.71 | 0.25 | 0.23 | 0.06 | 61 |
| strands-decider-2b | 0.69 | 0.23 | 0.55 | 0.01 | 150 |
| kev-0.8b | 0.53 | 0.22 | 0.04 | 0.01 | 73 |
| laya-multilingual | 0.45 | 0.23 | 0.25 | 0.20 | 57 |