| Model | Quality | p99 latency | Cost per request | vs today | Result |
|---|---|---|---|---|---|
| Gemini 4 Argon Today's modelopenrouter/google/gemini-4-argon | 88.3% replayed on 180 gold rows | 6,059 ms benchmark replies on the same messages | ₹0.122 benchmark replies on the same messages | — | Today |
| dvar:gemma-4-31b-dhan-assist-ftopenai_compatible/dvar:gemma-4-31b-dhan-assist-ft | 96.3% replayed on 180 gold rows | 2,177 ms replayed on 180 gold rows | ₹0.0037 replayed on 180 gold rows | −97% | Passes |
| openai_compatible/dvar:gemma-4-31b-it | 55.6% replayed on 180 gold rows | 2,347 ms replayed on 180 gold rows | ₹0.0036 replayed on 180 gold rows | −97% | Fails 3 checks |
dvar:gemma-4-31b-it is fully right on 55.6% of gold rows, under the 90.0% quality bar; fails a critical check (investment advice, a fake order confirmation or a PII leak) on 39 gold rows. Dhan's release rule allows none; falls below the minimum on Orders and order status: 0.56 vs 0.90 (36 gold rows); F&O margin: 0.56 vs 0.90 (36 gold rows); KYC and account: 0.49 vs 0.90 (36 gold rows); Algo and backtest: 0.56 vs 0.90 (36 gold rows); Complaints and escalation: 0.60 vs 0.90 (36 gold rows). Weakest topic: KYC and account 0.49 | |||||
The fine-tuned model is Gemma 4 31B tuned on masked Dhan Assist chats (synthetic for this demo). Each round, its misses become training data: Fine-tune. | |||||
dvar:gemma-4-31b-dhan-assist-ft passes every check: 8.0 points higher than today's model at 97% less cost. Indicative only: not every judge check is trusted.
Prices are per request, at each provider's list price.
Judge: 3 of 5 checks trusted.
| Topic | Rows | Today · Gemini 4 Argon | Candidate · dvar:gemma-4-31b-dhan-assist-ft | Candidate · dvar:gemma-4-31b-it | ||
|---|---|---|---|---|---|---|
| Orders and order status | 36 | 0.92 | 0.97 | +0.05 | 0.56 | −0.36 |
| F&O margin | 36 | 0.89 | 0.95 | +0.06 | 0.56 | −0.33 |
| KYC and account | 36 | 0.81 | 0.96 | +0.15 | 0.49 | −0.32 |
| Algo and backtest | 36 | 0.89 | 0.94 | +0.05 | 0.56 | −0.33 |
| Complaints and escalation | 36 | 0.91 | 0.98 | +0.07 | 0.60 | −0.31 |
| All topics, weighted by traffic | 180 | 0.88 | 0.96 | +0.08 | 0.56 | −0.32 |