Synthetic inputs. Measured locally. Not a customer report.
A dimension became
an item count.
This example shows the kind of evidence an assessment can contain. It comes from a small local model experiment, not a production application.
01 / The task
Extract an order reference, requested action and item count from a support message.
The label says 12 cm. Please cancel H-808.
Expected
{
"order_id": "H-808",
"intent": "cancel",
"quantity": null
}Observed
{
"order_id": null,
"intent": "cancel",
"quantity": "12"
}02 / What we observed
- The order reference was lost.
- A dimension was extracted as an item count.
- The quantity was a string, not the required integer or null.
03 / Why it matters
An application trusting these fields could take an incorrect downstream action. This is a possible impact, not an observed customer incident.
04 / Recommendations
Validate output types and units. Add regression cases that distinguish measurements from counts and preserve references. Assess any model or prompt changes against a fresh held-out set.
05 / Experiment context
- Hardware
- Apple M5, 16 GB, macOS 26.0
- Model
- Qwen3 1.7B, MLX 4-bit
- Runtime
- MLX 0.28.0 / MLX-LM 0.26.3
- Test set
- 15 synthetic cases, repeated 3 times
- Overall correct outputs
- 34 / 45 (75.6%)
- Median response time
- 0.489 seconds
- Peak MLX allocation
- 1.434 GB; not total process memory
A small synthetic set cannot establish production reliability. Repetitions are not independent examples. No iPhone, battery or thermal measurements are claimed.
Model and test provenance
- Model revision
- 21457c6f51ed54a7c16e988c0844db973815c137
- Official chat-template revision
- 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
- Run identifier / case
- 20260905T204632308820Z / test-05