Synthetic inputs. Measured locally. Not a customer report.

A dimension became
an item count.

This example shows the kind of evidence an assessment can contain. It comes from a small local model experiment, not a production application.

Decision summary

The model did not consistently satisfy this extraction task. Investigate unit handling, reference preservation, and output types before relying on these fields. This limited experiment is not a release approval or a certification.

01 / The task

Extract an order reference, requested action and item count from a support message.

The label says 12 cm. Please cancel H-808.

{
  "order_id": "H-808",
  "intent": "cancel",
  "quantity": null
}

{
  "order_id": null,
  "intent": "cancel",
  "quantity": "12"
}

02 / What we observed

• The order reference was lost.
• A dimension was extracted as an item count.
• The quantity was a string, not the required integer or null.

03 / Why it matters

An application trusting these fields could take an incorrect downstream action. This is a possible impact, not an observed customer incident.

04 / Recommendations

Validate output types and units. Add regression cases that distinguish measurements from counts and preserve references. Assess any model or prompt changes against a fresh held-out set.

05 / Experiment context

Hardware

Apple M5, 16 GB, macOS 26.0

Model

Qwen3 1.7B, MLX 4-bit

Runtime

MLX 0.28.0 / MLX-LM 0.26.3

Test set

15 synthetic cases, repeated 3 times

Overall correct outputs

34 / 45 (75.6%)

Median response time

0.489 seconds

Peak MLX allocation

1.434 GB; not total process memory

A small synthetic set cannot establish production reliability. Repetitions are not independent examples. No iPhone, battery or thermal measurements are claimed.

06 / Reproduction and walkthrough

This finding is case test-05 in the recorded run. The same case failed all three repetitions. Reproduce it with the recorded model and template revisions, input, and task contract; changing the prompt, runtime, or sampling settings may change the result.

In a customer walkthrough, we would explain which findings matter to the agreed workflow, separate observed behavior from unverified causes, and discuss what to address or test next. That walkthrough is a service deliverable, not something claimed to have occurred for this synthetic example.