Skip to content
Stance

AI testing
on real Macs.

We assess on-device AI behavior in Mac apps.
You get clear findings, evidence and recommendations.

Discuss your project
Checked by a human.
  • Independent
  • Hands-on
  • Mac first

Small mistakes.
Real consequences.

Synthetic example
The model returned

quantity: "12"

A dimension became an item count. The order reference was also lost.

A dimension is not an item count.

01 / How it works

One assessment. A clear handover.

  1. 01

    Agree the scope

    We define the goals, workflows, target devices, and expected behavior with you.

  2. 02

    Design the tests

    We build a tailored, repeatable set of cases around the questions that matter.

  3. 03

    Run and investigate

    We run the tests, record the results, and investigate the important discrepancies.

  4. 04

    Report + walkthrough

    You get the report, supporting evidence, and a walkthrough of the findings.

Timing is agreed after scope and access are confirmed.

You receive:

Assessment report
What worked, what did not, and what to address next.
Reproducible evidence
Inputs, outputs, and steps to inspect the findings.
A findings walkthrough
A conversation about the results and your next decision.

Before testing, we agree on access and boundaries. New builds, missing access, or changed requirements may change the schedule.

Return for a separately scoped retest or your next release. No ongoing commitment.

02 / The evidence

See exactly
what failed.

The input. The answer. The expectation.
A finding you can understand and reproduce.

Synthetic experiment / Not a customer report

Input

The label says 12 cm.
Please cancel H-808.

Observed

quantity: "12"
order_id: null

A measurement became an item count. The order reference was lost.

Expected

quantity: null
order_id: "H-808"

No quantity was supplied. The order to cancel was explicit.

Why it matters An app trusting these fields could take an incorrect downstream action. This is a possible impact, not an observed customer incident.

Measured locally / 6 September 2026 (IST)

A result you
can inspect.

One small model experiment. Every square below represents a recorded run, grouped by test case.

01
02
03
04
05
06
07
08
09
10
11
12
13
14
15
Correct output Incorrect output

34 / 45correct outputs

11 of 15 cases correct on every repeat.

15 synthetic cases, repeated three times. Repeats are not independent examples, and this is not a production reliability estimate.

Qwen3 1.7B / MLX 4-bitApple M5 / 16 GB

03 / The assessment

What we assess.

On-device AI behavior, against agreed expectations.
A focused assessment of the workflows that matter to your release.

01

Correctness

Check outputs against an agreed answer or rubric, including the details that change the outcome.

Is the answer actually right?
02

Grounding

Trace claims and citations back to the supplied source. Separate supported answers from invented details.

Where did that claim come from?
03

Missing information

Test whether the model acknowledges gaps or asks for clarification instead of confidently filling them in.

Does it know when it does not know?
04

Instruction following

Check the requested format, constraints, and task boundaries, including distracting or conflicting input.

Did it follow the contract?
05

Consistency

Repeat cases and vary inputs deliberately to find unstable behavior. Similar wording is not always the same task.

Does the result hold up on another run?

Mac applications first. Standalone local models by agreed scope. General app QA, fixes, fine-tuning, and certification are not included.

04 / What you receive

Understand the failure.
Know the next step.

Not just a score. A clear record of what happened,
why it matters, and what to investigate or address next.

Assessment report

The version, workflows and devices assessed. Prioritized findings, observed impact, recommendations, and the limits of the assessment.

Understand the risk

Reproducible evidence

Inputs, expected behavior, actual outputs, reproduction steps, and relevant repeat-run results. Observations are separate from suspected causes.

Inspect the details

A findings walkthrough

A conversation about the important failures, their practical significance, and the next steps for your team.

Leave with clarity

05 / Independent. Hands-on.

Engineering-led.
Evidence-first.

We assess on-device AI behavior against clear requirements, investigate failures, and turn the results into reproducible evidence and practical recommendations. From scoping to handover, our focus is rigorous testing and clear communication.

Verified assessment environmentApple M5 / 16 GB / macOS 26.0

Additional device coverage is confirmed per engagement. Results apply to the configurations and cases actually assessed, not every Mac or every possible input.

Before we start

What do you need from us?

Access to the app or local model, the version to assess, intended behavior, target devices, and your priorities. We agree on a suitable access arrangement before testing. Please do not send confidential files through this website.

What does an assessment include?

Agreed AI workflows, a test set and evaluation criteria, recorded findings and evidence, prioritized recommendations, and a walkthrough. Coverage and limitations are explicit in the report.

Do we need to prepare all the test cases?

No complete suite is required. We design cases around your goals, and your team helps clarify domain-specific expectations and what a correct result means.

Will you fix the issues?

No. The engagement covers assessment, evidence, and recommendations. Your team implements changes. General application QA, fine-tuning, optimization, and certification are outside this scope.

How long does it take?

We agree on a delivery date after confirming scope and access. New builds, missing access, or changed requirements may change the schedule. There is no universal turnaround promise.

What happens after the report?

The first assessment is a standalone engagement. You can return for a separately scoped verification of fixes or an assessment of a future release. There is no subscription or automatic commitment.

Start with one assessment.

Tell us about your on-device AI workflow and the decision you need to make.

Checking form availability. All fields except app, model, or company are required. Please do not send confidential information. You can also email vaibhav@stance.run.

What are you providing?

Loading security check...

STANCE / EXAMPLE ASSESSMENT

Synthetic inputs. Measured locally. Not a customer report.

A dimension became
an item count.

This example shows the kind of evidence an assessment can contain. It comes from a small local model experiment, not a production application.

Decision summary

The model did not consistently satisfy this extraction task. Investigate unit handling, reference preservation, and output types before relying on these fields. This limited experiment is not a release approval or a certification.

01 / The task

Extract an order reference, requested action and item count from a support message.

The label says 12 cm. Please cancel H-808.

Expected

{
  "order_id": "H-808",
  "intent": "cancel",
  "quantity": null
}

Observed

{
  "order_id": null,
  "intent": "cancel",
  "quantity": "12"
}

02 / What we observed

  • The order reference was lost.
  • A dimension was extracted as an item count.
  • The quantity was a string, not the required integer or null.

03 / Why it matters

An application trusting these fields could take an incorrect downstream action. This is a possible impact, not an observed customer incident.

04 / Recommendations

Validate output types and units. Add regression cases that distinguish measurements from counts and preserve references. Assess any model or prompt changes against a fresh held-out set.

05 / Experiment context

Hardware
Apple M5, 16 GB, macOS 26.0
Model
Qwen3 1.7B, MLX 4-bit
Runtime
MLX 0.28.0 / MLX-LM 0.26.3
Test set
15 synthetic cases, repeated 3 times
Overall correct outputs
34 / 45 (75.6%)
Median response time
0.489 seconds
Peak MLX allocation
1.434 GB; not total process memory

A small synthetic set cannot establish production reliability. Repetitions are not independent examples. No iPhone, battery or thermal measurements are claimed.

06 / Reproduction and walkthrough

This finding is case test-05 in the recorded run. The same case failed all three repetitions. Reproduce it with the recorded model and template revisions, input, and task contract; changing the prompt, runtime, or sampling settings may change the result.

In a customer walkthrough, we would explain which findings matter to the agreed workflow, separate observed behavior from unverified causes, and discuss what to address or test next. That walkthrough is a service deliverable, not something claimed to have occurred for this synthetic example.

Model and test provenance
Model revision
21457c6f51ed54a7c16e988c0844db973815c137
Official chat-template revision
70d244cc86ccca08cf5af4e1e306ecf908b1ad5e
Run identifier / case
20260905T204632308820Z / test-05