# NewEyes visual Q&A comparison

Use the [CSV scorecard](/solutions/benchmark-scorecard.csv) to compare the NewEyes pilot API, a direct Gemini API integration and a direct OpenAI API integration on your product. The template awaits measured trials across all three routes. Enter a measured zero only when the logs support it. Leave unknowns blank and record the gap.

## Prepare one shared test set

Choose one product task, such as reading a price from a menu. Give every case a stable ID, the exact image bytes and SHA-256 hash, the user question, the expected result and a pass rule. For the menu task, the rubric might require the requested item's amount and currency. Include examples where the image lacks enough information and define an acceptable clarification. Agree on the mix of easy and difficult cases, trial count and acceptance criteria before running the test.

Use the same device or test client, network profile, region, image bytes, user prompt, answer format, deadline and retry allowance for each route. Record the prompt template, model or deployment revision, preprocessing, image detail, generation settings and any tools each integration uses. Give every route access to equivalent product context and document any differences. Keep tuning cases separate from the final test set.

## Run one matched trial per route

A trial is one case at one repeat number, sent through one route. Create the planned rows before running. Rotate route order within each case, spread repeats across the same time window, and record cache and warm-up conditions. Save the terminal outcome even if a call times out. A retry adds a call to the original trial.

Keep one row per trial in a private log. These headers can be pasted into a CSV file.

```csv
case_id,repeat_id,route,model_or_build,started_utc,elapsed_ms,outcome,call_count,answer_ref,pass_grade,usage_cost_usd,cost_evidence_ref,trace_ref
```

`outcome` is `usable_answer`, `timeout`, `error` or `invalid_response`. A valid clarification can count as a usable answer if the rubric allows it. Set `pass_grade` to 1 for a rubric pass and 0 for a failed or incorrect answer. Leave the grade blank while review is pending. Keep an itemized record for each billed call, including failed calls and retries. Reference images, responses and traces by their approved internal IDs.

## Fill in the scorecard

- **Accuracy and completion.** Hide route names from graders, score all usable answers against the same rubric and resolve disagreements. Count terminal failures as zero for task accuracy. Passed tasks plus wrong answers plus failed trials should equal attempted trials once grading is complete. Show how many planned trials were attempted. If one route missed assignments, report those gaps before comparing rates and compare the same matched cases across routes.
- **Latency.** Start the client timer when the image and question are ready to send. Stop when the complete answer is presented. This includes transmission, preprocessing, model work, retries and rendering. Report median and P95 for usable answers alongside the number timed and the timeout count. Use the inclusive percentile for P95, for example `PERCENTILE.INC` in Excel. Retain elapsed time to terminal failure separately in the log.
- **Cost.** Reconcile billed usage across successful calls, failures and retries. Include applicable image, text, cached input, output, tool and per-request infrastructure charges. For estimates, save the units, rates, tier and effective date. Use the agreed NewEyes quote or billing record, and the provider's rate schedule or invoice for each direct integration. Keep fixed fees separate. Report unit costs only when all attempted trials have known costs and the denominator is positive. [Gemini API pricing](https://ai.google.dev/gemini-api/docs/pricing) · [OpenAI API pricing](https://developers.openai.com/api/docs/pricing)
- **Engineering effort.** Give each route the same acceptance checklist for capture, API setup, answer rendering, failure handling, usage tracking and QA. Record actual person-hours by activity and note reused components. Count common client work consistently. State the hourly rate if converting effort to money.

Record results for each task and operating condition before pooling them. Keep counts beside percentages, distinguish unique cases from repeated trials, and leave metrics with a zero denominator blank. Share the completed scorecard with its test window, model versions, grading rules and evidence references.
