One Gateway, Different Models: Keep the Evaluation Contract Stable

Mistral Large 4 becoming open weight and OpenAI updating its Model Spec are reminders that “the model” is only one layer of an AI application. Mistral announcement · OpenAI update.\n\nDevelopers still need a repeatable request boundary, a model identifier taken from the current catalog, response validation, timeouts, logs that do not leak credentials, and a fallback policy. RelayRouter can reduce the connection work for supported routes by giving that boundary one place to live. It does not make every model identical, so capability checks remain part of the adapter.\n\nA beautiful generated frame can hide the exact failure your application cares about. An object may look right until the camera turns. A route may look plausible until someone tries to return along it.

Genie 4 was announced on October 27 as a limited research preview for interactive worlds. For developers, it raises a practical testing question: what must remain true as a scene changes? Official announcement.

The following is a proposed evaluation method for an interactive prototype. It is not a Genie API tutorial, and no generated-world benchmark is claimed here.

1. Define one invariant before judging the scene

Take a fictional ceramics showroom. Place one ivory cup on a walnut table beside a large window. The viewer walks around the table and returns to the entrance.

Decide what matters before the run: the cup stays ivory, there is one handle, it remains on the same table, and the route back to the entrance remains coherent. Decorative plants and the exact pattern of clouds might be less important for this particular test.

This distinction prevents an evaluation from becoming a contest of attractive still images. It also makes disagreements between reviewers easier to resolve: they are checking a stated property rather than relying on an overall impression.

2. Write a short interaction sequence

Use a reproducible sequence instead of random exploration. The example below is an application-level test plan you can store as JSON. It is not a model request schema.

{
  "case_id": "showroom-return-v1",
  "scene": "One ivory cup on a walnut table near a window",
  "steps": [
    "Observe the cup from the entrance",
    "Move to the left side of the table",
    "Change the light to late afternoon",
    "Return to the entrance viewpoint"
  ],
  "invariants": [
    "Exactly one cup remains visible where expected",
    "The cup keeps one handle and its ivory color",
    "The table and return route remain coherent"
  ],
  "review_status": "not_run"
}

Keep the initial state, interaction instructions, and resulting observations together. If the tool supports saved sessions or reproducible seeds, record them according to its own documentation. Do not assume a seed makes a nondeterministic system perfectly repeatable.

3. Inspect the transitions, including the return

Capture evidence at meaningful checkpoints: initial view, side view, edited lighting, and return view. Note where an invariant changes and whether the issue survives after the scene settles.

If audio is part of the experience, review it separately. A convincing scene with an unrelated sound event may be unsuitable for the intended interaction even if individual frames look excellent. Define what the user should hear in the same way you define what they should see.

The official Genie page identifies consistency and spatial limitations as continuing research areas. Use those limitations to select cases, not to invent measurements. Model page.

4. Distinguish four failure classes

Record uncertainty explicitly. A reviewer may be unable to tell whether an object is occluded or missing. That should be an unresolved observation, not automatically a success or failure.

5. Use an ordinary text route to prepare the brief

You can develop the planning part before you have access to a specialist world model. A text route can propose a checklist from a synthetic brief. Your application then validates the structure and a person decides whether the checklist is useful.

The included request_checklist.py demonstrates this bounded planning step through a documented Chat Completions-compatible route in RelayRouter. Configure RELAYROUTER_BASE_URL, RELAYROUTER_API_KEY, and RELAYROUTER_MODEL from your current console. It returns a proposal marked proposal_needs_review.

That status is deliberate. A text model describing a scene has not rendered a world. A generated checklist has not performed a test. Keep planning, execution, observation, and approval as distinct records.

6. Make the acceptance decision specific

For a mood exploration, small decorative drift might be acceptable. For product identity or spatial planning, the same drift could invalidate the artifact. Decide which class of use you are evaluating before comparing tools.

RelayRouter can provide the API connection for available planning and text routes; verify specialized media support separately. Start with the available routes, prepare one reviewable case, and preserve the sequence that explains the outcome. The useful evidence is what happened between the frames.