Decide whether a new model deserves the switch

What you need

Two models you can access, three representative tasks and an agreed spending limit. Preparing the sheet costs nothing. Actual model runs may use credits or API charges.

Read the Astra model reference

Work through the exercise

1. Choose three jobs from your real work: repair one small bug, extract facts from a supplied page, and write a creative brief. Replace client data with a shareable fixture when necessary. Write what a correct answer must contain before running either model.
2. Freeze the inputs and record the model names, date, reasoning setting where offered, available tools and maximum budget. Compare text models on text tasks. If one model has web access and the other does not, record that as a workflow comparison rather than a pure model ranking.
3. Run the same brief on both. Keep failures. Do not give one candidate repeated hidden hints while accepting the other's first attempt. If you allow a correction, permit the same correction budget for both and count the time spent writing it.
4. Hide the model names during review if practical. For a coding task, execute the relevant tests. For extraction, check every claimed fact against the fixture. For a Cirka creative brief, check that the supplied product constraints survive and the brief can be used without another interview.
5. Choose adopt only when the improvement matters to your workflow and survives the extra cost or latency. Choose hold when the sample is too small, access is uncertain or quality is mixed. Save the baseline and repeat after your next three real jobs. This is a practical trial, not a statistically conclusive benchmark.

You can complete this guide by saving the deliverable and review in your own folder. Sharing inside the club is optional and requires membership.

A prompt to adapt

Replace the bracketed inputs with your own approved material. Keep unknown facts marked unknown.

Evaluate the two anonymized outputs below against this task and source material.
Task: [task]. Sources: [approved inputs]. Acceptance criteria: [checks written before the run].
Output A: [paste]. Output B: [paste].
For each criterion mark pass, fail or cannot verify, with a concrete excerpt or observed result.
Separate factual errors from style preferences. Do not assume either output is correct.
List checks that require a human, code execution or access you do not have.
Recommend adopt, hold or reject only after I supply time, cost and independent test results.

Review your result

• Both candidates received equivalent inputs and correction budgets.
• Checks are grounded in sources or executed tests.
• Human corrections and failed runs count toward cost.
• The decision states what remains uncertain.

If it fails

If both answers pass, test a harder representative job or keep the cheaper workflow. If neither passes, repair the brief or tool access before concluding that the model is the problem.

Try recalling it later

Can a persuasive model judge replace running the code or checking the source?

Sources and tools

This is an original club exercise. The source-check date identifies this reference edition; it is not a result from running the prompt.

• Read the Astra model reference

Cirka is our founder's app. We use it as an optional creative workspace in these examples. A separate subscription or credits may be required. Check the tools and models actually available in your account; a model mentioned in a guide is not a claim of Cirka integration.

Continue from here

See how this exercise fits into the Vibe coding course, including its full public syllabus.

Browse all free guides for another task, or save your work and return to it later.