Skip to content
Software Development

The Counterweight Test: Three Questions Before Choosing an AI Coding Partner

Updated

The Counterweight Test is a pre-selection framework for AI coding partners. Instead of asking which model is smartest, it asks which model's defaults cover the gap in your own engineering temperament. The test consists of three self-directed questions and a practical evaluation step.

The Three Questions

  1. What do you over-supply? Identify the reflex you never need reminding to use: speed, invention, abstraction, or the reverse (caution, polish, completeness).
  2. What does your system quietly need protected? Usually the exact thing your over-supply runs over: boundaries, boring paths, narrow diffs, or alternatively, momentum and tolerance for rough edges.
  3. Whose defaults cover that gap without you asking? Not which model benchmarks highest, but which one drifts toward the thing you neglect when left alone on a real task.

The Practical Evaluation

Pick an underspecified task with two reasonable interpretations and drop it into a part of the codebase that already has conventions. Ambiguity forces the model back onto its defaults. Run the same task through two models and compare: does it ask or assume? Does it lean on existing structure or invent new abstractions? Does the diff stay narrow or spread?

Q&A

Why is an underspecified task better than a clear spec for evaluating a model?

A clear spec hides temperament because the model just follows instructions. A vague prompt forces the model back onto its working defaults, and those defaults are what you will live with across hundreds of sessions. The ambiguity reveals whether the model asks clarifying questions, invents structure, or traces existing conventions.

What does it mean to 'over-supply' a trait as a developer?

It means defaulting to a particular reflex so strongly that you produce too much of it. For example, a speed-oriented builder over-supplies invention and momentum but under-supplies patience with structure. A caution-oriented builder over-supplies polish and abstraction but under-supplies the willingness to ship rough work and learn from real usage.

How is the Counterweight Test different from checking benchmarks?

Benchmarks measure capability on a single axis and rank models from best to worst. The Counterweight Test treats model selection as a team-composition problem, where the right partner depends on what you already bring. A model that scores lower on a leaderboard might still produce better long-term results if its defaults compensate for your blind spots.

Can you change which model you use for different tasks?

Yes, and this is a key advantage over human pair programming. You can select a careful, boundary-respecting model for refactoring and a fast, inventive model for prototyping. With a human colleague, temperament is bundled with the person. With AI, temperament is a variable you set per task.