Found while fixing the litellm pricing breakage (#3722).
What happens
OpenAIProvider._get_pricing() falls back to (2.50, 10.00) for any model it cannot resolve — headroom/providers/openai.py:174, commented "GPT-4o tier as reasonable default". Nothing distinguishes that fallback from a real lookup at the call site.
litellm.model_cost is downloaded from GitHub at import time and BerriAI prunes retired models from it. On 2026-09-23 it dropped groq/llama-3.3-70b-versatile outright. So today:
_get_pricing("groq/llama-3.3-70b-versatile") -> (2.50, 10.00)
Groq's actual rate for that model was $0.59 / $0.79 per million. Headroom now reports it at $2.50 / $10.00 — roughly 4x the input cost and 12x the output cost — and says nothing.
tests/test_pricing_from_litellm.py::test_provider_prices_models_its_table_never_covered caught it, because its own comment names $2.50/$10.00 as the "unknown default" it was written to detect. That test is currently red on main for this reason.
Why it matters beyond a wrong number
check_budget is a hard control — it refuses requests. tests/test_cost_budget_basis.py exists precisely because a budget decision made on a guess used to be indistinguishable from one made on measured data, and that work added an "estimated basis" marker so the guess was visible.
This fallback reintroduces the same class of problem one layer down: the price is a guess, the basis marker says nothing about it, and a deployment can be refused on a number Headroom invented. A model priced 12x too high exhausts a budget 12x too fast.
It is also silent in the other direction — a model genuinely more expensive than GPT-4o gets under-billed.
Suggested fix
Make "unknown" representable rather than substituted:
- Have the resolver return
None (or a PriceLookup carrying resolved: bool) instead of a sentinel tuple, and let callers decide.
- Mark any cost derived from a fallback price the way
record_tokens already marks an estimated token basis, so check_budget and the cost card can say the number is a guess.
- Log once per unresolved model id — today there is no signal at all that a lookup failed.
- Keep a default only where a number is genuinely required, and make it visibly a default.
Minimum viable fix is (3) plus surfacing the flag in the cost card; the budget-refusal path is the part that should not run on a fabricated price.
Related
Found while fixing the litellm pricing breakage (#3722).
What happens
OpenAIProvider._get_pricing()falls back to(2.50, 10.00)for any model it cannot resolve —headroom/providers/openai.py:174, commented "GPT-4o tier as reasonable default". Nothing distinguishes that fallback from a real lookup at the call site.litellm.model_costis downloaded from GitHub at import time and BerriAI prunes retired models from it. On 2026-09-23 it droppedgroq/llama-3.3-70b-versatileoutright. So today:Groq's actual rate for that model was $0.59 / $0.79 per million. Headroom now reports it at $2.50 / $10.00 — roughly 4x the input cost and 12x the output cost — and says nothing.
tests/test_pricing_from_litellm.py::test_provider_prices_models_its_table_never_coveredcaught it, because its own comment names$2.50/$10.00as the "unknown default" it was written to detect. That test is currently red onmainfor this reason.Why it matters beyond a wrong number
check_budgetis a hard control — it refuses requests.tests/test_cost_budget_basis.pyexists precisely because a budget decision made on a guess used to be indistinguishable from one made on measured data, and that work added an "estimated basis" marker so the guess was visible.This fallback reintroduces the same class of problem one layer down: the price is a guess, the basis marker says nothing about it, and a deployment can be refused on a number Headroom invented. A model priced 12x too high exhausts a budget 12x too fast.
It is also silent in the other direction — a model genuinely more expensive than GPT-4o gets under-billed.
Suggested fix
Make "unknown" representable rather than substituted:
None(or aPriceLookupcarryingresolved: bool) instead of a sentinel tuple, and let callers decide.record_tokensalready marks an estimated token basis, socheck_budgetand the cost card can say the number is a guess.Minimum viable fix is (3) plus surfacing the flag in the cost card; the budget-refusal path is the part that should not run on a fabricated price.
Related