test(pricing): accept litellm's off-peak DeepSeek rate in cost_per_token check - #3684
Closed
Ayushraj06-bit wants to merge 1 commit into
Closed
Ayushraj06-bit wants to merge 1 commit into
Ayushraj06-bit wants to merge 1 commit into
Conversation
…ken check `test_cost_per_token_resolves_deepseek_v4_flash` compares `litellm.cost_per_token()` against the flat `input_cost_per_token` of the active `litellm.model_cost` entry. Between 2026-09-18 22:00 and 2026-09-19 12:00 UTC litellm's live pricing map gained an `off_peak_pricing` block for `deepseek-v4-flash` (input 0.15/M vs 0.30/M), and litellm 1.101.0 — the version in uv.lock — bills from it during DeepSeek's off-peak windows: Mon–Fri 10:00–01:00 and 04:00–06:00 UTC, all weekend. The two sides of the assertion now disagree for ~17 hours of every weekday and all weekend, so the test fails on `main` and on every PR whose `test (3)` shard runs then (main run 35462876818 at 19:21 UTC, headroomlabs-ai#3668 at 16:26 UTC Sunday) and passes only in the peak hours (headroomlabs-ai#3679 at 03:12 UTC Monday). Accept either tier from the active entry. With a litellm that has no off-peak block the set collapses to the single flat rate, so nothing changes there; with one, the assertion still pins cost_per_token to the entry's own rates and stays clock-independent. Verified under litellm 1.101.0 with the clock pinned to the failing instant (Fri 19:21 UTC): unpatched `assert 0.15 == 0.3`, patched passes; both pass at a peak instant.
Ayushraj06-bit
requested review from
DevanshiVyas,
JerrettDavis and
chopratejas
as code owners
September 21, 2026 04:27
Contributor
PR governanceThis PR follows the template and is marked ready for human review. |
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
21 tasks
This was referenced Sep 21, 2026
21 tasks
Collaborator
|
This test-only change is now superseded by merged #3699, which fixed the same wall-clock-dependent DeepSeek assertion on current |
Contributor
Author
|
Superseded: #3699 landed on main with the same clock-independent assertion (list rate or the documented off-peak half), so this is no longer needed. Closing. |
chopratejas
added a commit
that referenced
this pull request
Sep 23, 2026
…ired one (#3722) ## What broke `litellm` downloads `model_prices_and_context_window.json` from GitHub **at import time**, so `litellm.model_cost` is live third-party data we do not control. On 2026-09-23 BerriAI pruned retired Anthropic ids from that table. `claude-sonnet-4-20250514` disappeared, and every test that priced it started failing with `KeyError: 'input_cost_per_token'` — on every open PR simultaneously, with no change on our side. `main` was green at `5ff4ea1e` on 09-22 and the identical shards went red on #3720 and #3721, which share that base and touch no pricing code. ## What this does The model id in those tests is **incidental**. They assert that Headroom's cost arithmetic agrees with litellm's numbers — not that any particular model is priced correctly. So `tests/_pricing_models.py` asks for *a* model carrying the fields a test needs, newest first, and raises with an actionable message if litellm prices none of them. ```python MODEL = anthropic_pricing_model( "input_cost_per_token_above_200k_tokens", "cache_creation_input_token_cost", "cache_read_input_token_cost", ) ``` Retiring one candidate is now a no-op until the list runs dry. ## Why not pin to the wheel's vendored copy That was this PR's first attempt and **it was wrong** — I'm recording why, because it looks like the obvious fix. The live table and the copy vendored in the litellm wheel are *complementary, not ordered*: | | live | vendored | |---|---|---| | `claude-sonnet-4-20250514` | ✗ pruned | ✓ | | `claude-sonnet-5` | ✓ | ✗ predates it | | live-only entries | 1743 | — | | vendored-only entries | — | 314 | Pinning fixed the 25 `KeyError`s and **broke `test_proxy_savings_history`**, which prices `claude-sonnet-5`. Taking the union does not rescue it either: the vendored `claude-sonnet-4-20250514` entry is an older schema that has no `input_cost_per_token_above_200k_tokens` field at all, so `test_cost_card_attribution` still fails against it. ## A second, separate breakage in the same data drop `test_provider_prices_models_its_table_never_covered[groq/llama-3.3-70b-versatile]` was failing for a different reason: litellm dropped that model too, so `_get_pricing` fell through to the `$2.50/$10.00` GPT-4o default — which that test's own comment names as the "unknown default" it exists to catch. Repointed at `groq/llama-guard-3-8b`, which litellm still prices. **The product-side problem is filed separately as #3732**: an unresolvable model is silently billed at the GPT-4o tier, so cost tracking and `check_budget` — a hard control that refuses requests — can act on a fabricated number. For that groq model the default overstated output cost by ~12x. ## Verification Same six modules, same machine: ``` origin/main 25 failed, 82 passed this branch 107 passed ``` Ruff 0.16.7 `check` + `format --check` clean. ## Prior art #3683 / #3684 was the same class of failure (litellm started billing DeepSeek off-peak) and was patched narrowly at the assertion. This removes the category for model *identity*; #3732 covers what happens in production when identity goes missing.
JerrettDavis
pushed a commit
that referenced
this pull request
Sep 24, 2026
…3700) ## Description Bounds the `int()` conversion added by #3616 to the range where a double is exact, so a legacy `.xls` cell stops inventing digits (#3695). `_xls_cell` rendered any integral double with `int(value)`. xlrd hands back an IEEE-754 double, and above 2**53 consecutive integers are not representable, so the conversion prints the double's *exact* value rather than the number the sheet held: ```text typed 123456789012345678 stored 1.2345678901234568e+17 rendered 123456789012345680 ``` Two trailing digits, fabricated, delivered to an agent in a form that reads as an exact identifier. Before #3616 both loaders rendered the float and simply disagreed about `12.0` vs `12`; after it they agree on small whole numbers and disagree again -- in the worse direction -- at the top of the range, which is also where the PR's own goal ("render an .xls cell the way the .xlsx loader renders it") stops holding. Closes #3695 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/transforms/spreadsheet_ingest.py`: the numeric branch becomes `if kind == xlrd.XL_CELL_NUMBER and float(value).is_integer() and abs(value) <= 2**53:`, so values past the exactly-representable range`return value` and render as the float. The comment records why the bound exists so it is not "simplified" away again. - `tests/test_transforms_tabular.py`: adds `test_load_xls_and_xlsx_agree_above_the_exact_integer_range`, written against the same xlwt/xlrd fixture style the three existing `.xls` tests use. It pins both halves of the requirement: `12` still renders without a decimal point (the #3616 win survives the bound), and 1.2345678901234568e+17 renders as `repr()` of the double rather than an invented integer. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed Type checking was run on the changed module rather than the whole package, so that line means `mypy --config-file pyproject.toml headroom/transforms/spreadsheet_ingest.py`, not a full-package pass. ### Test Output Red, with the fix removed and the new test kept (this is also the state of `main` for a cell in that range): ```text $ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q E AssertionError: assert '123456789012345680' == '1.2345678901234568e+17' E E - 1.2345678901234568e+17 E ? - ^^^^ E + 123456789012345680 FAILED tests/test_transforms_tabular.py::test_load_xls_and_xlsx_agree_above_the_exact_integer_range 1 failed, 45 deselected in 4.10s ``` Green, with the fix: ```text $ python -m pytest tests/test_transforms_tabular.py -q 46 passed in 5.48s $ uvx ruff@0.16.7 check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py All checks passed! $ uvx ruff@0.16.7 format --check headroom/transforms/spreadsheet_ingest.py tests/test_transforms_tabular.py 2 files already formatted $ uvx mypy@1.14.1 --config-file pyproject.toml headroom/transforms/spreadsheet_ingest.py Success: no issues found in 1 source file ``` ## Real Behavior Proof - Environment: Windows 11 (10.0.26200) x64, Python 3.12.13, working tree at `origin/main` 85fac8c plus this one-file change. Optional extras present for this path only: `xlrd 2.0.2`, `xlwt 1.3.0`, `openpyxl`. No provider, no API key, no network -- the loader reads a local file. - Exact command / steps: build a real legacy workbook with `xlwt` holding one small whole number and one whole-valued double beyond 2**53, save it as `.xls`, and read it through the shipped `load_spreadsheet()`; then run the new test twice, once with the bound in place and once with the pre-fix line restored in the working tree. - Observed result: the same cell renders `123456789012345680` before this change and `1.2345678901234568e+17` after it, while `12` renders as `12` in both states. ```text $ python -c "xlwt writes [12, 1.2345678901234568e17] into legacy.xls; print(load_spreadsheet(...)['Data'].splitlines()[1])" post-fix row: 12,1.2345678901234568e+17 $ python -m pytest tests/test_transforms_tabular.py -k exact_integer_range -q # fix reverted in the tree E AssertionError: assert '123456789012345680' == '1.2345678901234568e+17' ``` The same cell, both states: the pre-fix render carries two invented digits (`123456789012345680`), the post-fix render is the double the sheet holds. The `12` half is identical in both, which is the point of bounding rather than deleting the conversion -- whole numbers at or below 2**53 keep the #3616 rendering, and only the range where a double cannot hold an integer reverts to the float. - Not tested: a workbook authored in real Excel or LibreOffice (the fixture is written by `xlwt`, which is also how the three existing `.xls` tests build theirs), and `.xls` cells in the 2**53..2**1024 range carrying `#`-formatted number styles, which `_xls_cell` does not see at all. One measurement worth flagging for whoever picks this up: I first asserted that the two loaders produce the same *text*, and it failed for a reason unrelated to this fix -- a value written and read back through `openpyxl` comes back as `1.234567890123457e+17`, because openpyxl serializes the literal to 15 significant digits, so the `.xlsx` path loses a digit on write while the `.xls` path loses one on render. The test therefore asserts the property that is this fix's subject (no invented integer; small integers still `12`) instead of cross-format textual equality, which the loaders do not actually guarantee for values in that range. ## Runtime Rollout Safety - Rollout-managed feature(s): none. One comparison in a pure function inside the spreadsheet loader; no flag service or staged rollout is involved. - Minimum rollout channel: n/a -- ordinary release. - Stable/default behavior changed: yes, narrowly. `.xls` cells holding a whole-valued double at or above 9007199254740992 now render as `1.2345678901234568e+17` instead of `123456789012345680`. Below that bound, and for every non-numeric cell type, output is byte-identical to today. - Kill switch / disable path: none needed and none added. Reverting the single commit restores the previous rendering exactly; there is no new code path to turn off, and no cache or stored format is involved (`load_spreadsheet` is a read path). - Unsafe override required: no. - Qualification impact: text of affected cells changes, so any test or prompt-cache fixture that captured `.xls` output for a cell in that range sees a diff. Existing tests: all 46 in `tests/test_transforms_tabular.py` pass, including the three `.xls` tests from #3616 whose expectation (`12`) is deliberately preserved. - Rollback path: revert the one commit; no migration, no persisted state, no store rewrite. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` -- it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) Not applicable; the observable artifact is the rendered CSV row, quoted in the Real Behavior Proof. ## Additional Notes Two judgement calls, both flagged for review: - The bound is `abs(value) <= 2**53`. #3695 proposed the exclusive form, and the review here correctly pointed out that binary64 represents 2**53 and -2**53 exactly while openpyxl loads those as integers, so excluding them left the two loaders disagreeing at the boundary itself. A stricter bound (e.g. 1e15, where decimal-style output still lo A stricter bound (e.g. 1e15, where decimal-style output still looks like a plain integer) would change more cells than necessary, and a looser one would keep inventing digits. - A negative whole number is also rendered as an int when small; `abs()` keeps that behavior symmetric, which is what the existing tests expect for positives and what `repr()` would otherwise contradict for negatives. Note on CI: `test (3)` on this branch is expected to show the same pre-existing `test_cost_per_token_resolves_deepseek_v4_flash` failure described in #3683 (litellm's off-peak pricing window), which #3684 addresses. Nothing in this diff is on that test's path. --------- Co-authored-by: sxh313 <250161920+sxh313@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
test_cost_per_token_resolves_deepseek_v4_flashfails onmainand on unrelated PRs whenever thetest (3)shard runs outside Mon–Fri 01:00–04:00 / 06:00–10:00 UTC (assert 0.15 == 0.3): main run 35462876818 (Fri 19:21 UTC) failed, #3668 (Sun 16:26 UTC) failed, #3679 (Mon 03:12 UTC) passed.Cause: between 2026-09-18 22:00 and 2026-09-19 12:00 UTC litellm's live pricing map gained an
off_peak_pricingblock fordeepseek-v4-flash(input 0.15/M vs 0.30/M; windows Mon–Fri 00:00–01:00, 04:00–06:00, 10:00–00:00 UTC, all weekend), and litellm 1.101.0 (uv.lock) bills from it incost_per_token. The test compares that clock-aware result against the flatinput_cost_per_tokenfield of the same entry, so the two sides now disagree most of the week.Fix: accept either tier from the active entry. With no off-peak block the accepted set is the single flat rate, so nothing changes for older litellm; with one,
cost_per_tokenis still pinned to the entry's own rates and the assertion no longer depends on the time of day. The test's purpose — thatresolve_litellm_modeladds thedeepseek/prefix socost_per_tokencan resolve the provider — is unchanged.#3575 rewrites this test as part of the V4.1 rate-card refresh and supersedes this; this is the one-hunk unblock for every open PR until it lands.
Closes #3683
Type of Change
Changes Made
tests/test_providers/test_deepseek.py: intest_cost_per_token_resolves_deepseek_v4_flash, build the accepted input/output rate sets fromactive_pricingplus itsoff_peak_pricingblock (if any) and assertcost_per_tokenmatches one of them.Testing
pytest) —tests/test_providers/test_deepseek.py: 20 passed (local litellm 1.97.0, no off-peak block → identical behaviour to before); under litellm 1.101.0 the single test passes at a peak instant and at the exact failing instant (below).ruff check .) —All checks passed!(ruff 0.16.7)mypy headroom) — tests only, no source changed.Test Output
Real Behavior Proof
litellm==1.101.0(theuv.lockversion) loading litellm's live pricing map as CI does; repo litellm 1.97.0 for the unchanged-behaviour check.litellm.cost_per_token(model="deepseek/deepseek-v4-flash", prompt_tokens=1_000_000, completion_tokens=1_000_000)withllm_cost_calc.utils.datetime.nowpinned to Fri 2026-09-19 19:21 UTC (main's failing run), Sun 2026-09-20 16:26 UTC (feat(opencode): exclude hosts from transport plugin routing #3668's failing run) and Mon 2026-09-21 03:00/08:00 UTC (peak); then the test itself at the same instants, patched and unpatched.input=0.150 output=0.600, peak instantsinput=0.300 output=1.200; unpatched test fails at off-peak withassert 0.15 == 0.3, patched passes at all four instants. Independently, re-running feat(opencode): exclude hosts from transport plugin routing #3668'stest (3)inside the peak window (03:29–03:37 UTC Monday) turned it green with no code change.Runtime Rollout Safety
Review Readiness
Checklist
CHANGELOG.md— it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this)Additional Notes
CONTRIBUTING asks not to open test-only PRs chasing a known
mainfailure; this one was not tracked anywhere, so the linked issue documents it, and this is the smallest change that stops it failing other people's PRs. Happy to close it in favour of #3575 if that is landing first.