Commit 683eda8
authored
feat(bigtable): add SessionPoolImpl (two-tier pool + scaling + debug) (#20225)
## Summary
Second of five PRs porting the session pool infrastructure from
`feat/bigtable-sessionz-debug` (which powers the recycle-repro fleet).
Adds `SessionPoolImpl`: the concrete two-tier read/write session pool
for one resource. ~4000 LOC across five files plus matching tests.
## Stack
- [ ] PR-1: sessionList (#20224) — per-AFE bucketing data structure.
**Not yet merged.**
- [x] **PR-2 (this)** — SessionPoolImpl (pool + scaling + debug +
snapshot).
- [ ] PR-3 — `SessionPool` / `Invoker` interfaces + `sessionClient` /
`sessionTable` factory wiring.
- [ ] PR-4 — Debug pages (`sessionz` / `afez` / `flightz` / `loadz`
under `bigtable/debugview/`).
- [ ] PR-5 — `bigtable.Client` integration + release notes.
**Because PR-1 has not landed, this PR is opened against `main` and the
diff includes PR-1's commits.** Once #20224 merges the base can be
re-targeted (or this branch rebased) so only PR-2's own delta shows.
## What lands
**SessionPoolImpl** (5 files, ~2400 LOC prod + ~2200 LOC tests):
- `session_pool.go` — struct + constructor + `Invoke` +
`CheckoutSession` (waiter queue, deadline propagation) + pluggable
picker via the AFE picker from #20204.
- `session_pool_lifecycle.go` — `SessionHooks` wiring,
consecutive-failure breaker, `Close` (5-phase teardown),
`WaitGoroutines` / `spawns.Wait` choreography so no session-owned
goroutine outlives the pool.
- `session_pool_scaling.go` — `Tick` loop, `createSession` (dial +
`OpenSession` + hook registration), `pendingStarts` / `startingSessions`
accounting so scale-up decisions never double-count in-flight opens.
Uses the channel-pool pick hint (`ChannelPickHintInto`, added to
`connpool.go`) to attribute each session to its underlying channel.
- `session_pool_debug.go` — `PoolSnapshot` / slow-vRPC ring /
per-close-reason counters / scaling-history buffer / `pickHistory` ring
— the input to the sessionz / afez / loadz debug pages (landing in a
later PR).
- `session_snapshot.go` — the value-typed snapshot record the debug
surface consumes; no live locks escape.
**Session helpers added** (pool-facing additions to files already
touched by prior PRs, kept minimal):
- `Session.loops sync.WaitGroup` + `WaitGoroutines()` — pool teardown
blocks on this so `readLoop` / `heartbeatLoop` and their `notifyClosed →
recordClose` callback chains fully unwind before `Close` returns.
Prevents session goroutines from racing metric-var writes across test
boundaries.
- `Session.closeErr atomic.Pointer[error]` + `setCloseErr` /
`closeError` — preserves the raw `Recv` error handed to `handleClose`.
Pool surfaces this on consecutive-failure breaker trips so operators see
the underlying server rejection (e.g. `FailedPrecondition` when the
resource is still being created) instead of only the sentinel.
**Supporting additions to existing files:**
- `afe_picker.go` — const `defaultAfeRandomSubsetSize = 2`
(power-of-two-choices K-choice default; matches Java).
- `debug_tracer.go` — three new tag constants:
`tagSessionPoolCreatePanic`, `tagSessionPoolConsecutiveFailuresTripped`,
`tagSessionPoolCheckoutFailedCINil`.
- `connpool.go` — `ChannelPickHintInto(ctx, *atomic.Int32)` context
helper. No-op when the channel pool doesn't consume the hint.
## What does NOT land yet
- `SessionPool` / `Invoker` interfaces (follow-up PR alongside
sessionClient / sessionTable).
- `bigtable.Client` integration (PR-3+).
- Debug pages under `bigtable/debugview/` (later PR).
## Test plan
- [x] `go build ./...` passes.
- [x] `go vet ./internal/transport/` clean.
- [x] `go test ./internal/transport/ -race -count=1 -short -skip
'AfeLbSim' -timeout=180s` — passes (32s wall). ~2200 LOC of new tests
across pool lifecycle, scaling, consecutive-failure breaker, AFE
integration, debug surface, snapshot rendering, plus a K-choice bench.
---
# Reviewer guide
## Guide 1 — mutianf (human)
### What this PR does
Adds `SessionPoolImpl`, the layer that sits above the per-AFE
`sessionList` shipped in #20224 and consumes it via a two-tier picker
(AFE first, then a ready session in that AFE). It owns the session
lifecycle (open / active / closing / close hooks), server-driven scaling
via `PoolSizer`, a consecutive-failure circuit breaker, and the
debug/observability surface (histograms + ring buffers) that feeds
sessionz/loadz. New files: 5 source, 6 test, ~4.9k LOC. Nothing outside
`session_pool*.go` / `session_snapshot*.go` is new logic — the small
edits elsewhere are hook-plumbing scaffolding already vetted by the
session/AFE subagent reviewers.
### Recommended read order
1. **`session_pool.go`** — start here. Struct field layout with
per-field ownership comments (`:104-179`), the `waiter` FIFO shape
(`:94-101`), `CheckoutSession` two-tier pick + parking (`:235-310`),
`Invoke` (`:465-559`), `Stats` (`:361-397`), `UpdateConfig`
(`:402-431`), `pickerFromLoadBalancing` (`:439-461`). Skim
`session_pool_test.go` (28 tests) — the FIFO waiter, Stats, and
UpdateConfig behaviors are all covered there.
2. **`session_pool_lifecycle.go`** — hooks (`onActive:255`,
`onClosing:308`, `onClose:336`), `recordSessionClose` once-CAS on
`Session.poolCloseRecorded` (`:117-130`), `Close`'s 6-phase teardown
(`:154-247`), `noteAbnormalCloseIfAny` breaker (`:363-392`), the three
ticker loops (`:426-538`). Skim `session_pool_lifecycle_test.go` — every
hook + `Close`.
3. **`session_pool_scaling.go`** — `Tick` (`:81-162`), `createSession`
worker (`:164-274`), `scalingReason` (`:278-299`),
`noDeadlineButCancellableContext` (`:301-311`). Skim
`session_pool_scaling_test.go` — the `scalingInProgress` gate and
panic-safety are the only non-obvious contracts.
4. **`session_pool_debug.go`** — `poolMetrics` (`:36-72`), `latencyHist`
log2 histogram (`:160-228`), the four ring buffers (slow-vRPC,
time-series, lifetimes, pick-history), `recordPickDecision`
(`:366-387`). Skim `session_pool_debug_test.go` — mostly ring-cap and
rate-computation coverage.
5. **`session_snapshot.go`** — mostly type defs. Focus on `PoolSnapshot`
(`:452-594`) and `LoadBalancingSnapshot` (`:414-436`) as the debug-view
contract.
6. **`session_pool_consecutive_failures_test.go`** and
**`session_pool_afe_test.go`** — end-to-end behavior verification;
useful for confirming intent.
### Flow of events
- **CheckoutSession → Invoke → release.** `CheckoutSession`
(`session_pool.go:235`) opportunistically kicks Tick if
`sl.ReadyCount()==0`, snapshots the picker under `p.mu`, then two-tier
picks outside the lock: `ReadyAfes()` → `PickAfe` → `Checkout(afeID)`
(`:259-268`). Miss → park in the FIFO waiter queue (`:286-289`), bracket
`waitersCount` for the sizer (`:291,300`). `Invoke` (`:465`) checks out,
runs `sh.session.Invoke`, records latencies (`:508-523`), logs a
slow-vRPC row if over threshold (`:524-557`); the deferred
`sh.DecOutstanding()` + `noteVRpcOutcome` (`:493-496`) hands the
OK-gated latency to the per-AFE PeakEwma tracker. Session release itself
is driven by `OnSlotDrained` (installed at
`session_pool_scaling.go:228-231`), which returns the handle to
`sessionList` and calls `signalFree` — separate from the `defer` in
`Invoke`.
- **Background Tick.** `startTickLoop` (`session_pool_lifecycle.go:426`)
fires every 1 s → `tickOnce` debounces via `tickPending` CAS
(`:447-458`) → `Tick` (`session_pool_scaling.go:81`) samples uptimes,
gates on `scalingInProgress`, calls `sizer.Decide()`, and on a positive
delta reserves `pendingStarts += delta` + `spawns.Add(delta)` under
`p.mu` (`:131-138`) then fans out one goroutine per session. Each
`createSession` acquires the budget outside `p.mu`, dials via
`streamFactory`, transfers `pendingStarts → startingSessions` in one
lock (`:246-249`), starts the session, and blocks on `WaitGoroutines` so
it stays on `p.spawns` until the session dies.
- **Abnormal close → breaker trip.** `onClose`
(`session_pool_lifecycle.go:336`) CAS's `closeRecorded`, calls
`noteAbnormalCloseIfAny` (`:363`), which bumps `consecutiveFailures` and
stores the raw error into `lastAbnormalCloseErr`. Crossing the threshold
snapshots the poison, CAS-resets the counter, and calls
`drainWaitersWithErr` — waiters get `*consecutiveFailureError` wrapping
the last cause (so `errors.Is(err, ErrConsecutiveFailures)` and
`status.Code(err)` both still work, `:60-82`). Counter only resets in
`onActive` (`:292-293`) — a successful open, not a healthy vRPC.
### Key invariants
1. **Two-tier pick, no re-entrant `p.mu`.** `CheckoutSession` reads
`p.picker` under `p.mu` (`session_pool.go:249-255`) then unlocks before
calling picker/sessionList. `recordPickDecision` takes `pickerName` as a
**parameter** (`session_pool_debug.go:366`, `session_pool.go:260-262`)
precisely because the caller already holds no lock — but any new pool
method that reads `p.picker.Name()` from a hot path must not re-take
`p.mu`.
2. **Waiter FIFO with `waitersCount` bracketed.** Every `PushBack` bumps
`waitersCount` (`session_pool.go:291`); every wake path (`ctx.Done`,
`w.ready`) decrements it (`:294,300`). `removeWaiter` (`:316`) is
idempotent via `w.elem != nil`; `signalFree` and `drainWaitersWithErr`
nil out `elem` under `waitersMu` (`:329-358`). `Stats().PendingCount`
reads `waitersCount.Load()` — this is the sizer's queue-depth input.
3. **Close-exactly-once accounting.** `sessionsClosed` and
`closesByReason` bumps are gated by
`Session.poolCloseRecorded.CompareAndSwap(false, true)` inside
`recordSessionClose` (`session_pool_lifecycle.go:117-130`).
`sh.closingRecorded` and `sh.closeRecorded` are per-handle CAS's
protecting the lifetime histogram + the `OnClose` branch. `Close`'s
Phase 1 pre-flips both CAS's on every handle (`:187-193`) so a
concurrent mid-flight onClosing can't double-count.
4. **Breaker resets only on `onActive`.** `consecutiveFailures.Store(0)`
and `lastAbnormalCloseErr.Store(nil)` live at
`session_pool_lifecycle.go:292-293`. Not on per-vRPC OK — otherwise one
long-lived healthy session would mask a run of failed opens.
5. **Hot path is atomics/RLocks; debug views take snapshots.** `Stats`
is the only per-request path that briefly takes `p.mu`
(`session_pool.go:362`); everything else on the vRPC path is atomic.
Debug snapshotters copy under lock and format after release
(`session_snapshot.go:452-594`).
### What NOT to worry about
- **Session / vRPC layer itself** — shipped in #20213 / #20215 (state
machine, one-in-flight, PeerInfo timing, retry oracle, heartbeat).
- **Per-AFE `sessionList` I1-I6** — shipped in #20224, has its own
tests.
- **`PoolSizer` scaling formula** — already upstream (`pool_sizer.go`);
this PR only wires it and consumes `ScaleDecision`.
- **AFE pickers (`SimpleAfePicker` / `LeastInFlight` / `LeastLatency`)**
— already upstream (`afe_picker.go`); this PR only builds them via
`pickerFromLoadBalancing`.
- **`SessionThrottler` / `AdaptiveSessionThrottler`** — already
upstream; this PR consumes `Acquire` / `Release` / `UpdateConfig`.
- **`ClientConfigurationManager` polling** — this pool receives
`UpdateConfig` calls; the polling itself is elsewhere.
### Danger zones
- **Re-entrant `p.mu` on picker access.** `recordPickDecision`
intentionally takes `pickerName` as a param
(`session_pool_debug.go:366`). Adding a new pool method that reads
`p.picker.Name()` from within a `CheckoutSession` code path is a
re-entrant deadlock; pass the name in or snapshot up-front.
- **`startingSessions` / `pendingStarts` accounting.** Tick reserves
`pendingStarts` under `p.mu` (`session_pool_scaling.go:131-138`),
`createSession`'s `reserved` defer releases it on any early return
(`:172-179`), and the transfer at `:246-249` is atomic under `p.mu`.
`onActive` deletes from `startingSessions`
(`session_pool_lifecycle.go:265`). Any new failure branch in
`createSession` must preserve the invariant `pendingStarts +
len(startingSessions) + Ready = "in-flight scale-up capacity"`.
- **`budget.Acquire` blocks; must run OUTSIDE `p.mu`.** Currently at
`session_pool_scaling.go:181`, deliberately after the `defer reserved`
block and before any `p.mu.Lock()`. Moving it under the lock deadlocks
scale-up under budget exhaustion.
- **Slow-vRPC is fire-and-forget.** `recordSlowVRpc`
(`session_pool_debug.go:301`) appends into a mutex-guarded ring; it's on
the vRPC return path but bounded by threshold gating and one small
mutex. Do not add I/O, logging fanout, or channel sends here.
- **`newTestPool` / bootstrap defaults.** Test helpers construct the
pool without a `ClientConfigurationManager`, so the bootstrap defaults
from `defaultPoolConfig()` (`session_pool.go:218-226`) are the only
config those tests see. Real callers always get `UpdateConfig`
synchronously on registration — verify by reading `NewSessionPoolImpl`
end-to-end, not by trusting the bootstrap values.
- **`Close` phase ordering.** Phase 4 (`poolCancel`) runs AFTER Phase 3
(`wg.Wait` on graceful closes) because Phase 2's `closeCtx` derives from
`p.poolCtx`; swapping order strands in-flight graceful closes on a
cancelled ctx. Phase 5 (`p.spawns.Wait`) blocks on every createSession
goroutine's `WaitGoroutines` — that's why createSession blocks on
`s.WaitGoroutines()` at `session_pool_scaling.go:272`.
---
## Guide 2 — mutianf-bot (automated reviewer)
### REAL HAZARDS to flag
- **Re-entrant `p.mu` in pool methods called from `CheckoutSession`.**
Anchor: `session_pool.go:235-310`. `p.mu` is dropped at `:255` before
`PickAfe` / `Checkout` / `recordPickDecision` fire. Flag any newly-added
helper called from that block that re-acquires `p.mu`, or any new method
that reads `p.picker.Name()` without taking the name as a parameter (see
the intentional parameter pattern at `session_pool_debug.go:366`).
- **`budget.Acquire` under `p.mu`.** Currently correctly outside the
lock at `session_pool_scaling.go:181`. `SessionThrottler.Acquire` blocks
on the budget semaphore; calling it while holding `p.mu` would deadlock
scale-up. Flag any code path that acquires `p.mu` before line `:181` or
moves `Acquire` inside a `Lock`/`Unlock` bracket.
- **`sync.Map` allocations on hit paths.** `bumpCloseReason` uses `Load`
first, `LoadOrStore(k, new(atomic.Int64))` only on miss
(`session_pool_lifecycle.go:102-111`) — this is the correct pattern.
Flag any new `sync.Map.LoadOrStore(key, new(...))` call on a hot path
that isn't gated by a preceding `Load` — that allocates on every hit.
- **Waiter counter drift.** `waitersCount.Add(+1)` at
`session_pool.go:291`, `Add(-1)` on both the `ctx.Done` branch (`:294`)
and the `w.ready` branch (`:300`). Flag any new wake path, timeout
branch, or early-return between `:291` and `:308` that doesn't
decrement, and any new enqueue site that doesn't increment. Drift here
corrupts the sizer's `PendingCount` input.
- **Unbalanced `pendingStarts` / `startingSessions`.** Tick increments
`pendingStarts` under `p.mu` at `session_pool_scaling.go:131-138`;
`createSession`'s `reserved` defer at `:172-179` releases on early
return; the transfer to `startingSessions` at `:246-249` is atomic;
`onActive` deletes at `session_pool_lifecycle.go:265`; failed-start
deletes at `session_pool_scaling.go:253-255`. Flag any new failure
branch in `createSession` that returns without either the `reserved`
defer or an explicit transfer/cleanup.
- **Missing CAS on close-once flags.** `sessionsClosed.Add(1)` sits
downstream of `s.poolCloseRecorded.CompareAndSwap(false, true)` at
`session_pool_lifecycle.go:121`. Lifetime histogram sits downstream of
`sh.closingRecorded.CompareAndSwap(false, true)` at `:187` and `:317`.
`recordSessionClose` re-entry is dedup'd at `:190` and `:347`. Flag any
new counter bump / histogram record that isn't gated by the
corresponding CAS.
- **Uncontrolled goroutine spawns.** Every pool-spawned goroutine must
be tracked so `Close` phases 3/5/6 can drain: createSession workers use
`p.spawns.Add`/`Done` under `p.mu`
(`session_pool_scaling.go:137-138,149`); `spawnTickOnce` uses the same
pattern (`session_pool_lifecycle.go:464-476`); the mid-Close onActive
dispatch adds at `:274`. Session-owned goroutines
(`readLoop`/`heartbeatLoop`) are tracked on `Session.loops` and drained
via `WaitGoroutines`. Flag any bare `go func(){…}()` in these files with
no WaitGroup entry.
- **Hot-path lock churn.** `p.mu` is briefly held on the fast path only
at `session_pool.go:239-241` and `:249-255`. `Stats` takes `p.mu`
(`:362`). Flag any new `p.mu.Lock()` inserted into `Invoke` (`:465-559`)
or the pre-park stretch of `CheckoutSession` outside the two existing
points.
### FALSE-POSITIVE SUPPRESSION
- **"Package name should be `transport`, not `internal`."** The
directory is `bigtable/internal/transport/` but the `package`
declaration is `package internal` (confirmed at `session_pool.go:20`).
Package name is correct as-is; do NOT flag.
- **Local variable shadowing an exported type** where the local name is
idiomatic (e.g. `afeID` local vs `AfeID` type in `CheckoutSession`).
Already resolved intentionally; do NOT re-raise variants.
- **`goimports` / `gofmt` / column-alignment / trailing-newline nits.**
CI (`goimports -l`, `gofmt -l`, `go vet`) already gates these. Bot echo
is noise.
- **Comments referencing PR #20213 / #20215 / #20224.** Stacked-PR
context, not stale references; do NOT suggest removal.
- **Test coverage complaints for `pool_sizer.go`, `afe_picker.go`,
`session_list.go`, `session.go`, `session_vrpc.go`,
`session_throttler.go`, `client_configuration_manager.go`,
`default_client_config.go`.** All shipped in earlier PRs (#20213,
#20215, #20224) with their own tests; out of scope here.
- **"Missing error wrapping"** on internal-only calls where the caller
already annotates via `fmt.Errorf("POOL %s ...: %w", ...)` or via
`btopt.Debugf`. Do NOT suggest adding a second wrap.
- **Retry loop / context propagation questions on `Session.Invoke`.**
That's the Session layer (`session_vrpc.go`), out of scope for this PR.
- **"Consider using `sync.RWMutex` instead of `sync.Mutex` on `p.mu`."**
The pool holds `p.mu` for tens of nanoseconds at a time and never for
read-heavy loops; the added atomic on `RLock`/`RUnlock` would cost more
than it saves. Do NOT suggest.
- **"Consider extracting anonymous goroutine into named function."**
Style-only; do NOT suggest for the three ticker loops or the
createSession worker.
### SCOPE BOUNDARY
Comment ONLY on:
- `bigtable/internal/transport/session_pool.go`
- `bigtable/internal/transport/session_pool_lifecycle.go`
- `bigtable/internal/transport/session_pool_scaling.go`
- `bigtable/internal/transport/session_pool_debug.go`
- `bigtable/internal/transport/session_snapshot.go`
- `bigtable/internal/transport/session_pool_*_test.go`
- `bigtable/internal/transport/session_snapshot_test.go`
Do NOT comment on additions to:
- `session.go` / `session_vrpc.go` (WaitGoroutines / closeError
additions — vetted)
- `connpool.go` (`ChannelPickHintInto` helper — vetted)
- `afe_picker.go` (`defaultAfeRandomSubsetSize` constant — vetted)
- `debug_tracer.go` (3 new tags — vetted)
These are supporting scaffolding, already reviewed by the 3 subagent
reviewers in this stack. Only re-raise if something looks actively
unsafe.
### EFFORT SCALING
- ~4.9k LOC across 12 files. Do NOT paginate uniformly.
- **First pass — the 4 hot source files, in this order:**
1. `session_pool.go` (559 LOC)
2. `session_pool_lifecycle.go` (538 LOC)
3. `session_pool_scaling.go` (311 LOC)
4. `session_pool_debug.go` (416 LOC)
- **Second pass ONLY if a first-pass finding needs corroboration:**
`session_snapshot.go` (594 LOC, mostly type defs), and the tests. Tests
use `newTestPool`, which skips config wiring — do NOT flag bootstrap
defaults on tests as if they were production paths.
- If a first-pass finding is a real hazard from the list above, cite the
file:line and the exact anchor pattern it violates. Do not file
speculative "consider" comments.1 parent 5d4c193 commit 683eda8
19 files changed
Lines changed: 5103 additions & 15 deletions
File tree
- bigtable/internal/transport
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
21 | 27 | | |
22 | 28 | | |
23 | 29 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1066 | 1066 | | |
1067 | 1067 | | |
1068 | 1068 | | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
116 | 116 | | |
117 | 117 | | |
118 | 118 | | |
119 | | - | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
120 | 145 | | |
121 | 146 | | |
122 | 147 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
60 | 66 | | |
61 | 67 | | |
62 | 68 | | |
| |||
87 | 93 | | |
88 | 94 | | |
89 | 95 | | |
90 | | - | |
| 96 | + | |
91 | 97 | | |
92 | | - | |
93 | | - | |
94 | 98 | | |
95 | 99 | | |
96 | 100 | | |
97 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
98 | 105 | | |
99 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
100 | 116 | | |
101 | 117 | | |
102 | 118 | | |
| |||
110 | 126 | | |
111 | 127 | | |
112 | 128 | | |
113 | | - | |
114 | | - | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
115 | 135 | | |
116 | 136 | | |
117 | 137 | | |
| |||
196 | 216 | | |
197 | 217 | | |
198 | 218 | | |
199 | | - | |
200 | | - | |
| 219 | + | |
| 220 | + | |
201 | 221 | | |
202 | 222 | | |
203 | 223 | | |
| |||
233 | 253 | | |
234 | 254 | | |
235 | 255 | | |
236 | | - | |
| 256 | + | |
237 | 257 | | |
238 | 258 | | |
239 | 259 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
195 | 195 | | |
196 | 196 | | |
197 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
198 | 210 | | |
199 | 211 | | |
200 | 212 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
405 | 405 | | |
406 | 406 | | |
407 | 407 | | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
78 | | - | |
79 | | - | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
80 | 85 | | |
81 | 86 | | |
82 | 87 | | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
83 | 95 | | |
84 | 96 | | |
85 | 97 | | |
| |||
450 | 462 | | |
451 | 463 | | |
452 | 464 | | |
| 465 | + | |
453 | 466 | | |
454 | 467 | | |
455 | 468 | | |
| |||
0 commit comments