Skip to content

Latest commit

 

History

History

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

README.md

Batch Processing

Process multiple job offers in parallel via headless workers. Each worker runs the full evaluation pipeline (A-F report + PDF + tracker line) autonomously. See the Headless / Batch Mode table in AGENTS.md for the correct command per CLI.

Quick Start

  1. Add offers to batch-input.tsv (tab-separated: id, url, source, notes):

    id	url	source	notes
    1	https://jobs.example.com/role-a	LinkedIn	
    2	https://greenhouse.io/company/role-b	Greenhouse	priority
  2. Dry run to preview what will be processed:

    ./batch/batch-runner.sh --dry-run
  3. Run the batch:

    ./batch/batch-runner.sh
  4. Results are automatically merged into data/applications.md, processed offers are reconciled out of the data/pipeline.md inbox, and integrity is verified with verify-pipeline.mjs at the end of the run.

Options

Flag Default Description
--parallel N 1 Number of concurrent headless workers
--dry-run off Preview pending offers without processing
--retry-failed off Only retry offers marked as failed in state
--resume-paused off Resume offers paused after a Claude session/rate limit
--start-from N 0 Skip offers with ID below N
--limit N 0 Max number of offers to process in this run (0 = no limit)
--max-retries N 2 Max retry attempts per offer before giving up
--rate-limit-sleep N 300 Maximum adaptive retry delay in seconds (0–2147483647); use 0 to pause the batch immediately

Adaptive rate-limit retries (Claude only)

Transient rate limits start at min(30, N) seconds, double for each retry already consumed by the offer (including persisted retries on resume), and saturate at N. A random integer percentage from 0 through 20 is added, rounded down to whole seconds, then capped at N. With the default ceiling, the first waits are 30–36s and 60–72s. Small ceilings can leave no jitter room.

The current failed attempt's log may override exponential growth with a standalone Retry-After: <non-negative integer seconds> line (case insensitive, surrounding whitespace and CRLF allowed). Its delay is max(min(30, N), seconds) plus the same capped upward jitter, so it never retries before the requested time. If several valid lines exist, the largest wins. Values exceeding N pause the batch without consuming a retry, rather than retrying too early. HTTP dates, fractions, signed values, JSON/embedded prose and malformed lines are ignored; no worker log text is evaluated as code. Each attempt overwrites its log, so a prior attempt's header cannot affect the next wait.

There is no new base-delay flag. Waits within the ceiling remain in the worker, including in serial mode. --max-retries still bounds retries; once exhausted, the existing failure path applies. --rate-limit-sleep 0 and session-limit pauses retain their existing behavior. Actual waits appear in the console and the rate_limited state note; the TSV schema is unchanged.

Directory Layout

batch/
  batch-runner.sh          # Orchestrator script
  batch-prompt.md          # Prompt template sent to each worker
  batch-input.tsv          # Input offers (you create this)
  batch-state.tsv          # Processing state (auto-managed, resumable)
  logs/                    # Per-offer worker logs ({report_num}-{id}.log)
  tracker-additions/       # TSV lines produced by workers
    merged/                # TSVs already merged into applications.md

How It Works

  1. batch-runner.sh reads batch-input.tsv and batch-state.tsv to determine which offers need processing.
  2. For each pending offer, it assigns a report number and launches a headless worker with batch-prompt.md as the system prompt (placeholders like {{URL}}, {{REPORT_NUM}} are resolved).
  3. Each worker evaluates the offer, writes a report to reports/, generates a PDF to output/, and writes a tracker TSV to tracker-additions/.
  4. After all workers finish, batch-runner calls merge-tracker.mjs to merge TSVs into data/applications.md, reconcile-pipeline.mjs to move processed offers out of the data/pipeline.md inbox, and verify-pipeline.mjs to check integrity.

Tracker Merge

Workers write one TSV per offer to batch/tracker-additions/. The merge script (npm run merge) handles:

  • Deduplication by company + role fuzzy match and report number
  • Column order conversion (TSV has status before score; applications.md has score before status)
  • In-place updates when a re-evaluation scores higher than the existing entry
  • Moving processed TSVs to tracker-additions/merged/

Run npm run merge manually if you need to merge outside of a batch run.

Pipeline Reconcile

Batch mode reads offers from batch-input.tsv, but the data/pipeline.md inbox is a separate list. Without reconciliation, an offer evaluated by a batch run stays in the pipeline "Pendientes" section and gets surfaced again on the next scan or /career-ops pipeline run -- producing duplicate reports.

reconcile-pipeline.mjs (run as npm run reconcile) closes that gap: after the tracker merge, every completed or skipped offer in batch-state.tsv whose URL is still in pipeline "Pendientes" is moved to "Procesadas" with its report link and score (entries without a report file on disk are left in place). It is idempotent -- safe to run after every batch, or manually.

Resumability

batch-state.tsv tracks the status of every offer (pending, processing, completed, failed, skipped, rate_limited, paused_rate_limit). If the batch is interrupted, re-running batch-runner.sh picks up where it left off -- completed offers are skipped automatically. rate_limited is a non-completed state used while the runner waits before retrying, so interrupted rate-limited jobs are eligible on the next normal run.

paused_rate_limit is different: a worker hit a Claude session/usage limit, zero-wait pause, or a Retry-After above the configured ceiling, so the runner stopped scheduling new offers and preserved the retry count. Resume those rows explicitly after the limit resets:

./batch/batch-runner.sh --resume-paused

A PID-based lock file (batch-runner.pid) prevents concurrent batch runs. If a previous run crashed, the stale lock is detected and removed automatically.

Prerequisites

  • Your CLI in PATH (see Headless / Batch Mode table in AGENTS.md)
  • Node.js >= 18, Playwright chromium installed (npm run doctor to verify)
  • batch-input.tsv with at least one offer