@jiajun0413/pi-provider-manager

Roundrobin failover engine for the pi coding agent.

Registers a virtual roundrobin provider that pools multiple real model candidates behind one virtual model, with sticky failover, idle-timeout guards, and per-candidate cooldown.

The virtual wrapper advertises api: openai-responses so it resolves through a real registered API identity even when the underlying candidate pool is heterogeneous.

Attribution

This package is a fork and rebuild of @arcaneorion/pi-provider-manager@0.3.9 (upstream npm package; no public Git repo). The roundrobin failover engine architecture is the original author's work, used here under MIT.

This fork fixes a mid-stream idle-timeout bug (a candidate stream that starts then goes silent no longer hangs forever), hardens failover against unstable relays (upstream aborted is now a cooldown-able failure, not a turn-killing cancel; auth lookup is bounded by the idle guard), and removes the older visual config panel and on-disk health store. The current extension also includes the root-session routing-policy editor and durable policy store described below.

Install

pi install npm:@jiajun0413/pi-provider-manager

Restart pi. (pi loads .ts extensions via jiti — no build step.)

Features

• Roundrobin failover — one virtual roundrobin provider pools multiple provider/model candidates. Sticky strategy with automatic failover on first-response timeout or stream error. Each preset becomes a selectable virtual model.
• Nested pools — a candidate may reference another roundrobin/<preset> virtual model. Nested candidates are dispatched directly to the local failover engine rather than through Pi's remote API dispatcher.
• Mid-stream idle timeout — a candidate stream that starts then goes silent errors out after timeoutMs instead of hanging forever (prevents ghost subagent sessions).
• No mid-stream replay — once content or tool calls have been emitted, an error is terminal rather than replayed on another provider, to avoid duplicating output.
• Upstream-aborted is a failure, not a cancel — a relay that drops the socket / cancels upstream and surfaces stopReason:"aborted" is cooled down and the pool rotates, instead of killing the whole turn. Only a real user abort (the request's own AbortSignal) short-circuits.
• Auth-bounded idle guard — API-key/header resolution races against the same timeoutMs + attempt abort as stream chunks, so a hung key lookup can't stall the turn.
• Entry-time model population — virtual models register at load time with placeholder candidates so they appear in /model and static lists (pi-web's /api/models) before a session starts; session_start rebuilds with the real registry.
• virtualModel inheritance — the virtualModel block is optional; omit it and metadata is inherited from the group's first candidate (see Inheritance).
• Root-session routing policy — /rr-routing can restrict which configured terminal destinations Roundrobin may choose. Nested roundrobin/<preset> pools are traversed, not matched. Direct non-Roundrobin /model selections stay unrestricted. This is a cooperative Roundrobin guardrail, not a quota, billing, or security boundary (see Root-session routing policy).

Usage

1. Create ~/.pi/agent/roundrobin/config.json (the "default" group):

   {
     "virtualModel": {
       "name": "GLM-5.2",
       "reasoning": true,
       "input": ["text"],
       "contextWindow": 1000000,
       "maxTokens": 128000,
       "thinkingLevelMap": { "max": "max" },
       "compat": { "thinkingFormat": "deepseek" }
     },
     "candidates": [
       { "provider": "k", "model": "glm-5.2" },
       { "provider": "wg", "model": "glm-5.2" }
     ],
     "timeoutMs": 16000,
     "cooldownMs": 120000,
     "strategy": "round-robin",
     "sticky": true,
     "log": true
   }
   

2. (Optional) Add more groups as *.json files under presets/. Each file's stem (name without .json) becomes a roundrobin/<name> model.

3. In pi, use /model to select roundrobin/default (or any preset name).

4. Requests now flow through the failover engine. If the current candidate times out or errors before the stream starts, the next candidate is tried automatically. If every candidate fails, the engine waits for the earliest cooldown and retries from the preferred candidate — bounded to 3 passes before the turn ends with the last error.

A mid-stream error is returned immediately rather than replayed (content or tool calls may already have been emitted).

Selection strategy and stickiness

strategy chooses a new candidate within the highest healthy priority tier:

• primary — try the first candidate in the tier first.
• round-robin (default) — rotate that tier's start index after each success.
• random — choose a fresh random start in that tier for each selection/retry pass.

sticky independently keeps the last successful candidate, including one from a lower priority tier, until its lease ends or it fails/cools down. Plugin-wide allowedFallback mappings can redirect after the currently sticky candidate fails when onSticky is enabled. When onExhausted is enabled and every policy-eligible configured candidate is cooling solely because of a provider cooldown hint, mapped targets are selected by the normal tier/strategy order. Ordinary failure and manual cooldowns do not trigger this path. Lease expiry and policy-driven reselection do not trigger sticky fallback. The mapped from and to identities share the sticky lease: timed session age carries across the switch, while last-request and while-cached refresh from each successful mapped request. Configure mappings in ~/.pi/agent/roundrobin/roundrobin.json, not per-preset:

• true, "true", "yes", or "on": sticky until failure/cooldown or Pi restart.
• false, "false", "no", or "off": select anew each request.
• "session": sticky until stickyTime after it first became sticky.
• "last-request": sticky until stickyTime after its latest successful completed request.
• "while-cached": sticky until the estimated prompt-cache TTL after its latest successful completed request. TTLs are currently hardcoded: 60 minutes for names containing claude, 30 minutes for names containing gpt, 5 minutes for cursor/cursor-grok and cursor/composer, and 5 minutes by default. Matching is case-insensitive; provider/model identities are matched as a whole substring.

session and last-request require stickyTime, which accepts positive composite durations such as 1h, 5h30m, and 2d. while-cached uses its model-derived TTL and does not take stickyTime. A failed request never refreshes a last-request or while-cached lease.

When a while-cached group considers nested roundrobin candidates, it queries their next eligible selection without authenticating or dispatching. A live child sticky lease is retained; otherwise nested candidates are ranked by the tier their child would select (lower index wins). Non-nested candidates count as tier 0. The parent’s own priority tiers remain dominant, and its configured strategy breaks equal-rank ties. The query is advisory; normal policy and health checks still apply at dispatch.

{
  "allowedFallback": {
    "mappings": [{ "from": "billing/(*)/included", "to": "billing/\\\\1/api" }],
    "allowNonExplicit": true,
    "onSticky": true,
    "onExhausted": true
  }
}

(*) captures any substring (including /) and \\1 references the first capture. A mapped destination must be registered; with allowNonExplicit: false it must also be explicitly listed in that group's candidates. With true, it may be selected outside the candidate list, but remains subject to routing policy and cooldown checks. Both switches default to false; provider hints require an absolute future cooldown deadline and a list of affected candidates.

Legacy strategy: "sticky" and strategy: "random-sticky" remain accepted as deprecated aliases for round-robin + sticky: true and random + sticky: true; Pi logs a startup configuration warning for each affected group.

Per-process caveat: cursors, sticky leases, and automatic failure cooldowns live only in extension memory. They reset on restart and are not shared with subagent processes. Provider hint cooldowns persist (see below).

Provider cooldown hints

Any provider can add this optional field to an error AssistantMessage to cool exact configured candidates until an absolute epoch-millisecond deadline:

routingHint: {
  version: 1,
  action: "cooldown",
  until: 1760000000000,
  candidates: ["billing/cursor/claude-sonnet-5/included"]
}

Roundrobin validates the version, action, future bounded deadline, and exact non-empty candidate list. It applies the deadline without shortening an existing cooldown, to matching candidates across all loaded presets. Hint deadlines are written to ~/.pi/agent/roundrobin/hint-cooldowns.json so a new Pi process skips the same candidates until until. Invalid hints are ignored and the originating candidate receives its ordinary cooldownMs. This is a generic protocol: no provider ID is special-cased.

Manually cycle a candidate

Use /rr-cycle-candidate while a roundrobin/<preset> model is currently selected. It requires an active roundrobin selection — running it with any other model selected (or none) is rejected with an error instead of doing nothing silently.

It marks the current candidate as bad and immediately advances the cursor to the next non-cooling candidate, so the very next request uses it rather than waiting for a future failed request to trigger failover. By default, it applies the owning preset's cooldownMs. Pass a positive duration to override it for this one cooldown:

/rr-cycle-candidate 1d
/rr-cycle-candidate 5h
/rr-cycle-candidate 30m

Supported units are ms, s, m, h, and d, including composites such as 5h30m. The cooldown remains process-local and is reset when pi restarts.

Nested chains and routing policy — if the selected preset's current policy-eligible candidate is itself another roundrobin/<preset> (see Nested pools), the command descends through eligible wrappers to the real upstream leaf and marks that candidate bad, never the nested wrapper. For example, given default → usage-based → openai-codex/gpt-5.6-terra (all eligible), running /rr-cycle-candidate while default is selected cools down openai-codex/gpt-5.6-terra inside the usage-based group. Descent does not treat a wrapper as a leaf: if a nested group is disabled, a cycle is detected, or no destination remains policy-eligible, the command errors and applies no cooldown. The advertised next candidate is also policy-eligible. The command requires a readable routing policy; if policy state is unavailable it errors instead of cycling.

Time out a provider or model by name

Unlike /rr-cycle-candidate, /rr-mark-failed does not require a roundrobin model to be selected — it targets candidates by name across every configured group (config.json and every preset) at once, including nested ones:

/rr-mark-failed openai-codex
/rr-mark-failed openai-codex/gpt-5.6-sol
/rr-mark-failed openai-codex 1d
/rr-mark-failed openai-codex/gpt-5.6-sol 5h

• <provider> alone (e.g. openai-codex) marks every candidate for that provider bad, in every group it appears in.
• <provider>/<model> (e.g. openai-codex/gpt-5.6-sol) marks only that exact candidate bad.
• An optional trailing duration (same units as /rr-cycle-candidate: ms, s, m, h, d) overrides each affected group's own cooldownMs for this cooldown; otherwise each group's own cooldownMs applies independently.

For every group where the matched candidate was the group's current pick, its cursor is advanced to the next non-cooling candidate immediately — the same automatic-reselection behavior as /rr-cycle-candidate — so any group actively routing through that provider/model switches away right away instead of waiting for a failed request. Groups where the match wasn't the current pick are just cooled down for future selection. If nothing matches across any group, the command errors instead of silently doing nothing.

Manual cooldowns from either command persist in ~/.pi/agent/roundrobin/manual-cooldowns.json, so they survive extension reloads and Pi restarts. Expired entries are discarded automatically; automatic failure cooldowns remain process-local.

Multi-preset routing

Each preset under ~/.pi/agent/roundrobin/ registers as an independent virtual model whose model id equals the preset file stem (the .json suffix is stripped). config.json is the reserved "default" group; every *.json file under presets/ is an additional group. In /model all virtual models appear at once — pick roundrobin/<name> to route to that group's pool. Health and currentIndex are isolated per preset.

Root-session routing policy

The /rr-routing command opens a transactional TUI editor for the current session's Roundrobin policy. It restricts only Roundrobin's configured terminal destinations; direct model selections and non-Roundrobin calls remain outside this policy. The status key is labelled RR only and remains visible when another model is selected. This is a cooperative routing guardrail, not a security sandbox, quota monitor, or financial guarantee.

The editor is available only in the TUI. In print, RPC, and JSON modes, routing is still enforced but the editor is not opened. A root session can edit. A child, grandchild, or fork sees a read-only view and must resume the owning root session to edit. Draft changes are local until Apply; Cancel, Escape, an invalid draft, a stale revision, or an I/O failure leaves the stored policy unchanged. Apply validates and publishes one revision. There is no autosave.

Quick setups and rules

The editor offers:

• Included usage only — whitelist billing/*/included.
• Included usage from… — select one or more real providers; each creates billing/<provider>/*/included. Selecting none creates an empty whitelist, which permits nothing.
• Custom rules — edit the mode and individual allow/deny patterns.

Quick setups replace the draft, not the preset or billing configuration. If the draft is dirty, replacement asks for confirmation. Add-rule flows offer a billing route builder or a free pattern. The builder chooses a provider, an exact model or all models, and included, api, or both. Choosing both creates two explicit patterns. Exact model IDs preserve slashes and escape supported glob metacharacters. Picker choices are local and do not authenticate or contact a provider; choices not reachable from a configured preset are warned about.

Rules use full, case-sensitive provider/model destination identities. Supported flat globs are *, ?, and positive, range, or negated character classes such as [abc], [a-z], [!abc], and [^abc]. * spans /; matching is anchored to the complete string. Bare-model, suffix, basename, brace, extglob, and leading-! semantics are not supported, and other regular-expression metacharacters are literal. Empty patterns, control characters, invalid ranges, and more than 256 total rules are rejected. A pattern is limited to 1,024 characters and stored pattern text to 64 KiB.

A destination is eligible when no deny pattern matches and either:

• mode is allow-unless-denied; or
• mode is whitelist-only and at least one allow pattern matches.

Deny always wins. In allow-unless-denied mode, allow rules are editable but unused. An empty whitelist permits no destination. The default unrestricted policy is allow-unless-denied with empty allow and deny lists. A direct non-billing reference, such as anthropic/model, is matched literally and warned about; it is not treated as an included or API billing route. Billing-shaped examples are full identities, such as billing/openai-codex/gpt-5.6-sol/included.

Recommended route patterns, when the corresponding billing routes are configured, include:

{
  "mode": "whitelist-only",
  "allow": ["billing/*/included"],
  "deny": []
}

For one provider, use billing/openai-codex/*/included. To permit both variants for that provider while explicitly denying API routes, use allow billing/openai-codex/* and deny billing/*/api. These are recommendations only: the package does not migrate or rewrite existing user presets, billing routes, provider configuration, or authentication configuration.

Preview and details

The modal's compact preview is intended to show unique configured eligible routes, usable presets, fully policy-blocked presets, direct non-billing reference diagnostics, and zero-match rule warnings. The counts are policy eligibility only — not quota, authentication, registry availability, or health. A route may be eligible but unavailable in the registry or unavailable because of its provider's quota or health. Configuration problems such as empty, missing, or cyclic nested presets are diagnosed separately from a fully policy-blocked preset.

Press Ctrl+E to expand the details view; it starts collapsed every time the modal opens. The expanded view is scrollable and shows full deduplicated eligible route identities, all transitively reachable preset names, fully blocked preset names, and direct non-billing references with their owning presets. Ctrl+E is intercepted before text inputs and list controls, so it neither submits nor applies the draft. When the details control is focused, use arrows, PageUp/PageDown, Home, and End to scroll; Tab and Shift+Tab move focus. Enter activates only the focused control, so Apply is always explicit.

Enforcement and lifecycle

Policy filtering occurs before Roundrobin selection and health/cooldown logic, before authentication, immediately before dispatch, and again after authentication. A denied destination receives no authentication, dispatch, selection-cache write, cooldown, or retry-budget charge. If all configured terminal destinations are denied, Roundrobin returns a clear policy-blocked error with zero usage instead of trying an API route. A permitted provider failure keeps the normal failover, retry, cooldown, and no-mid-stream-replay behavior. Policy edits affect subsequent selection and fallback points; an already dispatched request can finish.

Nested roundrobin/<preset> references are traversal nodes, not matchable destinations. Route rules apply to the terminal leaves reached through them, with path-local cycle handling. The same policy is used by sticky selection, fallback, nested routing, current-candidate reporting, and manual cycling. /rr-mark-failed is an explicit administrative action: it may cool matching configured destinations even when they are currently denied, while its reported next candidate remains policy-eligible. Policy skips themselves never create cooldowns.

Each session binds to a root policy. A fresh root starts unrestricted. Descendants and grandchildren inherit the root's live policy reference and are read-only; root edits are visible to them, including explicit loosening. /tree and compaction preserve the binding. /reload restores the same session binding. Root /new starts a new unrestricted root while existing descendants keep the old root. Descendant /new remains dependent and cannot escape. /fork, /clone, CLI --fork, and ephemeral forks are read-only dependents of the same live root, including before the first saved entry. Two independent roots do not share a policy. Quitting retains the durable records.

The extension must be loaded in every participating child or subagent process. Descendants that were already running when the feature was installed or reloaded do not gain retroactive context; restart those descendants (and restart/reload the relevant tree as needed) before relying on the policy. A root reload alone does not inject the context into an already-running child. A malformed or missing known policy/binding fails closed for Roundrobin rather than falling back to unrestricted routing.

Policy storage

The durable policy store is separate from the expiring authenticated-leaf cache:

~/.pi/agent/roundrobin/routing-policy/
├── roots/<encoded-root-id>.json
├── sessions/<encoded-session-id>.json
├── locks/<encoded-root-id>.json.lock
└── locks/<encoded-root-id>.json.lock.recover

Root records contain the version, root ID, owner session ID, revision, and rules. Session records contain the session ID, root ID, owner session ID, and immutable owner or dependent role. A roundrobin-routing-binding custom session entry supplements the sidecar as an adoption marker; it is not a historical policy snapshot. The process handoff reference is carried in PI_ROUNDROBIN_POLICY_CONTEXT; it contains identity and store metadata, not rules or credentials.

Directories are created with 0700 permissions and records with 0600 where supported. Writes are validated and atomic. Policy publication uses a short per-root symlink lock. On Linux, the lock records the PID, kernel boot ID, and /proc/<pid>/stat start time, so a crashed owner's lock is reclaimed even if its PID has been reused; a live matching owner is never stolen. Concurrent recovery is fenced by locks/<encoded-root-id>.json.lock.recover. On non-Linux hosts, process generations are unavailable and stale locks deliberately fail closed rather than using a TTL or PID-only recovery. Legacy PID-only locks and leftover recovery fences also fail closed, including on Linux. If an Apply remains locked after a crash without a verified generation, confirm no editor or writer is active for that root, then remove only that root's locks/<encoded-root-id>.json.lock and locks/<encoded-root-id>.json.lock.recover files and retry. Records are retained across reload, restart, and quit; cacheRetention does not apply to them. The process-local automatic failure cooldowns, cursors, and sticky leases remain separate and reset with the process. Manual cooldowns continue to use ~/.pi/agent/roundrobin/manual-cooldowns.json. Provider hint cooldowns use ~/.pi/agent/roundrobin/hint-cooldowns.json.

Configuration

~/.pi/agent/roundrobin/roundrobin.json

Plugin-wide settings:

Field	Default	Meaning
allowedFallback	(none)	Plugin-wide mapped fallback controls (onSticky, onExhausted, mappings, allowNonExplicit). Both switches default to false.
logUnknownModel	true	When true, append a skip unknown model provider/model line to roundrobin.log for each configured pair that is missing from the registry. Set false to stay silent.

~/.pi/agent/roundrobin/config.json

Field	Default	Meaning
virtualModel	(optional)	Virtual model metadata: name, reasoning, input, contextWindow, maxTokens, thinkingLevelMap, compat. id is forced to the group name. Omitted fields are inherited from the first candidate (see Inheritance); omit the whole block for homogeneous groups.
candidates	[]	A flat { provider, model }[] primary tier, or a list of ordered candidate tiers ([[...primary], [...secondary]]). Unknown pairs are skipped. Mixed flat/nested shapes are invalid; empty tiers are ignored with a warning.
timeoutMs	30000	Idle timeout per stream event — first byte and every mid-stream chunk race against this.
cooldownMs	60000	How long a failed candidate is skipped before it can be retried.
strategy	round-robin	round-robin / primary / random; controls order within a priority tier. See Selection strategy and stickiness.
sticky	true	true/false, systemd-style yes/no/on/off, session, last-request, or while-cached.
stickyTime	(required for timed sticky)	Positive duration such as 1h or 5h30m; used only with sticky: "session" or "last-request". while-cached derives its TTL from the active candidate name.
log	true	Append failover events to ~/.pi/agent/roundrobin/roundrobin.log. Each line includes Pi's session ID, so entries from concurrent sessions can be distinguished.
cacheRetention	7d	Retain per-session authenticated-leaf cache files for this positive duration. Supports ms, s, m, h, and d, including composites such as 5d5h.

Presets

*.json files under ~/.pi/agent/roundrobin/presets/<name>.json — each is a standalone config (same shape as config.json) and becomes a roundrobin/<name> virtual model (id = <name>, the .json suffix stripped). bak.* files are skipped. A corrupt preset is skipped silently; it never breaks the engine.

Inheritance

The virtualModel block is optional. When you omit a field (or the whole block), the engine fills it from the group's first resolved candidate — the Model from modelRegistry.find(provider, model) in your models.json. Precedence is three-tier:

1. Explicit virtualModel.<field> in the preset wins.
2. Else inherit from the first resolved candidate's Model.
3. Else a built-in default.

A homogeneous group (multiple providers of the same model) needs no virtualModel at all:

{
  "candidates": [
    { "provider": "k", "model": "glm-5.2" },
    { "provider": "wg", "model": "glm-5.2" }
  ],
  "timeoutMs": 16000,
  "cooldownMs": 120000,
  "strategy": "round-robin",
  "sticky": true
}

For heterogeneous groups (mixed models, e.g. a "power" pool of Grok + Claude + Deepseek), keep an explicit virtualModel to unify behavior across the pool (force a single compat.thinkingFormat, or a conservative contextWindow).

Inheritance happens at session_start, when the real model registry is available. Pre-session_start the engine registers with placeholder candidates so virtual models still show in /model and static lists; fields are then filled from real candidates once a session starts.

Architecture

index.ts                          ← entry: registers the roundrobin provider
extensions/
├── model-roundrobin.ts           ← failover engine (streamRoundRobin → tryCandidates)
├── rr-config.ts                  ← read config.json + presets/ → GroupConfig[]
├── candidate-chain.ts            ← pure index-rotation + nested-chain descent (unit tested)
├── duration.ts                   ← duration string parsing/formatting (unit tested)
├── routing-policy.ts             ← validated full-destination glob rules
├── routing-preview.ts            ← configured graph preview and billing builders
├── routing-policy-store.ts       ← durable root/session policy records
├── routing-policy-session.ts     ← inherited lifecycle bindings and status
└── routing-policy-ui.ts          ← transactional TUI editor and details view

Automatic failure cooldowns are process-local in-memory — they reset when pi restarts. Provider hint cooldowns persist in ~/.pi/agent/roundrobin/hint-cooldowns.json and are merged across local Pi processes, the same way as manual bans.

Authenticated-leaf cache

After authenticating a real upstream candidate, the plugin writes its selection to ~/.pi/agent/cache/roundrobin/<session-id>.json. The file contains the selection directly (no enclosing sessions map and no session ID in its contents). Each Pi session owns its file, so cooperative local sessions do not need a shared lock or a read-modify-write merge. Files older than cacheRetention are removed when a cache entry is written.

Older installations may leave roundrobin.json and roundrobin.json.lock behind. They are no longer read or used and can be removed safely.

License

MIT