The signatures were verified against the specified public key
What's Changed
chore(release): backport proxy request-handling maintenance and refresh runtime deps for 1.96.1 by @yuneng-berri in https://github.com/BerriAI/litellm/pull/36494
## TLDR Problem this solves: - Blocked model returns 403 even when a healthy fallback is configured - The fallback group never gets a chance to serve the request How it solves it: - Check the fallback chain before raising at the proxy block gate - Raise 403 only when no fallback reaches an unblocked group ## User Flow Before: a developer whose primary model
## Relevant issues <!-- e.g., "Fixes #000" --> ## Linear ticket <!-- if you are an internal contributor, add "Resolves " followed by the Linear ticket e.g., "Resolves LIT-1234" to link the Linear ticket to the GitHub PR. If you don't have one, leave the section blank rather than guessing --> ## Pre-Submission checklist **Please complete all items before aski
## TLDR Problem this solves: - Copilot reasoning models silently ignore `reasoning_effort` - Invalid effort values return 200 instead of a validation error How it solves it: - Override `map_openai_params` so the params survive mapping - Gate them on the model's reasoning capability, not its name ## User Flow Before: a developer calling a reasoning-capable Co
## TLDR Problem this solves: - `reasoning_effort: "max"` is silently ignored on responses-bridge models - Callers get a normal 200 at default reasoning depth How it solves it: - Add `max` to the accepted effort levels - Replace the per-level if/elif chain with a membership check ## User Flow Before: a developer raising reasoning depth to the highest level on
## Summary - `atranscription` / `aspeech` / `aimage_generation` now pass `custom_llm_provider` into `get_llm_provider`, matching `aembedding`. Proxy deployments that register an unprefixed model with a separate `custom_llm_provider` (no `hosted_vllm/` prefix) no longer fail with `LLM Provider NOT provided`. - Hosted vLLM transcription URL joining now follows
## Summary - Pass-through endpoints can set an optional `model` on `PassThroughGenericEndpoint`. It is written to spend/logging when the request body has no `model` (document parsers and other non-LLM forwarders). - Request-body `model` still wins when present, so LLM passthroughs keep logging `gpt-4o` instead of the endpoint hint. - Without this, those endp
## TLDR Problem this solves: - Guardrail scan payload repeats the same conversation up to three times - Image heavy requests upload roughly 95% redundant bytes - The proxy also pays to serialize that duplicate How it solves it: - Drops `messages` and `input` from the forwarded request data - Drops the same two keys from the forwarded logging details - Trims
## TLDR Problem this solves: - Async worker loops strand pending logging tasks when event loops change - Stranded tasks emit shutdown errors and lose queued callbacks How it solves it: - Keeps a logging worker registry isolated by running event loop - Starts bounded callback runners only while each loop has queued work - Drains and removes workers during exp
## TLDR Problem this solves: - Nvidia Riva is missing from the Add Model provider list - Admins cannot add a Riva deployment from the UI - Riva logs and model rows render the raw slug How it solves it: - Adds a Riva entry to the provider fields endpoint - Exposes api_base, api_key and nvcf_function_id in the form - Registers the Riva display name, slug and l
## TLDR Problem this solves: Knowledge Base direct ingestion targets Agent Runtime PUT passthrough requests are signed as POST How it solves it: Routes direct ingestion to Bedrock Agent Signs requests using the original HTTP method Preserves existing Knowledge Base Retrieve routing Adds regression coverage for both routes ## User Flow Before: a developer can
## TLDR Problem this solves: - Azure Foundry `FW-Kimi-K3` rejects Claude-style sampling / medium `reasoning_effort` - Multi-turn tool calls fail without `reasoning_content` on assistant messages How it solves it: - Mirror Moonshot Kimi shaping on `azure_ai` for FW-Kimi / kimi-* models - Drop K3 fixed sampling params and invalid `reasoning_effort` - Inject /
## TLDR Problem this solves: - Presidio upstream added 37 recognizers litellm's entity list doesn't expose - German identifiers (Steuer-ID, Personalausweis, KFZ, ...) can't be masked or blocked - Korea, Canada, Sweden, Thailand, Turkey, Nigeria, Philippines, South Africa also missing How it solves it: - Adds the 37 entities to `PiiEntityType` with 9 new cate
## TLDR Problem this solves: - Request Logs cannot be narrowed to one authenticated internal user - End User often appears as an opaque JSON blob containing a device ID, so it is not a usable way to find an internal user - Using the global user directory would exclude team admins and expose users outside the caller's Request Logs scope How it solves it: - Ad
## TLDR Problem this solves: - Bedrock Converse streams emit content after the terminal chunk. - Strict clients reject the extra assistant-role delta. How it solves it: - Metadata-only events no longer carry an assistant role. - Existing empty-chunk filtering removes the stripped metadata carrier. ## User Flow Before: a developer streaming Bedrock chat compl
## TLDR Problem this solves: - `/v1/messages` streams skip configured fallbacks after upstream errors - Failed provider streams can be logged as successful How it solves it: - Detects SSE errors and iterator transport failures - Uses fallbacks before response content reaches the client - Suppresses success logging for failed provider streams ## User Flow Bef
## Relevant issues Related to #13137, #11703 ## Linear ticket <!-- n/a --> ## Pre-Submission checklist - [x] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_code) - [
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Summary Prometheus metrics that include resolved `end_user` labels could grow one child series per distinct end user without a cleanup path. This rebuilds the cardinality cap fix on the current staging branch, keeps the existing custom-metadata label handling intact, and applies Bugbot feedback by sharing the bounded tr
## Relevant issues ## Linear ticket Resolves LIT-2881 ## Pre-Submission checklist - [x] I have Added testing - 8 Playwright specs in `ui/litellm-dashboard/e2e_tests/tests/agents/` (one per validation criterion in LIT-2881) - [ ] My PR passes all unit tests on `make test-unit` - [x] My PR's scope is as isolated as possible, it only solves 1 specific problem -
## Relevant issues ## Linear ticket Resolves LIT-2890. Stacks on top of: - LIT-2877 / PR #27330 (Epic A — `/v2/sessions` endpoint) - LIT-2878 / PR #27335 (Epic B1 — VM provider abstraction + AMI) - LIT-2891 / PR pending (Epic G — agent settings UI + `LiteLLM_AgentVMConfig`) Those PRs land first; my PR's diff against `litellm_internal_staging` shows their cha
## Relevant issues <!-- e.g. "Fixes #000" --> ## Linear ticket <!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR --> ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [ ] I have Added testing in the [`tests
## Relevant issues Fixes #33055 ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a mainta
## TLDR Problem this solves: - Bedrock Invoke streaming drops `cacheRead`/`cacheWrite` token counts from usage → cache-heavy Claude traffic billed as fresh input (4–7× over-report) - The usage-only recovery fallback also drops cache-write tokens on large/agentic streams How it solves it: - Map the two cache fields in the Invoke decoder, and read `cache_creat
## TLDR Problem this solves: - Anthropic→OpenAI tool translation mutates the caller's `input_schema` in place - Reused tool lists get polluted with non-schema keys on the second translation How it solves it: - Shallow-copy `input_schema` before assigning it to `parameters` - The vendor-kwargs loop now writes to the copy, never the source ## Relevant issues F
## TLDR Problem this solves: - Four mirror test files never execute their own module - They pass against arbitrarily broken implementations of it - Their coverage numbers are false confidence How it solves it: - Removes the four files, 1,581 lines - Each removal backed by per-file mutation evidence below - Misplaced sibling findings kept for retargeting, not
## Summary Fixes #34896 `ResetBudgetJob._reset_expired_window()` stripped tzinfo from `reset_at` **without first converting to UTC**. When `litellm_settings.timezone` is a non-UTC zone (e.g. `Asia/Tokyo`), `reset_at` carries that offset (`+09:00`), so the local wall-clock value was compared against naive UTC `now` and the window reset fired late by the timez
## Summary Fixes #34905 When a unified `pre_call` guardrail creates `litellm_metadata` alongside the Router's `metadata` dict, `add_missing_spend_metadata_to_litellm_metadata` only copied keys containing `user_api_key`. Router-set keys like `model_group`, `model_id`, and `deployment` were dropped, leaving `SpendLogsPayload.model_group` empty and breaking spe
## TLDR Problem this solves: Spend data takes too much effort to read. * Significant digits are not vertically aligned, preventing easy scanning * Most views do not need four decimal places of precision, which adds noise without helping users make decisions. > <img width="270" height="641" alt="Spend values before alignment" src="https://github.com/user-atta
## Problem When adding a Generic Guardrail API guardrail via the Admin UI (Guardrails > Add New Guardrail > Add Provider Guardrail), selecting "Generic Guardrail API" as the provider renders "No configuration fields available for this provider." The expected API Base, API Key, and optional parameter fields never appear. Closes #34927 ## Root Cause In guardra
## TLDR Problem this solves: - One poisoned managed batch aborts the entire `CheckBatchCost` poll cycle - Sibling batches in the same cycle never reconcile or bill - A metrics-backend failure while recording the error can also abort the cycle How it solves it: - Wrap each per-job poll body in its own error boundary - Make Prometheus error recording best-effo
## TLDR Problem this solves: - `proxy-server / Run tests` is red on every staging PR - #36020 made `get_agent_list` return a tuple, one assert still expects a list How it solves it: - Compare against `()` instead of `[]`, keeping the same emptiness assertion ## User Flow Not user-facing: a test-only assertion fix to unblock CI on the staging branch ## Releva
## Relevant issues ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [x] I have received a Gr
## TLDR Problem this solves: - ToolSearch results vanish during Responses translation - Providers reject the unmatched function call How it solves it: - Converts tool references into loaded-tool text - Adds the official type and regression coverage ## User Flow Before: a Claude Code user cannot continue after deferred tools are discovered 1. They send POST h
## Title `register_model()` with an empty payload silently converts "model isn't mapped" errors into $0.0 costs — `Router.__init__` triggers it for every deployment configured without pricing. ## Relevant issues None filed — found while working in this code path. Repro below. ## The bug `register_model()` always ran: ```python litellm.model_cost.setdefault(m
## TLDR <!-- Fill in the bullets below and keep each one short and concrete: one line per bullet, roughly 10 words max This section must be extremely human parsable, comprehensible, and readable: its target audience is humans, not AI agents --> Problem this solves: - <blah> - ... How it solves it: - <blah> - ... ## User Flow <!-- Two ordered lists, Before an
## TLDR Problem this solves: - Three page tests are red on `litellm_internal_staging` - They assert a denied role fires no request at all - Their names and next line say: no request for this page's data - Resolving org-admin status legitimately fetches `/organization/list` - That unrelated request fails the blanket assertion How it solves it: - Drops the bla
## TLDR Problem this solves: - Cost Optimization still used legacy dashboard UI primitives How it solves it: - Moves the route shell to installed shadcn primitives - Preserves permissions, tab behavior, and data flows ## User Flow Before: a proxy admin can use Cost Optimization, but its route shell follows the legacy dashboard visual system 1. They open `htt
## TLDR Problem this solves: - Cost Tracking helpers still used legacy UI primitives How it solves it: - Moves calculator helpers to installed shadcn primitives - Adds complete keyboard behavior to the export menu ## User Flow Before: a proxy admin can calculate discounts and export estimates, but these helpers use the legacy dashboard visual system 1. They
## TLDR Problem this solves: - Admin UI e2e tests stop at "a success toast appeared" - A save that persists nothing looks identical to one that worked - No UI test ever added a model and then called it - MCP servers had no edit or delete coverage at all How it solves it: - Every mutating flow now asserts the outgoing request body - Then reads the resource ba
## TLDR Problem this solves: - The Internal Users route shell still used legacy dashboard UI primitives How it solves it: - Moves the form-free route shell to installed shadcn primitives - Preserves proxy-admin tabs, table actions, loading, selection, and tab-panel state ## User Flow Before: a proxy admin can manage internal users, but the route shell follow
## TLDR Problem this solves: - Guardrail logs silently drop rows near the range edges - Only bites viewers off UTC, by their offset - On IST that hides 5.5 hours of every range How it solves it: - Send the picker's local day as UTC instants - Endpoint already takes a timestamp, so nothing changes server side - Tests pin non-UTC zones, where the bug is visibl
## TLDR Problem this solves: - Fully blocked primary deployments reject requests before configured fallbacks run. - Healthy fallback deployments therefore cannot recover availability. How it solves it: - Preserve the blocked-model error when no fallback deployment is available. - Let the Router fallback chain run when a fallback remains unblocked. - Add mock
## Title The dict arm of streaming usage conversion drops `prompt_tokens_details` / `completion_tokens_details` — its two sibling arms preserve them. ## Relevant issues None filed for this specific defect. **This is deliberately NOT a fix for #36168** — that issue is about SDK `CompletionUsage` objects on choices-bearing chunks and already has three open PRs
## TLDR Problem this solves: - DashScope tiered pricing was billed with graduated, income-tax-style slicing - Alibaba Model Studio instead picks one tier by total input tokens and bills the whole request at it - large Qwen requests were under-charged, and logged spend disagreed with the budget reservation code How it solves it: - select one tier from the req
## TLDR Problem this solves: - 39 test functions are collected every CI run and always skip - 30 more are byte-identical duplicates under a second name - 9 are shadowed by a later definition, so they never run - Coverage numbers count all of them as tests How it solves it: - Delete the 18 always-skipped files and the 1 demo script - Delete one of each duplic
## TLDR Problem this solves: - Eight tests are silently unreachable, shadowed by a same-named `def` - Python keeps only the last binding, so pytest never collects them - Each dead copy asserts something its survivor does not - The files still look like they cover those scenarios How it solves it: - Rename each dead copy to say what it actually covers - Only
## TLDR Problem this solves: - Shared guardrail and duration controls still depend on antd - Those dependencies block the shared-component migration track How it solves it: - Replaces both controls with installed shadcn primitives - Preserves behavior with pre-migration characterization tests ## User Flow Before: an admin sees legacy controls while managing
## TLDR Problem this solves: - Shared search and user controls still depend on antd - Those dependencies block the shared-component migration track How it solves it: - Replaces both controls with installed shadcn primitives - Preserves behavior with pre-migration characterization tests ## User Flow Before: an admin sees legacy controls while managing models