The signatures were verified against the specified public key
What's Changed
chore(release): backport proxy request-handling maintenance and refresh runtime deps for 1.96.1 by @yuneng-berri in https://github.com/BerriAI/litellm/pull/36494
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? When using litellm with a local ollama instance and `ollama_chat` connection type, after setting the `supports_function_calling: true`, the tool calling reached the ollama provider, but the schema is missing in the UI. ###
## What happened? When an `auto_router/` marker shares its `model_name` with a plain deployment (the tag-based split: tagged requests go through the semantic router, untagged ones to the plain model), the pre-routing rewrite that swaps the alias for the routed tier model also copies litellm_params from whichever entry happens to be listed first in `model_lis
## What happened? With `enable_tag_filtering: true`, a `model_name` served by both an untagged plain deployment and an `auto_router/` marker carrying `tags: ["route"]` is meant to split traffic: requests tagged `route` go through the semantic router, untagged requests fall through to the plain deployment. Instead every request to the name goes through the se
## What happened? With `enable_tag_filtering: true` and an `auto_router/` marker carrying `tags: ["route"]`, a request tagged `route` correctly selects the marker and the semantic router picks its tier, then the request dies with a 401: `Not allowed to access model due to tags configuration. Passed model=gemini-flash and tags=['route']`. After the rewrite, d
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. --- ### What happened? **In brief:** `POST /openai_passthrough/v1/embeddings` returns 200 and consumes real billable tokens, but LiteLLM writes **no `LiteLLM_SpendLogs` row at all** — not a zero-cost row, nothing. No `x-litellm-response-cost`
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. Closest existing issues are #29187 (Bedrock **passthrough** route, error arriving *mid-stream* inside the binary event stream) and the closed #24609 (`async_sse_wrapper` has no try/except). This one is a different route and a different code p
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? As a customer / premium user, while trying to setup OAuth2 authentication on our proxy, I ran into errors with [OAUTH_TOKEN_INFO_ENDPOINT](https://github.com/BerriAI/litellm/blob/f64479e74d9d20d0edc806139b36d2a0db55a28c/lit
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? I have a model alias (`DeepSeek-Flash`) whose primary target is one model group (`DeepSeek-V-Flash-00`), with a `fallbacks` entry configured to escalate to a second model group (`DeepSeek-V-Flash-10`) if the primary fails.
# LiteLLM issue draft: Cohere Embed v4 mixed inputs are sent as texts ## Title `[Bug]: Cohere Embed v4 mixed text/image inputs are sent in the texts field` ## Check for existing issues - [x] I searched open and closed issues for Cohere Embed v4 multimodal inputs and did not find a duplicate. ## What happened? Cohere Embed v4 supports an `inputs` array whose
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? ## What I verified - Upstream logs show LiteLLM **does** forward the tools (55 TOOL) and preserves prompt caching (`CACHE` read/write present), format stays `claude→claude`. - `anthropic_beta_headers_config.json` in the con
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### The Feature ## Feature request Allow administrators to replace client-facing `budget_exceeded` messages so exact spend and budget values are not exposed. Current response: ```json { "error": { "message": "Budget has been exceeded! Current
### What happened? When calling `/v1/realtime/client_secrets` (or `/realtime/client_secrets`, `/openai/v1/realtime/client_secrets`) through the proxy using a model group name that is different from the actual underlying model string in `litellm_params.model`, the wrong model can end up being sent to the backend provider. The Router resolves the deployment co
### What happened? `OpenTelemetry.set_attributes()` writes to a caller-owned `parent_span` without checking whether that span is still recording. The guard immediately above it is `hasattr(parent_span, "set_status")`, which is true for every span object — ended or not — so the check never actually does what its own comment says. `litellm/integrations/opentel
## Summary Running `litellm --version` (the bare CLI) tracebacks with `ModuleNotFoundError: No module named 'proxy_server'`. Reproduced on `litellm[proxy]` installs across multiple versions. ## Repro (deterministic on a clean pipx environment) ```sh $ pipx install 'litellm[proxy]' $ litellm --version Traceback (most recent call last): File ".../litellm/proxy
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? ## What happened? User enables `redact_user_api_key_info: true` in `litellm_settings` with OpenTelemetry tracing enabled (`success_callback: ["otel"]`). When checking the OTEL traces in Jaeger, sensitive metadata fields lik
## What happened? On the standard Bedrock chat path (Converse API, not passthrough), 1-hour TTL prompt cache writes get billed at the 5-minute rate. This is a different root cause than #29432 (which is scoped to the Bedrock passthrough route), but the symptom is the same: 1h cache writes undercounted by about 37.5%. ## Root cause `AmazonConverseConfig._trans
### Check for existing issues Searched open + closed issues and PRs (bedrock streaming / empty chunk / chunk after finish_reason / trailing chunk / `is_model_response_stream_empty`). The closest matches are #36168, #28735 and #14106, but none cover this symptom. #25389 / #25410 are the vLLM analogue of the same code path. ### What happened? Since **v1.94.0**
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. Related (same fallthrough, different ask — please do not close this as a duplicate): - #34351 — usage dropped on `response.incomplete` / `response.failed`. PR #34352 still maps `failed` → `finish_reason=stop`. - #27186 — `response.incomplete`
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? I'm using litellm-helm to deploy my litellm, I've configured it to have a default_priority as this : ``` proxy_config: router_settings: default_priority: 10 ``` When using my client, it does not send a priority and so I get
### check for existing issues - [x] i have searched the existing issues and checked that my issue is not a duplicate. ### what happened? on litellm proxy 1.96.2, a caller can put `"api_key": "<some token>"` in the json body of `post /v1/chat/completions` even when `allow_client_side_credentials` is not set. the proxy then authenticates to the configured prov
## Summary Fixes #20456 When providers like vLLM return embedding results out of order (i.e. `result.data[0].index != 0`), the async batch embedding + Redis cache flow produces index misalignment, duplication, and missing entries in the final response. This is a regression of the same class of bug fixed in #2990. ### Root Cause (3 interrelated bugs) 1. **Cac
## Relevant issues <!-- e.g. "Fixes #000" --> ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](http
## Relevant issues <!-- e.g. "Fixes #000" --> ## Linear ticket <!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR --> ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [ ] I have Added testing in the [`tests
## Description Adds [HermesToken](https://hermestoken.top) as a new LLM provider in `model_prices_and_context_window.json`. HermesToken is an LLM token trading platform that provides access to Anthropic Claude and OpenAI models. This PR adds pricing information for five models currently available through their API. ### Models added | Model | Input ($/1M) | O
## Relevant issues ## Linear ticket Resolves LIT-2888 (parent: LIT-2878 Epic B — Cursor SDK on LiteLLM full-VM provisioning) ## Pre-Submission checklist - [ ] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory — N/A, this is a throwaway spike under `infra/spikes/` to answer Epic B's
## Relevant issues ## Linear ticket Resolves LIT-2878 ## Pre-Submission checklist - [x] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory — 42 mocked tests + 3 slow-marked real-cloud tests. - [x] My PR passes all unit tests on `make test-unit`. - [x] My PR's scope is as isolated as
## Summary When a Responses API request carries `encrypted_content` (or an encoded `encitem_…` ID) and the originating deployment is **cooled down** (e.g. during an Azure 429 storm), the affinity check in `encrypted_content_affinity_check.py` was falling back to the full healthy pool. The router then picked a different deployment, which Azure rejected with `
## Relevant issues <!-- e.g. "Fixes #000" --> ## Linear ticket <!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to magically link the Linear ticket to the GitHub PR --> ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [ ] I have Added testing in the [`tests
## Summary Bedrock Converse with custom **application inference profiles** rejects any field in `additionalModelRequestFields` that the underlying inference profile does not whitelist. Some clients (OpenAI Codex CLI, Claude Code, others adopting OpenAI's Responses API conventions) send a top-level `client_metadata` field for run-level telemetry. LiteLLM forw
## Summary Bedrock Converse requires every `toolResult` block to live in the `user` message **immediately** following the assistant message that emitted the matching `toolUse`. The OpenAI Responses API does not enforce strict adjacency, so upstream clients can legally inject a `message` between a `function_call` and its `function_call_output`. When such a pa
## Relevant issues Fixes GitHub Copilot Responses requests that include Codex's local `internal_chat_message_metadata_passthrough` field ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [x] My PR passes the targeted checks listed below - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [ ] I
## TLDR Problem this solves: - Google GenAI generateContent logging crashes with ValueError when the adapter path builds a ModelResponse without populating httpx_response - The handler unconditionally read model_call_details["httpx_response"] and raised before doing any useful work How it solves it: - Early-return when result is already a ModelResponse, skip
## TLDR Problem this solves: - Copilot now exposes MAI-Code-1.1-Flash - LiteLLM lacks its metadata and pricing How it solves it: - Adds the model to both bundled model maps - Adds focused metadata and cost coverage ## Relevant issues ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR*
## Relevant issues Fixes zero token counts reported by Gemini CLI when proxied through LiteLLM. ## Pre-Submission checklist - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR scope is as isolated as possible; it only solves 1 specific problem ## Type 🐛 Bug Fix ✅ Test ## Changes **Root cause:**
## Relevant issues ## Linear ticket ## Pre-Submission checklist - [x] I have added testing in the `tests/test_litellm/` directory - [ ] My PR passes all unit tests - [x] My PR scope is isolated to model pricing metadata - [ ] I have requested and received a Greptile review ## Screenshots / Proof of Fix The cost registry includes the six Nebius model IDs with
## Relevant issues Successor of #33614 (closed: GitHub kept a DIRTY merge state / huge file list after rebase even though the tip is a single commit on `main`). ## Pre-Submission checklist - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 s
## Relevant issues Heroku chat needs `HEROKU_API_KEY`. `validate_environment` previously skipped this provider, so missing keys looked configured (false all-clear). ## Pre-Submission checklist - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves
# fix(responses): merge assistant message + tool calls into a single chat-completions choice ## Relevant issues Fixes the message+function_call split that happens when responses-mode models return both. Related to the empty-output cluster #25429/#26179/#26309 but INDEPENDENT - this PR is model-agnostic and does not touch the chatgpt provider's SSE recovery.
## Summary Fixes #33941 When calling on an end-user that has **no existing budget** () with both and in the request body, a new budget row was created in but and were silently dropped — the budget never auto-resets. ## Root Cause Analysis The bug had **two layers**: 1. **Request parsing silently drops **: (in ) did not declare a field. Since uses pydantic v2