The signatures were verified against the specified public key
What's Changed
chore(release): backport proxy request-handling maintenance and refresh runtime deps for 1.96.1 by @yuneng-berri in https://github.com/BerriAI/litellm/pull/36494
## What is wrong `azure/gpt-5.6-terra` and `azure/gpt-5.6-luna`, along with their `azure/us/` and `azure/eu/` data-zone siblings, currently hold OpenAI's direct post-cut prices. Azure never made that cut. OpenAI reduced Terra by 20% and Luna by 80% on 2026-07-30. #35481 then applied the same reduction to the twelve Azure rows on 2026-08-05. Its stated basis
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? LiteLLM's native Azure Responses path forwards `additional_tools` input items without normalizing nested namespace tools. Codex CLI 0.147.0 can send a default `functions` namespace with `"description": ""`; Azure rejects th
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? `streaming_iterator.py`'s `_should_start_new_content_block` accesses `chunk.choices[0]` unconditionally. Some OpenAI-format backends send a usage-only chunk with `choices: []` near stream end. This chunk crashes the adapter
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? Guardrails Monitor does not show evaluations for five configured global `litellm_content_filter` guardrails, even though the associated request metadata lists all six guardrails in `applied_guardrails`. For recent successfu
https://github.com/BerriAI/litellm/blob/0e9cd9893e9de3221c04c6a96542a589b32b6f19/model_prices_and_context_window.json#L7539-L7594 We also need to update [model_prices_and_context_window_backup.json](https://github.com/BerriAI/litellm/blob/0e9cd9893e9de3221c04c6a96542a589b32b6f19/litellm/model_prices_and_context_window_backup.json). gpt-image-1.5 and gpt-imag
This adds a small cookbook example showing how to plug Agent Threat Rules into LiteLLM as a CustomGuardrail. ATR is an open detection standard for AI agent threats (prompt injection, tool poisoning, MCP attacks, skill compromise) released under Apache-2.0 at https://github.com/Agent-Threat-Rule/agent-threat-rules. The new file cookbook/atr_detection_callback
## Summary - add Azure AI Foundry grok-4.3 metadata with pricing from the Microsoft announcement - include 200k context metadata, cached input pricing, and capability flags - keep the backup model cost map in sync and cover routing/lookup behavior in tests ## Tests - parsed both model cost JSON files with python -m json.tool - uv run pytest tests/test_litell
Fixes #27946 ## Summary When converting Anthropic assistant messages with thinking blocks to OpenAI Chat Completions format, the `reasoning_content` field was missing. DeepSeek reasoning models (and OpenAI o-series) require this field on assistant messages in multi-turn conversation history. ## Changes In `litellm/llms/anthropic/experimental_pass_through/ada
## Relevant issues Closes #31075 (Add zai/glm-5.2 pricing entry — GLM-5.2 missing from model cost map) ## What & Why Zhipu AI (智谱) released **GLM-5.2** as its new flagship on 2026-06-13, and official pricing is now published. LiteLLM currently has `zai/glm-5` but **no `zai/glm-5.2`**, so downstream cost-resolving tools misattribute GLM-5.2 traffic to GLM-5 a
## Problem Vendors like ZhipuAI (ZAI) and DashScope return error codes **outside the standard HTTP 100–599 range** inside SSE streaming chunks. For example: - ZAI rate-limit: `1302` - ZAI quota exceeded: `1305`, `1308` - DashScope throttle codes: `4001`, `4002`, … The old `_parse_event_data_for_error()` only returned the code when `100 ≤ code ≤ 599`. Codes o
Fixes #31385 ## Problem `AgenticAnthropicStreamingIterator` and `AsyncResponsesAPIStreamingIterator` never set `completion_start_time` on the logging object. As a fallback, `litellm_logging.py` substitutes `end_time` when `completion_start_time is None`, which makes TTFT equal to total latency. ## Fix Stamp `completion_start_time` on the first non-None chunk
## Relevant issues Addresses #32086 — the core stream-swallowing (ask 1 of the issue). Asks 2 (server-side failure logging/metrics) and 3 (`output_tokens` always 0 on this path — which looks like dict-vs-attribute access in the same `response.completed` branch) are intentionally left for follow-ups to keep this PR isolated. ## Linear ticket ## Pre-Submission
## Relevant issues Fixes #33193 ## Linear ticket Resolves LIT-4261 ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific pro
## Relevant issues Non-admin teams cannot upload documents to, read the details of, or run most of the query endpoints on an Azure AI Search index through the passthrough, even when an admin has granted them access ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit te
## Summary Fixes #33772 OpenAI models that report **cache-write** tokens (`cache_write_tokens`) were being mis-priced because LiteLLM's cost path only recognized Anthropic's field name (`cache_creation_tokens`). This caused `response_cost` to be **understated** for any request that writes to the prompt cache -- cache-write tokens were billed at the plain inp
## TLDR Problem this solves: - a request needs to know which admin-owned destinations its identity is granted, resolved from its team and org - a client must not be able to name its own export destinations - an identity no grant reaches must get nothing rather than everything How it solves it: - resolves destinations at auth time and anchors them on a server
## TLDR Problem this solves: - Realtime translation endpoints were not first-class - New transcription models lacked SDK streaming support - Direct WebRTC bypasses spend tracking and budget enforcement How it solves it: - Adds translation WebSocket and WebRTC proxy paths - Uses OpenAI SDK 2.52 native Realtime methods - Adds typed transcription streaming and
## TLDR Adds `/guardrails/apply_guardrail` support for Azure guardrails (`AzureContentSafetyPromptShieldGuardrail` and `AzureContentSafetyTextModerationGuardrail`) Problem this solves: - Allows `/guardrails/apply_guardrail` to work with Azure guardrails properly How it solves it: - Both Azure classes (`AzureContentSafetyPromptShieldGuardrail` and `AzureConte
## What Router._pre_call_checks() blocks the event loop on async paths because it calls sync litellm.token_counter() directly. Offload the existing sync logic to asyncio.to_thread() so async callers can await without stalling. ## Evidence - litellm/router.py:10245 _pre_call_checks() called from async completion/embedding/etc. paths at lines 10771, 11355, 115
## What DB proxies with managed files route all `GET /v1/batches` through the managed-files hook. Filtered requests (`provider` or `target_model_names`) should fall through to provider routing instead of hitting a bare `Exception`. ## Root Cause `list_batches` unconditionally calls `managed_files.list_user_batches()` when the hook is present, even for filter
## What Router now applies cache_kwargs (e.g. type=disk) even when Redis is not configured. ## Evidence - router.py:530-535 — cache_kwargs applied before Redis block - router.py:560 — removed duplicate cache_kwargs.update() inside Redis block ## Fix cache_kwargs are now extracted and applied before the Redis conditional. The 'type' key is popped to avoid dup
## What Bedrock file upload now reports correct bytes in FileObject instead of always 0. ## Evidence - transformation.py:863-865 — _file_content_size passed through litellm_params - transformation.py:1030-1034 — fallback to _file_content_size when Content-Length missing ## Fix S3 PUT responses often omit Content-Length. We capture the actual file size during
## TLDR Problem this solves: - APScheduler anchors an `interval` job at `now + interval`, so every scheduled background job registered in one proxy startup shares a single firing instant for the life of the process, and every replica a rollout brought up together shares that instant as well - Each tick, the spend flush, the daily tag spend flush, the gateway
## TLDR Problem this solves: - Uncostable batches held poll slots forever - Once they filled the page, newer batches went uncosted - Terminal rows escaped the staleness sweep entirely How it solves it: - Retire rows with no model id in their unified id - Retire rows the provider 404s as gone - Sweep completed rows never costed past the staleness cutoff ## Us
## TLDR Problem this solves: - A DB outage silently loses every queued spend log - Prisma's P1001 masquerade escaped the retry clause entirely - Failed batches were dropped, never put back on the queue How it solves it: - Classify failures with the existing transport-error helper - Requeue the batch at the head after retries exhaust - Route every enqueue, pr
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? In our setup we only use Models which are added through the WebUI and stored in the database. No config file whatsoever is used. With this approach however all available (and working) models are not visible on the AI Hub Ma
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? **Description** When adding a vector store and specifying the `embedding_model` using a model mapping public model name, the request fails with a `BadRequestError` indicating that the LLM provider is not provided. **Additio
### Summary The Usage dashboard's **Ask AI** feature fails if the selected model is a LiteLLM model name / model group such as `mylitellmmodel`. In my setup, `mylitellmmodel` is a valid LiteLLM proxy model name that routes to GPT-4.1. It works for normal proxy requests, but the Usage AI chat feature fails with: ```text litellm.BadRequestError: GetLLMProvider
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? Hi, We're seeing an issue in a distributed setup where responses occasionally seem to be returned to the wrong client. Environment - OpenShift (multiple replicas) - Image: litellm-database (v1.81.3+) - Redis Cluster enabled
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### The Feature Allow customizing transport layer (HTTP) by passing a custom `httpx` client as a "client" argument. The customization will provide a simplified path to resolve simple tasks as an alternative to a more complex setup of litellm
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? Hello, `max_parallel_requests` counter in Redis monotonically increases when clients cancel streaming `/v1/messages` requests mid-stream. Eventually every request hits Limit type: `max_parallel_requests. Current limit: N, R
## Summary When LiteLLM is used as a **Bedrock passthrough** proxy (`/bedrock/.../invoke-with-response-stream`, e.g. Claude Code with `CLAUDE_CODE_USE_BEDROCK=1`), Anthropic cache-creation tokens written with a **1-hour TTL** (`cache_control: {"type":"ephemeral","ttl":"1h"}`) are billed at the **5-minute** rate. Bedrock **does** return the split in `usage.ca
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? ## What happened? All replicas of our LiteLLM proxy (Docker Swarm, 4 replicas) entered a continuous crash-loop: every container was killed with `task: non-zero exit (137): dockerexec: unhealthy container` every 4–8 minutes,
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? Similar to https://github.com/BerriAI/litellm/issues/19781, internal users who have been previously set a budget cannot be set back to Unlimited. <img width="2918" height="670" alt="Image" src="https://github.com/user-attac
## Check for existing issues - [x] I searched the existing issues and checked that this is not a duplicate ## What happened? When Azure OpenAI image generation or image editing is configured with `api_version: v1` or `preview`, LiteLLM Proxy builds a deployment-scoped URL instead of the integrated Azure v1 image endpoint The request is sent to `/openai/deplo
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? A bug happened!## Summary The "Ask AI" chat feature under Usage calls `litellm.acompletion()` directly (`litellm/proxy/management_endpoints/usage_endpoints/ai_usage_chat.py`), which cannot resolve proxy model aliases define
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### The Feature ### Problem The `GET /spend/logs/v2` endpoint currently filters logs **only by `startTime`** (when the request started). This creates a critical gap for billing and monitoring systems that need to track when requests **complet
## Summary Qdrant semantic cache is completely non-functional due to 4 cascading bugs. Multiple code paths access `litellm.cache.cache` directly, which only exists for Redis-based backends. For Qdrant, this raises `AttributeError: 'Cache' object has no attribute 'cache'` at startup and on every request. ## Changes 1. **proxy_server.py**: Use `getattr(litellm
## Summary Corrects the "supports_xhigh_reasoning_effort" configuration for gpt-5.4-mini models to match their OpenAI counterparts. ## Changes Fixed models (changed "supports_xhigh_reasoning_effort" from false → true): - azure/gpt-5.4-mini - azure/gpt-5.4-mini-2026-03-17 - azure/gpt-5.4-nano - azure/gpt-5.4-nano-2026-03-17 Also updated "supports_none_reasoni
## Summary The Responses API streaming transform (`LiteLLMResponsesAPIStreamingIterator`) did not handle the `response.incomplete` event type, which is sent by Azure OpenAI when generation ends due to `max_output_tokens` being reached or a content filter trigger. The event fell through to the `else: pass` branch, silently discarding `incomplete_details` and
## Relevant issues ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [x] I have received a Gr
Fixes #31078. ### Description When an internal_user has their budget exhausted, `GET /v1/models` and `GET /models` were returning `400 budget_exceeded` instead of the model list, whereas proxy admins with an exhausted budget were able to successfully access the route. The root cause was that `auth_checks.py::common_checks()` correctly identifies that `MODEL_
## Relevant issues Fixes #31233 ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [ ] I have
## Relevant issues Addresses the usage-extraction finding in #32086 ("`message_delta.usage.output_tokens` is always 0 on the openai-provider → `/v1/messages` streaming path, and every spend row records 0/0 tokens"). Complements #32475, which covers the error-event half of that issue. ## Linear ticket ## Pre-Submission checklist **Please complete all items be
## Relevant issues ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [x] I have received a Gr
## Summary - Add `litellm_settings.mcp_tool_search_default_top_k` for a global default when the model omits `top_k` - Add per-key `object_permission.mcp_tool_search_top_k` override (with DB migration) - Expose the resolved default in the virtual `mcp_tool_search` tool schema Fixes #33440 ## Test plan - [x] Added unit tests for default resolution (per-key, gl
## TLDR Problem this solves: - `safe_deep_copy` and `safe_dumps` crash on `dictionary changed size during iteration` - A concurrent hook inserting a key mid-iteration fails the request (intermittent 500s) How it solves it: - Snapshot `.items()` with `list(...)` before iterating, in both helpers ## Relevant issues Fixes #34471 ## Pre-Submission checklist - [x
## TLDR Problem this solves: - Spend rows popped for a flush vanish if the flush is cancelled - Nothing writes the spend logs queue on proxy shutdown How it solves it: - Requeue the popped batch when the write is cancelled - Drain the queue at shutdown before Prisma disconnects ## User Flow Before: a platform team rolls their proxy pods and the requests bill
## TLDR Problem this solves: - Cancelled spend flush drops the dequeued batch forever - No requeue and no shield around the DB write How it solves it: - Shield the write so cancellation lets it finish - Requeue the spend log batch at the queue head on cancellation - Requeue a cancelled tool usage batch only once its shielded flush is known to have failed, so
## TLDR Problem this solves: - there was no way to store a trace destination that belongs to the admin rather than to a team's own callback settings - a destination needs an access grant that says which identities it may receive traces for, validated when it is written rather than trusted at read time - collector tokens must never come back out of the creden