The signatures were verified against the specified public key
What's Changed
chore(release): backport proxy request-handling maintenance and refresh runtime deps for 1.96.1 by @yuneng-berri in https://github.com/BerriAI/litellm/pull/36494
## TLDR Problem this solves: - `tool_choice: {"type": "auto"}` reaches OpenAI verbatim - OpenAI rejects it on `tool_choice.type` with a 400 - Breaks every tool call, web search included - Cursor and Claude Code both send this shape How it solves it: - Unwrap the object to the bare string - Fixed at the validation site and the Responses bridge - Function and
## What append_system_prompt_messages now handles both string and list-of-content-blocks forms of system prompt. ## Evidence - litellm_logging.py:4720 — removed isinstance(system, str) check ## Fix Pass system value through as-is (string or list), without dropping the list form. ## Test plan test_append_system_prompt_messages test case 7: verifies list form
## TLDR Problem this solves: - The PR template's rules hide inside HTML comments - Rendered PRs and agent harnesses both drop those comments - Authors, human and AI, routinely miss the rules How it solves it: - New CI job checks PR bodies against the comment rules - Enforces short bullets, no placeholders, no stray QA runbook - Re-runs on every body edit and
## TLDR Problem this solves: - `OpenTelemetry._tracer_provider_cache` (the v1 OTEL logger's per-credential-set `TracerProvider` cache) was an unbounded, unsynchronized dict that never shut anything down. Each entry owns a `BatchSpanProcessor` worker thread that only stops on `shutdown()`, so every distinct team or key scoped credential set added one live thr
## TLDR Problem this solves: - Auxiliary DB jobs run once per pod, times uvicorn workers - Seven jobs hand-roll election, with four incompatible outage policies - A lease has no renewal, so a slow job loses it mid-run - No way to keep aux work off pods serving traffic - The job-ownership gauge only ever reports zero How it solves it: - One shared helper, two
## TLDR Problem this solves: - Models and Endpoints still used legacy dashboard UI primitives How it solves it: - Moves sixteen route-owned surfaces to installed shadcn primitives - Preserves retry, pricing, health, credentials, and test flows ## User Flow Before: a proxy admin can manage model settings, but the route-owned surfaces follow the legacy dashboa
## TLDR Problem this solves: - OpenAI passthrough embeddings returned 200 with no key spend - Budget limits could be bypassed on `/openai_passthrough/v1/embeddings` How it solves it: - Bill OpenAI passthrough `/v1/embeddings` like other supported OpenAI routes - Stop Cohere `/v1/embed` matching from stealing OpenAI `/v1/embeddings` ## User Flow Before: a dev
## TLDR Problem this solves: - Module always created its own VPC, subnets, NAT - No way to reuse an existing Postgres - No way to reuse an existing Redis - Customers cannot deploy in locked-down accounts How it solves it: - `vpc_id` + subnet id inputs skip all networking - `create_database` false takes a `database_url` - `create_redis` false takes a `redis_u
## Relevant issues Reported internally: budget checks silently skipped for `auto_router` model groups, while real spend accrues on the routed deployment ## Pre-Submission checklist - [x] I have Added testing in the `tests/test_litellm` directory - [x] I have added a screenshot / evidence of my new feature working ## Type 🐛 Bug Fix ## Changes A strategy-route
## TLDR Problem this solves: - Parallel AI is search-only in LiteLLM today - Its OpenAI Responses-compatible endpoint is unreachable through the gateway - Several v1 search params (after_date, fetch_policy, location) need hand-built nesting - Flat per-request pricing tracked as $0 spend How it solves it: - Registers `parallel_ai` as an LLM provider with a na
## TLDR Problem this solves: - MCP tool grants on a key stayed stale for a minute - Revoked tools kept being served after the update - `/key/info` showed the new grants, MCP did not How it solves it: - Key updates now drop the cached permission row too - Regenerating a key drops it as well - Clears both the old and new permission ids - Broadcasts the evictio
## TLDR Problem this solves: - Guardrail sits on the user's critical path, always - Whole transcript re-sent every turn, O(n^2) per session - Base64 images sent to text-only guardrails - Embeddings and audio calls scanned pointlessly - No way to scope a guardrail to some requests How it solves it: - `fire_and_forget` dispatches the call, request proceeds imm
## TLDR Problem this solves: - The `langfuse` callback is pinned to Langfuse Python SDK v2, which Langfuse has retired. v2 clients are unsupported against Langfuse v4 servers, and Langfuse Cloud's current ingestion path expects v4 - The v2 API this callback is built on (`trace()`, `trace.generation()`, `trace.span()`) does not exist in v4, along with `langfu
## TLDR Problem this solves: - basedpyright reportAny / reportExplicitAny counts sit far above zero - silent `Any` values skip type checking and hide real bugs How it solves it: - replaces `Any` with precise types across 28 hotspot files - Protocols, TypedDicts, and Pydantic validation replace untyped access - cuts reportAny by 967 and reportExplicitAny by 2
## TLDR Problem this solves: - Gemini 3.7 Flash launched today with no cost map entry - Requests route fine but spend tracking silently records nothing How it solves it: - Adds `gemini/`, `vertex_ai/`, and bare cost map entries - Launch pricing: $0.75 input / $3.75 output per 1M tokens - Cache, batch, flex, and priority tiers scaled at the same 50% discount
## TLDR Problem this solves: - SIEM scan events don't say which virtual key caused them - Attribution needs a manual lookup in the proxy database - Key aliases are optional, renameable and reusable — unreliable alone How it solves it: - `send_meta` forwards the calling key's identity and attributes - `meta_identity_format` can send `{alias, key_id}` instead
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### The Feature When the LiteLLM router falls back from one deployment to another (e.g., `gpt-4o` on Azure East ? `gpt-4o` on Azure West, or `gpt-4o` ? `gpt-4-turbo`), the `model` field in the response and the `x-litellm-model` response heade
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### The Feature When configuring the OCI provider in LiteLLM, the parameter servingType is automatically set to ON_DEMAND by default. However, there is no option in the UI to change this value to DEDICATED. ### Motivation, pitch I'm working o
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? When configuring vector stores through the config.yaml ### Steps to Reproduce 0. Docker compose file: ``` services: litellm: image: docker.litellm.ai/berriai/litellm:v1.83.3-stable volumes: - ./litellm-config.yaml:/app/conf
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? ## Summary When the parallel request limiter returns a 429 response, the JSON error body includes the full 64-character SHA-256 hash of the offending virtual key in the `error.message` field. This identifier is then visible
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? I set liteLLM global_max_parallel_requests <img width="1324" height="185" alt="Image" src="https://github.com/user-attachments/assets/23d27165-3b79-46c9-8c02-47d47ad7cc94" /> It is hoped that the number of requests can be l
My setup is as follows: I deployed Qwen3.5-9B-AWQ locally using VLLM and used litellm as the aggregator. My litellm --config litellm_config.yaml configuration is as follows: general_settings: host: 0.0.0.0 port: 4000 debug: true master_key: "123456" model_list: - model_name: "Qwen3.5-9B-AWQ" litellm_params: model: "openai/Qwen3.5-9B-AWQ" api_base: "http://10
## Describe the bug When using Ollama with thinker models (Qwen3, DeepSeek-R1 variants), `reasoning_content` is always `null` in the LiteLLM response even though the model generates extensive internal reasoning. This means any downstream observability (Langfuse, etc.) loses the reasoning chain entirely. ## Environment - LiteLLM version: 1.83.10 - Provider: O
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? when trying to use the template MCP Security: Block Unregistered Servers, ti fails creating the mcp-security-block guardrails ### Steps to Reproduce 1. go to policies/template 2. try to create a policy from Block Unregister
### Check for existing issues - [x] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? When multiple requests are sent through a **single shared virtual key**, each carrying a different OpenAI-style `user` field, the `end_user` column in `LiteLLM_SpendLogs` (and the per-end-user spend in `LiteLLM_EndUserTable
### What happened? DB-stored auto-router / complexity-router models (`litellm_params.model = "auto_router/..."`) permanently disappear from `/v1/models`, `/model/info`, and `/model_group/info` — even for the master key — while continuing to serve completions. Once delisted they cannot be re-listed without deleting the row AND restarting the proxy. Verified l
In `litellm/proxy/management_endpoints/usage_endpoints/ai_usage_chat.py`, both `stream_usage_ai_chat()` and `_stream_final_response()` call `litellm.acompletion(model=<alias>, ...)` directly. Bare `litellm.acompletion` only infers a provider from a real provider-prefixed model string — it has no concept of virtual proxy aliases registered in `model_list`. Si
## Bug description When a `pre_call` guardrail that mutates `structured_messages` is active on the native Anthropic `/v1/messages` route, `document` content blocks (e.g. base64 PDFs) sent by the client are corrupted during the Anthropic→OpenAI→Anthropic round-trip: they are converted to OpenAI `image_url` parts on the forward translation and rebuilt as Anthr
## Summary `AnthropicMessagesConfig._normalize_system_role_messages` (introduced in #33807) hoists mid-conversation `role: "system"` messages into the top-level `system` field for Claude models without `supports_mid_conversation_system` (pre-4.8 generation). This avoids the 400 those models throw on partner endpoints (Vertex, Azure Foundry, Bedrock Invoke) —
### Check for existing issues - [ ] I have searched the existing issues and checked that my issue is not a duplicate. ### What happened? ### Summary #24516 reported `Invalid type NoneType for attribute 'gen_ai.system' value` crashing/spamming the OpenTelemetry exporter, and was fixed via #24545 and #26713 by guarding the span-attribute call site with `cast_a
Description When LiteLLM converts an OpenAPI spec into MCP tools, the generated tool's inputSchema is missing all request-body field definitions whenever the request body schema is a $ref (which is how FastAPI/Pydantic generates every spec). The tool ends up like this, giving the LLM no field information: ```` { "name": "tool_kubectl_get_post", "description"
## Summary Fixes the streaming case for Ollama tool calling that was missed in #18924. ### The Problem Ollama sends `tool_calls` in one chunk and `done: true` in a **separate** chunk. The existing fix from #18924 only checked for `tool_calls` in the current chunk when setting `finish_reason`, causing streaming to incorrectly return `finish_reason: "stop"` in
## Relevant issues <!-- N/A - Internal rebranding from Javelin to Highflame guardrails --> ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have Added testing in the [`tests/litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/litellm) directory, **Adding at least 1 test is a hard
## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have Added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory, **Adding at least 1 test is a hard requirement** - [see details](https://docs.litellm.ai/docs/extras/contributing_c
## Linear ticket Resolves LIT-2880 ## Changes New TypeScript package `@litellm/agent-sdk` at `sdks/typescript-agent-sdk/`. Mirrors Cursor SDK style with the corrected 3-level hierarchy: `Agent` (definition) -> `SessionHandle` (VM) -> `Run` (execution). Targets the `/v2/agents`, `/v2/sessions`, `/v2/sessions/{sid}/runs` HTTP API (the `/v1/agents` namespace is
## Relevant issues Resolves part 1/2 of LIT-2891. Part 2/2 (5 antd UI screens + Playwright e2e) tracked in LIT-2894. ## Linear ticket Resolves LIT-2891 (part 1/2 — backend slice) ## Pre-Submission checklist - [x] I have added testing in the [`tests/test_litellm/`](https://github.com/BerriAI/litellm/tree/main/tests/test_litellm) directory — 47 unit tests acro
Ambient is an SVM-compatible Layer-1 blockchain providing verified AI inference via Proof-of-Logits consensus. Its inference API provides OpenAI-compatible chat completions at https://api.ambient.xyz/v1. ## Relevant issues <!-- e.g. "Fixes #000" --> ## Linear ticket <!-- if you are an internal contributor, add the Linear ticket e.g. "Resolves LIT-1234" to ma
## Relevant issues ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [ ] I have received a Greptile Confidence Score of at least 4/5 after the latest changes ## Delays in PR merge? If you're seeing a delay in
## Relevant issues Fixes !31187 ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code) - [x] My PR's scope is as isolated as possible; it only solves 1 speci
## Relevant issues Fixes #31206 ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [ ] My PR passes all unit tests on [`make test-unit`](https://docs.litellm.ai/docs/extras/contributing_code) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [ ] I have requested a Greptile review by commenting
Implements ADEPT adaptive delegation for agentic routing in LiteLLM Proxy based on template matching and progressive SLM specialization from production traffic ## Pre-Submission checklist Please complete all items before asking a LiteLLM maintainer to review this PR - [x] I have added meaningful tests - [ ] My PR passes all CI/CD checks (lint, format, unit t
## Relevant issues Relates to #30941 (Bedrock Mantle Responses-only models over the Anthropic surface). Companion to the OpenAI-only messages -> Responses routing; a scoped subset of the capability-driven direction in #26088. ## Linear ticket ## Pre-Submission checklist - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, form
`get_supported_openai_params` now includes `reasoning_effort` when the Azure AI model registry marks a model as reasoning-capable. The parameter is forwarded after inherited OpenAI mapping for non-OpenAI reasoning models without removing it from the caller-override path. GPT-5 models continue through inherited effort validation and dictionary-form effort nor
VLLMPassthroughConfig did not override `logging_non_streaming_response`, so the base stub returned `None` and no `StandardLoggingPayload` was built for usage-bearing `/vllm/*` passthrough requests. The implementation now builds an `EmbeddingResponse` only for pooling, embeddings, classify, score, and rerank endpoints; defaults a missing `total_tokens` value
## Relevant issues Fixes #33872 ## Linear ticket ## Pre-Submission checklist **Please complete all items before asking a LiteLLM maintainer to review your PR** - [x] I have added meaningful tests - [x] My PR passes all CI/CD checks (e.g., lint, format, unit tests) - [x] My PR's scope is as isolated as possible; it only solves 1 specific problem - [x] I have
## Summary Support Azure OpenAI v1 image generation and editing while preserving preview behavior ## Reproduction Configure Azure image generation or image editing with `api_version: v1` or `preview`. Requests resolve to deployment-scoped endpoints instead of the integrated `/openai/v1` image endpoints ## Fix Use `/openai/v1/images/generations?api-version=pr
## TLDR <!-- Problem this solves --> - `global_spend_refresh()` leaks a DB connection on every `/global/spend/refresh` call - Leak accumulates until Postgres `max_connections` is hit → proxy-wide auth failures <!-- How it solves it --> - The per-call dedicated `PrismaClient` is now always disconnected via `try/finally` - Its long `timeout:6000` (needed for s
## TLDR Problem this solves: - Soniox SRT/VTT cues were uniform ~3s blocks, not speech-aligned - Cues split words in half, e.g. "pac" / "e." - Pauses in speech were bridged instead of separating cues - Untimestamped translated tokens could leak into subtitle cues How it solves it: - Merges subword tokens into whole words before grouping - Breaks cues on sile
## TLDR Problem this solves: - Rust `/v1/messages` only supported Anthropic and Azure - Bedrock Anthropic invoke was Python-only - Claude Code could not target the Rust gateway for Bedrock How it solves it: - Adds a Bedrock Messages provider (invoke + streaming) - Signs Bedrock requests with SigV4 on the Axum host - Transcodes AWS event-stream frames into An
## Summary Fixes #34890 `aresponses` and `anthropic_messages` route through `_ageneric_api_call_with_fallbacks_helper`, which spreads the deployment's `litellm_params` into the underlying provider call. The router-level `silent_model` directive was **not** removed there, so any deployment configured with `silent_model` made the primary Responses / Anthropic