Hugging Face's logo

Spaces:
DesertWolf
/
test3
Configuration error

App Files Community
test3 / docs /my-website /release_notes /v1.71.1-stable /index.md
DesertWolf's picture
DesertWolf
Upload folder using huggingface_hub
447ebeb verified
preview code
|
raw
history blame contribute delete
15.5 kB
metadata
title: v1.71.1-stable - 2x Higher Requests Per Second (RPS)
slug: v1.71.1-stable
date: 2025-05-24T10:00:00.000Z
authors:
  - name: Krrish Dholakia
    title: CEO, LiteLLM
    url: https://www.linkedin.com/in/krish-d/
    image_url: >-
      https://media.licdn.com/dms/image/v2/D4D03AQGrlsJ3aqpHmQ/profile-displayphoto-shrink_400_400/B4DZSAzgP7HYAg-/0/1737327772964?e=1749686400&v=beta&t=Hkl3U8Ps0VtvNxX0BNNq24b4dtX5wQaPFp6oiKCIHD8
  - name: Ishaan Jaffer
    title: CTO, LiteLLM
    url: https://www.linkedin.com/in/reffajnaahsi/
    image_url: >-
      https://pbs.twimg.com/profile_images/1613813310264340481/lz54oEiB_400x400.jpg
hide_table_of_contents: false

import Image from '@theme/IdealImage'; import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem';

Deploy this version 

docker run
-e STORE_MODEL_IN_DB=True
-p 4000:4000
ghcr.io/berriai/litellm:main-v1.71.1-stable

pip install litellm==1.71.1

Key Highlights

LiteLLM v1.71.1-stable is live now. Here are the key highlights of this release:

  • Performance improvements: LiteLLM can now scale to 200 RPS per instance with a 74ms median response time.
  • File Permissions: Control file access across OpenAI, Azure, VertexAI.
  • MCP x OpenAI: Use MCP servers with OpenAI Responses API.

Performance Improvements

<Image img={require('../../img/perf_imp.png')} style={{ width: '800px', height: 'auto' }} />


This release brings aiohttp support for all LLM api providers. This means that LiteLLM can now scale to 200 RPS per instance with a 40ms median latency overhead.

This change doubles the RPS LiteLLM can scale to at this latency overhead.

You can opt into this by enabling the flag below. (We expect to make this the default in 1 week.)

Flag to enable

On LiteLLM Proxy

Set the USE_AIOHTTP_TRANSPORT=True in the environment variables. 

export USE_AIOHTTP_TRANSPORT="True"

On LiteLLM Python SDK

Set the use_aiohttp_transport=True to enable aiohttp transport. 

import litellm

litellm.use_aiohttp_transport = True # default is False, enable this to use aiohttp transport
result = litellm.completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello, world!"}],
)
print(result)

File Permissions

<Image img={require('../../img/files_api_graphic.png')} style={{ width: '800px', height: 'auto' }} />


This release brings support for File Permissions and Finetuning APIs to LiteLLM Managed Files. This is great for:

  • Proxy Admins: as users can only view/edit/delete files they’ve created - even when using shared OpenAI/Azure/Vertex deployments.
  • Developers: get a standard interface to use Files across Chat/Finetuning/Batch APIs.

New Models / Updated Models

  • Gemini VertexAI, Google AI Studio
    • New gemini models - PR 1, PR 2
      • gemini-2.5-flash-preview-tts
      • gemini-2.0-flash-preview-image-generation
      • gemini/gemini-2.5-flash-preview-05-20
      • gemini-2.5-flash-preview-05-20
  • Anthropic
    • Claude-4 model family support - PR
  • Bedrock
    • Claude-4 model family support - PR
    • Support for reasoning_effort and thinking parameters for Claude-4 - PR
  • VertexAI
    • Claude-4 model family support - PR
    • Global endpoints support - PR
    • authorized_user credentials type support - PR
  • xAI
    • xai/grok-3 pricing information - PR
  • LM Studio
    • Structured JSON schema outputs support - PR
  • SambaNova
    • Updated models and parameters - PR
  • Databricks
    • Llama 4 Maverick model cost - PR
    • Claude 3.7 Sonnet output token cost correction - PR
  • Azure
    • Mistral Medium 25.05 support - PR
    • Certificate-based authentication support - PR
  • Mistral
    • devstral-small-2505 model pricing and context window - PR
  • Ollama
    • Wildcard model support - PR
  • CustomLLM
    • Embeddings support added - PR
  • Featherless AI
    • Access to 4200+ models - PR

LLM API Endpoints

  • Image Edits
    • /v1/images/edits - Support for /images/edits endpoint - PR PR
    • Content policy violation error mapping - PR
  • Responses API
    • MCP support for Responses API - PR
  • Files API
    • LiteLLM Managed Files support for finetuning - PR PR
    • Validation for file operations (retrieve/list/delete) - PR

Management Endpoints / UI

  • Teams
    • Key and member count display - PR
    • Spend rounded to 4 decimal points - PR
    • Organization and team create buttons repositioned - PR
  • Keys
    • Key reassignment and 'updated at' column - PR
    • Show model access groups during creation - PR
  • Logs
    • Model filter on logs - PR
    • Passthrough endpoint error logs support - PR
  • Guardrails
    • Config.yaml guardrails display - PR
  • Organizations/Users
    • Spend rounded to 4 decimal points - PR
    • Show clear error when adding a user to a team - PR
  • Audit Logs
    • /list and /info endpoints for Audit Logs - PR

Logging / Alerting Integrations

Authentication & Security

  • JWT Authentication
    • Support for applying default internal user parameters when upserting a user via JWT authentication - PR
    • Map a user to a team when upserting a user via JWT authentication - PR
  • Custom Auth
    • Support for switching between custom auth and API key auth - PR

Performance / Reliability Improvements

  • aiohttp Transport
    • 97% lower median latency (feature flagged) - PR PR
  • Background Health Checks
    • Improved reliability - PR
  • Response Handling
    • Better streaming status code detection - PR
    • Response ID propagation improvements - PR
  • Thread Management
    • Removed error-creating threads for reliability - PR

General Proxy Improvements

  • Proxy CLI
    • Skip server startup flag - PR
    • Avoid DATABASE_URL override when provided - PR
  • Model Management
    • Clear cache and reload after model updates - PR
    • Computer use support tracking - PR
  • Helm Chart
    • LoadBalancer class support - PR

Bug Fixes

This release includes numerous bug fixes to improve stability and reliability:

  • LLM Provider Fixes

    • VertexAI:
      • Fixed quota_project_id parameter issue - PR
      • Fixed credential refresh exceptions - PR
    • Cohere: Fixes for adding Cohere models through LiteLLM UI - PR
    • Anthropic:
      • Fixed streaming dict object handling for /v1/messages - PR
    • OpenRouter:
      • Fixed stream usage ID issues - PR

  • Authentication & Users

    • Fixed invitation email link generation - PR
    • Fixed JWT authentication default role - PR
    • Fixed user budget reset functionality - PR
    • Fixed SSO user compatibility and email validation - PR

  • Database & Infrastructure

    • Fixed DB connection parameter handling - PR
    • Fixed email invitation link - PR

  • UI & Display

    • Fixed MCP tool rendering when no arguments required - PR
    • Fixed team model alias deletion - PR
    • Fixed team viewer permissions - PR

  • Model & Routing

    • Fixed team model mapping in route requests - PR
    • Fixed standard optional parameter passing - PR

New Contributors

Demo Instance

Here's a Demo Instance to test changes:

Git Diff