Job description

### **TLDR** LiteLLM is an **open-source LLM Gateway with 28K+ stars on GitHub** and trusted by companies like **NASA, Rocket Money, Samsara, Lemonade, and Adobe.** We’re rapidly expanding and seeking a performance engineer to help scale the platform to handle 5K RPS (Requests per second). We’re based in San Francisco. ### **What is LiteLLM** LiteLLM provides an **open source Python SDK and Python FastAPI Server that allows calling 100+ LLM APIs (Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic) in the OpenAI format** We just hit **$2.5M ARR** and have raised a **$1.6M seed round from Y Combinator, Gravity Fund and Pioneer Fund.** You can find more information on our [**website**](https://www.litellm.ai/), [**Github**](https://github.com/BerriAI/litellm) and [**Technical Documentation.**](https://docs.litellm.ai/docs/) ### About the Role We're hiring a Python performance engineer to own maximizing throughput, minimizing latency and ensuring our platform is reliable in production. **Roadmap for Performance Engineer:** * By end of this year our RPS and latency overhead should be at parity with industry benchmarks. Cover stream + non-stream for /chat/completions, /completions, /embeddings, /realtime, /audio/transcriptions * Reduce e2e overhead latency for cache misses. Currently at 100ms-500ms - ensure we meet industry standards. * Reduce e2e overhead latency for cache hits - ensure we meet industry benchmarks. * Ensure overhead latency scales well when other components are added to the platform - e.g Redis, Redis Cluster, DB, Non-Admin Virtual Keys * Ensure overhead latency scales well with payload size - 1MB prompt with streaming should be sub 100ms * Address customer specific and pipeline specific latency issues. * e.g. Enterprise customers reporting high overhead - this person should be able to debug these issues, get on support calls and help address any environment specific settings. * Address paying customer memory leaks * Enterprise clients have ongoing memory leaks that need resolution * Longer term - should add coverage over new endpoints - /realtime, /audio/transcriptions/, /audio/speech

Backend Performance Engineer

Backend Performance Engineer · LiteLLM

Get the Saturday tech briefing