feat: add LiteLLM backend - #1332
Conversation
|
Thanks for this @prodmanpd . This is a large patch, and I am not very familiar with LiteLLM. Help me understand: If we adopted LiteLLM, the CLTK backend could use only one API by which to call upstream providers (ChatGPT, Anthropic, etc.)? |
I think so, but one has to run a LiteLLM instance that handles the various models. It is quite a heavy architecture for our audience, though it's still a nice-to-have for teams with the resources for it. |
|
Yes, that's the idea. cltk would call one interface and LiteLLM routes to whichever provider you configure (OpenAI, Anthropic, Gemini, local models), swapped through config rather than code. On the architecture, there are two ways to do this and I'm happy to go with whichever you prefer:
Both stay opt in and off by default, so nothing changes for existing users. Happy to take whichever direction fits cltk better. |
Summary
Adds LiteLLM as a first-class CLTK generative backend for synchronous and asynchronous NLP pipelines.
The backend uses LiteLLM's OpenAI-compatible Chat Completions endpoint, so a CLTK pipeline can select any model alias configured by the gateway, including aliases backed by OpenAI, Anthropic, Azure, or gateway fallback routes.
Changes
litellmto CLTK's supported backend configurationLiteLLMBackendConfigwith model alias, base URL, API key, temperature, and retry settingsUsage
your-proxy-model-aliasis a gateway alias, not a fixed CLTK model. DifferentNLPinstances may select different aliases, and LiteLLM can apply provider routing or fallback behavior behind each alias.Tests
62 passed, 2 skipped, 6 deselected15 passed, 1 opt-in E2E skippedwithout live environment variablesThe running gateway's configured Azure Anthropic deployment currently rejects requests at the upstream account-access layer; CLTK correctly surfaces that SDK failure as
OpenAIInferenceError. Provider-specific routing remains owned by LiteLLM, while CLTK consumes the same OpenAI-compatible response contract for every alias.Compatibility
cltk[openai]extra