Skip to content

feat: add LiteLLM backend - #1332

Open
prodmanpd wants to merge 1 commit into
cltk:masterfrom
prodmanpd:feat/litellm-provider
Open

feat: add LiteLLM backend#1332
prodmanpd wants to merge 1 commit into
cltk:masterfrom
prodmanpd:feat/litellm-provider

Conversation

@prodmanpd

Copy link
Copy Markdown

Summary

Adds LiteLLM as a first-class CLTK generative backend for synchronous and asynchronous NLP pipelines.

The backend uses LiteLLM's OpenAI-compatible Chat Completions endpoint, so a CLTK pipeline can select any model alias configured by the gateway, including aliases backed by OpenAI, Anthropic, Azure, or gateway fallback routes.

Changes

  • add litellm to CLTK's supported backend configuration
  • add LiteLLMBackendConfig with model alias, base URL, API key, temperature, and retry settings
  • add synchronous and asynchronous LiteLLM connections
  • route dependency parsing, morphosyntax, enrichment, and translation through LiteLLM
  • preserve CLTK's structured fenced-response parsing and token accounting
  • reject missing credentials, empty model aliases, empty responses, and malformed responses with explicit errors
  • document environment-variable and structured-config usage
  • add mocked, error-path, configuration, async, and opt-in live E2E tests

Usage

import os

from cltk import NLP

os.environ["LITELLM_API_KEY"] = "your-gateway-key"
os.environ["LITELLM_BASE_URL"] = "https://litellm.example.com/v1"

nlp = NLP(
    "lati1261",
    backend="litellm",
    model="your-proxy-model-alias",
)

doc = nlp.analyze(text="Gallia est omnis divisa in partes tres.")

your-proxy-model-alias is a gateway alias, not a fixed CLTK model. Different NLP instances may select different aliases, and LiteLLM can apply provider routing or fallback behavior behind each alias.

Tests

  • full test suite: 62 passed, 2 skipped, 6 deselected
  • LiteLLM-focused tests: 15 passed, 1 opt-in E2E skipped without live environment variables
  • Ruff lint and formatting checks pass for every changed Python file
  • live E2E passed against a running LiteLLM proxy with an Azure OpenAI-backed model alias, exercising the real Chat Completions response shape and CLTK fenced-block parser
  • verified SDK error handling for invalid credentials, unknown models, rate limits, context-window errors, and timeouts
  • verified retry exhaustion and explicit rejection of missing choices, null content, and unfenced responses

The running gateway's configured Azure Anthropic deployment currently rejects requests at the upstream account-access layer; CLTK correctly surfaces that SDK failure as OpenAIInferenceError. Provider-specific routing remains owned by LiteLLM, while CLTK consumes the same OpenAI-compatible response contract for every alias.

Compatibility

  • no new required dependency; the backend reuses the existing cltk[openai] extra
  • existing backends retain their current configuration paths
  • API keys default to environment variables and are not logged
  • LiteLLM requests are intentionally non-streaming because the existing CLTK generative pipelines parse complete structured responses

@kylepjohnson

Copy link
Copy Markdown
Member

Thanks for this @prodmanpd . This is a large patch, and I am not very familiar with LiteLLM. Help me understand: If we adopted LiteLLM, the CLTK backend could use only one API by which to call upstream providers (ChatGPT, Anthropic, etc.)?

@clemsciences

Copy link
Copy Markdown
Member

Thanks for this @prodmanpd . This is a large patch, and I am not very familiar with LiteLLM. Help me understand: If we adopted LiteLLM, the CLTK backend could use only one API by which to call upstream providers (ChatGPT, Anthropic, etc.)?

I think so, but one has to run a LiteLLM instance that handles the various models. It is quite a heavy architecture for our audience, though it's still a nice-to-have for teams with the resources for it.

@prodmanpd

prodmanpd commented Aug 6, 2026

Copy link
Copy Markdown
Author

@kylepjohnson @clemsciences

Yes, that's the idea. cltk would call one interface and LiteLLM routes to whichever provider you configure (OpenAI, Anthropic, Gemini, local models), swapped through config rather than code.

On the architecture, there are two ways to do this and I'm happy to go with whichever you prefer:

  1. What the PR does now: reuse cltk's existing OpenAI client with a base URL setting. It adds no new dependency to cltk and works with any OpenAI-compatible endpoint (a LiteLLM proxy, Azure, OpenRouter, or a local model). The tradeoff is the user needs to already have such an endpoint running.

  2. Use LiteLLM's Python library directly. Here you just set the provider's own API key and pick the model, and it calls that provider's API for you. Nothing to host and no endpoint to point at, but it adds litellm as a dependency to cltk.

Both stay opt in and off by default, so nothing changes for existing users. Happy to take whichever direction fits cltk better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants