Skip to content

feat(llm): provider-aware LiteLLM catalog lookup - #8742

Open
Vidit Ostwal (Vidit-Ostwal) wants to merge 7 commits into
mainfrom
viditostwal/litellm-catalog-lookup
Open

Vidit Ostwal (Vidit-Ostwal) wants to merge 7 commits into
mainfrom
viditostwal/litellm-catalog-lookup

Conversation

@Vidit-Ostwal

@Vidit-Ostwal Vidit Ostwal (Vidit-Ostwal) commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Summary

Adds a provider-aware lookup into the bundled LiteLLM model catalog and exposes the resolved entry on every LLM as BaseLLM.catalog_entry.

Changes

  • New crewai/llms/model_catalog.py
    • Owns catalog loading and model-name normalization (bare_model_name, catalog_provider_prefixes), moved out of context_window.py.
    • find_model_catalog_entry(model, provider=None) returns the LiteLLM entry for an unambiguous match, or None.
    • Lookups are scoped to the provider (with CrewAI aliases such as azure_openai, aws, google, claude) and never cross provider catalogs.
    • Match precedence: exact provider/model key, then Bedrock-style deployment id (:0 / -v1 stripped), then normalized name. A match shared by several entries is left unresolved instead of picking one.
    • Without a provider, only a literal catalog key matches. The provider is lowercased once, so matching is case-insensitive for unknown providers too.
  • BaseLLM.catalog_entry: lazy, cached property resolved from model and provider on first access. It does not rely on model_post_init, so subclasses that override it keep working.
  • context_window.py: now imports the shared catalog and helpers from model_catalog.py; behavior is unchanged. The duplicated :suffix / -vN stripping is shared between bare_model_name and the deployment-name helper.
  • Entries are the catalog's own dicts, documented as read-only. No deep freezing, to keep the change small.

Tests

  • tests/llms/test_model_catalog.py: provider scoping, aliases, qualified keys, Bedrock deployment ids, missing properties, unknown provider or model, empty model, and case-insensitive provider. The ambiguity case runs against a synthetic catalog fixture, so it won't break when the bundled LiteLLM data is refreshed.
  • tests/llms/test_base_llm.py: catalog_entry uses the LLM's provider, is None for unknown models, survives a model_post_init override, and is cached.

tests/llms/test_model_catalog.py, test_base_llm.py and test_context_window.py pass (72 tests); ruff and mypy are clean. The full suite was not run.

🤖 Generated with Claude Code


Note

Medium Risk
Changes how every native LLM decides whether to send tools to providers, which can alter runtime behavior for edge models (e.g. Azure Mistral vs GPT) though explicit supports_tools and broad test coverage mitigate regressions.

Overview
This PR introduces a shared LiteLLM model catalog (model_catalog.py) with provider-scoped find_model_catalog_entry, refactors context-window sizing to use it, and exposes a cached BaseLLM.catalog_entry on every LLM.

Tool use is now resolved consistently: optional supports_tools on BaseLLM overrides the catalog’s supports_function_calling, then the provider default. Callers must use supports_function_calling()—not supports_tools—when deciding whether to attach tools. Native providers (Anthropic, Azure, Bedrock, Gemini, OpenAI, Snowflake) and the LiteLLM fallback path follow that order; request builders gate tools / toolConfig on the resolved capability instead of hard-coded flags (e.g. Azure non–OpenAI models can send tools when explicitly enabled).

Docs (AGENTS.md, LLM concept pages) describe supports_tools. Tests cover catalog lookup, catalog entry caching, cross-provider function-calling resolution, and Azure tool payload behavior.

Reviewed by Cursor Bugbot for commit 58b38d0. Bugbot is set up for automated code reviews on this repo. Configure here.

@strix-security

strix-security Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Strix Security Review

Warning

This pull request has 41 commits after the last Strix review (431dae2). Strix has not reviewed these changes.
Automatic review on push is off for this repository. To review the latest changes, tag @strix-security in a comment, or turn on re-review on push.

✅ No open security findings.

PR summary: Moves LiteLLM catalog loading and model-name normalization into a new `crewai/llms/model_catalog.py`, adds a provider-scoped `find_model_catalog_entry` lookup, and exposes the resolved entry on every LLM through a cached `BaseLLM.catalog_entry` property.

Review summary

This PR relocates the bundled LiteLLM catalog loading and model-name normalization into a new crewai/llms/model_catalog.py, adds find_model_catalog_entry(model, provider) with provider scoping, and exposes a lazily cached BaseLLM.catalog_entry. I reviewed all changed files, and the context-window refactor was verified to be value-identical (same catalog, same per-provider window tables, identical bare_model_name results), with no removed symbols still referenced elsewhere and no user-controlled input reaching the new lookup. The new lookup has real but currently unreachable quality defects: provider alias-group shadowing can return a sibling provider's entry, catalog entries are handed out as mutable module globals, and the per-instance cache does not track later model/provider changes. No production code in this repository reads catalog_entry or find_model_catalog_entry, so these cannot influence any limit, billing, or provider-trust decision today and were rejected as latent, non-exploitable issues. No security vulnerabilities are introduced by the changed code.

Ruled out (3)
  • Can a caller mutate the catalog dict returned by `catalog_entry` and poison shared catalog state process-wide?
    The entries are shared module-level dicts, but no code in the repository reads or writes `catalog_entry`, the context-window tables are independent copies, and mutating them requires already executing Python in the process, so no trust boundary is crossed.
  • Does the alias-group shadowing in `find_model_catalog_entry` let a caller obtain another provider's token limits?
    The defect reproduces (e.g. a Gemini TTS model resolving to a Vertex entry with 1,048,576 vs 8,192 max input tokens), but nothing consumes the value, pricing is unaffected, and `model`/`provider` come from developer configuration rather than attacker input.
  • Can the one-shot `catalog_entry` cache return a stale entry after `model` or `provider` is reassigned?
    The cache is intentional and pinned by a test, no library code reassigns those fields on a `BaseLLM`, and the cached value has no security-relevant consumer.

Review 1 on this PR · 431dae2 · requested Oct 28 17:01 UTC when the PR was opened


Reviewed by Strix
Re-run review · Configure security review settings

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@Vidit-Ostwal
Vidit Ostwal (Vidit-Ostwal) force-pushed the viditostwal/litellm-catalog-lookup branch from 1c4d4cf to 1f0574e Compare November 4, 2026 13:10

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

* feat(llm): add provider-aware catalog lookup

* feat(llm): read tool support from catalog

* fix(llm): respect supports_tools opt-out and log dropped tools

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(llm): merge catalog function-calling helpers and drop tool-drop warning

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* feat(llm): add explicit supports_tools setting that overrides the catalog

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(llm): let provider and user supports_tools take precedence over the catalog

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(llm): drop provider supports_tools defaults so the catalog decides

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(llm): restore provider supports_tools defaults; explicit None defers to catalog

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(llm): route litellm function-calling check through shared resolver

Let _resolve_supports_function_calling accept and return None so LLM can
reuse it before falling back to litellm; name the default explicitly in
the Anthropic and Bedrock providers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* refactor(gemini): let unset supports_tools defer to catalog then version

Drop the validator assignment that always overwrote supports_tools so
Gemini resolves explicit setting, catalog, then version heuristic like
the other providers.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* docs: document supports_tools and tool-support resolution

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(azure): gate tool attachment on supports_function_calling

_prepare_completion_params attached tools only for OpenAI model names,
while supports_function_calling() now honors supports_tools and the
LiteLLM catalog. Non-OpenAI Azure models reporting tool support silently
sent requests without tools.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* test(cli): suppress flow events in conversational fixture

* test(cli): scope conversational flow event suppression

* fix(llm): remove rebase conflict markers

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@mintlify

mintlify Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
crewai 🟢 Ready View Preview Nov 12, 2026, 7:01 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@github-actions github-actions Bot added size/XL and removed size/L labels Nov 12, 2026

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 58b38d0. Configure here.

def supports_function_calling(self) -> bool:
"""Check if the model supports function calling."""
return not self.is_o1_model
return self._resolve_supports_function_calling(not self.is_o1_model)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OpenAI path still sends disabled tools

Medium Severity

supports_function_calling() can now be false from supports_tools=False or the LiteLLM catalog, and Azure, Anthropic, Gemini, and Bedrock already omit tools from the request in that case. OpenAI native completions and Responses still attach tools whenever they are passed, and the LiteLLM fallback still puts tools on the completion params. OpenAI-compatible and Snowflake inherit that completions path, so an explicit disable does not keep tools off the wire for those providers.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 58b38d0. Configure here.

This branch was successfully deployed

1 active deployment
staging - docs — 58b38d05 Deployed Nov 12, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants