Skip to content

feat(openai-compat): let a custom server declare each model's reasoning-effort levels - #100

Open
PranitSakhuja wants to merge 1 commit into
DuskBotAI:mainfrom
PranitSakhuja:feat/openai-compat-effort
Open

PranitSakhuja wants to merge 1 commit into
DuskBotAI:mainfrom
PranitSakhuja:feat/openai-compat-effort

Conversation

@PranitSakhuja

Copy link
Copy Markdown
Contributor

What and why

A custom OpenAI-compatible server never offered a reasoning effort. openai-compat.ts gave effort only to Gemini's and OpenAI's own addresses, with a fixed low/medium/high, so a relay serving a model that takes reasoning_effort (the report's gpt-6.1-sol, which takes up to max) showed no effort step and was never sent one.

An OpenAI-compatible server can now declare the levels each model takes, in its instance config:

"instances": {
  "relay": {
    "driver": "openai-compat",
    "config": {
      "url": "https://relay.example/v1",
      "effortLevels": { "gpt-6.1-sol": ["low", "medium", "high", "xhigh", "max"] }
    }
  }
}

A declared model takes exactly those levels, and they go out as reasoning_effort through the existing chosenEffort path. A model that isn't listed keeps what the address implies: OpenAI's and Gemini's rules on their own APIs, and none anywhere else. The engine advertises the union of its levels with effortByModel, so the picker offers a level only on the models that take it, as it already does for OpenAI and xAI.

decodeConfig refuses anything other than a map of model IDs to known levels, with a message naming the levels. The instance PATCH merges into the existing config, so a declared map survives the reply-wait setting being changed.

Closes #98 for servers set up in config.json.

How it was checked

server/drivers/openai-compat.test.ts:

  • a relay declaring five levels for one model advertises them, puts them on that model's catalog entry only, sends reasoning_effort: "xhigh" for it and nothing for its other model;
  • a relay declaring nothing has no effort capability, as before;
  • a malformed declaration (an unknown level, or a list instead of a map) is refused.

The first two fail on main and pass with the change. pnpm exec vitest run server/drivers/openai-compat.test.ts server/drivers/api-effort.test.ts passes all 55 tests.

  • tsc with the server config passes and pnpm lint is clean. vitest run server/drivers/openai-compat server/drivers/api-effort server/drivers/openai-chat server/routes/instances server/config passes all 603 tests. The full pnpm test runs in CI.

  • New behaviour has a test

  • No secrets, personal data or local paths in the diff

Notes for reviewers

This is the server half only. The custom-server form in Settings doesn't have an effort field yet, so for now the levels are set in config.json. If you're happy with the config shape, I can add the field to the form as a follow-up.

…ng-effort levels

Effort was offered only on OpenAI's and Gemini's own addresses. An effortLevels map in the instance config now names the levels each model takes; they are offered on that model alone and sent as reasoning_effort. Models not listed keep what the address implies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Custom OpenAI-compatible endpoints have no reasoning-effort option, even when the model supports it

1 participant