docs: Fix max_tokens placement and response schema on Public Endpoints model pages - #847
promptless[bot] wants to merge 1 commit into
Conversation
…l pages On the native /run and /runsync route, sampling parameters must sit at the top level of the input object; nesting them under sampling_params is silently ignored. Flatten the request examples on the Moonshot Kimi and IBM Granite 4.0 pages, add the reasoning_effort parameter for kimi-k3, and correct the Moonshot Kimi response schema to the actual output array / result shape.
![promptless[bot]](https://avatars.githubusercontent.com/in/979718?s=60&v=4){kind=small}
| <ParamField body="input.sampling_params.max_tokens" type="integer" default="2048"> | ||
| Maximum number of tokens to generate. | ||
| <ParamField body="input.max_tokens" type="integer" default="2048"> | ||
| Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`. This cap counts reasoning tokens as well as visible output tokens. |
There was a problem hiding this comment.
Citation
Customer empirical A/B test on kimi-k3 native /runsync route: nested input.sampling_params.max_tokens=200 was silently ignored (23,094 completion tokens, finish_reason "stop", $0.3464), while input.max_tokens=200 at the top level was honored (200 completion tokens, finish_reason "length", $0.0030). Basis for requiring top-level placement of max_tokens on the native route.
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
| </ParamField> | ||
|
|
||
| <Note> | ||
| Sampling parameters such as `max_tokens` must sit at the top level of the `input` object on the native route. Nesting them inside a `sampling_params` object has no effect: the values are ignored, and generation continues until the model stops on its own. Check `finish_reason` in the response (`"length"` means the `max_tokens` cap was reached, `"stop"` means the model finished on its own) to confirm the cap took effect. |
There was a problem hiding this comment.
Citation
Same customer test: nesting sampling parameters inside a sampling_params object on the native route produces no error but has no effect, so generation runs unbounded; finish_reason distinguishes the two cases ("length" = cap applied, "stop" = cap ignored / model stopped on its own).
|
|
||
| <ParamField body="input.sampling_params.temperature" type="float" default="1"> | ||
| <ParamField body="input.reasoning_effort" type="string" default="max"> | ||
| Controls how much reasoning the model performs before answering. One of `low`, `high`, or `max`. Specific to the `kimi-k3` variant. |
There was a problem hiding this comment.
Citation
Moonshot AI's own docs confirm reasoning_effort accepts low/high/max (default max) and is specific to Kimi K3 as the top-level replacement for the K2.x thinking configuration.
|
|
||
| <ParamField body="input.sampling_params.temperature" type="float" default="1"> | ||
| <ParamField body="input.reasoning_effort" type="string" default="max"> | ||
| Controls how much reasoning the model performs before answering. One of `low`, `high`, or `max`. Specific to the `kimi-k3` variant. |
There was a problem hiding this comment.
Citation
Runpod's Kimi K3 technical FAQ corroborates reasoning_effort values low/high/max for the K3 variant served on this endpoint.
| { | ||
| "cost": 0.0030309, | ||
| "result": { | ||
| "id": "chatcmpl-6aaa2dce25da6d26c138da0d", |
There was a problem hiding this comment.
Citation
Response shape (output is an array; output[].result is an OpenAI-style chat completion with choices[0].message.content/reasoning_content and finish_reason; usage uses prompt_tokens/completion_tokens plus completion_tokens_details.reasoning_tokens; cost at output[].cost) comes from the customer's verified kimi-k3 run with top-level max_tokens=200.
| <ParamField body="input.sampling_params.max_tokens" type="integer" default="512"> | ||
| Maximum number of tokens to generate. | ||
| <ParamField body="input.max_tokens" type="integer" default="512"> | ||
| Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`. |
There was a problem hiding this comment.
Citation
Customer's empirical test of the top-level-vs-nested max_tokens bug was performed on kimi-k3; extrapolated here to Granite 4 on the same Public Endpoints native-route input contract. Not independently verified against a Granite-4-specific test (see open question in verification report).
| <ParamField body="input.sampling_params.max_tokens" type="integer" default="512"> | ||
| Maximum number of tokens to generate. | ||
| <ParamField body="input.max_tokens" type="integer" default="512"> | ||
| Maximum number of tokens to generate. On the native route (`/run` and `/runsync`), this must be set at the top level of `input`. |
There was a problem hiding this comment.
Citation
Internal corroboration: sibling Public Endpoints model pages (Cogito 671B, GPT-OSS 120B, Qwen3 32B) already document max_tokens/temperature/top_p/top_k at the top level of input on the native route, with no sampling_params wrapper, consistent with the corrected placement.
Source: cogito-671b.mdx:31 · 1ac8c64
1 participant
Open in Promptless
The Moonshot Kimi and IBM Granite 4.0 Public Endpoints model pages placed
max_tokensand the other sampling parameters inside a nestedsampling_paramsobject. On the native/runand/runsyncroute those nested values are silently ignored, somax_tokenshas no effect and generation runs unbounded, driving up latency and cost with no error to signal the problem.This corrects both pages so the sampling parameters sit at the top level of
input, matching the convention the other text Public Endpoints already use. On the Moonshot Kimi page it also adds thereasoning_effortparameter (top-level,low/high/max, specific tokimi-k3), adds a note that the token cap counts reasoning tokens and thatfinish_reason("length"vs"stop") confirms the cap took effect, and updates the response schema and example to the actual shape (anoutputarray whose entries carrycostand aresultchat completion, with usage reported asprompt_tokens/completion_tokens). The Granite 4.0 change is limited to the request side.Trigger Events
Tip: Enable auto-create PR in your Configuration to review suggestions directly in GitHub 🤖