Conversation
…nctions Close gaps found by auditing the Python API against upstream DataFusion 55.1.0. - Add the any_value aggregate. - Add array_add, array_subtract, array_scale, array_sum, array_avg, array_product, and array_first, each with its list_* alias. - Add Spark monthname, weekday, atan2, hypot, pow/power, quote, and concat_ws. concat_ws calls the UDF directly so *cols stays variadic. - Fix the aggregations guide, which listed regr_slope twice and omitted regr_sxy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t kwargs - Add rand and substring_index as aliases of random and substr_index. - Add input_file_name and file_row_index, which report the source file and row offset during a file scan. - Add a distinct argument to bit_and, bit_or, mean, percentile_cont, quantile_cont, and string_agg. distinct goes before filter, matching sum and avg; the upgrade guide covers positional callers. - Fix mean, which passed filter into avg's distinct slot and raised a TypeError whenever filter was given. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add getbit, dateadd, datediff, datepart, sha, ceiling, printf, char_length, and character_length as aliases of their Spark primaries. - Add substr, whose len argument is optional as in pyspark. It calls the UDF directly because upstream expr_fn::substring always takes a length. - Rename the spark.last_day parameter from col to date to match pyspark, with an upgrade-guide note for keyword callers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tring, substr - btrim, ltrim, rtrim, and trim take an optional characters argument naming the set to strip. - array_to_string and its aliases take an optional null_string that is written in place of NULL elements. - substr takes an optional length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace NaN in floating-point columns, optionally limited to a subset. Mirrors fill_null and wraps upstream DataFrame::fill_nan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
IGNORE_NULLS skips null values when counting shift_offset rows, matching SQL LEAD/LAG ... IGNORE NULLS. The argument is appended after order_by, so existing calls are unaffected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
timsaucer
force-pushed
the
feat/upstream-coverage-gaps
branch
from
September 25, 2026 13:22
5f33af0 to
7cf8fb5
Compare
…xplain Expose the remaining upstream ExplainOption fields as keywords on DataFrame.explain. Each defaults to None, which falls back to the matching datafusion.explain.* session setting, so existing calls are unaffected. New ExplainAnalyzeLevel and ExplainMetricCategory enums sit beside ExplainFormat. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Upstream ExprFunctionExt methods on an Expr start from an empty builder, so build() resets every option not set again. The Python function wrappers already apply their keyword options, so calls such as string_agg(..., order_by=...).distinct().build() silently dropped the ordering, and .filter() dropped order_by, and so on. Seed the builder from the expression's existing params instead. A window frame equal to the default for its order_by is left unset so build() derives it again from the final order_by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AggregateUDF already accepted the capsule returned by __datafusion_aggregate_udf__ as well as an object exposing it (apache#1277). Extend the same to ScalarUDF and WindowUDF, with matching overloads on udf and udwf, so the three UDF kinds import the same way. Add FFI example tests for all three, including the previously untested AggregateUDF path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ABC is defined in datafusion.catalog beside CatalogProvider and SchemaProvider, which are in its __all__, but was only listed in the package root's __all__. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
str(df) is the Pythonic way to get a string and already uses __repr__ and the configurable formatter, so check-upstream should not flag the upstream to_string as a gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
timsaucer
force-pushed
the
feat/upstream-coverage-gaps
branch
from
September 25, 2026 13:45
7cf8fb5 to
5d43576
Compare
user_defined.py imported CapsuleType from _typeshed, which does not define it. Pyright reports `"CapsuleType" is unknown import symbol`, so _PyCapsule resolved to Unknown and every overload and parameter typed with it accepted any argument. This affected the udaf/from_pycapsule hints added in apache#1277 as well as the new udf/udwf ones. Import it from types on Python 3.13+ and from typing_extensions (already a dependency below 3.13) otherwise. Checked with pyright at --pythonversion 3.10 and 3.13: udf/udaf/udwf and each from_pycapsule accept a CapsuleType and return the right wrapper, and WindowUDF.from_pycapsule(1) is now rejected where it previously passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A built window function always stores a concrete frame, so the builder could not tell a frame the user chose from the default. A frame equal to the no-order_by default was treated as unset and re-derived as the running frame once order_by was chained, silently changing results. Record on the Python Expr whether over() or window_frame() set the frame explicitly, and pass that to builder_from_expr so the frame is kept. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Defaulting to NullTreatment.RESPECT_NULLS passed Some(RespectNulls) to Rust, which adds "RESPECT NULLS" to the generated column name and breaks code that refers to an un-aliased lead/lag output by name. Default to None instead; respecting nulls is already the behavior when unset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calling over() on an expression that was already a window function rebuilt it from an empty builder, silently dropping its order_by, null_treatment, and explicit window frame. For example, lead(v, order_by="i", null_treatment=IGNORE_NULLS).over(Window(partition_by=[g])) lost both the ordering and IGNORE NULLS. Start from builder_from_expr instead, and apply only the options the Window sets. Pass the Python-side explicit-frame flag through so a frame equal to the default is not re-derived when over() adds an order_by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
range() required all three of start, stop, and step as Expr, although upstream also accepts range(stop) and range(start, stop). Make stop and step optional in the bindings for both range and gen_series, so a single argument is the upper bound starting at 0, like Python's built-in range. Accept plain ints for start, stop, and step, coerced to literals, so callers no longer need lit(). Passing step without stop raises ValueError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
timsaucer
marked this pull request as ready for review
September 25, 2026 16:30
timsaucer
changed the title
Contributor
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Public explain enums need consistent exports, and identified documentation and distinct-option coverage gaps remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Balanced
Findings: 1
Open (3)
What changed in this PR
Expands Python coverage for DataFusion 55.1.0 APIs and fixes option preservation across expression builders.
Changes:
- Adds array, aggregate, metadata, Spark, range, and DataFrame APIs.
- Preserves aggregate/window builder options and supports bare FFI capsules.
- Adds documentation, migration guidance, and Python integration tests.
| File | Description |
|---|---|
.ai/skills/check-upstream/SKILL.md |
Records to_string audit guidance. |
crates/core/src/dataframe.rs |
Binds explain options and fill_nan. |
crates/core/src/expr.rs |
Preserves expression-builder options. |
crates/core/src/functions.rs |
Binds new functions and arguments. |
crates/core/src/spark_functions.rs |
Adds Spark function bindings. |
crates/core/src/udf.rs |
Imports bare scalar UDF capsules. |
crates/core/src/udwf.rs |
Imports bare window UDF capsules. |
docs/source/user-guide/common-operations/aggregations.md |
Updates aggregate catalog. |
docs/source/user-guide/upgrade-guides.md |
Documents breaking signature changes. |
examples/datafusion-ffi-example/python/tests/_test_aggregate_udf.py |
Tests bare aggregate capsules. |
examples/datafusion-ffi-example/python/tests/_test_scalar_udf.py |
Tests bare scalar capsules. |
examples/datafusion-ffi-example/python/tests/_test_window_udf.py |
Tests bare window capsules. |
python/datafusion/catalog.py |
Exports TableProviderFactory. |
python/datafusion/dataframe.py |
Exposes explain options and fill_nan. |
python/datafusion/expr.py |
Tracks explicit window frames. |
python/datafusion/functions/__init__.py |
Adds functions and optional arguments. |
python/datafusion/functions/spark.py |
Adds Spark functions and aliases. |
python/datafusion/user_defined.py |
Accepts bare UDF capsules. |
python/tests/test_aggregation.py |
Covers aggregate additions. |
python/tests/test_dataframe.py |
Covers DataFrame and window options. |
python/tests/test_expr.py |
Covers builder option preservation. |
python/tests/test_functions.py |
Covers scalar and array additions. |
python/tests/test_lambda.py |
Covers array_first. |
python/tests/test_spark_functions.py |
Covers Spark additions and aliases. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| """Graphviz DOT format for graph rendering.""" | ||
|
|
||
|
|
||
| class ExplainAnalyzeLevel(Enum): |
| Returns NULL if every value in the group is NULL. Which value is returned | ||
| is not specified and may differ between runs. | ||
|
|
||
| If using the builder functions described in ref:`_aggregation` this function ignores |
Comment on lines
+367
to
+368
| ("bit_and_distinct", f.bit_and(column("b"), distinct=True), [4]), | ||
| ("bit_or_distinct", f.bit_or(column("b"), distinct=True), [6]), |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue filed. The gaps were found by auditing the Python API against upstream DataFusion 55.1.0 with the
check-upstreamskill. Gaps that already have open issues (#1571–#1575, #1577, #1668, #1669) are left out.Rationale for this change
Several upstream functions, optional arguments, and DataFrame methods were not reachable from Python. The audit also turned up bugs where options on aggregate and window functions were silently dropped.
What changes are included in this PR?
array_first,any_value, file metadata functions, and a batch of Spark functions and pyspark aliases.distinct=on several aggregates,characters=on the trim functions,null_treatment=onlead/lag, and newDataFrame.explainoptions.DataFrame.fill_nan.ScalarUDFandWindowUDFaccept a bare PyCapsule, and the capsule overloads now type-check correctly.rangeandgen_series: they accept plain ints, andrange(stop)works like Python's built-in..over()onto an aggregate or window function no longer discards options already set on it (ordering, null treatment, explicit window frame).mean(x, filter=...)no longer raisesTypeError.Every new function has a doctest and pytest coverage. The same options are still dropped when
.over()converts an aggregate to a window function; that is tracked separately in #1764.Are there any user-facing changes?
Yes, new functions, arguments, and methods. Two breaking changes are documented in
docs/source/user-guide/upgrade-guides.md:distinctis inserted beforefilterinbit_and,bit_or,mean,percentile_cont,quantile_cont, andstring_agg, matchingsumandavgin 54.0.0. Passfilterby keyword.spark.last_day's parameter is renamed fromcoltodateto match pyspark.Code that chained builder methods or
.over()onto a configured function may see different results; it was previously getting silently wrong ones.🤖 Generated with Claude Code