Skip to content

Fix pandas conversion of dynamic data with heterogeneous properties - #9769

Merged
jhonabreul merged 8 commits into
QuantConnect:masterfrom
jhonabreul:bug-pandas-dynamic-data-columns
Sep 1, 2026
Merged

Fix pandas conversion of dynamic data with heterogeneous properties#9769
jhonabreul merged 8 commits into
QuantConnect:masterfrom
jhonabreul:bug-pandas-dynamic-data-columns

Conversation

@jhonabreul

@jhonabreul jhonabreul commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Description

QuantBook.UniverseHistory with flatten=True fails when the universe constituents are DynamicData instances that don't all carry the same properties, e.g. per-security rows carrying security-specific factors plus a market-wide row carrying a different one:

ValueError: Length of values (1) does not match length of index (504)

The same call without flatten works, one row per day with the constituent objects in the cell:

symbol                        time
MY-UNIVERSE.MyFactorsData 2S  2026-08-01    [MyFactorsData: AAPL, MyFactorsData: SPY, MyF...
                              2026-08-02    [MyFactorsData: AAPL, MyFactorsData: SPY, MyF...

With the fix, flatten=True returns one row per constituent, with NaN for the properties a row doesn't carry:

                              factor_a  market_factor
time       symbol
2026-08-01 AAPL R735QTJ8XC9X  0.019643            NaN
           SPY R735QTJ8XC9X  -0.004292            NaN
           $MARKET 2S              NaN           3.78
2026-08-02 AAPL R735QTJ8XC9X  0.013058            NaN
...

Cause: DynamicData columns come from each instance's storage dictionary, so data points can contribute different column sets, which the pandas conversion assumed were homogeneous.

PandasData stores a symbol's data as one series per column and both data frame builders assume every series has exactly one entry per row. Everything in this PR exists to restore that guarantee for DynamicData, filling the missing entries with NaN.

The fix:

  • PandasData.ToPandasDataFrame(IEnumerable<PandasData>, ...) (multi-symbol frame) back-fills missing values for the symbols that don't have a given series, so every column stays aligned with the symbol index. The scan only runs for symbols that don't have every known series, so homogeneous data doesn't pay for it.
  • PandasData now keeps every series aligned with the rows added. A series created on demand for a property that first appears in a later data point (previously ArgumentException: <name> key does not exist in series dictionary) is filled with missing values for the previous rows, and the series a data point doesn't have are filled for that row. Same lazy scan as above.

The second point also fixes two bugs already in master for a series that stops having a property:

  • Same symbol, same EndTime: the value silently broadcasts onto the rows that never had the property.
  • Flattened single-symbol collection: ValueError: Length of values (1) does not match length of index (2).

Related Issue

N/A

Motivation and Context

Universe and history requests for dynamic data types (custom C# DynamicData and PythonData) with per-row properties should produce NaN for the properties a row doesn't carry instead of crashing or returning wrong values.

Requires Documentation Change

No

How Has This Been Tested?

  • PandasConverterTests.FlattensBaseDataCollectionOfDynamicDataWithHeterogeneousProperties: flattens a universe collection with 3 constituents carrying different properties. Asserts row count, index names and NaN placement. Reproduces the ValueError without the fix.
  • PandasConverterTests.HandlesDynamicDataWithPropertiesAddedInLaterDataPoints: a property first appears in the second data point of a series. Asserts values and NaN for the first point. Reproduces the ArgumentException without the fix.
  • PandasConverterTests.HandlesDynamicDataWithHeterogeneousPropertiesAtTheSameTime: two data points at the same EndTime, the property present in only one of them (both orders). Asserts NaN on the other row. Reproduces the silent broadcast without the fix.
  • PandasConverterTests.FlattensSingleSymbolBaseDataCollectionOfDynamicDataWithHeterogeneousProperties: same shape through a flattened single-symbol collection. Reproduces the ValueError without the fix.
  • End-to-end QuantBook.UniverseHistory(universe, start, end, flatten=True) over a PythonData universe with 504 heterogeneous constituents per day: reproduces the exact reported error without the fix, returns the expected flattened frame with it.
  • Regression suites (PandasConverter*, QuantBook*, PythonData, AlgorithmHistoryTests, AlgorithmChainsTests): 1708 passed, 0 failed.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (non-breaking change which improves implementation)
  • Performance (non-breaking change which improves performance. Please add associated performance test and results)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Non-functional change (xml comments/documentation/etc)

Checklist:

  • My code follows the code style of this project.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

@jhonabreul
jhonabreul merged commit f24fc0d into QuantConnect:master Sep 1, 2026
7 of 8 checks passed
@jhonabreul
jhonabreul deleted the bug-pandas-dynamic-data-columns branch September 1, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants