Fix pandas conversion of dynamic data with heterogeneous properties - #9769
Merged
jhonabreul merged 8 commits intoSep 1, 2026
Merged
Conversation
Martin-Molinero
approved these changes
Sep 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
QuantBook.UniverseHistorywithflatten=Truefails when the universe constituents areDynamicDatainstances that don't all carry the same properties, e.g. per-security rows carrying security-specific factors plus a market-wide row carrying a different one:The same call without
flattenworks, one row per day with the constituent objects in the cell:With the fix,
flatten=Truereturns one row per constituent, withNaNfor the properties a row doesn't carry:Cause:
DynamicDatacolumns come from each instance's storage dictionary, so data points can contribute different column sets, which the pandas conversion assumed were homogeneous.PandasDatastores a symbol's data as one series per column and both data frame builders assume every series has exactly one entry per row. Everything in this PR exists to restore that guarantee forDynamicData, filling the missing entries withNaN.The fix:
PandasData.ToPandasDataFrame(IEnumerable<PandasData>, ...)(multi-symbol frame) back-fills missing values for the symbols that don't have a given series, so every column stays aligned with the symbol index. The scan only runs for symbols that don't have every known series, so homogeneous data doesn't pay for it.PandasDatanow keeps every series aligned with the rows added. A series created on demand for a property that first appears in a later data point (previouslyArgumentException: <name> key does not exist in series dictionary) is filled with missing values for the previous rows, and the series a data point doesn't have are filled for that row. Same lazy scan as above.The second point also fixes two bugs already in master for a series that stops having a property:
EndTime: the value silently broadcasts onto the rows that never had the property.ValueError: Length of values (1) does not match length of index (2).Related Issue
N/A
Motivation and Context
Universe and history requests for dynamic data types (custom C#
DynamicDataandPythonData) with per-row properties should produceNaNfor the properties a row doesn't carry instead of crashing or returning wrong values.Requires Documentation Change
No
How Has This Been Tested?
PandasConverterTests.FlattensBaseDataCollectionOfDynamicDataWithHeterogeneousProperties: flattens a universe collection with 3 constituents carrying different properties. Asserts row count, index names andNaNplacement. Reproduces theValueErrorwithout the fix.PandasConverterTests.HandlesDynamicDataWithPropertiesAddedInLaterDataPoints: a property first appears in the second data point of a series. Asserts values andNaNfor the first point. Reproduces theArgumentExceptionwithout the fix.PandasConverterTests.HandlesDynamicDataWithHeterogeneousPropertiesAtTheSameTime: two data points at the sameEndTime, the property present in only one of them (both orders). AssertsNaNon the other row. Reproduces the silent broadcast without the fix.PandasConverterTests.FlattensSingleSymbolBaseDataCollectionOfDynamicDataWithHeterogeneousProperties: same shape through a flattened single-symbol collection. Reproduces theValueErrorwithout the fix.QuantBook.UniverseHistory(universe, start, end, flatten=True)over aPythonDatauniverse with 504 heterogeneous constituents per day: reproduces the exact reported error without the fix, returns the expected flattened frame with it.PandasConverter*,QuantBook*,PythonData,AlgorithmHistoryTests,AlgorithmChainsTests): 1708 passed, 0 failed.Types of changes
Checklist:
bug-<issue#>-<description>orfeature-<issue#>-<description>