feat(UTILS): Add dfextensions packages and perfmonitor - #2180

Closed

miranov25 wants to merge 132 commits into

AliceO2Group:masterfrom

miranov25:feature/groupby-optimization

Commits on May 4, 2025

make diff of time series
miranov25
committed
e6940e9

Commits on May 29, 2025

`AliasDataFrame` is a small utility that extends `pandas.DataFrame` functionality by enabling:
miranov25
committed
8337e9a

Commits on May 31, 2025

adding perfmonitor
miranov25
committed
8ddfbf7
adding PerfromanceLogger extracted from calibration code
miranov25
committed
350f786
supressing linter warning
miranov25
committed
1ba0686

Commits on Jun 1, 2025

Add dtype support and alias dependency graph to AliasDataFrame
miranov25
committed
4a7d520
Add support for dtype persistence and alias filtering in save/load
miranov25
committed
54de3fd
Save aliases directly to pyarrow metadata
miranov25
committed
b8e241e

Commits on Jun 2, 2025

add FormulaLinearModel.py used for the dEdx and distortion calibration
miranov25
committed
fcb9bb9
add FormulaLinearModel.py used for the dEdx and distortion calibration
miranov25
committed
cfe72d4
special treatment for constants - should be enver materialized but used
miranov25
committed
9087f54
special treatment for constants
miranov25
committed
60e26cb
special treatment for constants
miranov25
committed
b188456

Commits on Jun 3, 2025

`Add ROOT SetAlias export and Python-to-ROOT AST translation for aliases`
miranov25
committed
f77f57c

Commits on Jun 4, 2025

Add dependency-aware alias materialization with optional cleanup and verbosity
miranov25
committed
664db50

Commits on Jun 9, 2025

Extended usnit test for the sub_frames
miranov25
committed
679141b
Add extended unit tests for AliasDataFrame including lazy join and error handling
miranov25
committed
6561696
fixed - Circular dependency detection
miranov25
committed
3aae8ee
fixing all unit test - except oth the automatic materialization
miranov25
committed
6759c26
fixing automatic materialization test + working in the distrtion calibration test
miranov25
committed
071a860
fixing circular depndency test - all test are OK now
miranov25
committed
ea7c0d6

Commits on Jun 10, 2025

adding unit test for the export_import tree
miranov25
committed
7389cda
add failing test for export/import of the subframes
miranov25
committed
da90789
make test_export_tree_read_tree_with_subframe already OK
miranov25
committed
64b27cb

Commits on Jun 11, 2025

adding metadata to all trees
miranov25
committed
2a6bd71
Updated documentation
miranov25
committed
9b7a038
AliasDataFrame: add index-based subframe join and robust error handling
miranov25
committed
c2e7ca6

Commits on Jun 12, 2025

AliasDataFrame: Add __getattr__ support for subframes + docstring/type hint improvements
miranov25
committed
3753500
AliasDataFrame: Add support for __getattr__ access to subframes and chained aliases
miranov25
committed
718259a
Refactor GroupByRegressor with robust fit logic, dtype casting, and unified min_stat
miranov25
committed
d55b796
Fix: ensure regression outputs are preserved for underpopulated groups
miranov25
committed
c45e5d0
Fix NaN handling in robust regression and enable predictor-specific min_stat threshold
miranov25
committed
4f4f425
Add NaN filtering and robust fit fallback logic to GroupByRegressor
miranov25
committed
22ce23c
Add flexible regression model selection via `fitter` parameter
miranov25
committed
2785bc4

Commits on Jun 13, 2025

* removing pylint warning
miranov25
committed
c3d3617
* adding __init__.py
miranov25
committed
67e3699
* adding protection for infinite recursion
miranov25
committed
27c9fbe
pylint fix
miranov25
committed
e9da107

Commits on Jun 23, 2025

adding test for the logger
miranov25
committed
d4d20e6

Commits on Jun 25, 2025

adding conversions to the function list
miranov25
committed
4d44bb2

Commits on Jun 27, 2025

adding chunksize and compression as argument
miranov25
committed
cb4b5d1
adding chunksize and compression as argument
miranov25
committed
87fa521

Commits on Aug 14, 2025

adding df drawing interface similar to the tree::Draw
miranov25
committed
4ef6973

Commits on Oct 11, 2025

docs(quantile_fit_nd): add v3.1 Δq-centered ND quantile fitting spec
miranov25
committed
512323d
Commit latest working version of AliasDataFrame
miranov25
committed
257d2ea
Commit latest working version of perfoemance_logger.py
miranov25
committed
fc54430
Commit latest working version of groupby_regression.py
miranov25
committed
161f0f0
feat(DataFrameUtils): Enhance docstrings and error handling for scatter plots
miranov25
committed
53db0b8
feat(dfextensions): add ND quantile fitting (Δq-centered) + tests & bench
miranov25
committed
0ae7eac
test(dfextensions): fix quantile ND tests vs synthetic truth; add robust edge expectations
miranov25
committed
273d6f8
fix(quantile_fit_nd): exclude q_center from nuisance axes; silence single-groupby warning
miranov25
committed
6d65a12
fix(dfextensions/quantile_fit_nd): evaluator axis bug + window-local b_min + stable inversion
miranov25
committed
b4b5b41
tests(quantile_fit_nd): snapshot pre-fix state with rich diagnostics (exclude IDE files)
miranov25
committed
a578c17
fix(quantile_fit_nd): do not floor degenerate Δq windows; keep NaN and record reason
miranov25
committed
5d9cacd
tests(quantile_fit_nd): handle Poisson via randomized PIT pre-processing
miranov25
committed
30b7ee7
docs(quantile_fit_nd): add Discrete Inputs policy and utilities
miranov25
committed
12d5fe4
bench(quantile_fit_nd): correct scaling assertions — α_b≈−0.5, α_rt≈0.0
miranov25
committed
1b2ed00
docs(quantile_fit_nd): add contextLLM.md (cold-start guide + policies)
miranov25
committed
8625857
docs(quantile_fit_nd): add contextLLM.md (cold-start guide + policies)
miranov25
committed
2b27e47
Forgottend commit of refernce test and bench log
miranov25
committed
ec9f424

Commits on Oct 22, 2025

feat(bench): add single-file GroupBy regression benchmark + reports
miranov25
committed
cd63f42
docs(groupby_regression): add Performance & Benchmarking section + fix Markdown tables
miranov25
committed
57b3293
docs(bench): set default to 5k groups; document 30% outlier scenario
miranov25
committed
7d215d3
docs(restartContext): update with 5k/5 default, 30% outliers, and leverage-outlier plan
miranov25
committed
bb51bc0
feat(groupby_regression): add optional per-group diagnostics (diag, diag_prefix)
miranov25
committed
5c9d14b

Commits on Oct 23, 2025

feat(bench): integrate class-level diagnostics summary into benchmark report
miranov25
committed
aa024b0
docs(restartContext): record diagnostics integration and real-data validation
miranov25
committed
a71cc4d
docs(restartContext): record diagnostics integration and real-data validation
miranov25
committed
cc1ecb4
use faster compression by default
miranov25
committed
5cf7431
removed accidentally comitted code
miranov25
committed
de11f93
feat(groupby_regression): add make_parallel_fit_fast (Phase 3 NumPy baseline)
miranov25
committed
3b3171c
feat(groupby): Phase 3 v4 (Numba) — 33–36× faster than v2 on tiny groups
miranov25
committed
225437c

Commits on Oct 24, 2025

feat(groupby): Phase 3 v4 (Numba) — 33–36× faster than v2 on tiny groups
miranov25
committed
8af2860
feat(groupby_regression): finalize v4 diagnostics + 200× speedup
miranov25
committed
0c4961c
test(v2/v3/v4): add verbose diagnostics for multi-target layout
miranov25
committed
4b7fa67

Commits on Oct 25, 2025

remove junk files
miranov25
committed
718ed1d
feat: create groupby_regression package structure
miranov25
committed
823c60a
refactor: move files to package structure (preserve history)
miranov25
committed
e43f332
refactor: update imports for package structure
miranov25
committed
4d90f73
test: add cross-validation tests with realistic tolerances
miranov25
committed
2997c0b
fix: handle suffix in benchmark column names
miranov25
committed
0898280
bench: limit quick mode to 2k groups, add performance warnings
miranov25
committed
a7bdc58
fix: remove relative import in fallback path (bench_groupby_regression.py)
miranov25
committed
7701367
benchmarks: add optimized-only benchmark for GroupByRegressor (v2/v3/v4 engines)
miranov25
committed
d4e54b8
bench: add optimized-only benchmark (v2/v3/v4)
miranov25
committed
ba768f6
bench: add optimized-only benchmark (v2/v3/v4)
miranov25
committed
94386df
bench: add optimized benchmark + visualization
miranov25
committed
59702d7
cleanup: remove .bak files from repository
miranov25
committed
4489253
add: .gitignore for Python project
miranov25
committed
0be66c9
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
0f28d18
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
c33b995
cleanup: remove diff.txt and prevent re-adding
miranov25
committed
d0aa5e9
refactor: remove obsolete optimized/ directory
miranov25
committed
2c3e180
refactor: complete migration to new package structure
miranov25
committed
7871783
rm diff.txt
miranov25
committed
36b1043
cleanup: finalize package structure cleanup
miranov25
committed
4290510

Commits on Oct 26, 2025

docs: finalize README.md for v2.0.0 release
miranov25
committed
f0d552d
docs: finalize README.md for v2.0.0 release
miranov25
committed
c21812d

Commits on Oct 27, 2025

docs: apply Section 1 corrections (scope, measurements)
miranov25
committed
eef2087
docs: Section 2 cleanup and focus
miranov25
committed
5b4ddb5
docs: Section 5 accuracy corrections
miranov25
committed
6a797c7
git add docs/SLIDING_WINDOW_SPEC_DRAFT.md
miranov25
committed
5fd11a5
Complete Section 6 specification with all reviewer feedback
miranov25
committed
9ae2976

Commits on Oct 28, 2025

fixing bencmark path
miranov25
committed
9271574
docs: Add Phase 7 implementation plan with synthetic data spec
miranov25
committed
34cab7d
test version 1
miranov25
committed
18bdeb3
cd ~/alicesw/O2DPG/UTILS/dfextensions/groupby_regression
miranov25
committed
194142d
feat: Add realistic TPC distortion synthetic data and validation
miranov25
committed
87724b7

Commits on Nov 8, 2025

cd ~/alicesw/O2DPG/UTILS/dfextensions/groupby_regression
miranov25
committed
97498ea

Commits on Nov 9, 2025

Add column compression support to AliasDataFrame
miranov25
committed
f2e537f
Fix precision measurement in AliasDataFrame compression
miranov25
committed
6ebf223
Add .pylintrc to suppress non-functional pylint warningsRetry
miranov25
committed
687c548
Add bidirectional atan2/arctan2 support for ROOT compatibility
miranov25
committed
39bb762
Remove IDE and system files from tracking, update .gitignore
miranov25
committed
6d19bf9
Remove IDE and system files from tracking, update .gitignore
miranov25
committed
8acda25
Remove .idea/ directory from git tracking
miranov25
committed
5ef0f6b
Remove .idea/ from git tracking and add to .gitignore
miranov25
committed
ea5965e

Commits on Nov 10, 2025

Add selective compression mode (Pattern 2) to AliasDataFrame
miranov25
committed
cc02d74
Refactor: Move AliasDataFrame to subdirectory
miranov25
committed
6359214
"Refactor: Move AliasDataFrame to subdirectory
miranov25
committed
70a2c3b
Docs: Organize documentation structure
miranov25
committed
5893349
Style: Fix pylint issues
miranov25
committed
309969a
Style: Fix pylint issues in AliasDataFrame
miranov25
committed
6c0dc8b
Style: Fix pylint issues in groupby_regression
miranov25
committed
b41160d
Style: Verify pylint scores in quantile_fit_nd All files already ≥9.0 ✅
miranov25
committed
cdff407
Refactor: Reorganize root utilities into subdirectories
miranov25
committed
cbbb57b
Style: Fix pylint issues in perfmonitor
miranov25
committed
733e5dc
Fix: Update imports after reorganization
miranov25
committed
0c098be
fix: Resolve 4 pylint errors
miranov25
committed
8ca5e2d
fix: Resolve 4 pylint CI errors
miranov25
committed
9186aec
fix: Skip aspirational formula validation test
miranov25
committed
4cb2571
fix: Skip formula validation tests and resolve pylint errors
miranov25
committed
cb0fa0a