Restructure PIPT and POPT around explicit schemes, analyses and optimizers - #166

Merged

rolfjl merged 341 commits into

Python-Ensemble-Toolbox:mainfrom

MathiasMNilsen:pr/restructure-pipt-popt

Sep 21, 2026

This pull request is big! We're only showing the most recent 250 commits

Commits on Jan 14, 2026

Comment out log.info
MathiasMNilsen
committed
f88fe0e
Reorganize code with ensemble-matrix
MathiasMNilsen
committed
c8ae932
Fix small bug
MathiasMNilsen
committed
c6ed492

Commits on Jan 15, 2026

Fix bug relating to data_mismatch for multilevel
MathiasMNilsen
committed
fc4668c
Remove unused imports for multilevel
MathiasMNilsen
committed
3cc09aa
Add progbar_settings to multilevel
MathiasMNilsen
committed
5f935aa
Update popt/update_schemes/linesearch.py

KriFos1
and
Copilot
authored
2950568
Update pipt/update_schemes/update_methods_ns/hybrid_update.py

KriFos1
and
Copilot
authored
542c51e
Update pipt/misc_tools/extract_tools.py

KriFos1
and
Copilot
authored
559c9ba
Update pipt/loop/ensemble.py

KriFos1
and
Copilot
authored
af200ca
Update optimize.py
rolfjl
authored
16ed42e
Fix bugs
MathiasMNilsen
committed
f8f1a74
Merge branch 'main' of github.com:MathiasMNilsen/PET
MathiasMNilsen
committed
e3cc78e

Commits on Jan 16, 2026

Fix bugs
MathiasMNilsen
committed
b52f83f
Update linesearch.py
MathiasMNilsen
authored
602bb6b
Update ensemble_tools.py
MathiasMNilsen
authored
fc8bd60
Merge branch 'Python-Ensemble-Toolbox:main' into main
MathiasMNilsen
authored
10ac754

Commits on Jan 21, 2026

update TrustRegion
MathiasMNilsen
committed
b227367
Fix serious bug regarding scaling of enX for popt
MathiasMNilsen
committed
32dfeab
Merge branch 'main' of github.com:MathiasMNilsen/PET
MathiasMNilsen
committed
1890720

Commits on Jan 22, 2026

Remove one-liner
MathiasMNilsen
committed
6673b4e
Improve some logic for ensemble gradient
MathiasMNilsen
committed
43eee87

Commits on Feb 3, 2026

Include adjoints from simulator
MathiasMNilsen
committed
bfc4ee1
fix bug
MathiasMNilsen
committed
903f262

Commits on Feb 4, 2026

Fix iter bug for line-search
MathiasMNilsen
committed
2d9b20d
Maake line-search logger fancier
MathiasMNilsen
committed
ea5b356
Make line-search logger fancier
MathiasMNilsen
committed
bc77354

Commits on Feb 10, 2026

Update TrustRegion and Logger
MathiasMNilsen
committed
6e2ee74

Commits on Feb 11, 2026

Include adjoints for LM-EnRML
MathiasMNilsen
committed
602846a
Small fix
MathiasMNilsen
committed
4da0778
Merge branch 'main' into main
rolfjl
authored
e0dee27
Merge branch 'Python-Ensemble-Toolbox:main' into main
MathiasMNilsen
authored
c9f3d74

Commits on Mar 3, 2026

Update truncSVD
MathiasMNilsen
committed
dd4cf6c
Change order of log_update
MathiasMNilsen
committed
6286664

Commits on Mar 4, 2026

Fix ensemble size handling and staticvar import logic
MathiasMNilsen
committed
fd9f528
Fix bug
MathiasMNilsen
committed
c915ef1

Commits on Mar 5, 2026

Merge branch 'Python-Ensemble-Toolbox:main' into main
MathiasMNilsen
authored
594f6f9

Commits on Mar 12, 2026

Update adjoint handling
MathiasMNilsen
committed
c941c2a

Commits on Mar 16, 2026

Fix logic for logging in LM-EnRML
MathiasMNilsen
committed
9fbdfdd
Include ensemble_mistfit for LM-EnRML
MathiasMNilsen
committed
af8e3c1

Commits on Mar 17, 2026

Include enAdj for ESMDA
MathiasMNilsen
committed
3e29201
Fix logger table column centering
MathiasMNilsen
committed
2c84fff

Commits on Mar 18, 2026

Merge branch 'Python-Ensemble-Toolbox:main' into main
MathiasMNilsen
authored
2866655

Commits on Mar 19, 2026

Add PETDataFrame and PETStateArray data structures
MathiasMNilsen
committed
d80dc61
Use PETDataFrame for adjoint ensemble
MathiasMNilsen
committed
38454fc

Commits on Mar 23, 2026

Add sampling function to PETStateArray
MathiasMNilsen
committed
d02c50f

Commits on Mar 24, 2026

Add tests for PET structures
MathiasMNilsen
committed
b70d7ff

Commits on Mar 25, 2026

Add CSV reader to PETDataFrame
MathiasMNilsen
committed
f2dcb13

Commits on Mar 27, 2026

Start to use PETDataFrame for data and predictions
MathiasMNilsen
committed
ceae3aa

Commits on Apr 7, 2026

Commits on Apr 8, 2026

Preserve ensemble metadata in matrix-based update flow
MathiasMNilsen
committed
fb2ae07

Commits on Apr 9, 2026

Add DataReader class and some refactoring
MathiasMNilsen
committed
0d67993

Commits on Apr 10, 2026

Include some tests for DataReader
MathiasMNilsen
committed
1cebb7b

Commits on Apr 13, 2026

Fix popt and add tests
MathiasMNilsen
committed
3d36d1b
Merge branch 'main' into feature/pet_structures
MathiasMNilsen
committed
c65105d
Move structures package into src and remove stray test artifact
MathiasMNilsen
committed
909b9b2
Fix bug
MathiasMNilsen
committed
4fd6b9e
Update Rosenbrock test
MathiasMNilsen
committed
31d6e7a
Fix bug
MathiasMNilsen
committed
b282f69
Rename some input variable
MathiasMNilsen
committed
e040a45

Commits on Apr 14, 2026

Add adjoint to EnKF
MathiasMNilsen
committed
4e8c83c
Add tests for truncSVD
MathiasMNilsen
committed
25d5da6

Commits on Apr 15, 2026

Move remove_outliers to anlysis_tools
MathiasMNilsen
committed
25a6c59
Move remove_outliers to anlysis_tools
MathiasMNilsen
committed
2386093

Commits on Apr 16, 2026

Fix bugs
MathiasMNilsen
committed
57f8927
Fix prediction save
MathiasMNilsen
committed
704203a
Small fix
MathiasMNilsen
committed
8289c00

Commits on Apr 17, 2026

Update ML with PET structures
MathiasMNilsen
committed
65112c9
Fix small bugs
MathiasMNilsen
committed
3cc273c
Fix small bugs
MathiasMNilsen
committed
375ff55

Commits on Apr 21, 2026

Fix ML bugs
MathiasMNilsen
committed
3e25b2e

Commits on Apr 22, 2026

Add scaling methods for PETDataFrame
MathiasMNilsen
committed
37761ec
Add scaling methods for PETDataFrame
MathiasMNilsen
committed
21bf9b2
Include scaling in code
MathiasMNilsen
committed
722be26

Commits on Apr 28, 2026

Update Gauss-Newton
MathiasMNilsen
committed
9278a70
Fix some bugs
MathiasMNilsen
committed
ef6a055

Commits on May 5, 2026

Small bugfix
MathiasMNilsen
committed
9a1bfd9
Add VanderPol simulator for testing
MathiasMNilsen
committed
779ba74
Add tests data assimilation workflows
MathiasMNilsen
committed
13b22ee

Commits on May 13, 2026

Add adjoint in Approx update
MathiasMNilsen
committed
5e8c158

Commits on May 21, 2026

Refactor Assimilate with AI
MathiasMNilsen
committed
78c3103
Fox bugs for sparse data run
MathiasMNilsen
committed
ae1cb8b
Fix small bug
MathiasMNilsen
committed
e96f6d2
Remove old data-reader methods
MathiasMNilsen
committed
c1e1ce3
Save DataFrames as list of records in debug files
MathiasMNilsen
committed
f1a3a70

Commits on May 22, 2026

Add report-pont reader
MathiasMNilsen
committed
ced3315

Commits on May 26, 2026

Commits on May 27, 2026

Update tests
MathiasMNilsen
committed
ef60115
Remove NPV function, it is relocated to SimulatorWraps
MathiasMNilsen
committed
97b70cd

Commits on Jul 3, 2026

small fix
MathiasMNilsen
committed
b419905

Commits on Jul 6, 2026

Remove npv import
MathiasMNilsen
committed
45d3d3b

Commits on Jul 7, 2026

Add new base class for optimization loop
MathiasMNilsen
committed
ce61785
Remove scaling from POPT ensembles and cleaned them up.
MathiasMNilsen
committed
765d637
Update TrustRegion
MathiasMNilsen
committed
a693852
Small change in func wrapper
MathiasMNilsen
committed
4a5c87f
Fix bug
MathiasMNilsen
committed
221584f
Refactor Enopt and test
MathiasMNilsen
committed
4e2cc8f
Re-write SmcOpt with AI
MathiasMNilsen
committed
c7f80f6
small test fix
MathiasMNilsen
committed
c048932
Improve docstrings for EnOpt and TrustRegion
MathiasMNilsen
committed
8d665f7
Better name structure for PET ensemble
MathiasMNilsen
committed
fbabf66
Update install dependencies
MathiasMNilsen
committed
d279997
Fix install bug
MathiasMNilsen
committed
01ddc69
Fix install bug
MathiasMNilsen
committed
15dca2c
Replace row_stack (depricated)
MathiasMNilsen
committed
15da17b

Commits on Aug 11, 2026

refactor: introduce pipt.localization package and modernise update schemes
MathiasMNilsen
committed
63af4c0
Small logic fix
MathiasMNilsen
committed
14debe8

Commits on Aug 12, 2026

fix(distance_loc): correct CSV field order, actnum+z_range, and import taper
MathiasMNilsen
committed
fdc4f7f
Improve localizations
MathiasMNilsen
committed
5c7959f
Fix logic
MathiasMNilsen
committed
b7cd6f7

Commits on Aug 13, 2026

Modernize packaging, tooling, and CI
claude
committed
0ee6f4c
Add pet CLI for validating and converting config files
claude
committed
5cd9c6f
Clean up lint findings and fix latent bugs across src
claude
committed
b52d359
Merge feature/refactor_loc: adopt pipt.localization package
claude
committed
e0206d0
Add AssimilationSchemeBase, the PIPT counterpart to popt's OptimizerBase
claude
committed
adc2946
Add AnalysisStrategy base, deduplicating the analysis-flavour helpers
claude
committed
751d2c0
Replace importlib scheme dispatch with an explicit registry
claude
committed
8f0b602
Add algorithm-level scheme constructors, flavour as an argument
claude
committed
994af67
Replace the daalg config key with scheme, and add `pet migrate`
claude
committed
3dba9c9
Tidy repo root: relocate example CSV, drop stray artifact
claude
committed
b1ed4dd
Add CHANGELOG documenting the breaking config change
claude
committed
52fd87e

Commits on Aug 14, 2026

Add characterisation tests pinning current numerical output
claude
committed
6dbfe0a
Split pipt/loop/ensemble.py into a pipt/ensembles/ package
claude
committed
d6fb280
Add Phase 8 handover document
claude
committed
a22162c
Preserve formatting and comments in `pet migrate`
claude
committed
83e8855
Move the forecast from the assimilation loop onto the ensemble
claude
committed
d2b0ff2
Add characterisation coverage for es before migrating it
claude
committed
ceff647
Migrate esmda onto AssimilationSchemeBase
claude
committed
b4417ee
Migrate lmenrml and gnenrml onto AssimilationSchemeBase
claude
committed
3721753

Commits on Aug 17, 2026

Fix enkf's dangling full_cov_data reference and characterise it
claude
committed
193a91a
Fix ES discarding its own update
claude
committed
c17f93e
Migrate es and enkf onto AssimilationSchemeBase
claude
committed
3dd91ac
Delete Assimilate; schemes own their loop
claude
committed
3b56980
Make Scheme.assimilate() take what the constructor takes
claude
committed
d18cc7c
Stop the factory's default from overriding the config's analysis
claude
committed
d531e04
Let an analysis strategy bind to a scheme instead of being mixed in
claude
committed
502f67a
Write strategy attributes through to the bound scheme
claude
committed
20962dd
Collapse the eighteen scheme classes into five
claude
committed
c02be5c
Drive the characterisation cases through Scheme.assimilate()
claude
committed
b25625f
Move the remaining tests onto the class entry points
claude
committed
a8f1b4b

Commits on Aug 18, 2026

Commits on Aug 19, 2026

Make the prior forecast's misfit available to the iteration-0 artifacts
claude
committed
0dc9b65
Rename analysisdebug to savedata and its output to assimilation_result_{i}
claude
committed
545b87e
Completed PIPT tutorial
MathiasMNilsen
committed
275900a
Rename the loop methods for the job they do, in both packages
claude
committed
1336345
Remake the popt tutorial to match the pipt one
claude
committed
7987cf2
Make the popt tutorial's run cells safe to re-execute
claude
committed
4b70346
Stop popt swallowing TypeErrors raised inside the objective
claude
committed
8a4f2d5
Clear En_<member> scratch folders before each prediction, not only at init
claude
committed
166a9b6
Report an objective that fits neither call shape, rather than guessing
claude
committed
92eed49

Commits on Aug 20, 2026

Collapse eighteen per-flavour scheme classes into COMPATIBLE_ANALYSES dicts

MathiasMNilsen
and
claude
committed
ad595a8
Check convergence after every attempt, not only accepted iterations

MathiasMNilsen
and
claude
committed
fa7d92a
Replace the popt 3WELL tutorial with a 5SpotLineSearch walkthrough

MathiasMNilsen
and
claude
committed
73fc94a
Move the pipt tutorial into TinyBox/, add a GN-EnRML/margis example

MathiasMNilsen
and
claude
committed
008d37f
Remove stale Phase 8 handover doc

MathiasMNilsen
and
claude
committed
8edae63

Commits on Aug 24, 2026

Make logit/logger_name real options, defaulting to ASSIM.log

MathiasMNilsen
and
claude
committed
c16df9a
Replace scheme->ensemble __getattr__ with declared properties; unify naming

MathiasMNilsen
and
claude
committed
ab20d4c
Report the criterion that stopped a converged EnRML run; fix lint

MathiasMNilsen
and
claude
committed
dff6dc1
Finish check_state_convergence: snapshot enX_old in the loop

MathiasMNilsen
and
claude
committed
8f1efe6
Trim inline commentary in scheme_base to the load-bearing parts

MathiasMNilsen
and
claude
committed
b945dcd
Assign step_accepted directly from update_step()

MathiasMNilsen
and
claude
committed
2ce5fde
add localization tutorials
MathiasMNilsen
committed
c77ecf7
Add tutorials for writing a new analysis and a new scheme

MathiasMNilsen
and
claude
committed
9c05db8
Move after_analysis off the scheme base

MathiasMNilsen
and
claude
committed
0d1a2f6
Replace the optional-hooks table with the actual call order

MathiasMNilsen
and
claude
committed
40931ed
Document what update_step() must leave behind

MathiasMNilsen
and
claude
committed
b15df91
Return a StepReport from update_step() instead of a bool

MathiasMNilsen
and
claude
committed
b249dce
Rename data_misfit to data_misfit_mean on the schemes

MathiasMNilsen
and
claude
committed
838c11d
Fix a docstring the data_misfit_mean rename missed

MathiasMNilsen
and
claude
committed
f684b0c
Move the state commit into StepReport and the loop

MathiasMNilsen
and
claude
committed
6b87184
Say why enX_temp is cleared, and drop an import the change orphaned

MathiasMNilsen
and
claude
committed
5166c49
Pass the state to forecast instead of parking it in enX_temp

MathiasMNilsen
and
claude
committed
0ac5454
Update the scheme tutorial's prose for the forecast(state) change

MathiasMNilsen
and
claude
committed
9de972b
Add the state field to the tutorial's StepReport snippet

MathiasMNilsen
and
claude
committed
a6cfc04
Move the shared log_update row to the scheme base

MathiasMNilsen
and
claude
committed
a5440d4
Give the scheme tutorial a config it actually reads

MathiasMNilsen
and
claude
committed
a2a173c
Bring the scheme checklist in line with the contract

MathiasMNilsen
and
claude
committed
5c06046
Make the tutorial scheme log its iterations

MathiasMNilsen
and
claude
committed
5f67d0d
Trim the loop's commentary back to what a reader can get wrong

MathiasMNilsen
and
claude
committed
16b5676
Tidy the execution counts in the auto-adaptive localization tutorial

MathiasMNilsen
and
claude
committed
bae3533

Commits on Aug 25, 2026

Move the extending tutorials under pipt

MathiasMNilsen
and
claude
committed
841ff69
Add comments
MathiasMNilsen
committed
8de7339

Commits on Aug 28, 2026

Move the inner damping loop into update_step, and scoring into score()

MathiasMNilsen
and
claude
committed
5d5de88

Commits on Sep 7, 2026

Keep at least the requested energy fraction in truncSVD

MathiasMNilsen
and
claude
committed
a371527
Fix the idX offset for the second and later prior variables

MathiasMNilsen
and
claude
committed
6391524
Move a replaced member's adjoint together with its state

MathiasMNilsen
and
claude
committed
45badac
Stop the analyses allocating what they never use

MathiasMNilsen
and
claude
committed
82d7387
Import QA/QC, wavelets and geostat only where they are used

MathiasMNilsen
and
claude
committed
03fc1bd
Drop mako, psutil and six; pin geostat to a commit

MathiasMNilsen
and
claude
committed
fe7f90d
Drop two full-ensemble copies that nothing reads

MathiasMNilsen
and
claude
committed
c1669df
Remove code with no callers

MathiasMNilsen
and
claude
committed
cdff13d
Make co_lm_enrml and gn_enrml thin names for the algorithms they were

MathiasMNilsen
and
claude
committed
2e2c8f6
Run every test in a temporary directory; seed the pipeline tests

MathiasMNilsen
and
claude
committed
af394aa
Root the API reference at src and rewrite the developer guide for this repository

MathiasMNilsen
and
claude
committed
e0eb2bd
Give every module a docstring so the API reference lists it

MathiasMNilsen
and
claude
committed
e55ce9a
Pin the five untested scheme/analysis pairs in the golden suite

MathiasMNilsen
and
claude
committed
d4006de
Remove the twelve analysis_tools functions nothing calls

MathiasMNilsen
and
claude
committed
fb89d9c
Clip through PETStateArray in EnKF; drop the duplicate clipper and three dead ensemble_tools functions

MathiasMNilsen
and
claude
committed
e2d96e1
Let esmda_hybrid inherit update_step and check_convergence

MathiasMNilsen
and
claude
committed
f73b275
Write down the simulator contract as ensemble.protocols.ForwardSimulator

MathiasMNilsen
and
claude
committed
3039b08
Record the unwhitened subspace anomalies as a known issue, with a strict xfail

MathiasMNilsen
and
claude
committed
2b41a93
Include pet_cli in the standalone-import check

MathiasMNilsen
and
claude
committed
9dc4222

Commits on Sep 8, 2026

Whiten the predicted anomalies in the subspace analysis

MathiasMNilsen
and
claude
committed
01767a6
Score EnKF and ES against the data covariance, not its square root

MathiasMNilsen
and
claude
committed
24f1b80
Apply the state scaling in the approx and full analyses

MathiasMNilsen
and
claude
committed
9efdf2f
Replace a crashed member's state along with its prediction; make the null localization picklable

MathiasMNilsen
and
claude
committed
34961d6
Place localization kernels with their own axes

MathiasMNilsen
and
claude
committed
0d8796a
Refuse the unsupported screendata option with an explanation

MathiasMNilsen
and
claude
committed
7b1688b
Fix five verified bugs in popt's numerical subroutines

MathiasMNilsen
and
claude
committed
9b2254e
Fix six small crash and correctness bugs across misc, simulator and pipt

MathiasMNilsen
and
claude
committed
acb5637
Port QA/QC to the ensemble's frames

MathiasMNilsen
and
claude
committed
1318945
Return analysis results instead of writing them onto the scheme

MathiasMNilsen
and
claude
committed
dfd9c70
Collapse LM-EnRML and GN-EnRML onto one IterativeEnRML base

MathiasMNilsen
and
claude
committed
f6b4324
Let schemes take a ready-made ensemble; select localization from a registry

MathiasMNilsen
and
claude
committed
236bf41
Give each log file its own logger; raise instead of exiting; create folders only when writing

MathiasMNilsen
and
claude
committed
2a1b976
Draw every random number from the ensemble's stream; add a `seed` option

MathiasMNilsen
and
claude
committed
7a199fd
Honour restart_sim_results.pkl only on a restart; fix popt's save_prediction

MathiasMNilsen
and
claude
committed
ce1ce58
Let OptimizerBase own the choreography around a step

MathiasMNilsen
and
claude
committed
6c9c9bc
Unify restart on RestartMixin: one checkpoint the scheme writes and reads

MathiasMNilsen
and
claude
committed
fff9583
Split calc_prediction into orchestration and four named steps

MathiasMNilsen
and
claude
committed
8d59232

Commits on Sep 9, 2026

Remove the uncalled module-level readers from read_input_csv

MathiasMNilsen
and
claude
committed
0895fbd
Add DataLayout; build the observation vector from it

MathiasMNilsen
and
claude
committed
c44529f
Fill the prediction matrix directly from the member outputs

MathiasMNilsen
and
claude
committed
d4767ea
Keep the forecast as the members returned it; adjoints as an array

MathiasMNilsen
and
claude
committed
2469fae
Build the observation variance from the layout; retire construct_data_cov

MathiasMNilsen
and
claude
committed
4cbed9f
Replace PETStateArray with a plain state array and a StateLayout

MathiasMNilsen
and
claude
committed
b4f273e
Tutorials: describe the layout-ordered matrices, not the frame flatten

MathiasMNilsen
and
claude
committed
fb590f1

Commits on Sep 17, 2026

Move StateLayout into layout.py

MathiasMNilsen
and
claude
committed
5c1e370
Compress seismic vintages as the prediction matrix is filled

MathiasMNilsen
and
claude
committed
c1f9619
Drop the schemes' max_iter attribute; keep maxiter

MathiasMNilsen
and
claude
committed
4e85819
max_iter counts update iterations, not the prior forecast

MathiasMNilsen
and
claude
committed
6a0d4ac
One configuration boundary: normalise once, validate by key

MathiasMNilsen
and
claude
committed
54cbb06
Document the configuration and the architecture; a public popt API

MathiasMNilsen
and
claude
committed
4285094
Merge upstream/main: reconcile the restructuring with upstream's 15 commits

MathiasMNilsen
and
claude
committed
d49a323
Write the PET log file as UTF-8

MathiasMNilsen
and
claude
committed
3ecbb0c
Record an HPC extraction failure against its own member, not the batch

MathiasMNilsen
and
claude
committed
5019717
Converge the outer EPF loop on the constraint violation, not the step size

MathiasMNilsen
and
claude
committed
0f336b3
Understand a LOCALIZATION block that names no mode

MathiasMNilsen
and
claude
committed
c894f3e
Build the distance localization operator over the active cells

MathiasMNilsen
and
claude
committed
9fc7010
Read a pickled localization file into localization entries

MathiasMNilsen
and
claude
committed
7306525
Add the subspace2 analysis: Gauss-Newton on the ensemble transform

MathiasMNilsen
and
claude
committed
d77d72b
Restore GenOpt and CMA on the current optimizer base

MathiasMNilsen
and
claude
committed
1ccd840
Let the optimizer reject a crashed evaluation instead of ending the run

MathiasMNilsen
and
claude
committed
f45657b