Repository navigation
feat: add an untested TM1 package in systems/tm1, for others to run - #54
Conversation
FPABench-TM1 builds the model on IBM Planning Analytics TM1 v11 as four cubes, with rules, feeders and TurboIntegrator processes, and runs the tests through TM1's REST API with its own PowerShell harness. It has not been run on a TM1 server, and it ships no results. Anyone with a licensed TM1 server can install and run it, then check the answers with scripts/compare.mjs.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4617175bd8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for (const run of Object.keys(runs)) { | ||
| const file = path.join(dir, `${test}_${run}.csv`); | ||
| let got = readTm1(file); | ||
| if (got === null) continue; |
There was a problem hiding this comment.
When TM1 fails to create an expected answer file, readTm1 returns null and this branch silently skips that run. Consequently compare.mjs exits successfully even if every requested answer is missing; in particular, the installer's Check stage can report a successful L1 comparison without validating any TM1 output. Treat a missing expected file as a comparison failure.
Useful? React with 👍 / 👎.
| if ($r.exit -ne 0) { throw "L2 exit $($r.exit)" } | ||
| $times += $r.ms | ||
| Run-Check 'L2' @{} (Join-Path $answers "L2_$i.csv.check") | ||
| Restore-State @($j) |
There was a problem hiding this comment.
If L2 execution or its post-run check throws, control skips this restore, the outer catch records the failure, and the harness continues with M7 actuals and close flags still applied. The same success-only restore pattern is used for L3/W2/M1–M4 below, so a failed model or load test can contaminate every subsequent measurement. Put restoration in a finally block as the write-test branch already does.
Useful? React with 👍 / 👎.
| const [, key, value] = splitCsv(line); | ||
| const obj = {}; | ||
| for (const part of (key || '').split(';').filter(Boolean)) { const i = part.indexOf('='); obj[part.slice(0, i)] = part.slice(i + 1); } | ||
| rows.push({ key: obj, value: Number(value) }); |
There was a problem hiding this comment.
If an answer row is truncated or contains an error token, Number(value) produces NaN; later Math.abs(a - b) > tol is false for NaN, so the malformed answer is reported as ok. Validate that parsed values are finite and count invalid rows as differences rather than allowing corrupt output to pass the correctness gate.
Useful? React with 👍 / 👎.
| do { $r = Invoke-TI 'z.FPB.Read' @{ pTest = 'R8'; pOut = (Join-Path $answers "R8_$i.csv") } -Quiet; if ($r.exit -ne 0) { Start-Sleep -Milliseconds 100 } } while ($r.exit -ne 0 -and $sw.Elapsed.TotalSeconds -lt 900) | ||
| $sw.Stop(); $times += $sw.Elapsed.TotalMilliseconds |
There was a problem hiding this comment.
When the restarted service never becomes usable within 900 seconds, this loop exits because of the elapsed-time condition but never checks that $r.exit became zero. The timeout is then appended as a valid R8 timing and the test remains ok, despite having no successful proving read. Throw after the loop when the final call failed.
Useful? React with 👍 / 👎.
… from FPABench README.md and the methodology doc lose their repeated notices, which NOTICE holds. A new Known gaps section lists where the harness times or checks differently from FPABench's other entries. Wrong statements are fixed: W8 and the .check files are not compared, -Quick times 3 runs of each single-cell write, and the rules and feeders do name the Actual version.
…Bench entry is The installer downloads the answers and checks RAMA recorded for the published dataset, and checks that they belong to the dataset, instead of recording its own reference with the DuckDB entry. The harness plan takes W8's thresholds from it. compare.mjs now checks W8 and the check files too, and fails a missing answer or check.
…very check The harness's L1 now destroys the four cubes and every FPB dimension and builds, sets up and loads them inside the timer, as FPABench's L1 loads master data from an empty system. L3 ends with SaveDataAll, as the other loads do. The check process reads every check expected.json has, each into its own file, and compare.mjs reads them all.
FPABench keeps one changelog for the repository.
The repository's licences cover it. The IBM notices and the TM1-specific rules for contributions move into the package README.
Motivation
Farseer will not measure TM1 itself. This package lets anyone with their own licensed IBM Planning Analytics TM1 v11 server build the FPABench model and run the tests.
Changes
systems/tm1/: FPABench-TM1, a self-contained package.FPB_Drivers,FPB_Reporting,FPB_FX,FPB_Control), plus the M1 and M4 variants..tisources.README.md,LICENSE(Apache 2.0),NOTICE,CONTRIBUTING.mdandCHANGELOG.md.Nothing outside
systems/tm1/changes. The runner and the website don't read this folder: it has nodb.mjs, noentry.jsonand no results.Testing completed
None on a TM1 server. The package says so at the top of its README: it has not been installed, run or validated on TM1 v11 or v12.
Before pushing I read every file. There are no credentials, server addresses or machine names. Its
.gitignorekeeps local config with credentials out of the repo.Notes for review
NOTICEandREADME.mdsay the package is not affiliated with, or endorsed by, Farseer. That wording was written for a separate repo, and it reads oddly inside Farseer's own repo..gitattributes.Run-FPBench.ps1keeps its BOM.🤖 Generated with Claude Code