Conversation
--seed=N, and a seed set at the propertyAn unseeded run cannot be replayed: `IO.stdGenRef` is seeded from the OS at startup, so the input that broke a property died with the run that drew it. `--seed=N` fixes the whole run. Every property draws from `N` mixed with its own name, so properties still differ from each other and `--only=` replays exactly what the full run gave that property; `N` also seeds the process-wide RNG, which is what the self-driving `IO` properties, the diagnostics and the Tyche panels draw from. A property that fails reports the seed to keep. `@[strata_property (seed := N)]` pins one property instead, whatever the run was given, which is what turns a defect on a rare draw into one that fails every run — and so into one `knownFailure` can watch. The pin beats the flag: a pin is what makes that failure reliable, so letting the command line displace it would turn the property flaky again. One attribute with one argument rather than `@[strata_property, seed = 42]`: in Lean that is a list of two independent attributes, so `seed` would be a global name, meaningless alone and a silent no-op where `strata_property` is absent.
`--seed=N` gave each property `N` mixed with its own name, so that properties still drew different inputs from each other. That is a second mechanism to understand and to keep stable, and it is not what a reader expects `--seed=N` to mean: the number in the report was the mixed seed, not the one they passed. Every property now runs at `N` itself, as in `hspec` (whose runner says "the same seed is used for all properties") and `tasty-quickcheck`. The report prints the seed the reader typed, and both routes it suggests now hold: `--seed=N` replays the run, and a pin of the same number keeps that one draw. The cost is real and now documented rather than engineered around: two properties over one input type draw the same inputs as each other, so a seeded run covers less than an unseeded one. That is the trade the flag is for. Reach for it to reproduce a failure, not to gate a merge.
…l English ASD-STE100, and the comment rules of CR-296832157 (`strata-comment-quality`): short active sentences, 1 idea in 1 sentence, and no buzzwords. The rules removed the word "load-bearing" from `effectiveSeed`, cut the rationale that 4 files each repeated, and dropped the figures of speech. The behaviour does not change. 1 message changes its punctuation: a failure now reports `seed: 7. Replay with ...` in place of an em dash, which STE does not permit. The 2 doc headings in `docs/writing-properties.md` lose their gerunds, so their anchors change with them: "Reproducing a draw" is now "Get the same inputs again", and "Pinning a seed" is now "A seed for 1 property".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The suite gets its inputs from the random number generator of the operating system.
If a property fails, you cannot get the same inputs a second time. You cannot look at
the failure again. A defect that only 1 input in 300 makes visible has no reliable test.
What this change adds
There are 2 new controls: a flag for the full suite, and an attribute argument for one
property.
--seed=NGive this flag to
testor totest-plain:Each property then makes its inputs from the seed
Nand from its own name. If you givethe same command 2 times, the suite makes the same inputs. A property that fails shows its
seed in the report:
The seed also goes to the random number generator of the process. Because of this, the
self-driving
IOproperties, the diagnostics and the Tyche panels also give the sameoutput again.
Each property gets
Nitself.hspecandtasty-quickcheckdo the same. Because ofthis, 2 properties of 1 input type get the same inputs, and a suite with a seed covers
less than a suite without one. Use the flag to get a failure again, not to gate a merge.
A bad value stops the suite with an error.
--seed=abcdoes not make a suite without a seedthat the header shows as a suite with one.
A seed in the declaration
The property then makes the same inputs each time, and it ignores
--seed=.@[strata_properties (seed := N)]does the same for each member of a list.withSeed Nis the term form.
This is the reason for the feature. A defect on a rare input now fails each time, so
knownFailurecan watch it. Before this change, such a property could not have a mark.The seed in the declaration has a higher precedence than the flag. The seed makes the
failure reliable. If the flag replaced the seed, the property becomes unreliable again.
Its
knownFailuremark then fails the suite on each input that does not show the defect.The cost is clear: a property with a seed in its declaration stops the search for new
defects, because it makes the same inputs each time. Give a seed to 1 property, and remove
the seed with the
knownFailuremark that it helps.Why 1 attribute with an argument
Lean reads
@[strata_property, seed = 42]as 2 independent attributes. Aseedattributeis then a global name that other packages cannot use. It also does nothing if
strata_propertyis not there, which is a silent fault. This change puts the argument inthe attribute, as Lean core does for
@[deprecated (since := "…")].Checks
--quick --seed=7on all 147 properties give the same output, byte forbyte. The Tyche JSONL file is also the same, except for its clock field.
--seed=flag. The seed came from the report of an earlier
--seed=7test.distribution.
Notes for the reviewer
main. The second commit removes a seed mixer fromthe first one. The suite gave each property a mix of
Nand the name of the property.1 seed for each property is enough for basic use, and the report then shows the number
that the reader gave.
docs/writing-properties.mdhas the new section Reproducing a draw, with theprecedence rules and the costs.