Repository navigation
Conversation
|
Please note that the test_sanity critic failure is the |
…ges from flag_activity
… drop the backfill script
…net change per request
…the user activity stats
…vity change in the bug modal
…questee is removed
…ield and the activity by product
| # bugs_activity, which has them all; flag_activity has only some of them, | ||
| # written to both tables, which must not be listed twice. Older rows also don't | ||
| # always know their who: it was filled in from the setter. | ||
| sub newer_than_bugs_activity_sql { |
There was a problem hiding this comment.
per-bug MAX(bug_when) cutoff hides flag_activity rows whenever a later flagtypes.name row lands in bugs_activity, e.g. an old pod still writing both tables during the rolling deploy makes earlier flag_activity-only changes on that bug vanish from history, reports and changedby search, could we use a fixed cutoff stored at migration time instead
There was a problem hiding this comment.
Yes sure, would a row in the params table (written once by checksetup, undeclared param then it won't appear in admin UI) works for you, or would you prefer a dedicated one-row table (most explicit but it's a schema change for a single value)?
There was a problem hiding this comment.
a dedicated one-row table please. an undeclared param doesn't survive well: migrate_params in Bugzilla/Config.pm drops params with no definition from the hash on every checksetup run and then pushes that hash to memcached, so Bugzilla->params loses the value even though the db row is still there (saving editparams does the same). a table like flag_activity_migration (cutoff DATETIME NOT NULL) filled once with NOW() next to the start rows, then read once per request via request_cache, keeps it out of the admin ui and is easy to drop later. with a fixed cutoff, bugs_activity flag rows need bug_when <= cutoff and flag_activity rows need flag_when > cutoff, and the per-row subquery and bugs_activity_bug_field_when_idx can probably go
There was a problem hiding this comment.
Thanks, and you're right about migrate_params, I missed that. I'll do it as you describe:
flag_activity_migration (cutoff DATETIME NOT NULL), filled once withLOCALTIMESTAMP(0)at the end of the start rows migration, which also uses it as its run-once guard. Read once per request throughrequest_cache.flag_activityrows after the cutoff, andbugs_activityflag rows up to it in the readers that list or count changes: history, bugmail, the user activity report and the per-product counts. The ones that take aMAXor anEXISTS(non-bot last change, last user activity, touched, the search conditions, ReleaseTracking) only need theflag_activityside, as a change listed twice doesn't change their result.- The per-row subquery and
bugs_activity_bug_field_when_idxgo away.
I'll push it with the NOT NULL step once #2808 is merged.
There was a problem hiding this comment.
Done as described:
flag_activity_migration (cutoff) is filled with LOCALTIMESTAMP(0) at the end of the start rows migration, which uses it as its run-once guard, and Bugzilla::FlagActivity->cutoff reads it once per request. History, bugmail, the user activity report and the per-product counts read the bugs_activity flag rows up to the cutoff and the flag_activity rows after it. The other readers only use the flag_activity side. bugs_activity_bug_field_when_idx is gone.
The tests set the cutoff with a new set_flag_activity_cutoff helper in Bugzilla::Test::Util. The history and user stats tests also have a change after the cutoff written to both tables, as a pod on the older code does, and check that it's listed and counted once.
Fixed in 3c0d478
# Conflicts: # t/bmo/bugmail-flag-events.t
| {TYPE => 'INT3', REFERENCES => {TABLE => 'profiles', COLUMN => 'userid'}}) | ||
| if !$who; | ||
| $dbh->do('UPDATE flag_activity SET who = setter_id WHERE who IS NULL'); | ||
| $dbh->bz_alter_column('flag_activity', 'who', |
There was a problem hiding this comment.
during the rolling deploy old pods still insert into flag_activity without who. strict mode is turned off in Bugzilla/DB/Mysql.pm so mysql fills in 0, then the new fk to profiles rejects it and flag changes on old pods fail until the rollout ends (if the fk isn't there yet, the row is saved with who = 0 and the inner join on profiles in history drops it). could we ship this in two steps: first add who as nullable and write it while still writing bugs_activity, then make it NOT NULL and stop writing bugs_activity in a follow-up. or at least keep the column nullable for now and backfill who = setter_id WHERE who IS NULL OR who = 0 on every checksetup run
There was a problem hiding this comment.
Right, thanks! I went with the two steps:
- Bug 2088705 - Record the acting user in flag_activity.who, as a nullable column #2808 (bug 2088705) adds
whoas a nullable column, with its foreign key and index, and writes it on every flag change, still writingbugs_activity. checksetup fillswho = setter_idon the rows without it, on every run. - Once it's deployed, this PR makes
whoNOT NULL (after checksetup has filled the rows left without it) and stops writingbugs_activity. During its rolling deploy, the old pods already writewho.
So #2808 needs to go out in an earlier release than this one, I'll update this PR once #2808 is merged.
There was a problem hiding this comment.
Done in 5c3d058
master with #2808 is merged in, and _set_flag_activity_who_not_null makes who NOT NULL right after _add_flag_activity_who has filled the rows left without it.
Checking it on MySQL showed one more thing: once #2808 has added the foreign key, MySQL refuses to make who NOT NULL ("Cannot change column 'who': used in a foreign key constraint"). The step now drops the key first, and bz_setup_foreign_keys adds it back at the end of checksetup, as the oauth2 migration does.
| # bugs_activity, which has them all; flag_activity has only some of them, | ||
| # written to both tables, which must not be listed twice. Older rows also don't | ||
| # always know their who: it was filled in from the setter. | ||
| sub newer_than_bugs_activity_sql { |
There was a problem hiding this comment.
a dedicated one-row table please. an undeclared param doesn't survive well: migrate_params in Bugzilla/Config.pm drops params with no definition from the hash on every checksetup run and then pushes that hash to memcached, so Bugzilla->params loses the value even though the db row is still there (saving editparams does the same). a table like flag_activity_migration (cutoff DATETIME NOT NULL) filled once with NOW() next to the start rows, then read once per request via request_cache, keeps it out of the admin ui and is easy to drop later. with a fixed cutoff, bugs_activity flag rows need bug_when <= cutoff and flag_activity rows need flag_when > cutoff, and the per-row subquery and bugs_activity_bug_field_when_idx can probably go
# Conflicts: # Bugzilla/DB/Schema.pm # Bugzilla/Install/DB.pm # t/flag-activity-who.t
…ctivity, the later ones from flag_activity
| $flag_bug_when AS bug_when, | ||
| CASE WHEN prev.id IS NULL THEN '' | ||
| WHEN prev.status = '?' AND preq.login_name IS NOT NULL | ||
| THEN CONCAT(ft.name, '?(', preq.login_name, ')') |
There was a problem hiding this comment.
nit: raw CONCAT(...) here skips the DB layer, could use $dbh->sql_string_concat like the rest of the codebase. same in BugMail.pm and Reports/UserActivity.pm
There was a problem hiding this comment.
Done => now the four CONCAT(...) come from $dbh->sql_string_concat, built next to the other $flag_* SQL fragments.
Fixed in cc875d3
| fsa.flag_when AS bug_when, | ||
| CASE WHEN prev.id IS NULL THEN '' | ||
| WHEN prev.status = '?' AND preq.login_name IS NOT NULL | ||
| THEN CONCAT(ft.name, '?(', preq.login_name, ')') |
There was a problem hiding this comment.
nit: same raw CONCAT as in Bug.pm, $dbh->sql_string_concat would keep it portable
There was a problem hiding this comment.
Done the same way in _get_diffs
Fixed in 715c65f
| . " AS ts, | ||
| CASE WHEN prev.id IS NULL THEN '' | ||
| WHEN prev.status = CHAR(63) AND preq.login_name IS NOT NULL | ||
| THEN CONCAT(ft.name, prev.status, '(', preq.login_name, ')') |
There was a problem hiding this comment.
nit: same raw CONCAT, could use $dbh->sql_string_concat
There was a problem hiding this comment.
Done here too (the CHAR(63) for the ? stays, the status column still provides it).
Fixed in dc04011
| my $dbh = Bugzilla->dbh; | ||
|
|
||
| my @args = ($bug->id); | ||
| my @args = ($bug->id); |
There was a problem hiding this comment.
nit: alignment is off here, looks like perltidy wasn't run. perl scripts/perl-fmt.pl on the changed files should fix this plus the long $flag_suppjoins line in Bug.pm and the hash indent in Install/DB.pm
There was a problem hiding this comment.
Ah yes, done with the repo's .perltidyrc...
ran perl scripts/perl-fmt.pl on the files this PR modifies. Most weren't perltidy-clean on master, so this also reformats existing code in them.
Fixed in c1c4690
| # SQL matching a flag_activity row against a history value like | ||
| # "review+" or "review?(user@example.com)". Returns nothing if the value | ||
| # isn't a flag state. | ||
| # SQL condition on flag_activity row $fa for changedto or changedfrom $value |
There was a problem hiding this comment.
nit: two doc comments stacked above _flag_state_changed_sql, one can go
There was a problem hiding this comment.
My fault: in fact the first one is the _flag_state_match comment, I think it got separated when I added _flag_state_changed_sql...:/
=> Moved it back above that sub rather than deleting it
Fixed in af5884e
|
Besides the 5 previous points, found 4 small things, fixed in separate commits:
I also merged master: |
| my $verified = $params->{product_change_confirmed}; | ||
|
|
||
| # BMO - if everything is ok then we can skip the verfication page when using bug_modal | ||
| # BMO - if everything is ok then we can skip the verfication page when using bug_modal |
There was a problem hiding this comment.
please drop the unrelated perltidy reformatting, it hides the real change and pushes some comments out to column 0 (like this one and the one in _migrate_profiles_modification_ts in Install/DB.pm)
There was a problem hiding this comment.
Fixed in ca55922: Changes to code this PR does not touch are now reverted, including comments.
| # can look up the exact timestamp without relying on flag-type name matching. | ||
| my $dbh = Bugzilla->dbh; | ||
| my ($flag_when, $who) = $dbh->selectrow_array( | ||
| "SELECT flag_when, who FROM flag_activity WHERE flag_id = ? |
There was a problem hiding this comment.
this runs one flag_activity query per flag on every bug page, could you load the last rows for all the bug's flags in one query
There was a problem hiding this comment.
Done in 7f1ecfc: The first call now loads the last flag_activity row of every flag of the bug in one query (in flag_activity_bug_id_idx) and keeps them for the other flags, with a new case in t/flag-activity-stream.t for two flags on one bug.
…in one query in the activity stream
Summary
Stops writing flag changes to
bugs_activity: new ones are only recorded inflag_activity, and everything that reads flag changes now reads both tables,bugs_activityfor the changes up to a cutoff stored at checksetup andflag_activityfor the later ones.Part 2 of 1806896, whose part 1 (#2657) moved
flag_state_activityto core asflag_activityand writes every flag change to both tables.Changes
Bugzilla::BugandBugzilla::Attachmentno longer logflagtypes.nametobugs_activity; the user merge ineditusers.cgiwrites its flag changes toflag_activity.flag_activity.who, the user who made the change, becomes NOT NULL. #2808 (bug 2088705) added it as a nullable column and writes it on every change, so it must be deployed in an earlier release than this PR. On MySQL its foreign key is dropped for the change and added back at the end of checksetup.flag_activityrow (flags last changed beforeflag_activitylogged every type) a row with its current state, dated at its last change, in batches of 10,000 flags. History and bugmail don't show it, but the flag's next change gets its previous state. It then stores the cutoff, the time it ran, in the new one-row tableflag_activity_migration, which also keeps it from running again.GetBugActivity), bugmail (_get_diffs), the BMO user activity report and the UserProfile per-product counts read theflagtypes.namerows inbugs_activityup to the cutoff and theflag_activityrows after it (Bugzilla::FlagActivity->cutoff), so a change written to both tables, by a pod still on the older code during the deploy, is listed once. A request that changes a flag several times shows its net change, compared with the flag's last row before the request.changedbefore/changedafter/changedfrom/changedto/changedby/everchangedon flags, and the ReleaseTracking report, match either table.flag_idinflag_activity.flag_activity: the non-bot last change time, the UserProfile stats, the TagNewUsers first activity, owner idle time, andchanged*searches on any field.flag_activityrows can only be deactivated, not deleted, as its history would be deleted with it. The user deletion check ineditusers.cgicountsflag_activityrows, andscripts/remove-non-public-data.plkeepswho.t/flag-activity-history.t,-search.t,-who.t,-user-stats.t,-stream.t,-start-rows.tandt/flag-type-delete.t. They set the cutoff with the newset_flag_activity_cutoffinBugzilla::Test::Util.The old
flagtypes.namerows stay inbugs_activity; nothing is deleted.Deploy
flag_activity_migration, which checksetup creates: until then, bug history, bugmail and flag searches fail with a missing-table error. Until checksetup stores the cutoff at the end, they also miss the older flag changes inbugs_activity. The flag backfill can take a while on production.whoand log to both tables, and the new code lists those changes once.The order is based on what I checked in the code:
Test plan
bugs_activitygets no newflagtypes.namerowneedinfo?" and the user activity report find both older and new changeseditusers.cgi: the reassigned requests appear in bug historyReferences