Skip to content

Send pull replications to the location pulled from, with the destinat… - #2931

Draft
SylvainSenechal wants to merge 1 commit into
development/9.7from
improvement/BB-913/pull-from-source-location-v2
Draft

SylvainSenechal wants to merge 1 commit into
development/9.7from
improvement/BB-913/pull-from-source-location-v2

Conversation

@SylvainSenechal

Copy link
Copy Markdown
Contributor

Issue: BB-913

Draft to compare with #2876

@bert-e

bert-e commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue BB-913 contains:

  • 9.6.6

  • 9.7.0

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.7.0

Please check the Fix Version/s of BB-913, or the target
branch of this pull request.

*/
getCopyDestination() {
return this.isPullReplication() ?
this.getAttribute('pullDestination') : this.getAttribute('toLocation');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A pull replication action without pullDestination (e.g. still in Kafka from before this change, where toLocation was the destination) returns undefined here. LifecycleUpdateTransitionTask would then write dataStoreName/storage class as undefined into the object metadata. Fall back to toLocation:

Suggested change
this.getAttribute('pullDestination') : this.getAttribute('toLocation');
(this.getAttribute('pullDestination') ?? this.getAttribute('toLocation')) :
this.getAttribute('toLocation');

@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 32.09877% with 55 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.15%. Comparing base (64dd89f) to head (41461de).
⚠️ Report is 37 commits behind head on development/9.7.

Files with missing lines Patch % Lines
extensions/replication/tasks/CopyLocationTask.js 0.00% 34 Missing ⚠️
...xtensions/replication/ReplicationQueuePopulator.js 20.00% 8 Missing ⚠️
lib/models/ActionQueueEntry.js 77.27% 5 Missing ⚠️
extensions/replication/ReplicationAPI.js 57.14% 3 Missing ⚠️
...s/lifecycle/tasks/LifecycleUpdateTransitionTask.js 0.00% 2 Missing ⚠️
extensions/replication/ReplicationMetric.js 0.00% 2 Missing ⚠️
extensions/lifecycle/LifecycleMetrics.js 50.00% 1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (32.09%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
lib/constants.js 100.00% <100.00%> (ø)
extensions/lifecycle/LifecycleMetrics.js 97.43% <50.00%> (-0.23%) ⬇️
...s/lifecycle/tasks/LifecycleUpdateTransitionTask.js 94.51% <0.00%> (-0.61%) ⬇️
extensions/replication/ReplicationMetric.js 91.66% <0.00%> (-2.39%) ⬇️
extensions/replication/ReplicationAPI.js 90.12% <57.14%> (-1.60%) ⬇️
lib/models/ActionQueueEntry.js 97.90% <77.27%> (-1.12%) ⬇️
...xtensions/replication/ReplicationQueuePopulator.js 93.81% <20.00%> (-2.56%) ⬇️
extensions/replication/tasks/CopyLocationTask.js 74.10% <0.00%> (-2.37%) ⬇️
Components Coverage Δ
Bucket Notification 85.50% <ø> (ø)
Core Library 86.87% <79.16%> (-0.02%) ⬇️
Ingestion 83.38% <ø> (ø)
Lifecycle 86.21% <25.00%> (-0.04%) ⬇️
Oplog Populator 89.48% <ø> (ø)
Replication 68.25% <11.32%> (-0.40%) ⬇️
Bucket Scanner 89.37% <ø> (ø)
@@                 Coverage Diff                 @@
##           development/9.7    #2931      +/-   ##
===================================================
- Coverage            81.24%   81.15%   -0.10%     
===================================================
  Files                  209      209              
  Lines                44290    44342      +52     
===================================================
+ Hits                 35985    35986       +1     
- Misses                8305     8356      +51     
Flag Coverage Δ
bucket-scanner 89.37% <ø> (ø)
ft_test:queuepopulator 17.42% <32.09%> (+0.06%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@SylvainSenechal SylvainSenechal Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • can't rely on 'toLocation' anymore, need to go through the helper getCopyDestination to check if we are in a pullReplication context
  • New Kafka message field pullDestination, used in that new getCopyDestination helper because we still need to know the destination that isn't passed in toLocation anymore but in pullDestination now
  • we still need that new _getDestination helper everywhere to decide which location to write to
  • Potentially misleading message logs (at least one spotted, fixable but some more if else thing)

But compared with the other pr, we don't have the "siteInCharge" part

const siteInCharge = actionEntry.isPullReplication() ?
                actionEntry.getAttribute('fromLocation') : actionEntry.getAttribute('toLocation');

Ater looking again at both pr, I think the first one is less complex than this one, I would keep the first pr, unless we absolutely do not want to have the data processor invert the copy data

objMD.getTransitionTime();
LifecycleMetrics.onLifecycleStarted(log, getCopyLocationMetricsType(actionEntry),
actionEntry.getAttribute('toLocation'),
actionEntry.getCopyDestination(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not be needed?

actionEntry.getAttribute('metrics');
ReplicationMetrics.onReplicationProcessed(
origin, fromLocation, this.site, contentLength,
origin, fromLocation, this._getDestination(actionEntry).location, contentLength,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not be needed?

// fields, shared with cloudserver
const { location } = entry.getAttribute('results');
const newLocationName = entry.getAttribute('toLocation');
const newLocationName = entry.getCopyDestination();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of tweaking LifecycleUpdateTransitionTask, should be handled in CopyLocationTask::_publishCopyLocationStatus

Comment on lines +639 to +640
StorageType: storageType,
StorageClass: location,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I think this maybe the critical point

  • we pass the storageClass/Type in many places : so this makes the approach as risky as the other one...
  • ...but actually we don't need to change these places : these commands are only used in actual transition, and not in the pull-replication case

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • ...but actually we don't need to change these places : these commands are only used in actual transition, and not in the pull-replication case

sorry, I mixed things up. This is indeed the write on the (real) "destination" data location, and must be fixed indeed.

transitionTime: params.transitionTime,
});
if (params.pullDestination) {
action.setAttribute('pullDestination', params.pullDestination);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • this is used by LifecycleUpdateTransitionTask.js, not CopyLocationTask : so not needed
  • in copyLocationTask, the actual "target" storageClass is already available in the location pointer

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants