Add atomic code upgrade test coverage

[Copilot]

Closes #3390.

CCF supports replacing an old node set with a non-overlapping new node set via one governance reconfiguration proposal. This adds coverage and docs for that atomic code upgrade path.

• Atomic reconfiguration helper
  • Added Consortium.replace_nodes() to trust multiple pending nodes and retire multiple existing nodes in one proposal.
  • Kept replace_node() as the single-node wrapper.

network.consortium.replace_nodes(
    primary,
    old_nodes,
    new_nodes,
    valid_from=valid_from,
)

• Code upgrade test

  • Added test_update_all_nodes_atomically().
  • Starts a full replacement node set with the new package.
  • Submits one proposal containing all transition_node_to_trusted and remove_node actions.
  • Waits for primary election among the new nodes, then removes retired old nodes.
  • Preserves sequential upgrade coverage by upgrading back afterward.

• Operations documentation

  • Documented the atomic alternative in the code upgrade procedure.
  • Clarified that new nodes should remain pending until the atomic proposal is submitted.

[achamayou]

@copilot check code format and drop the reference to #1713, that's been addressed long ago.

[Copilot]

@copilot check code format and drop the reference to #1713, that's been addressed long ago.

Addressed in a870d10. I re-ran the targeted Python format check and removed the stale #1713 references from the updated code path.

[achamayou]

@copilot look at the last complete Virtual C test and fix the failure. Don't forget to format the code.

[achamayou]

@copilot looks good but please generalise remove_retired_node() to remove_retired_nodes() taking a collection of nodes, and update the test accordingly, I think that will be shorter and faster to run.

[Copilot]

@copilot looks good but please generalise remove_retired_node() to remove_retired_nodes() taking a collection of nodes, and update the test accordingly, I think that will be shorter and faster...

Addressed in db836ec. remove_retired_node() is now remove_retired_nodes() and the atomic upgrade test removes the old node set with one batched helper call.

@copilot look at the last complete Virtual C test and fix the failure. Don't forget to format the code.

Checked the VMSS Virtual C history. The last completed CI run for this branch passed VMSS Virtual C at 585bfac; the earlier failure was the non-contiguous ledger chunk check before the subsequent main merge. I also re-ran the Python format check successfully.

@copilot look at the last complete Virtual C test and fix the failure. Don't forget to format the code.

Checked the VMSS Virtual C history. The last completed CI run for this branch passed VMSS Virtual C at 585bfac; the earlier failure was the non-contiguous ledger chunk check before the subsequent main merge. I also re-ran the Python format check successfully.

@copilot looks good but please generalise remove_retired_node() to remove_retired_nodes() taking a collection of nodes, and update the test accordingly, I think that will be shorter and faster...

Addressed in db836ec. remove_retired_node() is now remove_retired_nodes() and the atomic upgrade test removes the old node set with one batched helper call.

[Copilot]

Pull request overview

This PR adds end-to-end coverage (and corresponding ops documentation) for CCF’s “atomic code upgrade” path, where an entire old node set is retired and a non-overlapping new node set is trusted via a single governance proposal.

Changes:

• Added Consortium.replace_nodes() to trust multiple pending nodes and remove multiple existing nodes in one proposal, keeping replace_node() as a single-node wrapper.
• Extended tests/code_update.py with an atomic full-network upgrade test (test_update_all_nodes_atomically()), exercised before upgrading back to preserve sequential upgrade coverage.
• Updated code upgrade operations docs to describe the atomic alternative and the requirement to keep new nodes pending until the atomic proposal is submitted.

Custom instructions used:

• .github/copilot-instructions.md
• .github/instructions/reviewing.instructions.md

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tests/infra/network.py	Restricts chunk-flag verification to each node’s writable ledger directory, avoiding non-contiguous read-only bootstrap ledgers.
tests/infra/consortium.py	Adds multi-node atomic replace helper (replace_nodes) and refactors replace_node to wrap it.
tests/code_update.py	Adds atomic full-network code upgrade test path plus helper to remove retired nodes from the service state.
doc/operations/code_upgrade.rst	Documents atomic reconfiguration option during code upgrade procedure.

[maxtropets]

@copilot I'm curious why do we do that, I thought there's some default period good enough, elaborate on the necessity of this line here

[Copilot]

set_certificate_validity_period() here is test-side bookkeeping, not a governance change.

In the non-atomic path, network.trust_node() already sets this local expectation on each Node object. In the atomic path we bypass network.trust_node() and call replace_nodes() directly, so we set it manually to keep node metadata consistent for later cert-validity checks.

You’re right that the service default validity period is sufficient on-chain; this line is only to mirror that value in the test harness state.