So two nodes migrated, but other have the following error when we try to remove from registry

 dfx canister --network ic call rwlgt-iiaaa-aaaaa-aaaaa-cai remove_node_directly '(record { node_id = principal "ebih2-65fto-thxfj-knxfn-xxdmq-z42dj-oty4c-n3fbj-qry6f-g3oe2-nqe" })'

Error: The Replica returned an error: code 5, message: "Error from Canister rwlgt-iiaaa-aaaaa-aaaaa-cai: Canister called `ic0.trap` with message: Panicked at '[Registry] do_remove_node_directly: Cannot remove a node that is a member of a subnet. This node is a member of Subnet: subnet_record_pzp6e-ekpqk-3c5x7-2h6so-njoeq-mt45d-h3h6c-q3mxf-vpeq5-fk5o7-yae', rs/registry/canister/src/mutations/node_management/do_remove_node_directly.rs:49:13.
Consider gracefully handling failures from this canister or altering the canister to handle exceptions. See documentation: http://internetcomputer.org/docs/current/references/execution-errors#trapped-explicitly"

Figured it out
https://dashboard.internetcomputer.org/proposal/131977

Aparently I only added one node, will add the 3 after this accepted

./ic-admin \
    --nns-url https://ic0.app \
    -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-remove-nodes-from-subnet \
    --summary "Removing cm1 nodes for new IPv6 and IPv4 Redeployment" \
    --proposer 10477137729120770817 \
    ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae \
    zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe \
    zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae```

Code snippet

Figured it out
https://dashboard.internetcomputer.org/proposal/131977

Aparently I only added one node, will add the 3 after this accepted

./ic-admin \
    --nns-url https://ic0.app \
    -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-remove-nodes-from-subnet \
    --summary "Removing cm1 nodes for new IPv6 and IPv4 Redeployment" \
    --proposer 10477137729120770817 \
    ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae \
    zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe \
    zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

Figured it out
https://dashboard.internetcomputer.org/proposal/131977

Apparently I only added one node, will add the 3 after this accepted

./ic-admin \
    --nns-url https://ic0.app \
    -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-remove-nodes-from-subnet \
    --summary "Removing cm1 nodes for new IPv6 and IPv4 Redeployment" \
    --proposer 10477137729120770817 \
    ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae \
    zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe \
    zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

Figured it out
https://dashboard.internetcomputer.org/proposal/131977

Apparently I only added one node, will add the rest of the 3 nodes after this NNS proposal is accepted accepted

./ic-admin \
    --nns-url https://ic0.app \
    -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-remove-nodes-from-subnet \
    --summary "Removing cm1 nodes for new IPv6 and IPv4 Redeployment" \
    --proposer 10477137729120770817 \
    ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae \
    zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe \
    zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

Figured it out
https://dashboard.internetcomputer.org/proposal/131977

Apparently I only added one node, will add the rest of the 3 nodes after this NNS proposal is accepted

./ic-admin \
    --nns-url https://ic0.app \
    -s ~/.config/dfx/identity/node-provider-hotkey/identity.pem \
    propose-to-remove-nodes-from-subnet \
    --summary "Removing cm1 nodes for new IPv6 and IPv4 Redeployment" \
    --proposer 10477137729120770817 \
    ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae \
    zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe \
    zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

Hi Malith. Thank you for trying to take action yourself. Sadly, these proposals have some problems. I believe that Sven already reached out to you to explain the problems.
On a high level, the problem is that if we remove a node from a subnet, the subnet stays with one node less, which can be seen as a (not a huge one but still..) security concern since the threshold keys will be split across a smaller number of nodes in the subnet. Our team also hit a number of proposals rejected in the past for the same reason.
Therefore, it's recommended to use change-subnet-membership proposals.

I understand that there was little documentation on how to do this, and I've made an attempt now to improve the situation.

In particular, I:

• Updated ic-admin to remove the deprecated proposals to separately add or remove nodes from a subnet: https://github.com/dfinity/ic/pull/1103
• Added a notice that the DRE tool should be used instead, with a link to the DRE repo
• Added instructions in the DRE repo on how the subnet replacement proposals should be submitted
• Added docs with the explanation of how decentralization is calculated and why is it implemented the way it is
• Added instructions on how nodes in a subnet can be replaced
• Added instructions on how nodes from a particular DC can be removed from a subnet https://dfinity.github.io/dre/nns-proposals.html#removing-nodes-from-a-subnet

Someone should also update the wiki to point to the DRE documentation for this operation.

Please take a look at the new docs and let me know if the described process looks reasonable to you.

Hi Saša Tomić [DFINITY] and Sven Fischer any update on removing the nodes ?
The following nodes are now offline
ebih2-65fto-thxfj-knxfn-xxdmq-z42dj-oty4c-n3fbj-qry6f-g3oe2-nqe

ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae

zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe

zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

For DRE tool to recognize and remove them from subnet

Hi Saša Tomić [DFINITY] and Sven Fischer any update on removing the nodes from subnet ?
The following nodes are now offline
ebih2-65fto-thxfj-knxfn-xxdmq-z42dj-oty4c-n3fbj-qry6f-g3oe2-nqe

ylm4h-mc2vy-vmwvf-g2qhn-36rbw-5cunq-wpj2j-ttofo-xc27c-vfdh7-aae

zpjxp-c6l3m-x54lj-47zz5-dpy6u-lpdkt-hc7pp-gdfuu-i4i7i-jpp4w-lqe

zppi4-egu2x-ylkfm-fttld-d5v43-wq7zs-hnszi-6plxh-6ol3h-ibceq-xae

For DRE tool to recognize and remove them from subnet

In reply to @malithh:matrix.org
I ran the DRE Tool heal function, seems to be working so far

zppi4
ebih2
ylm4h
zpjxp

All 4 nodes get replaced and the best part is the two nodes we migrated and online

bv2x3
shtwt

It will be in the proposal as a replacement. Great job guys! on building the tool. I will wait for you to submit the proposals

@malithh:matrix.org we found a bug in the tool, that it may readd an unhealthy node back to the subnet. Please wait with submission, a fix is under way.

In reply to @sasa-tomic:matrix.org
Likely network connectivity. I don't see the node online.