| 2 Jul 2024 |
 Saša Tomić [DFINITY] | We're making some healthy progress this time. Already updated 190 nodes, with very few failures.
INFO dre::operations::hostos_rollout > 190 nodes are already on the new version, out of those 6 unhealthy
WARN dre::operations::hostos_rollout >
***** WARNING: Unhealthy nodes found with the new version: *****
32yxi-hqan7-bhpqw-ypgjn-zq64m-6uqxs-htba2-sldaq-dkm7c-af3o5-vae: Dead DC hk4 (-) Node Provider Web3game
weww5-eanvf-5ryld-mz57k-7nukv-o5amt-ny5dl-6c22g-7iyvp-gnkf5-rqe: Dead DC zh4 (zh4-dll05) Node Provider Tomahawk.vc
fneyt-nqoj2-ehhfp-vat74-ookwk-othuy-dj2nz-giiio-tuzqi-dkbsw-mqe: Dead DC an1 (an1-dll16) Node Provider Allusion
rbr7v-biuga-7kvyt-bmptt-qzqaf-mizsl-5be3p-tfszx-akpur-j62co-cqe: Dead DC an1 (an1-dll04) Node Provider Allusion
lf2qh-kizev-gkppo-tsz4a-tvev3-y7be3-gnyjz-5tmzr-ds56j-ylpwe-3ae: Dead DC hk4 (-) Node Provider Web3game
vreb7-z3w7h-bmkze-eyuio-e265o-qxyfr-3uzik-htfw6-otpjr-2rrct-aqe: Dead DC mr1 (mr1-dll27) Node Provider DFINITY Operations SA
| 15:36:29 |
 Bhavik Vadhia | The sever is still in degraded mode with IC_Orchestrator_VersionBehind status. Any update? | 15:41:39 |
| 3 Jul 2024 |
| Saša Tomić [DFINITY] | We have some node failing to upgrade HostOS. Louise from Aviate Labs tried rebooting some of his nodes and most recovered. So seems like nodes get stuck during the reboot to the new OS.
INFO dre::operations::hostos_rollout > 266 nodes are already on the new version, out of those 7 unhealthy
WARN dre::operations::hostos_rollout >
***** WARNING: Unhealthy nodes found with the new version: *****
32yxi-hqan7-bhpqw-ypgjn-zq64m-6uqxs-htba2-sldaq-dkm7c-af3o5-vae: Dead DC hk4 (-) Node Provider Web3game
weww5-eanvf-5ryld-mz57k-7nukv-o5amt-ny5dl-6c22g-7iyvp-gnkf5-rqe: Dead DC zh4 (zh4-dll05) Node Provider Tomahawk.vc
kwgrq-dixu3-iztnh-3nucy-zfo73-lstek-xtehy-4lywo-ojb3n-7wbdl-dqe: Dead DC bt1 (-) Node Provider MB Patrankos šūvis
lf2qh-kizev-gkppo-tsz4a-tvev3-y7be3-gnyjz-5tmzr-ds56j-ylpwe-3ae: Dead DC hk4 (-) Node Provider Web3game
scjri-zcrz3-27aoo-a2ggm-alu4o-fcu4t-r5jy4-7hdeh-prukj-4cols-qae: Dead DC mb1 (mb1-dll12) Node Provider Fractal Labs AG
kcv4a-pltml-ux6to-476gm-bihz5-vb5rc-zi7d5-moydx-irkw4-qt3ru-qae: Dead DC an1 (an1-dll08) Node Provider Allusion
game888 web3 Rokas Ambrazaitis Gorazd Ocvirk : Could you please also try to reboot these stuck nodes listed above and see if they would recover? If not, can you please share if you see anything standing out during the boot process?
| 09:21:29 |
 77676m | Saša Tomić [DFINITY]: this update also applies to uzr34 subnet ? | 09:54:01 |
 Malith Hatanachchige | Hi Saša Tomić [DFINITY]
One of our nodes were too on the subnet that failed longtime back
https://dashboard.internetcomputer.org/node/onlqg-nqo75-pcmvj-ctydh-hy5c6-gb2sz-7uvko-c7pnw-5g22a-fej4n-qae
Should we do something abit this ? We are available to assist
| 10:51:33 |
| Malith Hatanachchige | * Hi Saša Tomić [DFINITY]
One of our nodes were too on the subnet that failed longtime back
https://dashboard.internetcomputer.org/node/onlqg-nqo75-pcmvj-ctydh-hy5c6-gb2sz-7uvko-c7pnw-5g22a-fej4n-qae
Should we do something about this? We are available to assist
| 10:53:12 |
| Saša Tomić [DFINITY] | 77676m: no, the node team is still working on the recovery image. It's almost ready. Apologies for the delay, I keep pushing them. | 10:53:17 |
| Saša Tomić [DFINITY] | Malith Hatanachchige: that onlqg node was also in the uzr34 subnet. I'll ping eerokelly and andrewbattat [DFINITY] , in case they have an update to share. | 10:59:02 |
| Saša Tomić [DFINITY] | * Malith Hatanachchige: that onlqg node was also in the uzr34 subnet, so the error is not related to the HostOS upgrade. I'll ping eerokelly and andrewbattat [DFINITY] , in case they have an update to share. | 10:59:37 |
| Malith Hatanachchige | Great, ping anytime, if you guys need assist on rebooting and getting logs from the screen. | 11:00:12 |
 Oleksandr | 
Download 23.46.44.png | 21:50:13 |
| Oleksandr |
After the update.
https://dashboard.internetcomputer.org/proposal/130960
my noda is not working, I rebooted it and now I get logs like this | 21:50:26 |
| 4 Jul 2024 |
| Saša Tomić [DFINITY] | In reply to @alexnod:matrix.org
After the update.
https://dashboard.internetcomputer.org/proposal/130960
my noda is not working, I rebooted it and now I get logs like this Thanks for sharing, @Oleksandr . We have already seen similar behavior with some other nodes, but I'll ping @andrewbattat [DFINITY] @eerokelly @raymondk For awareness.
In the meantime, this probability means that you will have to redeploy the node. Sorry.
| 08:09:20 |
| Oleksandr | In reply to @sasa-tomic:matrix.org
Thanks for sharing, @Oleksandr . We have already seen similar behavior with some other nodes, but I'll ping @andrewbattat [DFINITY] @eerokelly @raymondk For awareness.
In the meantime, this probability means that you will have to redeploy the node. Sorry.
How do I reinstall if my node is logged into Subnet, does it automatically create a new Node Operator ID or do I need to log out of Subnet before redeploy? | 11:43:11 |
| Saša Tomić [DFINITY] | In reply to @alexnod:matrix.org
How do I reinstall if my node is logged into Subnet, does it automatically create a new Node Operator ID or do I need to log out of Subnet before redeploy? We will tomorrow submit all proposals to free up unhealthy nodes. Should be executed on Monday. | 14:33:00 |
| Saša Tomić [DFINITY] | The HostOS upgrade is finished! This means that the public node metrics feature should be live!
Sadly, some nodes failed during the upgrade.
Louise from Aviate Labs tried rebooting some of his nodes and most of his nodes recovered and became healthy again.
It would be appreciated if you could try to upgrade your nodes if you see this message in time.
Tomorrow we'll submit proposals to free up unhealthy nodes for redeployment, by removing them from subnets.
INFO dre::operations::hostos_rollout > 1368 nodes are already on the new version, out of those 42 unhealthy
WARN dre::operations::hostos_rollout >
***** WARNING: Unhealthy nodes found with the new version: *****
weww5-eanvf-5ryld-mz57k-7nukv-o5amt-ny5dl-6c22g-7iyvp-gnkf5-rqe: Dead DC zh4 (zh4-dll05) Node Provider Tomahawk.vc
ct3c3-aaohu-xben4-mbsjf-evtcb-bvf4i-qax6r-i4lm2-tcn6j-ybqpz-wae: Dead DC dr1 (-) Node Provider Privoxy Solutions, LLC
2a7fq-jivc5-upcob-l5e6k-v3l6a-xo4fc-5rfga-7t5ng-cnv72-t6z7d-wqe: Degraded DC hk1 (-) Node Provider Wancloud limited
kwgrq-dixu3-iztnh-3nucy-zfo73-lstek-xtehy-4lywo-ojb3n-7wbdl-dqe: Dead DC bt1 (-) Node Provider MB Patrankos šūvis
6syks-la2v3-ua4aj-iyt6f-oivnq-6yosi-cqxwo-ahzno-ieuvx-bl7pf-vqe: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
ofemt-rbrvy-o6hne-hqs4p-xbhmf-go5aj-3gbs4-gj3it-akve3-2kklg-jae: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
plm5s-lz6v5-kifr5-zdcfy-5dacp-nwc2i-72rsw-nj5cg-d75um-s3jy4-kae: Dead DC zh7 (zh7-dll04) Node Provider Sygnum Bank
hlc73-cklhu-zo4mt-waxb6-ew5tx-gm4or-coy2j-xnwmy-xsycn-j6f4e-iae: Dead DC hk3 (-) Node Provider Power Meta Corporation
scjri-zcrz3-27aoo-a2ggm-alu4o-fcu4t-r5jy4-7hdeh-prukj-4cols-qae: Dead DC mb1 (mb1-dll12) Node Provider Fractal Labs AG
za7cm-bcsh6-dkayq-wbbbi-5pw7l-5dsqk-7isoh-osqdi-rmcot-pswxd-aae: Dead DC hk3 (-) Node Provider Power Meta Corporation
2cov2-kcta2-rsiae-jvmot-jhp4z-kmq7n-75e35-4dvf7-tobjm-ml4lt-bqe: Dead DC an1 (an1-dll05) Node Provider Allusion
lmfy6-g2wta-3pp67-cjv5u-qws2j-kf3ci-tkze3-iqucl-mscsw-3u7bx-jae: Dead DC rg1 (-) Node Provider Maksym Ishchenko
nbkmo-z2yx3-l4tjn-vvx4c-qj5ck-maicu-o7o44-ztlm5-5nm6d-it36c-hae: Dead DC ph1 (ph1-dll14) Node Provider MI Servers
3ecdy-dk5hn-gmh2r-4wral-uqy4v-4kly2-k6tb2-zqxyl-7ljhv-3xhes-cqe: Dead DC br1 (br1-dll16) Node Provider Allusion
mmx5c-ac7r2-kz47l-w6aur-w7wab-hslkz-zs6g5-g64tr-bve3c-2uvon-7ae: Dead DC an1 (an1-dll01) Node Provider Allusion
m3ju2-x3nqm-fcrha-zbyll-tl7no-7iwen-ph3y4-5xxwa-cniii-yu2ti-zae: Dead DC dr1 (-) Node Provider Privoxy Solutions, LLC
kcv4a-pltml-ux6to-476gm-bihz5-vb5rc-zi7d5-moydx-irkw4-qt3ru-qae: Dead DC an1 (an1-dll08) Node Provider Allusion
vreb7-z3w7h-bmkze-eyuio-e265o-qxyfr-3uzik-htfw6-otpjr-2rrct-aqe: Dead DC mr1 (mr1-dll27) Node Provider DFINITY Operations SA
b74lr-4ly45-pyqyx-jo4yd-eptv5-7fvps-zjeca-eugl3-ih77p-yjyu5-7ae: Dead DC ge1 (ge1-dll01) Node Provider Archery Blockchain SCSp
akbo3-imbii-tzc4p-swdwz-hh2tg-i2ffj-tnzjy-zycuo-3xjcq-zxbth-aqe: Dead DC hu1 (hu1-dll13) Node Provider 43rd Big Idea Films
67t6p-i4h3c-msv6p-kmbmm-rr6gj-z3nix-d6lo2-mq3q4-3h6rb-lwkbc-lae: Dead DC nd1 (-) Node Provider Marvelous Web3
235hh-hmjhq-dejel-3q5oi-pdz66-dygbp-yi2sy-zmuiq-rj7r7-65hue-wae: Dead DC dr1 (-) Node Provider Privoxy Solutions, LLC
f4xs2-aejji-yt4ks-7sjyb-g7zxl-ayuc3-lw6tv-khrii-nlh7j-ttbqy-zae: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
yzcei-zmxrz-tjso4-rgnbn-rr65u-gsgzz-hrsiu-iamm6-i6rtn-shqk4-3ae: Dead DC an1 (an1-dll02) Node Provider Allusion
nlxmx-rm4y4-vai2n-g2vb7-effvv-rh75h-tz7di-dsw7d-frcya-7jlig-fqe: Dead DC sg2 (sg2-dll03) Node Provider OneSixtyTwo Digital Capital
wdg4t-afemy-nttou-sdh2u-b6lik-2cuxv-frfoy-wjige-z4qy5-wheg6-kae: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
iwjry-3vgec-6gxlo-ujqqm-dvquc-dryh3-qzwyr-tzqbv-7blol-wdpg6-uae: Dead DC ch3 (ch3-dll06) Node Provider MI Servers
2ftux-ivgjc-t4i2d-lq5hs-myjdg-3h2pz-l2ddl-nsxfc-wlaq7-fkbc5-6ae: Dead DC sl1 (-) Node Provider Neptune Partners
j7mu5-sfjx5-rflso-aqhd5-ymijf-glw4a-ovi6i-f3blz-o54sg-a2rfk-bae: Dead DC li1 (-) Node Provider Ivanov Oleksandr
zgtrt-4vlgr-pbytl-t2yqq-qf4nk-wyoos-vrfpu-hxqcw-tcnfu-73kjb-pae: Dead DC bn1 (-) Node Provider Maksym Ishchenko
mjte3-dopva-7tttu-5hnnu-q572b-p3wwy-b56of-4q6h7-hbtf4-f3m32-tae: Dead DC pa1 (-) Node Provider Ivanov Oleksandr
kpvfz-zg5cs-5ofal-k5zng-uop7b-gjm4v-z4ug3-sggef-4zu7x-ihvcw-yqe: Dead DC bu1 (bu1-dll07) Node Provider Iancu Aurel
vte5d-zw5lg-axpbi-yrc6s-nm3mj-knu44-piwp3-ukrtj-xb4s5-mleiq-lae: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
mplx3-z67a3-qjn42-zw7ll-ui2i2-ed4n4-zwlfp-e36a7-57uzi-xoxxt-cae: Dead DC an1 (an1-dll15) Node Provider Allusion
wazkf-2xehg-y26ek-2uq64-pkrbh-6frmq-4q7lf-5gaya-44q4p-v5r5z-cqe: Dead DC jb2 (-) Node Provider Honeycomb Capital (Pty) Ltd
p5h3q-2pght-vje5y-2jmsf-cmdfi-52txk-qkrux-mnvmj-zzira-nugjd-dae: Dead DC ge1 (ge1-dll20) Node Provider Archery Blockchain SCSp
jemyk-uhint-w6ftv-s3xpy-ne4hr-xbcpu-aqskm-6a4cy-orm5q-aqqwd-gqe: Degraded DC hk1 (-) Node Provider Wancloud limited
uaxbw-jxkhf-lscbn-lie7h-qiwsz-jrjfp-bt4rp-vl7ds-yuayj-4wp4f-uae: Dead DC mr1 (mr1-dll24) Node Provider DFINITY Operations SA
q4ksz-lpvfy-ikkem-wyuaa-abopq-nmu5k-ovh6h-xfoqk-hx3x3-g2bk6-4qe: Dead DC jb3 (-) Node Provider Wolkboer (Pty) Ltd
xy4br-2xvw3-f4vuz-duieu-qloxb-7f5v6-mbzkc-puiej-3wqio-qyy24-5qe: Dead DC rg1 (-) Node Provider Ivanov Oleksandr
6kfcs-57wu7-37qb6-mkh2a-sl2gt-z7cwa-jwihc-o5ui4-zu3lw-nf5xq-nae: Dead DC kr2 (-) Node Provider Web3game
pojjf-vh65n-rephu-6rm4t-3zdx3-5pnon-csrf2-wlm32-z3tak-qyefb-fae: Dead DC gn1 (-) Node Provider ACCUSET SOLUTIONS
| 16:12:08 |
 Rokas Ambrazaitis | I have one dead and one degraded as MB Patrankos šūvis provider, should i try to restart them ? | 16:24:40 |
 Louise Velayo | Aviate Labs | Yeah this fixed a couple of nodes for me yesterday | 16:25:57 |
| Louise Velayo | Aviate Labs | Otherwise I would try to redeploy if the reboot doesn’t solve the issue | 16:26:25 |
| Rokas Ambrazaitis | Since it belongs to subnet i think I cant reinstall yet | 16:27:51 |
| Louise Velayo | Aviate Labs | Oh yes you’re right | 16:43:30 |
 marvelousweb3 | Hello Louise Velayo | Aviate Labs
Recently, we performed a redeployment of one of our nodes to the latest HostOS version due to observed degradation. Following this upgrade, we noticed that the node ID has been changed unexpectedly.
Could you please confirm if this change is a normal occurrence after redeplyment ?
| 18:29:40 |
| Louise Velayo | Aviate Labs | In reply to @marvelousweb3:matrix.org
Hello Louise Velayo | Aviate Labs
Recently, we performed a redeployment of one of our nodes to the latest HostOS version due to observed degradation. Following this upgrade, we noticed that the node ID has been changed unexpectedly.
Could you please confirm if this change is a normal occurrence after redeplyment ?
Yes that is normal behaviors. Every time you redeploy a node, it will have a new node id! But if you reboot a node, node id stays the same | 20:21:40 |
| marvelousweb3 | In reply to @louisevelayo:matrix.org
Yes that is normal behaviors. Every time you redeploy a node, it will have a new node id! But if you reboot a node, node id stays the same Thank you for your recent insights. Additionally, I would like to inquire whether we need to resubmit our reward proposal following the change in node ID? | 20:33:48 |
| marvelousweb3 | In reply to @louisevelayo:matrix.org
Yes that is normal behaviors. Every time you redeploy a node, it will have a new node id! But if you reboot a node, node id stays the same * Thank you for your insights. Additionally, I would like to inquire whether we need to resubmit our reward proposal following the change in node ID? | 20:33:57 |
| Louise Velayo | Aviate Labs | In reply to @marvelousweb3:matrix.org
Thank you for your insights. Additionally, I would like to inquire whether we need to resubmit our reward proposal following the change in node ID? As they were onboarded using your node operator key, I don’t think you need to. But Saša Tomić [DFINITY] please correct me if I am wrong | 20:44:00 |
| Saša Tomić [DFINITY] | In reply to @louisevelayo:matrix.org
As they were onboarded using your node operator key, I don’t think you need to. But Saša Tomić [DFINITY] please correct me if I am wrong No need to, that is correct. | 21:26:18 |
| Oleksandr | Redacted or Malformed Event | 22:40:28 |
| Oleksandr | Redacted or Malformed Event | 22:40:41 |
| Oleksandr | Redacted or Malformed Event | 22:44:35 |
