feat(migration): move to new my - #1609
Draft
edospadoni wants to merge 30 commits into
Draft
Conversation
edospadoni
changed the title
edospadoni
changed the title
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
2 times, most recently
from
April 23, 2026 12:02
2f327f7 to
9f5e205
Compare
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
April 23, 2026 14:01
9f5e205 to
3debb8f
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Apr 23, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
April 24, 2026 07:27
3debb8f to
0962cdc
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Apr 24, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
April 24, 2026 07:45
0962cdc to
15e4b9f
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Apr 24, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
changed the title
26 tasks
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
June 17, 2026 09:23
15e4b9f to
98e0d88
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Jun 17, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Jun 29, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
June 29, 2026 10:42
ee14f9e to
4dfc096
Compare
This was referenced Jul 3, 2026
Member
Author
|
Added the hotspot login via My Nethesis ( |
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
July 9, 2026 10:15
308209b to
04c409e
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Jul 9, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
|
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
July 20, 2026 13:32
079f1e6 to
949538a
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Jul 20, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
July 27, 2026 09:59
ec8ade5 to
3171d81
Compare
edospadoni
added a commit
to NethServer/ns8-core
that referenced
this pull request
Jul 27, 2026
Final migration PR for ns8-core, mirror of the nethsecurity cutover on NethServer/nethsecurity#1609. After this commit, nsent clusters talk directly to the my collect API with native my credentials; the legacy my.nethesis.it /api/ and /isa/ endpoints and the /proxy/* translation routes are gone from the hot path. nscom clusters keep using the legacy my.nethserver.com / backupd.nethesis.it infrastructure — that is explicitly out of scope for the my migration. Credential rotation (existing nsent clusters): - New /var/lib/nethserver/cluster/bin/migrate-to-my: idempotent bash one-shot, gated on provider=nsent. Calls the translation proxy's /proxy/credentials with the legacy Basic-Auth pair, reads back the mapped my system_key/system_secret and atomically HSETs cluster/subscription in Redis. Preserves the legacy pair under legacy_system_id / legacy_auth_token for audit and manual rollback, (re)asserts collect_url, and sets the migrated='1' marker that stops the helper from running again. A single HSET guarantees no half-migrated state. - send-cluster-backup / send-heartbeat / send-inventory invoke migrate-to-my up front on the nsent branch so the first successful cron/timer tick flips a pre-migration cluster over. Native my registration (fresh nsent subscriptions): - set-subscription subscribe_nsent now POSTs my.nethesis.it/backend/api/systems/register with {system_secret: <pasted>} and stores the returned system_key as cluster/subscription system_id. collect_url is written alongside the VPN metadata and migrated='1' is set so migrate-to-my is a no-op. Community subscribe is untouched; it keeps using the dartagnan endpoint on my.nethserver.com. - terminate_nsent routes /api/Utils/freekey through the preserved legacy_system_id / legacy_auth_token pair when available, so migrated clusters can still release their slot on my-old at unregister time. On pre-migration clusters behaviour is unchanged. - get-subscription fetch_subscription_info_nsent queries collect /info with the rotated credentials and synthesises the legacy envelope the UI consumes (system_url, plan_name, expires, expire_date, status, with_remote_support). The new my data model no longer tracks a subscription plan at the system level, so plan_name falls back to the organization name and expires/expire_date to an "unbounded" default — the UI keeps rendering the same row layout without a KeyError. A pre-migration cluster falls back to a "pending" snapshot instead of raising, so the subscription page stays usable during the first rotation window. Single-path send scripts: - send-heartbeat nsent: POST $collect_url/heartbeat with native Basic-Auth. The primary my.nethesis.it/isa/heartbeats/store and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/heartbeats/store. - send-inventory nsent: POST $collect_url/inventory with a phonehome payload. The primary my.nethesis.it/isa/inventory path, the my-old /api/systems/info registration-date refresh and the proxy shadow are gone; nscom continues on ${dartagnan_url}/machine/inventories/store. - send-cluster-backup: POST $collect_url/backups for nsent, $TYPE-less community path preserved via a dedicated nscom branch (backupd.nethesis.it/community/api/v2/backup/). Failure mode: - A /proxy/credentials outage during an nsent cluster's upgrade window leaves the cluster on legacy credentials against collect, which returns 401. migrate-to-my is re-invoked every time one of the send-* services fires, so the cluster recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
2 times, most recently
from
July 30, 2026 12:25
3a49fef to
8bcd72f
Compare
Reach the new my via the Render prod proxy so migrated units work before the my.nethesis.it DNS flip; reverts at the flip.
Align with main cf2d305 — the option was added during a transition phase and is no longer required. The my-cutover branch predated that removal, so the rebase re-introduced it; remove it again to avoid resurrecting code dropped from main.
Pin ns-ui to the my-native backup UI (nethsecurity-ui#746) so a migrated appliance can browse/restore the backups it now sends to my collect. Temporary git-ref pin — replace with the released tag before merging (see Makefile note).
…ted units subscription-info threads organization.name explicitly; ns.subscription info returns it as the organization and defaults the plan to "Nethesis Enterprise" for enterprise units. Community units keep the real plan name untouched.
register exits 2 on the backend 409 (the my system key is one-shot and never freed) and ns.subscription returns 'system_already_registered' instead of collapsing every failure into the generic 'invalid_secret_or_server_not_found', so the UI can explain a new system is needed.
Align the bundled ns-ui with the re-register UX fix in nethsecurity-ui#746.
Realign the bundled ns-ui after the prettier formatting fix.
ns.subscription info now returns system_url (my-proxy-prod.onrender.com/systems/<uuid>) for enterprise, so the UI can link the System ID to the portal — parity with ns8.
Community units now get system_url=my.nethserver.com/servers/<id> so the UI links the System ID for community as well, matching the enterprise/ns8 behavior.
register sent the inventory before the heartbeat, so a freshly-registered unit stayed unknown/pending on my until the (slower) inventory completed. Send the near-instant heartbeat first so the system flips to active right away; the inventory follows. Registration result is still reported on the heartbeat, as before.
ns.dedalo gains oidc-start/oidc-poll: the unit starts a device pairing on the hotspot manager, hands the verification URL to the browser and polls for the session token, then connects exactly like the password login did. Remote curl calls get timeouts; account name and user are stored in uci so the UI can show who the unit is linked to, and are cleared on password login and unregister.
Follow the rebase of nethsecurity-ui#746 onto its current main.
Follow the lint fix on nethsecurity-ui#746.
Follow the rebase of nethsecurity-ui#746 onto its current main.
Follow the hotspot connection UI restructure on nethsecurity-ui#746.
migrate-to-my rotated the my credentials but never re-ran the feed hooks, so the blocklists and the apk repository kept authenticating as the legacy system: they work only while the old my still answers for it. Re-run the feed hooks after the rotation. ts-ip added banip's ban_allowurl entry only when no bl.nethesis.it one was present, so an entry written before a rotation kept the old pair for good. Enforce a single entry holding the current credentials.
edospadoni
force-pushed
the
feat/backup-my-cutover
branch
from
August 28, 2026 07:13
6b0cd48 to
e5c9286
Compare
Tbaile
requested changes
Aug 28, 2026
Tbaile
left a comment
Tbaile
left a comment
Collaborator
There was a problem hiding this comment.
This review is harsh due to just the many credentials leak that will happen in case the curl hangs for some reason since they get printed directly into ps. Sadly the system is taunted with many of them.
One of the other major issues is the migration to the new my that gets called whenever we run the backups, heartbeat or inventory.
I can't figure another way to implement this without breaking the old implementation, a init.d would be nice but if it doesn't run we encounter issues with all the scripts, it's not acceptable.
We could've got the chance to rewrite many of the components with python to streamline and avoid bash directly, sadly this won't be the case and I don't think right now is possible.
I'll try and put my head around it if I find some solution to the migration, keep it like this for the moment.
| # Native my Mimir alertmanager, derived from collect_url, e.g. | ||
| # https://my-proxy-prod.onrender.com/collect/api/systems | ||
| # -> https://my-proxy-prod.onrender.com/collect/api/services/mimir/alertmanager | ||
| notifier_url="${collect_url%%/collect/*}/collect/api/services/mimir/alertmanager" |
Collaborator
There was a problem hiding this comment.
Variable sub is always nasty on initd, try to keep away from it, a second uci variable is more suited
Collaborator
There was a problem hiding this comment.
the "migration" section should end up inside the python3-nethsec facts library
Comment on lines
+172
to
+176
| # Refuse the call before running sysupgrade/uploading, and emit | ||
| # valid JSON so the HTTP API wraps it as a 422 ValidationError | ||
| # the UI can render (the previous form printed a Python dict | ||
| # repr, which was silently dropped upstream and caused the run | ||
| # modal to stay open after a successful upload). |
Collaborator
There was a problem hiding this comment.
AI generated a way verbose comment to address a simple validation error, that's the case for all the changes, we should keep it simple
| # (legacy) creds and POSTs to /proxy/alerts until the next reload. Reloading | ||
| # now makes the alert path flip together with the credential rotation. Safe | ||
| # no-op if victoria-metrics is not installed. | ||
| /etc/init.d/vmalert reload 2>/dev/null || true |
Collaborator
There was a problem hiding this comment.
Restart will be better fitted here, leave reload for the service triggers through config, comment is too verbone and returns a (now with 8.8) wrong affirmations, victoria-metrics is always installed
Comment on lines
+102
to
+104
| for feed_setup in /usr/sbin/ts-ip /usr/sbin/ts-dns /usr/sbin/distfeed-setup; do | ||
| [ -x "$feed_setup" ] && "$feed_setup" || true | ||
| done |
Collaborator
There was a problem hiding this comment.
ts-ip, ts-dns and distfeed might not work with the new credentials. They still point to old my to check, we need to plan the migration for them.
Comment on lines
+40
to
+52
| err=$(/usr/bin/curl -m 30 --retry 3 -L -sSf -X POST \ | ||
| --user "$SYSTEM_ID:$SYSTEM_SECRET" \ | ||
| -o /dev/null -w 'HTTP %{http_code}' \ | ||
| "$COLLECT_URL/heartbeat" 2>&1) || \ | ||
| logger -t send-heartbeat "heartbeat send failed: $err" | ||
| ;; | ||
| community) | ||
| URL=$(uci -q get ns-plug.config.alerts_url)"heartbeats/store" | ||
| /usr/bin/curl -m 180 --retry 3 -L -s \ | ||
| --header "Authorization: token $SYSTEM_SECRET" \ | ||
| --header "Content-Type: application/json" \ | ||
| --header "Accept: application/json" \ | ||
| --data-raw '{"lk": "'$SYSTEM_ID'"}' "$URL" >/dev/null |
Collaborator
There was a problem hiding this comment.
This was already here, but we have leaking credentials here too
Comment on lines
+29
to
+34
| if [ -n "$LEGACY_ID" ] && [ -n "$LEGACY_SECRET" ]; then | ||
| curl -s -m 180 --retry 3 -L \ | ||
| -H "Content-type: application/json" -H "Accept: application/json" \ | ||
| -d "{\"lk\":\"$LEGACY_ID\",\"secret\":\"$LEGACY_SECRET\"}" \ | ||
| https://my-proxy-prod.onrender.com/api/Utils/freekey >/dev/null | ||
| fi |
Comment on lines
+46
to
+49
| resp=$(/usr/bin/curl -f -s -m $timeout --retry-delay 1 --retry 2 -L \ | ||
| -H "Accept: application/json" \ | ||
| --user "$system_id:$secret" \ | ||
| "$collect_url/info") || exit $? |
Collaborator
There was a problem hiding this comment.
This file has the same issues, the migration is fragile and credentials leak
Comment on lines
+58
to
+64
| resp=$(/usr/bin/curl --silent --location-trusted --fail-with-body \ | ||
| --max-time 30 --retry 2 \ | ||
| --user "$SYSTEM_ID:$SYSTEM_SECRET" \ | ||
| https://my-proxy-prod.onrender.com/proxy/credentials 2>/dev/null) || { | ||
| logger -t migrate-to-my "credential fetch failed; will retry on next run" | ||
| exit 0 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final migration PR that takes the appliance off
backupd.nethesis.it, off my-old (/api/,/isa/) and off themy.nethesis.it/proxy/*translation layer, and moves it onto themycollect API with its own native credentials. After this ships,type=enterpriseunits talk directly to the my collect endpoints;type=communityunits are left on the legacymy.nethserver.com/backupd.nethesis.itinfrastructure — that is explicitly out of scope for the my migration.What it does
migrate-to-my(new idempotent one-shot, enterprise-only). Runs at the top of every send-* script./proxy/credentialswith the legacy Basic-Auth pair.ns-plug.config.system_id/secretto the mapped my credentials.legacy_system_id/legacy_secret(audit + rollback).collect_url, stampsmigrated_atand setsmigrated='1'so subsequent runs are no-ops.Scripts cutover.
send-heartbeat,send-inventory,send-backup,remote-backupall drop the dual-send to backupd + proxy and do a single call to$collect_url/{heartbeat,inventory,backups/*}with rotated credentials on the enterprise branch. Community branches are byte-for-byte untouched.Native registration.
/usr/sbin/registerenterprise branch now POSTsmy.nethesis.it/backend/api/systems/registerwith{system_secret: <pasted>}, reads backsystem_key, writescollect_url+migrated='1'so a fresh unit lands on the new my without passing through the proxy. Community register is unchanged.Subscription info read path.
/usr/sbin/subscription-infoenterprise branch now queries collect/infowith the rotated credentials and emits a legacy-shaped envelope sons.subscription infoand the existing Vue UI keep parsing the same keys. Plan / expiration fields fall back to the organization name / null — the new my data model no longer tracks a subscription plan at the system level.Migration fingerprint in phonehome.
phonehomenow publishes afacts.migrationblock with:from_legacy_system_id: the pre-rotation UUID when the unit was migrated through the proxy,nullon fresh natively-registered units.migrated_at: ISO 8601 timestamp of the rotation,nullon fresh units.This lets
mycount how many legacy units have completed the rotation and decide whenbackupd.nethesis.itand the translation proxy can be decommissioned.Resilience
A
/proxy/credentialsoutage during an enterprise unit's upgrade window leaves the unit on legacy credentials against collect, which returns401.migrate-to-myis re-invoked every 10 minutes viasend-heartbeat's cron entry, so the unit recovers automatically once the proxy is back up. Accepted trade-off: no dual-mode in the scripts; the simpler single-send path is preferred.Blocking / coordination
Warning
Do not merge this PR until
myis in production.Once this ships, enterprise appliances stop talking to
backupd.nethesis.itand my-old entirely. Ifmyis not serving the/collect/api/systems/*endpoints and the/proxy/credentialsrotation helper in production, the unit goes silent frommy's perspective until the services come up.Requires in order:
Post-merge operational follow-up
/proxy/credentialssuccess rate during the upgrade rollout.facts.migration.from_legacy_system_id != null. When this count equals the rows inproxy_mappings, every legacy unit has rotated andbackupd.nethesis.it+ the translation proxy can be decommissioned.Tracking issue: NethServer/my#83.