Prevent mesh firmware checks from interrupting node updates

[ExtremeFiretop]

Fix a new bug identified and introduced by commit: bae3640
And PR: #539

This may also be related to a report from JimbobJay here:
https://www.snbforums.com/threads/merlinau-v1-6-8-the-ultimate-firmware-auto-updater.96306/post-999444

This bug appears to be a concurrent firmware management collision, essentially a race while flashing a node while checking it for updates from the primary at the same time.

I accidently ran into this, the issue seems to be that the node had prepared for the update, authenticated to its own WebUI, and was just about to hand the beta1 image to /upgrade.cgi

When at that exact time, I started an upgrade from the primary, which logged into the node to check for updates, triggered start_webs_update, and resulted in putting the node’s ASUS firmware update system into a competing update/check state and ultimately MerlinAU rebooted the node without flashing any firmware.

[ExtremeFiretop]

@Martinski4GitHub

Have a look at this...
The node had already started its own MerlinAU update.
It restarted httpd, then logged into its own WebUI as 192.168.50.3, and shortly afterward mentioned the beta1 flash:

• NODE SYSTEM LOGS BELOW:

Sep 15 08:57:49 rc_service: service 21766:notify_rc restart_httpd
Sep 15 08:57:49 RT-BE92U: start https:8443
Sep 15 08:57:49 RT-BE92U: start httpd:80
Sep 15 08:57:49 httpd: Succeed to init SSL certificate...8443
Sep 15 08:57:49 httpd: S:RT-BE92U-2340 Server Certificate, I:GT-BE98_Pro-DE30 Root Certificate 20240101000048, 2026/6/14 ~ 2046/6/15
Sep 15 08:57:49 rc_service: service 21775:notify_rc restart_leds
Sep 15 08:57:51 rc_service: service 21787:notify_rc restart_leds
Sep 15 08:57:53 rc_service: service 21974:notify_rc restart_leds
Sep 15 08:57:55 rc_service: service 21986:notify_rc restart_leds
Sep 15 08:57:57 rc_service: service 21997:notify_rc restart_leds
Sep 15 08:57:59 MerlinAU_[14179]: The email notification was sent successfully [START_FW_UPDATE_STATUS].
Sep 15 08:57:59 HTTPD: [LOGIN][http][Web] successful (192.168.50.3)
Sep 15 08:57:59 rc_service: service 22057:notify_rc restart_leds
Sep 15 08:58:01 MerlinAU_[14179]: ---------------------------------------------------------------
Sep 15 08:58:01 MerlinAU_[14179]: Cron job #1: [45 */6 * * * /jffs/addons/amtm/routerdate cron #amtm_RouterDate#]
Sep 15 08:58:02 MerlinAU_[14179]: Cron job [amtm_RouterDate] was removed successfully.
Sep 15 08:58:02 MerlinAU_[14179]: Cron job #2: [18 5 * * * /bin/sh /jffs/addons/amtm/sc_update.mod -run #amtm_ScriptsUpdateNotification#]
Sep 15 08:58:03 MerlinAU_[14179]: Cron job [amtm_ScriptsUpdateNotification] was removed successfully.
Sep 15 08:58:03 MerlinAU_[14179]: Cron job #3: [21 2 * * * sh /jffs/scripts/backupmon.sh #RunBackupMon#]
Sep 15 08:58:04 MerlinAU_[14179]: Cron job [RunBackupMon] was removed successfully.
Sep 15 08:58:04 MerlinAU_[14179]: Cron job #4: [20 3 * * * sh /jffs/scripts/MerlinAU.sh run_now #MerlinAU_FWUpdate#]
Sep 15 08:58:05 MerlinAU_[14179]: Cron job [MerlinAU_FWUpdate] was removed successfully.
Sep 15 08:58:05 MerlinAU_[14179]: Cron job #5: [08 0 * * * /bin/sh /jffs/addons/amtm/led_control.mod -set #amtm_LEDcontrol_set#]
Sep 15 08:58:06 MerlinAU_[14179]: Cron job [amtm_LEDcontrol_set] was removed successfully.
Sep 15 08:58:06 MerlinAU_[14179]: Cron job #6: [5 3 * * * sh /jffs/scripts/MerlinAU.sh checkupdates #MerlinAU_ScriptUpdate#]
Sep 15 08:58:07 MerlinAU_[14179]: Cron job [MerlinAU_ScriptUpdate] was removed successfully.
Sep 15 08:58:07 MerlinAU_[14179]: Cron jobs [6] from 3rd-party add-ons were found.
Sep 15 08:58:07 MerlinAU_[14179]: ---------------------------------------------------------------
Sep 15 08:58:22 MerlinAU_[14179]: Post-update email notification hook was added successfully to '/jffs/scripts/services-start' script.
Sep 15 08:58:22 MerlinAU_[14179]: Flashing RT-BE92U_3006_102.9_beta1_nand_squashfs.pkgtb... Please wait for reboot in about 4 minutes or less.

Exactly 4 seconds after the node entered that flashing stage, the primary logged into it, and invoked the firmware-check process as found below:

Sep 15 08:58:22 MerlinAU_[14179]: Flashing RT-BE92U_3006_102.9_beta1_nand_squashfs.pkgtb... Please wait for reboot in about 4 minutes or less.
Sep 15 08:58:26 HTTPD: [LOGIN][http][Web] successful (192.168.50.1)
Sep 15 08:58:26 rc_service: httpd 21769:notify_rc start_webs_update
Sep 15 08:58:28 MerlinAU_[14179]: START of Unmount/Eject USB Drive...

When I checked the primary router MerlinAU logs for timestamps it confirms it:

• PRIMARY MERLINAU LOGS BELOW:

2026-09-15 08:58:25 MerlinAU v1.6.8
2026-09-15 08:58:26 Running the update task now. Checking for F/W updates...
2026-09-15 08:58:26 Successful Login for AiMesh Node [192.168.50.3].

After MerlinAU finished ejecting the USB device, and the flash attempt never moved on to the next step which is the normal ASUS upgrades path. (No calls to stop_upgrade or start_upgrade)
Instead, the next firmware related activity is another firmware check, followed by a plain reboot 3 minutes later by MerlinAU.

[ExtremeFiretop]

Actually... I'm testing another possibility/theory:

• node logs in, that creates cookie "A"
• then the primary logs in, that creates cookie "B"
• Router then invalidates original cookie "A"
• node finishes USB eject and sends upgrade.cgi using Cookie "A"
• authentication rejected/redirected, and we see no no stop_upgrade and no start_upgrade

Hold tight...

[ExtremeFiretop]

@Martinski4GitHub

Confirmed on my side, this is likely what happened. The primary is able to null the cookie of the node.
The primary logged into the node to check for updates, in the vulnerable window between restarting it's own WebUI, and unmounting it's USB.

That USB window can be anywhere from 10 seconds to 45 seconds+ based on the new USB unmount code. Unloading cron jobs just as long.
That window is vulnerable for something else to login and null the first cookie, such as the primary.

Once that happens, the cookie is now null, and the flash fails.
I kept the guard in place, but adjusted the timing so we restart the WebUI RIGHT before we login to flash.

Also adjusting the flash order so we delete cron jobs before unloading Entware.
(Incase a cron tries to fire for an entware script that is unloaded)

This may also be related to a report from JimbobJay here:
https://www.snbforums.com/threads/merlinau-v1-6-8-the-ultimate-firmware-auto-updater.96306/post-999444

[ExtremeFiretop]

Unfortunately, because we previously never kept /tmp/upload_response.txt even though we generated it.
The true answers of what happened will need to be assumed, we won't know.

It has to be one or the other...
Either

1. a invalidated cookie, which seems more and more likely.
2. Or some type of internal concurrent firmware management collision when checking for updates while running an update.

This PR now attempts to address both.
I also added extra logging so if it happens again, we save /tmp/upload_response.txt in the future.

I tested it, and in case it does ever happen, we now should see this in the logs:

Sep 16 02:01:55 MerlinAU_[8709]: Successful Login for AiMesh Node [192.168.50.3].
Sep 16 02:01:55 MerlinAU_[8709]: AiMesh Node [192.168.50.3] entered an active MerlinAU F/W update before start_webs_update. Skipping firmware check.

[Martinski4GitHub]

@ExtremeFiretop,

That's a very interesting scenario, and a great catch, bud!!

So, essentially, when the primary router logs into a node to check for F/W updates (which also triggers execution of the node's webs_update.sh script), it seems to "kick the node out" of its own login session by invalidating a previously obtained login cookie when/if the node happens to be in the middle of running a F/W update and is about to flash the image.

This scenario creates a conflict between the primary router actions and the node's attempt to flash the F/W image because the primary router is completely unaware that the node is, in fact, in the middle of a F/W update.

Unfortunately, I don't have a mesh setup in my home network, so I cannot help you test, debug, or validate any of the changes. But I'll take a look, review the code, do some sanity checks, and run the Linter tool after merging the PR. Give me until Friday evening to get to it (I'll be busy in the next couple of days).

BTW, a friend of mine reported an issue where he's not able to do a simple 'Copy&Paste' operation when entering the password string on the CLI menu. I found the problem, so I'll fix it soon.

Talk to you later, bud.

[Martinski4GitHub]

@Martinski4GitHub

Confirmed on my side, this is likely what happened. The primary is able to null the cookie of the node. The primary logged into the node to check for updates, in the vulnerable window between restarting it's own WebUI, and unmounting it's USB.

That USB window can be anywhere from 10 seconds to 45 seconds+ based on the new USB unmount code. Unloading cron jobs just as long. That window is vulnerable for something else to login and null the first cookie, such as the primary.

Once that happens, the cookie is now null, and the flash fails. I kept the guard in place, but adjusted the timing so we restart the WebUI RIGHT before we login to flash.

Yeah, that makes sense. Good call!

Also adjusting the flash order so we delete cron jobs before unloading Entware. (Incase a cron tries to fire for an entware script that is unloaded)

In that case, the cron job would simply fail silently; but it's better to avoid such a scenario by changing the order. Good call.

This may also be related to a report from JimbobJay here: https://www.snbforums.com/threads/merlinau-v1-6-8-the-ultimate-firmware-auto-updater.96306/post-999444

In this case, the reported issue was that it failed to remove some cron job. This is independent of whether the cron job itself fails to execute. These are completely separate events.

[ExtremeFiretop]

@ExtremeFiretop,

That's a very interesting scenario, and a great catch, bud!!

Not going to lie, to me, this is the most interesting thing that's come up for merlinAU in a while 😂 I had to investigate!!! Happened 2 days ago on the 15th early morning and then I spent whatever time I could until 2AM on the 16th trying to make sense of what happened to me.

So, essentially, when the primary router logs into a node to check for F/W updates (which also triggers execution of the node's webs_update.sh script), it seems to "kick the node out" of its own login session by invalidating a previously obtained login cookie when/if the node happens to be in the middle of running a F/W update and is about to flash the image.

Bingo, you got it. That seems like the most likely scenario to me anyways and I was able to independently reproduce that scenario. Unlike my initial firmware management collision theory.

I initially thought it was a concurrent firmware management bug deeper in the firmware. (Because let's be honest, when a user normally starts a flash, it starts a spinning wheel and locks up the WebUI, people can't usually login a second time and trigger an update check which is essentially with the primary router did in this case)

But the most likely scenario is the window between the WebUI restart and the actual flash is simply too large and allows for other things to login.

This scenario creates a conflict between the primary router actions and the node's attempt to flash the F/W image because the primary router is completely unaware that the node is, in fact, in the middle of a F/W update.

Ive tried to build in some "communication" between the two. It's rudimentary. We don't really have any real comms between them like SSH. But now the node sets a specific nvram value when it starts the critical flash process.

The primary can then pull that nvram value using the same method it pulls all the other nodes values, if it finds that nvram value is set to 1, it says "node must be flashing, skip whatever I can for the node update check and trigger process"

Unfortunately, I don't have a mesh setup in my home network, so I cannot help you test, debug, or validate any of the changes. But I'll take a look, review the code, do some sanity checks, and run the Linter tool after merging the PR. Give me until Friday evening to get to it (I'll be busy in the next couple of days).

No rush, this is extremely rare and uncommon. An "edge case" if I ever saw one 😉
But I figured I'd protect this vulnerable window as much as possible once I found it.

My guess/assumption is between skipping as much as we can from the primary in an activate node flash, plus restarting the WebUI right up against the start of the flash, is the best way forwards to protect this window.

BTW, a friend of mine reported an issue where he's not able to do a simple 'Copy&Paste' operation when entering the password string on the CLI menu. I found the problem, so I'll fix it soon.

Talk to you later, bud.

I see the PR! I'll replicate if I can and merge.
Talk later buddy!

[ExtremeFiretop]

@Martinski4GitHub
Confirmed on my side, this is likely what happened. The primary is able to null the cookie of the node. The primary logged into the node to check for updates, in the vulnerable window between restarting it's own WebUI, and unmounting it's USB.
That USB window can be anywhere from 10 seconds to 45 seconds+ based on the new USB unmount code. Unloading cron jobs just as long. That window is vulnerable for something else to login and null the first cookie, such as the primary.
Once that happens, the cookie is now null, and the flash fails. I kept the guard in place, but adjusted the timing so we restart the WebUI RIGHT before we login to flash.

Yeah, that makes sense. Good call!

Also adjusting the flash order so we delete cron jobs before unloading Entware. (Incase a cron tries to fire for an entware script that is unloaded)

In that case, the cron job would simply fail silently; but it's better to avoid such a scenario by changing the order. Good call.

Ah yes, funny enough I went back through past PRs and we had this discussion once before. I thought it rang a bell hearing you say that.

Well this time that fact silently went over my head and I just changed the order so we limit that possibly anyways 😜

This may also be related to a report from JimbobJay here: https://www.snbforums.com/threads/merlinau-v1-6-8-the-ultimate-firmware-auto-updater.96306/post-999444

In this case, the reported issue was that it failed to remove some cron job. This is independent of whether the cron job itself fails to execute. These are completely separate events.

Fair enough!

[Martinski4GitHub]

Looking good and approved!!

[Martinski4GitHub]

@ExtremeFiretop,

Sorry for the delayed response. Friday evening, my wife and I went out to dinner with a few friends and came back home a little bit past midnight.

Anyway, I reviewed the code, and everything looks good. I'm going to merge it, but I'm already too tired to run it through the Linter tool. I'll do that tomorrow, Saturday evening.

Enjoy the weekend and take care, bud!!

[Martinski4GitHub]

This call to restart the HTTP daemon was invalidating the previously authenticated login session token/cookie, which would cause the call to flash the F/W to fail.