libc/atexit: honor registrations made during exit processing

[Junbo-Zheng]

Summary

atexit_call_exitfuncs() evaluated its loop bound once on entry (for (idx = aehead->nfuncs - 1; idx >= 0; idx--)), while atexit_register() appends new entries at funcs[nfuncs] and never decrements nfuncs. Any function registered by an exit handler via atexit() / on_exit() / __cxa_atexit() therefore lands above the cached bound and is never invoked -- even though the registration returns OK. This is the standard C++ code path where a function-local static's destructor is registered by __cxa_atexit at construction time, and that construction first runs during exit processing.

This contradicts the exit(3) documentation that NuttX mirrors verbatim in its own exit() docstring (libs/libc/stdlib/lib_exit.c):

It is possible for one of these functions to use atexit(3) or
on_exit(3) to register an additional function to be executed
during exit processing; the new registration is added to the
front of the list of functions that remain to be called.

The fix restructures the consumption loop into while (aehead->nfuncs > 0): the newest entry is claimed under
ta_lock, cleared, and nfuncs-- frees the slot (a registration made during exit lands exactly there and runs on the next iteration, before the older remaining entries -- the documented semantics); the handler is invoked with the lock released.

The same restructure closes a second defect: atexit_call_exitfuncs() previously read and cleared the task-group-shared ta_exit list without holding ta_lock, while atexit_register() takes it ("The following must be atomic"). Claiming entries under the lock closes that data race; calling handlers with the lock released keeps a handler re-entering atexit_register() deadlock-free on the non-recursive nxmutex.

Evidence: exit(3) man page, DESCRIPTION -- https://man7.org/linux/man-pages/man3/exit.3.html
NuttX mirrors this passage verbatim in its own exit() docstring --

nuttx/libs/libc/stdlib/lib_exit.c

Lines 65 to 70 in 5a209a8

	* called, in the reverse order of their registration. (It is
	* possible for one of these functions to use atexit(3) or
	* on_exit(3) to register an additional function to be executed
	* during exit processing; the new registration is added to the
	* front of the list of functions that remain to be called.) If one
	* of these functions does not return (e.g., it calls _exit(2), or

Impact

• Users: functions registered during exit processing (from an atexit handler, or via __cxa_atexit from a C++ destructor chain) are now actually invoked, in the documented order (newest first, ahead of older remaining handlers). Previously they were silently dropped while atexit() reported success.
• Build: None.
• Hardware: None -- pure libc-level fix, no board specifics.
• Documentation: None -- the code now matches the behavior already documented in the exit() docstring and the exit(3) man page.
• Security & Compatibility: closes a data race on the task-group shared exit-function list (consume side now holds ta_lock). No API change.

Testing

Simulated (sim:nsh host build on Ubuntu x86-64, CONFIG_LIBC_MAX_EXITFUNS=8 -- the sim default is 1).

Build and run:

cmake -B build -DBOARD_CONFIG=sim:nsh -GNinja
cmake -S . -B build   # after setting CONFIG_LIBC_MAX_EXITFUNS=8 # in build/.config
cmake --build build -j$(nproc)
(echo hello; echo poweroff) | ./build/nuttx

"hello" runs the test at the NSH prompt; poweroff terminates the sim.

The test was carried by apps/examples/hello/hello_main.c (scratch
only, not part of this PR); its diff:

--- a/examples/hello/hello_main.c
+++ b/examples/hello/hello_main.c
@@ -24,6 +24,7 @@

 #include <nuttx/config.h>
 #include <stdio.h>
+#include <stdlib.h>

 /****************************************************************************
  * Public Functions
@@ -33,8 +34,29 @@
  * hello_main
  ****************************************************************************/

+static void handler_b(void)
+{
+  printf("ATEXIT-TEST: handler B called (registered during exit)\n");
+}
+
+static void handler_a(void)
+{
+  int ret;
+
+  printf("ATEXIT-TEST: handler A called\n");
+  ret = atexit(handler_b);
+  printf("ATEXIT-TEST: atexit(handler_b) inside A returned %d\n", ret);
+}
+
+static void handler_c(void)
+{
+  printf("ATEXIT-TEST: handler C called\n");
+}
+
 int main(int argc, FAR char *argv[])
 {
   printf("Hello, World!!\n");
+  atexit(handler_c);   /* older entry, must run LAST */
+  atexit(handler_a);   /* registers handler_b during exit */
   return 0;
 }

Before the fix:

Hello, World!!
ATEXIT-TEST: handler A called
ATEXIT-TEST: atexit(handler_b) inside A returned 0
ATEXIT-TEST: handler C called

(handler B is never invoked although its registration returned 0)

After the fix:

Hello, World!!
ATEXIT-TEST: handler A called
ATEXIT-TEST: atexit(handler_b) inside A returned 0
ATEXIT-TEST: handler B called (registered during exit)
ATEXIT-TEST: handler C called

Signed-off-by: Junbo Zheng zhengjunbo1@xiaomi.com

[github-actions]

MemBrowse Memory Report

No memory changes detected for:

• arduino-mega2560
• esp32-devkitc
• hifive1-revb
• mirtoo
• qemu-armv8a
• qemu-intel64
• rx65n-rsk2mb
• s698pm-dkit
• stm32-nucleo-f103rb