diff --git a/Documentation/components/fdpic.rst b/Documentation/components/fdpic.rst new file mode 100644 index 0000000000000..1abaca3f3f797 --- /dev/null +++ b/Documentation/components/fdpic.rst @@ -0,0 +1,420 @@ +.. _fdpic: + +============= +FDPIC Modules +============= + +Overview +======== + +An FDPIC module is an ELF shared object whose read-only and writable +segments are placed independently of one another. NuttX uses the +read-only segment where it already lies on the media and never copies +it; only the writable segment is copied to RAM, once per running +instance. A module's code and ``.rodata`` therefore cost no RAM at all, +and several instances of one module share them. + +FDPIC is not a separate binary format and has no loader of its own. An +object announces itself in its OS/ABI byte, +``e_ident[EI_OSABI] == ELFOSABI_ARM_FDPIC`` (65), which ``readelf -h`` +reports as *OS/ABI: ARM FDPIC*, and the ELF loader takes it from there. +Everything else -- ``exec()``, ``posix_spawn()``, ``dlopen()``, the +symbol table -- is the ordinary ELF path. + +What FDPIC adds over the position independent ELF support already in the +tree is a function pointer that carries its own data base. That is what +lets a module be called back on a thread it did not create, and what +lets a module and the libraries it uses hold distinct data bases at the +same time. + +Function descriptors +-------------------- + +Code reaches its own data through a base register -- **r9** on ARM -- +holding the address of that object's GOT. Because code and data are +placed independently, a bare code address is not enough to call a +function: the callee needs its data base too. FDPIC therefore +represents a function pointer as a two word *descriptor*: + +=========== ============================================================== +Word Contents +=========== ============================================================== +``entry`` Code address, including its Thumb bit +``got`` Data base to install in the PIC base register before + branching +=========== ============================================================== + +Building those descriptors is most of what relocation does. Because +each one names its own base, a pointer handed to the base firmware +carries everything needed to call back into the module later, from any +thread. + +A module links against nothing. libc and everything else are undefined +imports, resolved at load time against the globally registered symbols +first, then any shared libraries the module names, then the symbol table +``exec()`` supplied. + +Placement +--------- + +The loader asks the filesystem where the file lies on its media. Two +mechanisms exist and they are not interchangeable: + +* ``XIPFSIOC_PIN`` is for a filesystem that can move a file's blocks. It + returns an address together with a pin that holds the extent still, and + the pin is given back with ``XIPFSIOC_UNPIN`` when the module is + unloaded. :doc:`XIPFS ` is the one in tree. + +* ``FIOC_XIPBASE`` is for a filesystem whose layout never changes, which + has nothing to hold and answers with a bare address. ROMFS and TMPFS + are those. + +The pin is asked for first, because a filesystem that needs one is not +safe without it. The loader asks for a pin only if it can hold one, +which is the flat build, or the pin would stay for ever. + +A filesystem that answers neither is still usable. The loader then +copies the text to RAM, as it does for any other module. The module +loses the shared text and the flash saving, but it runs. + +The writable segment is allocated and copied per instance, and a pool of +function descriptors is reserved behind it for the relocations that ask +the loader to manufacture one. When the task starts, +``up_initial_state()`` installs the object's data base -- ``DT_PLTGOT``, +or the GOT immediately after ``PT_DYNAMIC`` in an object with no +imports -- into the PIC base register. + +Shared libraries +---------------- + +A module may name shared libraries in ``DT_NEEDED``. Each is loaded +during relocation by calling ``dlopen()`` on the name, and the module's +undefined symbols are then bound against that library's exports. This +requires ``CONFIG_LIBC_DLFCN``; without it, a module carrying +``DT_NEEDED`` is refused, because there is no way to bring in what it +asks for. ``CONFIG_LIBC_ELF_MAXNEEDED`` caps how many one module may +name. + +Because ``dlopen()`` does the work, libraries are found the way it finds +them: an absolute path is used as given, and a bare name is searched for +along ``LD_LIBRARY_PATH``, which needs ``CONFIG_LIBC_ENVPATH`` and is +seeded from ``CONFIG_LDPATH_INITIAL``. + +A library lands in the module registry, which holds one instance per +name, so its data is shared by everything that opens it. A module +started with ``exec()`` is different: that path loads a fresh copy each +time, so two running instances of one module have separate data while +sharing one copy of the text in flash. + +Comparison with NXFLAT and PIC ELF +================================== + +All three run position independent code from flash on a target with no +MMU, and all three give several instances of one module a shared +``.text`` with private ``.data``. They differ in what a *pointer* can +express and in what the toolchain has to provide. + +========================= ============== ============== ============= +Property NXFLAT PIC ELF FDPIC +========================= ============== ============== ============= +Format NuttX only ELF ELF +Extra build tools yes none assembler and + linker +Data base per task task object +Shared libraries no no yes +Foreign-thread callback no no yes +Instruction set ARM, Thumb-2 unrestricted Thumb-2 only +========================= ============== ============== ============= + +:ref:`NXFLAT ` is a NuttX-specific format. A module imports +symbols from the base firmware but cannot export any, so shared +libraries are not possible, and the build needs ``mknxflat`` to generate +a thunk, ``ldnxflat`` to link, and one of the ``binfmt/libnxflat`` linker +scripts to place the sections. + +**PIC ELF** needs no extra tools. With ``CONFIG_PIC`` the ELF loader +allocates the writable sections separately and, when the filesystem +answers ``FIOC_XIPBASE``, leaves the read-only ones on the media. Two +limits follow from having one base register per task: a shared object is +loaded as a single allocation, because the distance between its text and +its data is compiled into it, and the data base is installed once per +task, so every object in a task shares one. + +**FDPIC** pays for its descriptors with an ``arm-uclinuxfdpiceabi`` +assembler and linker, and gets back the two things a single register +cannot express. A task or pthread that a module starts inherits the +module's D-Space, so a register would be enough there; a work queue +worker was created at boot and carries no module base, and a descriptor +supplies one, which is how ``SIGEV_THREAD`` notifications reach module +code. + +Requirements +============ + +**An ARM Thumb-2 core.** The boundary is the instruction set, not the +core profile: GCC rejects FDPIC in Thumb-1 mode. + +========================= ========================== ===== +Core Architecture FDPIC +========================= ========================== ===== +Cortex-M3 / M4 / M7 ARMv7-M / ARMv7E-M yes +Cortex-M33 ARMv8-M Mainline yes +Cortex-M0 / M0+ / M23 ARMv6-M / ARMv8-M Baseline no +========================= ========================== ===== + +RISC-V has no FDPIC ABI -- the psABI addendum is an unmerged proposal and +no ``EI_OSABI`` value is assigned -- so a RISC-V target cannot use this. + +**Flash that is memory mapped and executable**, exposed by a filesystem +that answers ``XIPFSIOC_PIN`` or ``FIOC_XIPBASE``. This is what gives +execute in place. Without it the module still loads, but from RAM. + +**An FDPIC assembler and linker.** A stock ``arm-none-eabi`` GCC +compiles correct FDPIC code for both C and C++, but the assembler has to +be in FDPIC mode to accept the relocations that code produces, and only +``arm-uclinuxfdpiceabi`` binutils carry the ``armelf_linux_fdpiceabi`` +emulation the link needs. ``arm-none-eabi-ld``, rather than failing, +marks its output *UNIX - System V*, which the loader will not treat as +FDPIC. + +No distribution packages that target, so build binutils for it -- which +takes about a minute and needs nothing else:: + + configure --target=arm-uclinuxfdpiceabi --prefix=$HOME/fdpic \ + --disable-nls --disable-werror + make && make install + export PATH=$HOME/fdpic/bin:$PATH + +An FDPIC GCC is not needed. + +**The base firmware must reserve r9.** It is not enough for the module +to be well behaved: a firmware routine calling back into module code +arrives with the module's data base in r9 only if the compiler was never +free to allocate that register elsewhere. ``CONFIG_FDPIC`` selects +``CONFIG_PIC``, under which ``arch/arm/src/common/Toolchain.defs`` adds +``--fixed-r9``; see :ref:`nxflat` for why it goes into ``ARCHCFLAGS`` +rather than ``CFLAGS`` and how to check that it arrived. + +Configuration +============= + +``CONFIG_FDPIC`` lives under ``CONFIG_ELF``. A working configuration +also needs a symbol table for modules to import from and a filesystem +that can expose its media:: + + CONFIG_ELF=y + CONFIG_FDPIC=y + CONFIG_LIBC_EXECFUNCS=y + CONFIG_EXECFUNCS_HAVE_SYMTAB=y + CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y + CONFIG_FS_XIPFS=y + +Shared libraries need three more, the last two so that a library can be +named rather than spelled out as an absolute path:: + + CONFIG_LIBC_DLFCN=y + CONFIG_LIBC_ENVPATH=y + CONFIG_LDPATH_INITIAL="/mnt/xipfs" + +``CONFIG_ELF_STACKSIZE`` gives the stack a module runs with. A module +that needs a different one can export an ``nx_stacksize`` symbol, which +the loader prefers when present. + +Building a module +================= + +Select ``CONFIG_FDPIC`` and a module is built by the ordinary in-tree ELF +build: the same ``MODULE = m`` in the same application Makefile as any +other, and the same ``crt0``. Nothing else is needed. + +What the build does differently is add two compiler flags and use a +different linker:: + + arm-none-eabi-gcc -mcpu=cortex-m33 -mthumb -mfdpic -fPIC -Os \ + -fno-builtin -D__NuttX__ -I$NUTTX/include -c mod.c -o mod.o + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e _start -T $NUTTX/libs/libc/elf/gnu-elf.ld \ + -o mod $NUTTX/arch/arm/src/crt0.o mod.o + +Only the link needs the FDPIC toolchain. The stock compiler emits correct +FDPIC objects for both C and C++, its assembler included. That linker is in +the NuttX CI image; ``tools/ci/docker/linux/Dockerfile`` shows how it is +built. ``FDPIC_CROSSDEV`` names a different prefix, and the build says so +if it is missing. + +Five flags carry weight: + +* ``-mfdpic`` is stated rather than assumed, so a mis-set toolchain fails + loudly instead of producing a plain ELF the loader will not recognize. + +* ``-fPIC`` is not implied by ``-mfdpic`` on a bare-metal target, and + without it the link emits ``TEXTREL``. Text relocations cannot work + against text executed from read-only flash. + +* ``-shared`` preserves the ``R_ARM_FUNCDESC_VALUE`` relocations for + imported symbols. A PIE link with ``--unresolved-symbols=ignore-all`` + appears to work but degrades every import to ``R_ARM_NONE``, and the + module branches to zero on its first call into the firmware. + +* ``-m armelf_linux_fdpiceabi`` is required: this linker supports several + emulations and will not guess. + +* ``-e _start`` names the entry point. ``crt0.c`` is the module's own + start-up file, the one every other module uses: it walks ``.init_array`` + and then calls ``main``. A shared library is never entered, so it is + linked with ``-e 0`` and without ``crt0``. + +A module links with ``-shared``, so importing something the firmware does +not export links cleanly and fails only on the target, as a bare +``-ENOENT``. Checking the module's undefined symbols against the generated +``libs/libc/exec_symtab.c`` catches that at build time, and is what +``apps/examples/fdpicxip/modules`` does for its own fixtures. + +Building a shared library +------------------------- + +A library is built the same way, with a soname and no entry point, and +the module names it on its link line:: + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e 0 -soname libfoo.so -o libfoo.so libfoo.o + + arm-uclinuxfdpiceabi-ld -m armelf_linux_fdpiceabi -shared -z now \ + -e main -o mod.fdpic mod.o libfoo.so + +At run time the library must be reachable under its soname along +``LD_LIBRARY_PATH``. + +Calling back into a module +========================== + +A module's function pointer is the address of a descriptor in its +writable segment. Firmware that stores one and later branches to it +would jump into RAM data, so an entry point that accepts a callback from +a module has to resolve the descriptor first. ``CONFIG_FDPIC`` makes +these do so: + +``qsort``, ``bsearch``, ``pthread_create``, ``signal``/``sigaction``, +``task_create``/``task_create_with_stack``, ``task_spawn``, +``pthread_once``, ``scandir``, and ``mq_notify``/``timer_create`` with +``SIGEV_THREAD``. + +Whether a pointer is a descriptor is decided by reading the PIC base +register: a module's task runs with its data base there, a firmware task +with zero, so a kernel caller is unaffected. + +A new entry point that takes a module callback must resolve it too, under +three rules: + +* **Resolve once, in the innermost common routine.** Resolving twice + treats a code address as a descriptor. ``qsort()`` recurses, so its + public entry resolves and the recursive body does not; ``signal()`` + does not resolve because ``nxsig_action()`` does it for both paths; + ``scandir()`` resolves its filter but not the comparison function it + hands to ``qsort()``. + +* **Exclude sentinel values by hand.** ``fdpic_callback()`` declines to + dereference NULL and nothing else. ``sigaction()`` excludes + ``SIG_IGN``, ``SIG_DFL``, ``SIG_HOLD`` and ``SIG_ERR`` -- the integers + 0, 1, 2 and -1. + +* **A callback on a shared thread needs its base installed.** A + ``SIGEV_THREAD`` notification runs on a work queue worker that carries + no module base, so resolving the entry is not enough. Capture the base + at registration with ``fdpic_base()``, in the module's own context, and + install it around the call with ``fdpic_invoke()``. + +Everywhere else the callback runs in a task that inherited the module's +D-Space, so only the code address needs resolving. + +Limitations +=========== + +**A library's constructors run on the loading task, not on the task that +uses it.** A module that ``exec()`` runs is entered at its own ``crt0``, +which walks ``.init_array`` on the task that runs the module, so its +constructors see that task's group and its own data base. A library is +different: ``dlopen()`` constructs it through ``libelf_insert()``, on +whichever task called the loader, entered through ``fdpic_invoke()`` so +that a global object still reaches its own storage. A library constructor +that reads task-local state, its own pid or its environment, sees that +task's. + +A library named in ``DT_NEEDED`` is constructed before the module that +needs it, because the module's own relocation is what opens it, and +destroyed after, at the last ``dlclose()``. Since the library is one +instance, its constructors run once however many modules name it. + +Destructors are walked from ``DT_FINI_ARRAY`` at unload, in +``libelf_uninit()``, and not from ``crt0``, so they run once whichever way +the object was loaded. + +Reference +========= + +Object layout +------------- + +A linked module already has the layout execute in place needs, with no +linker script:: + + LOAD vaddr 0x00000000 R E .text .rodata .hash .dynsym .dynstr + LOAD vaddr 0x00001244 RW .dynamic .got .data .bss + DYNAMIC DT_PLTGOT -> .got + +``.rodata`` lands in the read-only segment on its own, reached PC +relative or GOT indirect. That matters: in the writable segment it would +be copied to RAM with ``.data``, and most of the saving would evaporate +silently, with everything still working. + +The FDPIC marker is the OS/ABI byte alone. ``e_flags`` reads as an +ordinary ``0x5000000, Version5 EABI``. + +Relocations +----------- + +The static link resolves ``R_ARM_GOT_BREL`` and ``R_ARM_GOTFUNCDESC`` +into the GOT already, so only three types carry work into a linked +module. + +``R_ARM_RELATIVE`` + An address needing its segment's base added. + +``R_ARM_FUNCDESC_VALUE`` + A descriptor the linker has laid out, for the loader to fill in. This + is what a *call* to an imported function produces. When the symbol + resolves to a function in another FDPIC object, both words are copied + from that object's own descriptor, so the callee runs with its own data + base; otherwise the entry is the resolved address and the base is this + object's. + +``R_ARM_FUNCDESC`` + A pointer to a descriptor that does not exist yet, which the loader + manufactures from the pool behind the writable segment. This is what + *taking the address* of a function produces -- a different thing from + calling one, and both can appear for the same symbol. + +Constants, from binutils ``include/elf/arm.h`` and mirrored in +``arch/arm/include/elf.h``: ``R_ARM_GOTFUNCDESC`` 161, +``R_ARM_GOTOFFFUNCDESC`` 162, ``R_ARM_FUNCDESC`` 163, +``R_ARM_FUNCDESC_VALUE`` 164. + +Which table an imported function's descriptor lands in is the linker's +decision: with ``-z now`` imports stay in ``DT_REL``, without it they go +to ``DT_JMPREL``. Both are bound eagerly, so either link works, but the +two are not walked identically. In ``DT_REL`` the word being overwritten +is the addend and is added to the resolved value; in ``DT_JMPREL`` it is +the lazy binding bootstrap and the descriptor is overwritten outright. + +``.rofixup`` is skipped. It is the self-relocation list a *static* +executable's ``crt0`` walks to find its own GOT. A module's ``crt0`` does +not: the loader supplies the data base, in the PIC base register, before +the module is entered. + +Exported functions +------------------ + +A function exported by a module or library is published to ``dlsym()`` +as a descriptor rather than a code address, taken from the same pool, so +that an FDPIC caller can branch through what it gets back. diff --git a/Documentation/components/index.rst b/Documentation/components/index.rst index edd1415ede23d..7c13b4c1b0109 100644 --- a/Documentation/components/index.rst +++ b/Documentation/components/index.rst @@ -15,6 +15,7 @@ case, you can head to the :doc:`reference <../reference/index>`. concurrency/index.rst iterable_sections.rst drivers/index.rst + fdpic.rst nxflat.rst nxgraphics/index.rst paging.rst diff --git a/Documentation/platforms/arm/rp23xx/boards/pimoroni-pico-2-plus/index.rst b/Documentation/platforms/arm/rp23xx/boards/pimoroni-pico-2-plus/index.rst index 8269cc7788c05..6c4045f7ada64 100644 --- a/Documentation/platforms/arm/rp23xx/boards/pimoroni-pico-2-plus/index.rst +++ b/Documentation/platforms/arm/rp23xx/boards/pimoroni-pico-2-plus/index.rst @@ -177,6 +177,21 @@ xipfs XIPFS mounted on the on-board flash, with the ``xipfs`` command and the XIPFS test suite. +xipfs-fdpic +----------- + +Same as ``xipfs``, plus the FDPIC module loader and the +``fdpicxip`` demo, so the ``fdpic`` and ``reject`` sections of the XIPFS +test suite have something to run. The demo carries its modules as +committed byte arrays, so nothing beyond the ordinary ARM toolchain is +needed to build it; rebuilding those from source needs +``arm-uclinuxfdpiceabi`` binutils. See :doc:`/components/fdpic`. + +``CONFIG_ELF_STACKSIZE`` is 4096 here rather than the 2048 the rest of +the board's tasks use. A module that calls into the firmware's printf +family overflows 2048, and with no MPU that is a lockup rather than a +diagnostic. + xipfs-nxflat ------------ diff --git a/boards/arm/rp23xx/pimoroni-pico-2-plus/configs/xipfs-fdpic/defconfig b/boards/arm/rp23xx/pimoroni-pico-2-plus/configs/xipfs-fdpic/defconfig new file mode 100644 index 0000000000000..a06c5c1b82618 --- /dev/null +++ b/boards/arm/rp23xx/pimoroni-pico-2-plus/configs/xipfs-fdpic/defconfig @@ -0,0 +1,66 @@ +# +# This file is autogenerated: PLEASE DO NOT EDIT IT. +# +# You can use "make menuconfig" to make any modifications to the installed .config file. +# You can then do "make savedefconfig" to generate a new defconfig file that includes your +# modifications. +# +# CONFIG_NSH_ARGCAT is not set +# CONFIG_NSH_CMDOPT_HEXDUMP is not set +# CONFIG_NSH_DISABLE_DATE is not set +# CONFIG_NSH_DISABLE_LOSMART is not set +# CONFIG_STANDARD_SERIAL is not set +CONFIG_ARCH="arm" +CONFIG_ARCH_BOARD="pimoroni-pico-2-plus" +CONFIG_ARCH_BOARD_COMMON=y +CONFIG_ARCH_BOARD_PIMORONI_PICO_2_PLUS=y +CONFIG_ARCH_CHIP="rp23xx" +CONFIG_ARCH_CHIP_RP23XX=y +CONFIG_ARCH_RAMVECTORS=y +CONFIG_ARCH_STACKDUMP=y +CONFIG_BINFMT_CONSTRUCTORS=y +CONFIG_BOARDCTL_RESET=y +CONFIG_BOARD_LOOPSPERMSEC=10450 +CONFIG_BUILTIN=y +CONFIG_DEBUG_FULLOPT=y +CONFIG_DEBUG_SYMBOLS=y +CONFIG_DEFAULT_TASK_STACKSIZE=4096 +CONFIG_ELF=y +CONFIG_EXAMPLES_FDPICXIP=y +CONFIG_EXAMPLES_HELLO=y +CONFIG_EXECFUNCS_HAVE_SYMTAB=y +CONFIG_EXECFUNCS_SYSTEM_SYMTAB=y +CONFIG_FDPIC=y +CONFIG_FS_PROCFS=y +CONFIG_FS_PROCFS_REGISTER=y +CONFIG_FS_XIPFS=y +CONFIG_FS_XIPFS_FAULT_INJECT=y +CONFIG_HAVE_CXX=y +CONFIG_INIT_ENTRYPOINT="nsh_main" +CONFIG_INIT_STACKSIZE=16384 +CONFIG_LDPATH_INITIAL="/mnt/xipfs" +CONFIG_LIBC_DLFCN=y +CONFIG_LIBC_ELF_HAVE_SYMTAB=y +CONFIG_LIBC_ENVPATH=y +CONFIG_LIBC_EXECFUNCS=y +CONFIG_MTD=y +CONFIG_NFILE_DESCRIPTORS_PER_BLOCK=6 +CONFIG_NSH_BUILTIN_APPS=y +CONFIG_NSH_READLINE=y +CONFIG_RAM_SIZE=532480 +CONFIG_RAM_START=0x20000000 +CONFIG_READLINE_CMD_HISTORY=y +CONFIG_RP23XX_FLASH_MTD=y +CONFIG_RR_INTERVAL=200 +CONFIG_SCHED_HPWORK=y +CONFIG_SCHED_WAITPID=y +CONFIG_SIG_EVTHREAD=y +CONFIG_START_DAY=9 +CONFIG_START_MONTH=2 +CONFIG_START_YEAR=2021 +CONFIG_SYSLOG_CONSOLE=y +CONFIG_SYSTEM_NSH=y +CONFIG_SYSTEM_XIPFS=y +CONFIG_TESTING_FS_XIPFS=y +CONFIG_TESTING_FS_XIPFS_MTD="/dev/rpflash" +CONFIG_UART0_SERIAL_CONSOLE=y