hm2_eth: Create support library's

[hdiethelm]

This PR has tree parts. If desired, I can split them also up in to separate PR's.

For now it is more about the concepts. No need to review the code from a style standpoint.

Part1: Create support library's
The issue is, that as soon as hm2_eth is linked to libevl, this module can not be used any more without libevl installed.
By creating liblinuxcnc-hm2_eth_net_evl.so and liblinuxcnc-hm2_eth_net_xenomai.so and loading them only when needed, the hard dependency is broken and the library's can be deployed in a linuxcnc-uspace-xenomai / linuxcnc-uspace-evl Debian package.

The Debian packages are not yet created but I am planning to do so if this approach is approved.

The firewall functions are moved out of the Ethernet support modules, so not so many exports are needed for the library's. The function order changes slightly but this should have no side effects.

Alternatives:

• hm2_eth / hm2_eth_xenomai / hm2_eth_evl
  • Three hm2_eth components, main core but different Ethernet code linked.
  • Disadvantages
    • You can not run one Ethernet card with EVL and the other with posix. Niche application if you have a low priority thread communicating with a second mesa card.
    • Three components that could have been a config flag.
• hm2_eth_xenomai / hm2_eth_evl linuxcnc components
  • Disadvantages:
    • More components
    • Empty rtapi_app_main
    • Needs entry in hal file
    • Makefile makes it hard to define custom C / LD flags for specific modules. I did not figure out how to do this.
• Just leave it as it is and depend on libevl
  • Disadvantage: Users would have to compile it from source or three different full linuxcnc-uspace packages are needed.

What do you think about the support library approach?

Part 2: Xenomai3 support
It was easy to do. Note that sendto(), recvfrom() and other syscalls are overwritten by the Xenomai C / LD flags, so even thoutght it looks like posix, behind are xenomai syscalls. This is just how xenomai3 posix skin works.

TBD: if anyone needs this. I just wanted to try it to see any differences to xenomai4. It needs more testing but on my hardware, performance is similar. However, setup is a bit more cumbersome. The two scripts rtnet_up.sh / rtnet_down.sh are just temporary and should go to the man page with some description how to set it up when done.
This will go to a separate PR. Code is here: https://github.com/hdiethelm/linuxcnc-fork/tree/hm2_eth_lib_xenomai3

Part 3: Xenomai4 OOB improvements
Moved to #4503 which should be merged first.

[BsAtHome]

To me it certainly feels like a bad choice doing dlopen in a realtime component.

Is the target for compilation ever both uspace/posix, Xenomai3 and EVL at the same time? If the answer is no, then there should be only one hm2_eth component/driver, which adapts properly at compile time to the necessary layout(s) and format(s). That can still be supported by libraries, but whether they should be .so is doubtful. The compilation generates a (mostly) static driver. Only upwards are symbols matched (to the hostmot2 component). Requiring this also downwards is unnecessary when you know at build-time what your target system is.

So, what is the answer to the question, whether the build can only target one system at a time or multiple simultaneously?

[hdiethelm]

To me it certainly feels like a bad choice doing dlopen in a realtime component.

Why? It only changes from 2x to 3x dlopen in the rtapi_app main thread. But of course there is no dlopen in the realtime tasks which would be bad.
First two:

Load RT extension:

linuxcnc/src/rtapi/uspace_rtapi_main.cc

Line 1533 in fa910f1

dll = dlopen(dllName.c_str(), RTLD_NOW);

Load Component:

linuxcnc/src/rtapi/uspace_rtapi_main.cc

Line 576 in fa910f1

void *module = modules[name] = dlopen(what.c_str(), RTLD_GLOBAL | RTLD_NOW);

With this PR, there is a third dlopen in the same thread in the rtapi_app_main path called just after load:

linuxcnc/src/rtapi/uspace_rtapi_main.cc

Line 583 in fa910f1

int (*start)(void) = DLSYM<int (*)(void)>(module, "rtapi_app_main");

Is the target for compilation ever both uspace/posix, Xenomai3 and EVL at the same time? If the answer is no, then there should be only one hm2_eth component/driver, which adapts properly at compile time to the necessary layout(s) and format(s). That can still be supported by libraries, but whether they should be .so is doubtful. The compilation generates a (mostly) static driver. Only upwards are symbols matched (to the hostmot2 component). Requiring this also downwards is unnecessary when you know at build-time what your target system is.

At build time, if you have libxenomai / libevl installed, all three (+posix) variants are compiled in. But that also means that if you create a debian package, it will automatically depend on these libs.

I use this all day for testing, just boot a different kernel and it works out of the box with the correct back-end, no rebuild needed.

Looks like the reason liblinuxcnc-uspace-xenomai.so / liblinuxcnc-uspace-xenomai-evl.so for rtapi_app also exist to break this dependency, and there is even a control file to pack these in a different debian package: https://github.com/LinuxCNC/linuxcnc/blob/master/debian/control.uspace-xenomai.in But it looks like the rest was either removed or never finalized, so these packages are not built. If this is finalized, you would have:

linuxcnc-uspace: Posix, not depending on libxenomai / libevl
linuxcnc-uspace-xenomai: Xenomai support, depending on libxenomai
linuxcnc-uspace-evl: Xenomai EVL support, depending on libevl

So, what is the answer to the question, whether the build can only target one system at a time or multiple simultaneously?

It can target all of the existing uspace realtime systems at the same time but right now, linuxcnc-uspace will then depend on libxenomai / libevl which is unnecessary but only an issue if these libs are not available in the linuxcnc package repo.

[grandixximo]

This is best argument I heard so far, for splitting the deb...

[hdiethelm]

This is best argument I heard so far, for splitting the deb...

This is also the main intent besides being able to use linuxcnc built with libevl on a system withouth.

However, with a single deb, it will still depend on libevl.

Might be it can be changed to recommended, i could look into how debian packages are configured and if shlibdeps can be modifyed. But I think separate packages are nicer anyway.

[zz912]

Hello,

I have two questions from a user’s perspective:

1. What do you see as the main user-facing reason for supporting Xenomai 3 in hm2_eth? Is there a specific use case for it in LinuxCNC, or is it more about keeping the door open for future use?

2. If Xenomai 4/EVL is the direction LinuxCNC’s realtime support is heading, wouldn’t it be more beneficial for users to focus the effort on a ready-to-use Xenomai 4 + LinuxCNC image/Live CD rather than supporting multiple realtime backends?

[hdiethelm]

Hello,

I have two questions from a user’s perspective:

1. What do you see as the main user-facing reason for supporting Xenomai 3 in hm2_eth? Is there a specific use case for it in LinuxCNC, or is it more about keeping the door open for future use?

Basically, Xenomai3 support is already in since some time (2016). Now that the framework is in to have different backends for hm2_eth, it took only 140 lines of code for Xenomai3 support.

It might be of use for some people, so why not add it, especially due to it took only a few hours to create it.

All I have read so far about realtime performance, there is no way to predict which system works best on your PC. The only real way is to test it. And you can only do that if the support is there. It might well be that for some people, Xenomai3 works better. This was also the main reason for me to implement this: Being able to test how well it works.

2. If Xenomai 4/EVL is the direction LinuxCNC’s realtime support is heading, wouldn’t it be more beneficial for users to focus the effort on a ready-to-use Xenomai 4 + LinuxCNC image/Live CD rather than supporting multiple realtime backends?

This is a question for the maintainers creating live images. Basically, the actual framework would also support a live image with tree different kernels and depending on which you start, the matching RT framework is used.

However, to have generic Debian packages which can be installed with or without libevl / libxenomai, part 1 of this PR (or one of the alternatives) is needed.

[zz912]

Thanks for the explanation. I now understand why you want to keep the individual variants separate – mainly because of the dependencies and because, as a developer/tester, you can have a single build and test it against different kernels.

I’m looking at it a little differently, though, from the perspective of a beginner who is using LinuxCNC for the first time.

Someone coming from Windows or Mac is often also working with Debian for the first time. Such a user really doesn’t know what POSIX, Xenomai 3 or EVL mean. I don’t think it’s ideal to expect them to make this decision before they have even run LinuxCNC for the first time.

A Debian beginner also tends to have exactly the opposite approach from an experienced developer: they would rather have “everything” installed so that it works. For a developer this may be a nightmare, but for a beginner it is completely normal. :-)

So I could imagine something like a simple meta-package called linuxcnc (just a suggestion for the name), which would install all the required variants:

linuxcnc
   ├── linuxcnc-uspace
   ├── linuxcnc-uspace-xenomai3
   └── linuxcnc-uspace-xenomai4

An experienced user could of course still install only linuxcnc-uspace or only linuxcnc-uspace-xenomai4 if they know exactly what they need. But for a beginner there would be a simple path: install linuxcnc and they don’t need to understand the differences between the various realtime environments beforehand.

This is just a suggestion from a UX perspective. I’m not saying that it is technically a better solution than your package separation.

One more small point about the naming. Personally, I would prefer:

linuxcnc-uspace
linuxcnc-uspace-xenomai3
linuxcnc-uspace-xenomai4

instead of linuxcnc-uspace-xenomai and linuxcnc-uspace-evl.

Again, the main reason is the beginner. “EVL” doesn’t mean anything to them, whereas xenomai3 and xenomai4 immediately make it clear that these are two generations of Xenomai. I know that Xenomai 4 uses EVL and that technically it is a bit more complicated, but from a user’s perspective xenomai4 seems like a much more understandable name to me.

So perhaps it makes sense to keep the packages separate for clean dependencies, while also providing one simple “default” path for people who are just getting started with LinuxCNC and Debian.

[hdiethelm]

Basically, not keeping the add-ons separate would mean either:

• linuxcnc-uspace depends on libevl / libxenomai which are not available for install (for now) except from my github
• Build from source or deploy separately as it is the case right now

If you don't know what this means, either google it or read the doc. I still have to update the main section about real time tough. For beginners, it's probably the best to stay with PREEMPT_RT, at least until there is some experience from more advanced users.

linuxcnc -> this is already taken by the RTAI variant

Suggestion which (mostly) matches the naming already used in the source:
linuxcnc-uspace suggests: linuxcnc-uspace-xenomai, linuxcnc-uspace-xenomai-evl in the case these packages are built
linuxcnc-uspace-xenomai
linuxcnc-uspace-xenomai-evl

[hdiethelm]

@BsAtHome Would you prefer if I separate the Xenomai4 OOB improvements to a separate PR?

The library part seams to need some more discussion. But it is also needed for Xenomai3 due to Xenomai3 uses a many C and LD flags to replace posix functionality which would probably be a bad idea to apply to all components.

[hdiethelm]

To simplify review and testing, I decided to create a PR only concerning the Xenomai4 improvements: #4503 It should be merged first due to there will be some conflicts but they are easy to resolve.

[BsAtHome]

Separating the PR's library question from other things would be a good thing because it separates features. I do have some worries about the proper separation of the CFLAGS/LDFLAGS necessities.

Having this in-tree means it needs to be maintained. And, you already noticed, some parts have been bitrotting. Fixing that is a good thing. Adding to it makes things more complex. That should be no problem if there is a way to maintain it properly.

Question 1: Can the combined packages for uspace/xenomai/evl be built in CI? You don't need a running kernel for the build process afaik.
Question 2: How is this going to be tested in real life? Is there a live/install image coming with three kernels so? And, how is that being maintained?

[hdiethelm]

Separating the PR's library question from other things would be a good thing because it separates features. I do have some worries about the proper separation of the CFLAGS/LDFLAGS necessities.

Done, I removed all code not needed for the library creation. However, the commit moving the firewall functionality to hm2_eth (also included in #4503) is needed. The library change is: 9978fee. The original branch is at https://github.com/hdiethelm/linuxcnc-fork/tree/hm2_eth_lib_xenomai3 until this has settled.

Yes, I also don't like the way it is now with CFLAGS/LDFLAGS for components. There is no way to change them for a single component as much as it looks. All files are added to a huge list and then compiled with the same settings, except the path based filter for cruckig. So it's either all, nothing or a complex filter.

If there is a way to build certain components with different flags, an other viable option would be to build hm2_eth + back-end in 3 variants: hm2_eth / hm2_eth_evl / hm2_eth_xenomai and then instead of using a config flag, select the back-end by loading the component. This was indeed my first approach and it would not need dlopen(). I discarded it because it is less flexible, you can not mix posix and evl cards.

For evl, CFLAGS/LDFLAGS is not a big issue, the only flag is -levl but the linker removes it if there is nothing to link. For Xenomai3, there are many flags which I really don't want to apply to all components.

Having this in-tree means it needs to be maintained. And, you already noticed, some parts have been bitrotting. Fixing that is a good thing. Adding to it makes things more complex. That should be no problem if there is a way to maintain it properly.

Question 1: Can the combined packages for uspace/xenomai/evl be built in CI? You don't need a running kernel for the build process afaik.

Yes, I supply the packages needed here:
https://github.com/hdiethelm/xenomai3-linuxcnc
https://github.com/hdiethelm/xenomai4-linuxcnc

I can install them in CI similar than we do for the RTAI build. To build the linuxcnc:
Xenomai4 needs libevl / libevl-dev / linux-libc-evl-dev
Xenomai3 needs libxenomai / libxenomai-dev

However, with the package structure right now, linuxcnc-uspace will then depend on libevl + libxenomai due to shlibdeps detecting the dependency. This is somewhat hidden, as soon as the libraries are installed, configure detects them and thus builds the additional files which then add dependency's to the debian packages. Creating separate packages will fix this.

BTW: This is nothing introduced by my changes. This issue exists probably since the separate xenomai packages where removed: 6f285604ac1a

Question 2: How is this going to be tested in real life? Is there a live/install image coming with three kernels so? And, how is that being maintained?

Right now, I maintain the two repos above. A live image would surely be possible but I did not look yet into how this is done. Otherwise, you can install all which is needed from my repo or build it from source with the supplied scripts and configs.

For Xenomai4, I run my machine with this Kernel + hm2_eth, so this will stay up to date at least for the near future and is tested.

For Xenomai3, I just use this for testing. There it depends on the interest. If there is no feedback or tickets incoming, it might start to bitrot. The main intent there was to compare it to Xenomai4 but if there is interest, I will update it from time to time.

[BsAtHome]

With respect to CI, I don't think we should build all the .deb variants in one runner. Most people won't see that case. I was more thinking about the way the RTAI runner works. That leaves the normal builds without added dependencies. A dedicated CI runner will build the xenomai/evl debs (and only targeted at latest Debian stable). We need to get that right so no one gets confused which deb is which. Importantly the linuxcnc-uspace deb from either runner targeting the same Debian release should be bit-equal (i.e. reproducible build).
That surely means, we need to do the package split right now so we never ever get into the problem in the first place. Confusion is the last thing we want to create.

It may be a necessity to support Xenomai3 as long as Xenomai4 lacks some features (or not?). That is, unless there is a reason only to support the head, which may be less problematic for us in the long run. But that only works if we really don't need Xenomai3 for supporting older/other installs. It also depends on the kernel support for each Xenomai version and the prospects for future support. Once we formalize a way, we have a hard time getting rid of it again, so we need to be very sure to go one or another path.