Initialize Configuration#Concurrency from /sys/fs/cgroup/... by Al2Klimov · Pull Request #7845 · Icinga/icinga2

Al2Klimov · 2020-02-18T16:36:57Z

... not to try to consume more hardware resources than the admin assigned.

Al2Klimov · 2020-02-18T17:36:03Z

$ docker run --rm -itv "$(pwd):/src" icinga2-dev
+ export PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export CCACHE_DIR=/src/docker-dev/ccache
+ CCACHE_DIR=/src/docker-dev/ccache
+ cd /src
+ mkdir -vp docker-dev/ccache
+ cd docker-dev
+ '[' -e build ']'
+ cd build
+ make -j2
(...)
+ make -j2 install
(...)
+ cd ../root
+ exec ./sbin/icinga2 daemon
[2020-02-18 17:32:04 +0000] information/cli: Icinga application loader (version: r2.11.2-1; debug)
[2020-02-18 17:32:04 +0000] information/cli: Loading configuration file(s).
[2020-02-18 17:32:05 +0000] information/config: 2
(...)
[2020-02-18 17:32:06 +0000] information/IcingaApplication: Icinga has shut down.
$ docker run --rm --cpuset-cpus=0 -itv "$(pwd):/src" icinga2-dev
+ export PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export CCACHE_DIR=/src/docker-dev/ccache
+ CCACHE_DIR=/src/docker-dev/ccache
+ cd /src
+ mkdir -vp docker-dev/ccache
+ cd docker-dev
+ '[' -e build ']'
+ cd build
+ make -j2
(...)
+ make -j2 install
(...)
+ cd ../root
+ exec ./sbin/icinga2 daemon
[2020-02-18 17:34:10 +0000] information/cli: Icinga application loader (version: r2.11.2-1; debug)
[2020-02-18 17:34:10 +0000] information/cli: Loading configuration file(s).
[2020-02-18 17:34:11 +0000] information/config: 1
(...)
[2020-02-18 17:34:43 +0000] information/IcingaApplication: Icinga has shut down.

Al2Klimov · 2020-08-24T09:25:20Z

TODO

handle --cpus

Al2Klimov · 2020-08-24T10:23:28Z

... e.g. --cpus 2.25 => /sys/fs/cgroup/cpu/cpu.cfs_quota_us = 225000, default -1.

Al2Klimov · 2020-08-24T13:00:09Z

MacOS

➜  icinga2 git:(feature/concurrency-docker-7842) prefix/sbin/icinga2 console
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
8.000000
<2> =>

CentOS

[centos@aklimov-7842 ~]$ icinga2 console
[2020-08-24 12:36:04 +0000] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed to adjust resource limit for open file handles (RLIMIT_NOFILE) with error "Operation not permitted"
[2020-08-24 12:36:04 +0000] warning/Application: Failed adjust resource limit for number of processes (RLIMIT_NPROC) with error "Operation not permitted"
Icinga 2 (version: v2.11.0-525-g666c307)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> =>

Docker

➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it icinga/icinga2 icinga2 console
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:56:02 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
4.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0,2 icinga/icinga2 icinga2 console
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:57:01 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpus 2.25 icinga/icinga2 icinga2 console
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:57:39 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0,2 --cpus 3 icinga/icinga2 icinga2 console
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:58:05 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000
<2> => exit(0)
➜  icinga2 git:(feature/concurrency-docker-7842) docker run --rm -it --cpuset-cpus 0-2 --cpus 2 icinga/icinga2 icinga2 console
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Initializing /data as we're the init process (PID 1)
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/etc/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/etc/icinga2" to "/data/etc/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/cache/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/cache/icinga2" to "/data/var/cache/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/lib/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/lib/icinga2" to "/data/var/lib/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/log/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/log/icinga2" to "/data/var/log/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/run/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/run/icinga2" to "/data/var/run/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/data/var/spool/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Copying "/data-init/var/spool/icinga2" to "/data/var/spool/icinga2"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Checking "/var/lib/icinga2/certs/ca.crt"
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Looking up "icinga2" in $PATH
[2020-08-24 12:58:22 +0000] information/DockerEntrypoint: Running "/usr/sbin/icinga2"
Icinga 2 (version: v2.11.0-524-g66b84be01)
Type $help to view available commands.
<1> => Concurrency
2.000000

Al2Klimov · 2021-08-04T12:22:57Z

@cla-bot check

julianbrost

Doesn't work with cgroup2, there the information is in /sys/fs/cgroup/cpu* files (i.e. not in extra subdirectories).

In general, an overview of how the Docker options map to cgroup attributes (and therefore which of these options this PR takes into account) would be nice. Looking at the corresponding systemd options is probably also interesting and they do properly document the cgroup options affected.

julianbrost · 2023-01-24T08:30:59Z

-int Configuration::Concurrency{static_cast<int>(std::thread::hardware_concurrency())};
+
+#ifdef __linux__
+static std::string ReadSysLine(const char *file)


This function looks overly complex for what it does. Shouldn't this basically be the same as std::ifstream + std::getline? The only thing I see that wouldn't be possible with that is the check for ENOENT. If this is just for kernel version compatibility, using an extra check would be fine. If this file can (dis)appear at runtime (changing the cgroup options, moving the process between cgroups), have a look at boost::iostreams like it's already used in AtomicFile.

julianbrost · 2023-01-24T08:33:05Z

+			return "";
+		}
+
+		throw system_error(error_code(errno, system_category()));


Do we really want to fail starting in this case? I think logging a warning and continuing with the value from std::thread::hardware_concurrency() would be a better option.

In "this case" a kernel subsystem(!) is out of service. The best reason to panic.

Logging? We're in a static constructor.

Al2Klimov · 2023-01-25T17:05:51Z

Shall we still support v1?
Not sure what for. Icinga 2 itself has only /sys and no idea what option caused the status quo.

julianbrost · 2023-01-26T13:38:06Z

v1 isn't dead yet, just checked on Ubuntu 20.04 with Docker 20.10.22 and that's still using that. So still supporting it wouldn't be wasted. But in general, I see a huge benefit in this PR just by replacing all hard-coded instances of hardware_concurrency with the config option. Better detection for the default value is more like a bonus.
I've mentioned the systemd documentation because it most likely affects the same cgroup settings as Docker, but in contrast to the Docker documentation, it provides a list of options to limit CPU usage including an explanation which cgroup settings this affects in v1 and v2.

Al2Klimov · 2023-04-05T09:41:34Z

TODO @Al2Klimov

decouple from Always use Configuration#Concurrency, not std::thread::hardware_concurrency() #9643
rebase
await review
test

Al2Klimov · 2023-05-26T12:41:34Z

Doesn’t whoever controls the container spec also control the command it runs?

I.e. can't they just set -DConfiguration.Concurrency= in addition to --cpu-whatever?

I.e. do we need this PR?

julianbrost · 2023-05-26T12:58:10Z

Doesn’t whoever controls the container spec also control the command it runs?

I.e. can't they just set -DConfiguration.Concurrency= in addition to --cpu-whatever?

Most likely yes.

I.e. do we need this PR?

Not strictly needed as it can be worked around. However, the user has to be aware of this and it would be nice if Icinga would just do the correct thing by itself.

…t.cpus refs #7842

….cfs_quota_us refs #7842

…cpus.effective

Al2Klimov self-assigned this Feb 18, 2020

Al2Klimov removed their assignment Feb 18, 2020

Al2Klimov marked this pull request as ready for review February 18, 2020 17:36

Al2Klimov added this to the 2.13.0 milestone Mar 17, 2020

Al2Klimov marked this pull request as draft August 24, 2020 09:25

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 5793b4e to 66b84be Compare August 24, 2020 12:17

Al2Klimov changed the title ~~Initialize Configuration#Concurrency from /sys/fs/cgroup/cpuset/cpuset.cpus~~ Initialize Configuration#Concurrency from /sys/fs/cgroup/... Aug 24, 2020

Al2Klimov marked this pull request as ready for review August 24, 2020 13:00

julianbrost modified the milestones: 2.13.0, 2.14.0 May 31, 2021

cla-bot Bot added the cla/signed label Aug 4, 2021

Al2Klimov added the enhancement New feature or request label Aug 10, 2021

Al2Klimov requested a review from Thomas-Gelf November 23, 2021 15:08

Al2Klimov mentioned this pull request Jan 3, 2023

Icinga's initial threads are linked to core count in host with no way to adjust or limit #9147

Closed

julianbrost requested changes Jan 24, 2023

View reviewed changes

Al2Klimov mentioned this pull request Jan 27, 2023

Always use Configuration#Concurrency, not std::thread::hardware_concurrency() #9643

Merged

Al2Klimov self-assigned this Jan 27, 2023

Al2Klimov marked this pull request as draft January 27, 2023 14:38

Al2Klimov removed their assignment Jan 31, 2023

lippserd removed the request for review from Thomas-Gelf March 30, 2023 09:30

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 66b84be to 1d6ab00 Compare April 5, 2023 09:41

Al2Klimov requested a review from julianbrost April 5, 2023 09:43

Al2Klimov removed this from the 2.14.0 milestone May 5, 2023

Al2Klimov removed the request for review from julianbrost June 6, 2023 15:39

Al2Klimov self-assigned this Jun 6, 2023

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 1d6ab00 to cda810d Compare July 26, 2023 10:02

Al2Klimov requested a review from julianbrost August 8, 2023 16:17

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from cda810d to 26c8c91 Compare January 31, 2025 16:00

Al2Klimov marked this pull request as ready for review February 3, 2025 16:33

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from 26c8c91 to dde5e35 Compare April 30, 2025 15:33

Al2Klimov added 3 commits August 28, 2026 17:12

Initialize Configuration#Concurrency from /sys/fs/cgroup/cpuset/cpuse…

13ec5da

…t.cpus refs #7842

Initialize Configuration#Concurrency also from /sys/fs/cgroup/cpu/cpu…

2986776

….cfs_quota_us refs #7842

Initialize Configuration#Concurrency also from /sys/fs/cgroup/cpuset.…

7e9bbc7

…cpus.effective

Al2Klimov force-pushed the feature/concurrency-docker-7842 branch from dde5e35 to 7e9bbc7 Compare August 28, 2026 15:12

Uh oh!

Conversation

Al2Klimov commented Feb 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Al2Klimov commented Feb 18, 2020

Uh oh!

Al2Klimov commented Aug 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Uh oh!

Al2Klimov commented Aug 24, 2020

Uh oh!

Al2Klimov commented Aug 24, 2020

MacOS

CentOS

Docker

Uh oh!

Al2Klimov commented Aug 4, 2021

Uh oh!

julianbrost left a comment

Choose a reason for hiding this comment

Uh oh!

julianbrost Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

julianbrost Jan 24, 2023

Choose a reason for hiding this comment

Uh oh!

Al2Klimov Jan 27, 2023

Choose a reason for hiding this comment

Uh oh!

Al2Klimov commented Jan 25, 2023

Uh oh!

julianbrost commented Jan 26, 2023

Uh oh!

Al2Klimov commented Apr 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO @Al2Klimov

Uh oh!

Al2Klimov commented May 26, 2023

Uh oh!

julianbrost commented May 26, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Al2Klimov commented Feb 18, 2020 •

edited

Loading

Al2Klimov commented Aug 24, 2020 •

edited

Loading

Al2Klimov commented Apr 5, 2023 •

edited

Loading