Rework HA job queue entry claiming & add concurrent processing

[yhabteab]

This PR reworks the existing event queue processing logic to allow for more generic and concurrent processing of job queue entries. The existing logic was tightly coupled to the Event type and its processing, which made it difficult to extend the queue processing to other types of jobs in the future, such as Icinga Notifications quick actions. For this reason, the existing logic has been refactored to allow for more generic processing of job queue entries as discussed offline with @nilmerg and @BastianLedererIcinga.

First and foremost, the database queueing table is no longer called event_queue, but rather job_queue, and its Json column has been replaced with a versioned, typed envelope column, so the table can eventually carry different types of jobs in the future. As a consequence, the object_id and user_agent columns have been completely removed. Object identity is now computed at processing time based on the job type and its payload, and the user agent is replaced by the aforementioned typed envelope, which can carry any metadata that is relevant to the job type.

Now, since we no longer have object_id in the job queue table, we can no longer use it to claim and lock a job queue entry for processing. Instead, a new job_processing_lock table has been introduced, which is used exclusively by Icinga Notifications to claim and lock a job queue entry and its associated object for processing. The table's primary key object_id column, and a unique and not nullable job_queue_id referencing the job_queue table, provides the same guarantees as the previously used LEFT JOIN and object_id based locking mechanism. So, this inevitably leads to refactoring the existing job queue entry claiming and locking logic, and has now two main phases:

1. First, it plain selects a batch of job queue entries that are in pending state without any locking or claiming. Next, it iterates over that batch and tries to convert the job queue entry's envelope into a typed job (currently only Event). If the conversion fails, the job queue entry is marked as failed and skipped. If the conversion succeeds, it computes the object identity of the job and tries insert a row into the job_processing_lock table. If the insert succeeds, it then compares and updates the job queue entry's state to processing and streams the claimed job to a channel for concurrent processing. If the INSERT IGNORE affects zero rows, it means that the object is already being locked by another worker, so the tx is rolled back and the job queue entry is skipped. Though, the compare and update of the job queue entry's state to processing can fail too, if we race and lose to another worker that has just claimed the job queue entry. In that case, the tx is rolled back and the job queue entry is skipped as well. All of this, except the plain select of the batch of job queue entries, is done in a single tx without using SERIALIZABLE isolation level.
2. The second phase is the actual processing of the claimed job queue entries, which is done concurrently by a pool of workers (currently limited to 4). Each worker receives a claimed job queue entry from the channel and do the actual processing of the job. If the processing succeeds, the job queue entry is marked as completed, and the associated job_processing_lock row is deleted. If the processing fails, the job queue entry is marked as failed, and the associated job_processing_lock row is deleted as well. The only time that we might leave a locked job and its associated job_processing_lock row is when the worker panics or Icinga Notifications shuts down unexpectedly, in which case the existing retention.ResetPruner will eventually reset the job queue entry's state to pending and delete the associated job_processing_lock row, so the job can be retried later.

Example of the new envelope structure for an Event job queue entry:

{"version":1,"format":"event","payload":{"source_id":1,"id":"icinga-master2.devlab.com: ssh!1786712186500","name":"icinga-master2.devlab.com: ssh","url":"https://watchtower.devlab.com/icingaweb/icingadb/service?name=ssh\u0026host.name=icinga-master2.devlab.com","tags":{"environment":"0162e462daea7c02249e1995a7fff030c776380f","host":"icinga-master2.devlab.com","service":"ssh"},"severity":"warning","message":"connect to address icinga-master1.devlab.com and port 80: Connection refused HTTP CRITICAL - Unable to open TCP socket","muted":false,"muted_reason":"Checkable is not muted (no active downtime, no acknowledgement, and not flapping)","incident":true,"complete_relations":["object.type","host.name","host.display_name","services[*].name","services[*].display_name"],"relations":{"host":{"display_name":"icinga-master2.devlab.com","name":"icinga-master2.devlab.com"},"object":{"type":"service"},"services":[{"display_name":"ssh","name":"ssh"}]}}}
{"version":1,"format":"event","payload":{"source_id":1,"id":"icinga-master2.devlab.com: swap!1786712202311","name":"icinga-master2.devlab.com: swap","url":"https://watchtower.devlab.com/icingaweb/icingadb/service?name=swap\u0026host.name=icinga-master2.devlab.com","tags":{"environment":"0162e462daea7c02249e1995a7fff030c776380f","host":"icinga-master2.devlab.com","service":"swap"},"severity":"crit","message":"SWAP CRITICAL - 0% free (0MB out of 0MB) - Swap is either disabled, not present, or of zero size. ","muted":false,"muted_reason":"Checkable is not muted (no active downtime, no acknowledgement, and not flapping)","incident":true,"complete_relations":["object.type","host.name","host.display_name","services[*].name","services[*].display_name"],"relations":{"host":{"display_name":"icinga-master2.devlab.com","name":"icinga-master2.devlab.com"},"object":{"type":"service"},"services":[{"display_name":"swap","name":"swap"}]}}}

Why removing the object_id column from the job queue table?

In the previous implementation, the object_id was populated by the listener when the job queue entry was enqueued, and it was of course used to claim and lock the job queue entry for processing. However, having the object_id in the job queue table limits each and every job queue entry to be associated with a single object, and of course, it requires every client to pre-compute the object identity before enqueuing. Till now, this was not a problem because the only client that was enqueuing jobs, was the Icinga Notifications listener, which is aware of the object identity. However, with #450, Icinga Notifications Web will also be able to enqueue jobs for quick actions that are very different from the existing and more complex events. As opposed to events, quick actions can be batched for multiple objects/incidents, and having the object_id in the job queue table would not make sense anymore, because a single job queue entry can target multiple objects (not yet supported, but is planned for the future). So, Icinga Notifications Web will have to scan the object_id_tags table to determine the complete list of tags, compute the object ID for each object, and then enqueue a separate job queue entry for each of them. According to @nilmerg, this slows down the UI interactions dramatically, and wants to avoid in any way possible. Consequently, it was decided to remove the object_id column from the job queue table, and instead let the job enqueuer (whoever that is) to just provide the id tags in its payload, and let Icinga Notifications do the heavy lifting.

[yhabteab]

Added some basic enqueueing and dequeueing test cases for the new job queue processing logic now. The tests require #489 though.

[yhabteab]

This is the baseline for #493. I've tested both of them extensively and my dev setup is also using this branch and works fine.

PS: most of the code changes are new tests for job queue enqueueing and processing logic.