Process monitor refactor - #392
Conversation
Introduce Run Target handling in refactoring * Run Target exploration * Small improvements * Restore table format * Make active flag atomic Proposal for component event queue * Proposal for central event queue * Introduce MPSC queue for event queue * Make use of concurrency error domain * Remove atomics/mutexes that now obsolete * Apply review findings * Move doxyen above the line * Remove blocking push behavior * Change queue push API * EventQueue only accepting r-value refs * Fixes after rebase Fixes after rebase Fix transition to Off states during shutdown * Fix transition to Off * Fix transition to off v2 * Fix test flakyness * RunTarget optimization * Fix copyright Fix race condition for self-term proc Integrate recoveryclient with event queue * Integrate recoveryclient with event queue * Fix include header Fix build with new config Fix concurrency build file Fix compiler errors & warnings Fix component event queue Separate transition bookkeeping from DependencyGraph topology * poc * Improve the transition design * Rollout to production code * Improve documentation * Remove graph poc folder * Improvements and more testing * fix compiler errors * Traverse whole graph on deactivate * Remove initial transition special case * Remove special case for Off * Improve documentation * Cleanup * Remove unneeded file * Cleanup documentation * Remove private inheritance * Reduce component states to minimum * Remove redundant checks * Replace component state with flag * Simply tryReportState method
License Check Results🚀 The license check job ran with the Bazel command: bazel run --lockfile_mode=error //:license-checkStatus: Click to expand output |
|
The created documentation from the pull request is available at: docu-html |
| sigterm = static_cast<uint8_t>(score::lcm::ProcessState::kTerminating), | ||
| off = static_cast<uint8_t>(score::lcm::ProcessState::kTerminated)}; | ||
| off = static_cast<uint8_t>(score::lcm::ProcessState::kTerminated), | ||
| failed = static_cast<uint8_t>(score::lcm::ProcessState::kFailed)}; |
There was a problem hiding this comment.
I see that failed state is currently not reachable from Running state.
However, to be safe maybe we should nonetheless make sure the AliveSupervision treats a failed process state the same as off or sigterm
score/launch_manager/src/daemon/src/alive_monitor/details/supervision/Alive.cpp:
if (std::holds_alternative<ProcessStateSnapshot>(f_updateEvent))
{
const auto& snapshot = std::get<ProcessStateSnapshot>(f_updateEvent);
if (snapshot.eProcState == ifexm::ProcessState::EProcState::running)
{
return EUpdateEventType::kActivation;
}
if (snapshot.eProcState == ifexm::ProcessState::EProcState::sigterm ||
snapshot.eProcState == ifexm::ProcessState::EProcState::off ||
+ snapshot.eProcState == ifexm::ProcessState::EProcState::failed)
{
return EUpdateEventType::kDeactivation;
}
return EUpdateEventType::kNoChange;
}| ITerminationCallback(ITerminationCallback&&) = default; | ||
| ITerminationCallback& operator=(ITerminationCallback&&) = default; | ||
| }; | ||
| using namespace score::mw::lifecycle::internal; |
There was a problem hiding this comment.
I wonder if we should remove any "using namespace" from header files.
Anything that includes the header transitively pulls every name from score::mw::lifecycle::internal into score::lcm::internal scope, which could result in unexpected naming collisions.
Though this exists in several places: process_info_node.hpp:37, transition.hpp:34, component_of.hpp:25, dependency_graph.hpp:26, graph.hpp:47
It could be done in a separate PR to clean this up.
| return nodes.capacity(); | ||
| } | ||
|
|
||
| T& operator[](GraphIndex index) |
There was a problem hiding this comment.
should we add a const overload?
| } | ||
|
|
||
| /// @brief Traverse the graph, starting at @p start, performing @p per_node on each node and moving to the nodes | ||
| /// provided by the return value from @p per_node. Only nodes such that @c filter(node) is true are traversed. |
There was a problem hiding this comment.
nitpick: The provided start node is traversed whether the filter for this node is true or not
| /// provided by the return value from @p per_node. Only nodes such that @c filter(node) is true are traversed. | ||
| /// Nodes are visited at most once. | ||
| template <typename PerNodeFn, typename FilterFn> | ||
| void traverse(const GraphIndex start, PerNodeFn per_node, FilterFn filter) |
There was a problem hiding this comment.
Maybe the filter could be optional or removed entirely, as it is currently not being used by the code.
|
|
||
| public: | ||
| /// @param count The exact number of nodes that will be added. | ||
| DependencyGraph(std::size_t count) : traversal_queue(std::max(count, std::size_t(2)) - 1) |
There was a problem hiding this comment.
Maybe a comment why the queue size is determined like this would be good
| { | ||
| nodes[node].depends_on.push_back(depends_on); | ||
| nodes[depends_on].dependents.push_back(node); | ||
| } |
There was a problem hiding this comment.
I wonder if an assert makes sense that we are not pushing dependencies beyond the capacity.
At lest for now, I think this is the expectation.
danth
left a comment
There was a problem hiding this comment.
Various nitpicks and ramblings. I have seen all the code but did not check the unit tests in much detail yet.
| while (!traversal_queue.empty()) | ||
| { | ||
| const auto pop_res = traversal_queue.tryPop(); | ||
| SCORE_LANGUAGE_FUTURECPP_ASSERT_PRD_MESSAGE( |
There was a problem hiding this comment.
Why do some checks use PRD and some not?
| } | ||
| }; | ||
|
|
||
| /// @returns Iterator at the beginning of the nodes store. |
There was a problem hiding this comment.
I think it's @return not @returns, same below
There was a problem hiding this comment.
A few of these methods have docs for parameters which do not exist
| /// provided by the return value from @p per_node. Only nodes such that @c filter(node) is true are traversed. | ||
| /// Nodes are visited at most once. | ||
| template <typename PerNodeFn, typename FilterFn> | ||
| void traverse(const GraphIndex start, PerNodeFn per_node, FilterFn filter) |
| report_state_lambda, | ||
| pgm_->getProcessInterface(), | ||
| pgm_->getProcessMap()); | ||
| assert(index == process_id && "Graph indicies must line up with os process indices"); |
There was a problem hiding this comment.
Use the SCORE_LANGUAGE_FUTURECPP_ assertions
| { | ||
| if (0 >= --nodes_in_flight_) | ||
| auto from_state = getState(); | ||
| if (from_state < GraphState::kAborting) |
There was a problem hiding this comment.
Not sure if I like relying on the numeric value of the enum for comparisons. Feel free to argue
| break; | ||
| } | ||
| else | ||
| assert(desired_state.has_value() && "Ready condition is not implemented"); |
There was a problem hiding this comment.
If this must be implemented for every possible ready condition, removing the default case would let the compiler emit a warning if one is missing. Since we treat warnings as errors it would therefore fail to build.
Otherwise: Use SCORE_LANGUAGE_FUTURECPP_ assertions rather than normal assert
| } | ||
|
|
||
| void ProcessInfoNode::startProcess() | ||
| IComponent::RequestResult ProcessInfoNode::startProcess(score::cpp::stop_token stop_token) |
There was a problem hiding this comment.
Same thing with the SCORE_LANGUAGE_FUTURECPP_ assertions in this method 😄
| /// @details When transitioning to a target state, the Transition computes the nodes that need to be | ||
| /// deactivated (those nodes not reachable from the target state) and those that need to be activated (those reachable | ||
| /// from the target that are not yet active). This list is then used to drive the transition by iterating over the ready | ||
| /// nodes (@ref nextReady()) and marking them as finished once activated/deactivated (@ref onNodeFinished()). The | ||
| /// transition is split into two phases: Stopping phase and Starting phase. Stopping Phase: Every node that is currently | ||
| /// running (@ref stopped() is false) and is not part of the target subgraph is deactivated. The stop set is derived | ||
| /// from live component state across the whole graph rather than from a source subgraph, so it also captures nodes left | ||
| /// running by a previously aborted transition. Once all nodes in the Stopping phase are finished, the transition moves | ||
| /// to the Starting phase. Starting Phase: The nodes that need to be activated are processed next. Once all nodes in the | ||
| /// Starting phase are finished, the transition is complete. | ||
| /// |
There was a problem hiding this comment.
Could we break up this text into paragraphs?
| void beginSetup(GraphIndex target) | ||
| { | ||
| state_.target_root = target; | ||
| clearNextNodes(); | ||
| state_.pending = 0; | ||
| state_.phase = Phase::Stopping; | ||
| } |
There was a problem hiding this comment.
This beginSetup method is currently only used for setupDeactivation, I think the _target_root and phase lines could be removed, and done directly by the caller.
Then it could be called within setupActivation / setupDeactivation so the caller does not need to do additional setup first.
The std::fill(false) which is duplicated across both methods could also be moved into beginSetup.
The description below is not AI generated. It includes some key information needed to begin a review.
Overview
This is the completion of the refactor started in #370. With these changes, the following is achieved:
IComponent. This means that different types of component can now be implemented, such as containers.Graphnow processes events sequentially rather than in parallel. SeeComponentEventQueue.ProcessInfoNodeis no longer responsible for dependency resolution. This responsibility is moved toTransition.ProcessInfoNode&Graphcoupling resolved -ProcessInfoNodeno longer has access toGraph.New components
DependencyGraph- Data structure for storing and traversing component dependenciesTransition- Resolves run target activation logic. Activates and deactivates components as required.Behavioral change
Processes are now managed based on component state, rather than process state. This means that once a component is activated, the underlying process shall not be started again until the component becomes inactive (either due to an error or a run target activation). Previously, a self-terminating process would be restarted on every run target activation that included it.
Minor changes
kLaunchManagercomms type removed. This was legacy and no longer needed.WIP (while this PR is a draft)
Graph&ProcessInfoNodeReview strategy
As this PR is quite large, it may be necessary for reviewers each to tackle subsections of the changes. In terms of line numbers, these sections are roughly equal:
graph.cpp+graph.hpp(+ unit tests)processinfonode.cpp+processinfonode.hpp(+ unit tests)transition.hpp+transition_UT.hppdependency_graph.hpp+dependency_graph_UT.cpp+process_group_manager.cpp+process_group_manager.hpp