From cf91e019bccd34220878f4292b72ef1880eadfbf Mon Sep 17 00:00:00 2001 From: LautaroPetaccio Date: Sat, 10 Oct 2026 21:13:32 -0300 Subject: [PATCH] AsyncAPI 3.x: the kill switch is off by default for message-driven services The kill switch stops SUT code that is still running once the coverage of a test has been collected. It assumes a request handled on a thread of its own. A message-driven service consumes on long-lived threads instead: the first line probe executed by a raw consumer loop inside that window throws, the thread dies, and every later message goes unanswered. A Spring Kafka listener container survives, but the records that arrive inside the window are retried while the switch is still on and then dropped. So far every AsyncAPI E2E had to pass --killSwitch false to get anything done. For AsyncAPI the switch is off unless the user set it explicitly. Two things make this less direct than flipping a default: - problemType may still be DEFAULT when the options are checked, and only become ASYNCAPI once resolved from the driver in Main. So the option itself is never changed; EMConfig.isKillSwitchOn() decides when asked, and that is what the core sends to the driver with every coverage fetch. - A boolean option has no sentinel value, so EMConfig keeps the options set on the command line or in the configuration file, as properties, accumulated over every updateProperties call. They were computed already, for the experimental-settings warning and the dependency checks, but after the multi-field checks and then discarded. RemoteControllerImplementation.getTestResults is the only place in the core that reads the option. Schedule tasks never send it. EMConfigTest covers the default and the explicit cases, with the problem type given on the command line, resolved later as Main does it, and the switch set in a configuration file. NcsKafkaEMTest drops --killSwitch false. No change for any other problem type: isKillSwitchOn() is the option itself there. --- .../spring/asyncapi/kafka/NcsKafkaEMTest.java | 14 +--- .../kotlin/org/evomaster/core/EMConfig.kt | 32 ++++++++- .../service/RemoteControllerImplementation.kt | 2 +- .../kotlin/org/evomaster/core/EMConfigTest.kt | 67 +++++++++++++++++++ core/src/test/resources/config/killswitch.yml | 2 + docs/options.md | 2 +- 6 files changed, 102 insertions(+), 17 deletions(-) create mode 100644 core/src/test/resources/config/killswitch.yml diff --git a/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java b/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java index 9943228af7..fb05312521 100644 --- a/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java +++ b/core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java @@ -57,14 +57,9 @@ public void testRunEM() throws Throwable { args.add("3000"); /* - The kill switch stops SUT code that is still running once an individual - has been evaluated. It assumes a request handled on its own thread: here - the service consumes on one long-lived thread, so killing it stops the - service for good, and every later message goes unanswered. Any - message-driven SUT has this shape. + The kill switch would stop the service's one consumer thread for good. It + is off by default for AsyncAPI; a missed reply would fail NO_REPLY below. */ - args.add("--killSwitch"); - args.add("false"); Solution solution = initAndRun(args); @@ -139,7 +134,6 @@ public void testRunEMInJava() throws Throwable { setOption(args, "outputFormat", "JAVA_JUNIT_5"); setOption(args, "asyncApiReplyTimeoutMs", "3000"); - setOption(args, "killSwitch", "false"); Solution solution = initAndRun(args); @@ -172,10 +166,6 @@ public void testAPromisedReplyThatNeverArrivesIsAFault() throws Throwable { args.add("--asyncApiReplyTimeoutMs"); args.add("1"); - //as in testRunEM: the kill switch would stop the consumer thread for good - args.add("--killSwitch"); - args.add("false"); - /* Both AsyncAPI categories are experimental, so without this the outcome is still reached and still a target, but nothing is reported as a fault. diff --git a/core/src/main/kotlin/org/evomaster/core/EMConfig.kt b/core/src/main/kotlin/org/evomaster/core/EMConfig.kt index cddeea58fe..bad828f0c5 100644 --- a/core/src/main/kotlin/org/evomaster/core/EMConfig.kt +++ b/core/src/main/kotlin/org/evomaster/core/EMConfig.kt @@ -303,6 +303,16 @@ class EMConfig { } } + /** + * The options the user set explicitly, on the command line or in the configuration + * file, over every call to [updateProperties]. Values assigned in code do not count. + * + * A boolean option has no DEFAULT sentinel like an enum, so this is how a default that + * depends on the problem type, which may only be resolved from the driver after the + * checks here, tells a value the user chose from one left alone. + */ + private val explicitlySetOptions: MutableSet> = mutableSetOf() + /** * Update the values of the properties based on the options * chosen on the command line @@ -338,14 +348,15 @@ class EMConfig { //why was this done for each updateProperty??? excludedTargetsForImpactCollection = extractExcludedTargetsForImpactCollection() + val modifiedOptions = modifiedOptions(options, cff) + explicitlySetOptions.addAll(properties.filter { it.name in modifiedOptions }) + checkMultiFieldConstraints() handleDeprecated() handleCreateConfigPathIfMissing(properties) - val modifiedOptions = modifiedOptions(options, cff) - checkForExperimentalSettings(modifiedOptions.keys) checkForInternalSettings(modifiedOptions.keys) @@ -1118,6 +1129,18 @@ class EMConfig { */ fun usesDriver() = !blackBox || bbExperiments || problemType == ProblemType.ASYNCAPI + /** + * Whether the driver should be asked to stop SUT code still running once the coverage + * of a test has been collected. Read this, not [killSwitch], when talking to the driver. + * + * Off for AsyncAPI unless the user set [killSwitch] explicitly: a message-driven service + * typically consumes on long-lived threads, and stopping one stops the service for good. + * Decided here rather than while checking the options, as [problemType] may only become + * ASYNCAPI once resolved from the driver. + */ + fun isKillSwitchOn() = killSwitch + && (problemType != ProblemType.ASYNCAPI || EMConfig::killSwitch in explicitlySetOptions) + /** * Whether a test can be written at all in the chosen format. * @@ -2948,7 +2971,10 @@ class EMConfig { @Cfg("Try to enforce the stopping of SUT business-level code." + " This is needed when TCP connections timeouts, to avoid thread executions" + - " from previous HTTP calls affecting the current one") + " from previous HTTP calls affecting the current one." + + " Off by default when testing an AsyncAPI service, unless set explicitly:" + + " a message-driven service typically consumes on long-lived threads," + + " which the kill switch would stop for good") var killSwitch = true @Cfg("Number of milliseconds we are going to wait to get a response on a TCP connection, e.g., " + diff --git a/core/src/main/kotlin/org/evomaster/core/remote/service/RemoteControllerImplementation.kt b/core/src/main/kotlin/org/evomaster/core/remote/service/RemoteControllerImplementation.kt index 8cc30d6d14..2eb24ca902 100644 --- a/core/src/main/kotlin/org/evomaster/core/remote/service/RemoteControllerImplementation.kt +++ b/core/src/main/kotlin/org/evomaster/core/remote/service/RemoteControllerImplementation.kt @@ -346,7 +346,7 @@ class RemoteControllerImplementation() : RemoteController{ val response = makeHttpCall { getWebTarget() .path(ControllerConstants.TEST_RESULTS) - .queryParam("killSwitch", !ignoreKillSwitch && config.killSwitch) + .queryParam("killSwitch", !ignoreKillSwitch && config.isKillSwitchOn()) .queryParam("fullyCovered", fullyCovered) .queryParam("descriptiveIds", descriptiveIds) .queryParam(QUERY_FROM_DATABASE, !config.useInsertionForSqlHeuristics) diff --git a/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt b/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt index c9f3d2ad0a..b5425f114e 100644 --- a/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt +++ b/core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt @@ -21,6 +21,7 @@ internal class EMConfigTest{ private val sqli = "sqli" private val blackBox = "blackBox" private val generateMongoData = "generateMongoData" + private val killSwitch = "killSwitch" @Test @@ -831,6 +832,72 @@ internal class EMConfigTest{ } } + @Test + fun testKillSwitchIsOffByDefaultForAsyncApi(){ + + fun configFor(vararg args: String) = + EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse(*args)) } + + val rest = configFor("--problemType", "REST") + assertTrue(rest.isKillSwitchOn(), "the kill switch is on by default for REST") + + /* + A message-driven service consumes on long-lived threads, which the kill switch + would stop for good. The option keeps its value: only what is asked of the driver + changes. + */ + val asyncApi = configFor("--problemType", "ASYNCAPI", "--createTests", "false") + assertTrue(asyncApi.killSwitch) + assertFalse(asyncApi.isKillSwitchOn(), "the kill switch should be off by default for AsyncAPI") + } + + @Test + fun testKillSwitchSetExplicitlyIsRespectedForAsyncApi(){ + + fun configFor(vararg args: String) = + EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse(*args)) } + + val on = configFor("--problemType", "ASYNCAPI", "--createTests", "false", "--$killSwitch", "true") + assertTrue(on.isKillSwitchOn(), "a kill switch turned on explicitly should stay on for AsyncAPI") + + val off = configFor("--problemType", "ASYNCAPI", "--createTests", "false", "--$killSwitch", "false") + assertFalse(off.isKillSwitchOn()) + + val restOff = configFor("--problemType", "REST", "--$killSwitch", "false") + assertFalse(restOff.isKillSwitchOn(), "turning the kill switch off explicitly should work for REST too") + + //a configuration file counts as explicit too + val fromFile = configFor("--problemType", "ASYNCAPI", "--createTests", "false", + "--configPath", "src/test/resources/config/killswitch.yml") + assertTrue(fromFile.isKillSwitchOn(), "a kill switch turned on in the configuration file should stay on") + } + + @Test + fun testKillSwitchFollowsAProblemTypeResolvedFromTheDriver(){ + + /* + In white-box mode, a problem type left at DEFAULT is resolved from the driver + only after the options have been checked, the way Main does it. + */ + val left = EMConfig() + left.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false", "--createTests", "false")) + assertEquals(EMConfig.ProblemType.DEFAULT, left.problemType) + left.problemType = EMConfig.ProblemType.ASYNCAPI + left.checkMultiFieldConstraints() + assertFalse(left.isKillSwitchOn(), "the kill switch should be off once the run turns out to be AsyncAPI") + + val forced = EMConfig() + forced.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false", "--createTests", "false", "--$killSwitch", "true")) + forced.problemType = EMConfig.ProblemType.ASYNCAPI + forced.checkMultiFieldConstraints() + assertTrue(forced.isKillSwitchOn(), "a kill switch turned on explicitly should survive the resolution") + + val rest = EMConfig() + rest.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false")) + rest.problemType = EMConfig.ProblemType.REST + rest.checkMultiFieldConstraints() + assertTrue(rest.isKillSwitchOn()) + } @Test fun testSetFalseOnDependsOn(){ diff --git a/core/src/test/resources/config/killswitch.yml b/core/src/test/resources/config/killswitch.yml new file mode 100644 index 0000000000..133ef6e3b5 --- /dev/null +++ b/core/src/test/resources/config/killswitch.yml @@ -0,0 +1,2 @@ +configs: + killSwitch: true diff --git a/docs/options.md b/docs/options.md index 25660a46af..faab8e133f 100644 --- a/docs/options.md +++ b/docs/options.md @@ -159,7 +159,7 @@ There are 3 types of options: |`jaCoCoPort`| __Int__. Port used by JaCoCo to export coverage reports. *Constraints*: `min=0.0, max=65535.0`. *Default value*: `8899`.| |`javaCommand`| __String__. Command for 'java' used in the External Drivers. Useful for when there are different JDK installed on same machine without the need to update JAVA_HOME. Note that this only impact the generated output test cases. *Default value*: `java`.| |`jsControllerPath`| __String__. When generating tests in JavaScript, there is the need to know where the driver is located in respect to the generated tests. *Default value*: `./app-driver.js`.| -|`killSwitch`| __Boolean__. Try to enforce the stopping of SUT business-level code. This is needed when TCP connections timeouts, to avoid thread executions from previous HTTP calls affecting the current one. *Default value*: `true`.| +|`killSwitch`| __Boolean__. Try to enforce the stopping of SUT business-level code. This is needed when TCP connections timeouts, to avoid thread executions from previous HTTP calls affecting the current one. Off by default when testing an AsyncAPI service, unless set explicitly: a message-driven service typically consumes on long-lived threads, which the kill switch would stop for good. *Default value*: `true`.| |`labelForExperimentConfigs`| __String__. Further label to represent the names of CONFIGS sets in experiment scripts, e.g., exp.py. *Default value*: `-`.| |`labelForExperiments`| __String__. When running experiments and statistic files are generated, all configs are saved. So, this one can be used as extra label for classifying the experiment. *Default value*: `-`.| |`lastLineEpsilon`| __Double__. The Distance Metric Last Line may use several values for epsilon.During experimentation, it may be useful to adjust these values. Epsilon describes the size of the neighbourhood used for clustering, so may result in different clustering results.Epsilon should be between 0.0 and 1.0. If the value is outside of that range, epsilon will use the default of 0.8. *Constraints*: `min=0.0, max=1.0`. *Default value*: `0.8`.|