AsyncAPI 3.x: the kill switch is off by default for message-driven services

core-tests/e2e-tests/spring/spring-asyncapi-kafka/src/test/java/org/evomaster/e2etests/spring/asyncapi/kafka/NcsKafkaEMTest.java

@@ -57,14 +57,9 @@ public void testRunEM() throws Throwable {
                     args.add("3000");
 
                     /*
-                        The kill switch stops SUT code that is still running once an individual
-                        has been evaluated. It assumes a request handled on its own thread: here
-                        the service consumes on one long-lived thread, so killing it stops the
-                        service for good, and every later message goes unanswered. Any
-                        message-driven SUT has this shape.
+                        The kill switch would stop the service's one consumer thread for good. It
+                        is off by default for AsyncAPI; a missed reply would fail NO_REPLY below.
                      */
-                    args.add("--killSwitch");
-                    args.add("false");
 
                     Solution<AsyncApiIndividual> solution = initAndRun(args);
 
@@ -139,7 +134,6 @@ public void testRunEMInJava() throws Throwable {
 
                     setOption(args, "outputFormat", "JAVA_JUNIT_5");
                     setOption(args, "asyncApiReplyTimeoutMs", "3000");
-                    setOption(args, "killSwitch", "false");
 
                     Solution<AsyncApiIndividual> solution = initAndRun(args);
 
@@ -172,10 +166,6 @@ public void testAPromisedReplyThatNeverArrivesIsAFault() throws Throwable {
                     args.add("--asyncApiReplyTimeoutMs");
                     args.add("1");
 
-                    //as in testRunEM: the kill switch would stop the consumer thread for good
-                    args.add("--killSwitch");
-                    args.add("false");
-
                     /*
                         Both AsyncAPI categories are experimental, so without this the outcome is
                         still reached and still a target, but nothing is reported as a fault.

core/src/main/kotlin/org/evomaster/core/EMConfig.kt

@@ -303,6 +303,16 @@ class EMConfig {
         }
     }
 
+    /**
+     * The options the user set explicitly, on the command line or in the configuration
+     * file, over every call to [updateProperties]. Values assigned in code do not count.
+     *
+     * A boolean option has no DEFAULT sentinel like an enum, so this is how a default that
+     * depends on the problem type, which may only be resolved from the driver after the
+     * checks here, tells a value the user chose from one left alone.
+     */
+    private val explicitlySetOptions: MutableSet<KMutableProperty<*>> = mutableSetOf()
+
     /**
      * Update the values of the properties based on the options
      * chosen on the command line
@@ -338,14 +348,15 @@ class EMConfig {
         //why was this done for each updateProperty???
         excludedTargetsForImpactCollection = extractExcludedTargetsForImpactCollection()
 
+        val modifiedOptions = modifiedOptions(options, cff)
+        explicitlySetOptions.addAll(properties.filter { it.name in modifiedOptions })
+
         checkMultiFieldConstraints()
 
         handleDeprecated()
 
         handleCreateConfigPathIfMissing(properties)
 
-        val modifiedOptions = modifiedOptions(options, cff)
-
         checkForExperimentalSettings(modifiedOptions.keys)
         checkForInternalSettings(modifiedOptions.keys)
 
@@ -1118,6 +1129,18 @@ class EMConfig {
      */
     fun usesDriver() = !blackBox || bbExperiments || problemType == ProblemType.ASYNCAPI
 
+    /**
+     * Whether the driver should be asked to stop SUT code still running once the coverage
+     * of a test has been collected. Read this, not [killSwitch], when talking to the driver.
+     *
+     * Off for AsyncAPI unless the user set [killSwitch] explicitly: a message-driven service
+     * typically consumes on long-lived threads, and stopping one stops the service for good.
+     * Decided here rather than while checking the options, as [problemType] may only become
+     * ASYNCAPI once resolved from the driver.
+     */
+    fun isKillSwitchOn() = killSwitch
+            && (problemType != ProblemType.ASYNCAPI || EMConfig::killSwitch in explicitlySetOptions)
+
     /**
      * Whether a test can be written at all in the chosen format.
      *
@@ -2948,7 +2971,10 @@ class EMConfig {
 
     @Cfg("Try to enforce the stopping of SUT business-level code." +
             " This is needed when TCP connections timeouts, to avoid thread executions" +
-            " from previous HTTP calls affecting the current one")
+            " from previous HTTP calls affecting the current one." +
+            " Off by default when testing an AsyncAPI service, unless set explicitly:" +
+            " a message-driven service typically consumes on long-lived threads," +
+            " which the kill switch would stop for good")
     var killSwitch = true
 
     @Cfg("Number of milliseconds we are going to wait to get a response on a TCP connection, e.g., " +

core/src/main/kotlin/org/evomaster/core/remote/service/RemoteControllerImplementation.kt

@@ -346,7 +346,7 @@ class RemoteControllerImplementation() : RemoteController{
         val response = makeHttpCall {
             getWebTarget()
                     .path(ControllerConstants.TEST_RESULTS)
-                    .queryParam("killSwitch", !ignoreKillSwitch && config.killSwitch)
+                    .queryParam("killSwitch", !ignoreKillSwitch && config.isKillSwitchOn())
                     .queryParam("fullyCovered", fullyCovered)
                     .queryParam("descriptiveIds", descriptiveIds)
                     .queryParam(QUERY_FROM_DATABASE, !config.useInsertionForSqlHeuristics)

core/src/test/kotlin/org/evomaster/core/EMConfigTest.kt

@@ -21,6 +21,7 @@ internal class EMConfigTest{
     private val sqli = "sqli"
     private val blackBox = "blackBox"
     private val generateMongoData = "generateMongoData"
+    private val killSwitch = "killSwitch"
 
 
     @Test
@@ -831,6 +832,72 @@ internal class EMConfigTest{
         }
     }
 
+    @Test
+    fun testKillSwitchIsOffByDefaultForAsyncApi(){
+
+        fun configFor(vararg args: String) =
+            EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse(*args)) }
+
+        val rest = configFor("--problemType", "REST")
+        assertTrue(rest.isKillSwitchOn(), "the kill switch is on by default for REST")
+
+        /*
+            A message-driven service consumes on long-lived threads, which the kill switch
+            would stop for good. The option keeps its value: only what is asked of the driver
+            changes.
+         */
+        val asyncApi = configFor("--problemType", "ASYNCAPI", "--createTests", "false")
+        assertTrue(asyncApi.killSwitch)
+        assertFalse(asyncApi.isKillSwitchOn(), "the kill switch should be off by default for AsyncAPI")
+    }
+
+    @Test
+    fun testKillSwitchSetExplicitlyIsRespectedForAsyncApi(){
+
+        fun configFor(vararg args: String) =
+            EMConfig().apply { updateProperties(EMConfig.getOptionParser().parse(*args)) }
+
+        val on = configFor("--problemType", "ASYNCAPI", "--createTests", "false", "--$killSwitch", "true")
+        assertTrue(on.isKillSwitchOn(), "a kill switch turned on explicitly should stay on for AsyncAPI")
+
+        val off = configFor("--problemType", "ASYNCAPI", "--createTests", "false", "--$killSwitch", "false")
+        assertFalse(off.isKillSwitchOn())
+
+        val restOff = configFor("--problemType", "REST", "--$killSwitch", "false")
+        assertFalse(restOff.isKillSwitchOn(), "turning the kill switch off explicitly should work for REST too")
+
+        //a configuration file counts as explicit too
+        val fromFile = configFor("--problemType", "ASYNCAPI", "--createTests", "false",
+            "--configPath", "src/test/resources/config/killswitch.yml")
+        assertTrue(fromFile.isKillSwitchOn(), "a kill switch turned on in the configuration file should stay on")
+    }
+
+    @Test
+    fun testKillSwitchFollowsAProblemTypeResolvedFromTheDriver(){
+
+        /*
+            In white-box mode, a problem type left at DEFAULT is resolved from the driver
+            only after the options have been checked, the way Main does it.
+         */
+        val left = EMConfig()
+        left.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false", "--createTests", "false"))
+        assertEquals(EMConfig.ProblemType.DEFAULT, left.problemType)
+        left.problemType = EMConfig.ProblemType.ASYNCAPI
+        left.checkMultiFieldConstraints()
+        assertFalse(left.isKillSwitchOn(), "the kill switch should be off once the run turns out to be AsyncAPI")
+
+        val forced = EMConfig()
+        forced.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false", "--createTests", "false", "--$killSwitch", "true"))
+        forced.problemType = EMConfig.ProblemType.ASYNCAPI
+        forced.checkMultiFieldConstraints()
+        assertTrue(forced.isKillSwitchOn(), "a kill switch turned on explicitly should survive the resolution")
+
+        val rest = EMConfig()
+        rest.updateProperties(EMConfig.getOptionParser().parse("--$blackBox", "false"))
+        rest.problemType = EMConfig.ProblemType.REST
+        rest.checkMultiFieldConstraints()
+        assertTrue(rest.isKillSwitchOn())
+    }
 
     @Test
     fun testSetFalseOnDependsOn(){

core/src/test/resources/config/killswitch.yml

@@ -0,0 +1,2 @@
+configs:
+  killSwitch: true

docs/options.md

@@ -159,7 +159,7 @@ There are 3 types of options:
 |`jaCoCoPort`| __Int__. Port used by JaCoCo to export coverage reports. *Constraints*: `min=0.0, max=65535.0`. *Default value*: `8899`.|
 |`javaCommand`| __String__. Command for 'java' used in the External Drivers. Useful for when there are different JDK installed on same machine without the need to update JAVA_HOME. Note that this only impact the generated output test cases. *Default value*: `java`.|
 |`jsControllerPath`| __String__. When generating tests in JavaScript, there is the need to know where the driver is located in respect to the generated tests. *Default value*: `./app-driver.js`.|
-|`killSwitch`| __Boolean__. Try to enforce the stopping of SUT business-level code. This is needed when TCP connections timeouts, to avoid thread executions from previous HTTP calls affecting the current one. *Default value*: `true`.|
+|`killSwitch`| __Boolean__. Try to enforce the stopping of SUT business-level code. This is needed when TCP connections timeouts, to avoid thread executions from previous HTTP calls affecting the current one. Off by default when testing an AsyncAPI service, unless set explicitly: a message-driven service typically consumes on long-lived threads, which the kill switch would stop for good. *Default value*: `true`.|
 |`labelForExperimentConfigs`| __String__. Further label to represent the names of CONFIGS sets in experiment scripts, e.g., exp.py. *Default value*: `-`.|
 |`labelForExperiments`| __String__. When running experiments and statistic files are generated, all configs are saved. So, this one can be used as extra label for classifying the experiment. *Default value*: `-`.|
 |`lastLineEpsilon`| __Double__. The Distance Metric Last Line may use several values for epsilon.During experimentation, it may be useful to adjust these values. Epsilon describes the size of the neighbourhood used for clustering, so may result in different clustering results.Epsilon should be between 0.0 and 1.0. If the value is outside of that range, epsilon will use the default of 0.8. *Constraints*: `min=0.0, max=1.0`. *Default value*: `0.8`.|