From b9805b7576060ba326eddaf14fcc591b0b4ebf5b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 02:15:02 +0300 Subject: [PATCH 01/16] Update docs to suggest optional version pinning --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index a1577be..10dc11a 100644 --- a/README.md +++ b/README.md @@ -67,14 +67,13 @@ For the Helm chart option, use the OCI release below. This requires permission t ```bash helm upgrade --install autofission \ oci://ghcr.io/pomponchik/charts/autofission \ - --version VERSION \ --namespace fission \ --create-namespace \ --atomic \ --wait ``` -Replace `VERSION` with a published Autofission version. The Python package and Helm chart use the same version number. +> ⓘ For reproducible deployments, pin a published chart version by adding `--version VERSION`. Function Pods managed by Autofission must use a low, non-preempting PriorityClass. The Helm chart creates `autofission-runtime`; configure Fission to apply it to runtime Pods: From 9fd70d239b484474a6cf0a88c7ccc62c22ca61ff Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 13:18:08 +0300 Subject: [PATCH 02/16] Clarify rationale for low PriorityClass on Function Pods --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 10dc11a..6542e49 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,9 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, pin a published chart version by adding `--version VERSION`. -Function Pods managed by Autofission must use a low, non-preempting PriorityClass. The Helm chart creates `autofission-runtime`; configure Fission to apply it to runtime Pods: +Function Pods are opportunistic workloads and must yield cluster capacity to regular services. The Autofission Helm chart creates a PriorityClass named `autofission-runtime`. The class gives Function Pods a lower priority than ordinary Pods and prevents them from preempting other workloads. Higher-priority services can therefore reclaim their resources when necessary. + +Fission creates the Function Pods, so it must be configured to assign this PriorityClass to them: ```yaml runtimePodSpec: From 448c451167453928b3d025ee322d4dea6f6f52b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 13:19:33 +0300 Subject: [PATCH 03/16] Clarify version pinning instructions in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6542e49..9045acb 100644 --- a/README.md +++ b/README.md @@ -73,7 +73,7 @@ helm upgrade --install autofission \ --wait ``` -> ⓘ For reproducible deployments, pin a published chart version by adding `--version VERSION`. +> ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. Function Pods are opportunistic workloads and must yield cluster capacity to regular services. The Autofission Helm chart creates a PriorityClass named `autofission-runtime`. The class gives Function Pods a lower priority than ordinary Pods and prevents them from preempting other workloads. Higher-priority services can therefore reclaim their resources when necessary. From be455d143872ea0616c91a60b62ede1927484c44 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 13:22:14 +0300 Subject: [PATCH 04/16] Update runtime PriorityClass description in README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9045acb..bd3489d 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. -Function Pods are opportunistic workloads and must yield cluster capacity to regular services. The Autofission Helm chart creates a PriorityClass named `autofission-runtime`. The class gives Function Pods a lower priority than ordinary Pods and prevents them from preempting other workloads. Higher-priority services can therefore reclaim their resources when necessary. +Functions managed by Autofission are meant to use only spare cluster capacity, so their Pods need a lower Kubernetes priority than regular services. The Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime` for this purpose. Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. Fission creates the Function Pods, so it must be configured to assign this PriorityClass to them: From 70a8538daf23b04960cf8babe9218c3389c2d368 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 13:26:43 +0300 Subject: [PATCH 05/16] Update README PriorityClass explanation --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index bd3489d..0e7ccda 100644 --- a/README.md +++ b/README.md @@ -75,9 +75,11 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. -Functions managed by Autofission are meant to use only spare cluster capacity, so their Pods need a lower Kubernetes priority than regular services. The Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime` for this purpose. Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. +Functions managed by Autofission are meant to use only spare cluster capacity. They should fill resources left idle by regular services, then give those resources back when the cluster needs them for something more important. -Fission creates the Function Pods, so it must be configured to assign this PriorityClass to them: +Kubernetes represents this relationship with PriorityClasses. The Autofission Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime`. Function Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. + +Fission creates the Function Pods, so this setting belongs to Fission rather than Autofission. Add the following block to the Helm values file used for your Fission installation, then apply that file with the same `helm upgrade` command you use to manage Fission: ```yaml runtimePodSpec: From 8d94f4d536a9fdd280bdb15aebc4ddb45ec1a697 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 15:21:32 +0300 Subject: [PATCH 06/16] Add runtime priority class validation for managed Functions --- autofission/cli.py | 6 ++ autofission/controller.py | 91 ++++++++++++++++++- deploy/fission-values.yaml | 5 + deploy/helm/autofission/templates/NOTES.txt | 22 +++-- .../autofission/templates/deployment.yaml | 2 +- tests/helpers.py | 6 +- tests/smokes/test_package.py | 14 +++ tests/units/test_cli.py | 8 ++ tests/units/test_controller.py | 39 ++++++++ 9 files changed, 182 insertions(+), 11 deletions(-) create mode 100644 deploy/fission-values.yaml diff --git a/autofission/cli.py b/autofission/cli.py index b66c806..2d4cca3 100644 --- a/autofission/cli.py +++ b/autofission/cli.py @@ -125,6 +125,11 @@ def build_parser() -> argparse.ArgumentParser: '--managed-value', default=_env('MANAGED_VALUE', MANAGED_VALUE), ) + parser.add_argument( + '--runtime-priority-class', + default=_env('RUNTIME_PRIORITY_CLASS', ''), + help='expected PriorityClass for existing managed Function Pods', + ) tainted_nodes = parser.add_mutually_exclusive_group() tainted_nodes.add_argument( '--include-tainted-nodes', @@ -187,6 +192,7 @@ def main(argv: Sequence[str] | None = None) -> int: managed_label=arguments.managed_label, managed_value=arguments.managed_value, include_tainted_nodes=arguments.include_tainted_nodes, + runtime_priority_class=arguments.runtime_priority_class or None, ) gateway = _create_gateway( kubeconfig=arguments.kubeconfig, diff --git a/autofission/controller.py b/autofission/controller.py index d73dc2c..be59672 100644 --- a/autofission/controller.py +++ b/autofission/controller.py @@ -3,6 +3,7 @@ from __future__ import annotations import logging +from collections import defaultdict from collections.abc import Mapping from dataclasses import dataclass, field from typing import Mapping as TypingMapping @@ -22,6 +23,8 @@ MANAGED_VALUE = 'true' ANNOTATION_PREFIX = 'autoscaling.fission.io' INT32_MAX = 2**31 - 1 +_TERMINAL_POD_PHASES = frozenset({'Succeeded', 'Failed'}) +_FUNCTION_UID_LABEL = 'functionUid' class KubernetesGateway(Protocol): @@ -53,6 +56,7 @@ class ControllerConfig: managed_label: str = MANAGED_LABEL managed_value: str = MANAGED_VALUE include_tainted_nodes: bool = False + runtime_priority_class: str | None = None def __post_init__(self) -> None: if self.fetcher_request.cpu_millicores < 0: @@ -61,6 +65,8 @@ def __post_init__(self) -> None: raise ValueError('fetcher memory request cannot be negative') if not self.managed_label or not self.managed_value: raise ValueError('managed label and value cannot be empty') + if self.runtime_priority_class is not None and not self.runtime_priority_class: + raise ValueError('runtime priority class cannot be empty') @dataclass(frozen=True) @@ -75,6 +81,12 @@ def display_name(self) -> str: return f'{self.namespace}/{self.name}' +@dataclass(frozen=True) +class _FunctionPod: + display_name: str + priority_class: str | None + + def _object(value: object, context: str) -> JsonObject: if not isinstance(value, Mapping): raise CapacityError(f'{context} must be an object') @@ -129,6 +141,46 @@ def _execution(function: object) -> JsonObject: ) +def _index_function_pods(pods: list[object]) -> dict[str, tuple[_FunctionPod, ...]]: + indexed: defaultdict[str, list[_FunctionPod]] = defaultdict(list) + for pod in pods: + pod_object = _object(pod, 'pod') + status = _object(pod_object.get('status', {}), 'pod.status') + phase = status.get('phase') + if phase in _TERMINAL_POD_PHASES: + continue + if phase is not None and not isinstance(phase, str): + raise CapacityError('pod.status.phase must be a string') + + metadata = _object(pod_object.get('metadata', {}), 'pod.metadata') + labels_value = metadata.get('labels') + if labels_value is None: + continue + labels = _object(labels_value, 'pod.metadata.labels') + uid = labels.get(_FUNCTION_UID_LABEL) + if uid is None: + continue + if not isinstance(uid, str) or not uid: + raise CapacityError('pod functionUid label must be a non-empty string') + + name = _string(metadata.get('name'), 'pod name') + namespace_value = metadata.get('namespace') + if namespace_value is None: + display_name = name + else: + namespace = _string(namespace_value, 'pod namespace') + display_name = f'{namespace}/{name}' + + spec = _object(pod_object.get('spec'), 'pod.spec') + priority_class = spec.get('priorityClassName') + if priority_class is not None and ( + not isinstance(priority_class, str) or not priority_class + ): + raise CapacityError('pod.spec.priorityClassName must be a non-empty string') + indexed[uid].append(_FunctionPod(display_name, priority_class)) + return {uid: tuple(items) for uid, items in indexed.items()} + + class Controller: """Reconcile opt-in Fission Functions against current free capacity.""" @@ -148,9 +200,13 @@ def reconcile(self) -> ReconcileResult: f'{self._config.managed_label}={self._config.managed_value}', ) environments = index_environments(self._gateway.list_environments()) + pods = self._gateway.list_pods() + function_pods = ( + _index_function_pods(pods) if self._config.runtime_priority_class is not None else {} + ) snapshot = ClusterSnapshot.build( self._gateway.list_nodes(), - self._gateway.list_pods(), + pods, include_tainted=self._config.include_tainted_nodes, ) @@ -162,15 +218,22 @@ def reconcile(self) -> ReconcileResult: if not _is_managed(function, self._config): continue managed += 1 - if self._reconcile_function(function, environments, snapshot): + if self._reconcile_function( + function, + environments, + snapshot, + function_pods, + ): updated += 1 except Exception as error: # Each malformed CR must be isolated. display_name = self._safe_display_name(function) failures.append(display_name) + details = str(error) self._logger.exception( - 'failed to reconcile %s: %s', + 'failed to reconcile %s: %s: %s', display_name, type(error).__name__, + details, ) result = ReconcileResult(managed, updated, snapshot.ready_nodes) @@ -189,6 +252,7 @@ def _reconcile_function( function: object, environments: Mapping[tuple[str, str], JsonObject], snapshot: ClusterSnapshot, + function_pods: Mapping[str, tuple[_FunctionPod, ...]], ) -> bool: metadata = _metadata(function) identity = _identity(metadata) @@ -203,6 +267,8 @@ def _reconcile_function( f'{identity.display_name} uses unsupported executor {executor_type!r}', ) + self._validate_runtime_priority_class(identity, function_pods) + minimum = _integer(execution.get('MinScale'), 'MinScale', default=0) if minimum < 0 or minimum > INT32_MAX: raise CapacityError(f'MinScale must be between 0 and {INT32_MAX}') @@ -260,6 +326,25 @@ def _reconcile_function( ) return True + def _validate_runtime_priority_class( + self, + identity: _FunctionIdentity, + function_pods: Mapping[str, tuple[_FunctionPod, ...]], + ) -> None: + expected = self._config.runtime_priority_class + if expected is None: + return + for pod in function_pods.get(identity.uid, ()): + if pod.priority_class == expected: + continue + actual = pod.priority_class or '' + raise CapacityError( + f'{identity.display_name} has Function Pod {pod.display_name} with ' + f'priorityClassName {actual!r}; expected {expected!r}. Configure Fission ' + 'runtimePodSpec.podSpec.priorityClassName, restart the Fission executor, ' + 'and recreate existing Function workloads before opting them in', + ) + @staticmethod def _safe_display_name(function: object) -> str: try: diff --git a/deploy/fission-values.yaml b/deploy/fission-values.yaml new file mode 100644 index 0000000..f3b4b5f --- /dev/null +++ b/deploy/fission-values.yaml @@ -0,0 +1,5 @@ +# Pass this file alongside the existing values files used to manage Fission. +runtimePodSpec: + enabled: true + podSpec: + priorityClassName: autofission-runtime diff --git a/deploy/helm/autofission/templates/NOTES.txt b/deploy/helm/autofission/templates/NOTES.txt index 6e921c5..3e6af22 100644 --- a/deploy/helm/autofission/templates/NOTES.txt +++ b/deploy/helm/autofission/templates/NOTES.txt @@ -1,11 +1,21 @@ Autofission is installed in {{ .Release.Namespace }}. -Opt a newdeploy Function in: - kubectl label function -n \ - {{ .Values.controller.managedLabel }}={{ .Values.controller.managedValue }} +Before opting in a Function, configure Fission to assign the low-priority runtime +class created by this chart. Add this block to the Helm values used for Fission: + + runtimePodSpec: + enabled: true + podSpec: + priorityClassName: {{ include "autofission.runtimePriorityClassName" . }} + +Apply those values with the same helm upgrade command you use to manage Fission, +then restart its executor: -Protect existing workloads by configuring Fission runtime Pods to use: - {{ include "autofission.runtimePriorityClassName" . }} + kubectl rollout restart deployment/executor --namespace -See README.md#install-in-a-cluster for the required Fission Helm values. +Now opt a newdeploy Function in: + + kubectl label function -n \ + {{ .Values.controller.managedLabel }}={{ .Values.controller.managedValue }} +See https://github.com/pomponchik/autofission#installation for details. diff --git a/deploy/helm/autofission/templates/deployment.yaml b/deploy/helm/autofission/templates/deployment.yaml index 24c91d4..e89e430 100644 --- a/deploy/helm/autofission/templates/deployment.yaml +++ b/deploy/helm/autofission/templates/deployment.yaml @@ -47,6 +47,7 @@ spec: - --fetcher-memory-request={{ .Values.controller.fetcherMemoryRequest }} - --managed-label={{ .Values.controller.managedLabel }} - --managed-value={{ .Values.controller.managedValue }} + - --runtime-priority-class={{ include "autofission.runtimePriorityClassName" . }} - --log-level={{ .Values.controller.logLevel }} - --state-directory=/var/run/autofission {{- if .Values.controller.includeTaintedNodes }} @@ -98,4 +99,3 @@ spec: tolerations: {{- toYaml . | nindent 8 }} {{- end }} - diff --git a/tests/helpers.py b/tests/helpers.py index 872cd49..8b3f548 100644 --- a/tests/helpers.py +++ b/tests/helpers.py @@ -56,6 +56,7 @@ def pod( # noqa: PLR0913 *, phase: object = 'Running', function_uid: object = None, + namespace: str | None = None, extra_spec: Mapping[str, object] | None = None, ) -> dict[str, object]: labels: dict[str, object] = {} @@ -68,8 +69,11 @@ def pod( # noqa: PLR0913 spec['nodeName'] = node_name if extra_spec: spec.update(deepcopy(dict(extra_spec))) + metadata: dict[str, object] = {'name': name, 'labels': labels} + if namespace is not None: + metadata['namespace'] = namespace return { - 'metadata': {'name': name, 'labels': labels}, + 'metadata': metadata, 'spec': spec, 'status': {'phase': phase}, } diff --git a/tests/smokes/test_package.py b/tests/smokes/test_package.py index f6f85c3..5e1bfd8 100644 --- a/tests/smokes/test_package.py +++ b/tests/smokes/test_package.py @@ -127,6 +127,20 @@ def test_both_priority_classes_are_non_preempting_and_runtime_survives_uninstall assert 'helm.sh/resource-policy: keep' in template +def test_chart_guides_and_enforces_fission_runtime_priority_configuration() -> None: + deployment = (CHART / 'templates' / 'deployment.yaml').read_text(encoding='utf-8') + notes = (CHART / 'templates' / 'NOTES.txt').read_text(encoding='utf-8') + example = (ROOT / 'deploy' / 'fission-values.yaml').read_text(encoding='utf-8') + assert '--runtime-priority-class=' in deployment + assert 'runtimePodSpec:' in notes + assert 'priorityClassName:' in notes + assert '#installation' in notes + assert 'install-in-a-cluster' not in notes + assert yaml.safe_load(example)['runtimePodSpec']['podSpec']['priorityClassName'] == ( + 'autofission-runtime' + ) + + def test_repository_artifacts_do_not_embed_ipv4_addresses() -> None: ipv4 = re.compile(r'(? None: + monkeypatch.setenv('AUTOFISSION_RUNTIME_PRIORITY_CLASS', 'elastic-runtime') + + assert build_parser().parse_args([]).runtime_priority_class == 'elastic-runtime' + + def test_boolean_environment_and_cli_negation(monkeypatch: pytest.MonkeyPatch) -> None: monkeypatch.setenv('AUTOFISSION_INCLUDE_TAINTED_NODES', 'TRUE') assert build_parser().parse_args([]).include_tainted_nodes is True diff --git a/tests/units/test_controller.py b/tests/units/test_controller.py index c6ef638..aba7db7 100644 --- a/tests/units/test_controller.py +++ b/tests/units/test_controller.py @@ -153,6 +153,44 @@ def test_reconcile_observes_larger_real_function_pod_request() -> None: assert maximum == 4 +def test_reconcile_validates_existing_function_pod_priority_class( + caplog: pytest.LogCaptureFixture, +) -> None: + wrong = pod( + name='hello-abc', + namespace='fission-function', + function_uid='uid-1', + extra_spec={'priorityClassName': 'default'}, + ) + gateway = FakeGateway(pods=[wrong]) + config = ControllerConfig(runtime_priority_class='autofission-runtime') + + with caplog.at_level(logging.ERROR), pytest.raises(ReconcileError): + Controller(gateway, config).reconcile() + + assert gateway.patches == [] + assert 'fission-function/hello-abc' in caplog.text + assert "priorityClassName 'default'; expected 'autofission-runtime'" in caplog.text + assert 'Configure Fission runtimePodSpec.podSpec.priorityClassName' in caplog.text + + +def test_reconcile_accepts_expected_priority_and_ignores_terminal_old_pods() -> None: + expected = pod( + function_uid='uid-1', + extra_spec={'priorityClassName': 'autofission-runtime'}, + ) + old = pod( + name='old', + phase='Succeeded', + function_uid='uid-1', + extra_spec={'priorityClassName': 'wrong'}, + ) + gateway = FakeGateway(pods=[expected, old]) + config = ControllerConfig(runtime_priority_class='autofission-runtime') + + assert Controller(gateway, config).reconcile().updated_functions == 1 + + def test_reconcile_skips_unmanaged_and_terminating_functions() -> None: unmanaged = function(name='plain', managed=False) no_labels = function(name='none') @@ -352,6 +390,7 @@ def test_controller_does_not_mutate_input_objects(config: ControllerConfig) -> N ({'fetcher_request': Resources(0, -1)}, 'memory'), ({'managed_label': ''}, 'label'), ({'managed_value': ''}, 'value'), + ({'runtime_priority_class': ''}, 'priority class'), ], ) def test_controller_config_validation(kwargs: dict[str, object], message: str) -> None: From d97bc385aaef9680ccbdfa4e958488df03e574dd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 15:21:46 +0300 Subject: [PATCH 07/16] Document runtime priority class configuration and validation --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0e7ccda..2e457ca 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ Functions managed by Autofission are meant to use only spare cluster capacity. T Kubernetes represents this relationship with PriorityClasses. The Autofission Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime`. Function Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. -Fission creates the Function Pods, so this setting belongs to Fission rather than Autofission. Add the following block to the Helm values file used for your Fission installation, then apply that file with the same `helm upgrade` command you use to manage Fission: +Fission creates the Function Pods, so this setting belongs to Fission rather than Autofission. Add the following block to the Helm values file used for your Fission installation, then apply that file with the same `helm upgrade` command you use to manage Fission. The repository also includes the same block as a ready-to-use [values file](deploy/fission-values.yaml), which you can pass alongside your existing Fission values files: ```yaml runtimePodSpec: @@ -88,7 +88,7 @@ runtimePodSpec: priorityClassName: autofission-runtime ``` -CLI-only installations must provide an equivalent PriorityClass separately. See [Operations](#operations) for existing Fission workloads, [Configuration](#configuration) for credentials and resource-request settings, and [RBAC and security](#rbac-and-security) for required permissions. +CLI-only installations must provide an equivalent PriorityClass separately; pass its name with `--runtime-priority-class` to check existing Function Pods. See [Operations](#operations) for existing Fission workloads, [Configuration](#configuration) for credentials and resource-request settings, and [RBAC and security](#rbac-and-security) for required permissions. ## Quick start @@ -157,6 +157,7 @@ CLI flags take precedence over valid, non-empty environment variables, which tak | `--fetcher-memory-request` | `AUTOFISSION_FETCHER_MEMORY_REQUEST` | `controller.fetcherMemoryRequest` | `16Mi` | | `--managed-label` | `AUTOFISSION_MANAGED_LABEL` | `controller.managedLabel` | `autoscaling.fission.io/cluster-capacity` | | `--managed-value` | `AUTOFISSION_MANAGED_VALUE` | `controller.managedValue` | `true` | +| `--runtime-priority-class` | `AUTOFISSION_RUNTIME_PRIORITY_CLASS` | `priorityClasses.runtime.name` | disabled (CLI); rendered runtime class (chart) | | `--include-tainted-nodes` | `AUTOFISSION_INCLUDE_TAINTED_NODES` | `controller.includeTaintedNodes` | `false` | | `--log-level` | `AUTOFISSION_LOG_LEVEL` | `controller.logLevel` | `INFO` | | `--state-directory` | `AUTOFISSION_STATE_DIRECTORY` | — | `/tmp/autofission` (CLI); `/var/run/autofission` (chart) | @@ -193,7 +194,9 @@ The chart runs one replica with a `Recreate` strategy, preventing overlap during Function patches include a `resourceVersion` precondition. If a Function changes after Autofission reads it, the patch fails with a conflict instead of overwriting the concurrent change; the daemon retries the Function during its next cycle. -Fission's `runtimePodSpec` setting is global: Fission merges its supported fields, including defaults supplied by its chart, into both `poolmgr` and `newdeploy` runtime Pods. After enabling or changing it, restart the Fission executor so it reads the new setting. Fission may not update existing `newdeploy` Deployment templates, so verify that each Function Deployment managed by Autofission uses the configured runtime PriorityClass before opting in that Function. +Fission's `runtimePodSpec` setting is global: Fission merges its supported fields, including defaults supplied by its chart, into both `poolmgr` and `newdeploy` runtime Pods. After enabling or changing it, restart the Fission executor so it reads the new setting. Fission may not update existing `newdeploy` Deployment templates, so recreate or update them before opting in those Functions. + +The Helm-installed controller checks every existing, non-terminal Pod belonging to a managed Function and rejects that Function if its `priorityClassName` does not match the class created by the chart. The error explains how to correct Fission. A Function without any existing Pods cannot be checked this way; configure `runtimePodSpec` before opting it in so its first cold-start Pod receives the class. CLI-only runs can enable the same check with `--runtime-priority-class NAME`. Useful checks for the default release name, namespace, and ServiceAccount: From c2610d88baa917f514ccda3eb11e9899aea56a87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 15:32:07 +0300 Subject: [PATCH 08/16] Bump version to 0.0.6 --- pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index fef19ba..dc7a026 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ build-backend = 'setuptools.build_meta' [project] name = 'autofission' -version = '0.0.5' +version = '0.0.6' authors = [{ name = 'pomponchik' }] description = 'Capacity-aware autoscaling limits for Fission on Kubernetes' readme = 'README.md' From 2b55ebc0d33a8e7ba01a2348f863beae3012b77d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 15:32:53 +0300 Subject: [PATCH 09/16] Merge PriorityClass explanation into single paragraph --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index 2e457ca..b8f47b8 100644 --- a/README.md +++ b/README.md @@ -75,9 +75,7 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. -Functions managed by Autofission are meant to use only spare cluster capacity. They should fill resources left idle by regular services, then give those resources back when the cluster needs them for something more important. - -Kubernetes represents this relationship with PriorityClasses. The Autofission Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime`. Function Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. +Functions managed by Autofission are meant to use only spare cluster capacity. They should fill resources left idle by regular services, then give those resources back when the cluster needs them for something more important. Kubernetes represents this relationship with PriorityClasses. The Autofission Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime`. Function Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. Fission creates the Function Pods, so this setting belongs to Fission rather than Autofission. Add the following block to the Helm values file used for your Fission installation, then apply that file with the same `helm upgrade` command you use to manage Fission. The repository also includes the same block as a ready-to-use [values file](deploy/fission-values.yaml), which you can pass alongside your existing Fission values files: From 75a4c30d6588f6bf44ca2e6422f78d5a0c75ec91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Tue, 25 Aug 2026 15:37:37 +0300 Subject: [PATCH 10/16] Simplify PriorityClass explanation in README --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index b8f47b8..9a66c65 100644 --- a/README.md +++ b/README.md @@ -75,9 +75,11 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. -Functions managed by Autofission are meant to use only spare cluster capacity. They should fill resources left idle by regular services, then give those resources back when the cluster needs them for something more important. Kubernetes represents this relationship with PriorityClasses. The Autofission Helm chart creates a low, non-preempting PriorityClass named `autofission-runtime`. Function Pods using this class cannot evict other workloads, but higher-priority services can evict them and take their place when the cluster is full. +Functions managed by Autofission are background workloads. They should run only in space that regular services are not using. When a regular service needs that space, the Function Pods must move out of the way. -Fission creates the Function Pods, so this setting belongs to Fission rather than Autofission. Add the following block to the Helm values file used for your Fission installation, then apply that file with the same `helm upgrade` command you use to manage Fission. The repository also includes the same block as a ready-to-use [values file](deploy/fission-values.yaml), which you can pass alongside your existing Fission values files: +Kubernetes has a built-in mechanism for this: a PriorityClass tells the scheduler which Pods are more important. The Autofission Helm chart creates a low-priority class named `autofission-runtime`. Kubernetes will never remove another workload to make room for a Function Pod using this class. If a regular service needs the space occupied by such a Function Pod, Kubernetes can stop the Function Pod and use its place for the service. + +Autofission does not create Function Pods itself—Fission does. You must therefore tell Fission to assign `autofission-runtime` to every Function Pod it creates. Add the following block to the Helm values you already use for Fission: ```yaml runtimePodSpec: @@ -86,6 +88,8 @@ runtimePodSpec: priorityClassName: autofission-runtime ``` +Then upgrade Fission with those values, using the same `helm upgrade` command you normally use to manage it. Instead of copying the block manually, you can pass the ready-made [values file](deploy/fission-values.yaml) alongside your other Fission values files. + CLI-only installations must provide an equivalent PriorityClass separately; pass its name with `--runtime-priority-class` to check existing Function Pods. See [Operations](#operations) for existing Fission workloads, [Configuration](#configuration) for credentials and resource-request settings, and [RBAC and security](#rbac-and-security) for required permissions. From 326ec4f40e21ff1b0e472b4a388626a2c6e83b00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 00:18:37 +0300 Subject: [PATCH 11/16] Account for pending Pod resources in capacity calculations --- autofission/capacity.py | 153 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 144 insertions(+), 9 deletions(-) diff --git a/autofission/capacity.py b/autofission/capacity.py index 848e9d0..5b2f665 100644 --- a/autofission/capacity.py +++ b/autofission/capacity.py @@ -206,7 +206,9 @@ def _node_resources(node: object, *, include_tainted: bool) -> NodeResources | N @dataclass(frozen=True) class _PodAllocation: - node_name: str + identity: str + node_name: str | None + nominated_node_name: str | None resources: Resources function_uid: str | None @@ -221,13 +223,25 @@ def _pod_allocation(pod: object) -> _PodAllocation | None: if phase is not None and not isinstance(phase, str): raise CapacityError('pod.status.phase must be a string') - node_name = spec.get('nodeName') - if node_name in (None, ''): - return None - if not isinstance(node_name, str): + node_name_value = spec.get('nodeName') + if node_name_value is not None and not isinstance(node_name_value, str): raise CapacityError('pod.spec.nodeName must be a string') + node_name = node_name_value or None + + nominated_value = status.get('nominatedNodeName') + if nominated_value is not None and not isinstance(nominated_value, str): + raise CapacityError('pod.status.nominatedNodeName must be a string') + nominated_node_name = nominated_value or None metadata = _object(pod_object.get('metadata', {}), 'pod.metadata') + name = _name(metadata, 'pod') + namespace_value = metadata.get('namespace') + if namespace_value is None: + identity = name + elif isinstance(namespace_value, str) and namespace_value: + identity = f'{namespace_value}/{name}' + else: + raise CapacityError('pod.metadata.namespace must be a non-empty string') labels_value = metadata.get('labels') function_uid: str | None = None if labels_value is not None: @@ -236,7 +250,82 @@ def _pod_allocation(pod: object) -> _PodAllocation | None: if uid_value is not None and not isinstance(uid_value, str): raise CapacityError('pod functionUid label must be a string') function_uid = uid_value - return _PodAllocation(node_name, pod_requests(spec), function_uid) + return _PodAllocation( + identity, + node_name, + nominated_node_name, + pod_requests(spec), + function_uid, + ) + + +def _fits(resources: Resources, slots: int, request: Resources) -> bool: + return ( + slots > 0 + and resources.cpu_millicores >= request.cpu_millicores + and resources.memory_bytes >= request.memory_bytes + ) + + +def _reserve_unbound_pods( + nodes: Mapping[str, NodeResources], + ordinary_used: Mapping[str, Resources], + ordinary_used_slots: Mapping[str, int], + pending: Iterable[_PodAllocation], +) -> tuple[dict[str, Resources], dict[str, int]]: + """Virtually place ordinary pending Pods after reclaiming Function Pods.""" + free: dict[str, Resources] = {} + free_slots: dict[str, int] = {} + for name, node in nodes.items(): + free[name] = Resources( + max( + node.allocatable.cpu_millicores + - ordinary_used.get(name, Resources()).cpu_millicores, + 0, + ), + max( + node.allocatable.memory_bytes - ordinary_used.get(name, Resources()).memory_bytes, + 0, + ), + ) + free_slots[name] = max(node.pod_slots - ordinary_used_slots.get(name, 0), 0) + + reserved: defaultdict[str, Resources] = defaultdict(Resources) + reserved_slots: defaultdict[str, int] = defaultdict(int) + ordered = sorted( + (allocation for allocation in pending if allocation.function_uid is None), + key=lambda allocation: ( + -allocation.resources.cpu_millicores, + -allocation.resources.memory_bytes, + allocation.identity, + ), + ) + for allocation in ordered: + if allocation.nominated_node_name is not None: + candidates = [allocation.nominated_node_name] + else: + candidates = list(nodes) + fitting = [ + name + for name in candidates + if name in nodes and _fits(free[name], free_slots[name], allocation.resources) + ] + if not fitting: + continue + selected = min( + fitting, + key=lambda name: ( + free[name].cpu_millicores - allocation.resources.cpu_millicores, + free[name].memory_bytes - allocation.resources.memory_bytes, + free_slots[name] - 1, + name, + ), + ) + free[selected] -= allocation.resources + free_slots[selected] -= 1 + reserved[selected] += allocation.resources + reserved_slots[selected] += 1 + return dict(reserved), dict(reserved_slots) class ClusterSnapshot: @@ -250,6 +339,8 @@ def __init__( # noqa: PLR0913 function_used: Mapping[str, Mapping[str, Resources]], function_slots: Mapping[str, Mapping[str, int]], observed_requests: Mapping[str, Resources], + pending_reserved: Mapping[str, Resources], + pending_reserved_slots: Mapping[str, int], ) -> None: self._nodes = dict(nodes) self._used = dict(used) @@ -257,6 +348,8 @@ def __init__( # noqa: PLR0913 self._function_used = {uid: dict(resources) for uid, resources in function_used.items()} self._function_slots = {uid: dict(slots) for uid, slots in function_slots.items()} self._observed_requests = dict(observed_requests) + self._pending_reserved = dict(pending_reserved) + self._pending_reserved_slots = dict(pending_reserved_slots) @classmethod def build( @@ -287,10 +380,20 @@ def build( lambda: defaultdict(int), ) observed: defaultdict[str, Resources] = defaultdict(Resources) + pending: list[_PodAllocation] = [] for pod in pods: allocation = _pod_allocation(pod) - if allocation is None or allocation.node_name not in node_map: + if allocation is None: + continue + if allocation.node_name is None: + pending.append(allocation) + if allocation.function_uid: + observed[allocation.function_uid] = observed[allocation.function_uid].maximum( + allocation.resources, + ) + continue + if allocation.node_name not in node_map: continue used[allocation.node_name] += allocation.resources used_slots[allocation.node_name] += 1 @@ -300,6 +403,28 @@ def build( function_slots[uid][allocation.node_name] += 1 observed[uid] = observed[uid].maximum(allocation.resources) + total_function_used: defaultdict[str, Resources] = defaultdict(Resources) + total_function_slots: defaultdict[str, int] = defaultdict(int) + for resources_by_node in function_used.values(): + for name, resources in resources_by_node.items(): + total_function_used[name] += resources + for slots_by_node in function_slots.values(): + for name, slots in slots_by_node.items(): + total_function_slots[name] += slots + ordinary_used = { + name: used.get(name, Resources()) - total_function_used.get(name, Resources()) + for name in node_map + } + ordinary_used_slots = { + name: used_slots.get(name, 0) - total_function_slots.get(name, 0) for name in node_map + } + pending_reserved, pending_reserved_slots = _reserve_unbound_pods( + node_map, + ordinary_used, + ordinary_used_slots, + pending, + ) + return cls( node_map, used, @@ -307,6 +432,8 @@ def build( function_used, function_slots, observed, + pending_reserved, + pending_reserved_slots, ) @property @@ -329,8 +456,16 @@ def function_capacity(self, function_uid: str, request: Resources) -> int: own_used = self._function_used.get(function_uid, {}) own_slots = self._function_slots.get(function_uid, {}) for name, node in self._nodes.items(): - used = self._used.get(name, Resources()) - own_used.get(name, Resources()) - slots = self._used_slots.get(name, 0) - own_slots.get(name, 0) + used = ( + self._used.get(name, Resources()) + - own_used.get(name, Resources()) + + self._pending_reserved.get(name, Resources()) + ) + slots = ( + self._used_slots.get(name, 0) + - own_slots.get(name, 0) + + self._pending_reserved_slots.get(name, 0) + ) free_cpu = max(node.allocatable.cpu_millicores - used.cpu_millicores, 0) free_memory = max(node.allocatable.memory_bytes - used.memory_bytes, 0) free_slots = max(node.pod_slots - slots, 0) From 362345b466d18f05da4395f9e835832d7cf81a1a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 00:18:50 +0300 Subject: [PATCH 12/16] Reserve capacity for unbound ordinary Pods --- tests/e2e/test_autoscaling.py | 266 ++++++++++++++++++++++++++++++++- tests/units/test_capacity.py | 81 +++++++++- tests/units/test_controller.py | 18 +++ 3 files changed, 357 insertions(+), 8 deletions(-) diff --git a/tests/e2e/test_autoscaling.py b/tests/e2e/test_autoscaling.py index 125fac6..a911bf8 100644 --- a/tests/e2e/test_autoscaling.py +++ b/tests/e2e/test_autoscaling.py @@ -13,7 +13,9 @@ import pytest +from autofission.capacity import pod_requests from autofission.controller import ANNOTATION_PREFIX +from autofission.models import Resources from autofission.quantities import parse_cpu_millicores from tests.e2e.conftest import FissionFunction, FunctionFactory from tests.e2e.support import PROTECTED_LABEL, WORKER_LABEL, E2ECluster @@ -164,13 +166,174 @@ def test_capacity_contracts_while_ordinary_work_runs_and_recovers_afterward( _assert_protected_unchanged(e2e_cluster, protected_before) +def test_ordinary_service_preempts_saturated_fission_workload( + e2e_cluster: E2ECluster, + function_factory: FunctionFactory, + router_url: str, +) -> None: + """An ordinary Pod starts by preempting already-running elastic Function Pods.""" + worker_cpu = _largest_worker_cpu(e2e_cluster) + function = function_factory( + cpu_millicores=max(worker_cpu // 2, 500), + minimum=1, + target_cpu=20, + route=True, + ) + baseline = _wait_for_calculated_maximum(e2e_cluster, function, below=1_000) + assert baseline >= 2 + deployment = _wait_for_function_deployment(e2e_cluster, function.name) + hpa = _wait_for_function_hpa(e2e_cluster, deployment) + _wait_for_hpa_maximum(e2e_cluster, hpa, baseline) + protected_before = _protected_snapshot(e2e_cluster) + + assert function.path is not None + controller_running = True + with _LoadGenerator( + f'{router_url}{function.path}', + concurrency=max(baseline * 3, 12), + ): + e2e_cluster.wait_for( + 'Fission to occupy all calculated capacity', + lambda: baseline if _ready_replicas(e2e_cluster, deployment) == baseline else None, + timeout=300, + interval=5, + ) + function_pods = _function_pods(e2e_cluster, function.name) + function_uids = {_pod_uid(item) for item in function_pods} + request = _largest_pod_request(function_pods) + preempted_before = _preempted_pod_uids(e2e_cluster) + + try: + controller_running = False + _scale_autofission(e2e_cluster, 0) + _apply_competing_pod(e2e_cluster, 'autofission-e2e-preempting', request) + competing = e2e_cluster.wait_for( + 'ordinary Pod to become Ready after Function preemption', + lambda: _ready_named_pod(e2e_cluster, 'autofission-e2e-preempting'), + timeout=180, + ) + assert _pod_priority(competing) is None + victims = e2e_cluster.wait_for( + 'a saturated Function Pod to be preempted', + lambda: (_preempted_pod_uids(e2e_cluster) - preempted_before) & function_uids + or None, + timeout=120, + ) + assert victims + + _scale_autofission(e2e_cluster, 1) + controller_running = True + contracted = e2e_cluster.wait_for( + 'MaxScale contraction after the ordinary Pod starts', + lambda: ( + value + if (value := _function_maximum(e2e_cluster, function)) < baseline + else None + ), + timeout=120, + ) + assert 1 <= contracted < baseline + _assert_protected_unchanged(e2e_cluster, protected_before) + finally: + try: + _delete_named_pod(e2e_cluster, 'autofission-e2e-preempting') + finally: + if not controller_running: + _scale_autofission(e2e_cluster, 1) + + +def test_pending_nonpreempting_service_reduces_fission_maxscale_and_starts( + e2e_cluster: E2ECluster, + function_factory: FunctionFactory, + router_url: str, +) -> None: + """Pending ordinary demand is reserved before it has a spec.nodeName.""" + worker_cpu = _largest_worker_cpu(e2e_cluster) + function = function_factory( + cpu_millicores=max(worker_cpu // 2, 500), + minimum=1, + target_cpu=20, + route=True, + ) + baseline = _wait_for_calculated_maximum(e2e_cluster, function, below=1_000) + assert baseline >= 2 + deployment = _wait_for_function_deployment(e2e_cluster, function.name) + hpa = _wait_for_function_hpa(e2e_cluster, deployment) + _wait_for_hpa_maximum(e2e_cluster, hpa, baseline) + _shorten_hpa_scale_down(e2e_cluster, hpa) + protected_before = _protected_snapshot(e2e_cluster) + + assert function.path is not None + with _LoadGenerator( + f'{router_url}{function.path}', + concurrency=max(baseline * 3, 12), + ): + e2e_cluster.wait_for( + 'Fission to occupy all calculated capacity', + lambda: baseline if _ready_replicas(e2e_cluster, deployment) == baseline else None, + timeout=300, + interval=5, + ) + request = _largest_pod_request(_function_pods(e2e_cluster, function.name)) + preempted_before = _preempted_pod_uids(e2e_cluster) + _apply_competing_pod( + e2e_cluster, + 'autofission-e2e-pending-reservation', + request, + priority_class='autofission-controller', + ) + try: + pending = e2e_cluster.wait_for( + 'non-preempting ordinary Pod to wait without spec.nodeName', + lambda: _pending_named_pod( + e2e_cluster, + 'autofission-e2e-pending-reservation', + ), + timeout=120, + interval=0.5, + ) + assert _pod_priority(pending) == 'autofission-controller' + assert _pod_node(pending) is None + assert _pod_nominated_node(pending) is None + + contracted = e2e_cluster.wait_for( + 'MaxScale contraction for the unbound ordinary Pod', + lambda: ( + value + if (value := _function_maximum(e2e_cluster, function)) < baseline + else None + ), + timeout=120, + ) + _wait_for_hpa_maximum(e2e_cluster, hpa, contracted) + e2e_cluster.wait_for( + 'non-preempting ordinary Pod to become Ready after HPA scale-down', + lambda: _ready_named_pod( + e2e_cluster, + 'autofission-e2e-pending-reservation', + ), + timeout=300, + interval=3, + ) + assert _preempted_pod_uids(e2e_cluster) == preempted_before + _assert_protected_unchanged(e2e_cluster, protected_before) + finally: + _delete_named_pod(e2e_cluster, 'autofission-e2e-pending-reservation') + + e2e_cluster.wait_for( + 'MaxScale recovery after pending reservation disappears', + lambda: baseline if _function_maximum(e2e_cluster, function) == baseline else None, + timeout=120, + ) + + def test_oversized_elastic_work_stays_pending_without_preempting_services( e2e_cluster: E2ECluster, function_factory: FunctionFactory, ) -> None: """Impossible demand is bounded and non-preempting priority preserves existing Pods.""" protected_before = _protected_snapshot(e2e_cluster) - preempted_before = _preempted_event_uids(e2e_cluster) + preempted_before = _preempted_pod_uids(e2e_cluster) worker_cpu = _largest_worker_cpu(e2e_cluster) function = function_factory(cpu_millicores=worker_cpu + 1_000, minimum=1) @@ -197,7 +360,7 @@ def test_oversized_elastic_work_stays_pending_without_preempting_services( _wait_for_failed_scheduling(e2e_cluster, 'autofission-e2e-nonpreempting') _assert_priority_classes_never_preempt(e2e_cluster) _assert_protected_unchanged(e2e_cluster, protected_before) - assert _preempted_event_uids(e2e_cluster) == preempted_before + assert _preempted_pod_uids(e2e_cluster) == preempted_before finally: e2e_cluster.kubectl( '-n', @@ -362,6 +525,16 @@ def _function_pods(cluster: E2ECluster, function_name: str) -> list[Mapping[str, return _items(document, 'function pods') +def _largest_pod_request(pods: list[Mapping[str, object]]) -> Resources: + if not pods: + raise AssertionError('expected at least one Function Pod') + largest = Resources() + for item in pods: + spec = _mapping(item.get('spec'), 'Function Pod spec') + largest = largest.maximum(pod_requests(spec)) + return largest + + def _pending_function_pod( cluster: E2ECluster, function_name: str, @@ -444,6 +617,74 @@ def _apply_reservation(cluster: E2ECluster) -> None: ) +def _apply_competing_pod( + cluster: E2ECluster, + name: str, + request: Resources, + *, + priority_class: str | None = None, +) -> None: + spec: dict[str, object] = { + 'nodeSelector': {'autofission.io/e2e-worker': 'true'}, + 'terminationGracePeriodSeconds': 0, + 'containers': [ + { + 'name': 'ordinary-service', + 'image': 'registry.k8s.io/pause:3.10', + 'resources': { + 'requests': { + 'cpu': f'{request.cpu_millicores}m', + 'memory': str(request.memory_bytes), + }, + 'limits': { + 'cpu': f'{request.cpu_millicores}m', + 'memory': str(request.memory_bytes), + }, + }, + }, + ], + } + if priority_class is not None: + spec['priorityClassName'] = priority_class + cluster.apply( + { + 'apiVersion': 'v1', + 'kind': 'Pod', + 'metadata': {'name': name, 'namespace': 'default'}, + 'spec': spec, + }, + ) + + +def _delete_named_pod(cluster: E2ECluster, name: str) -> None: + cluster.kubectl( + '-n', + 'default', + 'delete', + f'pod/{name}', + '--ignore-not-found', + '--wait=true', + ) + + +def _scale_autofission(cluster: E2ECluster, replicas: int) -> None: + cluster.kubectl( + '-n', + 'fission', + 'scale', + 'deployment/autofission', + f'--replicas={replicas}', + ) + cluster.kubectl( + '-n', + 'fission', + 'rollout', + 'status', + 'deployment/autofission', + '--timeout=5m', + ) + + def _largest_worker_cpu(cluster: E2ECluster) -> int: document = cluster.kubectl_json('get', 'nodes', '-l', WORKER_LABEL) capacities = [] @@ -485,6 +726,12 @@ def _pending_named_pod(cluster: E2ECluster, name: str) -> Mapping[str, object] | return pod if _pod_phase(pod) == 'Pending' else None +def _ready_named_pod(cluster: E2ECluster, name: str) -> Mapping[str, object] | None: + pod = cluster.kubectl_json('-n', 'default', 'get', 'pod', name) + status = _mapping(pod.get('status', {}), 'Pod status') + return pod if status.get('phase') == 'Running' and _pod_is_ready(status) else None + + def _wait_for_failed_scheduling(cluster: E2ECluster, pod_name: str) -> None: def found() -> bool | None: document = cluster.kubectl_json( @@ -506,7 +753,7 @@ def _assert_priority_classes_never_preempt(cluster: E2ECluster) -> None: assert priority.get('preemptionPolicy') == 'Never' -def _preempted_event_uids(cluster: E2ECluster) -> set[str]: +def _preempted_pod_uids(cluster: E2ECluster) -> set[str]: document = cluster.kubectl_json( 'get', 'events', @@ -516,8 +763,8 @@ def _preempted_event_uids(cluster: E2ECluster) -> set[str]: ) result = set() for event in _items(document, 'Preempted events'): - metadata = _mapping(event.get('metadata'), 'event.metadata') - result.add(_string(metadata.get('uid'), 'event.metadata.uid')) + involved = _mapping(event.get('involvedObject'), 'event.involvedObject') + result.add(_string(involved.get('uid'), 'event.involvedObject.uid')) return result @@ -525,6 +772,15 @@ def _pod_priority(pod: Mapping[str, object]) -> object: return _mapping(pod.get('spec'), 'pod.spec').get('priorityClassName') +def _pod_uid(pod: Mapping[str, object]) -> str: + metadata = _mapping(pod.get('metadata'), 'pod.metadata') + return _string(metadata.get('uid'), 'pod.metadata.uid') + + +def _pod_node(pod: Mapping[str, object]) -> object: + return _mapping(pod.get('spec'), 'pod.spec').get('nodeName') + + def _pod_phase(pod: Mapping[str, object]) -> object: return _mapping(pod.get('status'), 'pod.status').get('phase') diff --git a/tests/units/test_capacity.py b/tests/units/test_capacity.py index 75a2938..54f39e6 100644 --- a/tests/units/test_capacity.py +++ b/tests/units/test_capacity.py @@ -209,12 +209,10 @@ def test_snapshot_rejects_malformed_ready_nodes(mutation: object, message: str) ClusterSnapshot.build([candidate], []) -def test_snapshot_ignores_terminal_unbound_and_unknown_node_pods() -> None: +def test_snapshot_ignores_terminal_and_unknown_node_pods() -> None: pods = [ pod('succeeded', phase='Succeeded'), pod('failed', phase='Failed'), - pod('unbound', node_name=None), - pod('empty-node', node_name=''), pod('elsewhere', node_name='missing'), ] snapshot = ClusterSnapshot.build([node(cpu='1', memory='1Gi', pods='2')], pods) @@ -222,6 +220,75 @@ def test_snapshot_ignores_terminal_unbound_and_unknown_node_pods() -> None: assert snapshot.function_capacity('new', Resources(500, 512 * 2**20)) == 2 +def test_snapshot_reserves_unbound_ordinary_pods_after_reclaiming_function_pods() -> None: + pods = [ + pod('own-a', cpu='500m', memory='512Mi', function_uid='mine'), + pod('own-b', cpu='500m', memory='512Mi', function_uid='mine'), + pod('waiting-a', node_name=None, cpu='500m', memory='512Mi', phase='Pending'), + pod('waiting-b', node_name='', cpu='500m', memory='512Mi', phase='Pending'), + ] + snapshot = ClusterSnapshot.build( + [node(cpu='2', memory='2Gi', pods='4')], + pods, + ) + + assert snapshot.function_capacity('mine', Resources(500, 512 * 2**20)) == 2 + + +def test_snapshot_does_not_reserve_unbound_function_pods() -> None: + waiting = pod( + 'waiting-function', + node_name=None, + cpu='1', + memory='1Gi', + phase='Pending', + function_uid='other', + ) + snapshot = ClusterSnapshot.build( + [node(cpu='2', memory='2Gi', pods='4')], + [waiting], + ) + + assert snapshot.observed_request('other') == Resources(1_000, 2**30) + assert snapshot.function_capacity('mine', Resources(500, 512 * 2**20)) == 4 + + +def test_snapshot_uses_nominated_node_for_unbound_pod_reservation() -> None: + waiting = pod( + 'waiting', + node_name=None, + cpu='1', + memory='1Gi', + phase='Pending', + ) + waiting['status']['nominatedNodeName'] = 'node-b' # type: ignore[index] + snapshot = ClusterSnapshot.build( + [ + node('node-a', cpu='2', memory='2Gi', pods='4'), + node('node-b', cpu='2', memory='2Gi', pods='4'), + ], + [waiting], + ) + + assert snapshot.function_capacity('mine', Resources(1_000, 2**30)) == 3 + + +def test_snapshot_ignores_unbound_pod_that_cannot_fit_after_reclaiming_functions() -> None: + waiting = pod( + 'impossible', + node_name=None, + cpu='3', + memory='1Gi', + phase='Pending', + ) + snapshot = ClusterSnapshot.build( + [node(cpu='2', memory='2Gi', pods='4')], + [waiting], + ) + + assert snapshot.function_capacity('mine', Resources(500, 512 * 2**20)) == 4 + + def test_snapshot_subtracts_other_pods_but_reclaims_own_function_pods() -> None: pods = [ pod('normal', cpu='1', memory='1Gi'), @@ -299,6 +366,14 @@ def test_snapshot_rejects_invalid_function_inputs( lambda item: item['spec'].update(nodeName=1), 'pod.spec.nodeName must be a string', ), + ( + lambda item: item['status'].update(nominatedNodeName=1), + 'pod.status.nominatedNodeName must be a string', + ), + ( + lambda item: item['metadata'].update(namespace=''), + 'pod.metadata.namespace must be a non-empty string', + ), ( lambda item: item['metadata'].update(labels=[]), 'pod.metadata.labels must be an object', diff --git a/tests/units/test_controller.py b/tests/units/test_controller.py index aba7db7..dfac75c 100644 --- a/tests/units/test_controller.py +++ b/tests/units/test_controller.py @@ -153,6 +153,24 @@ def test_reconcile_observes_larger_real_function_pod_request() -> None: assert maximum == 4 +def test_reconcile_reserves_capacity_for_unbound_ordinary_pod() -> None: + waiting = pod( + name='waiting-service', + node_name=None, + cpu='1', + memory='1Gi', + phase='Pending', + ) + gateway = FakeGateway(pods=[waiting]) + + Controller(gateway).reconcile() + + maximum = gateway.patches[0][2]['spec']['InvokeStrategy']['ExecutionStrategy'][ # type: ignore[index] + 'MaxScale' + ] + assert maximum == 11 + + def test_reconcile_validates_existing_function_pod_priority_class( caplog: pytest.LogCaptureFixture, ) -> None: From dce40d9b742278b88760cefdcaeabe346d5154b2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 00:19:04 +0300 Subject: [PATCH 13/16] Document pending Pod capacity reservation and placement model --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 9a66c65..3cfa41a 100644 --- a/README.md +++ b/README.md @@ -138,7 +138,7 @@ flowchart TD Given the same controller settings and Function, Environment, Node, and Pod data, each cycle produces the same result without unnecessary patches: 1. List opted-in Functions, along with Fission Environments, Kubernetes Nodes, and Pods. -2. Keep schedulable `Ready` nodes and subtract the CPU, memory, and Pod slots requested by their existing workloads. Pod requests follow Kubernetes scheduling semantics, including init containers, restartable sidecars, Pod-level requests, and Pod overhead. +2. Keep schedulable `Ready` nodes and subtract the CPU, memory, and Pod slots requested by their existing workloads. Active, unbound non-Function Pods are virtually placed after reclaiming Function Pods: an existing `status.nominatedNodeName` is honored, otherwise a deterministic best-fit placement is used. Pod requests follow Kubernetes scheduling semantics, including init containers, restartable sidecars, Pod-level requests, and Pod overhead. 3. Resolve the resources required by one replica of each Function from its Function, Environment, fetcher, and observed runtime Pod configuration. 4. For each Function independently, estimate how many total replicas fit across the remaining per-node capacity, including replicas that are already running. 5. Update `MaxScale` to the calculated capacity, but never below `MinScale` or `1`. @@ -240,9 +240,10 @@ Apply any required controller overrides described under [Configuration](#configu - Autofission calculates the full capacity that each managed Function could use by itself. This preserves burst capacity, but simultaneous cold bursts can leave Pods `Pending`. Later cycles account for Pods from other Functions after they are scheduled and may reduce the limits; Autofission is not a fairness scheduler. - With at least one eligible node, modeled capacity below one replica produces `MaxScale=1` because Fission and Kubernetes require a positive HPA maximum. An explicit `MinScale` is also honored even when it exceeds currently free capacity. With no eligible nodes, reconciliation fails and leaves existing limits unchanged. - The [Fission v1 API](https://fission.io/docs/reference/crd-reference/) is tested end to end with Fission `1.27.0` on Kubernetes `1.34`; the Fission `1.27.0` chart requires Kubernetes `1.32` or newer. Other version combinations are untested. Environment resource inheritance follows Fission's override semantics and is covered by unit tests. -- Capacity includes CPU, memory, and Pod slots. It does not model storage, GPUs and other extended resources, quotas, topology or affinity, per-Function scheduling constraints, image architecture, in-place resize status, or unscheduled third-party Pods. +- Capacity includes CPU, memory, and Pod slots. It does not model storage, GPUs and other extended resources, quotas, topology or affinity, per-Function scheduling constraints, image architecture, or in-place resize status. +- Pending ordinary Pods reserve modeled capacity only when their requests can be virtually placed on an eligible node after reclaiming Function Pods. The fallback placement does not model node selectors, tolerations, affinity, topology, storage, or scheduler plugins; a scheduler-nominated node is used when available. Pending Fission Function Pods are not reserved, preventing Function demand from reducing its own capacity budget. - Nodes with `NoSchedule` or `NoExecute` taints are excluded unless the `--include-tainted-nodes` CLI flag or `controller.includeTaintedNodes` Helm value is explicitly set. Only enable the option when Fission runtime Pods actually tolerate those taints. -- Larger requests caused by extra sidecars, Pod overhead, or runtime `Container`/`PodSpec` overrides are learned only from non-terminal Function Pods scheduled to eligible nodes. Until such a Pod exists, the Function/Environment-plus-fetcher estimate is used. +- Larger requests caused by extra sidecars, Pod overhead, or runtime `Container`/`PodSpec` overrides are learned only from non-terminal Function Pods that are unbound or scheduled to eligible nodes. Until such a Pod exists, the Function/Environment-plus-fetcher estimate is used. - When Fission is configured to use the low, non-preempting runtime PriorityClass shown under Installation, Function Pods cannot preempt existing workloads. This does not prevent node-pressure eviction when requests are inaccurate or nodes run at their physical limit; reserve headroom and set accurate requests. - Very large clusters should benchmark API-server load and controller memory before shortening the default interval. - Prefer one Autofission installation per cluster. To run multiple installations, first configure `managedLabel` and `managedValue` so the controllers select disjoint sets of Functions; two controllers must never manage the same Function. Also ensure that each release renders distinct names for its chart-created cluster-scoped resources. With the default chart settings, cluster-scoped RBAC objects and PriorityClasses use rendered full names, so identical names collide even across namespaces. Any custom cluster-scoped objects created by separate installations must also have distinct names. From 5046afe6b4349214c5dd8dea2087ebb95c686bf0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 00:27:52 +0300 Subject: [PATCH 14/16] Refactor allocation loop to use if-elif structure --- autofission/capacity.py | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/autofission/capacity.py b/autofission/capacity.py index 5b2f665..75fce56 100644 --- a/autofission/capacity.py +++ b/autofission/capacity.py @@ -392,16 +392,14 @@ def build( observed[allocation.function_uid] = observed[allocation.function_uid].maximum( allocation.resources, ) - continue - if allocation.node_name not in node_map: - continue - used[allocation.node_name] += allocation.resources - used_slots[allocation.node_name] += 1 - if allocation.function_uid: - uid = allocation.function_uid - function_used[uid][allocation.node_name] += allocation.resources - function_slots[uid][allocation.node_name] += 1 - observed[uid] = observed[uid].maximum(allocation.resources) + elif allocation.node_name in node_map: + used[allocation.node_name] += allocation.resources + used_slots[allocation.node_name] += 1 + if allocation.function_uid: + uid = allocation.function_uid + function_used[uid][allocation.node_name] += allocation.resources + function_slots[uid][allocation.node_name] += 1 + observed[uid] = observed[uid].maximum(allocation.resources) total_function_used: defaultdict[str, Resources] = defaultdict(Resources) total_function_slots: defaultdict[str, int] = defaultdict(int) From f5b722fb196cb2edaaabdcb5cf445e86a2dd0b73 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 00:38:10 +0300 Subject: [PATCH 15/16] Use HealthFiles to mark state as ready in test --- tests/units/test_cli.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/units/test_cli.py b/tests/units/test_cli.py index bdaa875..e703a0b 100644 --- a/tests/units/test_cli.py +++ b/tests/units/test_cli.py @@ -8,6 +8,7 @@ from autofission import __version__ from autofission.cli import _create_gateway, build_parser, main from autofission.errors import ConfigurationError +from autofission.health import HealthFiles from tests.helpers import environment, function, node @@ -96,7 +97,8 @@ def test_probe_does_not_construct_client(tmp_path: Path, monkeypatch: pytest.Mon ) state = tmp_path / 'state' state.mkdir() - (state / 'ready').touch() + monkeypatch.setattr('autofission.health.time.time', lambda: 100) + HealthFiles(state).mark_ready() assert ( main( From a6d4dc84e4526db655ff917a49a9812d1346cbb6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9=20=D0=91=D0=BB?= =?UTF-8?q?=D0=B8=D0=BD=D0=BE=D0=B2?= Date: Wed, 26 Aug 2026 01:14:43 +0300 Subject: [PATCH 16/16] Reword README to improve clarity and accuracy --- README.md | 43 ++++++++++++++++++++++--------------------- 1 file changed, 22 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index 3cfa41a..e029d36 100644 --- a/README.md +++ b/README.md @@ -17,19 +17,18 @@ ![Autofission](https://raw.githubusercontent.com/pomponchik/autofission/develop/docs/assets/logo.svg) +Autofission turns a Kubernetes cluster's spare capacity into a dynamic scaling budget for Fission Functions. It estimates available CPU, memory, and Pod slots from Kubernetes scheduling data and independently updates the maximum replica limit (`MaxScale`) for each explicitly opted-in Function. -Imagine a Kubernetes cluster that runs a set of services but still has unused capacity. Rather than leaving those resources idle, you could fill them with useful, elastic work that continuously adapts to whatever CPU, memory, and Pod capacity remains available. +Imagine a Kubernetes cluster that runs a set of services but still has capacity not reserved by existing Pods' resource requests. Rather than leaving those resources idle, you could fill them with useful, elastic work that continuously adapts to the CPU, memory, and Pod slots that remain available. That opportunistic workload must also yield when the cluster is needed for something else. If you deploy another service, the elastic work should make room for it instead of turning spare-capacity use into permanent resource contention. Independent, disposable units of work are a good fit for this role: they can be packaged as AWS Lambda-like functions, and the workload can grow or shrink by changing how many function instances run at once. Managing those functions on Kubernetes requires a framework that deploys them, starts them on demand, and scales them. [Fission](https://fission.io/) provides that foundation. -Fission is excellent at deploying and scaling functions, but it is not designed to treat unused cluster capacity as a dynamic resource budget. Instead, it expects an operator to decide in advance how far each Function may scale; it does not derive that ceiling from the cluster's currently unused capacity. Set the ceiling too low and useful capacity remains idle; set it too high and Fission can ask the cluster to run more functions than it has room for. +Fission is excellent at deploying and scaling functions, but it is not designed to treat unused cluster capacity as a dynamic resource budget. Instead, it expects an operator to decide in advance how far each Fission Function may scale; it does not derive that ceiling from the cluster's currently unused capacity. Set the ceiling too low and useful capacity remains idle; set it too high and Fission can ask the cluster to run more Function instances than it has room for. That approach works when the capacity available to Fission is roughly constant. A shared cluster is rarely that static: services appear and disappear, new nodes join, and old nodes leave. An operator must therefore either dedicate a fixed amount of capacity to Fission and size every Function for that budget, or continually recalculate the Functions' limits as the rest of the cluster changes. -Autofission automates the second approach. It continuously estimates how much capacity remains available in the cluster and updates the scaling limit of each explicitly opted-in Function. This keeps Fission aligned with the cluster's changing spare resources without requiring manual retuning. - Autofission is designed for elastic, bare-metal, homelab, and edge clusters, where nodes come and go and idle compute should remain available to Functions without allowing them to preempt existing services. ## Table of Contents @@ -47,14 +46,14 @@ Autofission is designed for elastic, bare-metal, homelab, and edge clusters, whe ## Installation -Autofission is a single Python application exposed through the `autofission` CLI. It can run one reconciliation cycle and exit, or remain running and repeat the cycle at a configured interval. +Autofission is a Python controller with an `autofission` CLI. It can calculate and apply scaling limits once and exit, or remain running and repeat the update at a configured interval. Autofission can be installed in two ways: - As a Python package for local or one-shot runs. - As a Helm chart for continuous operation inside the cluster. -Both modes require an [existing Fission installation](https://fission.io/docs/installation/) and access to the target Kubernetes cluster. +Both modes require an [existing Fission installation](https://fission.io/docs/installation/) and access to the target Kubernetes cluster; only `newdeploy` Functions are supported. For the Python package option, run: @@ -75,11 +74,13 @@ helm upgrade --install autofission \ > ⓘ For reproducible deployments, add `--version VERSION`, replacing `VERSION` with the Autofission release number published on [PyPI](https://pypi.org/project/autofission/). The Python package and Helm chart use the same version number. -Functions managed by Autofission are background workloads. They should run only in space that regular services are not using. When a regular service needs that space, the Function Pods must move out of the way. +If you installed the chart, configure Fission to assign Function Pods the low-priority `autofission-runtime` class before opting in. The chart creates this class, but Fission must be told to use it. + +Managed Functions are background workloads: they should use only space that regular services do not need and yield it when needed. -Kubernetes has a built-in mechanism for this: a PriorityClass tells the scheduler which Pods are more important. The Autofission Helm chart creates a low-priority class named `autofission-runtime`. Kubernetes will never remove another workload to make room for a Function Pod using this class. If a regular service needs the space occupied by such a Function Pod, Kubernetes can stop the Function Pod and use its place for the service. +A PriorityClass tells Kubernetes which Pods are more important. With `autofission-runtime`, Function Pods cannot displace other workloads, while higher-priority Pods can displace them. -Autofission does not create Function Pods itself—Fission does. You must therefore tell Fission to assign `autofission-runtime` to every Function Pod it creates. Add the following block to the Helm values you already use for Fission: +Fission, not Autofission, creates these Pods. Add the following block to the Helm values you already use for Fission: ```yaml runtimePodSpec: @@ -90,12 +91,12 @@ runtimePodSpec: Then upgrade Fission with those values, using the same `helm upgrade` command you normally use to manage it. Instead of copying the block manually, you can pass the ready-made [values file](deploy/fission-values.yaml) alongside your other Fission values files. -CLI-only installations must provide an equivalent PriorityClass separately; pass its name with `--runtime-priority-class` to check existing Function Pods. See [Operations](#operations) for existing Fission workloads, [Configuration](#configuration) for credentials and resource-request settings, and [RBAC and security](#rbac-and-security) for required permissions. +If you use the Python package without the chart, provide an equivalent low-priority, non-preempting class and configure Fission to use it; Autofission does neither. Pass its name with `--runtime-priority-class` to check existing Function Pods. See [Operations](#operations) for existing Fission workloads, [Configuration](#configuration) for credentials and resource-request settings, and [RBAC and security](#rbac-and-security) for required permissions. ## Quick start -Before opting in, ensure that the Function resolves to positive CPU and memory requests after inheritance from its Environment; otherwise, Autofission rejects that Function. +Before opting in, ensure that the Function's final CPU and memory requests are greater than zero after its overrides are applied to Environment defaults; otherwise, Autofission rejects it. Replace `hello` and `default` below with the name and namespace of the Function you want to manage, then apply the label to opt it in: @@ -105,9 +106,9 @@ kubectl label function hello \ autoscaling.fission.io/cluster-capacity=true ``` -This example assumes Fission's default same-namespace workload placement. If Fission sets a separate `functionNamespace`, keep the Function namespace for the Function commands and use the workload namespace for the HPA command. +The commands in this section assume that Fission creates workload Pods in the Function's namespace. With a separate `functionNamespace`, use the Function namespace for `kubectl label` and `kubectl get function`, and the workload namespace only for `kubectl get hpa`. -For a one-shot CLI run, apply the label and then run `autofission --once`, adding `--context` or another credential-selection option when needed. For a Helm installation, wait for the controller's next successful update cycle. Then inspect the Function's `MaxScale`, the recorded calculation, and the HPA. A Helm-installed controller waits 15 seconds between cycles by default; processing and transient failures can add delay: +With the Python package, run `autofission --once` after applying the label; add `--context` or another credential-selection option when needed. With the chart, wait for the controller's next successful cycle. It waits 15 seconds between cycles by default, although processing and transient failures can add delay. Then inspect the Function's maximum replica limit (`MaxScale`), the recorded calculation, and the HPA: ```bash kubectl get function hello --namespace default \ @@ -124,7 +125,7 @@ Removing the label stops future management. Autofission deliberately does not gu ## How it works -Autofission manages only Functions that use Fission's [`newdeploy` executor](https://fission.io/docs/usage/function/executor/). Each such Function has a fixed maximum replica count, `MaxScale`, which Fission uses as the upper bound of that Function's [Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/concepts/workloads/autoscaling/). Autofission changes only `MaxScale`, along with informational annotations. It does not scale replicas itself: Fission's executor, HPA, and idle reaper still decide when each Function grows and shrinks. +Each managed Function uses Fission's [`newdeploy` executor](https://fission.io/docs/usage/function/executor/) and has a fixed maximum replica count, `MaxScale`, which Fission uses as the upper bound of that Function's [Horizontal Pod Autoscaler (HPA)](https://kubernetes.io/docs/concepts/workloads/autoscaling/). Autofission changes only `MaxScale`, along with informational annotations. It does not scale replicas itself: Fission's executor, HPA, and idle reaper still decide when each Function grows and shrinks. ```mermaid flowchart TD @@ -138,12 +139,12 @@ flowchart TD Given the same controller settings and Function, Environment, Node, and Pod data, each cycle produces the same result without unnecessary patches: 1. List opted-in Functions, along with Fission Environments, Kubernetes Nodes, and Pods. -2. Keep schedulable `Ready` nodes and subtract the CPU, memory, and Pod slots requested by their existing workloads. Active, unbound non-Function Pods are virtually placed after reclaiming Function Pods: an existing `status.nominatedNodeName` is honored, otherwise a deterministic best-fit placement is used. Pod requests follow Kubernetes scheduling semantics, including init containers, restartable sidecars, Pod-level requests, and Pod overhead. +2. Keep schedulable `Ready` nodes and record the CPU, memory, and Pod slots used by scheduled workloads. When placing active, unbound non-Function Pods, the model first treats capacity occupied by Function Pods as reclaimable, then uses `status.nominatedNodeName` when present or deterministic best-fit placement otherwise. Pod requests follow Kubernetes scheduling semantics, including init containers, restartable sidecars, Pod-level requests, and Pod overhead. 3. Resolve the resources required by one replica of each Function from its Function, Environment, fetcher, and observed runtime Pod configuration. -4. For each Function independently, estimate how many total replicas fit across the remaining per-node capacity, including replicas that are already running. +4. For each Function independently, estimate its total replica capacity from the per-node space left by other workloads; its own running replicas count toward that total instead of reducing it. 5. Update `MaxScale` to the calculated capacity, but never below `MinScale` or `1`. -The controller processes each Function independently, so an invalid or conflicting Function does not block the others. However, any global or per-Function error prevents that cycle from refreshing the readiness marker. A failed cycle does not immediately invalidate the marker: the previous successful marker remains fresh until the configured maximum age expires (`60` seconds by default). The readiness probe must then fail for its configured `failureThreshold` before Kubernetes reports the Pod as `NotReady`. +The controller processes each Function independently, so an invalid or conflicting Function does not block updates to valid Functions. Any global or per-Function error still prevents the cycle from refreshing readiness. The controller Pod remains ready while its latest successful cycle is within the configured maximum age (`60` seconds by default); afterward, Kubernetes marks it `NotReady` once the probe fails `failureThreshold` consecutive checks. ## Configuration @@ -181,7 +182,7 @@ The Python package does not install RBAC. Credentials selected for a CLI-only ru The chart does not grant permission to read Secrets, create or delete Functions, or mutate Pods, Nodes, Deployments, or Services. Other bindings attached to the same ServiceAccount can grant additional permissions. -In normal operation, the controller lists all Nodes, Pods, and Environments and asks the API only for label-selected Functions. It patches only opted-in Functions and changes only `MaxScale` and its calculation annotations. Kubernetes RBAC cannot enforce these restrictions: the ClusterRole authorizes listing every Function and patching any Function field. Function, Pod, and Environment specifications can contain literal environment-variable values. Node access supplies eligibility and allocatable capacity, Pod access accounts for existing workloads, and Environment access resolves inherited requests. +In normal operation, the controller lists all Nodes, Pods, and Environments and asks the API only for label-selected Functions. It patches only opted-in Functions and changes only `MaxScale` and its calculation annotations. Kubernetes RBAC cannot enforce these restrictions: the ClusterRole authorizes listing every Function and patching any Function field. Listing Function, Pod, and Environment specifications can expose literal environment-variable values. Node access supplies eligibility and allocatable capacity, Pod access accounts for existing workloads, and Environment access resolves inherited requests. By default, the container runs as UID/GID `65532` with a read-only root filesystem. It drops all Linux capabilities, blocks privilege escalation, and uses a `RuntimeDefault` seccomp profile. The chart creates a NetworkPolicy with an empty ingress list; it does not restrict egress. Enforcement requires a compatible network plugin, and other NetworkPolicies can add allowed ingress because Kubernetes combines their rules. @@ -192,7 +193,7 @@ Treat permission to set the opt-in label as permission to consume the cluster's ## Operations -The chart runs one replica with a `Recreate` strategy, preventing overlap during Deployment-managed rollouts. This rollout behavior does not guarantee a single active controller at all times because Autofission has no leader election. After a node or cluster restart, the Deployment restores its controller replica and Autofission rebuilds its state from the API. +The chart runs one replica with a `Recreate` strategy, avoiding overlap in Deployment-managed rollouts. This is not a distributed lock: without leader election, Autofission cannot guarantee a single active controller under every failure scenario. After a node or cluster restart, the Deployment restores its controller replica and Autofission rebuilds its state from the API. Function patches include a `resourceVersion` precondition. If a Function changes after Autofission reads it, the patch fails with a conflict instead of overwriting the concurrent change; the daemon retries the Function during its next cycle. @@ -236,7 +237,7 @@ Apply any required controller overrides described under [Configuration](#configu ## Compatibility and limitations -- Only Fission `newdeploy` Functions are managed. `poolmgr`, empty executor, and `container` are rejected as opt-in configuration errors. +- Only Fission `newdeploy` Functions are managed. `poolmgr`, an empty executor value, and `container` are rejected as opt-in configuration errors. - Autofission calculates the full capacity that each managed Function could use by itself. This preserves burst capacity, but simultaneous cold bursts can leave Pods `Pending`. Later cycles account for Pods from other Functions after they are scheduled and may reduce the limits; Autofission is not a fairness scheduler. - With at least one eligible node, modeled capacity below one replica produces `MaxScale=1` because Fission and Kubernetes require a positive HPA maximum. An explicit `MinScale` is also honored even when it exceeds currently free capacity. With no eligible nodes, reconciliation fails and leaves existing limits unchanged. - The [Fission v1 API](https://fission.io/docs/reference/crd-reference/) is tested end to end with Fission `1.27.0` on Kubernetes `1.34`; the Fission `1.27.0` chart requires Kubernetes `1.32` or newer. Other version combinations are untested. Environment resource inheritance follows Fission's override semantics and is covered by unit tests. @@ -252,7 +253,7 @@ Apply any required controller overrides described under [Configuration](#configu ## Troubleshooting -`Autofission is NotReady` — inspect controller logs. A `403` usually indicates missing custom RBAC or a ServiceAccount mismatch. A `409` means the Function changed after it was listed; the daemon retries it on the next cycle, while a `--once` run must be repeated. Other errors name the Function where possible. +`Autofission is NotReady` — for the default Helm release, run `kubectl logs deployment/autofission --namespace fission`; adjust the release name and namespace if needed. A `403` usually indicates missing custom RBAC or a ServiceAccount mismatch. A `409` means the Function changed after it was listed; the daemon retries it on the next cycle, while a `--once` run must be repeated. Other errors name the Function where possible. `The calculated limit is smaller than expected` — check cordons, `Ready` status, taints, Pod requests, Pod slots, fetcher values, and per-node fragmentation. Capacity cannot combine spare CPU and spare memory located on different nodes.