Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion modules/ROOT/pages/kubernetes/gke.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@

https://cloud.google.com/kubernetes-engine[https://cloud.google.com/kubernetes-engine{external-link-icon}^]

Autopilot clusters are not suported, as the xref:secret-operator:index.adoc[secret-operator] requires special privileges that are not granted in Autopilot clusters.
Autopilot clusters are not supported, as the xref:secret-operator:index.adoc[secret-operator] requires special privileges that are not granted in Autopilot clusters.

Other than that no special steps are needed.
2 changes: 1 addition & 1 deletion modules/ROOT/pages/release-guide.adoc
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
:description: Work-in-progress internal guide with the git and cargo steps to cut a release and bump the next development version.


(this is work in progress that is why it's not explicitely linked in nav.doc)
(this is work in progress that is why it's not explicitly linked in nav.doc)

Prerequisites:

Expand Down
2 changes: 1 addition & 1 deletion modules/ROOT/partials/release-notes/release-22.9.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The following new major platform features were added:

===== OpenShift compatibility

We have made continued progress towards OpenShift compability, and the following operators can now be previewed on OpenShift.
We have made continued progress towards OpenShift compatibility, and the following operators can now be previewed on OpenShift.
Further improvements are expected in future releases, but no stability or compatibility guarantees are currently made for OpenShift clusters.

* https://github.com/stackabletech/airflow-operator/pull/127[Apache Airflow]
Expand Down
2 changes: 1 addition & 1 deletion modules/ROOT/partials/release-notes/release-24.3.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ The `spec.version` field has been removed.

The `spec.mode` field is now required and must be set to `cluster`.

The `spec.mainClass` field is now required and must point to a location on ths file system or S3 where the main class is located.
The `spec.mainClass` field is now required and must point to a location on the file system or S3 where the main class is located.
====

* https://github.com/stackabletech/spark-k8s-operator/pull/355[Remove usage of `userClassPathFirst` properties]
Expand Down
4 changes: 2 additions & 2 deletions modules/ROOT/partials/release-notes/release-24.7.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ Use "stackablectl release list" to list available releases.
Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`.

NOTE: The cluster name for the hello-world operator has been changed in this release so the CRD cannot be patched in-place. For this reason in the snipets below the CRD for this operator will be subject to a `delete` command (plus an `apply` as part of the operator rollout in the new release) instead of a `replace`.
NOTE: The cluster name for the hello-world operator has been changed in this release so the CRD cannot be patched in-place. For this reason in the snippets below the CRD for this operator will be subject to a `delete` command (plus an `apply` as part of the operator rollout in the new release) instead of a `replace`.

[source]
----
Expand Down Expand Up @@ -406,5 +406,5 @@ spec:
IMPORTANT: Do not override this property for the 1.27 cluster version.

This is necessary because the 2.x versions do not support the XML format for flow definitions anymore.
Support for the JSON format has been addded in version 1.16 and both formats have been maintained up to (excluding) version 2.0.
Support for the JSON format has been added in version 1.16 and both formats have been maintained up to (excluding) version 2.0.
The next SDP release 24.11 will automatically take care of this step for you.
2 changes: 1 addition & 1 deletion modules/compliance/pages/licenses.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,4 @@ https://github.com/stackabletech/stackablectl/blob/main/LICENSE[License{external
https://github.com/stackabletech/docker-images/blob/main/LICENSE[License{external-link-icon}^] for the product Docker images.

The Docker images are built on the https://catalog.redhat.com/software/containers/ubi9-minimal/61832888c0d15aff4912fe0d[Red Hat ubi9-minimal base image{external-link-icon}^].
It is https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf[licensed seperately{external-link-icon}^].
It is https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf[licensed separately{external-link-icon}^].
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The core artifacts of the Stackable Data Platform are container images of Kubern

== Images overview

Every operator is packaged into its own image and every product is also packaged into its own, seperate image.
Every operator is packaged into its own image and every product is also packaged into its own, separate image.
Products that require multiple different processes to run, such as a coordinator and a worker, still only run off of one image;
usually these products also only provide a single artifact that is used to run all processes.

Expand Down
6 changes: 3 additions & 3 deletions modules/concepts/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

== Overview

The xref:overview.adoc[Platform overview] is a good starting point to understand the Stackable Data Platform covering the overall architecture, deployement and configuration.
The xref:overview.adoc[Platform overview] is a good starting point to understand the Stackable Data Platform covering the overall architecture, deployment and configuration.

== General configuration mechanisms

Expand All @@ -19,8 +19,8 @@ Learn about how to access xref:experimental-arm64-support[ARM64-support].
== Connectivity

Many Platform components depend on other components or expose functionality that you can connect to.
This connectivity is achived with xref:service-discovery.adoc[service discovery ConfigMaps].
To access your Stackable operated products from outside the Kuberenetes cluster learn more about xref:service-exposition.adoc[].
This connectivity is achieved with xref:service-discovery.adoc[service discovery ConfigMaps].
To access your Stackable operated products from outside the Kubernetes cluster learn more about xref:service-exposition.adoc[].

== Security

Expand Down
2 changes: 1 addition & 1 deletion modules/concepts/pages/overrides.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ All override property values must be strings.

For a role or role group, at the same level of `config`, you can specify `podOverrides` for any of the attributes you can configure on a Pod.
Every Stacklet contains one or more StatefulSets, DaemonSets, or Deployments, which in turn contain a Pod template that is used by Kubernetes to create the Pods that make up the Stacklet.
The `podOverrides` allow you to specify a fragment of a Pod template that is then overlayed over the one created by the operator.
The `podOverrides` allow you to specify a fragment of a Pod template that is then overlaid over the one created by the operator.

An example for an HDFS cluster looks as follows:

Expand Down
2 changes: 1 addition & 1 deletion modules/concepts/pages/s3.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ spec:
==== Stand-alone resources

To reuse S3Buckets across different applications, they can be defined as stand-alone resources. In a similar fashion, S3Connections can also be defined as stand-alone resources.
In the example below, ony bucket is used by a DruidCluster and a second bucket is used by both a TrinoCluster and a SparkCluster. Both buckets reference the same S3Connection.
In the example below, one bucket is used by a DruidCluster and a second bucket is used by both a TrinoCluster and a SparkCluster. Both buckets reference the same S3Connection.

image::s3-fully-separated.drawio.svg[One S3Connection is referenced by two different S3Buckets. The first Bucket is referenced by a DruidCluster and the second bucket is referenced by a SparkCluster and TrinoCluster. No object is inlined.]

Expand Down
4 changes: 2 additions & 2 deletions modules/concepts/pages/service-discovery.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ The ConfigMap should look something like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: my-already-exisiting-hdfs
name: my-already-existing-hdfs
data:
core-site.xml: |
<here should be your core-site.xml file contents>
Expand All @@ -105,7 +105,7 @@ kind: HbaseCluster
metadata:
name: simple-hbase
spec:
hdfsConfigMapName: my-already-exisiting-hdfs
hdfsConfigMapName: my-already-existing-hdfs
...
----

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ As currently all participants are German, the first choice would be German, howe

== Decision Outcome

Chosen option: "English", because the desire to create an active community with participants from all over the world is much more important than the minor barrier that this decision might create initally.
Chosen option: "English", because the desire to create an active community with participants from all over the world is much more important than the minor barrier that this decision might create initially.

=== Positive Consequences

Expand All @@ -51,4 +51,4 @@ Chosen option: "English", because the desire to create an active community with
=== English

* Good, because it allows growing an international community
* Bad, would initially add a small language barrier that is not strictly speaking necessary
* Bad, would initially add a small language barrier that is not strictly speaking necessary
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Chosen option: "Review then commit", because it meets both decision drivers.

=== Review then commit

This option requires a full review and approval of all changes before they are commited to the development branch.
This option requires a full review and approval of all changes before they are committed to the development branch.
Who and how many people need to approve a change will need to be defined in the contribution guidelines at a later time.

* Good, because there is no need to keep track of unreviewed commits
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,6 @@ are injected into pods using a custom CSI provider or init container.
* Good, because it can enforce policy for generated identities (for example: tying TLS certificate to Pod or Node identity)
* Good, because it can pick pregenerated tokens based on Pod/Node identity
* Good, because it can have a cluster-global policy for picking between backends
* Good, because it can accomodate to whatever authn methods we end up supporting
* Good, because it can accommodate to whatever authn methods we end up supporting
* Bad, because it's another bespoke thing to maintain and develop
* Bad, because none of us are experienced with writing CSI providers
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Technical Story: https://github.com/stackabletech/issues/issues/208[#208]
== Context and Problem Statement

One of the main focus areas of Stackable is making components of the stack work together well.
This requires a high degree of composability where one component refers to another compenent in its definition.
This requires a high degree of composability where one component refers to another component in its definition.
While this is a simple concept in principle it requires that operators obtain information about the object that is referred to and inject this information into the configuration of the object that refers to it.
An example for this is a NiFi cluster definition referring to a ZooKeeper cluster definition via a znode name:

Expand Down Expand Up @@ -74,7 +74,7 @@ The target config would then be written either to a ConfigMap or a Secret, depen
* Bad, because it would allow referencing ConfigMaps and Secrets in different namespaces, which would dilute namespace separation
* Bad, because it makes debugging startup issues of products much harder, as pods will simply not appear if required ConfigMaps are missing, effectively bypassing well established k8s patterns (like a pod missing a mount not starting and writing events about why it is not starting)
* Bad, because the dependencies between pods and configmaps / secrets become very complex and hard to debug as they are not modelled in any way as k8s dependencies (mounts)
* Bad, because setting up the correct watches in the operator becomes very complex, with a likelyhood of inadvertently forgetting to add a new watch that becomes necessary as CRDs evolve
* Bad, because setting up the correct watches in the operator becomes very complex, with a likelihood of inadvertently forgetting to add a new watch that becomes necessary as CRDs evolve
* Bad, because it potentially propagates sensitive values to multiple places by copying secret content to additional places
* Bad, because the Operator is seeing the Secrets. This is not a problem _now_, but if we could avoid it that would be better

Expand All @@ -86,4 +86,4 @@ Accessing the content of these mounted objects would then need to be done in an

* Good, because mounting the ConfigMap/Secret directly into the pod allows the restart controller to automatically restart a Pod if a mounted object changes. Otherwise, changes would not be propagated or the Operator would have to watch the ConfigMap/Secret for changes itself. The watching/restarting is done once by Kubernetes + the restart controller and makes our Operators simpler
* Bad, because the mounted properties cannot be validated by the Operator. Although Kubernetes at least verifies that all the properties that should be mounted exist
* Bad, because it is more difficult to see which config is actually used to run the product, as the actual config is only finally assembled inside of the container
* Bad, because it is more difficult to see which config is actually used to run the product, as the actual config is only finally assembled inside of the container
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ spec:
...
----

In the example above, the application will log its events to the bucket specifed in `logFileDirectory`.
In the example above, the application will log its events to the bucket specified in `logFileDirectory`.
In addition, the application processes data from a S3 bucket configured within the `s3connection` section of the specification.

The operator will read the `s3Connection` and set up the `fs.s3a.aws.credentials.provider` and co (endpoint, accesskey, secretkey, tls, path-style access - basically all attributes of S3Connection) settings.
Expand All @@ -149,6 +149,6 @@ NOTE: the credentials used by the `HistoryServer` *do not* have to be shared wit

=== Advantages

* Fully flexible solution, which allows the logDir to be on a different S3 ednpoint than the data.
* Fully flexible solution, which allows the logDir to be on a different S3 endpoint than the data.
* If they are on the same endpojnt, a single S3BucketDef can be shared between HistoryServer and SparkApplication for ease of use.
* HDFS and/or other distributed filesystems can be added non-breaking later on
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Technical Story: https://github.com/stackabletech/listener-operator/pull/1
== Context and Problem Statement
// Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question.

Eventually, the products we host in Kubernetes will need to be accessed from outside of the cluster, as this is where the client is. Our current solution for this is NodePort services. They are a simple and common solution for on-premise clusters, where nodes are reachable hosts in the local network. To get traffic into a Kubernetes cluster that runs in a public cloud, NodePorts do not work; instead LoadBalancers are the preferred solution.
Eventually, the products we host in Kubernetes will need to be accessed from outside of the cluster, as this is where the client is. Our current solution for this is NodePort services. They are a simple and common solution for on-premise clusters, where nodes are reachable hosts in the local network. To get traffic into a Kubernetes cluster that runs in a public cloud, NodePorts do not work; instead LoadBalancers are the preferred solution.

While a Pods name is stable across restarts and rescheduling, the IP of the NodePort can change if a Pod is rescheduled to a different node. This means that external addresses from simple NodePorts are not stable. LoadBalancers are not tied to nodes, but they are often not available in on-prem clusters.
At the moment we deploy NodePort Services per RoleGroup; clients cannot connect to an individual Pod in a RoleGroup.
Expand All @@ -26,7 +26,7 @@ Additionally, Pods currently do not know the address under which they are reacha
Problems:

* **Unstable addresses** - Clients need stable addresses to connect to, but Kubernetes can move pods around. While the discovery ConfigMap is updated, it's not feasible to ask the client to pull the new info from there every time, clients will want to use static config files with static addresses to connect to.
* **Replicas not addressable** - In our current setup, there's no way to connect to a specific replica in a StatefulSet or Deployement - which is necessary for cases like the data nodes of HDFS.
* **Replicas not addressable** - In our current setup, there's no way to connect to a specific replica in a StatefulSet or Deployment - which is necessary for cases like the data nodes of HDFS.
* **Pods don't know their outside address** - The hostname and IP that the pods know about themselves is from _inside_ the cluster. The IP only works inside the overlay network. This means ProductCluster processes cannot link to other nodes of the cluster.

== Decision Drivers
Expand Down Expand Up @@ -72,7 +72,7 @@ spec:
serviceAnnotations:
networking.gke.io/load-balancer-type: Internal

ListenerClasses allow for various different ways of getting outside traffic into the cluster. A dedicated operator seperates the deployment of this out - the product operators need to only request the listeners.
ListenerClasses allow for various different ways of getting outside traffic into the cluster. A dedicated operator separates the deployment of this out - the product operators need to only request the listeners.

Requests for Listeners are made through annotated volume claim templates:

Expand Down Expand Up @@ -188,4 +188,4 @@ With ARP, the LoadBalancers appear as "real" IP addresses in the same subnet as

=== Calico

link:https://www.tigera.io/project-calico/[Calico] requires BGP, another component that we cannot make required for customer setups.
link:https://www.tigera.io/project-calico/[Calico] requires BGP, another component that we cannot make required for customer setups.
2 changes: 1 addition & 1 deletion modules/contributor/pages/adr/ADR026-affinities.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ spec:

==== Pros

* Enables definining only one of the two structs an the CRD
* Enables defining only one of the two structs an the CRD

==== Cons

Expand Down
4 changes: 2 additions & 2 deletions modules/contributor/pages/adr/ADR029-database-connection.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -141,13 +141,13 @@ spec:
database:
postgresql:
host: druid-postgresql # mandatory
port: 5432 # defaults to some port number - depending on wether tls is enabled
port: 5432 # defaults to some port number - depending on whether tls is enabled
schema: druid # defaults to druid
credentials: druid-postgresql-credentials # mandatory. key username and password
parameters: {} # optional
redis:
host: airflow-redis-master # mandatory
port: 6379 # defaults to some port number - depending on wether tls is enabled
port: 6379 # defaults to some port number - depending on whether tls is enabled
schema: druid # defaults to druid
credentials: airflow-redis-credentials # optional. key password
parameters: {} # optional
Expand Down
4 changes: 2 additions & 2 deletions modules/contributor/pages/adr/ADR032-oidc-support.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ Useful information to consider in the decision.
=== OIDC clients and product-client-mappings

To configure a product to use OIDC, it needs information about where to find the OIDC provider (typically the URL of the discovery endpoint), as well as client credentials to authenticate with the provider.
The connection information is generic and shared between all clients, but the crendentials are product specific.
The connection information is generic and shared between all clients, but the credentials are product specific.

It is best practice to have one client per connecting product.
This allows the user to define exactly from which hosts the client can be used, as well as valid redirect URLs.
Expand Down Expand Up @@ -85,7 +85,7 @@ Does not support reading an OIDC discovery URL but requires:

* separate API base URL, auth URL, token URL
* the `email`, `profile` and `openid` scopes
* that the auth and token URL are supplied at the `.well-known` endoint, but the base URL is not
* that the auth and token URL are supplied at the `.well-known` endpoint, but the base URL is not

We could opt to implement proper OIDC support using fab-oidc. This however needs maintenance work from us.

Expand Down
Loading