helm/2517 - Deis on OpenShift

[kingdonb]

I tried to launch Workflow on OpenShift and I have followed from helm/helm#2517 where it was explained how Helm can be used on OpenShift

kingdonb$ helm install --tiller-namespace myproject -n deis --namespace myproject workflow/
Error: release deis failed: User "system:serviceaccount:myproject:default" cannot create daemonsets.extensions in project "myproject"

Is this an issue with OpenShift or with Deis? My reading is that there is a resource kind that does not exist on OpenShift yet, that Deis leveraged on a high version of Kubernetes...

[kingdonb]

Here's what I've done:

minishift start
...
oc login -u system:admin
kubectl -n myproject create sa tiller
kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=myproject:tiller
helm --tiller-namespace myproject --service-account tiller init
oc policy add-role-to-user admin -z tiller -n myproject
helm install --tiller-namespace myproject -n deis --namespace myproject workflow/

which is the most permissive role I can imagine, still only to get

Error: release deis failed: User "system:serviceaccount:myproject:tiller" cannot create daemonsets.extensions in project "myproject"

I followed this, because I understood that OpenShift role based security has essentially merged with standard kube RBAC, so I guessed that either set of permissions would work. I don't really want to set the most permissive possible though, I'd love to understand why this just doesn't work.

https://gist.github.com/mgoodness/bd887830cd5d483446cc4cd3cb7db09d

[kingdonb]

I get the same results with https://docs.bitnami.com/kubernetes/how-to/configure-rbac-in-your-kubernetes-cluster/

$ helm install -n deis --namespace myproject workflow/
Error: release deis failed: User "system:serviceaccount:kube-system:tiller" cannot create daemonsets.extensions in project "myproject"

[bacongobbler]

Did you grant tiller RBAC permissions to create objects in the "myproject" namespace?

That gist you posted earlier is only relevant if tiller is installed in the kube-system namespace. It looks like from your commands you deployed tiller in the "myproject" namespace.

[bacongobbler]

openshift/origin#8242

[kingdonb]

Oh!

The OpenShift devs helped me out, two things I'm doing wrong...

1. When a tutorial says to use -z in oc policy, it's a shortcut for system:serviceaccount:[-n]:ARG but this is new in 3.5, and I had an old oc client 3.4.1.2 hanging around in my path -- why I thought it would be OK to drop the -z I don't know, but in the end I sometimes got errors telling I couldn't even list the namespace, which I was supposedly granting myself admin role for. I didn't have the permission applied to the actual serviceaccount properly.

I could see this was the case by looking at oc policy who-can ... but it was not so easy to figure this out in the documentation for OpenShift, unfortunately, and I didn't realize until later I had an old oc binary.

2. What you just found there... giving the SA an admin role is not enough, because admin role is not allowed to handle daemonsets for some reason. You can give the cluster-admin role though:

So this didn't work: oc adm policy add-cluster-role-to-user cluster-admin tiller -n myproject

but this did work:

oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:myproject:tiller -n myproject

[bacongobbler]

Did you make sure to install Workflow with RBAC permissions? You'll need to add --set global.use_rbac=true in your helm install invocation to install Workflow with RBAC permissions. See #812.

[bacongobbler]

Not sure about the other issues. We don't officially support Workflow on Openshift, but please continue to post here so others can learn.

[kingdonb]

You have been so helpful. Thank you! I did not do that.

[kingdonb]

This big "MISSING" at the bottom looks kind of ominous... those are my RBAC roles, and I'm guessing that Missing means they're using a resource kind that I don't have on my cluster. Maybe I would have better luck with a newer OpenShift version...

$ helm install --tiller-namespace myproject -n deis --namespace myproject workflow/
NAME:   deis
LAST DEPLOYED: Thu Aug 24 14:03:09 2017
NAMESPACE: myproject
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                   TYPE    DATA  AGE
minio-user             Opaque  2     12s
deis-router-dhparam    Opaque  1     12s
objectstorage-keyfile  Opaque  2     12s

==> v1/ConfigMap
NAME                  DATA  AGE
dockerbuilder-config  2     12s
slugbuilder-config    2     12s
slugrunner-config     1     12s

==> v1/ServiceAccount
NAME                   SECRETS  AGE
deis-builder           2        12s
deis-controller        2        12s
deis-database          2        12s
deis-logger-fluentd    2        12s
deis-logger            2        12s
deis-minio             2        12s
deis-monitor-telegraf  2        12s
deis-nsqd              2        12s
deis-registry          2        12s
deis-router            2        12s
deis-workflow-manager  2        12s

==> v1/Service
NAME                    CLUSTER-IP      EXTERNAL-IP                    PORT(S)                                                   AGE
deis-builder            172.30.114.189  <none>                         2222/TCP                                                  11s
deis-controller         172.30.141.36   <none>                         80/TCP                                                    11s
deis-database           172.30.152.48   <none>                         5432/TCP                                                  11s
deis-logger             172.30.117.93   <none>                         80/TCP                                                    11s
deis-minio              172.30.27.13    <none>                         9000/TCP                                                  11s
deis-monitor-grafana    172.30.184.206  <none>                         80/TCP                                                    11s
deis-monitor-influxapi  172.30.160.104  <none>                         80/TCP                                                    11s
deis-monitor-influxui   172.30.71.72    <none>                         80/TCP                                                    11s
deis-nsqd               172.30.41.90    <none>                         4151/TCP,4150/TCP                                         10s
deis-logger-redis       172.30.182.56   <none>                         6379/TCP                                                  10s
deis-registry           172.30.100.247  <none>                         80/TCP                                                    10s
deis-router             172.30.141.156  172.29.232.160,172.29.232.160  80:31300/TCP,443:30075/TCP,2222:30158/TCP,9090:32519/TCP  10s
deis-workflow-manager   172.30.208.185  <none>                         80/TCP                                                    10s

==> v1beta1/DaemonSet
NAME                   DESIRED  CURRENT  READY  UP-TO-DATE  AVAILABLE  NODE-SELECTOR  AGE
deis-logger-fluentd    0        0        0      0           0          <none>         10s
deis-monitor-telegraf  0        0        0      0           0          <none>         10s
deis-registry-proxy    0        0        0      0           0          <none>         10s

==> v1beta1/Deployment
NAME                   DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
deis-builder           1        1        1           0          10s
deis-controller        1        0        0           0          10s
deis-database          1        1        1           0          10s
deis-logger            1        1        1           0          9s
deis-minio             1        1        1           0          9s
deis-monitor-grafana   1        1        1           1          9s
deis-monitor-influxdb  1        1        1           0          9s
deis-nsqd              1        1        1           0          9s
deis-logger-redis      1        1        1           0          9s
deis-registry          1        1        1           0          9s
deis-router            1        1        1           0          8s
deis-workflow-manager  1        1        1           1          8s

==> MISSING
KIND                   NAME
clusterroles           deis:deis-builder
clusterroles           deis:deis-controller
clusterroles           deis:deis-logger-fluentd
clusterroles           deis:deis-router
clusterrolebindings    deis:deis-builder
clusterrolebindings    deis:deis-controller
clusterrolebindings    deis:deis-logger-fluentd
clusterrolebindings    deis:deis-router
roles                  deis-builder
roles                  deis-monitor-telegraf
roles                  deis-router
rolebindings           deis-builder
rolebindings           deis-monitor-telegraf
rolebindings           deis-router

[kingdonb]

... It is true, the OpenShift 3.4 security model had not yet started to merge with K8S RBAC because K8S was not fully baked when 3.4 was developed and released. That doesn't seem to be my problem anymore though.

In 3.5 and 3.6, this should work fine. I'll test it out and add notes when I find out. If someone needs to use 3.4, there is probably a way to apply the permissions to the service accounts, but in my case it looks like the only reason I'm having problems is that I have an old oc OpenShift client binary.

[kingdonb]

Almost... there is a bridge for roles (OpenShift has its own role kinds that predate Kubernetes formal implementation of RBAC), but the connector goes the wrong way for what I'm trying to do:

in 3.6, we go openshift -> rbac, *not* rbac to openshift. If you create an RBAC object, it will be stomped away

So if you have OpenShift roles implemented, and you want to access them as Kubernetes roles, you can do it, but if you try the reverse you will know it didn't work because helm very helpfully tells you the state of these resources you created after install is MISSING. It means just what it says.

There is supposedly a raw and fairly untested tool for doing the conversion of those resources and we are going to try it out against the output of helm --debug

[kingdonb]

these roles/bindings:
https://gist.github.com/enj/078cc916365a7c3db6da581ea4485bd6

and some combination of these scc settings:

# (for tiller in kube-system namespace, cluster-admin because daemonsets)
oc policy add-role-to-user edit system:serviceaccount:kube-system:default
oc adm policy add-cluster-role-to-user cluster-admin system:serviceaccount:kube-system:default

# for your deis roles, you don't need to use system:authenticated
# but I was having trouble making sure my policies were actually
# getting applied where I thought they were
oc adm policy add-scc-to-group anyuid system:authenticated
oc adm policy add-scc-to-group privileged system:serviceaccounts:NAMESPACE

will get you most of the way there, in the worst possible way... this still didn't quite work for me, most of the containers came up, but a few still didn't. It's a bit mysterious, but I'll try again today.

Controller for instance, which was complaining about not being able to create a directory /app/data, but why wasn't clear. (anyuid policy should have fixed that, adding that setting I was able to start pods and see them come up as root via kubectl. Another security feature of OpenShift is that containers are not allowed to start as root unless you specifically say they are allowed to.)

I'm saying this is the worst possible way because it's basically turning off all of the security, but I'll work on tightening it up for the PR once I figure out what's still missing!

[kingdonb]

Hm. Today on a clean minishift, it works! I believe with no more information than I already had posted here. Waiting for Builder to come up...

NAME                                     READY     STATUS    RESTARTS   AGE
deis-builder-904459970-rwms8             0/1       Running   1          2m
deis-controller-198627737-jqq9m          1/1       Running   3          2m
deis-database-4120584427-kp6l8           1/1       Running   0          2m
deis-logger-2717637750-l9g60             1/1       Running   2          2m
deis-logger-fluentd-zcz1d                1/1       Running   0          2m
deis-logger-redis-1281502559-srbsz       1/1       Running   0          2m
deis-minio-2522384958-kc60z              1/1       Running   0          2m
deis-monitor-grafana-1204143830-q77cx    1/1       Running   0          2m
deis-monitor-influxdb-2842921881-91s7x   1/1       Running   0          2m
deis-monitor-telegraf-dt6w3              1/1       Running   0          2m
deis-nsqd-1225249577-v1lf1               1/1       Running   0          2m
deis-registry-3907297837-msgxm           1/1       Running   2          2m
deis-registry-proxy-74ljd                1/1       Running   0          2m
deis-router-3453227723-x352n             1/1       Running   0          2m
deis-workflow-manager-3757950027-s2d00   1/1       Running   0          2m

$ deis register deis.deis:32423
username: yebyen
password:
password (confirm):
email: yebyen@gmail.com
Registered yebyen
Logged in as yebyen
Configuration file written to /Users/kingdonb/.deis/client.json

... there are lots of things to work out after that, but it looks like I can use Deis Workflow on MiniShift now! That's great news, I love interchangeable parts.

[kingdonb]

I have now deployed Workflow on OpenShift Origin as well as Minishift, and tackled some of the additional challenges that entails.

I think mainly, the struggle came because this time I did not grant any privileges to ambiguous any-user 'system:authenticated' and made it a point to trace down issues individually through each of the logs, so I would know what permissions I was granting, and what wouldn't work until they were granted. (Most things just worked on minishift as I granted the more permissive settings broadly.)

They were all some subset of these serviceaccounts that required additional 'scc' permissions, which is another OpenShift thing.

... bad example deleted

My search history seems to indicate I also needed

setenforce 0

at some point, or for some reason.

These pods all needed cluster-admin I think mostly because on OpenShift, you can only enumerate namespaces if you have cluster-admin. (Edit: Perhaps there is a different endpoint that users can query to get all of their own namespaces, or projects as they're sometimes called in OpenShift)

I'm not sure how we can provide CI validation for this for a release,

I have cleaned my rough work up further into almost draft shape for a pull request, I am just running the deployment again onto my OpenShift/CentOS VM now to ensure the chart with rewritten roles, and a new value in values.yaml is still working as expected after I have adjusted it to fit within the template.

Maybe a ten-minute YouTube video that starts from vagrant destroy and skipping up to the part where you install docker onto your CentOS 7.3 machine, will suffice for the purposes of validating the release as "Experimental OpenShift Support" with prominent notice marking it as such. Not sure if you will want to merge this at all when it's done, actually I foresee probably not...

I predict a fork that is committed to supporting Deis Workflow on OpenShift and tracking maintenance as new Kube releases are tracked against OpenShift 😁 the one thing I have not yet figured out is what will allow telegraf to collect metrics, as I was able to communicate with InfluxDB from the logged-in grafana, and read the basic InfluxDB stats, and the stats from router; the interface worked but no CPU or Memory statistics were gathered, and I gather that is telegraf's job.

[kingdonb]

I packed the changes up into one repo so they can be reviewed at-once, but

https://github.com/kingdonb/openshift-deis/commits/openshift-deis-v2.17

I will unpack them into PRs on the affected deis/* repos if there's any interest in merging them back. I'll have to tear down CentOS VM and do from scratch to ensure I got all of the oadm policy things in. Presumably there is also some kind of declarative way I can set those, though maybe not via helm.

I'll also need to touch base with someone else who has helm working on OpenShift realistically to get some kind of approval or review on this branch, since that is also not especially trivial or well documented. But it works! This can deploy Deis Workflow into a scratch project on OpenShift.

I'll try hooking it up to #857 next, since I personally will need both.