Separating GHC's CI from the forge (or: woodpecker)

[hasufell]

Rendered: https://github.com/hasufell/tech-proposals/blob/woodpecker/proposals/0000-woodpecker.md

[Bodigrim]

As an occasional small-scale contributor to GHC, I can witness that the existent CI is counterproductive, especially if you are not looking to spend the next week babysitting it (which might be not too bad for the core team, who hang around working on other patches anyway, but terribly disheartening for others). And that's despite years of efforts from GHC team and Bryan. My view is that improving CI experience is vital for GHC's long-term sustainability and gaining new contributors.

[chreekat]

I agree with this direction.

One thing that may need more explicit mention is the unsuitability of GHC pipelines, themselves. I think everyone agrees that we do not want a 1-1 mapping of (current GitLab pipelines) to (future Woodpecker workflows). Certainly not in the fail-always state they're currently in¹.

To be clear, I believe this is already implied in the plan, and it would fall out naturally from an iterative move. But there's a lot of work and coordination hidden in there that might be worth describing.

As one small example, moving away from ci-images might not be necessary for a move to Woodpecker, but it sure would make the migration a lot less painful than it needs to be.

Footnotes

1. I would refer to my "CI Success Stats" dashboard, but it looks like it got broken by some infrastructure change on the GitLab server. ↩

[chreekat]

Suggested change

	- a full time devops paid by the Haskell Foundation, where part of the responsibilities were GitLab maintenance
	- a 1-day-a-week devops paid by the Haskell Foundation, where part of the responsibilities were GitLab maintenance

[hasufell]

I think everyone agrees that we do not want a 1-1 mapping of (current GitLab pipelines) to (future Woodpecker workflows). Certainly not in the fail-always state they're currently in.

Sure, but we have to be mindful about scope-creep here.

My idea would be to migrate a relatively small part, utilize both docker and VM backends and then see how things fall out at the end.

We can still consider this an experiment and say afterwards "oh well, maybe it won't work". So my idea would be to make this time and task bound. If we burn 1-2 person months on it and realize it's a failure... it's still not too bad and the insights could inform future efforts/decisions on possibly a different CI migration.

[hasufell]

Something I maybe have to make more clear in the document too is that I believe we have to avoid the following at all cost:

• manually provisioned VMs
• manually provisioned bare metal boxes (e.g. macOS)

A tangential discussion has come up on ghc-dev and some other issue trackers recently and I believe that we have to reconsider the role of CI as a whole: CI shouldn't be every developer's testsuite executor. It must be easy to run CI locally in any environment. And I believe the docker/libvirt approach could potentially deliver this.

Woodpecker supports local ad-hoc execution via e.g.:

woodpecker-cli exec --backend-engine docker .woodpecker/my-workflow.yaml

[MangoIV]

I really like this proposal. I think the two things I support strongest and like best are:

1. de-risking our deployments by splitting up our deployment in systems that can be maintained (and replaced) separately.
2. carefully evaluating a new system before committing to it.

The reason for (b) is that while having used woodpecker myself, I am not 100% sure it can replace our GitLab CI system although, especially with the newly added support for VMs and @hasufell's evaluation of the code base and contribution experience, I am quite confident that this could be a successful experiment and I'm certain that we should do it.

---

I also agree with @chreekat -- while gitlab CI is the reason for many CI failures (we have dozens of "runner system failures" that have no machine related reason), I think improving reliability of CI will only work if we're putting work into building a maintainable abstraction that is not hacks on top of hacks. For that reason, I also support to only port a small subset of the pipelines we are already running during the trial phase, to make sure those pipelines that do exist can be triaged for reliability.

---

What I think this proposal lacks is concrete success criteria. These ones I find particularly important:

1. "out of X pipelines, only Y failures can be attributed to the CI / test system" -- this requires appropriate monitoring to be in place.
2. there is buy in: part of why we arrived at the current situations is that there is next to no buy in -- maintenance for both the CI system and the infra that hosts it is done by very few people, mostly because code has to be closed source, the CI system is very complex and spans multiple repos, is done differently in many places, code is not shared, etc. -- if we want to make sure this does not happen again, devs have to at least be aware of how to modify pipelines and have to be willing and on-board for this, this needs communication and documentation
3. "we are able to propose a concrete architecture for building, testing and releasing GHC" -- this architecture does not need to be implemented, it just needs to be plausible that it can be implemented within the new system in reasonable time. Notably, this includes being able to do this on all supported platforms i.e. architectures / OSs.

---

In summary, I support this proposal. While there are many steps that can be done towards reducing the risk of whole system failure, I think the CI system is one of the biggest chunks. Gitlab's CI system is long-term unsustainable in that way because it's strictly coupled to Gitlab. Gitlab's internal strategy hints at changes that are going to affect us that will make Gitlab medium to long term unsustainable.

Therefore, carefully preparing a partial migration needs to be started as soon as possible.

[TeofilC]

I'm in favour of this! The key thing we need to ensure is that we don't get stuck maintaining too systems indefinitely.

I agree that local first CI is a worthy goal.

Your example in woodpecker-ci/woodpecker#6985 (comment) seems to suggest that settings about the VMs is configured as part of the in-repo CI config. Are there security concerns with this approach (and how can we alleviate them)? For instance, if I create a branch that tells the config to mount /var/secrets into the VM?