wrong version of dub is tested

[MartinNowak]

dlang/dub#1579 (comment)

It's a bit confusing that https://buildkite.com/dlang/ci/builds/511#4c02ce8c-f1a7-4899-bc6d-bcf91ba4d9ee passed and https://buildkite.com/dlang/phobos/builds/582#3ddd20c7-ec9f-4455-838e-061bb4d3a14b/106-716 subsequently failed.

The culprit seems to be that no distinction is made between building dub (as in dub the tool) where we want to use the dev version, and dub (as in dub the project with testsuite) where we want to use the latest stable release.
There is no point in rerunning dub's test-suite with the dub development version, that task is the duty of dlang/dub itself.

ci/buildkite/build_project.sh

Line 52 in 73ebbe8

"https://github.com/dlang/dub" | \

The purpose of the project tester is always to test stable projects with the latest development versions.
Anything that falls outside of this (like style tests et.al.) should be clearly isolated, so that we don't end up with yet another confusing Frankenstein CI.

[wilzbach]

Hmm, but as we ship dub as part of DMD (and a new version gets automatically tagged once new codes gets released), doesn't it make sense to dub master as we do for e.g. tools too?

This would also help in cases like dlang/dmd#9017 where dub needs to be modified for a DMD PR to pass.

[wilzbach]

Anything that falls outside of this (like style tests et.al.) should be clearly isolated, so that we don't end up with yet another confusing Frankenstein CI.

I agree, but not sure whether we can group projects with Buildkite. It would be great to send different status notification for "style" and "projects" (and maybe one or two more), but at the moment it seems that it can only distinguish between all or one.

[wilzbach]

After getting a bit of feedback on a few PRs where we ran into errors (i.e. dlang/druntime#2376), I think it only makes sense to the version of the current PR if a project triggered the build.
In other words, if dlang/dub triggered the build, then its current state should be tested for dlang/dub or if dlang/phobos triggered the build, the instead of cloning phobos master, the current state should be used when testing dlang/phobos.
An optimization for this would be to remove dlang/{dub,tools} when they get triggered from themselves, but an advantage of not doing this optimization is that this way we can safely make changes to a repository and now that with the current PR merged, it would still pass with the current Buildkite setup.