python/java/csharp: a line lost while hashing the program changes the engine id, so a warm rebuild recompiles the engine for minutes #1593

Symptom

A warm rebuild after a one-line edit sometimes recompiles the whole engine in C++. On a loaded macOS machine one background refresh of a 7-file Python project took 2183 s instead of the usual 11 s. For that whole time every query answered from the previous graph, and an explicit axiomcode index waited on the build lock for more than 300 s. The build log shows:

run-souffle.sh: line 124: printf: write error: Interrupted system call
▶ engine id = 43e00efb… (rules + souffle 2.5)
▶ compiling souffle program (cache miss)...

The id is normally 33a53d0f…. The binary compiled under the wrong id is byte-for-byte the same size as the cached one, so the program was right and only its id was wrong. This happened in 3 of about 460 builds (Python twice, Java once) at a load average of 60 to 170.

Reproduction

Simulate the lost line with a sort on PATH that drops one relation name, then ask for the id. From a checkout:

$ mkdir /tmp/fake && printf '#!/bin/bash\n/usr/bin/sort "$@" | grep -vx dispatch_cap\n' > /tmp/fake/sort && chmod +x /tmp/fake/sort
$ bash graph/pipeline/run-souffle.sh --language python --print-engine-id; echo "exit $?"
33a53d0fb44a…
exit 0
$ PATH=/tmp/fake:$PATH bash graph/pipeline/run-souffle.sh --language python --print-engine-id; echo "exit $?"
c9a07c91d782…
exit 0

Root cause

graph/pipeline/run-souffle.sh:372 computes ENGINE_ID with engine_id() (:181). That function does not hash the program the build compiles (write_program "$PROG", :365). It calls write_program a second time, which goes through input_relations() (:123-:130) and input_relations | LC_ALL=C sort -u | while … (:154). The printf calls there run in a pipeline under /bin/bash 3.2, and a failed write is never checked, so a lost line gives a different id with exit 0. The same unchecked pipeline also writes $PROG. If the line is lost there instead, a program missing an .input gets compiled and cached under the correct id.

Measured on 0.1.3 at 0df257c. Still present on 0.1.4 at 3df2e33, after #1565: the same lines, and the same two commands still print two different ids with exit 0.

Expected: the engine id is the hash of the exact program that gets compiled, and a failed write while generating it fails the build.