New option to print parsed test file with actual stdout, stderr, and exit status

[schoettl]

Hi, I think it would be great to have an option --print-test-file-with-output which would print the test file together with the actual output, e.g.:

Running:

cat <<EOF > example.test
$ echo foo
>
bar

$ false

$ echo bar >&2

EOF
shelltest --print-test-file-with-output example.test

would yield this output:

$ echo foo
>
foo

$ false
>= 1

$ echo bar >&2
>2
bar

Example use:

shelltest --print-test-file-with-output example.test > tmp && mv tmp example.test
git diff

Or did I miss an option like that?

It would make it easy to generate expected results in new tests. We also could diff and review changes without manually updating parts of the test file.

I could probably implement this. But first a question: Can the program know, after parsing, what the shelltest input format was (v1, v2, v3)? So that it can generate the exact same format?

[simonmichael]

@schoettl no there's no option like this. I agree it would be great if it could optionally update results in test files for you, or at least print out the whole thing as you said.

I don't know if the input format is currently stored anywhere. I suspect the first thing to do would be clean up and simplify this messy three format situation somehow.

[schoettl]

@simonmichael thanks for the quick response. I'm looking into it right now.

I don't know if the input format is currently stored anywhere. I suspect the first thing to do would be clean up and simplify this messy three format situation somehow.

No, the input format is not stored, at least not in Types.ShellTest. Regarding the different formats: there should be a way to migrate between the formats (like hledger print for shelltest).

[...] it would be great if it could optionally update results in test files for you [...]

That would be also great, but I suspect this to be harder to be implemented.

Proposed CLI:

shelltest --print=v3 example.test
shelltest --print --actual-results example.test

--print[=<format>] behaves like hledger print and skips normal tests.
(Given optional option arguments are supported.)

--actual-results (for now) allowed only in combination with --print and makes printing actual results.

Example use:

shelltest --print --actual-results example.test > tmp
vim -d example.test tmp  # use a diff tool to update test file

I'd love that! :)

Oh, but comments would be gone. It may make sense to also parse comments before (and after) all ShellTests.

[schoettl]

Current CLI proposal:

Print test file:
     --print[=FORMAT]   Print test files in specified format (default: v3).
     --actual[=MODE]    Combined with --print, print test files with actual
                        results (stdout, stderr, exit status). Mode 'all'
                        prints all actual results (default). Mode 'update'
                        prints actual results only for non-matching results,
                        i.e. regular expressions in tests are retained.

[simonmichael]

Sorry for the delay, I'll respond soon.

[simonmichael]

This is sounding good. I'd be inclined to call --actual --print-with or something, to indicate that it's a modifier for --print and not usable by itself.

But, I wonder if we should focus entirely on --print to start with ? It's a nice standalone feature with plenty to discuss.

[simonmichael]

Yes, we'll need to preserve comments.

[simonmichael]

When you have time, let's review a PR (same one or a new one) that just adds --print.

Do we want to support printing v1 and v2 ? Any reason to encourage those ? I don't think so, except for the case where v3's short delimiters clash with your tests. But perhaps just v3 plus the ability to set custom delimiters would be best.

[schoettl]

This is sounding good. I'd be inclined to call --actual --print-with or something, to indicate that it's a modifier for --print and not usable by itself.

OK, what do you have in mind?

• --print[=v3] --print-with[=update]
• ( --print | --print-with[=update] ) [--format=v3]
• ...

But here, --format also doesn't indicate that it's tied to --print.

But, I wonder if we should focus entirely on --print to start with ? It's a nice standalone feature with plenty to discuss.

To be honest, I'd rather like to integrate it as one thing because the --actual thing is already done and it would be more work to extract it, keep it and re-introduce it later.

Yes, we'll need to preserve comments.

The problems are:

• Comments at begin of file are discarded because in v2/v3 they cannot be assigned to a test if they come before the "shared input".
• Comments are parsed and returned without the comment character. Because there are two types of comments and also empty lines, I cannot print them correctly. The simplest solution would be to parse the complete comment line as one string. That would also prevent whitespace diffs.

[simonmichael]

If it's too much work, no problem, I can have a go when I have time. It'll work better for me to tackle this in smaller chunks.

[schoettl]

Do we want to support printing v1 and v2 ?

Already implemented, I think. Especially v1 is easy to generate because it does not support shared input.

Any reason to encourage those ? I don't think so, except for the case where v3's short delimiters clash with your tests. But perhaps just v3 plus the ability to set custom delimiters would be best.

As long we support the old input formats for input, I see no problem in also letting shelltest produce the old format.

One concrete reason to allow the old formats: I use shelltest on Raspberry Pi in production and it supports only the old format. For this, I had to convert many files from v3 to v1.

[schoettl]

Another reason to support v1/v2: If it comes to generating/updating results, we should not force the user to migrate to v3, if they want to use their current format for any reason.

So I'd really like to not discard my efforts in printing different formats.

@simonmichael Regarding the --print CLI for the current PR, I have two proposals:

• --print [--format=v3]
• --print[=v3]

I don't mind – but we should also consider the future --print-with/--actual feature, see my suggestion in the previous comment

[simonmichael]

I think the multiple formats are just an artifact of the design process, they make maintaining and using the tool more complex and actually I'd quite like to kill off the old ones. Your print feature will make this possible. But it doesn't have to happen right now..

The simple --print[=ARG] sounds good to me for now.