Skip to content

dictParser rejects input files that lack a trailing newline - #232

Open
NRavoisin96 wants to merge 1 commit into
CambridgeNuclear:mainfrom
NRavoisin96:fix/dictParser_trailing_newline
Open

dictParser rejects input files that lack a trailing newline#232
NRavoisin96 wants to merge 1 commit into
CambridgeNuclear:mainfrom
NRavoisin96:fix/dictParser_trailing_newline

Conversation

@NRavoisin96

Copy link
Copy Markdown
Contributor

Mechanism

dictParser_func.f90 parses a file by streaming its contents into a charTape (readFileContents, which converts each
end-of-record to a space—noted in the code itself at :74), appending the top-level dictionary terminator (:162-163):

    ! Append with dictionary terminator
    call file % append('}')

and walking the tape with (:273):

    do while (pos < tape % length())

For a file that ends with a newline, the converted trailing space sits between the last token and the appended '}', so the loop condition still holds when the terminator is scanned. For a valid input file whose last line has no trailing newline, the appended '}' lands at exactly tape % length() with the previous token immediately before it: after that token is consumed, pos == length, the strict < exits the loop with the terminator unconsumed, and the parse aborts with:

End of file was reached without finding a termination symbol for a subdictionary. It means that '}' must be missing somewhere.

The same off-by-one is why charToDict carries an explicit workaround (:223-224):

    ! parseDict does not like '}}' ending so add an extra space
    call file % append(' }' )

with the leading space existing only to keep the terminator off the exact end of the tape when the caller's data itself ends in '}'.

Impact

Any input file whose final line lacks a newline is affected. The abort fires on an otherwise valid command-line invocation
(scone.f90:50 feeds the path straight to fileToDict), with an error message pointing at a missing '}' that is, in fact, present.

Provenance

Present since ed517f67 ("New dictionary parser", 2020-03-14).

Reproduction

printf 'a 1;' > f.txt        # no trailing newline

fileToDict on f.txt (e.g., via ./Build/scone.out f.txt) aborts with the message above.

Fix

Two small changes. First, fix the loop bound to include tape % length when parsing:

    do while (pos <= tape % length())

and, in charToDict, the workaround is removed to match fileToDict:

    ! Append with dictionary terminator.
    call file % append('}')

Note that this only makes the final position a valid starting position for the scan; it does not weaken any termination check. If an inner dictionary is genuinely unterminated, it consumes the top-level terminator early and the outer parse still reaches the same fatalError; a '{' at the tape end still errors in the recursive call.

Test plan

First, a new integration test file, IntegrationTestFiles/testDictionaryNoNewline, whose content ends in '}' with no trailing newline (git records and preserves the missing end-of-line).

Second, two new tests attached to the PR:

  • testFromFileNoTrailingNewline in dictParser_iTest.f90 asserting the above file parses. On current main this test aborts with the error message above.
  • A charToDict case with data ending exactly in '}' in dictParser_test.f90 that explicitly tests the workaround removal. (Note: the existing testFromChar tape also ends in '}' and doubles as a regression for this path.)

The scan loop in parseDict starts an iteration only while pos is strictly before the end of the tape, so the last character is never a valid starting position. Any input whose final character is the closing brace therefore ends with that brace unparsed, and the caller reports that the string was not fully read.

charToDict has been working around this by appending ' }' instead of '}', where the space serves only to enforce the closing brace being processed; fileToDict has no such padding, so a file whose last byte is '}' fails outright.

Fix the loop to include the final position, and drop the padding from the append. Add a test for a file without a trailing newline and one for character input ending with a brace, both of which produce the '}}' ending the padding used to avoid.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant