Parser Combinators: Reject Valid Prefixes With Invalid Tails

A parser can successfully recognize the beginning of a string while leaving an invalid tail untouched. If the application checks only whether parsing succeeded, 42unexpected may be accepted as the number 42. Full input consumption closes that gap when the contract expects one complete document or value.

The nom allconsuming combinator is one concrete example: it requires its child parser to consume the entire input. The broader design principle applies across parsing libraries. Decide whether leftover data is part of the next message or evidence that the current input is invalid.

Distinguish a token parser from a document parser

A token parser answers, “What can I recognize here?” A document parser answers, “Does this entire input conform to the grammar?” Both are useful, but they have different completion conditions.

For example, a parser reading size=42;mode=fast should leave input after the size token so another parser can read the mode. A parser validating a standalone size field should reject the same trailing material unless the field grammar explicitly permits it.

Name the entry points accordingly. An internal parse_integer_prefix function is less likely to be mistaken for a complete field validator than an ambiguous parse_integer function returning only a number. Preserve the remaining-input value until the outermost grammar has decided what to do with it.

Write the acceptance table first

For a nonnegative decimal count, consider this deliberately small contract:

These are application decisions, not universal rules for numbers. A signed measurement field would need a different grammar. Establishing the table prevents a convenience conversion function from quietly choosing your protocol.

Also define leading zero handling, maximum length, and range. Consuming every character does not make a value fit into a 32-bit integer or make a million-digit input cheap to process.

Place end-of-input at the outer boundary

The general structure is: optional allowed leading space, a complete value parser, optional allowed trailing space, then end-of-input. Put the whitespace policy inside the document grammar rather than stripping arbitrary characters before parsing.

In nom 8, a small complete-input example is:

use nom::{Parser, character::complete::digit1};
use nom::combinator::all_consuming;
use nom::error::Error;

fn decimal_text(input: &str)
    -> nom::IResult<&str, &str>
{
    all_consuming(digit1::<&str, Error<&str>>).parse(input)
}

This recognizes digit text, not a bounded numeric type. A separate checked conversion should enforce the application’s range. The example intentionally does not accept whitespace, signs, separators, or decimal points.

Do not add end-of-input to every child parser in a larger grammar. Doing so would make the first child reject the tokens that belong to the next child. Completeness belongs at the level that owns the whole message.

Keep streaming and complete-input behavior separate

When parsing a network stream, the current buffer may end before the message ends. “Need more data” can be a valid intermediate result. A standalone file upload, by contrast, has a known boundary once reading finishes.

Choose the library’s complete or streaming mode deliberately. Do not convert an incomplete streamed token into a successful document merely because the connection paused. Conversely, do not retry forever when an already-complete file ends in a malformed token.

If the transport supplies a message length, apply resource limits before accumulating that many bytes. Full-input validation should not imply unlimited buffering. Reject oversized inputs according to the product’s documented limit and return an actionable error.

Test ambiguity and character boundaries

Add a property test that appends a forbidden non-whitespace suffix to each valid example. The complete parser should reject it. Add separate tests for every allowed trailing character so the policy does not become stricter accidentally.

Test multi-byte text where the grammar allows it. A byte offset and a user-visible character position are not interchangeable. Report errors using the coordinate system the interface promises, and avoid slicing UTF-8 in the middle of a code point.

For alternative branches, include inputs where the first branch recognizes a prefix but the second branch would recognize the complete value. Confirm that your library’s backtracking or commitment behavior matches the intended grammar. End-of-input reveals leftover text but does not resolve every ambiguity on its own.

Keep error messages bounded and safe. Report an offset and a short sanitized context rather than echoing an entire uploaded document into logs. Add a test containing a secret-like token after a valid prefix and confirm that rejection does not leak that suffix through diagnostics.

Conclusion

Successful recognition is not always successful validation. Keep prefix parsers composable, require full consumption at the complete-message boundary, and test suffixes, whitespace, range, and streaming behavior as separate concerns.