Unexpected concatenation of field tokens

[zepinglee]

Describe the bug

In the following example, the value of field 10 # "~" # jan is expected to be 10~Jan. but the output of this library is 10 # "~".

BibTeX has three types of field tokens: nonnegative number, macro name (like jan), and a brace-balanced string delimited by either double quotes or braces. They can be concatenated by the # character. Although the first type is called "number", it behaves the same as a string and it can be applied with string slicing, text length, and concatenation in a .bst style.

BTW, I've also made a bib2json.bst style that may help testing. It reads .bib data and writes JSON format (though with some limitations) to the .bbl output.

Reproducing

Version: 2.0.0b2

Code:

import bibtexparser
bibtex_str = '''
@STRING{ jan = "Jan." }

@INBOOK{inbook-full,
   month = 10 # "~" # jan,
}
'''
library = bibtexparser.parse_string(bibtex_str)
month = library.entries[0].fields_dict['month'].value
print(month.__repr__())
assert month == "10~Jan."

Output:

'10 # "~"'

[MiWeiss]

Thanks a lot for the beautiful bug report. This will probably have to be adressed in two distinct PRs

• One PR to fix the splitter to contain the entire field, even if the field contains string concatenations.
• One follow-up PR to adapt StringInterpolationMiddleware (and probably add a further middleware) to properly handle concatenation.

The first of these PRs is likely nontrivial.

P.s. I have not actually reproduced the issue, but given the nice issue description and the fact that token concatenation is not yet supported, I still added the reproduced label.

[kmccurley]

Note that string concatenation can also be used inside @string, and I've seen this in cryptobib. An example is:

@string{asiacryptname =         "ASIACRYPT"}
@string{asiacrypt91name =       asiacryptname # "'91"}
@string{asiacrypt92name =       auscryptname # "'92"}

[claell]

I independently explored a broader version of this fix while auditing the parser's round-trip behavior. I have published the result as a comparison branch rather than opening a competing PR:

• Browse the branch
• Compare it with upstream main

In addition to resolving ordinary entry-field concatenations, the branch tests and handles:

• concatenations inside @string definitions;
• recursively chained string definitions;
• cyclic definitions without a recursion error;
• exact preservation of unresolved expressions instead of partial resolution; and
• preservation of unresolved quoted or braced multi-token expressions through RemoveEnclosingMiddleware.

The branch has two commits so the expression resolver and the enclosing-middleware safeguard can be reviewed independently. Validation is clean: 2584 passed, 12 skipped, plus the complete pre-commit hook suite. Please feel free to compare the implementation or cherry-pick/adapt any of its tests and cases for this PR.

For transparency, this comparison implementation and its tests were prepared with ChatGPT Codex using GPT-5.6 Sol with high reasoning effort. They should still receive normal maintainer review, especially because the work deliberately covers more cases than the original report.