Conversation
handle_data strips whitespace-only content inside a stressed tag, so <b> </b> reaches the closing tag with the opening ** as the last thing written and nothing after it. The closing marker went out anyway: Hello<b> </b>world -> Hello**** world <b>First</b><b> </b><b>Last</b> -> **First********Last** <em>First</em><em> </em>... -> _First_ ___Last_ The first leaves an empty marker pair in the output; the second and third lose the only space separating two words, which is the shape an editor produces when it emits one element per styled run. Take the opening marker back out of outtextlist, the way the anchor handler already does for an empty link, and keep the whitespace it stood for. An element with real content is untouched, and so is <b>a</b><b>b</b>, where the words genuinely are adjacent in the HTML.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A whitespace-only inline element leaves an empty marker pair in the markdown, and between two
styled runs it takes the only space separating two words with it.
handle_datastrips whitespace-only content inside a stressed tag:so
<b> </b>reaches the closing tag with the opening**as the last thing written andnothing after it. The closing marker is emitted regardless.
Measured with
CustomHTML2Text().handle(...):<p>Hello<b> </b>world</p>Hello**** worldHello world<p>Hello<em> </em>world</p>Hello __ worldHello world<p>Hello<s> </s>world</p>Hello~~~~ worldHello world<p><b>First</b><b> </b><b>Last</b></p>**First********Last****First** **Last**<p><em>First</em><em> </em><em>Last</em></p>_First_ ___Last__First_ _Last_The last two are the ones that matter: an editor that emits one element per styled run puts
the space between two bold words in an element of its own, and both rows above lose it
entirely.
**First********Last**renders asFirstLast, and___Last_opens bold-italicthat never closes.
List of files changed and why
crawl4ai/html2text/__init__.py— newdrop_empty_stress(), called from the three closingbranches (
em/i/u,strong/b,del/strike/s). It pops the opening marker back offouttextlistwhen that marker is still the last thing written, and restores the whitespacethe marker stood for. Taking an already-emitted token back out of
outtextlistis thetechnique the anchor handler in this file already uses for an empty link.
tests/unit/test_html2text_empty_emphasis.py— new.The guard only fires when the opening marker is still the last element, so an element with
real content never reaches it.
<b>a</b><b>b</b>also stays**a****b**: there is nowhitespace between those runs in the HTML, so the words really are adjacent.
How Has This Been Tested?
tests/unit/test_html2text_empty_emphasis.py, 19 cases:b strong em i u s del strike)<b><i>x</i></b>and thegenuinely adjacent
<b>a</b><b>b</b>14 of the 19 fail on
main; the 5 that pass are the controls. Seven mutants of the changedlines are killed by them, one per call site plus the whitespace restore in both directions,
the
" " + markmatch, and thepreceding_stressedreset.Checklist: