Skip to content

fix(html2text): keep the word boundary an empty emphasis pair swallowed - #2271

Open
L4XB wants to merge 1 commit into
unclecode:mainfrom
L4XB:fix/empty-emphasis-whitespace
Open

L4XB wants to merge 1 commit into
unclecode:mainfrom
L4XB:fix/empty-emphasis-whitespace

Conversation

@L4XB

@L4XB L4XB commented Sep 16, 2026

Copy link
Copy Markdown

Summary

A whitespace-only inline element leaves an empty marker pair in the markdown, and between two
styled runs it takes the only space separating two words with it.

handle_data strips whitespace-only content inside a stressed tag:

if self.stressed:
    data = data.strip()        # "  " -> ""

so <b> </b> reaches the closing tag with the opening ** as the last thing written and
nothing after it. The closing marker is emitted regardless.

Measured with CustomHTML2Text().handle(...):

input before after
<p>Hello<b> </b>world</p> Hello**** world Hello world
<p>Hello<em> </em>world</p> Hello __ world Hello world
<p>Hello<s> </s>world</p> Hello~~~~ world Hello world
<p><b>First</b><b> </b><b>Last</b></p> **First********Last** **First** **Last**
<p><em>First</em><em> </em><em>Last</em></p> _First_ ___Last_ _First_ _Last_

The last two are the ones that matter: an editor that emits one element per styled run puts
the space between two bold words in an element of its own, and both rows above lose it
entirely. **First********Last** renders as FirstLast, and ___Last_ opens bold-italic
that never closes.

List of files changed and why

  • crawl4ai/html2text/__init__.py — new drop_empty_stress(), called from the three closing
    branches (em/i/u, strong/b, del/strike/s). It pops the opening marker back off
    outtextlist when that marker is still the last thing written, and restores the whitespace
    the marker stood for. Taking an already-emitted token back out of outtextlist is the
    technique the anchor handler in this file already uses for an empty link.
  • tests/unit/test_html2text_empty_emphasis.py — new.

The guard only fires when the opening marker is still the last element, so an element with
real content never reaches it. <b>a</b><b>b</b> also stays **a****b**: there is no
whitespace between those runs in the HTML, so the words really are adjacent.

How Has This Been Tested?

tests/unit/test_html2text_empty_emphasis.py, 19 cases:

  • the whitespace-only element across all eight tags (b strong em i u s del strike)
  • the space between two styled runs, for five of them, asserting the exact markdown
  • whitespace already present outside the element is not doubled, on both sides
  • five control cases with real content, including the nested <b><i>x</i></b> and the
    genuinely adjacent <b>a</b><b>b</b>

14 of the 19 fail on main; the 5 that pass are the controls. Seven mutants of the changed
lines are killed by them, one per call site plus the whitespace restore in both directions,
the " " + mark match, and the preceding_stressed reset.

tests/unit/test_html2text_empty_emphasis.py          19 passed
tests/test_markdown_generator_validation_1880.py
  + tests/test_table_gfm_compliance.py               24 passed
tests/unit                                           same 28 pre-existing failures as main
                                                     (pdf and sitemap optional deps)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added/updated unit tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

handle_data strips whitespace-only content inside a stressed tag, so
<b> </b> reaches the closing tag with the opening ** as the last thing
written and nothing after it. The closing marker went out anyway:

  Hello<b> </b>world              ->  Hello**** world
  <b>First</b><b> </b><b>Last</b> ->  **First********Last**
  <em>First</em><em> </em>...     ->  _First_ ___Last_

The first leaves an empty marker pair in the output; the second and third
lose the only space separating two words, which is the shape an editor
produces when it emits one element per styled run.

Take the opening marker back out of outtextlist, the way the anchor handler
already does for an empty link, and keep the whitespace it stood for. An
element with real content is untouched, and so is <b>a</b><b>b</b>, where the
words genuinely are adjacent in the HTML.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant