Skip to content

Fix invalid and oversized table header colspans - #2277

Open
SkanderBog wants to merge 1 commit into
unclecode:developfrom
SkanderBog:codex/fix-header-colspan
Open

SkanderBog wants to merge 1 commit into
unclecode:developfrom
SkanderBog:codex/fix-header-colspan

Conversation

@SkanderBog

Copy link
Copy Markdown

Summary

A header such as <th colspan="">Product</th> currently raises during extraction, so extract_tables() drops the entire table. Header colspans also bypass COLSPAN_LIMIT: a value of 1001 expands to 1001 header entries even though body spans are capped at 1000.

Reuse the existing bounded span parser for explicit <thead> headers and inferred first-row headers. Invalid and non-positive header spans become 1; spans above 1000 are capped before list expansion. Body span parsing, table scoring, and row traversal keep their existing behavior.

This is a small follow-up to the grid fix merged in #2261. Related to #2258 and the open #2268: that proposal overlaps malformed-header handling, but does not cap header spans. This PR is based on current develop and does not replace its grid implementation.

List of files changed and why

  • crawl4ai/table_extraction.py: lift the existing local span parser into a shared private method and use it for both header paths and the existing body grid.
  • tests/test_table_header_spans.py: 34 regression cases covering explicit headers, inferred th/td headers, malformed/missing/non-positive values, the 1000 boundary, oversized spans, and preservation of adjacent tables.

How Has This Been Tested?

Python 3.12.14, Linux, Chromium 153.

  • Before the fix, the new file produced 22 failures and 12 passes against current develop.
  • After the fix, 63 tests passed:
    python -m pytest tests/test_table_header_spans.py tests/test_table_rowspan_grid.py tests/test_table_gfm_compliance.py tests/unit/test_pdf_html_escaping.py -q
  • 26 browser content regressions passed, with four network tests deselected:
    python -m pytest tests/regression/test_reg_content.py -q -m "not network"
  • Independent browser/pipeline check: an empty header colspan retains the expected product/price table, and a 1001 colspan is capped at 1000, matching Chromium's DOM property for these inputs.
  • Live smoke crawl of Quotes to Scrape still returned HTTP 200, ten complete quote/author records, Markdown, and a screenshot.
  • Black check for the new tests and git diff --check passed.

The full repository test suite was not run. No claim of complete HTML span-parsing conformance is made; this preserves the existing body parser's normalization rules.

Checklist

  • Based on and targeting develop.
  • Performed a self-review; only table extraction and its new tests are changed.
  • Added regression tests that reproduce the failure before the fix.
  • New and selected existing tests pass locally.
  • No public API or documentation changes are needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant