Upgrade to Tika 4.0.0

[abhinav-phi]

Upgrade to Apache Tika 4.0.0 — Closes #2063

This PR upgrades StormCrawler from Tika 3.3.2 to the first stable release of the Tika 4.x line (announcement), tracking the 4.0.0 milestone. Tika 4.0 ships a number of breaking changes; this PR migrates the codebase, the bundled configuration and the documentation accordingly. No behaviour change is intended for users of the JSoupParserBolt and the Tika ParserBolt beyond the Tika upgrade itself.

Dependency changes

• tika.version 3.3.2 → 4.0.0 in the root POM. Tika 4.x requires Java 17+, which StormCrawler already targets (the project builds on JDK 17-25, CI runs JDK 25), so no compiler settings were needed.
• In Tika 4.0, tika-parsers-standard-package changed from a jar to a POM that aggregates the individual tika-parser-*-module artifacts. external/tika now consumes it with <type>pom</type>, which pulls in all standard parser modules transitively — same effective set of parsers as before, including the OCR module.
• Added explicit tika-core and tika-serialization dependencies to external/tika: tika-serialization provides the new TikaLoader used to read the JSON configuration (see below). The existing exclusions (asm, slf4j-log4j12, and the BSD-licensed jai-imageio-core, still a compile dependency of the image parser module) are preserved.
• THIRD-PARTY.txt regenerated with license:aggregate-add-third-party (Tika modules now 4.0.0; new Tika artifacts: tika-serialization, tika-encoding-detector-html, tika-encoding-detector-mojibuster, tika-ml-core, tika-ml-junkdetect, tika-parser-datauri-commons; removals: tika-parsers-standard-package jar, vorbis-java-tika, POI poi-ooxml-full, PDFBox jempbox/xmpbox 3.0.8, org.tukaani:xz — replaced by Tika 4's internal dependencies).

Code migration

core/JSoupParserBolt.java (only core module using Tika, via tika-core):

• TikaConfig was removed in Tika 4. The mime-type detector, previously obtained from TikaConfig.getDefaultConfig().getDetector(), is now built with new DefaultDetector(), which is exactly what the removed default configuration wired (bytecode of Tika 4's Tika() facade confirms it constructs new DefaultDetector()).
• Detector.detect() now takes (TikaInputStream, Metadata, ParseContext); guessMimeType() wraps the content bytes with TikaInputStream.get(content) and passes an empty ParseContext.
• Metadata no longer implements HttpHeaders, so the Metadata.CONTENT_TYPE / Metadata.CONTENT_LENGTH constants are now taken from org.apache.http.HttpHeaders. Code referencing TikaCoreProperties constants is unaffected (the RESOURCE_NAME_KEY usage stays).

external/tika/ParserBolt.java:

• XML configurations are no longer supported in Tika 4; configuration moved to JSON. instantiateTika() now loads the configuration through TikaLoader.load(path, classLoader) (from the new tika-serialization module) and wires the result into the Tika facade via new Tika(loader.loadDetectors(), loader.loadAutoDetectParser()) — the equivalent of the removed new TikaConfig(url, classLoader) + new Tika(tikaConfig) pair. On failure the bolt still falls back to the default configuration.
• TikaLoader can only read configuration files from the filesystem. When the configuration is bundled inside a jar (the common case for topologies built as fat jars), urlToPath() copies it to a temporary file first.
• The parse itself now uses TikaInputStream.get(content) in a try-with-resources block, replacing the manual ByteArrayInputStream + finally close (Tika 4's Parser.parse() requires a TikaInputStream, and TikaInputStream holds TemporaryResources that must be released).
• Default value of parser.tika.config.file is now tika-config.json.

Configuration migration

• external/tika/src/main/resources/tika-config.xml → tika-config.json. The old XML only (a) excluded TesseractOCRParser from DefaultParser and (b) set service-loader warning handlers, which are the Tika 4 defaults. Using Tika's own XmlToJsonConfigConverter semantics, this becomes:

  {
    "parsers": [
      {
        "default-parser": {
          "exclude": ["tesseract-ocr-parser"]
        }
      }
    ]
  }

  (OCR stays disabled by default in StormCrawler — Tesseract is an optional native dependency.)

• Archetype crawler-conf.yaml files (core, OpenSearch, Solr) updated to parse.tika.config.file: "tika-config.json".

Documentation

• external/tika/README.md and docs/configuration.adoc updated for the JSON configuration and the new default file name.

Testing

• mvn verify with CI_ENV=true:
  • stormcrawler-core: 415 tests, 0 failures (includes JSoupParserBoltTest mime-type detection).
  • stormcrawler-tika: ParserBoltTest green — parses a recursive embedded .docx (TIKA-2096 scenario) through the new TikaLoader/JSON-config path, and exercises the mime-type whitelist.
  • stormcrawler-urlfrontier, stormcrawler-langid, archetypes, docs: green.
• -Prat -DskipTests verify -Dskip.format.code=false (CI's license + format job): the new tika-config.json is approved by RAT; google-java-format validation passes on all touched Java files.
• Two test failures encountered locally are pre-existing Windows-environment issues, reproduced identically on a pristine checkout of main and unrelated to this change: HttpRobotRulesParserRedirectTest (WireMock cannot bind port 8089 on Windows) and WARCHdfsBoltTest (Hadoop requires winutils.exe on Windows); Testcontainers-based tests (OpenSearch/Solr/SQL) require Docker, unavailable locally. CI runs on Ubuntu with Docker where these all pass.

Notes for reviewers

• The Tika 4 metadata key renames (tk: prefix) only affect keys emitted by parsers, which this bolt copies with a parse. prefix into StormCrawler metadata. Users with ParseFilters/indexing rules matching specific parse.* Tika keys may be affected by the renames; the upstream migration guide documents an opt-in legacy-key filter (metadata-migration-3x-4x.json) for that case.
• HTML remains the domain of JSoupParserBolt; the "default content handler is now Markdown" change does not affect StormCrawler, which always supplies its own BodyContentHandler/LinkContentHandler.

[tballison]

From my 🤖

Findings

  1. parse-context in the user's JSON is silently ignored — ParserBolt:223. Every parse builds new ParseContext() and sets only Parser.class and HtmlMapper.class; the loader's parse-context (where Tika 4 puts output limits,
  timeouts, exception reporting — the only new capabilities the JSON format brings) never reaches it. A user who adds "parse-context": {"output-limits": {"writeLimit": 100000}} sees no effect and no warning. Fix: hold the
  loader (or tikaLoader.loadParseContext()) next to tika and seed each per-parse context from it before setting Parser.class. Test: a config with a small writeLimit, assert the extracted text is truncated. edge-case, in
  scope.

  2. A broken config degrades silently to Tika defaults — with OCR on — instantiateTika, catch (Exception) → LOG.error → new Tika(). Pre-existing pattern, but JSON changes its odds: Tika 4's mapper rejects unknown keys, so
  one typo throws at load, and the topology quietly runs the full default parser set, TesseractOCRParser included wherever tesseract is installed. ParserBoltTest can't see this: it asserts tuple count and status, and passes
  identically with tika-config.json deleted (verified by reading the test — nothing checks what was loaded). Maintainer decision: fail fast in prepare() (my recommendation for a crawler, where a silent config downgrade is
  worse than a failed deploy) or keep the fallback; either way, a test that asserts the loaded parser excludes TesseractOCRParser, so the config path is proven rather than assumed. edge-case, in scope.

  3. urlToPath keeps the temp copy for the worker's lifetime — deleteOnExit, one file per bolt prepare(), never freed on a long-running worker. TikaJsonConfig.load(Path) reads the file eagerly inside TikaLoader.load, so
  delete it in a finally immediately after load returns. The workaround itself is legitimate: at 4.0.0 TikaLoader's constructor is private and there's no InputStream/TikaJsonConfig factory, even though
  TikaJsonConfig.load(InputStream) exists — that's a Tika gap (punt list). edge-case, in scope.

  Hygiene (one line each):
  - README links https://tika.apache.org/4.0.0/config.html → 404. The 4.0.0 index links https://tika.apache.org/docs/4.0.x/configuration/index.html (200).
  - The three archetype crawler-conf.yamls set parse.tika.config.file; code and docs read parser.tika.config.file, so the yaml value is ignored. Pre-existing and harmless only because it equals the default — the PR edits
    exactly those lines, so fix the key while there.
  - Description: the dropped service-loader handlers aren't "the Tika 4 defaults" — at 4.0.0 ServiceLoader has no LoadErrorHandler/InitializableProblemHandler at all, and the JSON service-loader section deserializes into an
    empty ServiceLoaderConfig. Dropping it is right; the reason is that it's inert. Cosmetic.
  - configuration.adoc says "Path to the Tika configuration file"; it's a classpath resource name (getResource) — the README has it right.
  

One area where the bot is wrong is that configurations around timeouts don't affect us because we aren't using tika pipes (we on Tika need to improve our documentation around that). If you want process isolation and want to avoid a couple of PDFs infinitely looping on a node and/or OOM'ing it, consider the PipesForkParser. It may not be a good fit for StormCrawler. Happy to chat, though.

[tballison]

Other subtlety, and we should have a unit test for this is that

        if (extractEmbedded) {
            parseContext.set(Parser.class, tika.getParser());
        }

I think we should invert this to put in the EmptyParser if ! extractEmbedded because the AutoDetectParser adds itself (and did in 3.x, I thought). So, I think this was already broken?