Thematic mapping: metadata enrichment, keyword cleanup, and area-label improvements

[chreman]

Thematic mapping: metadata enrichment, keyword cleanup, and area-label improvements

This branch brings together the thematic-mapping work. It has 221 commits across 371 files. About 80% of the added lines are test replay fixtures and contentprovider.json cache updates.

1. Deduplication and metadata enrichment (BASE, ORCID)

• Anchor records use an is_anchor flag, and anchor selection and ordering are now deterministic. Before this, ties on BASE's unstable response order could produce different results from run to run.
• DOI handling: DOIs are merged regardless of case, multiple and versioned DOIs are supported, an additional_doi fallback is used for enrichment, and SOLR escaping for DOIs with special characters is fixed.
• Enrichment of OA status, PDF links, abstracts, and subjects. Keywords from duplicates are merged with RapidFuzz. PDF link candidates from duplicates are exposed to the frontend.
• A title sanity check drops wrong DOIs from duplicate groups.
• Enrichment is enabled for BASE, and several deduplication bugs in the ORCID merge are fixed.

2. Keyword and classification cleanup (R preprocessing)

• A structured subject-cleaning module (subject_cleaning.R) replaces the chained gsub calls.
• MeSH: qualifiers are stripped, terms are de-inverted (with an exclusion list), and tree-depth resources are added.
• Classification codes are cleaned for LoC, FoR, SDG, JEL, AMS, PACS, arXiv, and grant schemes. TeX quotation marks and asterisks are cleaned too.

3. Area-label generation

• There are now area-title ranking modes 0–3, set with RANKING_MODE[_<SERVICE>]. Mode 0 is the default and reproduces the legacy output exactly.
• N-gram generation is consolidated and parameterized as settings 0–5, set with NGRAM_SETTING[_<SERVICE>] and INCLUDE_ABSTRACTS. Setting 0 is the legacy baseline.
• Heuristic keyword generation now handles punctuation, uses a global heuristic threshold, adds a label exclusion list, and expands the stopword list.
• Label casing is restored, with guards for ALL-CAPS titles and keywords and for punctuation inside words.
• Pandas sorting is now deterministic.

4. Tests and tooling

• New R unit tests and replay fixtures for each ranking mode.
• New Playwright e2e suites for keyword cleaning (including invariant tests), keyword enrichment, and OA status and PDF link enrichment for BASE and ORCID.
• Debug and dump tooling for label generation and deduplication.

5. Configuration and ops

• docker-compose.yml passes the new ranking and n-gram environment variables through. They are documented in local_dev/dev.env.example.
• ENRICHMENT_STRATEGY_FOR_SUBJECT is removed.
• ⚠️ The BASE worker now reads its log level from LOGLEVEL instead of LOG_LEVEL. Check the deployment environment.
• flask-restx is pinned for Python 3.9, and the renv build order is fixed.