Thematic mapping: metadata enrichment, keyword cleanup, and area-label improvements - #883

Merged

chreman merged 222 commits into

Sep 23, 2026

Commits on Feb 10, 2026

bugfix: snapshots configuration
andreishket
committed
48f2ef0

Commits on Feb 11, 2026

feat: usage of dccollection
andreishket
committed
1419b03
refactor: usage of is_anchor flag instead of keep and is_latest
andreishket
committed
6fd83b8
refactor: filtering non_datasets and pure_datasets the same way
andreishket
committed
f16c5fb
feat: prioritize entries using doi and data provider
andreishket
committed
8b3d651
refactor: usage of doi and provider prioritization only for ORCID
andreishket
committed
78ee23c
feat: subject_orig replacement
andreishket
committed
45b483c
feat: paper_abstract replacement
andreishket
committed
84db510

Commits on Feb 12, 2026

feat: subject replacement
andreishket
committed
26541d0
refactor: group all enrichments in one cycle
andreishket
committed
b9219f4
feat: native merging strategy for subject and subject_orig
andreishket
committed
814be76
bugfix: dates in the request to base
andreishket
committed
c4ad1fe
bugfix: limit in the request to base
andreishket
committed
4cc4681
feat: merging keywords with usage of RapidFuzz library
andreishket
committed
f4ae820
feat: finding best OA status
andreishket
committed
8fb46f0

Commits on Feb 13, 2026

Commits on Feb 16, 2026

bugfix: some entries are not enriched
andreishket
committed
f073cfa
bugfix: mark the only entry in a group as is_anchor
andreishket
committed
e1207ad

Commits on Feb 17, 2026

feat: DOIs preparation before the BASE request
andreishket
committed
a90ec0f
feat: mapping DOIs back to initial case
andreishket
committed
9459892
doi metadata bugfix
chreman
committed
4dd20fa

Commits on Feb 18, 2026

updated query to include docs without date
chreman
committed
692d4ec
increased limits and size of possible return set
chreman
committed
dbd95dc
work-in-progress
chreman
committed
308a6c5
wip for merging with multiple DOIs
chreman
committed
2ff5c37
feat: preparing merged version of DOIs
andreishket
committed
cf93efb
feat: explode merged DOIs
andreishket
committed
0c66fc4
refactor: simplier explode function
andreishket
committed
231265d
feat: DOIs versions mapping
andreishket
committed
47e24d9
add timing logs
chreman
committed
98d2c7f
bugfix for column handling and OA state merging
chreman
committed
1eefbf4
improved doi replacement
chreman
committed
4889ec1

Commits on Feb 20, 2026

refactor: log that shows amount of not recieved DOIs from BASE
andreishket
committed
9a09e40
added waiting time analysis script
chreman
committed
8d67706
added new run time evaluation script
chreman
committed
9857010
README update
chreman
committed
68af5f0
datatype bugfix for oa_state enrichment; code & logging cleanup
chreman
committed
5d4f9fb
bugfix in unstable merge logic for oa_state
chreman
committed
228ebb8
tweaking batch size
chreman
committed
705d06e
further query simplification
chreman
committed
76f4ba9
reconcile code divergence
chreman
committed
b77e5b5
reconcile code divergence
chreman
committed
b321c2f
reconcile code divergence
chreman
committed
bc83ca3
reconcile code divergence
chreman
committed
04bd131
reconcile code divergence
chreman
committed
ed6641b
reconcile code divergence
chreman
committed
b7fbc8e
enable enrichment for BASE
chreman
committed
b057045
reconcile code divergence
chreman
committed
0830490
reconcile code divergence
chreman
committed
b79a7a0
bugfix for oa state priorisation
chreman
committed
e6b3749
activate enrichment pipeline for base
chreman
committed
3b62fed
revert to original approach
chreman
committed
8d38836
Merge branch 'feat/thematic-mapping-must-have-cherrypicks' into feat/thematic-mapping-must-have
chreman
committed
1446dde
bugfix for handling duplicates
chreman
committed
11b8d86

Commits on Feb 22, 2026

bugfix for batch retrieval
chreman
committed
5c8fb3a

Commits on Feb 24, 2026

Commits on Feb 26, 2026

Commits on Feb 27, 2026

analytic tool update
chreman
committed
a4510ce

Commits on Mar 1, 2026

group orcids before analysis
chreman
committed
995749a

Commits on May 11, 2026

Commits on May 12, 2026

e2e tests for ORCID enrichment tests
chreman
committed
c39eeae
first e2e tests for orcid enrichment bugfixes and regression testing
chreman
committed
10f58e6
add db warm-up and cache refresh to ORCID tests
chreman
committed
95d82be
deduplication test updates
chreman
committed
337453f
debug log expansion in deduplication pipeline
chreman
committed
02833dc
bugfix for wrong anchor record merge
chreman
committed
172fc51
deduplication test updates
chreman
committed
a7e86ae

Commits on May 13, 2026

Commits on May 14, 2026

debug logging for DOI case merging
chreman
committed
c186ee0
cleanup DOI case-based enrichment, replace with simplified merge
chreman
committed
5ba7980
reinstate some code
chreman
committed
727ed60
revert e2e test param change
chreman
committed
eed7e93
added tests for DOI case variants
chreman
committed
41a982b

Commits on May 15, 2026

Commits on May 19, 2026

test case update
chreman
committed
a939494
make test case titles unique
chreman
committed
2e6d31c
fix query handling in tests
chreman
committed
7371d5b
bugfix and tests for anchor document enrichment in DOI versioned datasets
chreman
committed
9f0368c
test fixes
chreman
committed
1615c38
test wrapup for flaky tests
chreman
committed
22667b4
update of contentprovider.json cache
chreman
committed
2b62546
test wrapup for flaky tests
chreman
committed
395baeb

Commits on Jun 3, 2026

e2e tests for OA status and PDF link enrichment; with some helper functions
chreman
committed
55724bb
remove link enrichment tests
chreman
committed
77cf6ad
cleanup of PDF link related debug logging
chreman
committed
f226599

Commits on Jun 5, 2026

Commits on Jun 8, 2026

update of contentprovider.json cache
chreman
committed
d538804

Commits on Jun 9, 2026

add tests for bugfix for escaping SOLR instructions in DOIs
chreman
committed
366d014
update of contentprovider.json cache
chreman
committed
e01500d

Commits on Jun 11, 2026

bugfix for SOLR parsing of DOIs with special characters
chreman
committed
96eeb02

Commits on Jun 15, 2026

Merge remote-tracking branch 'upstream/master' into feat/keyword-label-improvements
chreman
committed
d590678

Commits on Jun 16, 2026

Merge branch 'dev' into feat/keyword-label-improvements
chreman
committed
3cc0b01
comment out debug logging
chreman
committed
b753e71
clean up debug logging
chreman
committed
9dd38d2
add debug logging
chreman
committed
16c5ba9
clean up debug logging
chreman
committed
f6d2a0a

Commits on Jun 19, 2026

Commits on Jun 20, 2026

Commits on Jun 22, 2026

add docstring
chreman
committed
89fbfaf
add bugfix/expression for SDG
chreman
committed
d111266
add additional expression of FoR-classification
chreman
committed
56ca04b

Commits on Jun 23, 2026

Commits on Jun 25, 2026

Commits on Jun 26, 2026

add grant scheme cleanup; include tests
chreman
committed
fe0aa47
update of contentprovider.json cache
chreman
committed
de28cb3

Commits on Jun 30, 2026

add debug and logging functionality for summarize.R area title generation functions
chreman
committed
5809a6d

Commits on Jul 6, 2026

add wedge-in for ranking modes; config variables via env; basic tests
chreman
committed
bed26f6

Commits on Jul 7, 2026

Commits on Jul 9, 2026

Commits on Jul 10, 2026

Commits on Jul 11, 2026

added area label exclusion list, a more targeted approach than stopwords; only applied to TF-IDF label candidates
chreman
committed
a0c4dc8

Commits on Jul 12, 2026

Commits on Jul 13, 2026

mode 3 replay fixtures
chreman
committed
baf3d88
mode 3 replay fixtures
chreman
committed
d5d68c4
metadata handling bugfix
chreman
committed
c276fa1
metadata handling bugfix
chreman
committed
4fb4371

Commits on Jul 14, 2026

unit tests for metadata handling esp. of new columns
chreman
committed
7379cd0

Commits on Jul 17, 2026

update of contentprovider.json cache
chreman
committed
4bc33fb

Commits on Jul 20, 2026

update gitignore
chreman
committed
b77a730
Merge branch 'fix/base-request-spike'
chreman
committed
b738b95
Merge branch 'master' into feat/keyword-label-improvements-m3
chreman
committed
5d821e9
debug data dumps
chreman
committed
d03906d

Commits on Jul 27, 2026

add debug functionality for label generation
chreman
committed
c1f9436
add debug functionality for label generation
chreman
committed
3f19cc6

Commits on Aug 21, 2026

Commits on Aug 22, 2026

update e2e tests with new expected outcomes
chreman
committed
dc5886f
update of contentprovider.json cache
chreman
committed
bdb36d7

Commits on Aug 23, 2026

fix stability bug
chreman
committed
7853974

Commits on Aug 25, 2026

update of contentprovider.json cache
chreman
committed
205cfcb

Commits on Aug 26, 2026

bugfixes for different keyword cleaning issues in BASE
chreman
committed
b2c7da2
test updates for keyword cleaning improvements
chreman
committed
9e4a948

Commits on Aug 27, 2026

expand unit tests for keyword cleaning
chreman
committed
5de2a21
asterisk cleanup bugfix
chreman
committed
18895f0

Commits on Sep 2, 2026

Commits on Sep 3, 2026

test case updates
chreman
committed
183f7b8

Commits on Sep 4, 2026

ngram generation rework: foundational settings and test setup
chreman
committed
746725a

Commits on Sep 5, 2026

implementation of ngram construction; settings; incl. test updates
chreman
committed
db44d7d
update of contentprovider.json cache
chreman
committed
b95c632

Commits on Sep 7, 2026

Commits on Sep 10, 2026

settings bugfix
chreman
committed
0cbdf76

Commits on Sep 14, 2026

text update in ORCID researcher metrics modal
chreman
committed
4c56079

Commits on Sep 15, 2026

wip for deduplication bugfix
chreman
committed
868ee98
wip for deduplication bugfix
chreman
committed
67d5fe9
make pandas sorting deterministic
chreman
committed
90b849b
update of contentprovider.json cache
chreman
committed
0ba5adb
test cases for JEL, AMS, PACS classification cleanup
chreman
committed
c2b66ed
additional keyword cleanup for JEL, AMS, PACS classifications; arxiv classification parsing improvements; TeX quotation cleanup
chreman
committed
051fc42
update of contentprovider.json cache
chreman
committed
f32098e

Commits on Sep 16, 2026

case restoration bugfix work-in-progress
chreman
committed
f452b44
Merge branch 'feat/thematic-mapping-polishing-2' into feat/thematic-mapping-polishing-4

chreman
and
claude
committed
f24b055
adjustment of case restoration to ignore ALLCAPS titles
chreman
committed
664687e

Commits on Sep 17, 2026

Commits on Sep 21, 2026

add docstrings to deduplication.py functions
chreman
committed
c6084f3

Commits on Sep 22, 2026