Repository navigation
Add automatic 'See also' recommendations to every wiki article - #241
Shreenabh664 wants to merge 4 commits into
Conversation
4f6c490 to
ca3abce
Compare
|
Implementation looks good to me, but how does this feature interact with existing pages that have a statically written "See also" section? Will the article have two See alsos? |
Pre-render check in the client-side JS: scan the article body for any H2-H4 heading whose text matches "See also" (case-insensitive, optional trailing : or .) and bail out before fetching/rendering the auto recs. This addresses the duplicate-section concern raised by p-amyjiang on the PR -- about 57 wiki articles already ship a manually written See also list, and they should keep authorial curation untouched. The auto-injected `.sa-heading` is excluded from the detector so the guard stays idempotent if the script ever re-runs on the same DOM.
Fixed, we retain author's curation if present & dont inject a new section now. |
|
LGTM |
nevalsar
left a comment
There was a problem hiding this comment.
Thanks for building this. Cross-linking articles is something the wiki needs, and the browser side is clean. The scoring needs another pass before we merge, though. I ran a copy of it against the current articles and many picks are off-topic. For example, git recommends Cable Management and Open-Source Datasets, and tmux recommends Stereo Vision and Risk Management.
- Rare everyday words score highest. With only the first 400 characters of about 200 articles, words like "watch", "youtube", "free", "complete" and "list" land in the rare bucket and get 5 points each. tmux and Stereo Vision are linked because both say "watch a YouTube video". A larger stoplist and more of each article than the first 400 characters should help.
- Title words skip the frequency filter. One shared title word is worth 7 points and clears MIN_SCORE on its own, so words like "robot", "system", "introduction", "setup" and "guide" drive about 56 recommendations by themselves.
- Broken nav URLs. About 30 entries in _data/navigation.yml point at .md paths or wrong-case folders (for example /wiki/machine-learning/nlp_for_robotics.md/). Roughly 10% of all recommendations link to those and 404. Those 30 articles also never get a See also list, because their JSON key doesn't match the page path. Please skip entries whose page lookup fails and use page.url instead of the nav URL.
- Duplicate nav entries. A few articles are listed twice in the nav, so about 22 pages show the same recommendation twice. Please dedupe by URL.
- Short acronyms. Title words need 3 letters but body words need 4, so ROS, IMU, GPS and PID only match through titles.
- Glued words. strip_newlines removes line breaks without adding a space, so words merge, for example "bestpractices". The impact is small, but the fix is easy.
- Tuning and build time. The comments mention an ablation harness and 26 hand-labelled articles. Could you add those to the PR so we can rerun them after changes? Please also share build times on master with and without this change, since every article is scored against every other one.
One smaller thing: the include assigns to page, which shadows Jekyll's own variable, so a different name would be safer.
Thanks again. I'll take another look once these are in.
The algorithm auto-identifies the top N (≤ 4, based on confidence) most relevant articles to the current one and references them at the bottom in a "See also" section. It's a fully dynamically computed feature at build time - which means it doesn't require manual curation, new articles get matched automatically, and if a more relevant article to a specific existing topic is added in the future, it'll get auto-referenced.
This is a step to take this wiki closer to conventional wikis (like Wikipedia), which are very well cross-referenced - users can go on independent learning paths by simply following the suggested articles after reading any page.