Skip to content

Stop over-editing informal human text - #318

Merged
blader merged 1 commit into
blader:mainfrom
wahid-slimani:informal-text-and-cross-genre-study
Nov 19, 2026
Merged

blader merged 1 commit into
blader:mainfrom
wahid-slimani:informal-text-and-cross-genre-study

Conversation

@wahid-slimani

@wahid-slimani wahid-slimani commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Humanizer rewrites informal human text as if it were AI text. In isolated runs, a Slack message went from lowercase with "lmk" to formal sentences, "Worst. Service. Ever." became "Terrible service.", "Avoid this place" became "I'd skip this place", and "Tried restarting it" became "I restarted it".

Change

  • Voice: informal text (chat, comments, rants, angry reviews) is left alone unless two unrelated tells appear together. §22 chatbot residue still acts alone. Once two tells appear, the whole text is a rewrite target. Anger keeps its strength, and a text with no tell is returned unchanged.
  • §2: adds a closing "Overall" or "Ultimately" sentence that restates a verdict.
  • When not to act: a plain "let me know" in a message is not chatbot residue.
  • Source / README: cites El Attar et al., "A Systematic Analysis of Linguistic Features in AI-Generated English Text Detection Across Domains and Models" (30 LLMs, 10 text domains), once in SKILL.md (Source) and once in the README sources. It found that lexical richness held across models and domains and that most other signals changed with them. That is the basis for the informal-text caution.
  • Review changes: SKILL.md grows by 131 words over main (5,226 to 5,357 by len(SKILL.split()), cap 5,500). The citation moved out of the Why section. No version bump: the manifests and SKILL.md stay at 4.0.0, and the CHANGELOG entry is under "Unreleased". Rebased on main (v4.0.0).

Testing

Isolated claude -p runs (no hooks, tools, or other skills), SKILL.md as the system prompt, v3.1.0 vs the earlier version of this branch, 10 runs per text. Checks were written before the runs. Eighteen short texts: 8 human informal controls (including 4 held-out ones added after the first results) and 10 AI-written texts.

  • Human controls: all 8 came back unchanged (similarity 1.00). v3.1.0 changed 7 of the 8 (similarity 0.38 to 0.99) and dropped required words in 4 of them (for example "Worst", "Avoid", "lmk", "tbh").
  • AI texts: no measurable change. A blind pairwise judge, with fidelity first, gave 24 to this branch, 25 to v3.1.0, and 11 ties over 60 comparisons. Remaining tells are within noise (for example "not just a bug" survived 5/10 vs 3/10).
  • Not tested: longer texts, other models, other languages. These runs used the version before the trim in this rebase; the trimmed Voice paragraph keeps the same rules but was not re-run.

python scripts/validate-package.py passes on the rebased branch (package v4.0.0).

Copilot AI balanced review requested due to automatic review settings October 24, 2026 19:47

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

dgtlmonk pushed a commit to dgtlmonk/humanizer that referenced this pull request Oct 29, 2026
@blader

blader commented Nov 10, 2026

Copy link
Copy Markdown
Owner

Thanks, the human-control results make the case. Two things before I merge. SKILL.md is at 5,316 words, so this needs to come in at about 130; citing the study once, in Source, would get most of the way. And the arXiv page lists 30 LLMs and has "English" in the title, where this says 27 models. Please rebase on main and drop the version bump; I'll set the version at release.

@wahid-slimani
wahid-slimani force-pushed the informal-text-and-cross-genre-study branch from ddc830d to 5f9dbeb Compare November 10, 2026 09:05
@wahid-slimani wahid-slimani changed the title Stop over-editing informal human text (3.2.0) Stop over-editing informal human text Nov 10, 2026
@wahid-slimani

Copy link
Copy Markdown
Contributor Author

Done, all three:

  • Length: SKILL.md now adds 131 words over main (5,226 to 5,357). I cut the Voice paragraph, dropped the cross-reference in §2, and removed the citation from the Why section. The study is cited once in SKILL.md, in Source.
  • Citation: corrected to 30 LLMs, with "English" in the title as listed on the arXiv page. I fixed the title in the README sources too.
  • Rebase and version: rebased on main (v4.0.0). The version bump is gone: SKILL.md and both plugin manifests stay at 4.0.0, and the CHANGELOG entry is under "Unreleased" for you to name at release.

python scripts/validate-package.py passes. One caveat: the human-control results in the description come from the earlier, longer Voice paragraph. The trimmed one keeps the same rules but I have not re-run the 8 controls on it. I can do that if you want it before merge.

@wahid-slimani

Copy link
Copy Markdown
Contributor Author

Rebased on v4.0.0. SKILL.md is now +131 words, with the study cited once in Source (30 LLMs, 'English' in the title). The version bump is dropped, and the changelog entry is under Unreleased.

Chat messages, rants, and angry reviews were rewritten into polished
prose: lowercase and "lmk" formalised, fragments merged, "Worst. Service.
Ever." softened, "tried X" turned into "did X". Add a rule to Voice:
leave informal text alone unless two unrelated tells appear together
(chatbot residue excepted), keep the writer's anger at full strength,
and return the text unchanged if no tell is found.

Also add a restating "Overall"/"Ultimately" verdict to the closers in
pattern 2, note that a plain "let me know" is not chatbot residue, and
cite El Attar et al. (arXiv:2606.04177) once, in Source, which found
that most linguistic signals of AI text change with the model and the
text type.
@blader
blader force-pushed the informal-text-and-cross-genre-study branch from 5f9dbeb to 3e5f816 Compare November 19, 2026 10:07
@blader
blader merged commit a48c60c into blader:main Nov 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants