From 6b2d7add0131b7560a912d3c5d2298a6f18e447c Mon Sep 17 00:00:00 2001 From: Chen Wang Date: Wed, 16 Sep 2026 11:57:04 -0500 Subject: [PATCH] Add secret-scan: git history secret sweep tooling MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a script that sweeps the full git history of every IN-CORE repo with gitleaks and trufflehog and compiles a redacted report. Covers the two gaps in GitHub's own secret scanning: private repos (not available on the org's free plan, currently 9 of 29 repos) and credentials that don't match GitHub's ~200 partner patterns — DB passwords, Keycloak client secrets, connection strings, committed .pem files. Scans mirror clones so commits reachable only from deleted branches are included. TruffleHog can't read bare repos, so each is materialised as a temporary --shared working clone; objects are shared via alternates, and refs from every branch stay visible. Live credential verification is opt-in via VERIFY=1 and off by default, since it exercises any real key found against the provider's API. Co-Authored-By: Claude Opus 5 --- .gitignore | 4 + README.md | 7 +- secret-scan/README.md | 113 +++++++++++++++++++++ secret-scan/scan-secrets.sh | 189 ++++++++++++++++++++++++++++++++++++ 4 files changed, 312 insertions(+), 1 deletion(-) create mode 100644 secret-scan/README.md create mode 100755 secret-scan/scan-secrets.sh diff --git a/.gitignore b/.gitignore index cbb5e13..9d05566 100644 --- a/.gitignore +++ b/.gitignore @@ -70,3 +70,7 @@ keycloak/.env utils/geoserver_utils/data/ utils/gis_utils/data/ *env* + +# secret-scan working data — mirrors and raw scan output contain unredacted secrets +secret-scan/sweep/ +secret-scan/reports/ diff --git a/README.md b/README.md index 3bbed01..7e28f20 100644 --- a/README.md +++ b/README.md @@ -26,4 +26,9 @@ Parses existings fragility and mapping xml files and converts them to a POJO and Utilities for the INCORE V2 geoserver ### webdav_utils ### -Dataset obejct and utitlies to import webdav data and store in data repository \ No newline at end of file +Dataset obejct and utitlies to import webdav data and store in data repository +# secret-scan +Sweeps the full git history of every IN-CORE repository for committed secrets, using +gitleaks and trufflehog, and compiles a redacted report. Covers the gaps GitHub's own +secret scanning leaves: private repos (not available on the free plan) and credentials +that do not match GitHub's partner patterns. See [secret-scan/README.md](secret-scan/README.md). diff --git a/secret-scan/README.md b/secret-scan/README.md new file mode 100644 index 0000000..f0445c4 --- /dev/null +++ b/secret-scan/README.md @@ -0,0 +1,113 @@ +# secret-scan + +A one-time (or repeatable) sweep for secrets committed anywhere in the **history** +of the IN-CORE GitHub repositories. + +## Why this exists + +GitHub's own secret scanning is already enabled on the public IN-CORE repos, and +when you turn it on it backfills — it scans every commit on every branch, not just +new pushes. That coverage is real, but it has two gaps this script fills: + +1. **Private repos aren't covered.** The org is on the free plan, and secret + scanning for private repositories requires the paid GitHub Secret Protection + add-on. At the time of writing that is 9 of the org's 29 repos. +2. **Only partner patterns are matched.** GitHub detects roughly 200 credential + formats with recognisable shapes (AWS keys, GitHub PATs, Slack tokens). It will + not flag a hardcoded database password, a Keycloak client secret, a connection + string with inline credentials, or a `.pem` committed into a config directory — + which is where a platform like ours is most likely to leak something. + +So a clean bill of health from GitHub is necessary but not sufficient. This script +runs two independent scanners over full history to cover the rest. + +## What it runs + +| tool | what it is good at | +|---|---| +| [gitleaks](https://github.com/gitleaks/gitleaks) | regex + entropy rules; catches the generic stuff (passwords, connection strings, private keys) | +| [trufflehog](https://github.com/trufflesecurity/trufflehog) | detector-based, and can **verify** a credential by calling the provider's API to see if it still works | + +Both are run over `--all --full-history`, across every ref, including commits only +reachable from deleted branches. + +## Requirements + +```bash +brew install gitleaks trufflehog # also needs git, python3, and gh for --clone +``` + +On Intel macOS, trufflehog has no prebuilt bottle and will compile from source +(pulls the Go toolchain; budget 10-15 minutes). gitleaks installs from a bottle. + +## How to run + +```bash +./scan-secrets.sh --clone # mirror every IN-CORE repo into ./sweep, then scan +./scan-secrets.sh # scan the mirrors already in ./sweep +``` + +`--clone` skips repos already present, so it is safe to re-run. Cloning uses +`--mirror` deliberately: a normal clone would miss commits that are only reachable +from deleted branches, which is exactly where forgotten secrets tend to survive. + +Expect roughly 1.2 GB of mirrors and about 10 minutes for a full pass. + +### Verification mode + +```bash +VERIFY=1 ./scan-secrets.sh +``` + +Off by default. When **off**, trufflehog runs fully offline and reports every +candidate — nothing leaves your machine, but you hand-triage a larger pile. + +When **on**, trufflehog calls each provider's API to check whether a credential is +still live. This is what turns hundreds of candidates into a short actionable list, +and it is genuinely useful — but understand what it does first: any real key found +in history gets exercised against AWS/GitHub/Slack/etc. from the machine running +the scan, and a live hit may appear in that provider's audit log. Turn it on +deliberately, not by habit. + +## How to read the report + +Each run writes a timestamped directory: + +``` +reports// +├── report.md compiled summary — start here +├── scan.log full transcript, and the live progress view while a scan runs +└── raw/ per-repo gitleaks JSON and trufflehog JSONL, unredacted +``` + +`report.md` has a summary table of every repo sorted with verified-live findings +first, then a per-repo detail table giving tool, rule/detector, file, commit, date, +author, and the secret **redacted** to first/last four characters. That means the +report is safe to attach to an issue or paste into a ticket. Full values stay in +`raw/` — treat that directory as sensitive and do not commit it. + +`report.md` is only written once every repo has finished. While a scan is running, +tail `scan.log` to watch progress. + +### Triage + +**Raw counts are not findings.** Both tools are tuned to over-report, and most hits +are UUIDs, git SHAs, lockfile integrity digests, and test fixtures. Read the report +before acting on a number. A previous run flagged two "API keys" in a Playbook repo +that turned out to be DataWolf workflow parameter UUIDs. + +When something is a genuine leak: + +1. **Rotate the credential first.** This is the actual fix. +2. **Then decide about history.** Rewriting with `git filter-repo` or BFG breaks + every existing clone and fork, and GitHub keeps the old commits reachable by SHA + until you ask Support to garbage-collect them. A secret that was ever pushed to a + public repo should be treated as permanently compromised no matter what you do to + the history — which is why rotation is the fix and the rewrite is cleanup. + +## Ongoing scanning + +This script is for auditing history. To stop new secrets going in, use GitHub push +protection (already enabled on the public repos) plus a gitleaks pre-commit hook or +CI job, which also covers the private repos and the non-partner patterns GitHub +does not match. diff --git a/secret-scan/scan-secrets.sh b/secret-scan/scan-secrets.sh new file mode 100755 index 0000000..5232bfa --- /dev/null +++ b/secret-scan/scan-secrets.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash +# +# scan-secrets.sh — one-time secret-history sweep across the IN-CORE org. +# +# Iterates every mirror clone under ./sweep/*.git, runs gitleaks and trufflehog +# over the FULL history of every ref, and compiles a report. +# +# Usage: +# ./scan-secrets.sh # scan existing mirrors in ./sweep +# ./scan-secrets.sh --clone # (re-)mirror all IN-CORE repos first +# VERIFY=1 ./scan-secrets.sh # let trufflehog verify creds against live APIs +# +# VERIFY: off by default. When off, trufflehog runs fully offline and reports +# every candidate (noisier, but nothing leaves this machine). When on, trufflehog +# calls each provider's API to check whether a credential still works — that turns +# hundreds of candidates into a short actionable list, but it exercises any real +# key found against AWS/GitHub/Slack/etc. from this laptop, and live hits may show +# up in that provider's audit log. Turn it on deliberately. + +set -uo pipefail + +ORG=IN-CORE +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SWEEP="$ROOT/sweep" +STAMP="$(date +%Y%m%d-%H%M%S)" +OUT="$ROOT/reports/$STAMP" +RAW="$OUT/raw" +LOG="$OUT/scan.log" +REPORT="$OUT/report.md" +VERIFY="${VERIFY:-0}" + +WORKDIR="$(mktemp -d "${TMPDIR:-/tmp}/incore-sweep-XXXXXX")" +trap 'rm -rf "$WORKDIR"' EXIT + +mkdir -p "$RAW" + +log() { printf '%s %s\n' "[$(date +%H:%M:%S)]" "$*" | tee -a "$LOG"; } + +for bin in git gitleaks trufflehog python3; do + command -v "$bin" >/dev/null || { echo "FATAL: '$bin' not found on PATH" >&2; exit 1; } +done + +# ---------------------------------------------------------------- clone (opt) +if [[ "${1:-}" == "--clone" ]]; then + command -v gh >/dev/null || { echo "FATAL: gh required for --clone" >&2; exit 1; } + log "mirroring $ORG repos into $SWEEP" + mkdir -p "$SWEEP" + gh repo list "$ORG" --limit 200 --json name -q '.[].name' \ + | xargs -P4 -I{} sh -c "[ -d '$SWEEP/{}.git' ] || git clone --quiet --mirror https://github.com/$ORG/{}.git '$SWEEP/{}.git'" +fi + +shopt -s nullglob dotglob +REPOS=("$SWEEP"/*.git) +shopt -u dotglob +[[ ${#REPOS[@]} -gt 0 ]] || { echo "FATAL: no mirrors found in $SWEEP (run with --clone)" >&2; exit 1; } + +log "scanning ${#REPOS[@]} repos gitleaks=$(gitleaks version) trufflehog=$(trufflehog --version 2>&1 | head -1)" +log "trufflehog verification: $([[ $VERIFY == 1 ]] && echo 'ON (live API calls)' || echo 'OFF (offline)')" +log "output: $OUT" + +TH_FLAGS=(--json --no-update) +[[ "$VERIFY" == 1 ]] || TH_FLAGS+=(--no-verification) + +# ---------------------------------------------------------------------- scan +for repo in "${REPOS[@]}"; do + name="$(basename "$repo" .git)" + commits=$(git -C "$repo" rev-list --all --count 2>/dev/null || echo 0) + log "--- $name ($commits commits)" + + gitleaks git "$repo" \ + --report-format json \ + --report-path "$RAW/$name.gitleaks.json" \ + --log-opts="--all --full-history" \ + >>"$LOG" 2>&1 + gl_rc=$? + [[ -f "$RAW/$name.gitleaks.json" ]] || echo '[]' > "$RAW/$name.gitleaks.json" + + # trufflehog cannot read a bare/mirror repo (it stats for .git and an index), + # so materialise a temporary --shared working clone: objects are shared via + # alternates (no history duplication) and refs from every branch stay visible. + work="$WORKDIR/$name" + rm -rf "$work" + if git clone --quiet --shared "$repo" "$work" 2>>"$LOG"; then + trufflehog git "file://$work" "${TH_FLAGS[@]}" \ + > "$RAW/$name.trufflehog.jsonl" 2>>"$LOG" + th_rc=$? + else + log " WARN: could not create working clone; skipping trufflehog" + : > "$RAW/$name.trufflehog.jsonl" + th_rc=127 + fi + rm -rf "$work" + + gl_n=$(python3 -c "import json,sys;print(len(json.load(open(sys.argv[1]))))" "$RAW/$name.gitleaks.json" 2>/dev/null || echo 0) + th_n=$(wc -l < "$RAW/$name.trufflehog.jsonl" 2>/dev/null | tr -d ' ' || echo 0) + log " gitleaks=$gl_n (rc=$gl_rc) trufflehog=$th_n (rc=$th_rc)" +done + +# -------------------------------------------------------------------- report +log "compiling report" +VERIFY="$VERIFY" OUT="$OUT" RAW="$RAW" REPORT="$REPORT" STAMP="$STAMP" python3 <<'PY' +import json, os, glob, collections + +RAW, REPORT, STAMP = os.environ['RAW'], os.environ['REPORT'], os.environ['STAMP'] +VERIFY = os.environ['VERIFY'] == '1' + +def redact(s, keep=4): + s = (s or '').strip() + if len(s) <= keep * 2: return '*' * len(s) + return f"{s[:keep]}...{s[-keep:]} (len {len(s)})" + +rows, detail = [], collections.OrderedDict() + +# NB: glob.glob() skips dotfiles, which would silently drop dot-named repos +# such as the org's `.github` repo from the report. Enumerate the dir instead. +gl_files = sorted(os.path.join(RAW, x) for x in os.listdir(RAW) + if x.endswith('.gitleaks.json')) +for f in gl_files: + name = os.path.basename(f)[:-len('.gitleaks.json')] + try: gl = json.load(open(f)) + except Exception: gl = [] + + th, thfile = [], f'{RAW}/{name}.trufflehog.jsonl' + if os.path.exists(thfile): + for line in open(thfile): + line = line.strip() + if not line: continue + try: + o = json.loads(line) + if o.get('SourceMetadata'): th.append(o) + except Exception: pass + + verified = [o for o in th if o.get('Verified')] + rows.append((name, len(gl), len(th), len(verified))) + + items = [] + for x in gl: + items.append(dict(tool='gitleaks', rule=x.get('RuleID',''), file=x.get('File',''), + commit=(x.get('Commit') or '')[:10], date=x.get('Date',''), + author=x.get('Author',''), verified=None, + secret=redact(x.get('Secret','')))) + for o in th: + g = (o.get('SourceMetadata',{}).get('Data',{}).get('Git',{}) or {}) + items.append(dict(tool='trufflehog', + rule=f"{o.get('DetectorName','')}", file=g.get('file',''), + commit=(g.get('commit') or '')[:10], date=g.get('timestamp',''), + author=g.get('email',''), verified=bool(o.get('Verified')), + secret=redact(o.get('Raw','')))) + if items: detail[name] = items + +rows.sort(key=lambda r: (-r[3], -(r[1]+r[2]), r[0])) +tot = [sum(r[i] for r in rows) for i in (1,2,3)] + +with open(REPORT,'w') as fh: + w = fh.write + w(f"# IN-CORE secret-history sweep\n\n") + w(f"- Run: `{STAMP}`\n- Repos scanned: **{len(rows)}**\n") + w(f"- TruffleHog verification: **{'ON' if VERIFY else 'OFF (offline — Verified column not meaningful)'}**\n") + w(f"- Raw findings: gitleaks **{tot[0]}**, trufflehog **{tot[1]}**" + + (f", of which **verified live: {tot[2]}**\n\n" if VERIFY else "\n\n")) + w("> Raw counts are pre-triage and include false positives (UUIDs, hashes,\n" + "> test fixtures, lockfile digests). Triage before acting. Secrets below are\n" + "> redacted; full values are in `raw/`.\n\n") + + w("## Summary\n\n| repo | gitleaks | trufflehog | verified |\n|---|---:|---:|---:|\n") + for n,g,t,v in rows: + w(f"| {n} | {g} | {t} | {v if VERIFY else '–'} |\n") + w(f"| **total** | **{tot[0]}** | **{tot[1]}** | **{tot[2] if VERIFY else '–'}** |\n\n") + + w("## Findings by repo\n\n") + if not detail: w("_No findings._\n") + for name, items in detail.items(): + w(f"### {name} ({len(items)})\n\n") + w("| tool | rule/detector | file | commit | date | verified | secret |\n|---|---|---|---|---|---|---|\n") + for i in sorted(items, key=lambda x: (x['verified'] is not True, x['file'])): + v = '**LIVE**' if i['verified'] else ('no' if i['verified'] is False else '–') + w(f"| {i['tool']} | {i['rule']} | `{i['file']}` | `{i['commit']}` | {i['date'][:10]} | {v} | `{i['secret']}` |\n") + w("\n") + +print(f"repos={len(rows)} gitleaks={tot[0]} trufflehog={tot[1]} verified={tot[2]}") +PY + +log "report written: $REPORT" +echo +echo "====================================================================" +echo " report : $REPORT" +echo " raw : $RAW" +echo " log : $LOG" +echo "===================================================================="