A modular, config-driven pipeline for processing the Active Inference Institute video library.
Download Β· Transcribe Β· Extract Β· Export Β· Browse Β· Chat
- How It Works
- Status
- Features
- Quick Start
- Architecture
- Documentation
- Testing
- Journal v2 maintenance
- Transcription & speaker naming
- Environment
- Contributing
- InstituteOS sidecar
graph LR
subgraph Ingest
A[π¬ YouTube Channel] -->|yt-dlp| B[π₯ Download]
B --> C[π Transcripts]
B --> D[π΅ Audio]
end
subgraph Process
D -->|mlx-whisper / WhisperX| C
C -->|Cohere AI| E[π§ Entities & Graph]
C -->|5 formats| F[π Export]
end
subgraph Serve
C --> G[π Web Interface]
E --> G
G -->|Ollama RAG| H[Chat]
end
style A fill:#e63946,color:#fff
style G fill:#457b9d,color:#fff
style H fill:#2a9d8f,color:#fff
One command runs the configured application pipeline:
uv run python run.pyOne file controls all options:config.iniβ see reference β
Maintained and working. What runs without network or credentials:
- configuration loading, transcript export, the pytest suite (offline tests), the
static site builder (
scripts/build_pages_site.py, dry-run by default), and the journal v2 maintenance scripts in their read-only/dry-run modes.
What needs network, credentials or a private checkout:
run.py downloadand channel enumeration need YouTube access viayt-dlp; the YouTube Data API helpers additionally need OAuth credentials and spend quota.- Transcription needs WhisperX/GPU models (
HUGGINGFACE_TOKEN) or Apple Siliconmlx-whisper. - Entity extraction needs
COHERE_API_KEYand a live SurrealDB; the web interface needs the.[interface]group and, for chat, a running Ollama. - Journal maintenance commands (
journal-check, enrichment, repair, index generation) need the siblingActiveInferenceJournalcheckout passed explicitly.
|
Enumerate the Active Inference channel from a saved manifest. Download transcripts, audio, and video (cookie-free by default) with rate limiting and resume. Transcribe locally on Apple Silicon or GPU with speaker diarization. β Download Guide Β· Transcription Engines Β· YouTube Module |
FastAPI SPA with searchable video library, embedded YouTube player, transcript viewer, and category browser. Ollama-powered RAG chat with automatic context retrieval. β Web Interface Β· Chat Engine |
|
Batch-export to Markdown, JSON, HTML, PDF, and plaintext β each enriched with metadata headers (title, category, series, speakers, duration, URL, views). β Export Guide |
Cohere AI entity extraction (people, concepts, theories, organizations) and relationship mapping into a SurrealDB knowledge graph. β RAG Pipeline Β· Data & Database |
# 1. Clone & install
git clone https://github.com/ActiveInferenceInstitute/Journal_Utilities.git
cd Journal_Utilities
uv sync --all-extras
# 2. Run the default pipeline (Config β Validate β Export β Test β Serve)
uv run python run.pyuv run python run.py config # Show current configuration
uv run python run.py download # Download from YouTube
uv run python run.py export # Export transcripts to all enabled formats
uv run python run.py test # Run the pytest suite
uv run python run.py serve # Launch web UI at http://localhost:8000
uv run python run.py full # Full pipeline: download β export
uv run python run.py journal-check # Read-only journal integrity gategraph TB
subgraph "CLI Layer"
RUN["run.py β Pipeline Runner"]
S1["scripts/download_channel.py"]
S2["scripts/transcribe_missing.py<br/>scripts/transcribe_worklist.py"]
S3["scripts/scaffold_youtube_courses.py"]
S4["scripts/validate_journal.py"]
S5["scripts/speaker_cues.py<br/>scripts/apply_speaker_names.py"]
end
subgraph "src/journal_utilities/"
direction TB
YT["youtube/<br/>channel Β· playlist Β· categorizer"]
DL["download/<br/>downloader"]
TR["transcribe/<br/>mlx-whisper Β· WhisperX"]
EX["export/<br/>exporter (5 formats)"]
DATA["data/<br/>database Β· importer"]
RAG["rag/<br/>extractors Β· graph Β· models"]
IF["interface/<br/>app Β· chat_engine Β· data_loader"]
RN["render/<br/>renderer"]
end
subgraph "External Services"
OL["Ollama (LLM)"]
DB["SurrealDB"]
CO["Cohere AI"]
YT_API["YouTube (yt-dlp)"]
end
RUN --> EX & IF & DL & S4
S1 --> YT & DL
S2 --> TR
S3 --> RN
DL --> YT_API
RAG --> CO & DB
DATA --> DB
IF --> OL
style RUN fill:#e63946,color:#fff
style IF fill:#457b9d,color:#fff
style EX fill:#2a9d8f,color:#fff
Full directory tree
Journal-Utilities/
βββ src/journal_utilities/ # Main package
β βββ youtube/ # Channel enumeration, categorizer
β βββ download/ # yt-dlp download engine
β βββ transcribe/ # MLX-Whisper + WhisperX
β βββ data/ # SurrealDB client + Coda importer
β βββ interface/ # FastAPI SPA + Ollama chat
β βββ rag/ # Entity extraction pipeline
β βββ render/ # Course scaffolding
β βββ export/ # Multi-format transcript export
β βββ ingest/ # Channel enumeration/scaffold ingest helpers
β βββ site/ # Static GitHub Pages site builder
β βββ llm/ # LLM tool integration (placeholder)
β βββ utils/ # Shared utilities
βββ scripts/ # CLI tools
βββ tests/ # Pytest suite
βββ data/ # Input, output, database storage
βββ docs/ # Module and workflow guides
βββ run.py # Pipeline runner
βββ config.ini # All configuration
βββ pyproject.toml # Python 3.12+
All technical detail lives in docs/. The README you're reading is the overview and entry point.
See docs/JOURNAL_SCHEMA.md for the ActiveInferenceJournal v2 schema
and docs/REFACTOR_READINESS.md for the refactor pipeline
(scripts/refactor_journal.py).
| Guide | What You'll Find |
|---|---|
| Configuration | config.ini sections, environment variables, pipeline step control |
| YouTube | Channel enumeration, playlist parsing, title categorization |
| Download | Cookie auth, 403 troubleshooting, download strategies |
| Transcription | MLX-Whisper (Mac), WhisperX (GPU), model selection |
| Subtitles β Translations | Hosted OpenRouter subtitle translation |
| Export | Format details, metadata enrichment, library API |
| Web Interface | API endpoints, SPA frontend, development server |
| Chat Engine | Ollama RAG, prompt engineering, model auto-discovery |
| RAG & Graph | Cohere extraction, entity schema, knowledge graph |
| Data & Database | SurrealDB schema, Coda import, audit trails |
| Render | Playlist β course scaffolding, module.md format |
| Site | Static GitHub Pages bundle, URL contract, sitemap |
| M3 Ingest | Weekly channel enumeration, scaffold, reconciliation |
| Privacy | Processing boundaries, API usage, retention |
| Gate & CI | run.py gate arms, environment, measured wall time |
| Agent Guide | Architecture, code patterns, agent development rules |
uv run python run.py gate # lint, format, types, tests, docs, gnn
uv run python run.py gate test-full # everything on this host, with coverage
uv run python run.py journal-check # Journal corpus integrity gateThe gate is one entry point with separately runnable arms; CI calls the same
commands. Test counts and per-arm wall time are measured, not typed: see
docs/RUN_GUIDE.md and the generated
docs/_generated/gate-arms.md.
Journal-Utilities is the code-side source of truth for the generated ActiveInferenceJournal layout. The maintenance sequence is explicit and safe to repeat:
uv run python scripts/enrich_metadata.py --journal ../ActiveInferenceJournal
uv run python scripts/enrich_metadata.py --journal ../ActiveInferenceJournal --apply
uv run python scripts/repair_split_transcripts.py --journal ../ActiveInferenceJournal --utilities .
uv run python scripts/generate_journal_indexes.py --journal ../ActiveInferenceJournal
uv run python run.py journal-checkEnrichment is dry-run by default; only the command with --apply writes metadata.
The final journal-check command is read-only and blocks handoff when metadata,
indexes, transcript identities, duplicate handling, coverage, or the main
branch's no-audio/no-credentials boundary is inconsistent.
Journal transcripts follow a raw vs derived design (see
docs/JOURNAL_SCHEMA.md): transcript.json is the
immutable raw WhisperX diarization (SPEAKER_NN labels, never rewritten),
human speaker names live only in metadata.json parts[].speakers, and
transcript.txt is regenerated from the two. Private/unlisted videos are
never transcribed. All commands are dry-run by default.
# corpus status: diarized / captions-only / excluded / scheduled
uv run python scripts/transcription_status.py
# WhisperX + diarization for items missing transcripts (GPU, resumable)
uv run python scripts/transcribe_worklist.py # plan
uv run python scripts/transcribe_worklist.py --run
# identify speakers: timestamped YouTube links per SPEAKER_NN
uv run python scripts/speaker_cues.py # items still needing names
uv run python scripts/speaker_cues.py --item TextbookGroup/Namjoshi2026/Cohort_1/Session_024
# record parts[].speakers in the item's metadata.json, then regenerate transcript.txt
uv run python scripts/apply_speaker_names.py --item <item> --applyRequired in .env:
| Variable | Purpose |
|---|---|
HUGGINGFACE_TOKEN |
WhisperX speaker diarization |
COHERE_API_KEY |
Entity extraction (RAG) |
CODA_API_TOKEN |
Coda session data |
OLLAMA_MODEL |
Chat model (default: gemma3:4b) |
OLLAMA_BASE_URL |
Ollama API URL (default: http://localhost:11434) |
See CONTRIBUTING.md for the dev setup, gate workflow, and pull-request process. Security and cookie-handling policy: SECURITY.md. If you use or build on this work, please cite it β see CITATION.cff.
WhisperX pipeline & SurrealDB β Holly Grimm @hollygrimm (2024) YouTube download & local Whisper β 2025β2026 AssemblyAI scripts β Dave Douglass (2022)
This repository carries a committed .aii/ sidecar with curated
InstituteOS metadata (aii-sidecar/v1): category, maturity, ecosystem relations,
key artifacts, provenance, and portable tasks. It is read by
InstituteOS tooling and
does not affect the project's build or runtime.