Skip to content

download_huggingface_dataset hardcodes repo_type="model", so it cannot fetch datasets repos #549

Description

@MaxGhenis

policyengine_core.tools.hugging_face.download_huggingface_dataset (policyengine-core 3.32.5, policyengine_core/tools/hugging_face.py:92-94) always calls hf_hub_download(..., repo_type="model"). Run against the canonical US population release policyengine/populace-us (a Hugging Face datasets repo; certified tag populace-us-2024-spm-20260915), it raises RepositoryNotFoundError: 401, because there is no model repo of that name. Observed today while re-qualifying the PolicyEngine 6.0.0 bundle (registry re-qualification lane, 2026-09-15); the wrapper policyengine 6.0.0 avoids the helper by reading repo_type: "dataset" from its bundle manifest and downloading with the correct type, and recomputing the certified hash independently.

Suggested fix: accept a repo_type argument (default "model" for backward compatibility with policyengine-uk-data-private and the other model-typed data repos), or infer it by probing repo_info for both types; pass it through to hf_hub_download and to the private/gated check. This is separate from #538, which concerns token handling for public-but-gated repos.

🤖 Generated with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions