Bitcache

Bitcache is a distributed content-addressable storage (CAS) system.

_{[Features] |
[Prerequisites] |
[Installation] |
[Examples] |
[Reference] |
[Development]}

✨ Features

Available both as the command-line tool bitcache and a polyglot library.
Polyglot software also ^_(soon!) for Dart, Python, Ruby, Rust, and TypeScript.
Cuts red tape: 100% free and unencumbered public domain software.

⬇️ Installation

Installation of the CLI

Installation via Cargo Binstall

cargo binstall -y bitcache

Installation via mise

mise use -g github:artob/bitcache

Installation via Cargo

cargo install bitcache --locked --features=cli

Installation of the Library

Installation for Rust from Crates.io

Installation from Crates.io

cargo add bitcache

Installation for JavaScript/TypeScript from NPM

Installation from NPM

npm install bitcache.js
bun add bitcache.js
pnpm add bitcache.js
yarn add bitcache.js

Installation for Dart from Pub.dev

Installation from Pub.dev

dart pub add bitcache
flutter pub add bitcache

Installation for Python from PyPI

Installation from PyPI

pip install -U bitcache
uv add bitcache
poetry add bitcache
pdm add bitcache

Installation for Ruby from RubyGems

Installation from RubyGems

gem install bitcache
bundle add bitcache

👉 Examples

📚 Reference

Command-Line Interface

$ bitcache --help
Bitcache is a distributed content-addressable storage (CAS) system.

Usage: bitcache [OPTIONS] [COMMAND]

General commands:
  id      Compute the BLAKE3 hash of the given file(s)
  help    Print this message or the help of the given subcommand(s)

Current repository commands (`$BITCACHE_URL`, default `./.bitcache/`):
  init    Initialize a new repository in `./.bitcache/`
  list    List the IDs of the blobs in the repository, in ascending order
  has     Check whether the repository contains blob(s) with the given ID(s)
  get     Fetch blob(s) from the repository, writing their contents to stdout
  put     Store the given file(s) into the repository as blob(s)
  rm      Remove blob(s) with the given ID(s) from the repository
  clear   Remove all blobs from the repository
  export  Export all blobs in the repository into a tarball

Remote repository commands:
  push    Copy blobs missing from the given remote repositories to them
  pull    Copy blobs missing from the current repository from the given remotes
  sync    Synchronize with the given remote repositories, in both directions

Options::
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

      --license
          Show license information

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -V, --version
          Print version information

  -h, --help
          Print help (see a summary with '-h')

bitcache clear - Remove all blobs from the repository
bitcache export - Export all blobs in the repository into a tarball
bitcache get - Fetch blob(s) from the repository, writing their contents to stdout
bitcache has - Check whether the repository contains blob(s) with the given ID(s)
bitcache id - Compute the BLAKE3 hash of the given file(s)
bitcache init - Initialize a new repository in ./.bitcache/
bitcache list - List the IDs of the blobs in the repository, in ascending order
bitcache pull - Copy blobs missing from the current repository from the given remotes
bitcache push - Copy blobs missing from the given remote repositories to them
bitcache put - Store the given file(s) into the repository as blob(s)
bitcache rm - Remove blob(s) with the given ID(s) from the repository
bitcache sync - Synchronize with the given remote repositories, in both directions

`bitcache clear`

$ bitcache clear --help
Remove all blobs from the repository.

As a safety measure, this requires the `--force` flag; without it, nothing is removed and the command exits with a usage error.

Usage: bitcache clear [OPTIONS]

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -f, --force
          Actually perform the operation; without this, nothing is removed

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache export`

$ bitcache export --help
Export all blobs in the repository into a tarball

Usage: bitcache export [OPTIONS] --output <FILE>

Options:
      --color <COLOR>  Set the color output mode [default: auto] [possible values: auto, always, never]
  -o, --output <FILE>  The path to the tarball file to create
  -d, --debug          Enable debugging output
  -v, --verbose...     Enable verbose output (may be repeated for more verbosity)
  -h, --help           Print help

`bitcache get`

$ bitcache get --help
Fetch blob(s) from the repository, writing their contents to stdout.

Exits with a nonzero status unless all of the given blobs were found in the repository.

Usage: bitcache get [OPTIONS] [IDS]...

Arguments:
  [IDS]...
          The IDs of the blob(s) to fetch

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache has`

$ bitcache has --help
Check whether the repository contains blob(s) with the given ID(s).

With `--verbose`, prints `true` or `false` for each ID.

Exits with a nonzero status unless all of the given blobs were found in the repository.

Usage: bitcache has [OPTIONS] [IDS]...

Arguments:
  [IDS]...
          The IDs of the blob(s) to check for

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache id`

$ bitcache id --help
Compute the BLAKE3 hash of the given file(s).

Prints the ID each file would have as a blob, one per line, without accessing or modifying any repository.

Usage: bitcache id [OPTIONS] [FILES]...

Arguments:
  [FILES]...
          The paths to the file(s) to hash

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -f, --format <FORMAT>
          The format to use for the hash output

          Possible values:
          - hex:    Hexadecimal (aka Base16)
          - base58: Base58

          [default: hex]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache init`

$ bitcache init --help
Initialize a new repository in `./.bitcache/`.

Creates an empty repository in the `./.bitcache/` directory of the current working directory; `$BITCACHE_URL` is ignored.

Usage: bitcache init [OPTIONS]

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache list`

$ bitcache list --help
List the IDs of the blobs in the repository, in ascending order.

With `--verbose` (repeatable), appends further tab-separated columns to each line: the blob's byte size, media type, creation timestamp, last-update timestamp, last-access timestamp, and expiration
timestamp.

Usage: bitcache list [OPTIONS]

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -f, --format <FORMAT>
          The format to use for the hash output

          Possible values:
          - hex:    Hexadecimal (aka Base16)
          - base58: Base58

          [default: hex]

  -d, --debug
          Enable debugging output

  -p, --prefix <PREFIX>
          List only IDs whose hexadecimal encoding begins with this prefix

  -a, --after <ID>
          List only IDs ordered strictly after this one

  -n, --limit <COUNT>
          List at most this many IDs

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache pull`

$ bitcache pull --help
Copy blobs missing from the current repository from the given remotes.

Every blob present in a remote repository but absent from the current repository is copied into the current repository.

Usage: bitcache pull [OPTIONS] [URLS]...

Arguments:
  [URLS]...
          The URLs of the remote repositories to pull from

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache push`

$ bitcache push --help
Copy blobs missing from the given remote repositories to them.

Every blob present in the current repository but absent from a remote repository is copied to that remote repository.

Usage: bitcache push [OPTIONS] [URLS]...

Arguments:
  [URLS]...
          The URLs of the remote repositories to push to

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache put`

$ bitcache put --help
Store the given file(s) into the repository as blob(s).

Prints the ID of each stored blob, one per line. Since blobs are content addressed, storing already-present content is harmless: the blob is simply retained with the same ID.

Usage: bitcache put [OPTIONS] [FILES]...

Arguments:
  [FILES]...
          The paths to the file(s) to store

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -f, --format <FORMAT>
          The format to use for the hash output

          Possible values:
          - hex:    Hexadecimal (aka Base16)
          - base58: Base58

          [default: hex]

  -d, --debug
          Enable debugging output

      --ttl <DURATION>
          Expire the stored blob(s) after the given duration.

          Accepts a plain number of seconds (e.g. "90") or a human-friendly duration (e.g. "90s", "2m30s", "1h", "7d").

          Requires a repository backend that supports blob expiration (e.g., filesystem, Turso, or Valkey); exits with an error otherwise.

      --media-type <TYPE>
          Store an explicit media type (MIME type) for the blob(s)

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache rm`

$ bitcache rm --help
Remove blob(s) with the given ID(s) from the repository.

Exits with a nonzero status unless all of the given blobs were found in the repository.

Usage: bitcache rm [OPTIONS] [IDS]...

Arguments:
  [IDS]...
          The IDs of the blob(s) to remove

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

`bitcache sync`

$ bitcache sync --help
Synchronize with the given remote repositories, in both directions.

Equivalent to a `pull` followed by a `push` for each given remote repository: afterwards, the current repository and every given remote repository all contain the union of their blobs.

Usage: bitcache sync [OPTIONS] [URLS]...

Arguments:
  [URLS]...
          The URLs of the remote repositories to synchronize with

Options:
      --color <COLOR>
          Set the color output mode

          [default: auto]
          [possible values: auto, always, never]

  -d, --debug
          Enable debugging output

  -v, --verbose...
          Enable verbose output (may be repeated for more verbosity)

  -h, --help
          Print help (see a summary with '-h')

Storage Adapters

URL Scheme	Adapter Crate	`ls`	`get`	`put`	`rm`	`clear`
`file:`	bitcache-fs	✓	✓	✓	✓	✓
`git:`	bitcache-git	✓	✓	✓	✓	✓
`heap:`	bitcache-heap	✓	✓	✓	✓	✓
`opendal+azblob:`	bitcache-opendal	✓	✓	✓	✓	x
`opendal+fs:`	bitcache-opendal	✓	✓	✓	✓	✓
`opendal+ftp:`	bitcache-opendal	✓	✓	✓	✓	✓
`opendal+memcached:`	bitcache-opendal	x	✓	✓	✓	x
`opendal+memory:`	bitcache-opendal	✓	✓	✓	✓	✓
`opendal+mongodb:`	bitcache-opendal	x	✓	✓	✓	x
`opendal+gcs:`	bitcache-opendal	✓	✓	✓	✓	✓
`opendal+http:`	bitcache-opendal	x	✓	x	x	x
`opendal+redis:`	bitcache-opendal	x	✓	✓	✓	x
`opendal+s3:`	bitcache-opendal	✓	✓	✓	✓	✓
`opendal+sftp:`	bitcache-opendal	✓	✓	✓	✓	?
`opendal+sled:`	bitcache-opendal	✓	✓	✓	✓	✓
`redis:`	bitcache-valkey	✓	✓	✓	✓	✓
`sqlite:`	bitcache-turso	✓	✓	✓	✓	✓
`valkey:`	bitcache-valkey	✓	✓	✓	✓	✓

File System Adapter

BITCACHE_URL=file:.bitcache
BITCACHE_URL=file:/tmp/bitcache

Git Adapter

BITCACHE_URL=git://github.com/asimov-datasets/gutenberg.org.git

Heap (Memory) Adapter

BITCACHE_URL=heap:

OpenDAL Adapter

Azure Blob Storage (`azblob`) Service

BITCACHE_URL=opendal+azblob://my-container

Configuration for Floci AZ

BITCACHE_URL=opendal+azblob://my-container?endpoint=http://localhost:4577/devstoreaccount1&skip_signature=true

File System Service (`fs`)

BITCACHE_URL=opendal+fs:///tmp/bitcache

FTP Service (`ftp`)

BITCACHE_URL=opendal+ftp://localhost

Configuration for pyftpdlib

BITCACHE_URL=opendal+ftp://127.0.0.1:2121?user=anonymous&password=jhacker@example.org

Google Cloud Storage Service (`gcs`)

BITCACHE_URL=opendal+gcs://my-bucket/my-root

Configuration for Floci GCP

BITCACHE_URL=opendal+gcs://my-bucket/my-root?endpoint=http://localhost:4588&skip_signature=true

HTTP Service (`http`)

BITCACHE_URL=opendal+http://localhost:8000

Memcached Service (`memcached`)

BITCACHE_URL=opendal+memcached://localhost:11211

Memory Service (`memory`)

BITCACHE_URL=opendal+memory://

MongoDB Service (`mongodb`)

BITCACHE_URL=opendal+mongodb://localhost:27017/my-database/my-collection

Redis Service (`redis`)

BITCACHE_URL=opendal+redis://localhost:6379

S3 Service (`s3`)

BITCACHE_URL=opendal+s3://my-bucket

Configuration for Floci AWS

BITCACHE_URL=opendal+s3://my-bucket?region=us-east-1&endpoint=http://localhost:4566&skip_signature=true

SFTP Service (`sftp`)

BITCACHE_URL=opendal+sftp://my-host

Sled Service (`sled`)

BITCACHE_URL=opendal+sled:///tmp/bitcache

Miscellaneous Services

OpenDAL supports dozens more additional services; however, if we haven't validated them yet, we won't have a feature flag for them nor URL scheme support in Bitcache directly. (Submit a pull request to add support for your favorite service!)

Turso (aka SQLite) Adapter

BITCACHE_URL=sqlite:/tmp/bitcache.db

Valkey (fka Redis) Adapter

BITCACHE_URL=valkey://localhost:6379

👨‍💻 Development

git clone https://github.com/artob/bitcache.git

Name		Name	Last commit message	Last commit date
.config		.config
.github		.github
dart		dart
data		data
js		js
python		python
ruby		ruby
rust		rust
.gitattributes		.gitattributes
.gitignore		.gitignore
AUTHORS		AUTHORS
CHANGES.md		CHANGES.md
CITATION.cff		CITATION.cff
CREDITS.md		CREDITS.md
Makefile		Makefile
README.md		README.md
Rakefile		Rakefile
UNLICENSE		UNLICENSE
VERSION		VERSION
dist-workspace.toml		dist-workspace.toml

Latest commit

History

Folders and files

Repository files navigation

Bitcache

✨ Features

⬇️ Installation

Installation of the CLI

Installation via Cargo Binstall

Installation via mise

Installation via Cargo

Installation of the Library

Installation from Crates.io

Installation from NPM

Installation from Pub.dev

Installation from PyPI

Installation from RubyGems

👉 Examples

📚 Reference

Command-Line Interface

bitcache clear

bitcache export

bitcache get

bitcache has

bitcache id

bitcache init

bitcache list

bitcache pull

bitcache push

bitcache put

bitcache rm

bitcache sync

Storage Adapters

File System Adapter

Git Adapter

Heap (Memory) Adapter

OpenDAL Adapter

Azure Blob Storage (azblob) Service

Configuration for Floci AZ

File System Service (fs)

FTP Service (ftp)

Configuration for pyftpdlib

Google Cloud Storage Service (gcs)

Configuration for Floci GCP

HTTP Service (http)

Memcached Service (memcached)

Memory Service (memory)

MongoDB Service (mongodb)

Redis Service (redis)

S3 Service (s3)