Home / Storage

DataLad + git-annex workflow (orchestration layer)

handbook.datalad.org/en/latest/basics/101-139-s3.htmlAcademicstorage/datalad-git-annex-workflowJSON compare report an error provider JSON

Free allowance: N/A — DataLad is a wrapper over git-annex; quota is the special remote's.
Verdict: self-hosted, last resortverdict qualifies-with-caveat: usable, with a named limit or an unprobed read path. Self-hosted options are listed last, because this site is for users who have no hardware.
Cost: free tier
Identity: anonymousno card, no phone
Binding limit: noneno named constraint binds
Evidence: reported6 sources; a third party said so
Freshness: stale on 2026-11-05checked 2026-10-21 (inherited); stale on 2026-11-05

Free tier	N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish' a supported, documented workflow instead of hand-written annex commands. unit mixed; metered on unstated
Price	none stated
Identity	anonymous no account for the software: the user installs and runs DataLad and git-annex 10.20230802 or later from PyPI. The walk-through's worked target is an S3 bucket on an AWS account, and the page states that AWS signup wants an email address, a physical address and a credit card before verification, plus a GitHub account for the metadata sibling. DataLad itself asks for no card and no phone number; the card belongs to the S3 account the walk-through demonstrates.
Egress	Inherited; DataLad's S3 walkthrough specifically configures `publicurl` so consumers clone the git metadata from GitHub and stream file content from the bucket with no credentials.
Per-object cap	not stated
Retention	not stated
Interface	s3, git, cli s3
Tooling	`datalad` CLI (Python), `datalad create-sibling-s3`, `datalad push --to <sibling>`, `datalad get`. Requires git-annex >= 10.20230802 for S3 publicurl on post-April-2023 buckets.
Composes with	Combined with `datalad siblings` this is the cleanest way to shard one logical corpus across many free S3 accounts while keeping one coherent git history.
Reliability	Used in production by academic consortia (OpenNeuro etc.) precisely for multi-TB datasets on donated storage — the most battle-tested 'big dataset on free/cheap stores' pattern that exists.
Parsed numbers	has-free-tier 1

Evidence 6

date	source	what it proved
2026-10-21	handbook.datalad.org/en/latest/basics/101-139-s3.html	'This walk-through requires git-annex >= 10.20230802 ... Prior versions do not support public access via the publicurl parameter with S3 buckets created after April 2023'
2026-10-21	handbook.datalad.org/en/latest/basics/101-139-gitlfs.html	DataLad's Git LFS special-remote walkthrough on GitHub (linked as next chapter)
2026-11-01	handbook.datalad.org/en/latest/basics/101-139-s3.html	the accounts the walk-through needs are the target's, not the software's: a GitHub account for the metadata sibling and an AWS account for the bucket
2026-11-01	handbook.datalad.org/en/latest/basics/101-139-s3.html	the worked target's AWS signup takes an email address, a physical address and a credit card, and the software itself needs no account
2026-11-01	handbook.datalad.org/en/latest/basics/101-139-s3.html	the accounts the walk-through's worked example needs are a GitHub account and an AWS account, both stated on the page, so the identity kind is read rather than defaulted
2026-11-01	handbook.datalad.org/en/latest/basics/101-139-s3.html	the tooling requirement the record carries is unchanged on the page

Unknown values 4

per object cap notes: the field may be stated in a source that was not read
per object cap stated: the field may be stated in a source that was not read
retention notes: the field may be stated in a source that was not read
retention stated: the field may be stated in a source that was not read