{"eid":"datalad","name":"DataLad + git-annex workflow","domain":"datalad.org","resolved_by":"domain","topics":["storage"],"n_records":1,"best_vorder":1,"review":null,"aliases":["DataLad + git-annex workflow","DataLad + git-annex workflow (orchestration layer)","datalad.org"],"records":[{"rid":"storage/datalad-git-annex-workflow","card":{"rid":"storage/datalad-git-annex-workflow","eid":"datalad","topic":"storage","slug":"datalad-git-annex-workflow","provider":"DataLad + git-annex workflow (orchestration layer)","url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","domain":"datalad.org","category":"academic","tier":null,"verdict":"qualifies-with-caveat","vorder":1,"grp":"usable","self_hosted":1,"confidence":"reported","binding":"none","identity_kind":"anonymous","identity_basis":"stated","needs_card":0,"needs_phone":0,"needs_id":0,"needs_institution":0,"country":null,"crypto":null,"interfaces":"s3,git,cli","egress_metered":0,"per_object_bytes":null,"retention_days":null,"verified_at":"2026-11-01T18:05:17.593Z","verified_basis":"inherited","checked_on":"2026-10-21","stale_on":"2026-11-05","n_evidence":6,"usd_month":null,"usd_per_tb_month":null,"summary":"N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish...","duplicate_of":null,"n_unknown":7,"free_unit":"mixed"},"record":{"topic":"storage","slug":"datalad-git-annex-workflow","provider":"DataLad + git-annex workflow (orchestration layer)","url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","category":"academic","free_tier":{"stated":"N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish' a supported, documented workflow instead of hand-written annex commands.","unit":"mixed","metered_on":"unstated","source_url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","read_on":"2026-10-21","notes":null},"per_object_cap":{"stated":"not stated","source_url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","notes":"not stated"},"retention":{"stated":"not stated","source_url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","notes":"not stated"},"egress":{"terms":"Inherited; DataLad's S3 walkthrough specifically configures `publicurl` so consumers clone the git metadata from GitHub and stream file content from the bucket with no credentials.","metered":false},"interface":["s3","git","cli"],"interface_detail":"s3","tooling":"`datalad` CLI (Python), `datalad create-sibling-s3`, `datalad push --to <sibling>`, `datalad get`. Requires git-annex >= 10.20230802 for S3 publicurl on post-April-2023 buckets.","chaining":"Combined with `datalad siblings` this is the cleanest way to shard one logical corpus across many free S3 accounts while keeping one coherent git history.","limits":[],"limits_note":"Same as annex: no delete on LFS-backed remotes, per-store object caps. DataLad adds Python overhead and its own version-compat matrix (annex version pinned per feature).","identity_cost":{"kind":"anonymous","detail":"no account for the software: the user installs and runs DataLad and git-annex 10.20230802 or later from PyPI. The walk-through's worked target is an S3 bucket on an AWS account, and the page states that AWS signup wants an email address, a physical address and a credit card before verification, plus a GitHub account for the metadata sibling. DataLad itself asks for no card and no phone number; the card belongs to the S3 account the walk-through demonstrates.","country":null},"reliability_note":"Used in production by academic consortia (OpenNeuro etc.) precisely for multi-TB datasets on donated storage — the most battle-tested 'big dataset on free/cheap stores' pattern that exists.","confidence":"reported","confidence_note":null,"evidence":[{"url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","date":"2026-10-21","proved":"'This walk-through requires git-annex >= 10.20230802 ... Prior versions do not support public access via the publicurl parameter with S3 buckets created after April 2023'"},{"url":"https://handbook.datalad.org/en/latest/basics/101-139-gitlfs.html","date":"2026-10-21","proved":"DataLad's Git LFS special-remote walkthrough on GitHub (linked as next chapter)"},{"url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","date":"2026-11-01","proved":"the accounts the walk-through needs are the target's, not the software's: a GitHub account for the metadata sibling and an AWS account for the bucket","capture":"sha256:f5274fa9af9ba13cb49447c64e48e9c0470c7bdc6a4e6cbe221fcee6558e1864","quote":"Signup for a [GitHub account](https://github.com/join)"},{"url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","date":"2026-11-01","proved":"the worked target's AWS signup takes an email address, a physical address and a credit card, and the software itself needs no account","capture":"sha256:f5274fa9af9ba13cb49447c64e48e9c0470c7bdc6a4e6cbe221fcee6558e1864","quote":"The [AWS signup](https://aws.amazon.com) procedure requires you to provide your e-mail address, physical address, and credit card details before verification is possible."},{"url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","date":"2026-11-01","proved":"the accounts the walk-through's worked example needs are a GitHub account and an AWS account, both stated on the page, so the identity kind is read rather than defaulted","capture":"sha256:f5274fa9af9ba13cb49447c64e48e9c0470c7bdc6a4e6cbe221fcee6558e1864","quote":"Signup for a [GitHub account](https://github.com/join)"},{"url":"https://handbook.datalad.org/en/latest/basics/101-139-s3.html","date":"2026-11-01","proved":"the tooling requirement the record carries is unchanged on the page","capture":"sha256:f5274fa9af9ba13cb49447c64e48e9c0470c7bdc6a4e6cbe221fcee6558e1864","quote":"This walk-through requires git-annex >= 10.20230802"}],"verdict":"qualifies-with-caveat","verified_at":"2026-11-01T18:05:17.593Z","notes":"N/A — DataLad is a wrapper over git-annex; quota is the special remote's. Its value here is that it makes 'dataset + N special remotes + reproducible publish' a supported, documented workflow instead of hand-written annex commands.","extras":{},"binding":"none","binding_note":null,"self_hosted":true},"claims":[{"path":"$.free_tier","kind":"has-free-tier","num":1.0,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"ok"},{"path":"$.free_tier.stated","kind":"free-quota","num":null,"unit":null,"per":null,"currency":null,"usd":null,"fx_date":null,"parse":"unparsed"}],"unknown":[["$.binding_note","boilerplate"],["$.confidence_note","withheld"],["$.free_tier.notes","process"],["$.per_object_cap.notes","not-extracted"],["$.per_object_cap.stated","not-extracted"],["$.retention.notes","not-extracted"],["$.retention.stated","not-extracted"]],"history":[],"drift":[]}]}