Every ticket as the root of its provenance graph — goal, patch, approval, runs and (as the ticket-class build lands) mandate, manifestations, guarantees, journey, artifacts and children. Rendered through the <<<TicketGraphList>>> tag.
omni-git: stream blob put/get/verify + OMNI_CHUNK_BYTES RAM cap — safe large-file ingest (no whole-file-in-RAM)
goal
PROBLEM
omni-git is not safe for very large files today. The store STRUCTURE is fine (flat content-addressed
store, one file per blob at objects/<sha[:2]>/<sha[2:]> — NOT real git, so no packfile/delta/history
bloat). The real failure mode is MEMORY: the blob paths read whole files into RAM.
Evidence (omni/store.py):
* put_file(): return self.put_blob(fh.read()) # entire file -> RAM
* put_blob(data): sha = sha256_bytes(data); ... fh.write(data) # hashes+writes whole buffer
* _blob_is_intact(): sha256_bytes(fh.read()) # re-reads whole blob to verify (every dedup hit)
* get_blob()/pull path reads the whole object into RAM too
So a multi-GB image needs multi-GB of RAM to add, and again to verify/pull -> OOM. Worse for images:
omni stores the whole image as one inline blob PLUS (diskforge wiring) each guest file as its own blob.
FIX (in omni-git, NOT bakery; bakery is being retired)
Convert put_blob / put_file / get_blob / _blob_is_intact to STREAM in fixed-size chunks: a running
hashlib.sha256 updated chunk-by-chunk, written to the .tmp file as it goes, and verified by streaming.
Peak memory for blob I/O becomes ~one chunk regardless of file size. Fully backward-compatible: same
sha256 addresses, same objects/ layout, same dedup+self-heal semantics.
CONFIG (the RAM cap)
Expose OMNI_CHUNK_BYTES (default ~8 MiB) as the chunk size = the working-set RAM ceiling for large-file
ingest/pull. Unlike an external OS cap (which would just OOM-kill the current whole-file code), this lets
big files SUCCEED within a set budget. OS/container limits (docker --memory, systemd MemoryMax, ulimit)
remain a valid machine-level backstop.
ACCEPTANCE / VERIFICATION
* omni-git selftest stays 254/254 (no regression; addresses unchanged).
* New pin: a large fixture ingests + pulls + verifies with PEAK RSS bounded ~ OMNI_CHUNK_BYTES
(not file size); negative control = the old whole-file path blows the bound.
* A large-file add succeeds under a tight OS memory limit that the current code OOMs on.
SCOPE / CAVEAT
Streaming bounds the FILE-BYTES memory (the huge-single-file case — the target here). It does NOT bound
memory for an artifact with millions of tiny entries: manifest/index memory scales with ENTRY COUNT, a
separate dimension (far smaller than file bytes). Note it; do not conflate.
FILES: omni/store.py (put_blob, put_file, get_blob, _blob_is_intact; add streaming helper), possibly
omni/cli.py for OMNI_CHUNK_BYTES surfacing/docs; README config table. Homes: GitHub dudezilla/omni-git +
GitLab omni-git-entry-claude (kept consistent). Related: #493 (diskforge wiring), #495 (diskforge).
patch
none
approval
unapproved
runs
no runs recorded
mandate (clauses)
not yet recorded — lands with the ticket-class build
manifestations
not yet recorded — lands with the ticket-class build
guarantees
not yet recorded — lands with the ticket-class build
journey (blunders & successes)
not yet recorded — lands with the ticket-class build
artifacts (forge)
not yet recorded — lands with the ticket-class build