Vocabulary · Concepts · API · Command line · Undecided · Where each rule lives · Job schema · Logging API · Config API · Router API

The logging API reference is generated from its Thrift definition. Integration of that protocol into the installed service is still pending.

Vocabulary and API

The application holds an id; the work belongs to the machine. A lease that stops being renewed lapses, the next claim raises the epoch and inherits the checkpoint, and the successor resumes from the proven prefix and discards the rest — RESUME1 records that between two processes, and XLANG-DOWNLOAD between two languages. Go, Python and C++ agree on the record and the rules. Each implements them its own way.

The words

Every term names its ancestor or says it has none; where our name or shape differs from the established one, the difference is stated.

Concepts

Each: what it is, when it is met, an example that runs, remarks. Examples are lifted from the tests and the demo, not written for this page.

1. The record is the contract

Go, Python and C++ are not ports of each other and are not generated from a schema. They agree on one thing: the JSON on disk — one file per job, under a directory called the store — and the rules for taking it over. Field order, two-space indent, LF, UTC timestamps with exactly six fractional digits — because the bytes are compared across languages.

{
  "content":  ["abstraction.job/base@1", "abstraction.job/intent@1"],
  "critical": ["abstraction.job/base@1", "abstraction.job/intent@1"],
  "id": "1757000000000-deadbeef",
  "kind": "download",
  "state": "running",
  "spec": { "artifact": {}, "sources": [], "sink": {} },
  "checkpoint": { "verified_prefix": 400 },
  "progress":   { "done": 460, "updated_at": "…" },
  "lease":      { "owner": "go-worker", "epoch": 2, "expires_at": "…" },
  "intent":     { "want": "pause", "by": "comfyui@desktop:9184", "at": "…" },
  "created_at": "…", "updated_at": "…"
}

Strings are escaped exactly one way [JOB-E6]: " and \, the five short control escapes, every other character below U+0020 as lower-case \u00xx, and U+2028 and U+2029. Everything else is raw UTF-8 — &, <, >, /, every non-ASCII character, and characters above U+FFFF as four bytes rather than a surrogate pair; an ill-formed byte becomes �, one per byte. That is RFC 8785's string serialisation with two divergences declared on the contract page. The rule exists because Go's encoder escapes for HTML and Python's for transports that cannot carry high bytes, and a record with an ampersand in its URL came out three ways from three implementations that passed every check; the fixture locator-ampersand.json now carries it.

Remarks. A field not in the table on abstraction-job/README.md is refused, not ignored — the opposite of protobuf and of the robustness principle (be liberal in what you accept), and declared as a divergence. Anything else goes in extensions under a namespaced key and is written back untouched. Inside spec the rule flips: unknown keys are ignored, because a spec is payload the layer above extends. spec is the worst name on the page and cannot change: a renamed field stops every old reader on every record already on disk.

2. Ownership is a lease with an epoch

What. To work on a job, a process claims it for a ttl. The claim returns the record carrying the new holder's epoch, one higher than before. Every write the holder makes presents that epoch. When it stops renewing, the lease lapses; nothing tells it, and its next write is refused. Somebody else's claim then gets epoch+1 and inherits the checkpoint.

When. Every time anything moves bytes, and every time a process starts: look for orphans and adopt them. That pass is the sweep; a supervisor is a process that does nothing but sweep on a timer, and jobd is ours. Reclaiming is the mechanism; a polite release is only an optimisation, because a killed process never releases anything.

Example — claim, checkpoint, lapse, adopt

store, _ := job.NewFileStore("/var/lib/jobs")

rec := job.Record{Kind: "download"}
rec.SetSpec(mySpec)                  // opaque here; the download layer reads it
id, _ := store.Submit(rec)

r, _ := store.Claim(id, "my-worker", 30*time.Second)
store.Update(id, r.Lease.Epoch, func(r *job.Record) error {
    r.Progress.Done = 1 << 20
    return r.SetCheckpoint(myCheckpoint)   // what a successor would need
})

// …thirty seconds pass with no renew…
orphans, _ := store.Orphans()                                   // [the record]
adopted, _ := store.Claim(orphans[0].ID, "successor", time.Minute)   // epoch 2; inherits Checkpoint

_, err := store.Update(id, r.Lease.Epoch, func(*job.Record) error { return nil })
errors.Is(err, job.ErrStaleEpoch)                               // true

store = FileStore("/var/lib/jobs")
jid = store.submit(Record(id="", kind="download", spec={"what": "a thing"}))
held = store.claim(jid, "python-worker", ttl_seconds=30)

def progress(r):
    r.progress.done = 400
    r.checkpoint = {"proven": 400}
store.update(jid, held.lease.epoch, progress)

# …31 seconds pass with no renew…
orphans = store.orphans()               # [the record]
adopted = store.claim(jid, "go-worker", 60)
adopted.lease.epoch                     # 2
adopted.checkpoint["proven"]            # 400 — the successor inherits what was proven

store.update(jid, held.lease.epoch, lambda r: None)   # raises StaleEpoch

using namespace abstraction::job;
FileStore store(root);

Record r;
r.kind = "download";
r.spec = Json::parse(R"({"sink":{"final":"model.gguf"}})");
const std::string id = store.submit(std::move(r));

const Record claimed = store.claim(id, "cpp-worker", std::chrono::seconds(30));   // epoch 1
store.update(id, claimed.lease.epoch, [](Record& rec) {
    rec.progress.done = 460;
    rec.checkpoint = Json::parse(R"({"verified_prefix":400})");
});

// …the lease lapses…
store.orphans().size();                                          // 1
const Record adopted = store.claim(id, "successor", std::chrono::seconds(30));   // epoch 2
(*adopted.checkpoint)["verified_prefix"];                        // 400
store.update(id, 1, [](Record&) {});                              // throws StaleEpoch

Remarks. The lapse is silent: expires_at passing signals nothing. A holder learns at its next write, and the verdict — the single word an operation answers with, and the only part of a refusal a caller may branch on — tells it whether it has a successor: lease-expired while nobody has claimed since, stale-epoch once somebody has. A live lease held by another owner refuses a claim lease-held. The same owner may claim again while live and the epoch still rises. The lock is one machine's: two hosts writing one record over SMB lost 147–149 of 2150 updates; a record shared across hosts is handed off by delegation, not co-written.

3. A successor inherits what its predecessor proved — not what it did

"progress":   { "done": 460 },
"checkpoint": { "verified_prefix": 400, "verified": [[0, 400]] }

The predecessor wrote 460 units and proved 400. The successor resumes from 400 and discards the rest. For a download the resume point is the smaller of the checkpoint's prefix and what the file actually holds; the file is the floor and a partial is never deleted to make a record true. Ranges are half-open, sorted, merged where they touch. HTTP's byte ranges are inclusive at both ends — [0, 8) goes out as bytes=0-7; same word, different arithmetic, and every implementation converts at the wire.

4. Six states, and who writes each

Remarks. The store cannot decide transferred, complete or failed, because deciding them means knowing whether the work is done, and it does not know what the work is. Terminal is judged on the record as read, so the write that makes a record terminal lands and the next one of that epoch is refused: put everything into the final update — the error, the last checkpoint, the final done. A defer release() after a final write is a call that cannot succeed. The state names are BITS' with one exception, and the exception (complete) is a trap; see the words.

5. Intent — the one write with no epoch

What. The person who wants a job stopped is not the process doing it, and requiring a lease would mean stealing the job in order to stop it — the one thing the lease exists to prevent. So intent is written by anyone, holding nothing, and the holder must read it before starting and at least as often as it checkpoints, and move toward it.

When. A pause button, a cancel button, a supervisor on another machine that a UI here needs to stop.

Example — a person cancels while an owner is dead; the next owner honours it before moving a byte

submit A size=64 digest=good src=file
claim A alpha 400
intent A cancel
sleep 900
orphans            -> ok A
run A beta         -> state=cancelled done=0     [DL-R26] [JOB-I9]

Scenario cancel-adoption.txt; all three implementations produce this transcript byte for byte.

j := job.Open(store, id, "comfyui@desktop")   // a handle: look at it, give up on it
j.Cancel()                                      // records the request; succeeds while somebody holds the lease
if p, ok := j.(job.Pausable); ok { p.Pause() } // pause is an extended capability an application asks for

store.set_intent(jid, CANCEL, by="comfyui@desktop")   # no epoch, idempotent
store.set_intent(jid, PAUSE, by="ui")
store.set_intent(jid, RUN, by="ui")                    # resume records a value; it does not delete the field

store.set_intent(id, want::kCancel, "a-person");   // no epoch
store.set_intent(id, "nonsense", "a-person");       // throws: an unknown want is refused, never treated as run

Remarks. One word, four operations, and a caller has to know which it is holding:

Cancelling never stops an execution. It records a request; the acknowledgement is the holder's own write of cancelled. A caller that must know the work stopped waits for the state. cancel is honoured by everything; an owner that cannot honour pause fails the job with a reason rather than carrying on, because a pause that quietly does nothing is worse than no pause button. A paused job is not an orphan — unless its state is still running, which is an owner that died between the ask and the honouring; the sweep offers that one so the next owner can honour it, and the scenario above is exactly that case.

6. Recall — the issuer asks for the lease back

What. A Chubby lease expires; nothing recalls it. Systems that genuinely arbitrate a resource have the other half: WDDM tells a process its budget changed and evicts it if it does not trim. A recall presents the epoch the issuer read — the fencing token pointed the other way — and moves expires_at to until. The holder may still renew and checkpoint until then, never past it; it cannot shed the recall by re-claiming; the lapse is the eviction and nothing else is. A recall is about the resource; it leaves intent, the user's wish about the job, untouched.

claim  A alpha 5000
recall A beta  800 yield        -> stale-epoch     [JOB-R1]   beta never held it
recall A alpha 800              -> invalid         [JOB-R2]   no reason
recall A alpha 800 yield        -> ok  recall=yield want=run  [JOB-R3] [JOB-R8]
claim  A alpha 5000             -> lease-held      [JOB-R5]   cannot re-claim under recall

Scenario recall.txt. Command line: jobctl recall <id> --epoch N --reason WHY [--grace SECONDS] [--by who] — --epoch is the one the caller last read, not one it holds.

7. Acceptance — the instant ownership of a request transfers

Before it, an uncertain outcome means nothing was taken and retrying is free. After it, an uncertain outcome is unknown, and a retry starts a second job. Where a download runs is chosen from an ordered list — a NAS, a platform service, this process — and that order is the machine's tiers. Acceptance is a barrier in it: before acceptance a failure moves to the next tier; after it nothing may, because a second tier would start a second job.

8. The three answers

What. A request that did not succeed ended one of three ways, and a provider that collapses them has told the caller nothing:

The verdict classes — the named refusals in the API tables below — map onto them, and what each obliges is the whole reason they are separate:

Example — one application, three providers, nine situations

$ ABSTRACTION_STORE=/tmp/s fetch -provider curl get http://host/f.bin f.bin
provider=curl where=curl answer=delivered id=curl-0b55… detail="6291456 bytes sha256:e56b… at /tmp/f.bin"

Seven of the nine rows; the whole table is on Evidence, 2026-09-08. fetch is written once against download.Client and never names a provider. Exit codes: 0 delivered or working, 3 unavailable, 4 forbidden, 5 unknown.

9. Retries — not now against no

Retry classification, as old as SMTP's 4yz/5yz. A dropped connection, a full disk, a NAS that rebooted: not now. A request that can never work as written: no. Backwards in either direction costs: a 404 classed not now is re-fetched on every sweep forever; a dropped connection classed no throws away the case this project exists for.

• The class is attached where the error is defined, never kept in a list. Go and Python each kept a list; they disagreed by a row for as long as both existed.
• Over HTTP, the source refused is exactly 400 401 402 403 404 405 406 410 414 451. Listed, not ranged: an unrecognised 4xx is not now. That breaks RFC 9110 § 15 on purpose — 499 is nginx closing on its own client and 440 an IIS session timeout, and neither is a statement about the file.
• No is: the source refused, nothing can serve the job's sources, the sink escapes the store root or names the store's own layout, the record is invalid. Everything else is not now — including a sink written in the other platform's path convention, which is unusable here and fine where it was written.
• The attempt count is the lease epoch. A record carrying an error waits updated_at + min(15 s · 2^(epoch−1), 15 min); a record without one is adopted at once, because a killed owner wrote nothing.
• Taking a lease to publish a refusal is a write: it bumps the epoch and charges the submitter's backoff. So a sweep declines to claim a job whose sink it cannot write, and refuses out loud only when asked by name.

download.Permanent(err)            // true: trying again unchanged is pointless
download.RetryAfter(rec)           // time.Time; zero when the record carries no error

permanent(exc)                     # isinstance(exc, (Permanent, Invalid))
retry_after(rec)                   # POSIX timestamp; 0.0 when the record carries no error

download::Error e(...); e.permanent();   // the class rides on the exception

10. Declarations, and what an older reader does

There is no version number and no negotiation. A record says which schemas it carries (content) and which a reader must understand (critical); both are derived from the record on every write, never remembered. An unknown name in critical is refused with the verdict spelled unknown-model in a transcript and unknown_schema in the definition: the record is fine, this reader is too old, leave it alone. abstraction.job/terminal@1 and recall@1 name rules, not fields — a shape can be ignored, a rule about writes cannot, because the reader that does not know it is precisely the reader that breaks it. Measured: a reader published one day earlier walked a complete record back to pending under an unchanged base@1.

11. Delegation — handing the work to something that outlives the caller

Two shapes, and each half is somebody's published interface; the pair is not, and is the one concept on these pages with no ancestor. A Fetcher streams bytes through us and dies with the process (http, a local file, an SMB path). A Delegator does the work under its own account and hands back a handle (BITS, a NAS daemon). The Runner is what drives either one: it claims the record, resumes from what was proven, hashes, verifies and delivers. A caller asks for survives_process_exit; which shape provides it is the machine's business. When delegated, the handle is {system, external_id}, the lease is released — holding it would stop anyone else verifying and delivering the finished file — and progress is a cache. A delegate's "success" is verified by hashing the delivered file: BITS guarantees size and timestamp, not content. A handle that stops resolving returns the job to pending with its sources and checkpoint intact.

runner.Delegators = download.NewDelegators(bits.New())
err := runner.Delegate(ctx, id)      // returns as soon as the service has it; this process may now exit
n, _ := runner.ReconcileAll(ctx)     // any later process catches up with work handed to a service
n, _ := runner.Adopt(ctx)            // claims every orphan and finishes it here

runner.adopt()                       # claims every orphan and finishes it here

Runner runner(store, "cpp-worker");
runner.run(id);                      // claim, resume from what was proven, hash, verify, deliver

API

Signature, then what it refuses. Each language's shape is that language's; the refusal classes are the contract, their wording is not.

Store

Refusals by language: Go returns sentinel errors — job.ErrNotFound, ErrLeaseHeld, ErrStaleEpoch, ErrLeaseExpiry, ErrTerminal, ErrInvalid, ErrUnknownSchema — matched with errors.Is. Python raises NotFound, LeaseHeld, StaleEpoch, LeaseExpired, Terminal, Invalid, UnknownSchema, all under JobError. C++ throws the same names under JobError, with TerminalState for terminal. All three spell one Verdict enumeration declared in abstraction-job/job.thrift, from which the Schema page and the Go record type are generated.

Optional: a local area

A store backed by a filesystem can say so. A store backed by a service answers no, and a caller must have an answer for that.

if sc, ok := store.(job.Scratch); ok { sc.Root(); sc.WorkPath(id) }

store.root(); store.work_path(job_id)        # FileStore implements the Scratch protocol

if (auto* l = dynamic_cast<LocalStore*>(&store)) { l->root(); l->work_path(id); }

work/<id> is a name, not a shape: derived from the id so a successor finds what a predecessor left, and used by nothing else. Reserved(owner, path) (reserved in Python, abstraction::job::reserved in C++) says whether a path names the store's own layout — a sink of jobs/<id>.json would overwrite a record.

Watching

A live view of one kind's jobs: a snapshot first, then every visible change, and quiet once nothing has changed for a budget. A change is identity, state, progress.done, progress.total, the lease owner or the error. A lease renewal is not a change — a view that redraws on every heartbeat flickers.

sub := job.Watch(store, "download")          // Records(), Next(ctx), Changes() <-chan, Close()

sub = watch(store, "download", budget=0.0)  # records(), next(timeout) -> Notice, close()

#include <abstraction/job/watch.h>          // the same three operations

KeepAwake(store, claimed) holds the machine out of idle sleep for exactly as long as this process holds that lease and not a moment longer. Windows PowerRequestSystemRequired, macOS caffeinate -i, Linux systemd-inhibit. A queued or delegated job holds nothing. macOS and Linux are written and not run.

Download

The layer that can fetch. It adds to a job the three things a transfer needs and nothing about what the bytes are for:

a, _ := abstraction.Discover()             // reads the machine; no path, no flag in the caller
h, _ := a.Download().Get(url, dest)        // returns at once; the transfer outlives the caller
rec, err := h.Wait(ctx)                    // blocks until the bytes are here, then takes delivery

// a caller that knows more
h, _ := c.Submit(download.Spec{
    Artifact: download.Artifact{Digest: "sha256:74a4da…", Size: 491400032},
    Sources: []download.Source{
        {Scheme: "https", Locator: "https://huggingface.co/…/model.gguf"},
        {Scheme: "smb",   Locator: `\\nas\models\model.gguf`, Priority: 1},
    },
    Sink: download.Sink{Final: "models/model.gguf"},
}, "survives_process_exit")

h, out, _ := c.ResumeOrGet(url, dest)      // out.Decision: submitted · resumed · delivered · busy · paused
sub := c.Jobs()                            // every download on this machine, live; bind a UI to it
c.Where()                                  // "nas", "bits", "here" — display text, never a branch

svc = abstraction_download.discover()      # reads where this machine keeps its jobs
svc.deliver(svc.get(url, "models/x.gguf")) # ids, not handles; deliver waits and takes delivery

job_id = svc.submit(Spec(
    artifact=Artifact(digest="sha256:74a4da…", size=491400032),
    sources=[Source(scheme="https", locator="https://huggingface.co/…/model.gguf")],
    sink=Sink(final="models/model.gguf"),
), requires=["survives_process_exit"])
svc.jobs()                                 # a snapshot, not a live collection
svc.where()                                # "here", or the supervisor's tier

using namespace abstraction::download;
Spec spec;
spec.artifact = {"sha256:74a4da…", 491400032};
spec.sources.push_back({"https", "https://huggingface.co/…/model.gguf"});
spec.sink.final_path = "models/model.gguf";
const std::string id = submit(store, spec);   // job::Invalid when the spec is refused
Runner runner(store, "my-app");
runner.run(id);                               // this process does the transfer

switch (abstraction::discovery::ask(store_root, "abstraction.downloads")) {
    case Answer::Present: break;              // a supervisor would take it
    case Answer::Incompatible: break;         // a NEWER supervisor: do not
    case Answer::Absent: break;               // the ordinary case, not an error
}

Capabilities

requires is a placement constraint (Kubernetes nodeSelector, Nomad constraint). A job that asks for a property is not quietly served by something without it.

Command line

Undecided

Behaviour none of the sources settles, with what the three implementations do today. An adopter meets these as surprises; they are listed so the surprise is here instead.

1. Backward state moves. All three let a holder write transferred → pending. No scenario asserts it lands; no refusal class exists for it.
2. Same-owner re-claim while live. All three bump the epoch. It let one process race itself and report a delivered download as a failed install.
3. Verdict class for an empty owner. Go: a bare error. Python, C++: JobError. Neither is in the verdict list.
4. Whose clock. Each store reads its own. Two hosts on one share may assume nothing.
5. Delegated orphans. The store offers them; the download kind declines them.
6. Intent on a delegated record. Who honours cancel and pause once the holder released to a delegate.
7. Request identity on the write path. An update's lost answer cannot be asked about.
8. Possibly delegated. external_id == id is Go's encoding of "unsettled"; a Python or C++ reader sees a settled delegation it can never settle.
9. A repeated recall may leave until past expires_at.
10. unknown in the job layer. No sentinel, no verdict class, no transcript shape.
11. What refused names now that unknown-model exists.
12. Partial enumeration across languages — Go names the unreadable ids; Python and C++ skip.
13. Release after a final write has no distinguished answer.
14. Update, intent and release on transferred. All three answer ok; no page decides.
15. Where backoff lives. In the download kind, on the epoch. Whether the job layer owns the formula or only the counter.
16. The socket transport's decoder, and whether a connection declares what it enforces.
17. Renew on a never-held record (all three: lease-expired) and on a live lease (all three: ok).

The full list with what would settle each: abstraction-job/SPEC.md § 11.

Where each rule lives

Four places, each rule in exactly one:

Every rule on a contract page ends with a tag — [JOB-L1], [DL-E2] — and a scenario cites the tag it tests. A rule nothing cites is printed and counted UNEXERCISED; the tags minted in abstraction-job/SPEC.md § 13 are counted so until the harness reads that page. The count is an instrument's number and moves on every run, so it is not quoted here.