More than text

An agent that only moves text cannot look at the mockup you are asking it to edit, cannot hand back the video it rendered, and cannot pass an image from one stage to the next. Leviath moves typed parts instead. A part is one piece of content with a mime type: a paragraph, a PNG, a WAV clip, an MP4, a PDF, an OBJ model. Text is a part like any other. The only thing that sets it apart is that its bytes travel inside the entry and reach a model directly.

lev run sprite-editor --task "edit sprite image @hero_idle.png so the arm is longer"

That task lands as one entry with two parts: the sentence, and hero_idle.png. The model sees the image if it can take one, and a one-line stand-in for it if it cannot. The stage's tools see the typed file either way.

Where a part can go

Bytes are only touched at the edges. When a part arrives it is written once under <run>/blobs/<sha256>, and when a request is built the bytes are read back for the model. Everything between, the journal, the snapshots, the events, carries a reference: hash, type, size, dimensions, a token estimate. Deleting a run deletes its blobs.

The registry

Leviath does not know what an image is. A mime registry says what each type is, and you can extend it. Every type resolves to a row with a family, a text flag, a token rule, extensions, and optionally a magic prefix and a stand-in template. A row for image/* supplies defaults for every image subtype; */* is the last resort.

# ~/.leviath/mime_types.toml (lev mime init writes an example)
["model/obj"]
extensions = ["obj"]
text = true                      # UTF-8 under the hood: may reach a text model as text

["application/x-acme-scene"]
family = "model"
extensions = ["scene"]
magic = "41434D45"
tokens = { per_byte = 0.1 }
stand_in = "[{type} {size}] {name}"

A token rule is written out in full:

tokens = { per_byte = 0.25 }
tokens = { per_pixel = 750, max = 1600 }
tokens = { per_second = 32 }
tokens = { per_page = 2000 }
tokens = { fixed = 1000 }

In a magic prefix, ?? lets a tag past a length field be named, so 52494646????????57454250 is RIFF, four bytes of size, WEBP.

Rows layer, and a later row wins. The compiled defaults come first. Then the rows every configured Rhai provider ships for the types its models are built for. Then a [mime_types] table in your config, then mime_types.toml beside it, then the [mime_types] a blueprint carries. So your own rows and a blueprint's both win over a provider's. A row names only what it changes: adding an extension to image/png keeps its family and token rule. lev mime add <type> writes a row from the command line, and lev mime list prints the effective table with the source of every row. lev mime check <file> says what type a file resolves to, what its stand-in looks like, and how it reaches a model. lev models list --accepts <type> names the models that take it natively.

Every run reads its own copy of the table: the operator's rows with the blueprint's on top, built when the run spawns. Edit mime_types.toml or the config while runs are live and they pick the change up too, within the daemon's housekeeping interval of thirty seconds; a new run sees it at once.

A file's type is decided in a fixed order: the type the sender declared, then the registry's magic prefixes, then the extension, then valid UTF-8 counts as text/plain, and anything else is application/octet-stream.

A type is a claim, and the claim is checked in two places. Its spelling is checked wherever a type is written: lowercase type/subtype, no parameters, type/* only where a capability or an accepts list is declared. Whether the bytes are that type is checked only when a row names a check. The script runs once, where bytes are stored. An upload, a tool result, a read_file, a model's reply and an artifact are all refused with the reason when they are not what they claim. Without one, a declared type is taken at its word, as every provider takes it.

What a model sees

A model declares what it takes, as mime types. Anthropic and OpenAI models list image/* and application/pdf; Gemini and Meta's Muse Spark add audio/* and video/*; a local model you describe in [model_capabilities] lists whatever it can do. Where a provider publishes this per model, Leviath reads it: OpenRouter's catalogue carries each model's input and output modalities and Ollama's /api/show reports vision, and both win over the built-in tables. The vendors whose APIs stay quiet (Anthropic, OpenAI, Google direct) are carried in a compiled table. cargo xtask modalities refreshes it from OpenRouter's catalogue, the way cargo xtask prices refreshes list prices. So claude-3-haiku, which takes images but not PDFs, is described precisely rather than by a whole-vendor guess. lev models show names a model's input and output types. When a request is built, each stored part goes one of three ways:

A native part that the provider stores for you is named by id instead of resent. See Files and size limits.

The text bypass is what lets a model/obj file reach a model that has never heard of 3D models, while a tool declaring @accepts model/obj still receives it typed. The stand-in is what keeps every text-only model working: it always names the part, so the model can pass that name to a tool that can read it.

Stored parts are charged to their region like text is. The registry's token rule is the estimate; an image is billed by its pixels when the header could be read, and a provider's own count corrects the estimate after the first call.

What a model hands back

A model that draws or speaks answers with bytes as well as words. The image, video, speech, transcription and music models on OpenAI, Google, AWS Bedrock, xAI and Meta hand back images, MP4 videos, audio and transcripts this way. See Image, video and audio models. Meshy hands back 3D models the same way. A stage whose output routing or format names an image, video or audio type, and whose model answers with words only, is told so and asked again, up to three times. A reply with no bytes is usually a generation the vendor refused. An OpenAI-shaped provider reads them off the message as data URIs, from OpenRouter's images list and from image_url items in a content array, streamed or not; a Rhai provider returns them under parts. The runtime stores each one in the run's blob store and writes it beside the reply's text on the assistant turn, named as the provider named it (image-1.png when it did not). The next request, lev blobs, the dashboard's Context view and the API all see it, and a later context_export or submit_output can hand it on as an artifact. A part the run cannot keep (over [mime] max_part_bytes, or a world with no store) becomes a line in the reply saying what was dropped. The same line goes in the stage's log, and a warning in the daemon's. A stage left with nothing to hand back then reads as a ceiling, not a model that made nothing. A plain URL in a reply is never fetched.

Most models that draw cannot call tools (lev models show says supports_tools = false), and their providers refuse a request that carries a function call anywhere in it. A stage on such a model is offered no tool, whatever it grants. It sees what earlier stages' tools did as prose, so it can take a review stage's notes and draw again without the run dying on its second visit.

Regions hold typed inputs

A region can say what it accepts and how many stored parts it holds:

[context.regions]
brief         = { kind = "pinned", seed = "task_input", accepts = ["text/*"] }
voice_samples = { kind = "pinned", seed = "input", accepts = ["audio/*"] }
storyboard    = { kind = "pinned", seed = "input", accepts = ["image/*"] }

A stage's inputs are the regions it can see, so a stage that reads those three has three typed inputs and nothing new to declare. A write that does not match accepts is refused and says what the region does take. A region is bounded by its token budget: past it the oldest entry is evicted, or the write is refused under admission = "reject".

When a stage lists several models, the one that takes what the stage's regions accept goes first, so a stage reading a storyboard lands on the model that can see it. lev validate says what each stage takes. It warns (mime-unseen) when its models can see none of the types its regions take. When they see some and not the rest, it says so as information. That is how a pipeline that draws in one stage and builds in the next looks. Two keys under [stages.<name>.input] adjust this. accepts states the types outright. as_text names types whose parts reach the model as text whatever it takes, which is how a model/obj scene gets to a text model even when the registry calls it binary.

What a tool may be handed

A tool says what it takes (@accepts on a script, the built-in tables), and a stage can narrow that further for its own turn:

[stages.review.tool_accepts]
spawn_agent = ["image/*", "audio/*"]   # a sub-agent started here gets pictures and sound, never the video
context_export = ["text/*"]            # only text files may be written back into the workdir here

A stored part outside a tool's list is out of that tool's reach at the stage. spawn_agent's parts refuses it by name, and context_export refuses it the same way. A script's list_parts does not show it, and its read_part says which types the tool may be handed. Inline text is never hidden by a limit, and a tool absent from the table keeps whatever it takes itself. lev validate prints each stage's limits and warns (tool-accepts-ungranted) about a limit on a tool the stage does not grant; the dashboard's agent editor sets them on the Models and tools tab.

A Rhai tool makes a part with write_part, from bytes it got one of three ways. read_part takes a part the run already holds, and http_get_bytes takes a download. read_file_bytes takes a file in the working directory, such as an image its own shell command rendered or a design someone left there. A limit hides parts, not files: read_file_bytes is gated by the read_file permission instead.

Stages declare typed outputs

[stages.assemble]
available_tools = ["concat_video", "submit_output"]
[[stages.assemble.output.artifacts]]
name = "final"
type = "video/mp4"
required = true

submit_output names the files, Leviath checks that each exists inside the working directory and is the type the stage declared, and a missing required artifact is refused back to the model like a schema failure. Accepted artifacts land in the final_output region as parts, so the next stage sees them, and lev result and the API serve them. Final outputs has the details.

Attaching files

lev run storyteller --task "a 30 second trailer" \
  --attach voice.wav:voice_samples --attach frame1.png:storyboard
lev run reviewer --task "does @mockup.png match @spec.md?"
lev respond <id> --attach marked_up.png "the arm is still wrong, see the circle"

--attach path[:region][:type][:text|native|stand_in] puts a file in a region. An @path inside any text does the same for the region the text lands in, and keeps the text exactly as written so the model and the stand-in agree on the name. Write \@ for a literal @. A token that names no file is left alone, so an email address is never mistaken for one. On the command line, paths resolve from where you ran the command; over the API they resolve inside the run's working directory. A --<region> @file whose bytes are not text is attached to that region as a part rather than read as its seed. The daemon types every part with its own registry, so a file the CLI could not name still gets the type your [mime_types] rows give it. :type overrides that, and :text, :native or :stand_in override how the part reaches the model.

Over HTTP, POST /api/agents and POST /api/agents/{id}/message take multipart/form-data with any number of file parts, or a JSON parts list naming files already inside the workdir. The API page has the shapes.

Files and size limits

A provider with a Files API takes a part once and lets later requests name it by id. A 30 MB PDF then crosses the network once for the whole run, rather than on every turn and every retry. When a request carries a stored part its model takes natively, and the provider can store that type, Leviath uploads it the first time and names it by id after. The same part in a later stage, a retry, or the next turn reuses the upload.

Nothing is uploaded when:

• zero data retention is on. An upload is data the provider keeps, so the switch turns uploads off whatever else says. Anthropic's Files API is not eligible for zero data retention.
• [providers] file_uploads = false. The same switch is on the Defaults screen of lev setup ("Upload media to provider file storage") and is --file-uploads false headlessly.
• the run's store keeps nothing on disk, as an embedder's in-memory store does, since there would be no record to delete the uploads from.

A part sent inline is held to the provider's inline limits. One over a limit reaches the model as its stand-in with the reason, and when the provider could have stored it, why it was not uploaded:

[application/pdf, 60.0 MiB] report.pdf [not sent: 60.0 MiB is over the 50.0 MiB this provider takes inline; zero data retention is on, so nothing is uploaded]

Each run records its uploads in provider-files.json in its directory. The files are deleted from the provider when the run finishes, and when the run is deleted (from lev serve, the dashboard, or lev doctor). They are also deleted when the daemon starts for a run that finished while it was not running. Every upload also asks the provider to delete it after [mime] provider_file_ttl_secs (a day by default, clamped to what each provider takes; Google always keeps a file 48 hours). A request the provider refuses because a file it names is gone uploads the part again and is retried once.

Limits

[mime]
# max_part_bytes = 33554432             # one part, at every ingress
inline_text_bytes = 1048576             # text kept inside the entry before it is stored by hash
max_media_bytes_per_request = 20971520  # stored media one request carries, where a provider names no limit
provider_file_ttl_secs = 86400          # how long an upload lives in a provider's file storage

max_part_bytes, left unset, is the largest part a configured provider takes, by upload or inline: 1 GiB with Meta, 500 MiB with Anthropic, 32 MiB when no configured provider names a limit. A value you set always wins. lev mime list prints the ceiling in force, where it came from, whether uploads are on, and each configured provider's limits.

max_media_bytes_per_request is a backstop for the vendor request-size limits a token budget cannot see, for a provider that documents none of its own. An image costs the same few thousand tokens whatever its byte size, so a request can sit inside its context window and still be megabytes of media. Past it, the oldest stored parts are sent as their stand-ins.

lev doctor reports a [mime_types] row that will not load, or a check it cannot compile; the daemon keeps the built-in table until it is fixed.