Buffering, offline replay, and spool mode

Pick the upload mode that matches your training loop's durability needs, and know how to recover queued data when a process exits early.

Use the default async mode

By default, im.init(...) starts run creation in the background, returns a run handle immediately, and uses async metric/log/status uploads after the run exists. The first write waits for run creation if it has not finished yet.

import instantml as im

run = im.init(project="demo")
run.log({"train/loss": 0.2}, step=1)
run.finish(timeout=30)

Use run.wait_for_init() when a script should fail before expensive training starts:

run = im.init(project="demo")
run.wait_for_init()

Set async_init=False for fully synchronous run creation:

run = im.init(project="demo", async_init=False)

After initialization, scalar metrics, rank metrics, console logs, and final run status are queued locally and drained by the async uploader. Delivery failures surface through Run.upload_status(), dashboard upload-health metrics, and wait helpers rather than raising from hot-path log(...) calls.

run.log({"train/loss": 0.2}, step=1)
print(run.upload_status())
run.wait_for_processing(timeout=30)

Use upload_mode="sync" when a short script should raise foreground InstantMLError exceptions for normal post-init writes:

run = im.init(project="demo", upload_mode="sync")
run.log({"train/loss": 0.2}, step=1)
run.finish()

Recover the async queue after finish() times out

run.finish() drains the async queue within a time budget: an explicit finish(timeout=...) if you pass one, otherwise INSTANTML_FINISH_DRAIN_SECONDS if set, otherwise the client timeout (10 seconds by default). If the drain does not complete in time, the SDK warns and leaves the remaining rows on disk under the queue directory (.instantml/async by default, configurable with queue_dir= on init).

Upload the leftover rows with the bundled uploader:

instantml-uploader --queue-dir .instantml/async

Or give slow-network jobs a larger drain budget up front:

export INSTANTML_FINISH_DRAIN_SECONDS=120
python train.py

Batch writes in memory

Use buffering to batch post-init SDK calls in memory:

run = im.init(project="long-run", buffer_size=25)

for step in range(1000):
    run.log({"train/loss": 1.0 / (step + 1)}, step=step)

run.flush()
run.finish()

Call flush() before process exit or before you need writes visible in the UI.

Replay failed requests with offline_dir

Use offline_dir to store failed foreground requests and replay them later. It applies to requests that run in the foreground — sync-mode writes, buffered flushes, and calls such as configs, tags, artifacts, and objects. In the default async mode, hot-path writes go to the async queue instead (see above):

run = im.init(
    project="resilient-run",
    upload_mode="sync",
    offline_dir=".instantml/offline",
)

run.log({"train/loss": 0.2}, step=1)

# Later, after the server is reachable:
replayed = run.replay_offline()

offline_dir only replays requests made by an existing Run. For a run that never reaches the server at all — an egress-blocked HPC node, for example — use mode="offline" (see below), which reserves run identity locally and writes a self-describing run directory. offline_dir still requires online init().

Isolate uploads with spool mode

Use upload_mode="spool" when the training process should avoid post-init HTTP calls:

run = im.init(
    project="long-run",
    name="seed-42",
    upload_mode="spool",
    spool_dir=".instantml/spool",
)

Drain the spool from a separate process:

python -m instantml.uploader \
  --spool-dir .instantml/spool \
  --base-url https://api.instantml.ai

Metric and console-log events send their event ID as Idempotency-Key, so a compatible server can safely accept retries.

Run without a server with mode="offline"

mode="offline" never touches the network — not even at init() — and does not require credentials. It reserves the run identity locally and writes a self-describing run directory that instantml sync uploads later (sync ships in a follow-up release).

run = im.init(project="hpc-job", name="seed-42", mode="offline")

for step in range(1000):
    run.log_metrics({"train/loss": loss}, step=step)
run.log_config({"lr": 3e-4})
run.upload_file("checkpoints/policy.pt", artifact_type="checkpoint", step=1000)
run.finish()

The directory layout under <data_root>/offline/<run_id>/ is:

• run.json — an atomically-rewritten manifest (schema version 1) with the run id, deterministic producer session id, the verbatim create request, the finish signature, and per-event-class counters.
• segments/ — spool-format JSONL event segments with persisted, deterministic idempotency keys so a resumed upload never re-sends or duplicates delivered events.
• files/ — staged artifact bytes referenced by source_path.

Supported offline: config, tags, notes, scalar and rank metrics, console logs, text/histogram attributes, rich objects (tables, histograms, classification evals), and small artifact byte uploads via upload_file/log_artifact/ log_checkpoint_file. Not supported offline: versioned artifact uploads and the media helpers (Image/Video/Audio) that need a live upload response — these raise UnsupportedOfflineOperation naming the online alternative.

If a segment write fails (disk full, read-only filesystem), the event is dropped and counted in run.json instead of crashing the training loop; drops force an incomplete state at sync time rather than silent loss. A clean finish() writes {"clean": true}; a SIGTERM/SIGINT interruption writes {"clean": false}; a hard kill leaves finish null.

Modes and environment variables

Precedence for mode is init(mode=...) > INSTANTML_MODE > online. upload_mode only tunes online delivery and is ignored (with a debug notice) in offline and disabled modes.

Turn off logging with mode="disabled"

mode="disabled" returns a run whose full logging surface is present but every method is an inert no-op. It performs no network and no disk I/O, installs no signal/atexit handlers, and generates a local run id for API-shape parity — ideal for CI and test runs that import training code unchanged.

run = im.init(project="ci", mode="disabled")  # or INSTANTML_MODE=disabled
run.log_metrics({"loss": 1.0}, step=1)  # no-op
run.finish()

Pick a mode

Batch many scalar values into one metrics dictionary for high-frequency loops.

Next steps

• SDK logging — create runs and log metrics
• Metrics and steps — batching and step semantics
• Distributed training — rank metrics and multi-process runs
• Troubleshooting — first checks when uploads stall