Buffering, offline replay, and spool mode

Pick the upload mode that matches your training loop's durability needs, and know how to recover queued data when a process exits early.

Use the default async mode

By default, im.init(...) starts run creation in the background, returns a run handle immediately, and uses async metric/log/status uploads after the run exists. The first write waits for run creation if it has not finished yet.

import instantml as im

run = im.init(project="demo")
run.log({"train/loss": 0.2}, step=1)
run.finish(timeout=30)

Use run.wait_for_init() when a script should fail before expensive training starts:

run = im.init(project="demo")
run.wait_for_init()

Set async_init=False for fully synchronous run creation:

run = im.init(project="demo", async_init=False)

After initialization, scalar metrics, rank metrics, console logs, and final run status are queued locally and drained by the async uploader. Delivery failures surface through Run.upload_status(), dashboard upload-health metrics, and wait helpers rather than raising from hot-path log(...) calls.

run.log({"train/loss": 0.2}, step=1)
print(run.upload_status())
run.wait_for_processing(timeout=30)

Use upload_mode="sync" when a short script should raise foreground InstantMLError exceptions for normal post-init writes:

run = im.init(project="demo", upload_mode="sync")
run.log({"train/loss": 0.2}, step=1)
run.finish()

Recover the async queue after finish() times out

run.finish() drains the async queue within a time budget: an explicit finish(timeout=...) if you pass one, otherwise INSTANTML_FINISH_DRAIN_SECONDS if set, otherwise the client timeout (10 seconds by default). If the drain does not complete in time, the SDK warns and leaves the remaining rows on disk under the queue directory (.instantml/async by default, configurable with queue_dir= on init).

Upload the leftover rows with the bundled uploader:

instantml-uploader --queue-dir .instantml/async

Or give slow-network jobs a larger drain budget up front:

export INSTANTML_FINISH_DRAIN_SECONDS=120
python train.py

Batch writes in memory

Use buffering to batch post-init SDK calls in memory:

run = im.init(project="long-run", buffer_size=25)

for step in range(1000):
    run.log({"train/loss": 1.0 / (step + 1)}, step=step)

run.flush()
run.finish()

Call flush() before process exit or before you need writes visible in the UI.

Replay failed requests with offline_dir

Use offline_dir to store failed foreground requests and replay them later. It applies to requests that run in the foreground — sync-mode writes, buffered flushes, and calls such as configs, tags, artifacts, and objects. In the default async mode, hot-path writes go to the async queue instead (see above):

run = im.init(
    project="resilient-run",
    upload_mode="sync",
    offline_dir=".instantml/offline",
)

run.log({"train/loss": 0.2}, step=1)

# Later, after the server is reachable:
replayed = run.replay_offline()

Current limitation: init() still needs a reachable server because run creation is not spooled yet.

Isolate uploads with spool mode

Use upload_mode="spool" when the training process should avoid post-init HTTP calls:

run = im.init(
    project="long-run",
    name="seed-42",
    upload_mode="spool",
    spool_dir=".instantml/spool",
)

Drain the spool from a separate process:

python -m instantml.uploader \
  --spool-dir .instantml/spool \
  --base-url https://api.instantml.ai

Metric and console-log events send their event ID as Idempotency-Key, so a compatible server can safely accept retries.

Pick a mode

Batch many scalar values into one metrics dictionary for high-frequency loops.

Next steps

• SDK logging — create runs and log metrics
• Metrics and steps — batching and step semantics
• Distributed training — rank metrics and multi-process runs
• Troubleshooting — first checks when uploads stall