# Buffering, offline replay, and spool mode

Pick the upload mode that matches your training loop's durability needs, and
know how to recover queued data when a process exits early.

## Use the default async mode

By default, `im.init(...)` starts run creation in the background, returns a run
handle immediately, and uses async metric/log/status uploads after the run
exists. The first write waits for run creation if it has not finished yet.

```python
import instantml as im

run = im.init(project="demo")
run.log({"train/loss": 0.2}, step=1)
run.finish(timeout=30)
```

Use `run.wait_for_init()` when a script should fail before expensive training
starts:

```python
run = im.init(project="demo")
run.wait_for_init()
```

Set `async_init=False` for fully synchronous run creation:

```python
run = im.init(project="demo", async_init=False)
```

After initialization, scalar metrics, rank metrics, console logs, and final run
status are queued locally and drained by the async uploader. Delivery failures
surface through `Run.upload_status()`, dashboard upload-health metrics, and
wait helpers rather than raising from hot-path `log(...)` calls.

```python
run.log({"train/loss": 0.2}, step=1)
print(run.upload_status())
run.wait_for_processing(timeout=30)
```

Use `upload_mode="sync"` when a short script should raise foreground
`InstantMLError` exceptions for normal post-init writes:

```python
run = im.init(project="demo", upload_mode="sync")
run.log({"train/loss": 0.2}, step=1)
run.finish()
```

## Recover the async queue after finish() times out

`run.finish()` drains the async queue within a time budget: an explicit
`finish(timeout=...)` if you pass one, otherwise `INSTANTML_FINISH_DRAIN_SECONDS`
if set, otherwise the client timeout (10 seconds by default). If the drain does
not complete in time, the SDK warns and leaves the remaining rows on disk under
the queue directory (`.instantml/async` by default, configurable with
`queue_dir=` on `init`).

Upload the leftover rows with the bundled uploader:

```bash
instantml-uploader --queue-dir .instantml/async
```

Or give slow-network jobs a larger drain budget up front:

```bash
export INSTANTML_FINISH_DRAIN_SECONDS=120
python train.py
```

> **Warning:** Events still buffered inside the training process when it exits are lost — only rows already written to the on-disk queue can be recovered by the uploader.

## Batch writes in memory

Use buffering to batch post-init SDK calls in memory:

```python
run = im.init(project="long-run", buffer_size=25)

for step in range(1000):
    run.log({"train/loss": 1.0 / (step + 1)}, step=step)

run.flush()
run.finish()
```

Call `flush()` before process exit or before you need writes visible in the UI.

## Replay failed requests with offline_dir

Use `offline_dir` to store failed foreground requests and replay them later.
It applies to requests that run in the foreground — sync-mode writes, buffered
flushes, and calls such as configs, tags, artifacts, and objects. In the
default async mode, hot-path writes go to the async queue instead (see above):

```python
run = im.init(
    project="resilient-run",
    upload_mode="sync",
    offline_dir=".instantml/offline",
)

run.log({"train/loss": 0.2}, step=1)

# Later, after the server is reachable:
replayed = run.replay_offline()
```

Current limitation: `init()` still needs a reachable server because run
creation is not spooled yet.

## Isolate uploads with spool mode

Use `upload_mode="spool"` when the training process should avoid post-init HTTP
calls:

```python
run = im.init(
    project="long-run",
    name="seed-42",
    upload_mode="spool",
    spool_dir=".instantml/spool",
)
```

Drain the spool from a separate process:

```bash
python -m instantml.uploader \
  --spool-dir .instantml/spool \
  --base-url https://api.instantml.ai
```

Metric and console-log events send their event ID as `Idempotency-Key`, so a
compatible server can safely accept retries.

## Pick a mode

| Need | Mode |
| --- | --- |
| Small script, easiest behavior | Default async init plus async writes |
| Foreground exceptions on post-init writes | `upload_mode="sync"` |
| Fewer HTTP calls | `buffer_size` |
| Replay failed requests after temporary outage | `offline_dir` |
| Keep post-init HTTP out of the training process | `upload_mode="spool"` |

Batch many scalar values into one metrics dictionary for high-frequency loops.

## Next steps

- [SDK logging](/docs/sdk/logging.md) — create runs and log metrics
- [Metrics and steps](/docs/sdk/metrics-steps.md) — batching and step semantics
- [Distributed training](/docs/sdk/distributed-training.md) — rank metrics and multi-process runs
- [Troubleshooting](/docs/troubleshooting.md) — first checks when uploads stall

## Agent navigation

- [Docs index](/llms.txt)
- [Full docs bundle](/llms-full.txt)

### Get Started

- [Overview](/docs/index.md)
- [Quickstart](/docs/quickstart.md)
- [Core Concepts](/docs/concepts/core-concepts.md)
- [Pricing](/docs/pricing.md)
- [Benchmarks](/docs/benchmarks.md)
- [Examples](/docs/guides/examples.md)
- [Troubleshooting](/docs/troubleshooting.md)

### SDK

- [Installation Auth](/docs/sdk/installation-auth.md)
- [Logging](/docs/sdk/logging.md)
- [Tracing](/docs/sdk/tracing.md)
- [Metrics Steps](/docs/sdk/metrics-steps.md)
- [Config Tags Notes](/docs/sdk/config-tags-notes.md)
- [Artifacts Checkpoints](/docs/sdk/artifacts-checkpoints.md)
- [Rich Objects](/docs/sdk/rich-objects.md)
- [Distributed Training](/docs/sdk/distributed-training.md)
- [Console System Integrations](/docs/sdk/console-system-integrations.md)
- [Reliability](/docs/sdk/reliability.md) (current page)
- [Cli Login](/docs/sdk/cli-login.md)
- [Querying Data](/docs/sdk/querying-data.md)
- [Agent Mcp](/docs/sdk/agent-mcp.md)
- [Examples Patterns](/docs/sdk/examples-patterns.md)

### Integrations

- [Overview](/docs/integrations/overview.md)
- [Pytorch Lightning](/docs/integrations/pytorch-lightning.md)
- [Huggingface Transformers](/docs/integrations/huggingface-transformers.md)
- [Keras](/docs/integrations/keras.md)
- [Wandb](/docs/integrations/wandb.md)

### Dashboard

- [Tour](/docs/dashboard/tour.md)
- [Organizations Workspaces](/docs/dashboard/organizations-workspaces.md)
- [Runs Workspace](/docs/dashboard/runs-workspace.md)
- [Metrics Charts](/docs/dashboard/metrics-charts.md)
- [Run Detail](/docs/dashboard/run-detail.md)
- [Traces](/docs/dashboard/traces.md)
- [Compare Runs](/docs/dashboard/compare-runs.md)
- [Research Dashboards](/docs/dashboard/research-dashboards.md)
- [Artifacts Files](/docs/dashboard/artifacts-files.md)
- [Run Health](/docs/dashboard/alerts.md)
- [Datasets](/docs/dashboard/datasets.md)
- [Checkpoints](/docs/dashboard/checkpoints.md)
- [Reports](/docs/dashboard/reports.md)
- [Settings Api Keys](/docs/dashboard/settings-api-keys.md)
- [Api Tab](/docs/dashboard/api-tab.md)
- [Onboarding Team Billing](/docs/dashboard/onboarding-team-billing.md)

### Data

- [Imports](/docs/guides/imports.md)
- [W&B alternative](/docs/guides/wandb-alternative.md)
- [InstantML vs MLflow](/docs/guides/instantml-vs-mlflow.md)
- [W&B import guide](/docs/guides/wandb-import-guide.md)
- [W&B and Neptune imports](/docs/guides/wandb-neptune-imports.md)
- [Export Usage Limits](/docs/guides/export-usage-limits.md)
- [Pricing Limits Billing](/docs/guides/pricing-limits-billing.md)
- [Auth Billing Storage](/docs/guides/auth-billing-storage.md)
- [Customer Owned Clickhouse](/docs/guides/customer-owned-clickhouse.md)
- [Observability](/docs/guides/observability.md)

### API

**Practical API guides**

- [Authentication](/docs/api/authentication.md)
- [Errors And Limits](/docs/api/errors-and-limits.md)
- [Health Observability](/docs/api/health-observability.md)
- [Projects Runs](/docs/api/projects-runs.md)
- [Metrics Series](/docs/api/metrics-series.md)
- [Attributes Objects](/docs/api/attributes-objects.md)
- [Artifacts](/docs/api/artifacts.md)
- [Reports](/docs/api/reports.md)
- [Iframe Embeds](/docs/api/iframe-embeds.md)
- [Import Export Usage](/docs/api/import-export-usage.md)
- [Dashboard Control State](/docs/api/dashboard-control-state.md)
**Architecture**

- [System Overview](/docs/architecture/system-overview.md)
- [Service Planes](/docs/architecture/service-planes.md)
- [Storage Model](/docs/architecture/storage-model.md)
- [Google Clickhouse](/docs/architecture/google-clickhouse.md)
- [Auth Tenancy](/docs/architecture/auth-tenancy.md)
- [Schema Reference](/docs/architecture/schema-reference.md)

### API Reference

- [API Reference](/docs/api-reference.md)