Task-aware AutoDL orchestration for Codex research workflows.
Size the run, choose a region and GPU, inspect the worker, then keep a human at the billing boundary.

English · 简体中文

AutoDL Research Pilot

AutoDL Research Pilot gives Codex enough context to make a useful rental decision: the experiment contract, peak disk demand, prepared data regions, transfer conditions, current stock, expected throughput, budget, and deadline. It produces a reviewable plan, creates the resource only after confirmation, and can inspect the allocated worker over SSH before the expensive run begins.

Version 0.3 adds the pieces that turn allocation into an operating workflow:

• resilient Developer API reads with structured failure diagnosis;
• OS-keyring credentials and an explicit console compatibility mode for Pro;
• one typed view of Pro and Elastic resources;
• workload admission for disk, persistence, region, runtime, and network;
• live filesystem, CPU, memory, GPU, and bounded network probes;
• preview → confirm → receipt for paid operations.

The project uses the official AutoDL Developer API as its default control plane. Browser-session compatibility is separate, visible, and opt-in.

Quick start

Install

git clone https://github.com/chengxi271-commits/autodl-research-pilot.git
cd autodl-research-pilot
python -m venv .venv

Windows PowerShell:

.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[full]"

Linux / macOS:

source .venv/bin/activate
python -m pip install -e '.[full]'

full installs the MCP, OS-keyring, and SSH extras. Install .[auth], .[remote], or .[mcp] when only one integration is needed.

Save the Developer Token

The recommended login reads the token without echoing it and stores it under the fixed autodl-research-pilot service in the operating-system keyring:

autodl-pilot auth login
autodl-pilot auth status
autodl-pilot doctor --live

Remove the saved Developer Token with autodl-pilot auth logout.

For an ephemeral shell, use AUTODL_TOKEN instead. The environment takes precedence over the keyring and is resolved again for every request.

$env:AUTODL_TOKEN = "YOUR_TOKEN"

export AUTODL_TOKEN='YOUR_TOKEN'

The token stays out of project TOML, plans, receipts, logs, and request previews.

Describe the run before renting

cp examples/elastic.project.toml my-run.project.toml
cp examples/elastic.catalog.toml my-run.catalog.toml
autodl-pilot workload-inspect --project my-run.project.toml

The project file records four small contracts:

• runtime: mode, an audition or full phase, and a launch policy. Use
  product_default when comparing products: it resolves to
  manual_after_preflight on Pro and embedded_preflight on Elastic;
• capacity: system runtime, hot workspace, cache, persistent output and
  headroom as separate byte demands, plus workspace/persistent inode counts,
  CPU, total and working-set RAM, GPU count, and VRAM;
• storage: the concrete workspace, cache and persistent-output paths, which
  determine which filesystem pays each capacity demand, plus prepared regions;
• network: total download volume, every relevant source, the primary source
  used for transfer ETA, cache state, and measured throughput per region.

workload-inspect reports the compatible product set. It recommends a backend
only when exactly one product fits. When both Pro and Elastic fit,
recommended_backend remains null and requires_cost_time_comparison=true:
build both candidate plans and run portfolio-plan instead of defaulting to
Elastic. Missing measurements remain visible; they do not turn into optimistic
defaults.

Build and compare candidate plans

autodl-pilot discover --catalog my-run.catalog.toml --output live-context.json
autodl-pilot live-plan \
  --project my-run.project.toml \
  --catalog my-run.catalog.toml \
  --profile balanced \
  --output elastic.plan.json

# Local request preview: no mutation
autodl-pilot apply-live-plan --plan elastic.plan.json

The live planner treats each GPU-profile/region pair as a separate candidate. The selected request is bound to one observed region and the catalog's price_to ceiling. Confirmation refreshes wallet, stock, deployment-name, image, workload, region-storage, budget, and deadline conditions before calling the create endpoint.

With launch_policy = "product_default", the Elastic path requires
preflight_cmd in the catalog and sends one foreground command equivalent to
preflight_cmd && cmd. The embedded preflight must check mounts, actual free
space/inodes, runtime and fixed inputs before a large download or model load. A
failed preflight prevents the scientific command from starting and lets the
finite container stop.

If workload inspection reports both products compatible, build a Pro/local
plan from current quotes under the same project contract and compare the two:

autodl-pilot plan \
  --project my-run.project.toml \
  --offers current-offers.toml \
  --profile balanced \
  --output pro.plan.json

autodl-pilot portfolio-plan \
  --plan pro.plan.json \
  --plan elastic.plan.json \
  --profile balanced \
  --output portfolio.json

Follow portfolio.json.next_action. A candidate that was not the selected,
execution-ready result in its source plan must be replanned in that source
planner before preview. Once the chosen source plan is current, confirm only
that plan; for an Elastic winner:

autodl-pilot apply-live-plan --plan elastic.plan.json
autodl-pilot apply-live-plan \
  --plan elastic.plan.json \
  --receipt runs/my-run.receipt.json \
  --confirm

Candidates that miss VRAM, budget, deadline, inventory, or research-admission requirements are removed before scoring. For the rest, total time includes queue, setup, transfer, and workload-adjusted compute time; total cost uses the conservative billed time and approved hourly ceiling. For AutoDL Pro offers, the configured transfer time is floored at the slowest measured primary-source ETA among the allowed data centers, so a cheap offer cannot hide a region's cold-start download. economy chooses the lowest conservative cost, time chooses the fastest feasible run, balanced uses 45% cost and 55% time regret on the Pareto frontier, and custom accepts project weights. Performance factors should come from a short run of the same model, precision, batch semantics, and data path.

Inspect the allocated runtime

autodl-pilot resources

autodl-pilot resource-inspect elastic DEPLOYMENT_UUID \
  --container-uuid CONTAINER_UUID \
  --runtime

autodl-pilot runtime-preflight elastic DEPLOYMENT_UUID \
  --container-uuid CONTAINER_UUID \
  --project my-run.project.toml \
  --plan elastic.plan.json \
  --network

autodl-pilot resource-watch elastic DEPLOYMENT_UUID \
  --container-uuid CONTAINER_UUID \
  --samples 6 \
  --interval 60 \
  --runtime

runtime-preflight checks the actual filesystems containing the configured
workspace, cache and persistent output, groups paths that share a device, and
compares their combined byte/inode demand with real free capacity. Passing the
approved --plan also binds the observation to its GPU count/model and region:
Elastic must match its exact selected region, while Pro must be inside the
approved data_center_list. The SSH probe collects CPU/load, memory,
filesystem state, and GPU utilization through NVML, nvidia-smi, or AutoPanel
when available. Connection credentials live only long enough to open the
session; public output is sanitized.

For a full Pro run, product_default resolves to
manual_after_preflight. Leave start_command unset, create or power on the
instance, pass its approved plan to runtime-preflight, and start the workload
over SSH only after that gate passes. AutoDL documents that failure of a Pro
start_command does not fail instance startup, so provider state running
cannot stand in for this gate.

When the API “suddenly stops working”

A failed call does not identify its cause by itself. v0.3 keeps the useful distinctions in every AutoDLError:

Mutations never retry automatically. A timed-out create, power, stop, or delete call may already have reached AutoDL; the next step is state reconciliation, not a duplicate request.

The Pro API documentation specifies snapshot and status as GET requests carrying a JSON body. Some HTTP paths reject that shape. The default auto mode first sends the documented form, then tries a query-string compatibility form only after an explicit 400, 405, 422, or request-shape error. Authentication and permission failures never trigger that switch. Set AUTODL_GET_PAYLOAD_MODE=json to require the official form or query to select compatibility mode directly.

AutoDL's public API documentation describes where to obtain a Developer Token; it does not publish a refresh endpoint or token-lifetime contract. v0.3 reloads the environment/keyring value on every request and leaves replacement to autodl-pilot auth login. The client does not scrape the console for a new token.

autodl-pilot auth status --live and doctor --live probe capabilities separately. A working wallet call no longer implies that Pro instances, Elastic deployments, images, and stock all share the same permission. AutoDL documents separate eligibility for Pro, Elastic, and subaccounts.

auth status also reports the credential source that is actually active. When
AUTODL_TOKEN and keyring both exist, the environment wins and
keyring_shadowed_by_environment=true; after updating keyring, unset the stale
environment variable before expecting the new token to take effect.

auth status may also display the JWT-shaped aud, tenant, and exp fields
as unverified routing hints. Account identifiers and the credential itself are
omitted, and these hints never replace a live capability probe.

Explicit console compatibility for Pro

If the documented Pro endpoint is unavailable while the logged-in website still manages the same instance, v0.3 can use a human-supplied browser-session credential:

autodl-pilot auth console-login --app-version APP_VERSION

# Developer API first; console fallback for eligible Pro read failures
autodl-pilot --pro-control-plane auto resources
autodl-pilot --pro-control-plane auto resource-inspect pro PRO_UUID --runtime

# Select the compatibility client directly
autodl-pilot --pro-control-plane console api instances

# Recovery for an existing stopped Pro root filesystem; preview the price first
autodl-pilot --pro-control-plane auto pro-disk-expand PRO_UUID \
  --required-free-gib REQUIRED_FREE_GIB
autodl-pilot --pro-control-plane auto pro-disk-expand PRO_UUID \
  --required-free-gib REQUIRED_FREE_GIB \
  --current-free-gib CURRENT_FREE_GIB --confirm

autodl-pilot auth console-logout

console-login prompts for the current browser-session Authorization value and stores it with AppVersion in the OS keyring. Set AUTODL_CONSOLE_SUB_USER=1 for the observed subaccount route.

On a stopped Pro instance, --required-free-gib is the desired free space on the root filesystem, not the
create-time expansion increment. The command reads current root free space from
the snapshot and expands by ceil(required - current); pass
--current-free-gib only when runtime-preflight measured it but the snapshot
did not expose usable root usage. It does not resize /root/autodl-tmp or an
external mount.

These frontend routes are not part of AutoDL's published Developer API. They may change with the website, have no documented stability promise, and remain disabled unless --pro-control-plane console or auto is supplied. For Pro reads, auto falls back after auth, permission, contract, or request-shape failures; a transient Developer API outage remains visible. Confirmed power operations are never resent through another control plane. Inspect state, then explicitly select console and preview again if needed. Elastic operations and Pro creation continue through the documented Developer API.

Three resource surfaces

AutoDL exposes three operational surfaces with different identifiers and lifecycle rules:

The official Elastic overview documents the differences from marketplace instances: Elastic containers have a fixed 50 GB local data disk, no Jupyter entry, no restart after stop, and no durable local state after the lifecycle ends. Pro has its own endpoint family and pro-* identifiers. An instance missing from the ordinary “Container Instances” page can still exist and accrue usage under another control plane.

autodl-pilot resources merges Pro instances and Elastic deployments/containers into typed JSON while preserving kind, resource_id, parent deployment, state, region, GPU, and control-plane provenance. It also reports manual_market_instances.discovered=false, keeping the undocumented inventory gap explicit.

Storage and region admission

Disk demand is decided before the rental request:

• Path mapping: system_required_gb always belongs to the root filesystem.
  local_hot_required_gb, cache_required_gb, and
  persistent_required_gb are charged to the filesystems containing
  storage.workspace, storage.cache, and storage.persistent_output.
  /root/autodl-tmp is the local data role; /root/autodl-fs and
  /root/autodl-nas are external durable roles; other writable paths are on the
  system role. The runtime gate groups paths that resolve to the same device,
  sums their requirements, and adds headroom once to that device.
• Pro: create-time system expansion is
  ceil(system-role demand + headroom - 30 GB), clamped at zero. Moving the
  cache or workspace between the system disk and /root/autodl-tmp therefore
  changes the result. The Pro create API
  accepts 0–500 GB of system-disk expansion. A data-role demand above the
  default 50 GB is not rescued by system expansion because the Developer API
  has no data-disk size field.
• Elastic: all workspace and cache demand mapped to /root/autodl-tmp, plus
  headroom, must fit its fixed 50 GB local disk. Persistent output must map to a
  prepared durable mount in the selected region. Jobs that fail either rule are
  excluded before allocation.
• File storage: /root/autodl-fs is region-local shared storage. It must be initialized in each candidate region; it is durable and convenient, with lower I/O performance than local disk and a documented 200,000-inode limit. See file storage and the instance filesystem layout.

The Elastic catalog may list several dc_list regions. Stock is queried per region, each (profile, region) result stays separate, and a plan with persistent output cannot execute until its chosen region appears in prepared_regions.

For a Pro plan, start from a fresh account quote because the public Pro API has no pre-allocation stock or market-price endpoint:

autodl-pilot plan \
  --project examples/project.toml \
  --offers current-offers.toml \
  --profile balanced \
  --output pro.plan.json

autodl-pilot apply-plan --plan pro.plan.json
autodl-pilot apply-plan \
  --plan pro.plan.json \
  --receipt runs/pro-run.receipt.json \
  --confirm

The workload admission is embedded in the plan, including the derived system-disk expansion. A successful confirmed create writes the provider UUID and approved selection to a non-secret receipt, giving a later Codex session a stable handoff record.

Network-aware staging

AutoDL uses shared regional bandwidth, documented at roughly 3–10 Gbps for a region, so the path seen by one worker varies with load (network). workload-inspect therefore asks for download volume, all source classes, primary_download_source, cache state, and any available observed_mbps_by_region. The observed_mbps shorthand is valid only when one region is prepared. Runtime preflight requires every declared source to have a built-in or caller-supplied probe and tests them separately; only the primary source's regional observation is converted into the download ETA. A full live plan with uncached downloads and no primary-source observation for its selected region asks for a short network audition.

After a comparable slice finishes, convert its measured compute and transfer
time into regional scheduler inputs, inspect them, then rerun the plan:

autodl-pilot audition-calibrate \
  --project my-run.project.toml \
  --profile-id fast-24gb \
  --region westDC2 \
  --work-fraction 0.1 \
  --compute-seconds 900 \
  --download-bytes 1073741824 \
  --download-seconds 120

The download sample in this command must come from the project's
primary_download_source; the resulting regional Mbps/ETA is labelled with
that source rather than generalized to every origin.

Add --network to runtime-preflight to probe the declared entries in
sources. For a standalone resource-inspect, pass one credential-free
--network-url. Each route uses Range: bytes=0-1048575, accepts at most 1 MiB,
and uses a 10-second CLI timeout; the underlying bounded probe accepts at most
30 seconds, with at most eight sources in one preflight. Hugging Face and GitHub may test both direct
and command-scoped /etc/network_turbo, so their maximum is 2 MiB per source;
other origins use one route. Every declared source is required. A source without
a built-in mapping blocks the network preflight until its real,
credential-free probe object is supplied; repeat the option for multiple custom
sources:

autodl-pilot runtime-preflight elastic DEPLOYMENT_UUID \
  --container-uuid CONTAINER_UUID \
  --project my-run.project.toml --plan elastic.plan.json --network \
  --network-source-url private_oss=https://HOST/PATH/probe.bin

Omit --network only for a general capacity/telemetry inspection, not as proof
that a full download is ready. Proxy variables do not leak into later package
installs or training.

For a large HTTP object, the bundled downloader verifies byte-range support, keeps completed chunks across interruptions, and publishes the assembled file with an atomic rename:

autodl-resumable-http \
  "https://HOST/PATH/model.bin" \
  "/root/autodl-tmp/models/model.bin" \
  --workers 4

If the probe selected the academic route for GitHub or Hugging Face, scope it to this command:

bash -lc 'source /etc/network_turbo && autodl-resumable-http "https://HOST/PATH/model.bin" "/root/autodl-tmp/models/model.bin" --workers 4'

The academic acceleration guide limits that service to supported academic domains and gives no stability guarantee. Package installs should use an appropriate domestic pip/conda source without inherited proxy variables. When /root/autodl-tmp exists and has sufficient capacity, set HF_HOME=/root/autodl-tmp/cache to keep Hugging Face caches off the system disk (Hugging Face guide).

For a non-empty object, resume is admitted only when the server supplies a
strong ETag or Last-Modified validator, or the caller supplies an authoritative
--sha256. The .parts/object-identity.json binding records the sanitized URL,
length, validator and optional expected digest; range requests use If-Range,
and changed identity refuses old chunks. The query string and URL credentials
are never written to that record or JSON output. For bulk datasets, pre-stage
to same-region file storage or object storage and copy the hot subset to local
disk. No checksum sidecar is created when an expected digest was not supplied.

Observation and lifecycle

Pro snapshots expose CPU, memory, root/data-disk usage, connection details, and image progress. Elastic list/event APIs expose allocation and lifecycle state. Dynamic GPU metrics require the data plane: SSH, NVML/nvidia-smi, or AutoDL's in-container endpoint at 127.0.0.1:2022 for eligible enterprise accounts (performance monitoring).

resource-watch is bounded to 1–60 samples. --hourly-price-cny means the
price of one GPU-hour; pass the approved --gpu-count, and the estimate is
elapsed_hours × hourly_price_cny × gpu_count. Optional billing-start, budget,
and deadline arguments produce review alerts; the command reports
automatic_stop=false.

autodl-pilot resource-watch pro PRO_UUID \
  --samples 3 --interval 60 --runtime \
  --hourly-price-cny PRICE_PER_GPU --gpu-count GPU_COUNT \
  --billing-started-at ISO_TIME --budget-cny BUDGET

Stopping still uses the preview/confirm flow:

autodl-pilot container-stop CONTAINER_UUID
autodl-pilot container-stop CONTAINER_UUID --no-cache --confirm

autodl-pilot deployment-stop DEPLOYMENT_UUID
autodl-pilot deployment-stop DEPLOYMENT_UUID --confirm

autodl-pilot power-off PRO_UUID
autodl-pilot power-off PRO_UUID --confirm

Elastic cmd is the container lifecycle: the foreground command exits, then the container stops. Persist checkpoints during the run; the documented shutdown hook has a five-second limit. A reuse cache may contain previous files, so reusable deployments should initialize a unique run directory before training. See the Elastic operating guide.

CLI map

All ordinary results and controlled failures are JSON. Run autodl-pilot --help for the complete argument reference.

Codex and MCP

Load the repository as a local Codex plugin after installing the mcp extra. .mcp.json targets Windows; docs/mcp.unix.json is the Unix manifest. The Codex workflow lives in skills/autodl-research-pilot/SKILL.md.

MCP preflight_run defaults to network=false, so an ordinary capacity probe
does not contact download origins. For a full network gate, pass
network=true, the approved plan as approved_plan, and any custom mappings as
network_source_urls={"SOURCE":"https://HOST/PATH/probe.bin"}. Every source
declared by the workload must then have a probe result.

A useful first prompt is:

Inspect this project's workload contract, compare the prepared AutoDL regions, and show me the selected request and runtime preflight. Ask before any paid or destructive action.

Current boundaries

• The live scheduler currently creates one-GPU Elastic Container deployments. Job, ReplicaSet, and multi-GPU policy need separate workload evidence.
• The public Pro API does not expose a live pre-allocation catalog; Pro planning uses supplied, current offers.
• Ordinary marketplace instances are not enumerated because no public Developer API for them is documented.
• Console compatibility covers implemented Pro reads and lifecycle operations only. It is an observed frontend contract, requires current human-supplied credentials, and may change without notice.
• Runtime inspection executes a fixed probe set. It is not a general remote shell or training runner.
• resource-watch raises review alerts and never stops a resource automatically.
• Provider price fields and purchased duration packages are not treated as a complete billing ledger. Confirm the platform bill for long runs.

Endpoint and unit notes are collected in AutoDL API Boundaries. This project is community-maintained and is not affiliated with AutoDL.

Development

python -m unittest discover -s tests -v

See CONTRIBUTING.md, SECURITY.md, and the issue tracker.

License

Apache-2.0