Skip to content

Environment Capability Specification ​

Capability Identity ​

PropertyValue
EnumA2ECapability.ENV
String"env"
Plugin TypeEnvPlugin
Namespaceenv/*
Message Count25

Overview ​

The environment capability provides RL-style (Reinforcement Learning) step-wise interaction between Agent and Environment. It defines the core primitives for agentic loops: reset an episode, take a step, observe state, and receive rewards.

Core primitives:

  • reset — Initialize a new episode, return initial state
  • step — Execute action, receive (next_state, reward, done, info)
  • observe — Read-only current state without acting

Extended primitives:

  • close — Terminate an episode early
  • spaces — Discover action/state space definitions
  • render — Retrieve visual/multimodal representation
  • plan — Get environment-suggested affordances/actions
  • batch_step — Execute multiple steps in parallel

Server-initiated:

  • state/push — Incremental state update pushed to agent

Data plane (orthogonal to the episode lifecycle):

  • data/reset — Restore the data root to its pristine state
  • data/add — Stage task data into the data root (metadata only; host moves bytes)
  • data/get — Read artifacts back out of the data root

Host extension:

  • exp/list — Pull the transitions the host recorded for an episode (host-implemented; not in the base plugin)

Cross-capability integration:

  • Each env/step interaction can be auto-recorded as an (s, a, r, s', done) tuple in the ExperienceBuffer
  • Reward signals can be forwarded to the learning subsystem (learn/*)
  • Enables RL training loops, simulations, and CUA/browser environments

Protocol Flow ​

Message Types (25) ​

Reset (2) ​

env/reset/req — EnvResetRequest ​

Agent → Host. Initialize a new episode.

FieldTypeRequiredDefaultDescription
typestrYes"env/reset/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
env_namestrYes—Environment identifier
seedintNoNoneRandom seed for reproducibility
optionsdict[str, Any]No{}Environment-specific options

env/reset/resp — EnvResetResponse ​

Host → Agent. Returns initial observation.

FieldTypeRequiredDefaultDescription
typestrYes"env/reset/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
obsEnvObservationYes—Initial observation

Step (2) — Core RL Primitive ​

env/step/req — EnvStepRequest ​

Agent → Host. Execute an action in the environment.

FieldTypeRequiredDefaultDescription
typestrYes"env/step/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrYes—Active episode identifier
actiondict[str, Any]Yes—Action to execute

env/step/resp — EnvStepResponse ​

Host → Agent. Returns observation after action.

FieldTypeRequiredDefaultDescription
typestrYes"env/step/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
obsEnvObservationYes—Post-action observation

Observe (2) — Read-only State ​

env/observe/req — EnvObserveRequest ​

Agent → Host. Retrieve current state without acting.

FieldTypeRequiredDefaultDescription
typestrYes"env/observe/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrYes—Episode to observe

env/observe/resp — EnvObserveResponse ​

Host → Agent. Returns current observation.

FieldTypeRequiredDefaultDescription
typestrYes"env/observe/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
obsEnvObservationYes—Current observation

Close (2) — End Episode ​

env/close/req — EnvCloseRequest ​

Agent → Host. Terminate an episode early.

FieldTypeRequiredDefaultDescription
typestrYes"env/close/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrYes—Episode to close

env/close/resp — EnvCloseResponse ​

Host → Agent.

FieldTypeRequiredDefaultDescription
typestrYes"env/close/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
closedboolYesTrueWhether close succeeded

Spaces (2) — Action/State Discovery ​

env/space/req — EnvSpacesRequest ​

Agent → Host. Discover action and state space definitions.

FieldTypeRequiredDefaultDescription
typestrYes"env/space/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
env_namestrYes—Environment identifier

env/space/resp — EnvSpacesResponse ​

Host → Agent.

FieldTypeRequiredDefaultDescription
typestrYes"env/space/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
action_spacedict[str, Any]Yes—JSON Schema defining valid actions
state_schemadict[str, Any]Yes—JSON Schema defining state structure

Render (2) — Multimodal Support ​

env/render/req — EnvRenderRequest ​

Agent → Host. Retrieve visual/multimodal representation of current state.

FieldTypeRequiredDefaultDescription
typestrYes"env/render/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrYes—Episode to render
modestrNo"screenshot"Render mode: screenshot, rgb_array, text

env/render/resp — EnvRenderResponse ​

Host → Agent.

FieldTypeRequiredDefaultDescription
typestrYes"env/render/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
renderAnyYes—Rendered output (bytes, base64, or structured)

Plan (2) — Affordance Discovery ​

env/plan/req — EnvPlanRequest ​

Agent → Host. Get environment-suggested actions.

FieldTypeRequiredDefaultDescription
typestrYes"env/plan/resp"Message type (note: shares resp value)
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrNoNoneOptional episode context
statedict[str, Any]NoNoneOptional state context

env/plan/resp — EnvPlanResponse ​

Host → Agent.

FieldTypeRequiredDefaultDescription
typestrYes"env/plan/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
suggested_actionslist[dict[str, Any]]Yes[]Suggested actions with descriptions

Batch Step (2) — Parallel Execution ​

env/batch_step/req — EnvBatchStepRequest ​

Agent → Host. Execute multiple actions in parallel across episodes.

FieldTypeRequiredDefaultDescription
typestrYes"env/batch_step/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idslist[str]Yes—Episode identifiers
actionslist[dict[str, Any]]Yes—Actions (1:1 with episode_ids)

env/batch_step/resp — EnvBatchStepResponse ​

Host → Agent.

FieldTypeRequiredDefaultDescription
typestrYes"env/batch_step/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
resultslist[EnvStepResponse]Yes—Results (1:1 with request)

State Push (1) — Server-initiated ​

env/state/push — EnvStatePush ​

Host → Agent (server-initiated). Incremental environment state update.

FieldTypeRequiredDefaultDescription
typestrYes"env/state/push"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrYes—Episode identifier
step_idintYes—Step number
action_idstrNoNoneTie to specific EnvAction
event_typestrYes—Observation, tool_result, status, error
reasonstrNo""Reason: "proc_exit", "oom_warning", etc.
deltadict[str, Any]No{}Sparse diff (only changed fields)
rewardfloatNoNoneOptional reward signal
reward_infodictNo{}Additional reward metadata
terminalboolNoFalseMarks episode termination

State push use cases:

  • Async tool completion
  • External world changes (filesystem, browser DOM)
  • Long-running process updates
  • Safety/system signals (OOM, timeout)

Experience List (2) — Recorded Transitions ​

A host extension (wire-compatible with xa-agent-env) for pulling the transitions a host recorded during an episode — the (state, action, reward, …) tuples a training loop consumes. Neither the base EnvPlugin nor EnvAPI implements these; a host that records transitions opts in by handling the two messages itself.

env/exp/list/req — EnvExpListRequest ​

Agent → Host. Pull recorded transitions.

FieldTypeRequiredDefaultDescription
typestrYes"env/exp/list/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
episode_idstrNo""Restrict to one episode (empty ⇒ all)
limitintNo100Maximum transitions to return

env/exp/list/resp — EnvExpListResponse ​

Host → Agent. Recorded experiences for an episode (or all).

FieldTypeRequiredDefaultDescription
typestrYes"env/exp/list/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
episodeslist[dict]No[]Recorded transitions

Data Plane (6) — Task Data Lifecycle ​

The data plane is orthogonal to the episode lifecycle. env/reset starts an episode; the env/data/* messages move task data into and out of the env's data root (default /workspace/data). A host may reset an episode without touching data, and restore data without starting an episode.

Payloads carry METADATA ONLY — bytes never cross the wire. At multi-GB scale the host performs a local filesystem operation inside the container (bind-mounted path, archive extraction, …), so no large payload is ever buffered in a protocol message.

env/data/reset/req — EnvDataResetRequest ​

Agent → Host. Restore the data plane to its pristine state. This is not an episode reset and not a purge-to-empty: the host restores whatever it considers pristine (typically clearing the data root and re-staging the env's baseline corpus). A host with no baseline simply empties the root and reports restored: 0.

FieldTypeRequiredDefaultDescription
typestrYes"env/data/reset/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
scopestrNo"all"all restores the whole root; baseline re-stages only the pristine corpus
data_rootstrNo/workspace/dataData root to restore

env/data/reset/resp — EnvDataResetResponse ​

Host → Agent. Data plane restored. req_id is required for RPC match.

FieldTypeRequiredDefaultDescription
typestrYes"env/data/reset/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
okboolYesTrueWhether restore succeeded
restoredintNo0Number of items re-staged to reach the pristine state
data_rootstrNo/workspace/dataData root restored
detailstrNo""Host-supplied detail

env/data/add/req — EnvDataAddRequest ​

Agent → Host. Stage task data into the data root before an episode. Each parts entry is an EnvDataPart (metadata only).

FieldTypeRequiredDefaultDescription
typestrYes"env/data/add/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
partslist[EnvDataPart]No[]Items to stage (metadata only)
data_rootstrNo/workspace/dataData root to stage into

env/data/add/resp — EnvDataAddResponse ​

Host → Agent. What was staged. Failures are per-part, never fatal.

FieldTypeRequiredDefaultDescription
typestrYes"env/data/add/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
okboolYesTrueFalse when any part failed
addedintNo0Number of items successfully staged
errorslist[str]No[]Per-part failure messages
stagedlist[dict]No[]Echo of each staged item (uri/dest/size_bytes/checksum)

env/data/get/req — EnvDataGetRequest ​

Agent → Host. Read items back out of the data root — the graded artifact-collection path. The host lists matching entries; content is only populated when the caller asks for it and the item is within the size cap.

FieldTypeRequiredDefaultDescription
typestrYes"env/data/get/req"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
querystrNo""Glob/substring relative to the data root (empty lists everything)
include_contentboolNoFalseInline file contents; the host MUST refuse above its size cap
max_bytesintNo1048576Per-item ceiling when include_content is set (default 1 MiB)
data_rootstrNo/workspace/dataData root to read from

env/data/get/resp — EnvDataGetResponse ​

Host → Agent. Items read back from the data root. Each item is {uri, dest, size_bytes, checksum, mime, content?}.

FieldTypeRequiredDefaultDescription
typestrYes"env/data/get/resp"Message type
idstrYesautoMessage UUID
versionstrYes"1.0"Protocol version
tsfloatYesautoTimestamp
req_idstrYes""Echoes request ID
okboolYesTrueWhether the read succeeded
itemslist[dict]No[]Matching items
truncatedboolNoFalseAn inlined item hit the max_bytes cap
detailstrNo""Host-supplied detail

EnvDataPart ​

One staged item. Bytes live on the host filesystem, never on the wire. Declared with extra="allow".

FieldTypeRequiredDefaultDescription
srcstrNo""Host-side source the host copies from (bind-mount, archive, …)
deststrNo""Path relative to the data root; empty means the part's own name
uristrNo""Logical identifier, recorded and echoed back (not dereferenced by the SDK)
checksumstrNo""sha256:<hex> verified by the host after staging (empty skips verification)
size_bytesintNoNoneSize of the staged item
mimestrNo""Optional MIME type

Data Models ​

EnvState ​

Flexible state container with extra="allow" — accepts any fields the environment provides.

EnvObservation ​

FieldTypeRequiredDefaultDescription
episode_idstrYes—Episode this observation belongs to
step_numintYes—Step number within episode
stateEnvStateYes—Environment state
doneboolNoFalseEpisode is complete
truncatedboolNoFalseEpisode was truncated (time limit)
rewardfloatNo0.0Reward signal
created_atfloatNoautoObservation timestamp
metadatadict[str, Any]No{}Additional metadata

EnvAction ​

FieldTypeRequiredDefaultDescription
action_typestrYes—Action type identifier
payloaddict[str, Any]No{}Action payload
metadatadict[str, Any]No{}Additional metadata

EnvEvent ​

FieldTypeRequiredDefaultDescription
event_idstrNoauto UUIDEvent identifier
typestrYes—Event type
episode_idstrYes—Episode context
step_idintYes—Step context
action_idstrNoNoneTie to EnvAction
payloaddict[str, Any]No{}Event payload
timestampfloatNoautoEvent timestamp
metadatadict[str, Any]No{}Additional metadata

Error Codes — EnvErrorCode ​

CodeEnum ValueDescriptionRetryable
runtime_errorRUNTIME_ERRORGeneral runtime failureDepends
unknown_actionUNKNOWN_ACTIONAction type not recognizedNo
reset_deniedRESET_DENIEDEnvironment refused resetNo

Wire Examples ​

Reset and Step Loop ​

json
{"type":"env/reset/req","id":"er1","version":"1.0","ts":1716123456.789,"env_name":"browser","seed":42,"options":{"url":"https://example.com"}}
json
{"type":"env/reset/resp","id":"er2","version":"1.0","ts":1716123457.100,"req_id":"er1","obs":{"episode_id":"ep_abc","step_num":0,"state":{"url":"https://example.com","title":"Example"},"done":false,"truncated":false,"reward":0.0}}
json
{"type":"env/step/req","id":"es1","version":"1.0","ts":1716123458.100,"episode_id":"ep_abc","action":{"action_type":"click","payload":{"selector":"#button"}}}
json
{"type":"env/step/resp","id":"es2","version":"1.0","ts":1716123458.500,"req_id":"es1","obs":{"episode_id":"ep_abc","step_num":1,"state":{"url":"https://example.com/result","title":"Result"},"done":false,"truncated":false,"reward":1.0}}

State Push (Server-initiated) ​

json
{"type":"env/state/push","id":"sp1","version":"1.0","ts":1716123459.100,"episode_id":"ep_abc","step_id":1,"action_id":"act_1","event_type":"tool_result","reason":"proc_exit","delta":{"stdout":"done"},"reward":0.5,"terminal":false}

Data Plane (Metadata Only) ​

json
{"type":"env/data/add/req","id":"da1","version":"1.0","ts":1716123460.100,"data_root":"/workspace/data","parts":[{"src":"/host/tasks/task-B/orders.csv","dest":"orders.csv","uri":"orders","checksum":"sha256:9f2a...","mime":"text/csv"}]}
json
{"type":"env/data/add/resp","id":"da2","version":"1.0","ts":1716123460.400,"req_id":"da1","ok":true,"added":1,"errors":[],"staged":[{"uri":"orders","dest":"orders.csv","size_bytes":18432}]}
json
{"type":"env/data/get/req","id":"dg1","version":"1.0","ts":1716123470.100,"query":"*.csv","include_content":false,"max_bytes":1048576,"data_root":"/workspace/data"}
json
{"type":"env/data/get/resp","id":"dg2","version":"1.0","ts":1716123470.300,"req_id":"dg1","ok":true,"items":[{"uri":"orders.csv","dest":"orders.csv","size_bytes":18432,"checksum":"sha256:9f2a..."}],"truncated":false}

Security Considerations ​

  1. Episode isolation: Episodes must be scoped to prevent cross-session leakage
  2. Action validation: Actions must conform to action_space schema
  3. State push gating: Only emitted if agent negotiated env_push capability
  4. Batch limits: Host should enforce maximum batch_step size
  5. Reward signal integrity: Reward values must not be tamperable by the agent
  6. Data-root confinement: data_root/dest/src are host-interpreted paths — the host must confine them to the intended data root and reject traversal (..), never copying from or to arbitrary host paths
  7. No bytes on the wire: env/data/* payloads are metadata only; the host performs file I/O locally. The data_get include_content path is the sole exception and must enforce max_bytes

A2E Protocol v1.0 — Released under the MIT License.