Time as a Control Dimension in Robot Learning

Yinsen Jia, Boyuan Chen — Duke University

Paper · Video · Project website

This repository contains the implementation and paper artifacts for Time-Aware Policy Learning. It includes three Isaac Gym tasks, the PPO-family training code, N-P3O/PPO-Lagrangian/CPO variants, strict evaluation utilities, paper checkpoints, and real-robot interfaces.

Requirements

• Ubuntu 20.04 or 22.04
• NVIDIA GPU and a compatible driver
• Python 3.8
• At least 16 GB GPU memory for the default 16,384-environment training setup

Isaac Gym Preview 4 is bundled under the isaacgym directory. Consult isaacgym/docs/index.html for NVIDIA's runtime and driver requirements.

Installation

git clone https://github.com/generalroboticslab/TimeAwarePolicy.git
cd TimeAwarePolicy

conda create -n timeaware python=3.8
conda activate timeaware
pip install -r requirements.txt --no-cache-dir
pip install -e isaacgym/python

Verify the Python-side installation without starting a simulator:

python scripts/check_install.py
python scripts/verify_release_artifacts.py

Run a paper checkpoint

Rendering uses a Vulkan device ordinal, which can differ from the CUDA index. Find the correct ordinal with:

vulkaninfo --summary

Pass that ordinal through --graphics_device_id.

Cube stacking:

python tw_evaluation.py \
  --rendering --graphics_device_id 0 --num_envs 1 \
  --checkpoint 20250717_162724_tw_FrankaCubeStack \
  --index_episode best_rew --goal_speed 0.6 \
  --keyboard_ctrl --draw_scevel

Granular-media pouring:

python tw_evaluation.py \
  --rendering --graphics_device_id 0 --num_envs 1 \
  --checkpoint 20250715_123940_tw_FrankaGmPour \
  --index_episode best_rew --goal_speed 0.6 \
  --keyboard_ctrl --draw_scevel

Drawer opening:

python tw_evaluation.py \
  --rendering --graphics_device_id 0 --num_envs 1 \
  --checkpoint 20250730_151924_tw_FrankaCabinet \
  --index_episode best_rew --goal_speed 0.6 \
  --keyboard_ctrl --draw_scevel

Use the up/down arrow keys to increase/decrease the time ratio by 0.1.

Train

Training has four executable stages:

1. learn a task policy;
2. fine-tune a time-optimal policy;
3. distill it into a policy with temporal inputs;
4. optimize the time-aware CMDP objective.

See docs/training.md for copy-paste commands and the task-specific horizons.

Validation experiments

The following optional studies are separate from the four-stage training procedure used to produce a time-aware policy. Task-specific settings remain in the saved run configurations and training guide. Set the shell variables shown in each command before running it.

Initializer-quality sensitivity

Train and strictly calibrate one common initializer bank per task, then reuse its four selected checkpoints for every fine-tuning seed. Start from the initial-policy command in the training guide and enable candidate capture:

python tw_training.py \
  --saving --task_name "$TASK" \
  --num_updates "$INITIAL_UPDATES" --episodeLength "$INITIAL_HORIZON" \
  --num_envs 16384 --num-steps 32 \
  --fix_priv --gamma 0.995 \
  --successRewardScale 1000 \
  --quality_candidate_interval 5 \
  --quality_candidate_start_success 0.90

python scripts/calibrate_quality_candidates.py \
  --task_name "$TASK" \
  --producer "$INITIAL_TRAINING_FOLDER" \
  --output_bank "$CALIBRATED_BANK" \
  --num_envs 2000 --execute

The calibration records the measured held-out success of each nominal Q40/Q60/Q80/Q95 checkpoint. Launch matched time-optimal refinements by setting QUALITY and SEED for each run:

python tw_training.py \
  --saving --task_name "$TASK" \
  --checkpoint "$CALIBRATED_BANK" --index_episode "quality_$QUALITY" \
  --num_updates "$TIME_OPTIMAL_UPDATES" --episodeLength "$INITIAL_HORIZON" \
  --num_envs 16384 --num-steps 32 --seed "$SEED" \
  --fix_priv --reset_critic --warmup_iters 50 --no_dense \
  --gamma 0.995 --target_kl 2.5 \
  --successRewardScale 1000 \
  --epstimeRewardScale "[100, 100]"

CMDP solver comparison

Assume the temporal-input student, its fixed 1,000-configuration reference bank, and the N-P3O validation baseline have already been produced. Hold those inputs and all training settings fixed. Set TASK, TEMPORAL_STUDENT, TIME_AWARE_UPDATES, TIME_AWARE_HORIZON, METHOD, and SEED before running:

python tw_training.py \
  --saving --task_name "$TASK" \
  --checkpoint "$TEMPORAL_STUDENT" --index_episode best \
  --num_updates "$TIME_AWARE_UPDATES" --episodeLength "$TIME_AWARE_HORIZON" \
  --num_envs 16384 --num-steps 32 --seed "$SEED" \
  --reset_critic --warmup_iters 50 --no_dense \
  --time2end --time_ratio --ratio_range "[0.2, 1]" \
  --fixed_configs --use_cost --cmdp_method "$METHOD" \
  --lr 2e-4 --gamma 1.0 --target_kl 2.5 \
  --c_gamma "[1, 0.99]" --c_scale "[0, 1]" \
  --scevelRewardScale "[0, 0]" \
  --successRewardScale 1000 \
  --epstimeRewardScale "[100, 100]"

See docs/training.md for candidate-selection details, the main stage-wise recipe, and solver-specific defaults.

Evaluate and plot

Strict evaluation reports task success and metrics conditioned on successful episodes. Detailed evaluation protocols are documented separately.

See:

• docs/evaluation.md for strict evaluation commands and metric definitions;
• docs/checkpoints.md for included artifacts and their provenance;
• docs/reproducibility.md for determinism, randomization, and compatibility notes; and
• docs/results.md for versioned result profiles, preflight, and available evaluation figures.

Generate individual, seed-aggregate, and overview curves from a campaign status/log snapshot with:

python scripts/plot_training_curves.py \
  --campaign-dir /path/to/campaign

The available final-result launchers, validators, and figure builders are collected under scripts/paper/. Their JSON configurations are stored under scripts/paper/configs/; executable Python code does not rely on machine-specific campaign names.

Training writes to train_res/TASK_NAME and evaluation writes to eval_res/TASK_NAME unless the corresponding directory option is overridden.

Tests

The unit suite runs without a GPU:

CUDA_VISIBLE_DEVICES="" python -m pytest -q

Full simulator smoke tests require Isaac Gym and an NVIDIA GPU.

Run one environment rollout and optimizer update for every public task, while keeping logs and the validation manifest outside the repository:

python scripts/smoke_tasks.py \
  --gpu 0 --output-dir /tmp/timeaware-task-smokes

Repository layout

.
├── envs/                    # Simulation environments and assets
│   ├── assets/
│   └── isaacgymenvs/
├── model/                   # Policy architecture and normalization
├── training/                # PPO/CMDP updates and training lifecycle
├── evaluation/              # Shared and paired-stage evaluation logic
├── scripts/                 # Calibration, validation, and plotting tools
│   ├── paper/               # Paper evaluation and figure tools
│   │   └── configs/         # Paper-workflow JSON configurations
│   └── scene_visualization.py
├── train_res/               # Bundled base and time-aware policies
│   ├── FrankaCabinet/
│   ├── FrankaCubeStack/
│   └── FrankaGmPour/
├── eval_res/                # Bundled temporal calibration banks
│   ├── FrankaCabinet/
│   ├── FrankaCubeStack/
│   └── FrankaGmPour/
├── real_robot/              # Camera, estimation, and robot communication
├── isaacgym/                # NVIDIA Isaac Gym Preview 4
├── docs/                    # Training, evaluation, and deployment guides
├── tests/                   # CPU unit and structural tests
├── tw_training.py           # Training entrypoint
├── tw_training_utils.py     # Training arguments and utilities
├── tw_evaluation.py         # Evaluation entrypoint
├── tw_evaluation_utils.py   # Evaluation arguments and utilities
├── plot_utils.py            # Shared plotting utilities
├── runtime_utils.py         # Runtime and checkpoint helpers
├── tf_utils.py              # Transformation utilities
├── utils.py                 # General I/O utilities
├── requirements.txt         # Simulation dependencies
└── requirements-real.txt    # Additional real-robot dependencies

scripts/paper/configs/bundled_files.json verifies only files shipped with this repository. The other configurations are used only by the supplied paper-result tools. New checkpoints do not need to be added there.

Real robot

Real deployment additionally requires the Franka controller stack, RealSense drivers, camera extrinsic calibration, and optional hardware dependencies:

pip install -r requirements-real.txt --no-cache-dir
python scripts/check_install.py --real-robot

The controller address is deliberately explicit; the public code does not embed a lab-specific host. The entrypoint is:

python tw_evaluation.py \
  --num_envs 1 --real_robot \
  --controller_ip CONTROLLER_HOST \
  --checkpoint CHECKPOINT --index_episode best_rew \
  --goal_time 10

Hardware operation can cause injury or damage. Validate workspace limits, emergency-stop behavior, calibration, and controller gains before enabling motion. See docs/real_robot.md for the controller protocol, calibration layout, dry-run procedure, and recording options.

License

The repository is released under CC BY-NC-ND 4.0. Duke University has filed patent rights associated with this work. For additional rights, contact Duke's Office for Translation and Commercialization and reference OTC DU9041PROV.

Bundled Isaac Gym code, native libraries, and third-party assets retain their own terms; see THIRD_PARTY_NOTICES.md.

NVIDIA Isaac Gym Preview 4 is proprietary software. Its original license is retained verbatim at isaacgym/python/LICENSE.txt; use, modification, and redistribution of isaacgym/ are governed by those terms rather than the repository-level license.

Citation

@misc{jia2025timecontrol,
  title={Time as a Control Dimension in Robot Learning},
  author={Yinsen Jia and Boyuan Chen},
  year={2025},
  eprint={2511.07654},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2511.07654}
}