Training a DisplayPort Cable Insertion Policy and ROS Deployment

Training a DisplayPort Cable Insertion Policy and ROS Deployment#

This tutorial walks you through how to train a DisplayPort plug insertion reinforcement learning (RL) policy that transfers from simulation to a real Flexiv robot. The workflow consists of two main stages:

  1. Simulation Training in Isaac Lab: Train the policy in a high-fidelity physics simulation with domain randomization

  2. LEAPP Export and Real Robot Deployment: Export the trained policy with LEAPP, then deploy on hardware with Isaac ROS

This walkthrough covers the key principles and best practices for sim-to-real transfer using Isaac Lab.

Supported Robot:

  • Flexiv Rizon 4s: 7-DOF collaborative robot arm with Grav parallel gripper

../_images/rizon_4s.png

Flexiv Rizon 4s with the Grav gripper used for the DisplayPort insertion task.#

This environment has been successfully deployed and tested on a real Flexiv Rizon 4s robot without an IsaacLab dependency.

Task Details:

The DisplayPort insertion policy operates as follows:

  1. Initial State: The policy assumes the DisplayPort plug is already grasped by the gripper at the start of the episode

  2. Input Observations: The policy receives the current state of the robot together with the pose of the socket insertion point (position and orientation) from a separate perception pipeline. The robot state is the arm joint angles in the joint-space variants, or the flange / end-effector pose in the task-space variants

  3. Policy Output: The policy outputs an incremental command each step — delta joint positions (incremental changes to arm joint angles) in the joint-space variants, or a 6-DoF Cartesian end-effector pose delta in the task-space variants

  4. Task Goal: Insert the DisplayPort plug into a fixed socket until the mate point aligns within the success threshold

../_images/dp_insertion_validation.gif

DisplayPort insertion overview from the training course material.#

Both control spaces are supported end-to-end. Task space is the recommended route for real-robot deployment; see Choosing a Control Space for the trade-offs and for the extra robot-calibration step joint space requires.

Scope of This Tutorial:

This tutorial covers training and LEAPP export in Isaac Lab. For the complete on-robot workflow (vision pipeline, robot interface, ROS inference), refer to the Isaac ROS Documentation after exporting your policy.

Code Layout:

The task is packaged under the standalone isaaclab_training namespace:

  • src/isaaclab_training/tasks/displayport_insertion/displayport_insertion_env_cfg.py — shared task MDP (scene, assets, observations, rewards)

  • src/isaaclab_training/tasks/displayport_insertion/insertion_env.py — environment class that logs insertion success metrics during training

  • src/isaaclab_training/tasks/displayport_insertion/config/displayport_rizon_4s/ — Flexiv Rizon 4s + Grav robot-specific overrides and gym registrations

  • src/isaaclab_training/tasks/displayport_insertion/config/displayport_rizon_4s/joint_pos_env_cfg.py — joint-space (relative joint position) environment

  • src/isaaclab_training/tasks/displayport_insertion/config/displayport_rizon_4s/task_space_env_cfg.py — task-space (operational space control) environment

  • src/isaaclab_training/mdp/ — reusable observation, action, reward, event, noise, and termination helpers

  • uv run isaaclab train --rl_library rsl_rl — Isaac Lab’s packaged unified trainer

  • src/isaaclab_training/cli/play_policy_io.py / uv run isaaclab-training-play-policy-io — DisplayPort-specific inference, LEAPP validation, and CSV logging

  • src/isaaclab_training/cli/export_policy.py / uv run --extra leapp isaaclab-training-export — packaged LEAPP exporter

  • src/isaaclab_training/export/contracts.py — task-specific LEAPP I/O contracts such as displayport_task_space

Overview#

Successful sim-to-real transfer requires addressing three fundamental aspects:

  1. Input Consistency: Ensuring the observations your policy receives in simulation match those available on the real robot

  2. System Response Consistency: Ensuring the robot and environment respond to actions in simulation the same way they do in reality

  3. Output Consistency: Ensuring any post-processing applied to policy outputs in Isaac Lab is also applied during real-world inference

When all three aspects are properly addressed, policies trained purely in simulation can achieve robust performance on real hardware without any real-world training data.

../_images/rl_loop.png

Reinforcement-learning loop used during training.#

Debugging Tip: When your policy fails on the real robot, set up the real robot with the same initial observations as in simulation, then compare how the controller responds. This isolates whether the problem is from observation mismatch (Input Consistency) or physics/controller mismatch (System Response Consistency).

Asset Quality for Insertion Tasks#

For any contact-rich insertion task, the quality of the plug and socket assets matters more than most other sim-to-real knobs. DisplayPort insertion in particular operates at very small clearances between plug blades and the socket cavity. If the USD collision geometry, mass properties, or joint behavior are wrong, no amount of reward tuning or domain randomization will produce a policy that transfers well to hardware.

The DisplayPort plug and socket assets used by these environments are published on the Isaac asset server (under Props/Factory/display_port_cable_assets) and fetched at load time rather than vendored in the repository. They have been iterated extensively and work well for training policies that transfer sim-to-real. Expect considerable upfront effort to reach this quality for a new connector or cable type.

What to validate before training:

  1. Static insertion pose stability: Load the plug fully inserted into the socket at the goal pose (no robot, no gripper). The plug should remain seated without drifting, jittering, or being ejected by contact forces. Persistent separation or slow creep at the mated pose usually indicates incorrect collision meshes, rest offsets, or mass/inertia.

  2. Collision fidelity at clearance scale: Blade-to-cavity gaps are sub-millimeter. Convex hulls or coarse meshes often produce false contacts, snagging, or penetration. SDF or carefully authored triangle meshes with tuned contact_offset / rest_offset are typically required.

  3. Engagement behavior: Push the plug through the approach path by hand (or with scripted motion) and confirm contact feels plausible — no explosive pops, no tunneling through the socket wall, no sticky high-friction jamming unless that matches the real connector.

  4. Grasped plug stability: With the gripper closed at the training grasp width, the plug should not spin or slip unrealistically when the arm moves. Cable mass and plug COM should be representative of the real assembly.

  5. Mate-point alignment: Verify SOCKET_INSERTION_OFFSET, PLUG_INSERTION_OFFSET, and PLUG_GOAL_ROT in displayport_insertion_env_cfg.py match the intended physical mate frame. Reward and success metrics are computed from these offsets; a mismatch here looks like a perception error on the real robot.

The robot asset matters too. The checks above concern the plug and socket, but the arm’s USD must also represent your robot — particularly its kinematic parameters, which may vary unit to unit. This matters most for joint-space training, where the policy commands joints directly. These environments spawn the stock FLEXIV_RIZON4S_GRAV_GRIPPER_CFG asset, which describes a nominal Rizon 4s rather than any particular unit. Flexiv’s flexiv_calibration workflow exports a calibrated robot description for a specific arm; convert the result to USD and point the robot spawn config at it before training a policy you intend to deploy.

Practical workflow:

  1. Fix assets in isolation (drop-test or basic play env with fixed poses) before running full RL training.

  2. Compare sim behavior to real hardware video at the same poses — look for drift, bounce, and penetration, not policy success rate.

  3. Only after assets pass these checks, tune curriculum, rewards, and domain randomization.

Note

Poor asset quality often shows up as policies that learn high training success but fail on hardware with inconsistent contact behavior, or as training that never achieves high Metrics/success_rate despite reward tuning. Fix the assets first.

Part 1: Input Consistency#

The observations your policy receives must be consistent between simulation and reality. This means:

  1. The observation space should only include information available from real sensors

  2. Sensor noise and delays should be modeled appropriately

Using Real-Robot-Available Observations#

Your simulation environment should only use observations that are available on the real robot and not use “privileged” information that would not be available in deployment. The critic receives additional privileged observations (plug pose and joint velocities) to improve value estimation during training, but these are not passed to the actor at deployment time.

Observation Specification#

The DisplayPort insertion environment uses proprioceptive and exteroceptive (vision) observations. In every variant the actor sees the current state of the robot plus the socket insertion (mate) point supplied by perception. What differs between the two control spaces is how the robot state is represented: joint angles for the joint-space environments, and the flange / end-effector pose for the task-space environments.

../_images/dp_policy_inputs_outputs.png

Task-space policy inputs and outputs: end-effector pose, socket pose, and a relative Cartesian command.#

Joint-Space Observations (Flexiv Rizon 4s)#

Observation

Dim

Real-World Source

Noise

joint_pos (arm only)

7

Robot controller

None

joint_vel (arm only, optional)

7

Robot controller

None

socket_pos (insertion mate point)

3

Perception pipeline

±10mm

socket_quat

4

Perception pipeline

None

Recommended shipping configuration (NoJointVel variants): 14 policy dimensions (7 joint positions + 3 socket position + 4 socket quaternion).

Training configuration with joint velocity (Grav variants without NoJointVel): 21 policy dimensions.

Note

Within joint space, prefer the NoJointVel variants. Policies trained with joint_vel in the actor observation can achieve slightly higher success rates in simulation, but we consistently observe less stable behavior on the real Flexiv robot (jittery motions, inconsistent contact during insertion).

The NoJointVel configs remove joint_vel from the actor observation while keeping it in the critic observation group. This matches deployment setups where joint velocity is not exposed to the policy network but can still help the value function during training.

This is a choice among the joint-space variants; the task-space environments are recommended over joint space generally (see Choosing a Control Space), and they do not expose joint velocity to the actor at all.

Task-Space Observations (Flexiv Rizon 4s)#

Observation

Dim

Real-World Source

Noise

eef_pos (TCP position)

3

Robot controller

None

eef_rot_6d (TCP orientation)

6

Robot controller

None

socket_kp_pos (insertion mate point)

3

Perception pipeline

None by default

socket_kp_rot_6d

6

Perception pipeline

None by default

Shipping configuration (TaskSpace variants): 18 policy dimensions (3 + 6 end-effector pose + 3 + 6 socket keypoint frame).

Orientations use a 6D rotation representation (the first two rows of the rotation matrix) rather than a quaternion. This is continuous over SO(3), which avoids the sign-flip discontinuity a quaternion introduces and is easier for the network to regress against.

The end-effector position is reported at the tool center point (TCP), i.e. the flange body offset along its local +Z by _TCP_OFFSET in task_space_env_cfg.py. Note that the observation is taken at the TCP while the action is applied at the flange; see Action Space Design.

Implementation (base class):

@configclass
class PolicyCfg(ObsGroup):
    """Observations for policy group."""

    joint_pos = ObsTerm(
        func=mdp.joint_pos,
        params={"asset_cfg": SceneEntityCfg("robot", joint_names=[".*"])},
    )
    joint_vel = ObsTerm(
        func=mdp.joint_vel,
        params={"asset_cfg": SceneEntityCfg("robot", joint_names=[".*"])},
    )
    socket_pos = ObsTerm(
        func=mdp.rigid_object_pos_w,
        params={"asset_cfg": SceneEntityCfg("dp_socket"), "offset": SOCKET_INSERTION_OFFSET},
        noise=ResetSampledConstantNoiseModelCfg(
            noise_cfg=UniformNoiseCfg(n_min=-0.01, n_max=0.01, operation="add")  # ±10mm
        ),
    )
    socket_quat = ObsTerm(
        func=mdp.rigid_object_quat_w,
        params={"asset_cfg": SceneEntityCfg("dp_socket")},
    )

    def __post_init__(self):
        self.enable_corruption = True
        self.concatenate_terms = True

Rizon 4s overrides (in config/displayport_rizon_4s/joint_pos_env_cfg.py):

# Arm joints only — gripper joints are excluded from observations
self.observations.policy.joint_pos.params["asset_cfg"].joint_names = [
    "joint1", "joint2", "joint3", "joint4", "joint5", "joint6", "joint7",
]

Task-space actor group (in config/displayport_rizon_4s/task_space_env_cfg.py):

@configclass
class PolicyCfg(ObsGroup):
    """Actor observations: EEF pose + socket keypoint frame (18 dims)."""

    eef_pos = ObsTerm(
        func=mdp.eef_pos_w,
        params={"asset_cfg": SceneEntityCfg("robot"), "body_name": "flange", "offset": _TCP_OFFSET},
    )
    eef_rot_6d = ObsTerm(
        func=mdp.eef_rot_6d_w,
        params={"asset_cfg": SceneEntityCfg("robot"), "body_name": "flange"},
    )
    socket_kp_pos = ObsTerm(
        func=mdp.rigid_object_pos_w,
        params={"asset_cfg": SceneEntityCfg("dp_socket"), "offset": SOCKET_INSERTION_OFFSET},
    )
    socket_kp_rot_6d = ObsTerm(
        func=mdp.rigid_object_rot_6d_w,
        params={"asset_cfg": SceneEntityCfg("dp_socket")},
    )

In both control spaces the critic additionally receives privileged state that the actor never sees: joint positions and velocities plus the plug keypoint frame (task space) or the plug pose (joint space).

Why No Noise for Proprioceptive Observations?

As with the gear assembly task, policies trained without noise on proprioceptive observations transfer well to the Flexiv Rizon 4s. The controller provides sufficiently accurate joint state feedback that modeling sensor noise on joint states does not improve sim-to-real transfer for this task. The same reasoning carries over to the task-space variants: the end-effector pose is derived from those same joint encoders through forward kinematics, so it inherits their accuracy and is likewise left noise-free.

The socket pose, by contrast, comes from perception and is the observation worth corrupting. The joint-space environments apply ±10 mm of reset-sampled uniform noise to socket_pos by default. The task-space environments ship with observation noise disabled so the contract matches the deployed perception stack exactly; add it through the observation term’s noise field if your perception pipeline is less accurate.

Part 2: System Response Consistency#

Once your observations are consistent, ensure the simulated robot and environment respond to actions the same way the real system does. For DisplayPort insertion this involves:

  1. Physics simulation parameters (friction, contact properties, plug/socket collision meshes)

  2. Actuator modeling (PD controller gains, effort limits)

  3. Domain randomization and curriculum

Physics Parameter Tuning#

Accurate physics simulation is critical for contact-rich insertion. The DisplayPort plug and socket use SDF collision meshes with high solver iteration counts on the rigid bodies:

# From displayport_insertion_env_cfg.py — DisplayPortPlug / DisplayPortSocket
rigid_props=sim_utils.RigidBodyPropertiesCfg(
    solver_position_iteration_count=128,
    solver_velocity_iteration_count=1,
    max_depenetration_velocity=0.5,  # plug; socket uses 5.0
),
collision_props=sim_utils.CollisionPropertiesCfg(
    contact_offset=0.00001,   # plug
    rest_offset=-0.00005,
),

The Flexiv Rizon 4s arm uses lower solver iteration counts for performance, matching the gear assembly Flexiv configuration:

# From config/displayport_rizon_4s/joint_pos_env_cfg.py
rigid_props=sim_utils.RigidBodyPropertiesCfg(
    disable_gravity=True,
    max_depenetration_velocity=5.0,
    solver_position_iteration_count=4,
    solver_velocity_iteration_count=1,
    max_contact_impulse=1e32,
),
collision_props=sim_utils.CollisionPropertiesCfg(
    contact_offset=0.005,
    rest_offset=0.0,
),

Friction randomization (in config/displayport_rizon_4s/joint_pos_env_cfg.py):

plug_physics_material = EventTerm(
    func=mdp.randomize_rigid_body_material,
    mode="startup",
    params={
        "asset_cfg": SceneEntityCfg("dp_plug", body_names=".*"),
        "static_friction_range": (0.001, 0.001),
        "dynamic_friction_range": (0.001, 0.001),
    },
)

robot_physics_material = EventTerm(
    func=mdp.randomize_rigid_body_material,
    mode="startup",
    params={
        "asset_cfg": SceneEntityCfg("robot", body_names=".*finger.*"),
        "static_friction_range": (0.75, 0.75),
        "dynamic_friction_range": (0.75, 0.75),
    },
)

Low plug/socket friction reduces sticking during blade engagement. Gripper finger friction is set to match real grasp behavior.

Actuator Modeling#

The Rizon 4s uses ImplicitActuatorCfg with per-joint-group arm tuning from FLEXIV_RIZON4S_GRAV_GRIPPER_CFG, plus dedicated Grav gripper actuators:

# Grav gripper actuator configuration
self.scene.robot.actuators["gripper_drive"] = ImplicitActuatorCfg(
    joint_names_expr=["finger_joint"],
    effort_limit_sim=2.0,
    velocity_limit_sim=1.0,
    stiffness=2e3,
    damping=1e1,
)
self.scene.robot.actuators["gripper_passive"] = ImplicitActuatorCfg(
    joint_names_expr=[".*_knuckle_joint"],
    effort_limit_sim=1.0,
    velocity_limit_sim=1.0,
    stiffness=0.0,
    damping=0.0,
)

Note

Flexiv Rizon 4s: Domain randomization for actuator gains and joint friction is not included in the Rizon 4s EventCfg. The real-world Flexiv controller is stable and precise enough that the simulation policy transfers without these additional randomizations, consistent with the gear assembly Flexiv setup.

Action Space Design#

In both control spaces the policy commands only the arm; the gripper is not in the action space — the plug is held at a fixed grasp width for the episode. The action is always an incremental (delta) command, never an absolute target. What differs is the space that delta lives in.

The policy controls the 7 arm joints using incremental joint position control.

self.joint_action_scale = 0.025  # ±1.4 degrees per step

self.actions.arm_action = mdp.RelativeJointPositionActionCfg(
    asset_name="robot",
    joint_names=["joint1", "joint2", "joint3", "joint4",
                 "joint5", "joint6", "joint7"],
    scale=self.joint_action_scale,
    use_zero_offset=True,
)

Action dimension: 7 — one delta per arm joint, in radians.

The policy emits a 6-DoF Cartesian pose delta which an Operational Space Controller (OSC) converts into joint efforts. This removes joint-level system identification from the sim-to-real path: the same policy can drive the real robot through a task-space bridge.

_ACTION_SCALE = 0.025          # 25 mm and 0.025 rad per step
_STIFFNESS = (300.0, 300.0, 300.0, 30.0, 30.0, 30.0)

self.actions.arm_action = mdp.DeployOperationalSpaceControllerActionCfg(
    asset_name="robot",
    joint_names=_ARM_JOINTS,
    body_name="flange",
    controller_cfg=OperationalSpaceControllerCfg(
        target_types=["pose_rel"],
        motion_stiffness_task=_STIFFNESS,
        ...
    ),
    position_scale=_ACTION_SCALE,
    orientation_scale=_ACTION_SCALE,
)

Action dimension: 6 — [dx, dy, dz, dθx, dθy, dθz], the first three in metres and the last three in radians.

Two details matter for deployment:

  • The arm’s joint PD gains are zeroed (actuators[...].stiffness = 0.0); all compliance comes from the task-space stiffness above, so the controller — not the joint servo — sets the contact behavior.

  • The action is applied at the flange, while eef_pos is observed at the TCP. Observation and control therefore use different frames by design; the real-robot bridge must reproduce that split.

Action scale: 0.025 in both cases — read as radians per joint per step in joint space, and as metres / radians per step in task space.

../_images/dp_action_pipeline.png

Task-space action pipeline: scaled pose delta, task-space controller, joint targets, and robot response.#

Control frequency: sim.dt = 1/240 s with decimation = 8 → 30 Hz policy rate, identical in both control spaces.

Domain Randomization Strategy#

Socket pose randomization perturbs the fixed socket to cover perception and mounting variation:

randomize_socket_pose = EventTerm(
    func=mdp.reset_root_state_uniform,
    mode="reset",
    params={
        "pose_range": {
            "x": [-0.01, 0.01],                        # ±1 cm
            "y": [-0.01, 0.01],                        # ±1 cm
            "z": [-0.02, 0.02],                        # ±2 cm
            "roll": [-math.radians(2.0), math.radians(2.0)],
            "pitch": [-math.radians(2.0), math.radians(2.0)],
            "yaw": [-math.radians(2.0), math.radians(2.0)],
        },
        "asset_cfg": SceneEntityCfg("dp_socket"),
    },
)

Plug reset curriculum starts episodes with the plug near the goal pose and anneals toward farther approach poses:

reset_plug_curriculum = EventTerm(
    func=mdp.reset_plug_at_goal_curriculum,
    mode="reset",
    params={
        "at_goal_prob": 0.8,
        "at_goal_prob_final": 0.0,
        "anneal_start_iter": 0.0,
        "anneal_end_iter": 500.0,
        "num_steps_per_env": 512,
        "insertion_axis": [1.0, 0.0, 0.0],
        "at_goal_depth_range": [0.0, 0.015],      # 0–15 mm engaged
        "approach_depth_range": [0.02, 0.06],     # 20–60 mm approach
        "normal_pose_range": {
            "x": [-0.02, 0.02],
            "y": [-0.02, 0.02],
            "z": [0.0, 0.0],
        },
    },
)

At the start of training, 80% of resets place the plug near the inserted pose; this probability linearly anneals to 0% over 500 training iterations, forcing the policy to learn full approach and insertion.

../_images/dp_curriculum.png

Reset curriculum: early episodes often begin near insertion, then anneal toward full approach starts.#

Initial robot pose is set via inverse kinematics to a grasp pose on the plug at each reset. grasp_offset / end_effector_body_name / num_arm_joints are applied in Rizon4sGravDisplayportInsertionEnvCfg.__post_init__:

set_robot_to_grasp_pose = EventTerm(
    func=mdp.set_robot_to_object_grasp_pose,
    mode="reset",
    params={
        "target_object_name": "dp_plug",
        "grasp_offset": [0.0025, 0.0, -0.1875],  # plug local frame [m]
        "end_effector_body_name": "flange",
        "num_arm_joints": 7,
    },
)

Reward Shaping#

The environment uses keypoint-based rewards that measure alignment between the plug and socket insertion mate points. Reward terms are defined in displayport_insertion_env_cfg.py:

  • Keypoint tracking (plug_socket_keypoint_tracking): Penalizes L2 keypoint distance between plug and socket mate frames

  • Exponential keypoint tracking (plug_socket_keypoint_tracking_exp): Dense exponential reward for fine alignment

  • Action rate (action_rate_l2): Penalizes large action changes for smooth motions

../_images/rewards.png

Keypoint-pair reward intuition. Corresponding plug/socket keypoints collapse only when position and orientation agree.#

The Rizon 4s config sets the linear and exponential keypoint reward weights to a 1:1 ratio:

self.rewards.plug_socket_keypoint_tracking_exp.weight = abs(
    self.rewards.plug_socket_keypoint_tracking.weight
)

Terminations#

In addition to the episode timeout, the Rizon 4s config terminates early when:

  • Plug dropped: End-effector moves more than 15 cm away from the plug grasp point

  • Plug orientation exceeded: Roll or pitch deviation exceeds 15° relative to the grasp frame

Training Metrics#

Unlike gear assembly, this task uses a custom environment class (DisplayportInsertionEnv) to log insertion metrics to TensorBoard without changing the MDP:

  • Metrics/success_rate — fraction of environments within the 3 mm mate-point threshold

  • Metrics/plug_socket_pos_error_m — mean mate-point distance

  • Metrics/plug_socket_keypoint_dist_m — mean keypoint distance

  • Metrics/terminal_success_rate — success rate at episode reset

../_images/dp_tb_metrics.png

Task-level TensorBoard metrics: success rate, terminal success rate, mate-point distance, and keypoint distance.#

Tuning Hyperparameters for Better Performance#

After asset quality and physics look correct, the following hyperparameters are the main levers for improving training speed, final success rate, and sim-to-real robustness. Defaults below are the shipped Flexiv Rizon 4s values; adjust one group at a time and monitor Metrics/terminal_success_rate in TensorBoard.

Reward Weights#

Defined in displayport_insertion_env_cfg.py; the Rizon 4s config overrides the exponential weight in joint_pos_env_cfg.py.

Reward hyperparameters#

Parameter

Default

Effect

plug_socket_keypoint_tracking.weight

-1.5

Linear penalty on keypoint distance. More negative → stronger pull toward alignment.

plug_socket_keypoint_tracking_exp.weight

1.5 (matched to linear)

Exponential bonus near the goal. Increase relative to linear for sharper fine-insertion behavior; decrease if policy is brittle or stalls short of full insertion.

kp_exp_coeffs

[(50, 0.0001), (300, 0.0001), (600, 0.0001), (2000, 0.0001)]

Per-keypoint exponential scales. Higher first values tighten the reward basin around the goal.

keypoint_scale

0.15

Spatial extent of keypoint offsets. Affects how rotation errors contribute relative to translation.

action_rate.weight

-5e-6

Smoothness penalty. More negative → slower, smoother motions; too strong can prevent final insertion force.

1:1 linear-to-exponential weighting (current shipping default):

self.rewards.plug_socket_keypoint_tracking_exp.weight = abs(
    self.rewards.plug_socket_keypoint_tracking.weight
)

If the policy approaches but does not fully seat the plug, try increasing the exponential weight or tightening kp_exp_coeffs. If it rushes and bounces off the socket, increase action_rate magnitude or reduce the exponential weight.

../_images/dp_tb_episode_reward.png

Reward terms over training for the DisplayPort policy.#

Reset Curriculum#

Defined in config/displayport_rizon_4s/joint_pos_env_cfg.py → reset_plug_curriculum.

Curriculum hyperparameters#

Parameter

Default

Effect

at_goal_prob / at_goal_prob_final

0.8 → 0.0

Fraction of resets with plug near full insertion. Higher start values make early learning easier; anneal to zero for full approach behavior.

anneal_end_iter

500

Training iterations over which at-goal probability anneals. Extend (e.g. 800–1000) if success rate drops when curriculum gets harder; shorten if training is too slow to reach approach poses.

at_goal_depth_range

[0.0, 0.015] m

How deep the plug starts when sampled “at goal” (0–15 mm engaged). Narrow for fine final-insertion practice; widen slightly if the policy never sees near-mated contacts.

approach_depth_range

[0.02, 0.06] m

Standoff distance when not at goal (20–60 mm). Increase upper bound for harder long-range approach; decrease if the policy struggles to reach the socket mouth.

normal_pose_range

±2 cm lateral

Lateral misalignment when not at goal. Widen for more robustness to perception error; narrow if training fails to converge.

If Metrics/success_rate is high early but collapses after iteration ~500, the curriculum may be annealing too aggressively — extend anneal_end_iter or raise at_goal_prob_final temporarily.

../_images/dp_tb_episode_termination.png

Episode termination causes over training.#

Domain Randomization and Observations#

Randomization hyperparameters#

Parameter

Default

Effect

randomize_socket_pose ranges

±1 cm XY, ±2 cm Z, ±2°

Socket pose DR. Widen to match real perception/mount error; narrow if the policy cannot learn a baseline insertion.

socket_pos observation noise

±10 mm

Perception noise on mate point. Increase for more robust real-world pose error; decrease if sim policy is too conservative.

Plug/socket friction (startup)

0.001

Low friction reduces unrealistic jamming. Tune only after visual sim-vs-real comparison — wrong friction can dominate insertion feel.

Gripper finger friction

0.75

Affects grasp stability during insertion forces.

Actions and Grasp#

Action / grasp hyperparameters#

Parameter

Default

Effect

joint_action_scale

0.025

Max joint delta per step (~±1.4°). Increase if real robot stiction prevents reaching targets; decrease for finer final alignment.

grasp_offset

[0.0025, 0.0, -0.1875] m

EE-to-plug transform for IK reset. Wrong values cause dropped-plug terminations or misaligned approach.

hand_hold_width / hand_close_width

-0.05 / -0.155 rad

Grav finger_joint grasp command. Adjust if plug slips or is over-compressed during insertion.

Terminations and Success Metrics#

Termination / metric hyperparameters#

Parameter

Default

Effect

success_pos_threshold

3 mm

Mate-point distance counted as success in Metrics/success_rate. Tighten to match real acceptance criteria.

plug_dropped distance threshold

15 cm

Early reset if EE leaves plug. Tighten to discourage release; loosen if false positives during large motions.

Orientation thresholds (roll/pitch)

15°

Reset if plug tilts excessively relative to grasp frame.

RL Algorithm (PPO)#

Defined in config/displayport_rizon_4s/agents/rsl_rl_ppo_cfg.py.

PPO hyperparameters#

Parameter

Default

Effect

max_iterations

1500

Total training iterations. Extend if success rate is still climbing at the end.

num_steps_per_env

512

Rollout length per iteration. Affects curriculum annealing rate (tied to anneal_end_iter).

learning_rate

5e-4

PPO learning rate. Reduce if training is unstable; increase if learning is very slow.

desired_kl

0.008

Target KL for adaptive LR schedule.

init_noise_std

1.0

Exploration noise. Lower for fine-tuning a near-working policy.

Suggested tuning order: (1) confirm asset/physics quality, (2) curriculum depth and anneal schedule, (3) linear vs exponential reward balance, (4) socket pose DR and observation noise, (5) action scale, (6) PPO training length.

Part 3: Training the Policy in Isaac Lab#

Registered Gym Environments#

Train on the -ROS-Inference ids. They are the environments used to produce the policies deployed on hardware: in addition to the plain training configuration, they declare the observation/action metadata Isaac Manipulator and LEAPP consume (obs_order, policy_action_space, observation_space, action_space, action_scale) and pin the plug, socket and seed joint pose to the real deployment station. Training on them means the environment you trained in and the contract you export are the same thing, so a checkpoint can be exported and deployed with Isaac ROS without swapping configurations.

Flexiv Rizon 4s DisplayPort Insertion Environments#

Environment ID

Purpose

IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace-ROS-Inference

Task-space training and deployment — recommended. Carries the ROS / LEAPP export contract.

IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace

Task-space training without deployment metadata (ablations, experiments)

IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace-Play

Task-space evaluation / visualization (observation corruption disabled)

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference

Joint-space training and deployment. Carries the ROS / LEAPP export contract (14-dim actor obs).

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel

Joint-space training without deployment metadata

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-Play

Evaluation / visualization (observation corruption disabled)

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-ROS-Inference

Joint-space ROS inference with joint velocity in actor obs (21-dim)

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint

Training with joint velocity in actor obs (21-dim)

IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-Play

Evaluation with joint velocity in actor obs

The plain training ids remain useful when you are experimenting and do not intend to deploy that particular checkpoint — for example when sweeping rewards or randomization. If you later want to ship such a checkpoint, export it against the matching -ROS-Inference id so the traced layout matches deployment.

Note

Running the commands. uv is the recommended workflow (see isaaclab-installation-root), and the commands below use it from this standalone repository. Training goes through Isaac Lab’s packaged CLI (uv run isaaclab train ...), while DisplayPort export and inference use this repository’s entry points (uv run isaaclab-training-export and uv run isaaclab-training-play-policy-io). Add --extra leapp for LEAPP export and --group sim when you need the Isaac Sim dependency.

Note

Training and play use the unified RL entrypoints. You must pass --rl_library rsl_rl. This repository does not require an Isaac Lab source checkout.

Step 1: Visualize the Environment#

Launch training with a small number of environments and visualization enabled to verify the setup. Pick the tab for the control space you intend to deploy — task space is the recommended choice for real-robot transfer (see Choosing a Control Space):

uv run isaaclab train --rl_library rsl_rl \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference \
    --num_envs 4 \
    --max_iterations 100 \
    --visualizer kit

uv run isaaclab train --rl_library rsl_rl \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace-ROS-Inference \
    --num_envs 4 \
    --max_iterations 100 \
    --visualizer kit

What to Expect:

In early training, the robot moves the grasped plug toward the socket but will not insert reliably yet. Verify that:

../_images/dp_untrained_viz.gif

Early untrained behavior: the robot explores around the socket before the insertion behavior has been learned.#

  • The plug is grasped at reset and held throughout the episode

  • The socket pose randomization and plug curriculum produce varied starting configurations

  • Contact between plug blades and socket looks physically plausible

  • With the plug placed in the fully inserted pose (no policy), it stays seated without drift or instability

Stop training (Ctrl+C) once the environment looks correct, then proceed to full-scale training.

../_images/dp_insertion_validation.gif

Trained policy replay in Isaac Lab with the reset curriculum disabled.#

Step 2: Full-Scale Training with Video Recording#

Launch full training in headless mode with video recording:

uv run isaaclab train --rl_library rsl_rl \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference \
    --num_envs 256 \
    --viz none \
    --video --video_length 200 --video_interval 76800

uv run isaaclab train --rl_library rsl_rl \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace-ROS-Inference \
    --num_envs 256 \
    --viz none \
    --video --video_length 200 --video_interval 76800

Multi-GPU (distributed) training — for example on a cluster / OSMO workflow, launch the packaged Isaac Lab trainer under your cluster’s distributed wrapper and keep --distributed in the Isaac Lab arguments:

uv run isaaclab train --rl_library rsl_rl \
    --task <TASK_ID> \
    --num_envs <NUM_ENVS> \
    --viz none --distributed \
    agent.max_iterations=<MAX_ITERS> \
    --video --video_length 200 --video_interval 25600

The commands above train on the -ROS-Inference ids so the trained environment already carries the deployment contract. Swap in the plain ...-Joint-NoJointVel / ...-TaskSpace ids only for experiments you do not intend to deploy.

Command breakdown:

  • --rl_library rsl_rl: Selects the RSL-RL backend (required by the unified trainer)

  • --num_envs 256: Runs 256 parallel environments

  • --viz none: Disables all visualizers for throughput (the unified entry point has no --headless flag; pass --viz kit or --viz newton to enable a viewer)

  • --video_length 200: One episode per video (episode_length_s / (sim.dt * decimation) ≈ 200 steps)

  • --video_interval 76800: Records a video every 76,800 environment steps (~every 150 iterations with 512 steps/env)

  • --distributed: Required when launching under torch.distributed.run

Training uses a recurrent PPO agent (LSTM, 1500 max iterations, 512 steps per environment). Videos are saved under logs/.

Note

GPU Memory Considerations: The default configuration uses 4096 environments in the base config but 256 is recommended for most GPUs. The plug and socket SDF collision meshes and high rigid-body solver counts increase GPU memory usage compared to primitive-shape tasks. Reduce num_envs or solver_position_iteration_count on the plug/socket assets if you encounter out-of-memory errors.

Monitoring Training Progress with TensorBoard:

uv run python -m tensorboard.main --logdir logs/rsl_rl/displayport_insertion_rizon4s

Monitor Metrics/success_rate and reward curves to confirm learning. The curriculum anneals over the first 500 iterations — expect success rate to rise as the at-goal reset probability decreases.

Policy learning progress (during training): These TensorBoard curves show whether the policy is improving, whether terminal success follows the dense success metric, and whether the curriculum is still producing useful learning signal.

../_images/dp_tb_metrics.png

TensorBoard success metrics from the DisplayPort training run.#

Choosing a Control Space#

Both control spaces train with the same algorithm, reward, curriculum, and 30 Hz control rate; they differ only in what the policy outputs and what the real robot must reproduce.

Joint space

Task space (recommended)

Policy output

7 joint-position deltas

6-DoF Cartesian pose delta (OSC)

Actor observation

14 dims (or 21 with joint_vel)

18 dims (TCP pose + socket keypoint frame)

Real-robot requirement

Joint servo behavior must match sim; benefits from joint-level system identification

Task-space bridge that reproduces the OSC contract and the TCP-observe / flange-control split

Training ids

...-Joint-NoJointVel-ROS-Inference (deployable), ...-Joint-NoJointVel

...-TaskSpace-ROS-Inference (deployable), ...-TaskSpace

Task space is the recommended route. In our sim-to-real testing on the Flexiv Rizon 4s, task-space policies transfer better than joint-space ones: commanding a Cartesian pose delta keeps the policy independent of the arm’s joint servo behavior and of small errors in the robot’s kinematic model, both of which are difficult to reproduce exactly in simulation. Prefer it unless you specifically need joint-level control.

Use joint space when you must control the arm through its joint interface. Because the policy then commands joints directly, transfer depends on the simulated robot matching yours closely — see the note below on robot kinematics.

Important

Joint-space training needs a robot USD that matches your specific arm. Joint-space policies command joint positions, so any mismatch between the simulated kinematic parameters and your physical robot shows up directly as end-effector error during insertion — where the clearances are sub-millimetre. Individual arms differ from the nominal CAD model, so the shipped asset will not describe your unit exactly.

These environments spawn the stock FLEXIV_RIZON4S_GRAV_GRIPPER_CFG asset, which describes a nominal Rizon 4s rather than any particular unit. Before training a joint-space policy you intend to deploy, replace it with a description of your own arm. Flexiv publishes a per-robot calibration workflow for exporting an accurate robot description: flexiv_calibration. Follow it to generate the calibrated description for your setup, convert it to USD, and set usd_path on the robot spawn config in config/displayport_rizon_4s/joint_pos_env_cfg.py.

Task-space policies are less exposed to this: they are commanded in Cartesian space, so kinematic error affects the observed end-effector pose rather than being injected straight into the commanded joint targets. Getting the kinematics right is still worthwhile in both cases.

In each case, train on the -ROS-Inference id so the trained environment carries the deployment contract, and use the -Play id for evaluation and visualization. The task-space environment already uses the LEAPP-exportable DeployOperationalSpaceControllerActionCfg action, so a trained checkpoint can be exported directly (see Exporting a Task-Space Policy).

Step 3: Export and Deploy on Real Robot#

Recommended workflow: export the trained policy with LEAPP, validate the export in simulation, then deploy the LEAPP package with Isaac ROS / Isaac Manipulator on the Flexiv robot.

../_images/deploy-train-run.png

Boundary between Isaac Lab training and deployment.#

Export from the -ROS-Inference task id so the traced observation and action layout matches deployment. If you followed the training steps above this is the same id you trained on, and no configuration swap is needed. In joint space that is a NoJointVel variant (14-dim actor input); in task space it is -TaskSpace-ROS-Inference (18-dim actor input).

Exporting a Task-Space Policy#

A task-space checkpoint is exported with the packaged exporter and the displayport_task_space contract because its action is a scaled 6-DoF Cartesian pose delta rather than joint targets. The same exporter stays task-agnostic by loading task-specific contracts from src/isaaclab_training/export/contracts.py:

uv run --extra leapp isaaclab-training-export \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-TaskSpace-ROS-Inference \
    --checkpoint logs/rsl_rl/displayport_insertion_rizon4s/<run_timestamp>/model_<iteration>.pt \
    --export_save_path <output_dir> \
    --contract displayport_task_space

The export writes a DisplayPortTaskSpace LEAPP package (.onnx graph, .yaml semantics, and _initial_values.safetensors for the recurrent state). Its I/O contract is:

  • Inputs: eef_pos (3), eef_rot_6d (6), socket_kp_pos (3), socket_kp_rot_6d (6), plus the actor LSTM state.

  • Output: arm_action (6) — a pose_rel Cartesian delta [dx, dy, dz, dθx, dθy, dθz] consumed by the task-space bridge (isaaclab_connection: action:arm_action:pose_rel).

Validate it exactly as for the joint-space export above with isaaclab-training-play-policy-io and --leapp_model pointing at the exported directory.

Alternative: Raw Checkpoint Deployment#

For development or legacy Isaac Manipulator setups, you can deploy the RSL-RL checkpoint directly without a LEAPP export step. This path uses the .pt checkpoint with agent.yaml and env.yaml from:

logs/rsl_rl/displayport_insertion_rizon4s/<run_timestamp>/model_<iteration>.pt

This is not recommended for shipping — you must manually ensure observation ordering, action scaling, and LSTM state handling match training. Prefer the LEAPP export path above for production deployment.

Troubleshooting#

PhysX Collision Stack Overflow#

Error Message:

PhysX error: PxGpuDynamicsMemoryConfig::collisionStackSize buffer overflow detected

Cause: GPU collision buffer is too small for contact-rich plug/socket interaction across many parallel environments.

Solution: Increase gpu_collision_stack_size in displayport_insertion_env_cfg.py (default is 2**30):

sim: SimulationCfg = SimulationCfg(
    physics=PhysxCfg(
        gpu_collision_stack_size=2**31,  # Increase if overflow persists
        gpu_max_rigid_contact_count=2**23,
        gpu_max_rigid_patch_count=2**23,
    ),
)

CUDA Out of Memory#

Solutions (in order of preference):

  1. Reduce parallel environments:

    uv run isaaclab train --rl_library rsl_rl \
    --task <TASK_ID> \
    --num_envs 128 \
    --viz none

  2. Reduce plug/socket solver_position_iteration_count in displayport_insertion_env_cfg.py (trade-off: more penetration)

  3. Disable video recording during training

Deterministic Debugging (Play / Inference Script)#

For DisplayPort insertion, prefer the dedicated play entry point over the generic uv run isaaclab play path. It keeps DP-specific pose overrides, perception-error injection, and policy_io.csv logging out of the shared play entrypoint.

The examples below use the joint-space ROS-inference task. They apply unchanged to a task-space policy — substitute ...-TaskSpace-ROS-Inference for the --task id and point --checkpoint (or --leapp_model) at the corresponding run.

RSL-RL checkpoint (recommended shipping task):

uv run isaaclab-training-play-policy-io \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference \
    --checkpoint logs/rsl_rl/displayport_insertion_rizon4s/<run>/model_<iteration>.pt \
    --num_envs 1 \
    --socket_pos 0.476 0.127 0.07 \
    --observed_socket_pos 0.486 0.127 0.07 \
    --max_steps 200 \
    --log_dir logs/dp_inference_runs \
    --visualizer kit

Exact first observation from real/sim CSV (closed-loop; injects obs_* from the chosen row as the first policy input, then continues with live sim observations):

uv run isaaclab-training-play-policy-io \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference \
    --checkpoint logs/rsl_rl/displayport_insertion_rizon4s/<run>/model_<iteration>.pt \
    --init_obs_csv rollouts/sim_real_compare/real_policy_gt_sock.csv \
    --socket_pos 0.475 0.125 0.07 \
    --num_envs 1 \
    --max_steps 200 \
    --log_dir logs/dp_init_obs_runs \
    --visualizer kit

Unset --robot_joint_pos / --observed_socket_* are seeded from that CSV row so the scene starts aligned. Use --init_obs_step N to pick a non-zero row. Works with --leapp_model as well; not compatible with --replay_csv.

LEAPP-exported ONNX policy (same script; see Step 3 above for --leapp_model).

Open-loop replay of a logged policy_io.csv / real rollout:

uv run isaaclab-training-play-policy-io \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-ROS-Inference \
    --replay_csv rollouts/sim_real_compare/real_policy.csv \
    --socket_pos 0.475 0.125 0.07 \
    --log_dir rollouts/sim_real_compare/replay_run \
    --visualizer kit

Pose conventions:

  • --socket_pos / --observed_socket_pos are socket insertion-geometry world positions (mate point), matching fixed_asset_init_pos_center, not the USD root. The USD root is derived via compute_socket_root.

  • Quaternions are (x, y, z, w).

  • If --observed_socket_* is omitted, the policy observes the true simulated socket pose.

  • Telemetry is written to policy_io.csv (+ run_config.json) under --log_dir or <checkpoint_dir>/inference_logs/<timestamp>/.

Generic play (no DP pose / CSV knobs) still works for a quick smoke test:

uv run isaaclab play --rl_library rsl_rl \
    --task IsaacTraining-DisplayPortInsertion-Rizon4s-Joint-NoJointVel-Play \
    --num_envs 1 \
    --checkpoint <path_to_model.pt>

To match a specific real-world station layout, edit the workspace constants in config/displayport_rizon_4s/joint_pos_env_cfg.py (training layout) or config/displayport_rizon_4s/ros_inference_env_cfg.py (deployment layout):

# Training station layout (joint_pos_env_cfg.py)
_GEOMETRY_POS = (0.475, 0.125, 0.06)
_SOCKET_ROT = (0.5, 0.5, 0.5, -0.5)

# Deployment layout (ros_inference_env_cfg.py)
_DEPLOY_GEOMETRY_POS = (0.476, 0.127, 0.07)
_DEPLOY_SOCKET_ROT = (0.5, 0.5, 0.5, -0.5)

This workflow is useful for:

  • Comparing simulated and real-world policy behavior at a known socket pose

  • Validating LEAPP exports before hardware deployment

  • Verifying plug grasp and approach trajectories before full perception integration

  • Debugging insertion failures at a fixed station configuration

Further Resources#

  • Gear Assembly Sim-to-Real Tutorial: walkthrough_sim_to_real

  • Exporting Policies with LEAPP: /source/policy_deployment/05_leapp/exporting_policies_with_leapp

  • Isaac ROS Manipulation Documentation

  • RL Training Tutorial: tutorial-run-rl-training