DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Robot DevOps and MLOps — Ch.10: Hardware in the Loop CI [EN]

STATERA Library · Series 08 · Volume 10 · Chapter 10 · EN
Source: https://github.com/Statera-Library/08_Vol_10_Robot_DevOps_and_MLOps/blob/main/Chapter_10_Hardware_in_the_Loop_CI/Chapter_10_Hardware_in_the_Loop_CI_EN.md

Volume 10 Robot DevOps and MLOps

10. Hardware in the Loop CI

10.1. HIL CI Concepts Physical Robot in Automated Test Loop

Hardware-in-the-Loop Continuous Integration (HIL CI) extends conventional software testing by placing real robot hardware directly inside an automated verification pipeline. Instead of validating every change only with unit tests or simulation, selected software revisions are deployed to physical controllers, sensors, actuators, or complete robots. The resulting behavior is measured automatically and converted into reproducible pass-or-fail evidence before software progresses toward release.

This approach is especially important in robotics because correct software execution does not necessarily imply correct physical behavior. A controller may compile successfully and pass simulation while producing unexpected timing, communication, electrical, thermal, or mechanical effects on actual hardware. HIL CI therefore connects software verification with the physical characteristics of the robot and exposes integration failures that conventional CI environments cannot realistically reproduce.

A typical HIL CI workflow begins when a developer commits or merges a software change. Conventional CI stages first build the software and execute static analysis, unit tests, integration tests, and possibly simulation-based validation. Only revisions satisfying these inexpensive gates proceed to hardware testing. The CI system then reserves an available robot or test bench, installs the candidate software, configures the required test environment, and starts the predefined physical test sequence automatically.

The physical robot becomes a controlled CI resource rather than an independently operated laboratory prototype. Test orchestration software can power the system, initialize embedded computers, launch ROS 2 nodes, configure sensors, command actuators, collect telemetry, and restore the platform after each test. This concept forms the foundation for later robot-farm architectures in which multiple physical platforms are remotely allocated to different builds and test jobs.

Automation requires the physical test environment to be deterministic enough for meaningful comparison. Software versions, firmware revisions, calibration parameters, robot configuration, test fixtures, environmental conditions, and test datasets should therefore be recorded together. Without configuration traceability, a failed experiment may be impossible to reproduce because the software revision alone does not completely describe the physical system that executed the test.

HIL CI can operate at several levels of hardware integration. A low-level bench may contain only an embedded controller connected to simulated electrical interfaces, while an actuator bench may include actual motor drivers, motors, encoders, brakes, or steering mechanisms. Higher-level configurations can integrate sensors and computing platforms, and the most complete form places an operational robot inside a controlled test area where navigation, manipulation, perception, and safety behavior can be exercised.

This layered approach allows organizations to balance execution cost against physical realism. Fast controller-level HIL tests can run frequently, whereas complete robot tests may be reserved for important merges, nightly builds, release candidates, or safety-related modifications. HIL therefore complements rather than replaces software-in-the-loop and simulation testing. Each stage progressively increases fidelity while reducing the number of defective builds that consume scarce physical testing resources.

A fundamental design principle is separation between test orchestration and the system under test. The robot executes the software being evaluated, while an independent controller observes its behavior and determines whether expected conditions have been satisfied. External measurement equipment, cameras, power monitors, motion tracking systems, network monitors, or dedicated safety controllers may provide evidence that cannot reliably be obtained from the robot\'s own software.

For ROS 2 robots, HIL CI can validate interactions extending across nodes, middleware, embedded controllers, and physical devices. The pipeline may verify topic publication rates, message latency, lifecycle transitions, sensor availability, actuator responses, localization outputs, or navigation results. Such tests expose failures caused by timing and interface dependencies that may remain invisible when individual ROS 2 packages are evaluated separately in conventional software CI.

Motor and actuator testing provides another important HIL layer. Automated commands can exercise velocity, steering, braking, manipulator joints, or other mechanisms while measured feedback is compared against expected ranges. The broader chapter structure consequently separates automated motor and actuator HIL testing as a dedicated topic, together with sensor replay, navigation regression, safety-function validation, AI evaluation, and result reporting.

Sensor-oriented HIL testing introduces real interfaces into repeatable data-driven experiments. Recorded camera, LiDAR, radar, GNSS, IMU, or other sensor streams can be replayed while the target computing hardware executes the production software stack. This arrangement combines reproducible input data with realistic processors, drivers, middleware, accelerators, and operating-system behavior, making it useful for identifying timing regressions and hardware-dependent processing failures.

Complete physical-robot tests extend verification from interfaces to observable behavior. An AMR can receive a mission, localize itself, plan a route, move through a controlled environment, avoid predefined obstacles, stop at specified positions, and report completion. Measurements such as mission success, localization error, stopping accuracy, execution time, CPU or GPU utilization, communication latency, and safety events can then become quantitative regression criteria.

Safety changes require particularly careful HIL design because automated testing itself can generate hazardous motion. Hardware emergency stops, independent power isolation, restricted test zones, speed limits, watchdogs, physical barriers, and supervisory controllers should remain independent of the software under evaluation. Tests involving emergency stop, safe limited speed, or safe torque off should verify not merely that a software message exists, but that the physical system reaches the required safe condition.

The CI pipeline must also distinguish infrastructure failures from product failures. A disconnected charger, unavailable robot, damaged sensor, network interruption, or test-area obstruction should not automatically be interpreted as a software regression. HIL infrastructure therefore requires health checks and preconditions before execution, followed by structured diagnostics capable of classifying setup errors, hardware faults, software failures, and inconclusive results.

Resource scheduling becomes increasingly important as HIL adoption grows. Unlike virtual CI runners, physical robots cannot be duplicated instantly, and each platform may require charging, calibration, maintenance, environmental reset, or manual recovery. A practical HIL scheduler therefore tracks robot availability and configuration while coordinating reservations, exclusive access, test duration, recovery operations, and maintenance states. Parallel robot farms can eventually increase throughput for large development organizations.

Traceability connects HIL CI with broader DevOps and MLOps practices. Every execution should associate the tested software commit with firmware versions, container images, configuration files, AI model versions, calibration data, hardware identity, test definition, and collected results. This makes a HIL result reproducible and allows engineers to determine precisely which physical configuration demonstrated a regression or satisfied a release criterion.

HIL-generated telemetry also provides valuable evidence for debugging. ROS 2 bags or MCAP recordings, controller logs, sensor streams, actuator states, CPU and GPU metrics, network statistics, power measurements, and video can be captured during each run. When a regression occurs, these synchronized artifacts allow engineers to reconstruct the sequence of events rather than depending on manual observation or incomplete operator notes.

Pass-or-fail criteria should be machine-readable wherever possible. Instead of reporting that the robot appeared to operate normally, a test can define numerical limits for trajectory error, stopping distance, message latency, actuator response time, localization stability, inference latency, or mission completion. Automated evaluation converts these measurements into CI results and prevents subjective interpretation from becoming the primary release gate.

HIL CI also provides an important bridge between simulation and field validation. Simulation can execute enormous numbers of scenarios quickly and safely, while physical HIL confirms that selected behaviors survive real sensors, processors, communication buses, actuators, timing constraints, and mechanical dynamics. Field testing then evaluates conditions that remain difficult to reproduce in either environment. Together these stages create progressively stronger evidence about system behavior.

For AI-enabled robots, the same principle applies to perception and decision components. A model that achieves acceptable offline accuracy may behave differently when executed on the target GPU with real sensor timing, preprocessing, memory limits, and concurrent workloads. HIL evaluation therefore helps verify not only model accuracy but also inference latency, dropped frames, synchronization, resource consumption, and the interaction between AI outputs and downstream robot behavior.

The ultimate objective of HIL CI is not simply to automate laboratory experiments. It is to make physical behavior part of the software delivery discipline. When a robot can be provisioned, exercised, measured, evaluated, and reset automatically, hardware validation becomes a repeatable extension of continuous integration. This reduces the gap between software commits and real-world consequences and establishes the foundation for robot farms, continuous regression testing, and scalable DevOps for physical robotic systems.

10.2. HIL Test Infrastructure Design Robot Farm Architecture

A Hardware-in-the-Loop robot farm extends HIL CI from a single laboratory test platform into shared infrastructure where multiple physical robots, controllers, sensors, and actuator benches can execute automated tests. The central architectural objective is to make scarce physical hardware behave like schedulable CI resources while preserving safety, reproducibility, hardware identity, and reliable recovery between test executions.

A robot farm normally separates orchestration infrastructure from the physical systems under test. CI servers receive software changes and determine which HIL tests are required, while a dedicated orchestration layer selects suitable hardware and manages the test lifecycle. This separation prevents individual CI jobs from directly controlling laboratory resources and provides a common management boundary for scheduling, access control, health monitoring, and failure recovery.

Each hardware resource should be represented by an explicit machine-readable profile. A resource description can identify robot type, controller architecture, CPU and GPU platform, firmware compatibility, connected sensors, actuator configuration, safety capabilities, test-area assignment, and supported software versions. CI jobs can then request capabilities rather than relying on manually assigned robot names, allowing the scheduler to select compatible hardware automatically.

Resource scheduling is a fundamental robot-farm service because physical robots cannot execute unlimited parallel workloads. The scheduler maintains states such as available, reserved, testing, charging, recovering, calibrating, maintenance, and unavailable. When a pipeline requests a particular hardware configuration, the scheduler reserves a compatible resource, prevents conflicting access, establishes an execution lease, and releases the hardware only after testing and recovery have completed.

The robot farm requires a provisioning layer that transforms reserved hardware into a known test state. Provisioning can install operating-system packages, containers, ROS 2 workspaces, firmware, configuration files, calibration parameters, and AI models associated with a particular build. Before execution, the infrastructure should verify software versions and hardware configuration so that every test starts from a controlled baseline rather than an unknown state left by a previous experiment.

A physical test station typically combines the robot with supporting infrastructure. Depending on the platform, this can include controllable power distribution, chargers, network switches, wireless access points, emergency-stop circuits, cameras, external computers, measurement equipment, and environmental fixtures. These components should be treated as part of the HIL system because their state can directly influence whether a test is executable, reproducible, or safe.

Remote power management is particularly useful for unattended operation. Network-controlled power distribution units, relays, or dedicated supervisory controllers can power-cycle computers and selected devices when software becomes unresponsive. However, automated recovery should remain separated from safety functions. A CI server that controls software deployment should not be the only mechanism capable of stopping hazardous physical motion or disconnecting actuator power.

Robot-farm networking must support both operational communication and infrastructure management. ROS 2 traffic, sensor streams, telemetry, deployment data, management commands, and test artifacts can generate substantially different network requirements. Separating management and test traffic logically or physically can reduce interference and simplify diagnosis, while synchronized clocks are important when results from robot computers, sensors, external observers, and orchestration servers must be correlated.

The orchestration service manages the complete execution state machine. After reservation, it checks hardware health, verifies safety prerequisites, provisions the target software, launches required processes, executes test scenarios, monitors progress, collects results, and initiates cleanup. A successful test does not end when the robot stops moving; the infrastructure must also archive evidence, restore the test station, verify readiness, and return the resource to the available pool.

Pre-test health checks reduce false failures caused by infrastructure problems. The robot farm can verify battery state, charging status, network connectivity, disk capacity, sensor availability, actuator communication, emergency-stop state, localization readiness, calibration validity, and required external equipment. If these prerequisites are not satisfied, the job should be classified as an infrastructure exception rather than immediately reported as a software regression.

Safety architecture must remain independent from normal test automation wherever practical. Physical emergency stops, safety relays, protected test areas, speed restrictions, watchdogs, access detection, and independent supervisory systems provide protection if experimental software behaves unpredictably. Automated HIL execution increases the importance of these controls because robots may operate without an engineer continuously standing beside every platform.

Environmental design depends strongly on the robot class. A controller or actuator bench may need only an enclosed rack or fixture, whereas an AMR requires a controlled navigation area containing known landmarks, obstacles, docking targets, and reference positions. Manipulators may require fixtures and replaceable test objects, while legged robots need protected operating zones. The environment itself therefore becomes a versioned component of the test infrastructure.

Observability should cover both the system under test and the robot farm. Infrastructure metrics can include reservation utilization, queue time, test duration, charging time, recovery frequency, hardware faults, network health, and unavailable resources. Robot-level telemetry can include ROS 2 messages, controller states, sensor data, CPU and GPU utilization, power consumption, localization results, actuator feedback, and safety events collected during each execution.

External observation provides an independent reference when internal robot telemetry is insufficient. Cameras, motion-capture systems, fiducial markers, distance sensors, force measurement devices, or other instrumentation can determine whether physical behavior matches expected results. This is particularly valuable when validating stopping position, trajectory accuracy, manipulator motion, collision avoidance, or other behaviors where relying solely on the robot\'s own estimated state could hide systematic errors.

Test artifacts should be associated with a unique execution identity. The record can connect the CI job, Git commit, container or package version, firmware revision, AI model, hardware identifier, robot configuration, calibration data, test scenario, environment configuration, logs, telemetry, video, and final verdict. This traceability transforms robot-farm execution from an informal laboratory experiment into reproducible engineering evidence.

Recovery automation is essential because physical systems fail in ways that virtual CI runners do not. A robot may stop outside its expected position, lose localization, discharge its battery, disconnect from the network, encounter an obstacle, or leave an actuator in an abnormal state. Recovery procedures should first determine whether automated restoration is safe and possible; otherwise, the platform should be quarantined and marked for human intervention rather than repeatedly restarting a potentially unsafe test.

Charging and energy management become scheduling concerns when mobile robots are continuously tested. The infrastructure should avoid assigning a long mission to a robot without sufficient energy and should account for charging time when estimating availability. Automated docking can improve utilization, but charging status and battery health must still be incorporated into readiness checks because power conditions can influence computational performance and physical behavior.

Parallel execution improves throughput by distributing tests across multiple compatible platforms. Identical robots can process independent regression jobs concurrently, while heterogeneous farms can route controller tests, perception tests, navigation tests, and complete mission tests to specialized resources. Parallelism should nevertheless preserve isolation so that wireless traffic, moving robots, shared environments, or external equipment from one test do not invalidate another test.

A scalable architecture benefits from separating logical test definitions from individual physical robots. Test specifications describe required capabilities, initial conditions, actions, expected observations, timeout rules, and pass-or-fail criteria. The scheduler maps those requirements onto available resources at runtime. This abstraction allows additional robots or upgraded hardware to enter the farm without requiring every CI pipeline to be rewritten around specific laboratory devices.

Robot farms also require lifecycle management for the infrastructure itself. Hardware inventory, firmware baselines, calibration expiration, maintenance history, damaged components, replacement parts, and configuration changes should be tracked alongside software. Otherwise, gradual changes in physical platforms can introduce unexplained differences between historical and current results even when the tested software remains identical.

Access control is important because HIL infrastructure can command real actuators and potentially safety-critical functions. Developers, CI services, administrators, and maintenance personnel should receive only the permissions required for their roles. Authentication, reservation ownership, audit records, controlled deployment credentials, and restricted interfaces help prevent accidental interference between tests and provide accountability for changes made to shared hardware.

The robot-farm architecture ultimately turns physical validation into an infrastructure service for robotics development. CI pipelines request hardware capabilities, the scheduler allocates suitable resources, provisioning establishes a reproducible configuration, orchestration executes the experiment, monitoring captures physical behavior, and automated evaluation returns structured results. Recovery and maintenance then prepare the platform for the next job, completing a repeatable HIL lifecycle.

As the number of robots grows, this architecture becomes the physical counterpart of scalable software CI infrastructure. Virtual runners provide inexpensive and highly parallel software verification, while the robot farm supplies controlled access to real sensors, processors, communication interfaces, actuators, and mechanical dynamics. Combining both layers allows frequent software iteration while reserving expensive physical execution for changes that have already passed earlier verification stages.

10.3. ROS2 Launch Test Framework for HIL [w/Code]

ROS 2 launch testing provides a practical framework for connecting software-level continuous integration with physical Hardware-in-the-Loop execution. In conventional ROS 2 development, launch files start multiple nodes and configure their parameters, namespaces, remappings, and dependencies. In HIL, the same mechanism can establish a controlled test configuration containing real sensors, controllers, actuators, robot computers, and supporting test processes.

The ROS 2 launch system is particularly suitable for HIL because robotic behavior rarely depends on a single executable. A navigation or manipulation function may require drivers, state publishers, localization components, perception nodes, planners, controllers, and lifecycle-managed services to operate together. Launch-based testing allows this distributed software graph to be instantiated consistently before automated assertions evaluate its behavior.

A HIL launch test normally separates system startup from test evaluation. The launch description defines which production nodes and supporting processes must run, while the test logic observes their behavior through ROS 2 interfaces. This separation allows engineers to execute essentially the same robot software used during operation while adding dedicated test nodes, monitoring processes, stimulus generators, and verification logic around the system under test.

The launch_testing framework extends ROS 2 launch functionality with automated test coordination. Tests can start required processes, wait for initialization, execute assertions while nodes are active, and evaluate process behavior after shutdown. In a HIL environment, these capabilities can be combined with hardware preparation steps so that software execution begins only after the robot, controller, sensors, communication interfaces, and safety systems have reached the required state.

Lifecycle management is important when hardware initialization takes different amounts of time. A camera may require several seconds before publishing valid images, a LiDAR may need initialization, and a motor controller may require communication and fault checks before accepting commands. Rather than relying on arbitrary delays, the test framework should observe explicit readiness conditions and continue only when required components report valid operational states.

ROS 2 topics provide one of the primary observation channels for automated HIL tests. A test node can subscribe to odometry, joint states, sensor streams, localization estimates, diagnostics, velocity commands, or application-specific messages. It can verify publication frequency, timestamp behavior, numerical ranges, state transitions, and relationships between commands and measured responses while the actual robot hardware executes the candidate software.

Services and actions provide additional interfaces for controlling HIL scenarios. Services can reset components, modify operating modes, clear faults, or request system states, while ROS 2 actions can represent longer operations such as navigation goals or manipulator trajectories. Automated tests can issue these requests, monitor feedback, enforce timeouts, inspect final results, and determine whether physical execution satisfies predefined acceptance criteria.

Parameters must also be controlled because small configuration differences can significantly change physical robot behavior. HIL launch descriptions should load known parameter sets for controllers, sensor drivers, navigation components, safety limits, and AI modules. The exact parameter files used during testing should be preserved with the test result so that a successful or failed execution can later be reproduced using the same configuration.

Namespaces and remapping become increasingly important when the same test framework supports multiple robots or test stations. A generic HIL test can target different platforms by assigning robot-specific namespaces and remapping standardized logical interfaces to actual topics. This reduces duplication and allows one test definition to operate across compatible hardware while maintaining isolation when several robots execute tests simultaneously in a robot farm.

Before the main test begins, a launch-based HIL workflow should perform explicit preconditions. The framework can verify that required nodes are running, expected topics exist, sensors publish data, controllers are connected, transforms are available, and diagnostic states are acceptable. Physical prerequisites such as battery level, emergency-stop status, test-zone readiness, or actuator availability may be supplied through dedicated monitoring interfaces.

The test stimulus should be deterministic whenever practical. Instead of relying on an operator to manually command the robot, the framework can publish predefined velocity commands, send trajectories, request navigation goals, replay sensor recordings, or activate known test sequences. Deterministic stimulus improves regression testing because the same software behavior can be compared across commits, firmware revisions, hardware configurations, and repeated physical executions.

Timing requires special attention in HIL because ROS 2 software interacts with real devices and real clocks. Tests should distinguish between functional failure and normal initialization variation while still detecting excessive latency, missed deadlines, stale messages, or communication loss. Explicit timeout policies and timestamp checks provide more reliable evidence than fixed sleep statements scattered throughout test scripts.

Process monitoring is another important capability of launch-based testing. The framework should detect nodes that terminate unexpectedly, fail to start, repeatedly restart, or exit with abnormal return codes. A robot may appear stationary because a planner failed, a driver crashed, or a controller never activated. Capturing process-level state together with ROS 2 communication data makes the resulting HIL failure considerably easier to diagnose.

Assertions convert observed physical behavior into machine-readable CI results. A test can require that a sensor maintain a minimum publication rate, an actuator reach a commanded position within tolerance, localization remain within an error bound, or a navigation action finish before a timeout. Assertions should focus on measurable behavior and explicitly defined limits rather than subjective observations that depend on an engineer watching the robot.

Physical HIL assertions often require tolerance ranges rather than exact equality. Encoder noise, mechanical backlash, sensor uncertainty, communication jitter, friction, and battery conditions introduce natural variation that does not exist in deterministic unit tests. Acceptance thresholds should therefore represent meaningful engineering limits while remaining strict enough to detect regressions in control performance, timing, perception, or system integration.

Post-shutdown testing is useful for evaluating information that becomes available only after execution. The framework can inspect process return codes, verify that expected nodes terminated correctly, examine generated files, summarize logs, and confirm that no critical errors occurred during shutdown. HIL infrastructure can then collect ROS bags or MCAP files, diagnostics, performance metrics, videos, and other artifacts before resetting the physical station.

Failure handling should preserve evidence before attempting recovery. When an assertion fails or a process crashes, the test controller should record the current robot state, relevant ROS 2 messages, logs, hardware diagnostics, and execution metadata. Only after this evidence has been secured should automated recovery restart nodes, power-cycle equipment, return actuators to safe positions, or request intervention from the robot-farm infrastructure.

Safety supervision must remain independent from ordinary ROS 2 test logic. A launch test may request robot motion and verify safety-related responses, but software assertions should not replace certified or independent protective mechanisms. Emergency-stop circuits, safety controllers, watchdogs, physical barriers, speed restrictions, and supervisory systems should continue protecting the test area even if the ROS 2 graph freezes or the experimental software behaves incorrectly.

Integration with CI transforms ROS 2 launch testing from a local engineering utility into an automated quality gate. After build, static analysis, unit tests, integration tests, and simulation succeed, the pipeline can request an appropriate HIL resource and execute the launch test package. Test results can then be exported in CI-compatible formats together with logs and physical evidence, allowing the pipeline to determine whether the candidate build may progress.

Within a robot farm, the launch configuration should remain independent of unnecessary station-specific details. Infrastructure variables can identify the assigned robot, network interface, sensor configuration, test zone, or hardware revision at runtime. The same logical test can therefore execute on different compatible stations while the orchestration system records exactly which physical platform and configuration were used for each run.

Reusable launch fixtures and test utilities reduce duplication across a large HIL test suite. Common functions can manage node readiness, topic observation, action execution, timeout handling, diagnostic checks, artifact collection, and cleanup. Individual tests then concentrate on the behavior being verified rather than repeatedly implementing infrastructure logic, improving consistency across controller, sensor, navigation, manipulation, and AI HIL tests.

The ROS 2 launch test framework ultimately acts as the software control layer connecting CI orchestration with physical robot execution. It establishes the required ROS 2 graph, waits for hardware readiness, generates deterministic stimuli, observes real responses, evaluates measurable assertions, preserves diagnostic evidence, and shuts the system down predictably. This makes physical robot behavior repeatable enough to participate directly in continuous integration.

10.4. Automated Motor and Actuator HIL Testing [w/Code]

Automated motor and actuator Hardware-in-the-Loop testing brings real motion hardware into the continuous integration process so that control software can be verified against physical responses rather than software models alone. Motors, motor drivers, encoders, steering mechanisms, brakes, linear actuators, and robotic joints can become controlled HIL resources whose electrical and mechanical behavior is measured automatically whenever relevant software changes are introduced.

The purpose of actuator HIL is not merely to confirm that a command reaches a device. It verifies the complete path from application software through middleware, communication buses, embedded controllers, motor drivers, and physical mechanisms to measured feedback. This end-to-end approach exposes integration problems involving timing, scaling, direction, saturation, communication, firmware, and mechanical dynamics that may remain invisible in unit tests or simulation.

A typical automated test bench contains the production controller and motor drive connected to representative physical hardware. Depending on the system, the actuator may operate unloaded, against a controlled mechanical load, or through a dynamometer or fixture that reproduces expected resistance. Encoders, current sensors, voltage monitors, temperature sensors, force sensors, or external position references provide independent measurements of the resulting physical response.

Before motion begins, the HIL infrastructure should establish a known initial state. The test controller verifies hardware identity, firmware revision, communication status, supply voltage, emergency-stop condition, actuator temperature, encoder availability, and applicable calibration parameters. Any residual faults are cleared only when safe, and the actuator is placed in a defined position or disabled state before the automated sequence receives permission to proceed.

Command generation should be deterministic so that results can be compared across software revisions. The test framework can issue predefined position, velocity, torque, steering-angle, acceleration, or braking commands according to a repeatable profile. Step inputs, ramps, trajectories, direction reversals, start-stop cycles, and bounded dynamic sequences allow different aspects of controller and actuator behavior to be exercised under precisely defined conditions.

Measured feedback must be synchronized with the command stream to evaluate dynamic response accurately. Encoder position, rotational speed, motor current, torque estimate, joint angle, force, temperature, controller state, and timestamps can be captured together. External measurement devices may be added when the controller\'s internal feedback cannot provide an independent reference or when systematic sensor errors must be detected.

Position-control testing evaluates whether an actuator reaches the requested target within an acceptable tolerance and settling time. The automated evaluator can compare commanded and measured trajectories, calculate steady-state error, detect overshoot, identify oscillation, and determine whether the mechanism stabilizes before a timeout. Repeating the same motion in both directions can also reveal backlash, asymmetric friction, calibration errors, or direction-dependent controller behavior.

Velocity-control testing focuses on tracking performance across different operating ranges. The framework can command several speed levels and measure rise time, steady-state speed error, fluctuations, acceleration behavior, and response during transitions. For mobile robots, equivalent tests can be applied to individual wheel drives before complete vehicle-level navigation testing, helping isolate drivetrain problems from higher-level localization or planning failures.

Torque and current behavior provide important information about actuator loading and controller health. A HIL bench can compare requested torque with current consumption, measured force, or dynamometer output when suitable instrumentation is available. Unexpected current increases may indicate excessive mechanical resistance, incorrect control parameters, binding, or hardware degradation, while current limiting behavior can be tested without relying solely on software-reported values.

Steering systems require tests that combine position accuracy with mechanical response. Automated sequences can command multiple steering angles, center positions, reversals, and dynamic transitions while recording angle feedback and actuator current. The evaluator can measure steering error, response time, repeatability, hysteresis, and return-to-center behavior, providing regression evidence for mobile robots using independent or coordinated steering mechanisms.

Braking and stopping functions require physical verification because their performance depends on both software timing and actuator behavior. Test sequences can command motion followed by controlled deceleration or braking while measuring command latency, wheel speed reduction, actuator response, and stopping behavior. Where safety-related braking is involved, the automated test must operate inside an independently protected environment with appropriate physical safeguards.

Communication faults should also be represented in actuator HIL testing. The test infrastructure can intentionally interrupt or delay selected communication paths to verify watchdog behavior, command timeout handling, controller fault transitions, and recovery logic. The expected result is usually not continued normal operation but a predictable transition toward a predefined safe state when valid control commands are no longer available.

Limit conditions are another important test category. Software can be exercised near configured velocity, position, torque, current, or acceleration boundaries to verify that limits are enforced consistently across the control stack. Physical hard stops should not be used casually as test mechanisms; wherever possible, software and fixture design should keep automated motion inside controlled mechanical ranges while independently monitoring for abnormal excursions.

Thermal behavior becomes relevant during repeated or extended tests. An actuator that performs correctly during a short functional sequence may exhibit current limiting, reduced torque, communication faults, or protection shutdown after sustained operation. Temperature telemetry can therefore be recorded alongside performance measurements, allowing the test infrastructure to distinguish a software regression from behavior associated with known thermal operating limits.

Automated endurance sequences can repeat representative motions for many cycles to reveal intermittent failures that single executions may miss. Repeated acceleration, reversal, steering, lifting, or joint movement can expose communication instability, encoder anomalies, thermal accumulation, mechanical looseness, or controller-state errors. Such tests should remain bounded by defined duration, temperature, energy, and maintenance limits to avoid turning CI testing into uncontrolled lifetime testing.

Pass-or-fail criteria should translate physical performance into machine-readable engineering limits. Examples include maximum position error, velocity tracking error, response time, overshoot, settling time, current threshold, temperature threshold, communication timeout, or successful transition into a safe controller state. The thresholds should represent system requirements and hardware capability rather than arbitrary values chosen simply to make automated tests pass.

Natural physical variation requires statistical and tolerance-aware evaluation. Two actuator executions will rarely produce numerically identical traces because friction, temperature, supply voltage, sensor noise, mechanical loading, and communication timing vary. Regression analysis should therefore compare meaningful envelopes, summary metrics, or distributions while retaining sufficiently strict boundaries to identify real degradation in control performance.

Safety supervision must remain independent from the software being evaluated. Emergency-stop circuits, power contactors, safety controllers, mechanical restraints, guards, speed restrictions, and supervisory watchdogs should be capable of stopping hazardous motion even if the test program, ROS 2 graph, embedded controller, or communication network fails. Automated testing increases repeatability but must never make experimental software responsible for its own final protection.

When a test fails, the infrastructure should preserve evidence before resetting the bench. Command traces, encoder data, controller states, current and voltage measurements, temperatures, communication logs, ROS 2 messages, firmware information, and video can be associated with the CI execution. These synchronized artifacts allow engineers to determine whether the failure originated in application software, communication, embedded control, drive electronics, sensing, or mechanics.

Recovery procedures return the actuator bench to a known safe condition after successful and failed tests. Commands are disabled, stored energy is considered, moving mechanisms are stopped, faults are recorded, and the actuator is returned to a defined position when this can be done safely. If automated recovery cannot establish a trustworthy state, the hardware should be quarantined rather than immediately assigned to another CI job.

Within a robot farm, actuator benches can be registered as specialized HIL resources. CI jobs request capabilities such as motor type, controller family, communication interface, encoder configuration, load fixture, or measurement equipment, and the scheduler assigns a compatible station. This abstraction allows low-level actuator regression tests to execute independently from complete robots, preserving full platforms for higher-level navigation, manipulation, and mission testing.

The collected results can feed directly into the CI quality gate. Software changes that satisfy control accuracy, timing, safety, and hardware-health requirements can progress toward higher-level HIL tests, while regressions are blocked before reaching complete robot experiments. This layered strategy reduces debugging complexity because failures are detected close to the motor, controller, or actuator subsystem where their physical effects first become observable.

Automated motor and actuator HIL testing ultimately converts physical motion into a reproducible part of software verification. By combining deterministic commands, real production hardware, independent measurements, quantitative acceptance criteria, safety supervision, traceable artifacts, and automated recovery, robotics teams can continuously verify that changes in software and firmware still produce acceptable behavior in the mechanisms that physically move the robot.

10.5. Sensor Replay Based HIL Test MCAP Bag Playback [w/Code]

Sensor replay-based Hardware-in-the-Loop testing combines recorded robot sensor data with the actual computing hardware and production software stack used on the robot. Instead of requiring cameras, LiDAR, radar, GNSS, or IMUs to observe a live environment for every regression test, previously captured sensor streams are replayed through ROS 2 interfaces. This creates repeatable inputs while preserving hardware-dependent execution characteristics.

The method occupies an important position between simulation and complete physical robot testing. Simulation can generate controlled scenarios but may not reproduce every characteristic of real sensors, while live HIL experiments contain natural environmental variation. Sensor replay preserves measurements captured from real operation and feeds them repeatedly into target hardware, allowing software revisions to be evaluated against essentially the same observation sequence.

ROS 2 bag recording provides the basic mechanism for capturing and replaying topic-oriented robot data. A recording can contain camera images, point clouds, IMU measurements, odometry, GNSS information, transforms, diagnostics, control states, and other ROS 2 messages. When the same recording is replayed, downstream perception, localization, mapping, sensor-fusion, planning, and AI components can process a consistent stream without requiring the original physical environment.

MCAP provides a container format well suited to large robotics datasets and structured message streams. In ROS 2 workflows, MCAP-backed bag files can preserve multiple topics, timestamps, schemas, and associated metadata within a single recording structure. This makes the format useful for HIL regression testing where large camera images, dense LiDAR point clouds, high-rate IMU measurements, and lower-rate system information must remain temporally associated.

The quality of replay testing depends strongly on how the original data is recorded. A useful HIL dataset should capture the inputs required by the target software together with contextual information needed to reconstruct the experiment. Topic names, message types, frame identifiers, transforms, timestamps, calibration data, software configuration, sensor configuration, and scenario descriptions should be preserved so that the recording remains interpretable after the original experiment has ended.

Scenario selection determines the practical value of the replay library. Rather than recording only nominal operation, teams can preserve difficult intersections, reflective surfaces, low-light scenes, localization degradation, dynamic obstacles, GNSS disturbances, unusual object configurations, sensor dropouts, and previously discovered field failures. Each important real-world failure can become a permanent regression scenario that future software revisions must process successfully.

A replay-based HIL pipeline begins by provisioning the target robot computer or dedicated edge platform with the candidate software. The orchestration system then loads a selected recording, configures the ROS 2 graph, disables or isolates conflicting live sensor sources, and prepares downstream nodes to receive replayed data. The objective is to replace selected physical sensor inputs while keeping the production processing path as unchanged as practical.

Topic remapping can isolate recorded data from live interfaces without requiring application software to be rewritten. A recorded camera topic, for example, can be mapped to the logical input expected by a perception node, while the physical camera driver remains disabled. The same principle applies to LiDAR, radar, GNSS, IMU, odometry, or diagnostic streams, enabling a generic test configuration to support several robot and sensor arrangements.

Time handling is one of the most important design considerations in replay testing. Robotics algorithms frequently depend on message timestamps, transform availability, synchronization windows, timeout logic, and relative timing between sensors. The test environment must therefore define whether nodes operate from replayed simulation time or wall-clock time and ensure that the chosen clock model is applied consistently across the ROS 2 graph.

Playback rate can be controlled according to the objective of the test. Real-time playback evaluates whether the target computer can process recorded workloads at operational speed, while slower playback can support debugging or detailed analysis. Faster-than-real-time playback may be useful for selected offline algorithms, but it should not be interpreted as equivalent to operational HIL when scheduling, latency, buffering, or real-time processing behavior is part of the requirement.

Multi-sensor synchronization must be preserved when testing perception and localization pipelines. Camera frames, LiDAR scans, IMU samples, GNSS observations, radar detections, and odometry may arrive at different frequencies but represent related physical events. If replay alters their temporal relationships, failures may reflect the playback mechanism rather than the candidate software. Original timestamps and ordering therefore form part of the test evidence.

Replay also enables controlled fault injection. A test framework can intentionally omit selected messages, delay a topic, alter playback timing, interrupt a stream, or substitute a predefined degraded sequence to evaluate software behavior under abnormal sensing conditions. Such tests can verify timeout handling, sensor-health monitoring, fallback behavior, degraded operating modes, and recovery when valid data becomes available again.

Perception regression testing is a major application of sensor replay HIL. Recorded camera or LiDAR sequences can be processed by a candidate detector, segmenter, tracker, occupancy estimator, or multimodal perception stack on the target GPU. The resulting detections and inference timing can be compared with reference outputs or acceptance thresholds, exposing both algorithmic changes and hardware-dependent performance regressions.

Localization and SLAM systems also benefit from deterministic replay. The same sensor sequence can be processed by different software commits while trajectory estimates, map consistency, localization confidence, transform continuity, and processing latency are recorded. When suitable reference trajectories or landmarks are available, quantitative errors can be calculated; otherwise, regression metrics can compare candidate behavior with an established baseline.

Sensor-fusion testing requires special attention because synchronization itself is often part of the algorithm being evaluated. A fusion node may combine IMU, wheel odometry, GNSS, LiDAR, radar, and camera information using timestamp-dependent buffers. Replay-based HIL allows the same temporal sequence to be presented repeatedly, making changes in synchronization policy, filtering, queue sizes, or computational load observable across software revisions.

The actual target hardware remains important even though the physical sensors are replaced by recorded streams. CPU scheduling, GPU inference, accelerator behavior, memory bandwidth, storage throughput, ROS 2 middleware, serialization, and concurrent processes still influence execution. Replay HIL therefore reveals performance limitations that may not appear when the same dataset is processed offline on a powerful development workstation.

Resource monitoring should accompany functional evaluation. CPU and GPU utilization, memory consumption, inference latency, dropped messages, queue growth, disk throughput, and processing frequency can be captured while the recording is replayed. A candidate build may produce equivalent perception outputs yet consume substantially more resources, creating a regression that could affect other robot functions during real operation.

Deterministic evaluation requires a clearly versioned replay dataset. Each scenario should have a stable identifier and be associated with its recording, calibration information, metadata, expected outputs, evaluation configuration, and acceptance criteria. If recordings are modified or replaced without version tracking, differences in CI results may be incorrectly attributed to software changes rather than changes in the test input itself.

Large MCAP or bag collections require storage and data-management policies similar to other engineering datasets. Frequently executed regression scenarios may be cached near HIL runners, while larger archives can remain in centralized object storage or NAS infrastructure. Dataset integrity checks, retention policies, compression, indexing, and metadata catalogs help ensure that CI jobs retrieve the correct recording without silently using incomplete or corrupted files.

Test results should connect the replay input with the complete execution configuration. The CI record can associate the Git commit, container image, ROS 2 package versions, AI model, target hardware identity, firmware, recording identifier, playback configuration, parameter files, output topics, performance metrics, and final verdict. This traceability makes a failed replay test reproducible on the same or an equivalent HIL station.

Failure artifacts should preserve enough information for diagnosis without unnecessarily duplicating every source recording. Logs, output topics, selected message windows, performance traces, screenshots, generated maps, model predictions, and comparison reports can be stored with the CI job. The immutable source MCAP or bag can be referenced by dataset identity, allowing engineers to reconstruct the execution while controlling storage growth.

Replay-based tests can form a progressive quality gate. Fast and representative recordings can run for every important software merge, while larger scenario libraries execute nightly or before releases. Builds that pass sensor replay HIL can then progress to complete physical robot tests where real sensor acquisition, actuator behavior, environmental interaction, and closed-loop motion are evaluated together.

Sensor replay cannot replace live hardware testing because recorded data does not respond to actions produced by the current software. A planner changing the robot trajectory cannot cause a recorded camera or LiDAR sequence to observe a different environment. Replay is therefore fundamentally an open-loop or partially integrated technique for many applications, and closed-loop physical behavior still requires simulation with interactive environments or real robot execution.

Its major strength is reproducibility. A difficult field event can be captured once, converted into a managed regression asset, and executed repeatedly against future software and AI models on actual target compute hardware. By combining MCAP or ROS 2 bag playback, controlled timing, topic remapping, versioned scenarios, automated evaluation, performance monitoring, and CI traceability, sensor replay HIL turns real-world experience into continuously reusable verification evidence.

10.6. Navigation and SLAM Regression Test Automation [w/Code]

Navigation and SLAM regression test automation verifies that changes to localization, mapping, planning, and motion software do not degrade previously validated robot behavior. Instead of relying on engineers to repeatedly drive a robot through the same route and inspect results manually, predefined scenarios are executed automatically and evaluated against quantitative baselines. This turns navigation performance into measurable evidence that can participate directly in continuous integration.

The test architecture should separate navigation functions into observable stages while still preserving end-to-end behavior. Sensor processing supplies environmental observations, localization estimates the robot pose, SLAM maintains a map or trajectory, planners generate paths, and controllers convert those paths into motion commands. Automated regression testing monitors these interfaces together so that a change in one component can be related to its downstream physical effects.

Regression scenarios should represent the operating conditions that matter to the robot rather than only simple demonstrations. Tests can include straight corridors, intersections, narrow passages, static obstacles, dynamic obstacles, repeated loops, docking approaches, localization recovery, and route replanning. Difficult field situations discovered during deployment should be preserved as permanent regression cases so that previously solved failures do not silently return in later software versions.

Testing can begin with deterministic sensor replay before progressing to complete physical navigation. Recorded MCAP or ROS 2 bag sequences provide repeatable camera, LiDAR, IMU, GNSS, odometry, and diagnostic inputs to localization and SLAM components. Because every candidate build processes essentially the same observations, differences in estimated trajectories, maps, processing latency, or resource usage can be attributed more reliably to changes in software or configuration.

Localization regression testing focuses on the stability and accuracy of the estimated robot pose. Automated evaluation can measure absolute position error when an external reference is available, relative trajectory consistency, heading error, pose jumps, covariance behavior, transform continuity, and recovery after temporary degradation. Tests should also detect cases where localization appears active but gradually diverges from the robot\'s actual physical position.

SLAM evaluation extends these measurements to the generated map and trajectory. A candidate build can be compared with a validated baseline using trajectory error, loop-closure behavior, map consistency, landmark alignment, occupancy changes, or other application-specific metrics. Repeated execution is particularly useful because mapping failures may emerge only after accumulated drift, incorrect loop closure, synchronization errors, or computational overload.

Reference data significantly improves regression quality. Ground-truth trajectories may be obtained from motion-capture systems, surveyed landmarks, high-accuracy GNSS/RTK, external tracking systems, or carefully validated reference runs. The appropriate reference depends on the operating environment, but it should remain independent enough from the localization algorithm under test to avoid evaluating an estimator solely against its own internal state.

Navigation planning tests evaluate whether the system produces feasible and consistent paths under predefined conditions. The framework can provide identical start poses, goals, maps, costmaps, and obstacle configurations, then compare path validity, path length, planning time, clearance, replanning frequency, and goal success. Exact path equality is usually unnecessary because multiple valid trajectories may satisfy the same navigation objective.

Controller regression testing examines how planned paths become physical motion. During HIL execution, the robot can follow predefined routes while cross-track error, heading error, velocity tracking, oscillation, acceleration, stopping accuracy, and goal-position tolerance are measured. This physical stage reveals drivetrain, timing, and control interactions that sensor replay or purely simulated navigation cannot reproduce completely.

A complete automated navigation test should define explicit initial conditions. The robot\'s starting position, map version, localization state, software configuration, sensor calibration, battery condition, test-zone configuration, and navigation goal should be recorded before execution. If the starting state varies significantly between runs, differences in results may reflect the test setup rather than an actual software regression.

Environment repeatability is equally important. A physical HIL navigation area can contain fixed reference markers, standardized obstacle locations, docking targets, corridor structures, and measured checkpoints. Movable fixtures should have known placement references so that the environment can be restored between tests. External cameras or tracking systems can independently verify robot motion and provide evidence when internal localization outputs are questionable.

Dynamic obstacle testing introduces controlled environmental variation without sacrificing reproducibility. Automated obstacles, scripted moving targets, or carefully orchestrated test agents can cross predefined regions at known times. The navigation stack can then be evaluated for detection, local replanning, deceleration, stopping, clearance, and mission recovery while the infrastructure ensures that independent safety systems protect the physical test area.

Failure and recovery behavior should be tested deliberately rather than waiting for accidental faults. Selected sensor streams can be interrupted, localization can be initialized with controlled offsets, transforms can be delayed, or navigation goals can be made temporarily unreachable. The expected result may be safe stopping, degraded operation, replanning, localization recovery, or explicit mission failure rather than uncontrolled continuation.

Timing metrics are essential because navigation is a real-time distributed workload. Planning latency, localization update rate, controller frequency, sensor-processing delay, transform age, message loss, and action completion time should be monitored during regression tests. A software revision may generate geometrically correct paths while becoming computationally slower, eventually producing unstable control or delayed responses on the target robot computer.

CPU, GPU, memory, and network utilization should therefore be captured alongside navigation metrics. New perception or SLAM algorithms may improve one metric while consuming resources required by controllers or safety-related processes. Regression automation can identify these tradeoffs by comparing resource envelopes between a candidate build and an accepted baseline on the same target hardware.

Pass-or-fail evaluation should use engineering tolerances instead of requiring identical trajectories. Physical robots experience wheel slip, sensor noise, friction, communication jitter, and small differences in initial pose. Thresholds can therefore define acceptable localization error, trajectory deviation, obstacle clearance, planning latency, stopping position, mission duration, or success rate while still identifying meaningful degradation.

Repeated trials help distinguish genuine regressions from normal physical variation. Important scenarios can be executed several times, and distributions of trajectory error, completion time, localization stability, or control performance can be compared with established limits. A single unusual run can be retained for investigation without automatically being treated as equivalent to a consistent performance deterioration across repeated executions.

Automated evaluation should preserve intermediate results rather than only producing a final pass-or-fail label. Estimated trajectories, planned paths, costmaps, maps, localization states, controller commands, diagnostic messages, system metrics, and external reference measurements provide the evidence required to identify where a navigation regression originated. Visual overlays of baseline and candidate trajectories can further accelerate engineering analysis.

ROS 2 bags or MCAP recordings generated during physical tests provide reusable regression assets. When a physical navigation failure occurs, its sensor and system data can be preserved and later replayed against modified software without repeatedly reproducing the original environment. This creates a feedback cycle in which expensive physical failures become inexpensive deterministic tests for future development.

CI orchestration can organize navigation testing into progressively more expensive stages. Fast algorithmic and replay tests can run after important commits, simulation can evaluate broader scenario coverage, and physical HIL can verify selected high-value routes on real hardware. Only builds that satisfy earlier gates need to consume limited robot-farm time, improving both test throughput and fault isolation.

Within a robot farm, the scheduler can allocate a compatible AMR, test zone, sensor configuration, and external measurement system to a navigation job. The orchestration layer establishes the initial state, launches the ROS 2 navigation stack, sends goals, monitors execution, collects artifacts, and returns the robot to a known safe state. Failed recovery should quarantine the platform instead of allowing another test to begin from an uncertain condition.

Safety supervision must remain independent of the navigation software being evaluated. Emergency stops, safety scanners, speed limits, protected zones, collision-prevention mechanisms, and supervisory controllers should continue operating even if localization, planning, ROS 2 communication, or experimental software fails. Regression automation measures navigation behavior but does not replace the robot\'s independent protective architecture.

Traceability connects each navigation result with the exact software and physical configuration that produced it. Git commits, container images, ROS 2 package versions, maps, parameters, AI models, firmware, sensor calibration, robot identity, test-zone version, scenario identifier, reference data, and evaluation thresholds should be associated with the execution so that both successful and failed tests can later be reproduced.

Navigation and SLAM regression automation ultimately transforms robot mobility from a manually demonstrated capability into a continuously measured engineering property. By combining deterministic replay, controlled physical scenarios, independent reference measurements, quantitative metrics, repeated trials, automated CI evaluation, and traceable artifacts, teams can detect changes in localization, mapping, planning, and control before those regressions propagate into field operation.

10.7. Safety Function HIL Test E Stop SLS STO [w/Code]

Safety-function Hardware-in-the-Loop testing verifies that protective functions behave correctly when real controllers, drives, sensors, communication networks, and actuators are integrated with production robot software. Functions such as Emergency Stop, Safely-Limited Speed, and Safe Torque Off must be tested as physical system behaviors rather than software commands alone. HIL provides a controlled environment for measuring their activation, timing, state transitions, and recovery.

Emergency Stop, commonly abbreviated as E-Stop, is intended to initiate an emergency response when hazardous operation must be interrupted. In a robotic HIL environment, the test should verify the complete signal path from the physical emergency-stop device through safety inputs, safety logic, drive interfaces, and final motion response. The objective is to confirm that activation produces the defined protective state even when ordinary robot software is malfunctioning.

The architecture should maintain clear separation between functional robot control and the safety-related control path. Navigation, AI, ROS 2 nodes, motion planning, and ordinary motor commands may operate through the standard control stack, while safety functions are supervised by dedicated safety hardware or safety-rated control mechanisms. HIL testing should confirm that failure or freezing of the ordinary software stack cannot prevent the safety path from performing its intended function.

Safe Torque Off, or STO, prevents a drive from generating torque capable of producing commanded motor motion. HIL testing of STO should evaluate the actual drive state rather than merely checking whether a software flag has changed. The test infrastructure can observe STO input channels, drive status, motor current, encoder motion, controller diagnostics, and relevant electrical signals to confirm that torque-producing capability is removed as expected.

STO testing should also distinguish torque removal from mechanical stopping behavior. Removing motor torque does not inherently guarantee a controlled deceleration or holding position, particularly on slopes, vertical axes, or mechanisms with stored energy. The HIL scenario must therefore reflect the robot\'s mechanical architecture and verify interactions between STO, braking mechanisms, gravity, inertia, and other protective functions without assuming that one safety function performs another function\'s role.

Safely-Limited Speed, or SLS, supervises motion so that speed remains within a defined safety-related limit when the function is active. HIL testing should command representative motion while independently observing actual velocity. The test can verify activation thresholds, monitored speed, response to excessive velocity, transition into the required protective state, and restoration of normal operation after valid reset conditions have been satisfied.

Independent measurement is especially important for SLS because testing only the velocity value reported by the controller can hide common-mode errors. External encoders, reference sensors, motion-capture systems, or other suitable measurement equipment can provide a separate estimate of actual motion. The HIL evaluator can compare commanded velocity, controller-reported velocity, independently measured velocity, and safety-system state on a synchronized timeline.

Timing is a fundamental part of safety-function testing. The interval between an E-Stop event and the resulting protective response, or between an SLS violation and the corresponding safety action, can influence the distance or motion that occurs before the system reaches its required state. HIL instrumentation should therefore capture high-resolution timestamps across input activation, safety logic transitions, drive response, motor behavior, and final motion.

A deterministic test sequence begins from a verified safe initial condition. The infrastructure checks safety-controller communication, drive readiness, E-Stop state, STO status, sensor availability, actuator condition, and test-zone readiness before enabling motion. The robot is then placed into a predefined operating state so that repeated tests begin from sufficiently equivalent conditions and meaningful timing or stopping measurements can be compared.

Automated E-Stop tests can activate the emergency-stop input during several controlled operating conditions. The robot may be stationary, moving at a defined velocity, turning, carrying a representative load, or executing a predefined actuator sequence. Measurements can capture detection time, safety-state transition, drive response, velocity decay, final motion state, diagnostic reporting, and whether normal motion commands remain ineffective while the emergency condition persists.

Automated SLS tests can evaluate behavior below, near, and beyond the configured speed boundary. Commands within the permitted range should not cause unnecessary intervention, while a controlled violation should produce the defined safety response. Testing around the threshold is particularly useful for detecting incorrect scaling, configuration mismatches, measurement noise sensitivity, delayed supervision, or inconsistent behavior between forward and reverse motion.

Automated STO tests can exercise multiple operating states, including stationary drives, rotating motors, and predefined motion sequences. The evaluator can verify whether STO activation changes the drive state correctly and whether torque-producing commands remain ineffective while STO is active. Repeated activation also helps expose intermittent wiring, communication, drive-interface, or state-management problems that may not appear during a single manual test.

Fault injection extends testing beyond nominal activation. Selected input channels, communication paths, feedback signals, or state transitions can be disturbed within the limits of the test architecture to verify diagnostic behavior and fault responses. The objective is to determine whether the integrated system detects relevant abnormalities and moves toward its specified protective behavior rather than continuing operation based on invalid safety information.

Reset and restart behavior requires separate verification because a successful protective stop does not automatically imply safe recovery. After an E-Stop release, STO reset, or SLS-related event, the system should follow the defined restart sequence and should not unexpectedly resume hazardous motion. HIL automation can verify required acknowledgments, state transitions, command inhibition, controller readiness, and explicit conditions necessary before motion becomes available again.

The distinction between reset and restart should remain visible in test logic. Resetting a safety condition may clear or acknowledge a protective state, but it should not be assumed to authorize automatic motion in every architecture. Automated tests should therefore examine whether stored motion commands, pending navigation goals, actuator requests, or queued actions can produce unintended movement immediately after the safety system returns to an operational state.

ROS 2 and higher-level software can participate in safety HIL by recording commands, system states, diagnostics, and mission behavior, but they should not be treated as the sole source of safety evidence. The test framework should correlate ROS 2 information with safety-controller states, drive signals, physical measurements, and independent instrumentation. This creates an end-to-end record connecting application behavior with the actual protective response of the machine.

Pass-or-fail criteria should be based on explicitly defined system requirements. Relevant metrics can include activation recognition, state-transition completion, maximum permitted speed during SLS operation, response timing, residual motion, command inhibition during STO, diagnostic correctness, and restart behavior. Acceptance thresholds should originate from the robot\'s safety design and validation requirements rather than arbitrary values introduced by the CI framework.

Physical variation must be considered when evaluating stopping behavior. Vehicle mass, payload, floor friction, tire condition, battery voltage, actuator temperature, gradient, and initial velocity can influence the measured response. HIL records should therefore preserve relevant operating conditions so that changes in stopping distance or response time are not incorrectly attributed to software when the physical test configuration has materially changed.

Safety HIL infrastructure requires protection independent of the function currently under test. Testing an E-Stop cannot rely exclusively on that same E-Stop as the laboratory\'s final protective mechanism. Test cells may require separate emergency isolation, guarded areas, safety scanners, physical restraints, remote supervision, restricted energy, or other independent measures capable of bringing the experiment to a safe condition if the tested function fails.

Evidence collection is essential because safety tests may need later engineering review. The system should preserve synchronized safety-controller states, drive diagnostics, encoder traces, velocity measurements, current measurements, ROS 2 logs, event timestamps, configuration versions, and test observations. Video synchronized with measurement data can provide additional context for unexpected movement, mechanical behavior, or differences between commanded and actual responses.

Configuration traceability is particularly important for safety functions. The test record should identify the safety-controller configuration, drive parameters, firmware versions, safety limits, robot hardware revision, software build, sensor configuration, test procedure, and relevant parameter sets. A successful result is meaningful only when engineers can determine exactly which integrated configuration was tested and reproduce it when necessary.

Automated CI can schedule safety-function HIL after lower-risk software tests have completed successfully. Unit tests, integration tests, simulation, sensor replay, and ordinary actuator HIL can filter basic defects before physical safety scenarios consume specialized test resources. Safety HIL then becomes a controlled validation gate for selected builds rather than an uncontrolled experiment executed after every minor source-code modification.

Automation improves repeatability but does not replace formal machinery safety engineering or required validation activities. HIL results provide evidence about the tested integrated configuration and can reveal regressions in safety-related behavior, interfaces, timing, and diagnostics. The interpretation of those results must remain connected to the system\'s risk assessment, safety requirements, architecture, and applicable engineering lifecycle.

Safety-function HIL testing ultimately creates a measurable bridge between safety logic and physical robot behavior. By combining controlled E-Stop activation, SLS supervision, STO verification, independent measurements, synchronized timing, fault injection, recovery testing, configuration traceability, and protected test infrastructure, robotics teams can repeatedly verify that critical protective functions continue to behave as designed as software, firmware, and hardware evolve.

10.8. AI Model HIL Evaluation Perception Accuracy [w/Code]

AI model Hardware-in-the-Loop evaluation verifies whether perception models maintain required accuracy and execution performance when deployed on the actual computing hardware used by a robot. Offline dataset evaluation measures algorithmic quality, but it does not fully represent GPU behavior, memory pressure, preprocessing pipelines, ROS 2 communication, sensor timing, or concurrent workloads. HIL connects model accuracy with the conditions under which perception actually operates.

The evaluation pipeline should preserve the production inference path as closely as practical. Sensor messages pass through acquisition or replay, preprocessing, model inference, postprocessing, and ROS 2 output interfaces before results are evaluated. This approach tests more than the neural network itself because resizing, normalization, coordinate conversion, filtering, synchronization, and message serialization can all introduce errors that are invisible in isolated model benchmarks.

Perception HIL can use both recorded sensor sequences and live physical sensors. MCAP or ROS 2 bag replay provides deterministic camera, LiDAR, radar, and multimodal inputs that can be repeated across model versions. Live testing complements replay by exposing the system to changing illumination, motion blur, reflections, vibration, environmental geometry, and sensor behavior that cannot be completely represented by a fixed recording.

Reference annotations are required when quantitative perception accuracy is evaluated. Depending on the task, reference data may contain object classes, bounding boxes, segmentation masks, keypoints, tracks, depth, occupancy, or three-dimensional object poses. The evaluation framework associates these references with sensor timestamps and coordinate frames so that predictions can be compared with the correct physical observation.

Object detection evaluation can measure precision, recall, intersection over union, average precision, confidence behavior, and class-specific errors. A candidate model should not be judged only by one aggregate metric because a small overall change can hide substantial degradation for a safety-relevant or operationally important object class. HIL reports should therefore preserve per-class and scenario-specific results alongside overall model statistics.

Segmentation models require evaluation at the pixel or region level. Metrics such as intersection over union, class accuracy, boundary quality, and false-positive or false-negative regions can reveal whether environmental structures are being represented correctly. For mobile robots, errors affecting traversable space, obstacles, people, or restricted areas may be more operationally significant than equivalent errors in less important background regions.

Three-dimensional perception introduces additional geometric requirements. LiDAR-based or multimodal detectors can be evaluated using position error, orientation error, dimension error, three-dimensional overlap, range-dependent accuracy, and detection consistency. Coordinate transformations between sensor frames, robot frames, and map frames should also be verified because a correct model prediction can become physically incorrect after an erroneous calibration or transform is applied.

Tracking evaluation extends perception accuracy across time. Automated HIL tests can examine track continuity, identity switches, missed detections, false tracks, trajectory stability, and reacquisition after temporary occlusion. These temporal metrics are particularly important for navigation and interaction because downstream planners often consume object histories rather than independent frame-by-frame detections.

Perception performance should be analyzed across meaningful operating conditions. Scenario metadata can classify sequences by distance, object size, lighting, weather, motion, occlusion, background complexity, sensor modality, or robot speed. A model that improves average accuracy but becomes unreliable in low light or at long range may introduce a practical regression that would be hidden by a single dataset-wide score.

The actual target processor is a central part of AI HIL evaluation. GPU architecture, accelerator support, memory capacity, precision mode, runtime libraries, and thermal behavior can change both inference performance and numerical outputs. A model validated on a development workstation should therefore be tested again on the robot\'s edge computer using the same inference engine, model representation, and optimization settings intended for deployment.

Model conversion and optimization require explicit regression checks. Exporting a model through formats or runtimes such as ONNX or TensorRT can introduce precision changes, unsupported-operation substitutions, graph optimizations, quantization effects, or altered preprocessing assumptions. HIL evaluation should compare the deployed model against an accepted reference so that optimization does not silently produce unacceptable perception differences.

Latency is as important as accuracy for real-time perception. The system should measure sensor-to-model delay, preprocessing time, inference latency, postprocessing time, publication delay, and total end-to-end latency. Average latency alone is insufficient when occasional long delays can destabilize navigation or control, so percentile values, maximum observed latency, jitter, and deadline violations should also be captured.

Throughput evaluation determines whether the perception stack can sustain the required sensor rate. Camera frames or LiDAR scans may arrive faster than the model can process them, causing queues to grow or messages to be dropped. HIL monitoring should therefore relate input frequency, processing frequency, queue behavior, dropped frames, and output freshness rather than reporting only the maximum inference speed of the neural network.

Resource consumption must be evaluated simultaneously with model quality. GPU utilization, CPU utilization, memory usage, accelerator occupancy, power behavior, and temperature can reveal whether a new model leaves sufficient capacity for localization, planning, control, and other robot processes. An accuracy improvement may not represent a deployable improvement if it causes persistent resource saturation on the target platform.

Thermal effects become important during extended inference. A model may initially satisfy latency requirements but slow after the processor reaches a thermal limit. Long-duration HIL sequences can expose clock throttling, increasing latency, memory instability, or power-related performance changes. Monitoring temperature together with inference timing allows the test framework to distinguish algorithmic regression from hardware operating constraints.

Concurrent workload testing provides a more realistic deployment condition than running perception alone. Navigation, SLAM, logging, communication, visualization, and additional AI models may execute on the same computer. HIL can reproduce representative combinations of these workloads and determine whether perception accuracy, latency, throughput, or message delivery changes when the complete robot software stack competes for shared resources.

Multimodal AI models require synchronized evaluation of their input streams. Camera, LiDAR, radar, depth, and other observations may operate at different frequencies and latencies. The HIL framework should preserve or deliberately manipulate their temporal relationships to evaluate synchronization tolerance, missing-modality behavior, stale data handling, and degradation when one sensor becomes delayed or temporarily unavailable.

Controlled fault injection can test perception robustness beyond nominal accuracy. Frames can be dropped, sensor streams delayed, selected inputs removed, or predefined degraded recordings introduced. The expected response may include reduced confidence, diagnostic reporting, fallback to another modality, degraded operating mode, or explicit invalidation of perception outputs rather than continued publication of misleading high-confidence results.

Regression evaluation should compare candidate models with an approved baseline under identical scenarios. Differences in detections, segmentation, tracks, latency, memory use, and resource consumption can be calculated automatically. The objective is not necessarily to require identical predictions, but to determine whether the candidate remains within engineering tolerances defined for the robot\'s operational requirements.

Repeated physical tests are useful when live sensors are involved because environmental and sensor variation prevents perfectly identical inputs. Multiple runs allow accuracy, latency, and detection stability to be represented statistically rather than by a single measurement. Deterministic replay can then complement physical trials by confirming whether observed differences persist when the input sequence is held constant.

Pass-or-fail criteria should combine perception quality and deployment performance. A model may satisfy an accuracy threshold while violating latency, memory, thermal, or throughput requirements, or it may execute efficiently while producing unacceptable detection errors. The CI quality gate should therefore evaluate the model as a deployed robotic component rather than treating predictive accuracy and computing performance as unrelated properties.

Failure artifacts should contain sufficient evidence for model and system diagnosis. Predictions, confidence values, reference annotations, sensor frames, error cases, latency traces, resource measurements, ROS 2 messages, runtime logs, and visualization overlays can be retained. Engineers can then determine whether a failure originates from the model, preprocessing, calibration, synchronization, optimization runtime, hardware load, or downstream integration.

Traceability connects every evaluation result to the complete AI deployment configuration. The record should identify the Git commit, model version, training checkpoint, dataset or scenario version, preprocessing configuration, inference engine, precision mode, hardware identity, driver and runtime versions, ROS 2 packages, calibration data, evaluation thresholds, and generated artifacts. This allows a result to be reproduced and compared across future releases.

CI orchestration can organize AI evaluation into progressively more expensive stages. Fast offline model tests can screen basic regressions, deterministic sensor replay can validate the production inference stack on target hardware, and selected physical HIL scenarios can evaluate live sensing and integrated robot behavior. Only candidates that satisfy earlier gates need to consume scarce physical robot resources.

AI model HIL evaluation ultimately connects perception quality with deployability. By combining representative sensor data, reference annotations, target edge hardware, production inference pipelines, accuracy metrics, timing measurements, resource monitoring, fault injection, repeatable scenarios, and configuration traceability, robotics teams can verify that an AI model is not merely accurate in isolation but remains reliable when operating as part of the complete physical robot system.

10.9. HIL Result Reporting and CI Pass Fail Criteria

HIL result reporting converts physical test execution into structured engineering evidence that can be consumed by developers, reviewers, and Continuous Integration systems. A HIL job should not end with a simple statement that the robot moved correctly or that a test appeared successful. It should produce machine-readable results, quantitative measurements, execution context, diagnostic artifacts, and a clearly derived pass, fail, or infrastructure-related outcome.

The reporting architecture should separate raw observations from evaluated results. Raw data can include sensor streams, actuator states, controller messages, safety signals, system logs, timestamps, resource measurements, and external reference measurements. Evaluation logic transforms these observations into metrics and assertions, while the final report summarizes which requirements were tested, which criteria were satisfied, and which conditions caused failure.

Every report should identify the exact test configuration that produced the result. Relevant context includes the Git commit, software build, container image, ROS 2 package versions, model versions, firmware, robot identity, hardware revision, calibration data, parameter sets, scenario version, HIL station, and test start time. Without this traceability, a successful result cannot reliably demonstrate which integrated system configuration was actually validated.

Test cases should be connected to explicit engineering requirements whenever practical. A navigation test may verify localization accuracy or stopping position, while an actuator test may verify settling time or current limits. Safety tests may evaluate response timing and command inhibition, and AI tests may evaluate perception accuracy and inference latency. Requirement-linked reporting makes the meaning of a CI result clearer than isolated metric values.

Pass-or-fail criteria should be defined before test execution rather than selected after observing the result. Each criterion can specify a measured quantity, evaluation method, operating condition, acceptance range, and applicable test scenario. This prevents engineers from informally adjusting expectations to accommodate an unexpected result and enables the same engineering rule to be applied consistently across software revisions.

Not every criterion should be represented as a single fixed threshold. Some measurements require upper limits, lower limits, acceptable ranges, statistical distributions, or combinations of conditions. A localization error may require a maximum bound, an AI metric may require a minimum accuracy, and a thermal test may require both a temperature limit and sustained performance. The evaluation model should therefore support multiple forms of acceptance logic.

Physical HIL measurements contain natural variation that must be distinguished from meaningful regression. Wheel slip, friction, battery condition, sensor noise, environmental differences, mechanical tolerances, and communication jitter can change measured values between runs. Pass-or-fail logic should use engineering tolerances appropriate to the test rather than demanding numerical identity between executions that cannot physically be reproduced exactly.

Repeated trials can provide stronger evidence when physical variability is significant. Instead of evaluating only one stopping distance, trajectory error, or actuator response, the framework can evaluate several executions and calculate representative statistics. Acceptance rules may consider mean behavior, maximum values, variation, or the number of successful trials, provided these rules are defined by the relevant engineering requirements before evaluation.

Baseline comparison provides another method for detecting regressions. A candidate build can be executed under the same scenario as an accepted reference build, and differences in accuracy, latency, trajectory behavior, resource consumption, or control response can be calculated. Baselines should be versioned and explicitly identified so that the comparison does not silently change when reference software, hardware, parameters, or test data are updated.

Absolute requirements and regression criteria serve different purposes and can be evaluated together. A candidate may remain better than a previous baseline while still violating a required safety or performance limit, or it may satisfy the absolute requirement while showing a significant deterioration from previous behavior. Reports should preserve both perspectives rather than reducing all evaluation to a single relative comparison.

HIL infrastructure failures should be distinguished from product failures. A disconnected test sensor, unavailable robot, corrupted dataset, exhausted storage device, broken fixture, network outage, or invalid test initialization does not necessarily demonstrate a defect in the software under test. CI reporting should classify such executions separately so that infrastructure problems do not create misleading product-quality statistics or inappropriate release decisions.

Similarly, an aborted test should not automatically be treated as equivalent to a validated failure. Emergency intervention, facility shutdown, resource preemption, or loss of required instrumentation may prevent the test from producing sufficient evidence. The reporting system should record why execution ended, what data remained valid, and whether the job should be rerun rather than presenting an incomplete experiment as a conclusive engineering result.

Individual assertions should contribute to a hierarchical result structure. Measurements can be evaluated into assertions, assertions can belong to test cases, test cases can form scenarios, and scenarios can contribute to an overall HIL job. This hierarchy allows engineers to see that a complete job failed because one steering test exceeded its response-time limit instead of receiving only an opaque red status at the pipeline level.

Critical criteria may require different aggregation rules from ordinary performance criteria. A failure involving an explicitly required safety function may invalidate the test immediately, while a noncritical diagnostic warning may be recorded without failing the entire build. Such behavior should be determined by predefined requirement severity and test policy rather than by arbitrary logic embedded separately in individual CI scripts.

Result reports should include measured values alongside their acceptance limits. Reporting only "PASS" hides useful engineering information because a metric operating very close to its boundary may indicate decreasing margin. Showing the measured value, threshold, baseline, difference, and relevant uncertainty allows engineers to distinguish a comfortably passing system from one that is approaching a performance or safety limit.

Trend information extends HIL reporting beyond individual builds. Metrics such as localization error, inference latency, motor current, stopping distance, CPU load, or test duration can be tracked across commits and releases. A system may continue passing its formal threshold while gradually deteriorating. Trend analysis can expose this movement early and provide evidence for investigation before the quality gate is eventually crossed.

Artifacts should support rapid diagnosis when a result fails. Depending on the test, the report can link to ROS 2 bags or MCAP files, logs, command traces, sensor data, trajectories, maps, plots, videos, screenshots, model predictions, safety-controller events, and resource traces. Engineers should be able to move from a failed assertion to the evidence surrounding that failure without manually searching several unrelated storage systems.

Artifact retention should reflect both diagnostic value and storage cost. Small summaries and structured metrics may be retained for long periods, while large sensor recordings or videos may follow tiered retention policies. Important field failures, safety-related tests, release-validation evidence, and unusual regressions may justify longer preservation than routine passing runs. Metadata should remain available even when large artifacts are eventually removed.

Machine-readable output allows CI systems to consume HIL results automatically. Structured formats can represent test identifiers, status, duration, measurements, thresholds, failure messages, and artifact references. Human-readable dashboards or reports can then be generated from the same underlying evidence, reducing inconsistencies between what the automation evaluates and what engineers see during review.

The CI quality gate should aggregate HIL evidence according to an explicit policy. A build can advance when required tests have completed and mandatory acceptance criteria have been satisfied. Failed mandatory criteria can block progression, while infrastructure errors can request reruns or operator intervention. Optional or experimental tests can report information without necessarily controlling the release path until they become formally required.

Quality gates can also be staged according to test cost and risk. Fast sensor replay or component HIL can run frequently, while full navigation, safety, AI, or multi-robot tests may execute for selected builds or release candidates. Results from earlier stages determine whether a build is eligible for more expensive physical testing, preventing scarce robot resources from being consumed by software that already failed lower-level verification.

Flaky tests require explicit management because repeated inconsistent results weaken confidence in CI. The system should record reruns, failure frequency, environmental conditions, and infrastructure state rather than simply retrying until a pass appears. Tests with unresolved instability can be identified for engineering investigation so that randomness does not become an unofficial mechanism for bypassing the quality gate.

Manual overrides, when permitted by organizational policy, should remain visible and traceable. The original automated result should not be erased when an engineer authorizes continued testing or release for a documented reason. The report can preserve the failed criterion, responsible approval, justification, and subsequent action, ensuring that exceptional decisions remain distinguishable from genuine automated passes.

Dashboards should present information at several levels without hiding the underlying evidence. Teams may need an overall HIL status for rapid CI decisions, scenario-level summaries for subsystem owners, metric trends for performance analysis, and detailed artifacts for debugging. Effective reporting therefore connects high-level status with progressively deeper evidence rather than replacing engineering detail with a single status indicator.

HIL result reporting and CI pass-fail criteria ultimately transform complex physical experiments into reproducible quality decisions. By combining predefined requirements, quantitative thresholds, statistical treatment, baseline comparisons, failure classification, configuration traceability, diagnostic artifacts, trend monitoring, and explicit quality-gate policies, robotics teams can determine whether an integrated build is ready to progress while preserving the evidence needed to understand every decision.

10.10. Multi Robot HIL Parallel Test Execution Strategy

Multi-robot HIL parallel test execution extends Hardware-in-the-Loop validation from a single robot or test bench into a shared infrastructure where multiple physical platforms can execute independent or coordinated tests simultaneously. The objective is not simply to run more robots at once, but to increase test throughput while preserving isolation, repeatability, safety, traceability, and confidence in the result produced by every individual HIL job.

A parallel HIL architecture typically combines a CI orchestration layer with a robot-farm resource manager. Incoming jobs describe the hardware capabilities they require, such as robot type, compute platform, sensor suite, actuator configuration, test zone, charging state, or external measurement equipment. The scheduler matches these requirements to available resources and reserves the necessary combination before deployment begins.

Resource modeling should describe capabilities rather than relying only on fixed robot names. One test may require an AMR with LiDAR and an edge GPU, while another may require a manipulator, safety controller, and force sensor. Capability-based scheduling allows equivalent platforms to execute the same test and prevents CI pipelines from becoming unnecessarily dependent on one specific physical machine.

Parallel execution requires strong isolation between test jobs. Each robot should receive the intended software build, parameters, models, maps, credentials, and test scenario without contamination from another execution. ROS 2 domains, namespaces, network segmentation, container boundaries, dedicated workspaces, or equivalent mechanisms can prevent messages, configuration files, and runtime state from crossing between concurrent HIL sessions.

Network design becomes increasingly important as the number of simultaneous robots grows. Multiple platforms may stream camera images, LiDAR point clouds, logs, telemetry, and test artifacts while receiving software deployments and commands. The infrastructure should provide sufficient bandwidth and predictable communication behavior so that network congestion created by one HIL job does not cause timing failures or message loss in another unrelated test.

Physical test-space allocation must also be treated as a schedulable resource. Two robots cannot safely execute independent navigation tests in the same area unless the scenario explicitly supports coordinated operation. Test zones, lanes, docking stations, motion envelopes, safety barriers, and external tracking regions can therefore be represented as resources that are reserved together with the robot itself.

A reservation should be atomic when several resources are required simultaneously. A navigation test may need one AMR, one physical test zone, an RTK reference system, and an external camera. Starting the job after reserving only the robot can create deadlock or idle time if the remaining resources are unavailable. The scheduler should allocate the required resource set before the test enters its physical execution phase.

Robot health must be considered during scheduling. A platform that is technically idle may still be unsuitable because of low battery state, excessive temperature, unresolved faults, damaged sensors, calibration expiration, or maintenance requirements. Pre-test health checks allow the scheduler to distinguish between available, temporarily unavailable, quarantined, charging, maintenance, and ready-to-test resources.

Queue policies determine how limited physical resources are shared among development teams and CI pipelines. Jobs may be prioritized according to release criticality, failure investigation, normal regression testing, or experimental work. Fairness controls can prevent one project from continuously occupying the robot farm, while priority mechanisms allow urgent release-validation or defect-reproduction jobs to receive appropriate access.

Parallelism can occur at several levels. Independent tests can run simultaneously on different robots, different actuator benches can execute component tests, and multiple edge computers can process sensor-replay workloads without requiring physical motion. Separating these resource classes allows the infrastructure to exploit parallel capacity without unnecessarily reserving a complete robot for tests that need only a controller, GPU, or sensor interface.

Test sharding can further reduce regression duration. A large scenario suite can be divided into independent groups and distributed across compatible robots or benches. If twenty navigation scenarios do not depend on one another, several platforms can execute subsets concurrently and return their results to one CI job. Sharding should preserve scenario identity so that failures remain attributable to the exact robot, configuration, and execution.

Not every test is suitable for parallel execution. Long endurance experiments, shared-environment navigation, radio-interference tests, coordinated fleet behavior, or tests requiring unique instrumentation may need exclusive access. The scheduler should therefore support resource constraints and exclusivity rules instead of assuming that maximum concurrency is always the correct operating strategy.

Multi-robot interaction tests form a different class from independent parallel tests. Fleet coordination, traffic management, intersection negotiation, task allocation, and collision avoidance may intentionally require several robots in one scenario. In these cases, the scheduler reserves a group of compatible robots and a shared test area, while the orchestration layer launches synchronized software configurations and coordinates scenario start conditions.

Synchronized execution is important when several robots participate in the same test. Clocks, scenario triggers, logging, external measurements, and event timestamps should be sufficiently aligned to reconstruct interactions after execution. Common time references make it possible to determine which robot entered a region first, when a fleet command was issued, and whether communication or planning delays contributed to an observed behavior.

Safety supervision must scale with parallelism. Each robot requires an independently identifiable protective state, while facility-level mechanisms must account for interactions between platforms. Emergency stops, safety scanners, speed limits, geofenced zones, remote supervision, and test-area access control should remain effective even if the software under test, CI network, or orchestration service becomes unavailable.

A facility-wide emergency event should be distinguished from a robot-specific test failure. Stopping every robot because a person enters a protected test area does not necessarily indicate that all concurrent software builds failed. The orchestration system should preserve the reason for interruption and classify affected jobs appropriately so that environmental or infrastructure events do not become misleading product regressions.

Test deployment should be transactional whenever practical. Before execution, the system verifies that the correct software, container image, AI model, firmware compatibility, parameters, maps, and scenario assets are present on the assigned platform. If preparation fails, the robot should not enter motion. This prevents partially updated systems from producing ambiguous results or interacting unpredictably with other active HIL jobs.

Data collection must preserve isolation while supporting centralized analysis. Each execution should have a unique job identifier associated with logs, ROS 2 bags or MCAP recordings, metrics, videos, configuration snapshots, and evaluation reports. Central storage can aggregate these artifacts after execution, but the metadata must retain the robot identity, test station, scenario, timestamps, and software configuration that generated them.

Parallel HIL creates substantial storage and compute demand. Several robots recording high-rate cameras and LiDAR while running AI inference can generate large data streams simultaneously. Local buffering near each test station can protect execution from temporary storage congestion, while asynchronous artifact upload prevents centralized storage performance from directly controlling the timing of physical robot tests.

Result aggregation should maintain both individual and suite-level visibility. A parallel CI job may contain dozens of scenarios distributed across several platforms, but one failure must remain traceable to its exact execution context. The overall pipeline can summarize completion and quality-gate status while retaining per-robot, per-scenario, and per-assertion results for engineering diagnosis.

Infrastructure failures should trigger targeted recovery rather than unnecessary repetition of the entire suite. If one robot loses network connectivity while five other shards complete successfully, only the affected shard may need rescheduling when the completed results remain valid. This reduces robot-farm occupancy and prevents expensive physical tests from being repeated solely because an unrelated resource failed.

Post-test recovery determines whether a resource can safely return to the scheduling pool. The robot should stop, clear temporary test state, preserve required artifacts, restore the expected software or configuration state, and complete health checks. Charging or thermal cooldown may be required before another job begins. Resources that cannot reach the defined ready state should be quarantined automatically.

Scheduling efficiency can be improved using historical execution data. Expected test duration, battery consumption, thermal load, setup time, failure frequency, and required recovery time can help the orchestrator reduce idle periods and avoid unsuitable job sequences. These estimates should support resource planning rather than altering engineering acceptance criteria, which remain defined independently by test requirements.

Observability should cover the robot farm itself as well as individual tests. Operators need visibility into queue depth, resource utilization, robot readiness, charging state, test-zone occupancy, infrastructure failures, current executions, and upcoming reservations. This operational view complements CI test reports and helps distinguish a product regression from a capacity, maintenance, or facility problem.

Traceability becomes more important as concurrency increases. Every result should connect the Git commit, build, container, model, firmware, robot identity, hardware revision, calibration, scenario, test-zone version, scheduler allocation, start time, and generated artifacts. Without this information, parallel execution can produce large volumes of results that are difficult to reproduce or compare reliably.

Multi-robot HIL parallel execution ultimately transforms a collection of physical robots into a managed verification infrastructure. Capability-based scheduling, resource isolation, atomic reservation, test sharding, synchronized multi-robot scenarios, independent safety supervision, structured artifact collection, automated recovery, and complete traceability allow CI systems to increase physical test throughput without sacrificing the engineering integrity of each result.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)