DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Perception and Sensor Fusion — Chapter 10: Tactile and Force Perception [EN]

Chapter 10. Tactile and Force Perception

10.01. Tactile Sensor Technologies Capacitive Resistive Optical

Tactile perception provides robots with information that cannot be obtained reliably from vision, LiDAR, or other remote sensing modalities. It measures physical interaction at the contact interface, allowing a robot to estimate whether contact has occurred, where it occurs, and how pressure or deformation is distributed. This capability is fundamental to manipulation, grasping, human--robot interaction, and contact-rich Physical AI systems.

A tactile sensor converts mechanical interaction into measurable electrical or optical signals. Depending on its construction, the measured quantity may represent normal pressure, shear force, deformation, vibration, contact geometry, or combinations of these variables. Unlike a conventional force--torque sensor that usually measures aggregate loads at a joint or wrist, tactile arrays provide spatially distributed measurements across a contact surface.

The sensor architecture must therefore be selected according to the required spatial resolution, force range, bandwidth, mechanical compliance, durability, and integration constraints. A robotic fingertip used for precision manipulation may require dense sensing of small surface features, while a humanoid skin system may prioritize large-area coverage, robustness, flexibility, and safe contact detection. No single tactile technology optimally satisfies every requirement.

Capacitive tactile sensors measure changes in capacitance produced by mechanical deformation. A typical sensing element contains conductive electrodes separated by a dielectric material. When external pressure changes the electrode spacing or effective overlapping area, the capacitance changes accordingly. An array of these elements can form a pressure-sensitive surface capable of resolving the spatial distribution of contact forces.

Capacitive sensing offers high sensitivity, relatively low power consumption, and good compatibility with flexible substrates. It can detect small deformations and is therefore useful in robotic fingertips, artificial skin, wearable sensing surfaces, and compliant grippers. Dense arrays can provide pressure maps that support contact localization, object shape estimation, and grasp stability assessment rather than merely reporting binary contact.

However, capacitive sensors are sensitive to parasitic capacitance, electromagnetic interference, environmental conditions, and changes in dielectric properties. Large arrays also require careful electrode routing and multiplexing because wiring and readout electronics can introduce additional capacitance. Shielding, baseline calibration, temperature compensation, and mechanically stable packaging are consequently important elements of practical sensor design.

Resistive tactile sensors convert mechanical loading into changes in electrical resistance. Force-sensitive resistors, piezoresistive materials, conductive polymers, elastomer composites, and strain-sensitive structures are commonly used implementations. Applied pressure changes the conductive paths or material geometry, producing a resistance variation that can be sampled using comparatively simple analog electronics.

The major advantages of resistive sensing are structural simplicity, low cost, thin form factors, and straightforward integration into large sensing surfaces. These properties make resistive arrays attractive for gripper pads, robot skin, collision detection surfaces, and applications where exact reconstruction of fine contact geometry is less important than reliable detection of pressure magnitude and distribution.

Their limitations include hysteresis, nonlinear response, drift, creep, and variability between sensing elements. Repeated loading may also change material characteristics over time. Consequently, raw resistance should not automatically be interpreted as an accurate physical force. Calibration curves, normalization, filtering, compensation models, and periodic recalibration may be necessary when quantitative force estimation is required.

Optical tactile sensing follows a substantially different principle. Instead of directly measuring an electrical property modified by contact, an internal camera observes the deformation of a compliant transparent or translucent surface. Illumination systems reveal changes in geometry, texture, markers, or reflected light patterns, and computer vision algorithms transform these images into representations of the physical contact state.

Optical sensors can achieve exceptionally dense spatial measurements because their resolution is largely determined by the elastomer structure, optical configuration, and camera resolution. Systems based on concepts such as GelSight or DIGIT can capture fine contact geometry that resembles a high-resolution image of the touched surface. This enables perception of edges, textures, local curvature, contact patches, and subtle object features.

The resulting tactile image can be processed using conventional and learning-based vision techniques. Convolutional networks or vision transformers may infer contact state, material properties, object identity, pose correction, slip, or grasp quality directly from image sequences. Optical tactile sensing therefore creates a particularly strong connection between tactile perception and modern visual representation learning.

Optical designs nevertheless introduce their own engineering constraints. Cameras, illumination, elastomers, protective membranes, and internal optical paths occupy physical volume and must remain mechanically aligned. Elastomer wear or contamination can alter image characteristics, while illumination variation and material aging may produce domain shifts. High-resolution tactile video also generates substantially more data than simple pressure arrays.

The three technologies consequently occupy different design regions rather than forming a simple ranking. Capacitive sensing is attractive when sensitivity, flexible construction, and distributed pressure measurement are important. Resistive sensing favors inexpensive and mechanically simple implementations. Optical sensing becomes valuable when extremely rich spatial information and fine contact geometry justify greater computational and mechanical complexity.

Tactile perception also depends on temporal behavior. Static pressure indicates how contact forces are distributed, while rapid signal variations may reveal impact, vibration, incipient slip, or transitions between sticking and sliding. Sensor bandwidth therefore determines which interaction phenomena can be observed. A manipulation system may combine slowly varying pressure estimates with high-frequency components to characterize both grasp force and dynamic stability.

Spatial resolution presents a related trade-off. Increasing the number of tactile elements improves localization and enables reconstruction of detailed contact patterns, but it increases wiring, calibration effort, bandwidth, computation, and failure points. Robotic skin covering an arm or humanoid body may therefore use relatively coarse distributed sensing, whereas fingertips can concentrate high-resolution sensing where manipulation demands greater precision.

Mechanical design is inseparable from tactile sensor performance. Protective layers influence sensitivity, contact area, friction, and force transmission before signals reach the sensing elements. A very soft surface can improve conformity around objects but may blur localized forces, while a harder surface preserves sharper contact patterns but provides less compliance. Sensor packaging must therefore be designed together with the intended interaction mechanics.

Calibration establishes the relationship between raw sensor measurements and physically meaningful quantities. This process may include zero-offset correction, gain normalization, force calibration, spatial calibration, temperature compensation, and compensation for hysteresis or drift. For array sensors, calibration must also consider differences between individual taxels because nominally identical sensing elements can exhibit different responses.

Modern robotic systems increasingly treat tactile data as a perception modality rather than as a simple feedback signal. Pressure maps, tactile images, and temporal contact sequences can be encoded into learned feature representations and fused with vision, proprioception, force--torque measurements, and robot state. Such multimodal representations allow the robot to reason about interactions that remain ambiguous when observed from a distance.

This role becomes particularly important when visual perception is degraded by occlusion. During grasping, the robot hand itself may hide the object region where the most important interaction occurs. Tactile sensing continues to observe the contact interface and can detect whether the object has shifted, whether contact is asymmetric, or whether additional gripping force is required. Touch therefore complements rather than replaces external perception.

For Physical AI, tactile sensors provide a direct observation channel between intelligent computation and physical reality. Capacitive, resistive, and optical technologies convert contact into different representations, each with distinct advantages and limitations. Selecting among them requires balancing sensitivity, resolution, bandwidth, durability, cost, computation, and mechanical integration according to the robot's embodiment and intended interactions.

10.02. Force Torque Sensor Data Processing [w/Code]

Force--torque sensors provide robots with direct measurements of mechanical interaction between their bodies and the physical environment. A typical six-axis force--torque sensor estimates three orthogonal forces, Fx, Fy, and Fz, together with three moments, Mx, My, and Mz. These measurements are essential for manipulation, assembly, contact detection, compliant control, collision monitoring, and other interaction-intensive robotic tasks.

The sensing element commonly converts mechanical deformation into electrical signals using strain gauges, piezoelectric structures, capacitive elements, or related transducers. Internal electronics amplify and digitize these signals before applying a calibration matrix that maps sensor-channel measurements into physical force and torque values. The resulting six-dimensional wrench represents the combined translational and rotational loading acting at the sensor reference frame.

Raw force--torque measurements should rarely be used directly by robot control or perception algorithms. Sensor signals contain electronic noise, mechanical vibration, structural resonance, thermal drift, bias, quantization effects, and loads generated by the robot itself. Reliable processing therefore requires a pipeline that transforms raw measurements into calibrated, filtered, compensated, time-aligned, and physically interpretable wrench estimates.

Calibration establishes the mathematical relationship between raw sensor channels and actual mechanical loads. For a multi-axis sensor, this relationship is generally represented by a calibration matrix because deformation measured by one sensing element may respond to several force or torque components. Accurate calibration must therefore consider cross-axis sensitivity rather than treating each channel as an entirely independent measurement.

Zero-offset estimation is normally performed when the sensor is unloaded or when the expected static load is known. The measured baseline is stored and subtracted from subsequent samples. However, a single zeroing operation does not eliminate long-term bias because temperature changes, mounting stress, electronics drift, and mechanical aging can shift the baseline. Production systems may therefore require controlled rezeroing or online bias estimation.

Filtering is one of the most important stages of force--torque data processing. High-frequency electrical noise and structural vibration can cause unstable contact estimates or unnecessary control reactions. Low-pass filters are commonly applied when the desired interaction dynamics occupy a lower frequency range, while moving-average, Butterworth, exponential, or other digital filters can be selected according to latency and attenuation requirements.

Filter design must balance noise suppression against temporal response. Excessive smoothing can delay the detection of impacts, contact transitions, or slip-related events, while insufficient filtering can propagate vibration into the controller. In real-time manipulation, the cutoff frequency should therefore reflect the sensor sampling rate, robot structural dynamics, controller bandwidth, and frequency characteristics of the physical interaction being observed.

Notch filtering can be useful when force signals contain strong periodic disturbances at known frequencies. These disturbances may originate from motors, gearboxes, pumps, propellers, structural resonance, or other repetitive mechanisms. A carefully configured notch filter suppresses a narrow frequency region while preserving much of the remaining signal bandwidth, although changing operating conditions may require adaptive or multiple-frequency approaches.

Gravity compensation is necessary whenever the force--torque sensor supports a tool, gripper, payload, or other mass. As robot orientation changes, gravity produces different force components in the sensor coordinate frame and also generates moments when the center of mass is displaced from the sensing origin. Without compensation, these predictable loads can be incorrectly interpreted as external contact forces.

Gravity compensation requires estimates of payload mass, center-of-mass position, sensor pose, and gravitational acceleration. The gravity vector can be transformed from the world or base frame into the sensor frame using robot kinematics. The expected gravitational force and corresponding moment are then calculated and subtracted from the measured wrench, leaving an estimate that more closely represents external interaction.

Inertial compensation extends this principle to dynamic motion. Accelerating or rotating a manipulator generates forces and moments caused by payload inertia even without external contact. High-speed manipulation can therefore produce substantial wrench signals that resemble environmental interaction. Compensation can use robot joint states, kinematic estimates, payload inertial parameters, and sometimes IMU measurements to separate self-generated dynamics from external loads.

Coordinate transformation is another fundamental operation because a measured wrench is meaningful only relative to a defined reference frame. The sensor frame may differ from the tool center point, end-effector frame, robot base, or contact frame required by the application. Force vectors require rotational transformation, while torque transformation must additionally account for the displacement between reference points.

When a wrench is transferred from one reference point to another, a force acting through an offset generates an additional moment. Consequently, transforming torque is not equivalent to simply rotating three torque components. Correct wrench transformation uses the rigid-body spatial relationship between frames, ensuring that both orientation and lever-arm effects are represented consistently for perception and control.

Contact detection can be implemented by evaluating the processed wrench against force or torque thresholds. Simple thresholds are effective for clearly separated contact states, but real environments often contain vibration, changing payloads, and uncertain baselines. More robust systems combine thresholds with hysteresis, temporal persistence, derivative information, statistical tests, or learned classifiers to distinguish intentional contact from transient disturbances.

Force derivatives and rate-of-change measurements are especially useful for identifying sudden impacts. A moderate absolute force may still represent an important collision if it rises extremely rapidly. Conversely, a slowly increasing force may correspond to normal insertion or compliant contact. Combining force magnitude with temporal characteristics allows perception systems to distinguish different interaction modes more reliably.

Force--torque processing also supports estimation of contact properties. The direction of the measured force can indicate the approximate surface normal, while relationships between tangential and normal components can reveal frictional conditions or possible sliding. Torque patterns can provide information about off-center contact, rotational constraints, or contact location when the geometry of the tool and environment is sufficiently known.

In manipulation, processed wrench signals are tightly connected to compliant control. Impedance and admittance controllers use force information to regulate the relationship between robot motion and environmental interaction. The quality of the processed signal directly influences controller stability and responsiveness, making calibration accuracy, filter delay, sampling consistency, and coordinate-frame correctness critical system-level requirements.

Assembly operations illustrate the importance of six-axis measurements particularly well. During peg insertion, connector mating, surface following, polishing, or fastening, small lateral forces and moments may reveal misalignment before visual perception can identify the problem. A robot can use these signals to adjust its trajectory, reduce contact stress, search for alignment, or terminate an operation before excessive force causes damage.

Sampling and timestamp management become important when force--torque data are fused with joint encoders, cameras, tactile sensors, or IMUs. Measurements generated at different rates and latencies must correspond to consistent physical states. Timestamping near the acquisition source, maintaining deterministic communication where required, and synchronizing data streams reduce errors caused by combining measurements from different moments in robot motion.

Signal validation should accompany normal processing because sensor failures can create dangerous control behavior. Saturation, disconnected channels, abnormal offsets, excessive noise, implausible discontinuities, and communication loss should be detected before measurements reach safety-critical controllers. Systems can apply range checks, rate checks, health flags, redundant information, and fallback states to prevent corrupted wrench estimates from commanding unsafe motion.

Sensor mounting also affects data quality. Mechanical interfaces must provide sufficient stiffness and predictable load transfer while avoiding unintended preload or structural deformation. Cable forces, tool mounting errors, loose fasteners, thermal expansion, and mechanical cross-coupling can all appear in the measured wrench. Reliable perception therefore requires treating the sensor, mounting structure, tool, and calibration model as one integrated measurement system.

For Physical AI, force--torque sensing provides a direct representation of the consequences of actions. Vision may predict that an object has been contacted, but wrench measurements reveal how strongly and in what direction the robot is physically interacting with it. Properly processed force--torque data therefore connect perception, control, manipulation, and learning through a physically grounded feedback signal.

A complete force--torque processing pipeline consequently progresses from acquisition and calibration through bias correction, filtering, gravity and inertial compensation, coordinate transformation, synchronization, contact interpretation, and validation. The objective is not merely to produce smooth numerical signals, but to transform raw sensor measurements into reliable physical information that enables safe, responsive, and intelligent robot interaction.

10.03. Contact Detection and Classification from F T [w/Code]

Force--torque sensing enables a robot to determine when physical contact occurs and to infer the nature of that interaction from the measured wrench. A six-axis sensor provides forces Fx, Fy, and Fz together with moments Mx, My, and Mz, creating a compact representation of mechanical interaction. Contact detection transforms this continuous signal into meaningful states such as free motion, initial touch, sustained contact, impact, sliding, or constrained interaction.

Reliable contact detection begins with properly conditioned force--torque data. Raw measurements may contain sensor bias, electrical noise, structural vibration, gravity loads, and inertial forces generated by robot motion. Calibration, zero-offset correction, filtering, gravity compensation, inertial compensation, and coordinate transformation should therefore precede classification so that observed changes correspond as closely as possible to external physical interaction.

The simplest detection strategy compares force magnitude with a predefined threshold. The resultant force can be expressed as the norm of Fx, Fy, and Fz, while torque magnitude can similarly be calculated from Mx, My, and Mz. When either quantity exceeds an experimentally determined threshold, the system declares contact. This method is computationally inexpensive and appropriate for many industrial manipulation tasks with predictable operating conditions.

Single thresholds can nevertheless produce unstable decisions when signals fluctuate near the decision boundary. Hysteresis addresses this problem by defining different thresholds for entering and leaving the contact state. Contact may be activated at a relatively high force threshold but released only after the force falls below a lower threshold. This prevents rapid switching between contact and non-contact states caused by noise or small mechanical oscillations.

Temporal persistence provides another layer of robustness. Instead of declaring contact from one sample, the detector requires the threshold condition to remain valid for a specified number of samples or duration. Short disturbances can consequently be rejected while sustained interactions are preserved. The persistence interval must remain short enough to satisfy the reaction-time requirements of manipulation and safety functions.

Changes in force can contain information that absolute magnitude alone cannot provide. The temporal derivative of force highlights rapid transitions and is especially useful for detecting impacts or first contact. A rapidly increasing force may indicate collision even before its magnitude becomes large, whereas a slowly increasing signal may correspond to intentional compliant loading. Combining magnitude and derivative features therefore improves discrimination between interaction types.

A practical detector can represent robot interaction using a finite set of contact states. Free motion corresponds to forces close to the compensated baseline. Initial contact is characterized by a significant force transition, sustained contact by persistent loading, and separation by the return of force toward baseline. Additional states can describe impact, sliding, jamming, insertion, or other task-specific mechanical conditions.

Contact direction can be estimated from the three-dimensional force vector. After transforming the wrench into an appropriate tool or contact coordinate frame, the force direction provides information about where the environment is resisting robot motion. Normal and tangential components can then be separated when the approximate surface orientation is known, supporting more detailed interpretation of the contact mechanics.

The normal force represents loading perpendicular to a contact surface, while tangential forces represent loading along that surface. Their relationship is particularly important for distinguishing stable contact from possible sliding. When tangential loading approaches the frictional limit associated with the normal force, the interaction may transition toward slip. Force--torque sensing can therefore provide indirect evidence of contact stability even without dense tactile sensing.

Torque measurements add information that cannot be recovered from force magnitude alone. A contact occurring away from the sensor or tool reference point generates a moment proportional to the lever arm and applied force. Patterns across Mx, My, and Mz can therefore help identify eccentric loading, rotational constraints, misalignment, or changes in the effective contact location.

When tool geometry is known, force and torque measurements can sometimes be used to estimate the contact point. The measured moment reflects the cross product between the position vector and applied force, although the problem may be underdetermined without additional assumptions. Geometric constraints, known surfaces, robot kinematics, or tactile information can reduce ambiguity and improve contact localization.

Contact classification extends detection by determining what type of physical interaction is occurring. Useful input features include individual force and torque components, resultant magnitudes, temporal derivatives, moving statistics, frequency-domain characteristics, force ratios, and signal duration. These features can distinguish events that produce similar peak forces but have different temporal or directional structures.

Rule-based classifiers are attractive when interaction modes are well understood. An impact may be represented by a large positive force derivative followed by rapid decay, while sustained pressing produces a persistent normal load. Sliding can combine tangential force with characteristic fluctuations, and jamming may generate increasing forces and moments despite little commanded motion. Such rules are interpretable and computationally efficient.

Statistical classification becomes useful when fixed rules cannot capture sensor variability. Feature vectors derived from short temporal windows can be processed by methods such as logistic regression, support vector machines, decision trees, or probabilistic classifiers. Training examples associate characteristic wrench patterns with known contact classes, allowing boundaries to be learned from data rather than manually specified for every operating condition.

Deep learning can operate directly on force--torque sequences when interaction patterns are complex. One-dimensional convolutional networks can extract local temporal features, recurrent architectures can model sequential dependencies, and transformer-based models can represent longer interaction histories. These approaches can classify contact-rich actions but generally require larger labeled datasets and careful validation across tools, objects, speeds, and payload conditions.

Classification should not rely on force--torque data in isolation when complementary robot information is available. Joint position, velocity, commanded motion, motor current, tactile sensing, vision, and proprioceptive state provide context for interpreting wrench measurements. A large force during commanded pressing may be expected, while the same force during free-space motion may indicate an unexpected collision requiring immediate response.

Motion context is particularly important for separating inertial effects from external contact. Rapid acceleration of a heavy tool can generate substantial wrench changes without touching the environment. Conversely, contact may occur while the robot is nearly stationary. Combining compensated force--torque measurements with robot kinematics and motion commands therefore reduces false positives and improves interaction classification.

Real-time implementation requires explicit attention to latency. Filtering, temporal windows, persistence tests, and learned inference all introduce delay between physical contact and the classified result. Long windows may improve classification accuracy but can be unsuitable for collision response. Safety-oriented detection often uses a fast low-level path, while richer classification operates simultaneously on a longer temporal context.

Confidence estimation is useful when contact classes overlap or sensor conditions differ from training data. Rather than producing only a categorical label, the classifier can provide probabilities or confidence scores for possible interaction states. Low-confidence events can trigger conservative robot behavior, request additional sensing, or remain classified as unknown instead of forcing an unreliable decision.

Contact detection systems must also handle abnormal measurements. Sensor saturation, communication loss, sudden bias shifts, excessive vibration, or invalid compensation can resemble real interaction events. Range checks, signal-quality indicators, consistency tests, and sensor health monitoring should therefore operate alongside the contact classifier so that sensor faults are not mistakenly interpreted as environmental contact.

Manipulation and assembly provide important applications of force--torque contact classification. During insertion, the system can distinguish free approach, first touch, aligned insertion, edge contact, and jamming. During surface following, it can maintain desired normal contact while detecting excessive tangential loading. During grasping or handover, it can identify external interaction and adapt robot motion accordingly.

For Physical AI, contact classification converts mechanical signals into semantic information about action consequences. The robot does not merely observe that force exists; it interprets whether it has touched, collided, pressed, slid, become constrained, or completed a contact-dependent action. This interpretation creates a bridge between low-level physical sensing and higher-level reasoning about embodied interaction.

A robust contact perception pipeline therefore progresses from calibrated and compensated wrench measurements to feature extraction, contact detection, temporal state estimation, interaction classification, confidence evaluation, and control response. Combining force magnitude, direction, torque, temporal dynamics, robot motion, and contextual sensing enables the system to transform six-axis force--torque signals into reliable knowledge about physical contact.

10.04. Tactile Image Processing GelSight DIGIT [w/Code]

Tactile image processing transforms raw images acquired from optical tactile sensors into quantitative representations of contact geometry, pressure distribution, deformation, and interaction state. Unlike conventional pressure arrays that directly output electrical measurements from individual taxels, optical tactile sensors observe the deformation of an elastomeric surface using a camera and controlled illumination. GelSight and DIGIT represent important examples of this approach, where computer vision becomes the primary mechanism for interpreting touch.

A typical optical tactile sensor contains a compliant elastomeric sensing surface, an internal illumination system, a camera, and mechanical packaging that maintains a controlled optical relationship between these components. When an object presses against the sensing surface, the elastomer deforms and modifies the appearance of the internal sensing surface. The camera captures this change as an image, creating a tactile representation that contains both spatial and temporal information about the contact.

GelSight-style sensing emphasizes high-resolution reconstruction of local contact geometry. A compliant elastomer is observed through an internal camera, while illumination makes surface deformation and contact features visually distinguishable. The resulting tactile image can reveal object boundaries, surface texture, local curvature, and contact shape. This makes optical tactile sensing particularly useful when the robot must infer detailed physical properties that cannot be reliably determined from force magnitude alone.

DIGIT follows a compact optical tactile sensing concept designed for robotic fingertips and manipulation. Its small form factor allows tactile perception to be placed directly at the point of interaction while retaining image-based sensing. The resulting tactile frames can be processed using computer vision techniques to estimate contact regions, object features, deformation patterns, and interaction states. Compact construction is particularly important when tactile sensing must be integrated into articulated robotic hands.

The first processing stage normally converts the camera image into a stable tactile representation. Raw frames may contain illumination variation, sensor-specific artifacts, lens distortion, background structure, and changes caused by elastomer deformation. Image normalization, background subtraction, geometric correction, and illumination compensation can therefore improve the consistency of subsequent processing. The objective is to isolate changes caused primarily by physical contact.

Contact segmentation identifies the portion of the tactile image affected by an external object. A simple approach compares the current image with a reference image representing the unloaded sensor. Significant pixel differences indicate regions where the elastomer has changed. More advanced methods can use learned segmentation models to distinguish actual contact from noise, internal reflections, mechanical artifacts, or deformation unrelated to the target interaction.

Contact geometry can be extracted from the segmented tactile image. The shape, area, boundary, and local structure of the contact region provide information about how an object touches the sensor. Edges can indicate object boundaries, while local deformation patterns can reveal curvature or surface structure. When the sensor has been calibrated appropriately, image coordinates can be mapped into physical coordinates, allowing tactile measurements to become geometrically meaningful.

Pressure or force distribution can be estimated by learning a mapping between tactile image appearance and mechanical loading. The deformation of the elastomer produces image features correlated with applied forces. Calibration experiments using known loading conditions can therefore generate training data for regression models that estimate normal force, shear force, or spatial pressure distribution. Such models depend strongly on the mechanical properties and optical characteristics of the specific sensor.

Tactile image processing is inherently temporal because contact evolves during manipulation. A single frame may reveal where contact exists, but a sequence can reveal approach, impact, compression, stable contact, sliding, and release. Temporal filtering and frame-to-frame feature tracking can suppress noise and provide more stable estimates. Sequential models can further learn characteristic patterns associated with different interaction states.

Slip detection is an important application of temporal tactile processing. During stable grasping, the contact image may remain relatively stationary even when the object is under load. As the object begins to slide, local image features can move or change rapidly. Optical flow, correlation-based tracking, image gradients, or learned temporal representations can detect these changes and provide an early indication of incipient slip before the object is visibly lost.

Tactile images can also support object recognition and material-related inference. Contact geometry provides local information about object shape, while texture and deformation characteristics may provide additional clues about surface properties. Machine learning models can learn tactile representations that associate contact images with object categories, local surface features, or manipulation outcomes. Combining tactile and visual representations can further improve recognition when external vision is occluded or ambiguous.

Calibration is essential because the same physical force may produce different image patterns depending on elastomer properties, sensor geometry, illumination, camera characteristics, temperature, and mechanical aging. A practical calibration procedure establishes the relationship between image features and physical quantities under controlled conditions. Maintaining consistent mechanical construction and periodically monitoring calibration drift are important for reliable long-term operation.

Deep learning provides a powerful method for converting tactile images into higher-level perception outputs. Convolutional neural networks can extract spatial features from individual tactile frames, while temporal networks or transformers can process sequences of tactile observations. Models may be trained for contact classification, force estimation, slip detection, object recognition, grasp stability prediction, or pose refinement. However, performance depends strongly on the diversity and quality of the tactile training data.

Tactile image processing becomes more powerful when combined with other robot perception modalities. Vision provides global object and scene information, while tactile sensing provides local information at the actual contact interface. Force--torque sensing provides aggregate mechanical loading, whereas tactile images reveal spatial distributions. Fusion of these signals can therefore produce a more complete representation of manipulation state than any single sensor modality can provide.

For Physical AI, GelSight- and DIGIT-style tactile perception provides a direct interface between visual representation learning and physical contact. The robot can transform raw tactile images into contact geometry, force-related features, slip information, grasp stability, and interaction state. These representations can then be supplied to manipulation planners, controllers, learned policies, or multimodal foundation models, creating a perception pathway from physical touch to intelligent action.

A complete tactile image processing pipeline therefore progresses from optical image acquisition through normalization, background compensation, contact segmentation, geometric feature extraction, temporal analysis, force or pressure estimation, and higher-level classification. GelSight emphasizes detailed contact geometry, while DIGIT demonstrates how compact optical tactile sensing can be integrated into robotic manipulation. Together, these approaches illustrate how image-based touch can transform local physical interaction into structured perceptual information for advanced robotic systems.

10.05. Slip Detection from Tactile Signals [w/Code]

Slip detection from tactile signals enables a robot to identify when an object begins to move relative to a grasping surface. Unlike vision-based methods that observe the object from a distance, tactile sensing directly measures changes at the physical contact interface. This makes tactile slip perception particularly valuable when the object is partially occluded by the robot hand, when visual texture is weak, or when very small relative motions must be detected before the object is visibly displaced.

A tactile sensor can reveal slip through temporal changes in pressure, deformation, contact location, or local tactile image features. During a stable grasp, the contact pattern tends to remain relatively consistent, although small variations may occur because of vibration or sensor noise. When sliding begins, local features move across the sensing surface or the pressure distribution changes systematically. Detecting these changes over time is therefore more informative than interpreting a single tactile frame.

For optical tactile sensors such as GelSight or DIGIT, slip detection can be formulated as a visual motion estimation problem at the contact interface. Consecutive tactile images are compared to determine whether characteristic surface features have moved. Optical flow, image correlation, feature tracking, image gradients, and frame-to-frame displacement can reveal the direction and magnitude of local motion. The resulting motion field provides direct evidence that relative movement is occurring between the object and tactile surface.

A simple approach uses image correlation between consecutive tactile frames. A local contact region is divided into smaller areas, and each region is compared with the corresponding area in the next frame. A significant displacement of the image pattern indicates possible sliding. Correlation-based methods are computationally understandable and can operate efficiently on embedded systems, but their performance can decrease when contact geometry changes substantially or when the tactile image contains repetitive or ambiguous textures.

Optical flow provides a more detailed representation of tactile motion. Instead of estimating a single displacement for the entire contact region, optical flow calculates local motion vectors across the tactile image. The spatial distribution of these vectors can reveal whether the contact is moving uniformly, rotating, deforming, or experiencing localized slip. Flow magnitude and direction can therefore become important features for distinguishing stable grasping from different forms of object motion.

Slip can also be detected from changes in pressure or deformation without explicitly tracking image features. During stable contact, the pressure distribution may remain within an expected range. As an object begins to move, pressure may migrate from one part of the contact area to another, producing characteristic temporal patterns. A tactile system can monitor these changes using pressure differences, center-of-pressure movement, deformation gradients, or other spatial statistics.

Temporal processing is essential because slip is fundamentally a dynamic event. A single change in tactile appearance does not necessarily indicate slip because vibration, sensor noise, elastic recovery, or small changes in grasp force can produce similar variations. A temporal window can accumulate evidence across multiple frames, allowing the system to distinguish persistent directional motion from short-duration disturbances.

The rate of tactile feature motion can provide an early indication of incipient slip. In many grasping situations, small local movements occur before the object undergoes large-scale displacement. Detecting these early changes allows the robot to increase grasp force, modify finger motion, or adjust the manipulation strategy before the object is lost. This makes slip detection an important component of closed-loop grasp stabilization.

Slip direction is also useful for control. A tactile motion field can indicate whether the object is moving upward, downward, laterally, or rotationally relative to the contact surface. The robot can use this information to modify individual finger forces or adjust the end-effector pose. When multiple tactile sensors are available, their combined motion estimates can provide additional information about object-level motion and asymmetric contact conditions.

Threshold-based slip detection can be implemented by defining limits on displacement, optical-flow magnitude, pressure variation, or temporal change. When the measured signal exceeds the threshold for a sufficient duration, the system declares a slip event. Hysteresis and temporal persistence can reduce false detections caused by noise. However, fixed thresholds may require adjustment when object materials, grasp forces, sensor conditions, or manipulation speeds change.

Learning-based methods can capture more complex tactile slip patterns. A model can receive sequences of tactile images or extracted tactile features and learn the difference between stable contact, incipient slip, sustained slip, and other interaction states. Convolutional networks can extract spatial features, while recurrent or transformer-based models can represent temporal dependencies. Such models can also combine tactile information with force--torque measurements and robot state.

Training data for slip detection should represent the operating conditions in which the robot will actually work. Different object materials, shapes, surface textures, grasp forces, sensor configurations, and motion speeds can produce substantially different tactile signals. Labels may distinguish stable grasping, incipient slip, continuous slip, and recovery. The diversity of these examples is important because a model trained only on a narrow set of objects may not generalize reliably to new manipulation conditions.

Slip detection should be integrated with grasp control rather than treated as an isolated perception function. When incipient slip is detected, the controller can increase normal grasp force, change finger posture, slow the manipulation motion, or temporarily stabilize the object. The appropriate response depends on the task because simply increasing force may damage fragile objects or increase unnecessary energy consumption. Tactile perception therefore provides information for adaptive control rather than a simple binary alarm.

For Physical AI, tactile slip detection represents a direct connection between physical interaction and action adaptation. The robot observes how its contact state changes as a consequence of its own motion and environmental interaction, then uses this information to modify its behavior. Tactile sequences can consequently become part of learned manipulation policies, where slip-related features contribute to grasp stability estimation, action selection, and closed-loop decision making.

A complete tactile slip detection pipeline therefore proceeds from tactile signal acquisition through preprocessing, spatial feature extraction, temporal analysis, motion or deformation estimation, slip classification, confidence evaluation, and control response. Optical flow and image tracking are particularly useful for image-based tactile sensors, while pressure and deformation statistics provide complementary information. Combining these signals with force--torque measurements, robot motion, and learned models enables early and reliable detection of slip during contact-rich manipulation.

10.06. Tactile Based Grasp Stability Assessment [w/Code]

Tactile-based grasp stability assessment enables a robot to determine whether an object is securely held, approaching slip, or undergoing unstable contact by analyzing information directly at the grasp interface. Vision can estimate object pose and hand configuration, while force--torque sensing provides aggregate loading, but tactile sensing reveals how pressure and deformation are distributed across individual contact regions. This makes tactile perception particularly valuable for evaluating grasp quality under occlusion and during continuous physical interaction.

Grasp stability is not determined by force magnitude alone. A stable grasp depends on the relationship among normal force, tangential force, contact location, contact area, friction, object geometry, and the motion of both the robot and object. Tactile sensors can observe changes in these variables through pressure maps, deformation patterns, contact geometry, and temporal signal variations. Stability assessment therefore converts local tactile observations into an estimate of whether the current grasp can resist expected external disturbances.

A useful tactile representation begins with identifying the active contact regions on the fingers or gripper. The sensor can estimate contact area, pressure distribution, pressure center, local deformation, and spatial gradients. These measurements provide information about whether contact is evenly distributed or concentrated at an undesirable location. A grasp with sufficient total force but highly localized contact may behave differently from a grasp with a broader and more balanced contact distribution.

Normal force is an important component of grasp stability because it determines the available frictional resistance against tangential motion. When normal force is insufficient relative to the applied tangential load, the object can begin to slide. Tactile sensing can estimate the local normal loading across multiple contact regions and determine whether the available frictional margin is being consumed. This provides a physical basis for distinguishing secure contact from conditions approaching slip.

Tangential force and shear deformation provide additional evidence of grasp stability. When an object begins to move relative to a finger surface, the tactile pattern may shift even before large changes occur in total normal pressure. Local shear displacement can therefore act as an early indicator of incipient slip. Combining normal pressure with tangential motion allows the system to estimate whether the contact remains within a stable frictional regime or is approaching a transition to sliding.

Contact distribution across multiple fingers is also important. In a multi-finger grasp, each contact contributes to the overall ability to resist translation and rotation. Tactile measurements can reveal whether one finger is carrying excessive load while another is losing contact. Such asymmetry may indicate an unstable grasp even when the total measured gripping force remains within an acceptable range. The robot can use this information to redistribute finger forces and improve mechanical balance.

Temporal consistency is another key factor. A stable grasp should maintain contact patterns within an expected range while the robot performs the intended motion. Short fluctuations may result from vibration, sensor noise, or small elastic changes and should not automatically be interpreted as instability. Persistent movement of the pressure center, progressive reduction of contact area, increasing shear deformation, or repeated local slip events provides stronger evidence that grasp stability is degrading.

Tactile images from optical sensors such as GelSight or DIGIT can provide particularly rich stability information. Changes in local surface features can reveal object motion, contact deformation, and slip direction. Image correlation, optical flow, feature tracking, and temporal feature extraction can convert tactile image sequences into motion descriptors. These descriptors can then be combined with pressure or force estimates to distinguish stable grasping from incipient or continuous slip.

Stability assessment can initially be implemented using analytical or rule-based criteria. Thresholds can be defined for contact area, normal force, shear force, pressure-center movement, or tactile feature displacement. Hysteresis and temporal persistence can prevent unstable switching between stable and unstable states. These methods are transparent and computationally efficient, making them suitable for real-time embedded control when the operating conditions are well characterized.

Learning-based stability estimation becomes useful when grasp behavior depends on complex interactions among object geometry, material properties, finger configuration, and motion. A model can receive tactile images, pressure maps, force--torque measurements, and robot state information and learn to classify stable grasp, incipient slip, unstable contact, or recovery. Convolutional networks can encode spatial tactile information, while temporal networks or transformers can capture the evolution of contact patterns over time.

Grasp stability should be evaluated together with the intended manipulation task. A grasp that is stable while holding an object stationary may become unstable during acceleration, rotation, insertion, or transportation. Conversely, a temporary redistribution of pressure may be acceptable if it corresponds to an intentional manipulation action. Stability assessment must therefore consider robot motion, commanded action, object dynamics, and the expected contact state rather than treating every tactile change as a failure.

When instability is detected, tactile perception can support closed-loop grasp adjustment. The controller may increase or redistribute normal force, modify individual finger positions, change the end-effector orientation, slow the manipulation motion, or temporarily stabilize the object. The response should be proportional to the estimated risk because excessive gripping force can damage fragile objects, increase actuator load, or interfere with delicate manipulation. Tactile stability estimation therefore functions as a continuous feedback mechanism rather than a simple binary alarm.

For Physical AI, tactile-based grasp stability assessment provides a direct link between physical contact, perception, and adaptive action. The robot observes how its grasp behaves under changing loads and motion, estimates the current stability state, and modifies its behavior when the predicted margin becomes insufficient. By combining tactile geometry, pressure distribution, shear motion, temporal dynamics, force--torque information, and robot state, the system can develop a more physically grounded representation of grasp quality.

A complete tactile grasp stability pipeline therefore progresses from tactile acquisition and calibration through contact localization, pressure and deformation analysis, shear and slip estimation, temporal stability evaluation, multimodal fusion, stability classification, and closed-loop control. The resulting stability estimate can support grasp planning, manipulation execution, slip prevention, and recovery behavior. This approach allows tactile sensing to become an active component of robotic intelligence, transforming local contact measurements into continuous knowledge about whether an object is securely held and how the grasp should adapt during physical interaction.

10.07. Whole Body Skin Sensor for Humanoid Safety [w/Code]

Whole-body skin sensing provides humanoid robots with distributed awareness of physical contact across the torso, arms, hands, legs, and other externally accessible surfaces. Unlike conventional collision detection based mainly on motor current or joint torque, skin sensors observe contact closer to the actual interaction surface. This allows the robot to detect human contact, unexpected obstacles, pressure, and local mechanical interaction before the resulting load necessarily propagates through the entire mechanism.

A humanoid skin system is typically composed of many distributed sensing elements arranged across flexible or compliant surfaces. Each sensing element may measure pressure, deformation, shear, proximity, vibration, or a related physical quantity. The elements can be organized into local sensor patches and connected to regional acquisition electronics. The resulting architecture must provide sufficient spatial coverage while remaining compatible with the robot's curved geometry, joint motion, mechanical deformation, and external protective structures.

Whole-body coverage creates a fundamentally different sensing problem from fingertip tactile perception. A fingertip may require very high spatial resolution because it interacts directly with small objects, whereas body skin must cover much larger areas and detect unexpected contact from many directions. The design therefore requires a balance among spatial resolution, sensing range, response time, flexibility, durability, wiring complexity, power consumption, and manufacturing cost. Sensor density can be increased in regions where human interaction or collision risk is expected to be higher.

Contact localization is one of the primary functions of whole-body skin sensing. When multiple sensor elements respond simultaneously, their spatial pattern can be used to estimate the location and approximate extent of contact. A local pressure maximum may identify the center of interaction, while neighboring activated elements reveal the contact area. Maintaining a coordinate mapping between individual skin elements and the robot body model allows contact information to be expressed in a common robot frame for perception and control.

Safety-oriented skin sensing must also distinguish intentional interaction from potentially hazardous collision. A human may touch, push, hold, or guide a humanoid during normal operation, while an unexpected impact may require an immediate protective response. The system can use contact magnitude, rate of change, duration, location, direction, and robot motion context to classify these situations. Contact interpretation should therefore combine the tactile signal with the robot's commanded behavior and current operating state.

Response latency is particularly important for humanoid safety. A skin sensor may detect contact before conventional mechanical protection mechanisms or joint-level monitoring can identify the event. The sensing system must therefore acquire, process, and transmit contact information quickly enough to support the required safety response. Low-latency hardware paths can be used for immediate detection, while richer perception algorithms can analyze the same data over longer temporal windows for more detailed interpretation.

The sensor layer should be integrated with the robot's control architecture rather than treated as an isolated measurement system. Contact information can be transmitted to safety controllers, whole-body controllers, motion planners, or higher-level perception modules. Depending on the detected event, the robot may stop a motion, reduce joint torque, modify its trajectory, enter a compliant mode, or maintain controlled contact. The appropriate response depends on the location, severity, and context of the interaction.

Distributed skin sensing is especially important for physical human--robot interaction. A humanoid may need to recognize that a person has touched its arm, pushed its torso, blocked its movement, or made contact while the robot is carrying out another task. Because the skin covers regions that may be outside the field of view of cameras, it provides a complementary local sensing layer. This allows contact to be detected even when visual perception is affected by occlusion, lighting, or body configuration.

Whole-body tactile sensing can also contribute to collision avoidance and motion adaptation. If contact is detected on a body region during movement, the robot can incorporate that information into its immediate motion response. For example, an unexpected contact may cause the robot to reduce velocity, change direction, or stop a particular motion while maintaining safe behavior in other parts of the body. This creates a direct feedback path between physical interaction and whole-body motion control.

The mechanical construction of the skin is critical because the sensing layer must survive repeated contact while maintaining predictable measurement characteristics. Flexible substrates, compliant protective layers, segmented modules, and robust electrical connections can support integration over curved robotic surfaces. However, protective materials also affect force transmission and sensitivity. The mechanical stack must therefore be designed together with the sensor characteristics rather than treating the protective layer as an independent enclosure.

Calibration and health monitoring become challenging as the number of sensing elements increases. Individual elements may have different offsets, sensitivities, temperature responses, or aging characteristics. A practical system therefore requires baseline calibration, sensor normalization, detection of failed or saturated elements, and monitoring of communication quality. Regional redundancy can also help maintain useful contact information when individual sensing elements become unreliable.

Whole-body skin data can be combined with vision, force--torque sensing, joint-state information, motor current, and proximity sensing. Vision provides global environmental context, joint and motor measurements describe internal robot state, and skin sensors provide local physical contact information. Multimodal fusion can therefore determine not only that contact occurred, but also where it occurred, what the robot was doing, and whether the interaction was expected. This is particularly valuable for embodied intelligence and Physical AI.

For Physical AI, whole-body skin sensing provides a distributed physical observation layer between the robot and its environment. Instead of treating contact only as an abnormal event, the robot can continuously observe physical interaction and use it as part of its state representation. Contact patterns can support human--robot interaction, safe navigation, compliant manipulation, motion adaptation, and learning from physical experience. The skin consequently becomes an important component of the robot's embodied perception system.

A complete whole-body skin safety architecture therefore connects distributed tactile sensing with local signal acquisition, calibration, contact localization, temporal processing, contact classification, safety decision logic, and whole-body control. The architecture must operate across different body regions while maintaining predictable response time and fault handling. By combining broad physical coverage with contextual robot-state information, humanoid skin sensing can provide an additional layer of contact awareness that complements conventional perception and mechanical safety mechanisms.

10.08. Tactile Data Learning Contact Rich Manipulation [w/Code]

Tactile data learning enables robots to transform physical contact observations into representations that can support manipulation decisions and learned control. Unlike vision, which mainly describes objects and scenes before contact, tactile sensing observes what actually happens at the interaction interface. Pressure distributions, deformation patterns, contact geometry, slip signals, and force--torque measurements can therefore provide training information about the physical consequences of robot actions during contact-rich manipulation.

Contact-rich manipulation includes tasks in which physical interaction is continuous or repeatedly changes during execution. Examples include grasping, insertion, pushing, sliding, assembly, tool use, surface following, and handling deformable objects. In these tasks, visual observations alone may not fully reveal whether an object is aligned, constrained, slipping, jammed, or securely supported. Tactile data provides local evidence that can be used to infer these hidden interaction states and improve the robot's action selection.

A useful tactile learning dataset should preserve the relationship between sensory observations, robot state, actions, and task outcomes. A tactile frame or pressure map becomes more valuable when it is associated with joint positions, velocities, commanded actions, force--torque measurements, object state, and temporal context. Such synchronized records allow learning systems to estimate not only what contact looks like, but also how contact changes as the robot moves and how those changes influence task success.

Data representation is an important design decision because tactile sensors produce different signal types. Capacitive and resistive arrays may generate pressure maps, optical sensors may generate high-resolution tactile images, and force--torque sensors provide low-dimensional wrench measurements. These signals can be processed independently or encoded into common latent representations. Spatial features describe where contact occurs, while temporal features describe how contact evolves during manipulation.

Temporal information is particularly important for learning contact dynamics. A single tactile frame may show a stable contact configuration, but a sequence can reveal approach, impact, compression, slip, release, or jamming. Temporal convolution, recurrent models, transformers, or other sequence-learning methods can represent these changes. The resulting representation can be used to predict future contact states or determine which action is appropriate under the current physical condition.

Supervised learning can associate tactile observations with manually defined task labels. Labels may represent grasp success, contact type, slip state, insertion alignment, collision, or task completion. Classification models can then learn mappings from tactile features to these states. The quality of supervision is important because ambiguous or inconsistent labels can cause the model to learn sensor artifacts rather than meaningful physical interaction patterns.

Self-supervised learning can reduce dependence on manual tactile labeling. Instead of requiring a human to label every contact sequence, the robot can learn from naturally available signals such as temporal consistency, sensor reconstruction, cross-modal prediction, or future-state prediction. For example, a model can learn to predict the next tactile observation from the current tactile and robot-state history. Useful representations can emerge from these predictive relationships before task-specific fine-tuning.

Multimodal learning is especially valuable because tactile information is local while visual information is often global. A camera may identify the object and estimate its approximate pose, while tactile sensing determines the actual contact condition after grasping. Force--torque sensing provides aggregate mechanical loading, and proprioceptive data describes robot configuration and motion. Combining these modalities can produce representations that connect visual intent with physical execution.

Imitation learning can use tactile demonstrations to learn manipulation policies from successful physical interactions. Demonstration data may include tactile observations together with robot actions, joint states, and task context. The learned policy can then associate particular contact patterns with corrective or productive actions. Tactile information is especially useful when the visually identical situation can require different actions depending on friction, contact alignment, or whether an object has begun to slip.

Reinforcement learning can further optimize tactile-based manipulation through interaction. The robot receives observations containing tactile features and obtains rewards based on task progress, stability, force limits, or successful completion. The policy learns actions that improve expected outcomes under uncertain contact conditions. Careful reward design is important because maximizing task success without physical constraints can produce excessive forces, damaging contacts, or undesirable manipulation strategies.

Contact-rich learning also benefits from event-based representations. Instead of processing every tactile sample identically, the system can identify meaningful events such as first contact, peak compression, onset of slip, stable support, loss of contact, or collision. These events provide compact descriptions of interaction dynamics and can serve as inputs to higher-level task models. Event representations can also reduce computational requirements for long-duration manipulation sequences.

Data diversity is critical for tactile learning because contact signals depend strongly on physical conditions. Object material, surface texture, geometry, mass, friction, grasp force, sensor compliance, tool configuration, and manipulation speed can all change the observed tactile pattern. A model trained under only one combination of these conditions may fail when transferred to another object or sensor configuration. Dataset construction should therefore capture meaningful variation rather than simply increasing the number of nearly identical samples.

Calibration and sensor consistency are equally important for learning. Changes in sensor mounting, elastomer properties, illumination, temperature, electronics, or mechanical wear can shift the distribution of tactile observations. If these variations are not represented in training data or compensated during preprocessing, a learned model may interpret sensor drift as a physical change. Stable calibration procedures and sensor normalization can therefore improve the reliability of learned tactile representations.

Simulated tactile data can complement real-world collection, although the simulation must reproduce relevant contact physics and sensor characteristics sufficiently well. Synthetic data can provide large numbers of controlled interactions, including variations that are difficult or expensive to collect physically. Domain randomization and adaptation techniques can reduce the gap between simulated and real tactile observations. Real tactile data remains important for validating whether learned representations transfer to actual sensor behavior.

Tactile learning should ultimately support closed-loop manipulation rather than only offline recognition. The system observes contact, estimates the current interaction state, predicts the likely consequence of continued motion, and selects an action that maintains or improves task performance. When slip is detected, the policy may increase grasp force; when insertion resistance rises unexpectedly, it may adjust alignment; when contact becomes unstable, it may slow or modify the motion. This creates a direct perception--prediction--action loop.

For Physical AI, tactile data represents a particularly important source of grounded experience because it records the consequences of physical action at the point of interaction. Learning from these signals can help a robot acquire representations of contact, friction, compliance, stability, and manipulation outcomes. When combined with vision, force--torque sensing, proprioception, and action history, tactile learning can contribute to a multimodal model of how robot actions affect the physical world.

A complete tactile learning pipeline therefore connects sensor acquisition and calibration with synchronization, preprocessing, feature or representation learning, multimodal fusion, task labeling or self-supervised objectives, policy learning, evaluation, and closed-loop deployment. The central objective is not simply to classify tactile images, but to learn a relationship between contact observations, robot actions, and physical outcomes. Such contact-grounded learning provides a foundation for manipulation systems that can adapt their behavior according to what is actually happening at the physical interface.

10.09. Force Torque ROS2 Driver and Filter [w/Code]

A force--torque sensor driver provides the software interface between a physical six-axis sensor and the ROS2 perception and control system. The sensor typically reports three force components, Fx, Fy, Fz, and three torque components, Mx, My, Mz. The ROS2 driver must acquire these measurements reliably, apply the required sensor-specific interpretation, attach accurate timestamps, and publish standardized messages so that downstream contact detection, compliance control, manipulation, and tactile fusion modules can use the data consistently.

A production-quality driver should separate hardware communication from signal processing and ROS2 communication. The hardware layer manages protocols such as Ethernet, serial communication, CAN, or a vendor-specific interface, while the processing layer performs calibration, bias correction, filtering, coordinate transformation, and validity checking. The ROS2 node then publishes the processed wrench together with appropriate metadata. This separation makes the system easier to test and allows the same processing logic to be reused with simulated or recorded sensor data.

Sensor initialization should establish a deterministic operating state before measurements are accepted by downstream components. The driver may need to configure sampling rate, communication mode, sensor range, output format, internal filtering, and calibration parameters. Startup procedures should also verify communication and detect abnormal values before the sensor becomes available to the control system. When possible, ROS2 lifecycle management can be used so that configuration, activation, operation, and shutdown occur through explicit and predictable states.

Timestamp quality is important because force--torque measurements are frequently combined with joint states, tactile images, robot motion, and camera observations. A measurement should therefore receive a timestamp that represents the actual acquisition time as closely as possible rather than simply the time at which the ROS2 message was published. Hardware timestamps or synchronized clocks can reduce temporal uncertainty. When multiple sensors participate in contact estimation, consistent time synchronization becomes particularly important for interpreting rapid impacts and transient interactions.

Calibration converts raw sensor output into physically meaningful force and torque values. A sensor may require scale factors, axis alignment, bias compensation, and a complete calibration matrix depending on its characteristics. Zero-bias estimation is especially important because even a small offset can appear as a persistent contact force. The driver should provide a controlled procedure for capturing unloaded measurements and applying the resulting bias values without confusing normal payload or gravity effects with sensor errors.

Gravity and tool-load compensation should generally be distinguished from basic sensor calibration. Calibration corrects properties of the sensor itself, while compensation accounts for known mechanical loads applied to the sensor. In a manipulator, the tool mass, center of mass, and orientation can generate substantial wrench components even when the tool is not touching the environment. A higher-level compensation module can therefore use robot kinematics and current pose to remove predictable loads before contact detection or force control is performed.

Filtering is required because force--torque signals often contain electrical noise, structural vibration, motor-induced disturbances, and quantization effects. A low-pass filter can suppress high-frequency noise, while moving-average or exponential filters provide simple computational alternatives. However, excessive filtering introduces phase delay and can hide rapid contact events. Filter parameters should therefore be selected according to the intended application, with faster paths for collision or impact detection and smoother paths for force estimation or manipulation control.

Different filtering strategies may be required for force and torque channels because their noise characteristics and mechanical sensitivity can differ. A fixed filter applied identically to all six axes is simple but may not provide the best response. More advanced implementations can use separate cutoff frequencies, notch filters for known vibration frequencies, or adaptive methods when operating conditions change. The driver should expose filtering parameters clearly so that the system can be tuned without modifying the underlying hardware interface.

Coordinate-frame handling is another critical part of force--torque integration. The sensor measures a wrench in its own measurement frame, while manipulation algorithms may require the wrench in a tool, end-effector, robot base, or contact frame. Transforming a wrench requires correct treatment of both force and torque components, including the effect of changes in the reference point. ROS2 applications should therefore maintain a consistent TF2 frame structure and clearly document the frame associated with every published wrench.

ROS2 message design should preserve the semantic meaning of the measurement. A standard wrench message can contain force and torque vectors, while the message header provides the timestamp and reference frame. Additional diagnostic information can be published separately, including sensor status, saturation state, communication health, calibration state, and signal quality. This separation prevents application code from depending on undocumented vendor-specific fields while still allowing detailed monitoring of the hardware.

Real-time behavior becomes important when the force--torque signal participates directly in control loops. The acquisition and filtering path should avoid unnecessary dynamic memory allocation, uncontrolled blocking operations, and unpredictable processing delays. ROS2 executors, callback groups, QoS settings, and thread priorities should be selected according to the required update rate and latency. High-rate sensor data may also require appropriate history and reliability settings so that the communication layer does not introduce avoidable delays or excessive buffering.

Fault handling should be designed as part of the driver rather than added afterward. Communication loss, invalid packets, sensor saturation, missing samples, abnormal offsets, timestamp discontinuities, and values outside the physical operating range should be detected explicitly. The driver should distinguish between a valid near-zero measurement and a sensor that has stopped updating. Downstream controllers can then respond appropriately by rejecting invalid data, entering a safe mode, or switching to another sensing source rather than treating corrupted data as a real physical interaction.

Force--torque filtering should also be evaluated against the actual manipulation task. A filter that produces visually smooth signals during logging may be unsuitable for contact detection if its latency delays recognition of an impact. Conversely, a minimally filtered signal may contain too much vibration for stable force control. Practical evaluation should therefore consider noise level, step response, delay, overshoot, frequency response, contact-event detection time, and control stability. Recorded datasets can be particularly useful for comparing filter configurations before deployment on a physical robot.

For Physical AI, the ROS2 force--torque driver is more than a hardware abstraction layer because it establishes the reliability of a physical interaction signal used by perception, learning, and control. A consistent stream of calibrated, time-aligned, filtered, and frame-correct wrench measurements can be combined with tactile images, joint states, vision, and robot actions. These synchronized signals can support contact classification, slip detection, grasp stability estimation, compliant manipulation, and learning from contact-rich experience.

A complete Force--Torque ROS2 architecture therefore connects sensor communication, initialization, timestamping, calibration, compensation, filtering, frame transformation, ROS2 publication, diagnostics, and fault handling into a controlled data path. The final output should be predictable enough for real-time control while retaining sufficient physical information for higher-level perception and learning. This design allows force--torque sensing to function as a dependable interface between the robot's physical interaction with the environment and its ROS2-based intelligence stack.

10.10. Tactile Perception Integration in Manipulator Case

A practical manipulator integrates tactile perception as a continuous sensing layer that complements vision, joint-state feedback, and force--torque measurement. Vision can identify the object and provide an initial pose estimate, while tactile sensors observe the actual contact condition after the end-effector reaches the object. This integration is particularly important when the object becomes partially occluded, when geometric uncertainty remains after visual localization, or when successful manipulation depends on subtle contact changes that cannot be inferred reliably from external cameras.

A representative tactile manipulator architecture begins with an end-effector equipped with tactile sensors on the gripping surfaces and, when required, a wrist-mounted force--torque sensor. Tactile sensing provides local pressure, deformation, contact geometry, and slip information, while the wrist sensor provides the aggregate six-axis wrench. Joint positions, velocities, and commanded motion provide the robot-state context. These signals should be acquired with consistent timestamps so that the system can associate a tactile event with the exact robot motion and mechanical condition that produced it.

The perception pipeline first conditions each sensor stream before fusion. Tactile images or pressure maps may require normalization, background compensation, contact segmentation, and spatial feature extraction. Force--torque measurements require calibration, bias correction, gravity or tool-load compensation, filtering, and coordinate transformation. Joint-state data should be checked for validity and synchronized with the sensor streams. The objective is not simply to produce clean individual signals, but to create physically consistent observations that can be interpreted together at the contact interface.

Contact detection provides the first integrated decision layer. A manipulator approaching an object may initially operate in free space, then transition through first contact into sustained loading. Tactile activation can identify the local contact region, while the force--torque sensor can confirm that a measurable external wrench has developed. Combining these signals reduces dependence on a single threshold and provides contextual information about whether a measured force corresponds to intentional grasping, environmental contact, or an unexpected collision.

Grasp stabilization is the next stage of the integrated perception loop. Tactile sensors can estimate pressure distribution and contact area on individual fingers, while force--torque measurements describe the overall mechanical loading. If pressure migrates across the contact surface or local tactile features begin to move, the system can identify incipient slip. The controller can then adjust finger force or motion before the object is lost. This creates a closed-loop relationship between tactile observation, grasp-state estimation, and manipulation control.

Insertion and assembly tasks demonstrate the value of combining tactile and force information. Vision can guide the manipulator toward the approximate insertion location, but final alignment may depend on physical contact. During insertion, increasing lateral force or torque can indicate misalignment, while tactile patterns can identify which portion of the gripper or tool is contacting the object. The robot can respond by modifying its pose or trajectory rather than continuing to apply increasing force. Tactile perception therefore becomes an active source of information for contact-guided motion.

For optical tactile sensors such as GelSight or DIGIT, the manipulator can additionally extract detailed contact geometry and local motion. Image correlation, optical flow, and feature tracking can estimate displacement within the contact region, providing information about slip direction and local deformation. These image-based measurements complement the lower-dimensional force--torque signal. The force sensor indicates how much aggregate loading is occurring, while the tactile image indicates where that loading is occurring and how the contact pattern is changing.

ROS2 can provide the middleware layer connecting these sensing and control functions. Individual drivers publish calibrated tactile and force--torque measurements, while synchronization and transformation components establish a common temporal and spatial reference. The resulting data can be consumed by contact detection, grasp assessment, manipulation planning, and compliant control nodes. A consistent TF2 frame structure and standardized message interfaces are important because tactile measurements are meaningful only when their spatial relationship to the robot and tool is correctly maintained.

The integrated system should also distinguish perception functions according to their response requirements. Slip detection and collision detection may require a low-latency processing path because delayed information can reduce the effectiveness of corrective action. More computationally intensive tactile image interpretation or learned contact classification can operate on longer temporal windows. Separating fast safety or stabilization signals from richer perception allows the manipulator to maintain rapid reactions without discarding detailed information needed for higher-level reasoning.

A practical case can therefore use a layered decision process in which vision establishes the task context, tactile sensing identifies local contact state, force--torque sensing evaluates global mechanical interaction, and robot-state information provides motion context. A learned model can then combine these observations to estimate states such as free motion, first contact, stable grasp, incipient slip, constrained contact, insertion alignment, or abnormal interaction. The model output can be used by a controller or manipulation policy rather than serving only as an offline classification result.

Data collected from such an integrated manipulator can also become training material for contact-rich learning. Each interaction sequence can associate camera observations, tactile images, force--torque measurements, joint states, robot actions, and task outcomes. Successful and unsuccessful interactions can provide examples for supervised, self-supervised, imitation, or reinforcement learning. The resulting dataset captures not only what the object looked like, but how the physical interaction evolved in response to the robot's actions.

For Physical AI, tactile perception integration in a manipulator represents a transition from object-centered perception to interaction-centered perception. The robot does not rely solely on determining where an object is; it continuously estimates what is happening at the physical interface while acting on that object. Vision supplies global context, tactile sensing supplies local contact evidence, force--torque sensing supplies mechanical interaction information, and robot state supplies action context. Together these streams support perception, prediction, and adaptive action within a unified manipulation loop.

A complete manipulator tactile integration architecture therefore connects sensor acquisition, calibration, synchronization, coordinate transformation, contact detection, tactile feature extraction, force--torque interpretation, multimodal fusion, state estimation, learned perception, and closed-loop control. The final system should preserve low-latency physical feedback while providing sufficiently rich representations for manipulation reasoning and learning. This architecture allows tactile sensing to function not as an additional sensor attached to the manipulator, but as an integral perception layer that continuously informs grasping, insertion, assembly, slip prevention, and contact-rich physical interaction.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Original GitHub manuscript: https://github.com/Statera-Library/08_Vol_14_Perception_and_Sensor_Fusion/blob/main/Chapter_10_Tactile_and_Force_Perception/Chapter_10_Tactile_and_Force_Perception_EN.md

Top comments (0)