DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Perception and Sensor Fusion — Chapter 11: Open Vocabulary Perception [EN]

Chapter 11. Open Vocabulary Perception

11.01. Open Vocabulary Perception Motivation and Architecture

Open-vocabulary perception extends robot perception beyond a predefined list of object classes. Conventional detectors and segmentation systems are normally trained around a closed vocabulary, meaning that the categories recognized during deployment are largely determined during dataset construction and training. Robots operating in homes, factories, warehouses, streets, or unfamiliar environments cannot assume that every relevant object has been enumerated beforehand. :chatgpt-content-reference{index="0"}

The motivation for open-vocabulary perception therefore comes from the mismatch between finite training datasets and effectively unlimited real-world semantics. A mobile robot may encounter unfamiliar equipment, temporary obstacles, unusual containers, new tools, or objects described differently by individual users. Instead of requiring a new annotated dataset and retraining cycle whenever the vocabulary changes, the perception system should interpret novel semantic concepts from language and associate them with visual observations.

This capability changes the basic perception question. A closed-set detector asks which predefined category best explains an image region, whereas an open-vocabulary system can ask where an object described by an arbitrary phrase appears in the scene. Queries such as "red emergency box," "damaged package," "charging connector," or "tool used for tightening bolts" can become semantic inputs to perception. The vocabulary is consequently supplied partly at inference time rather than being completely embedded in the classifier.

The architectural foundation is a shared representation space between vision and language. A visual encoder converts an image, region, or image patch into feature vectors, while a text encoder converts words, category names, phrases, or task descriptions into compatible semantic embeddings. Similarity between these representations provides a mechanism for associating visual evidence with language without constructing a conventional independently trained classifier for every possible category.

Vision-language pretraining is central to this architecture because large collections of image-text pairs expose models to substantially richer semantic variation than conventional class-labeled datasets. The resulting representation can encode relationships among objects, attributes, activities, materials, and contextual concepts. Models derived from this principle can transfer semantic knowledge to downstream robot perception, while subsequent components specialize the representation for localization, detection, segmentation, navigation, or manipulation.

A practical architecture commonly separates semantic encoding from spatial grounding. Language encoders determine what concept the robot is searching for, while visual backbones extract features describing the observed scene. Cross-modal alignment or attention then connects the query with relevant visual regions. A grounding component predicts candidate boxes or spatial references, and a segmentation component can refine these coarse regions into pixel-level masks when accurate object boundaries are required.

This modular organization enables complementary foundation models to form a perception pipeline rather than forcing one network to perform every operation. A vision-language model can provide semantic correspondence, an open-set detector can localize language-specified objects, and a general segmentation model can obtain precise boundaries. The attached volume structure reflects this progression through CLIP-style feature alignment, Grounding DINO, SAM, open-vocabulary 3D segmentation, language-guided navigation, and manipulation. :chatgpt-content-reference{index="1"}

For robotics, open-vocabulary perception must ultimately connect two-dimensional semantics to physical space. Image detections alone do not tell a robot whether an object is reachable, traversable, graspable, or located at a particular three-dimensional coordinate. Camera depth, stereo vision, LiDAR, RGB-D sensing, robot pose, and calibrated sensor transformations can therefore be combined with language-grounded observations to project semantic information into point clouds, occupancy maps, object maps, or scene representations.

Temporal reasoning is equally important because robots perceive continuously rather than from isolated images. An object recognized with moderate confidence in one frame may become clear after the robot changes viewpoint. Tracking and temporal feature aggregation can maintain semantic identity while observations evolve. Persistent object representations also allow a robot to remember that an unfamiliar item observed earlier corresponds to a user-defined concept instead of rediscovering its meaning independently in every frame.

Open-vocabulary perception also provides an important interface between perception and task reasoning. A language instruction such as "go to the fire extinguisher near the exit" can be decomposed into semantic entities and spatial relationships, which perception attempts to ground in sensor observations. Navigation can then receive a physical target derived from semantic understanding. Similarly, manipulation systems can transform instructions such as "pick up the blue cleaning tool" into candidate objects and subsequent pose or grasp estimation.

However, semantic flexibility does not imply guaranteed recognition. Vision-language similarity may produce plausible but incorrect associations, especially for visually similar objects, uncommon terminology, small targets, domain-specific equipment, or scenes that differ substantially from pretraining data. Open-vocabulary models can also assign meaning to regions even when the requested object is absent. Robotics therefore requires confidence estimation, rejection mechanisms, geometric verification, temporal consistency, and explicit handling of unknown or ambiguous observations.

This distinction becomes particularly important in safety-related applications. A robot should not convert every language-grounding result directly into motion. Semantic perception can propose that a region corresponds to a requested object, but independent geometric and safety layers should determine whether approaching, traversing, or manipulating that region is permissible. Open-vocabulary perception is therefore best treated as an expanded semantic sensing capability embedded within a larger perception, planning, and safety architecture.

Computational cost is another architectural constraint. Large vision and language encoders, grounding transformers, segmentation networks, depth processing, and three-dimensional fusion may compete for the same edge GPU resources. A production robot may consequently use hierarchical inference: lightweight perception operates continuously, while expensive open-vocabulary grounding is activated by task queries, uncertain observations, scene changes, or lower-frequency semantic updates. Feature caching and selective segmentation can further reduce redundant computation.

The architecture can also separate fast operational perception from slower semantic interpretation. Conventional detectors remain useful for frequently encountered safety-critical classes because they can provide predictable latency and carefully validated performance. Open-vocabulary models can complement them by handling long-tail objects and dynamically specified concepts. Such hybrid perception avoids treating open-vocabulary capability as a replacement for every established perception component and instead assigns each model to the role where its properties are strongest.

The progression toward open-vocabulary perception represents a broader shift from category recognition toward language-addressable environmental understanding. The robot is no longer limited to answering "which trained class is present?" but can connect human concepts with visual and spatial observations as tasks change. This capability establishes a semantic bridge from sensor data to navigation, manipulation, scene reasoning, and Physical AI, while retaining geometric perception and safety validation as essential foundations.

11.02. CLIP and Vision Language Feature Alignment [w/Code]

CLIP, or Contrastive Language--Image Pre-training, provides a foundational mechanism for connecting visual observations with natural-language concepts. Instead of training an image classifier around a fixed set of predefined labels, CLIP learns representations from paired images and text. This approach creates a semantic interface through which robots can interpret objects and scenes using flexible language descriptions rather than relying exclusively on closed-set categories.

The core CLIP architecture contains two major processing paths: an image encoder and a text encoder. The image encoder transforms an input image into a compact visual feature vector, while the text encoder converts a textual description into a corresponding language feature vector. Both representations are projected into a shared embedding space, where their relationship can be evaluated directly through similarity rather than through a conventional fixed classification head.

Image encoders may be based on convolutional networks or Vision Transformers, with modern implementations commonly using transformer-based representations. The encoder divides visual information into features that capture shapes, textures, objects, contextual relationships, and higher-level semantics. After processing, the representation is projected and normalized so that it can be compared efficiently with text embeddings produced independently by the language processing branch.

The text encoder converts words or phrases such as "forklift," "red emergency box," or "open doorway" into semantic vectors. Tokenization first represents the language input as a sequence of tokens, after which a transformer models relationships among them. The resulting representation captures more than literal category names because attributes, contextual descriptions, and combinations of concepts can also occupy meaningful locations within the learned vision-language embedding space.

Feature alignment is learned through contrastive training. During training, a batch contains multiple image-text pairs, and the model learns to increase similarity between representations belonging to matching pairs while reducing similarity between mismatched combinations. This objective encourages images and their corresponding descriptions to become neighbors in the shared embedding space, while semantically unrelated observations are pushed farther apart.

Cosine similarity is commonly used to measure the relationship between normalized image and text embeddings. Given a visual feature vector and several candidate text vectors, the system calculates their similarities and identifies which textual concepts best correspond to the visual observation. A learned temperature or scaling parameter can control the distribution of similarity scores, allowing them to be transformed into relative probabilities for downstream interpretation.

This mechanism enables zero-shot classification without training a new classifier for every target category. Candidate class names can be converted into prompts such as "a photo of a forklift" or "a photo of a pallet," encoded into text features, and compared against an image representation. Changing the vocabulary then becomes primarily a matter of changing the language prompts rather than modifying and retraining the final classification layer.

Prompt design can significantly affect feature alignment because different linguistic descriptions may generate somewhat different text embeddings for the same underlying concept. A robot searching for a "charging station" might benefit from prompts describing its appearance, function, or environmental context. Prompt ensembles can combine several alternative descriptions and average their representations, producing a more stable semantic prototype than relying on a single short category label.

For open-vocabulary robotic perception, whole-image similarity alone is insufficient because robots must determine where relevant concepts occur. CLIP-style features can therefore be extracted from image regions, patches, object proposals, or dense feature maps. These localized representations can be compared with text embeddings to produce semantic similarity maps, enabling the system to associate language queries with particular portions of the observed scene.

Dense vision-language features provide a bridge from image-level recognition toward detection and segmentation. When spatially preserved visual features remain aligned with language embeddings, each region can be evaluated against arbitrary textual concepts. Subsequent detection or segmentation modules can convert these similarities into bounding boxes or masks. This principle supports architectures that combine vision-language representation learning with grounding and general-purpose segmentation models.

Robotics additionally requires vision-language features to interact with geometry. A semantic feature associated with a camera pixel or image region can be projected into three-dimensional space using depth information and calibrated sensor transformations. RGB-D cameras, stereo depth, or LiDAR-camera fusion can therefore attach language-compatible embeddings to points, voxels, objects, or map elements, creating spatial representations that remain queryable using natural language.

Such representations are particularly useful for semantic mapping. Instead of storing only fixed class labels such as wall, chair, or vehicle, a map can preserve richer feature embeddings associated with spatial locations. When a new language query arrives, the robot compares its text embedding with stored visual features to retrieve relevant locations. This allows semantic queries to evolve after mapping without requiring every useful category to have been specified beforehand.

Temporal aggregation can further improve feature reliability. A mobile robot observes the same object from different distances, viewpoints, lighting conditions, and levels of occlusion. Rather than treating every observation independently, features from multiple frames can be associated through tracking or geometric correspondence and fused into a persistent representation. Repeated observations can strengthen stable semantic evidence while reducing dependence on a single ambiguous image.

Vision-language alignment also creates an interface between perception and task-level language. An instruction such as "find the cart carrying the damaged package" contains objects, attributes, and relationships that can be converted into semantic queries. CLIP-like representations help identify visual candidates, while grounding, spatial reasoning, tracking, and scene understanding refine the result. The embedding space therefore serves as an intermediate semantic layer between raw sensor observations and higher-level robot reasoning.

However, similarity in an embedding space should not be interpreted as definitive physical understanding. CLIP can confuse visually related concepts, struggle with fine-grained industrial distinctions, or respond strongly to contextual correlations rather than the intended object itself. Small objects, unusual viewpoints, specialized equipment, text printed inside images, and environments far from the pretraining distribution can produce unreliable or poorly calibrated similarity scores.

Production systems should therefore combine vision-language similarity with additional evidence. Conventional detectors can continue handling validated safety-critical classes, while CLIP-style features provide flexible recognition for long-tail and dynamically specified concepts. Depth consistency, object tracking, geometric constraints, multiple viewpoints, confidence thresholds, and unknown rejection can prevent weak semantic similarities from being treated automatically as reliable physical facts.

Computational efficiency is especially important on edge robots because dense visual embeddings can consume substantial GPU memory and inference time. Feature extraction may therefore operate at reduced frequency, selected image resolutions, or only within candidate regions. Visual features can also be cached and reused for multiple text queries, because new language prompts can often be encoded and compared without repeatedly executing the expensive image backbone.

CLIP-style vision-language feature alignment fundamentally changes how semantic knowledge enters a perception system. Rather than encoding every required category into a dedicated output neuron, the robot maintains a representation in which visual observations and language concepts can be compared dynamically. This shared semantic space becomes a foundation for open-vocabulary detection, segmentation, 3D semantic mapping, language-guided navigation, novel-object manipulation, and increasingly general Physical AI perception.

11.03. Grounding DINO Open Set Object Detection [w/Code]

Grounding DINO extends transformer-based object detection from a fixed category vocabulary toward language-conditioned open-set detection. Instead of predicting objects only from class labels defined during training, it accepts textual concepts as queries and searches the image for regions that correspond to those descriptions. This capability allows a robot to request objects such as "fire extinguisher," "damaged package," or "red tool box" without requiring a dedicated output class for every possible concept.

The key architectural idea is to combine a strong visual representation with a language representation before producing detection results. An image backbone extracts multi-scale visual features from the input image, while a text encoder converts category names, phrases, or longer descriptions into language features. Grounding DINO then establishes correspondence between these two modalities so that detection is conditioned directly on the semantic information contained in the text query.

Unlike conventional detectors whose classification heads produce probabilities over a predetermined category set, Grounding DINO treats language as part of the detection process. Text prompts can contain multiple object concepts, and the detector evaluates visual regions against their associated language representations. Consequently, the vocabulary can be changed during inference without redesigning the detector output layer, which is a fundamental requirement for open-vocabulary robotic perception.

The visual backbone produces hierarchical image features representing objects at different spatial scales. Small tools, medium-sized packages, vehicles, doors, and large environmental structures may occupy dramatically different portions of an image, so multi-scale representations are important. These features retain spatial information needed for localization while providing increasingly semantic descriptions that can later interact with the language-conditioned detection transformer.

The language branch transforms an input prompt into token-level text features. Rather than reducing the complete prompt immediately to a single category vector, token-level representations preserve information about individual words and phrases. This allows the architecture to establish fine-grained correspondence between linguistic concepts and visual evidence, supporting prompts that contain object names together with attributes or descriptive information.

A central component is cross-modal feature fusion, where image and language information interact before final object prediction. Vision features are enhanced using linguistic context, while text features obtain information about relevant visual content. This deep interaction differs from architectures that perform visual detection first and attach language labels afterward. Grounding becomes part of the detection computation itself rather than merely a post-processing classification step.

Grounding DINO also builds on the query-based detection principles of DETR and DINO-style architectures. Detection queries are processed by a transformer decoder to predict candidate objects and their bounding boxes. Language-conditioned information guides these queries toward regions that correspond to the supplied textual concepts. The resulting predictions therefore contain both spatial localization and semantic correspondence with portions of the input text.

Query selection is particularly important because an image can contain a very large number of possible spatial locations while only a small subset corresponds to meaningful objects. Language-guided query selection helps identify visual features that are strongly related to the supplied text before decoder processing. This improves the connection between candidate regions and requested concepts while reducing dependence on completely generic object proposals.

The output of the detector typically consists of candidate bounding boxes together with scores describing their correspondence to textual tokens or phrases. Thresholds can then be applied to remove weak predictions. In a robot searching for a "yellow safety helmet," for example, the system can return spatial regions associated with the phrase rather than simply selecting among classes contained in a conventional detector's training vocabulary.

Open-set detection becomes especially useful when robot tasks are specified dynamically. A warehouse robot may receive instructions to locate a newly introduced container, a maintenance robot may search for an unfamiliar component described by an operator, and a service robot may need to identify household objects never included in its application-specific training dataset. Language-conditioned detection reduces the need to retrain the perception system whenever the operational vocabulary changes.

Grounding DINO can also serve as a front end for segmentation. A language query first identifies candidate objects and produces bounding boxes, after which a general segmentation model such as SAM can use those boxes as prompts to generate precise masks. This Grounding DINO--SAM combination transforms natural-language concepts into pixel-level object regions and provides a practical architecture for open-vocabulary segmentation in robotic perception.

For physical robots, the detected two-dimensional boxes usually need to be connected with depth and geometry. RGB-D measurements, stereo depth, or calibrated LiDAR-camera fusion can associate the detected image region with three-dimensional points. The system can then estimate object position, dimensions, orientation candidates, or occupied space and transform a language-grounded detection into information usable by navigation or manipulation modules.

Temporal integration further improves reliability. A mobile robot may detect the same object repeatedly while moving through the environment, allowing bounding boxes and language correspondence scores to be associated across frames. Tracking can distinguish persistent objects from transient false detections, while observations from different viewpoints provide additional evidence. The resulting object representation can then be inserted into a semantic map or persistent scene memory.

Grounding DINO also enables language-guided navigation by converting semantic goals into observable targets. A command such as "go to the red fire extinguisher near the exit" can provide object and attribute descriptions to the detector. Candidate detections can be combined with depth, localization, and semantic-map information to determine a navigable target position, while spatial reasoning can evaluate additional relations such as "near the exit."

Manipulation systems can use the same architecture to identify novel objects before performing more precise geometric processing. Grounding DINO may locate "the blue wrench" or "the damaged connector," after which segmentation, depth reconstruction, pose estimation, and grasp planning determine how the robot should interact with it. Open-set detection therefore acts as semantic object discovery rather than replacing the geometric perception required for physical manipulation.

Detection scores must nevertheless be treated carefully because language grounding does not guarantee that a requested object actually exists. Similar objects, contextual cues, unusual viewpoints, occlusion, small target size, and ambiguous prompts can produce incorrect associations. A robot should support an explicit unknown or not-found state instead of automatically accepting the highest-scoring candidate whenever every available detection has weak evidence.

Prompt formulation also influences detection quality. Short category names may work well for common objects, while specialized industrial components may require more descriptive phrases containing appearance, function, or attributes. However, excessively complicated prompts can introduce ambiguity. Production systems should therefore define prompt templates and vocabulary-management strategies while still retaining the flexibility that distinguishes open-set perception from conventional closed-set detection.

Real-time deployment introduces additional constraints because transformer-based visual processing and cross-modal fusion can require substantial GPU computation. Robots may reduce image resolution, restrict detection to selected cameras, activate open-set detection only when semantic queries occur, or execute the model at a lower frequency than safety-critical perception. Detected objects can then be tracked at higher rates using lighter-weight algorithms between expensive grounding updates.

A practical robot perception architecture therefore often combines Grounding DINO with conventional detectors rather than replacing them completely. Validated closed-set models can continuously detect pedestrians, vehicles, or other safety-critical classes with predictable latency, while Grounding DINO handles long-tail objects and task-specific language queries. Segmentation, depth sensing, tracking, and geometric verification can subsequently refine its detections before they influence robot behavior.

Grounding DINO represents an important transition from predefined object recognition toward language-addressable object discovery. By integrating visual features, token-level language representations, cross-modal fusion, language-guided queries, and transformer-based localization, it allows the robot's detectable vocabulary to evolve with its instructions. Within open-vocabulary perception, this capability forms a practical bridge from vision-language feature alignment to segmentation, 3D grounding, semantic navigation, and novel-object manipulation.

11.04. Segment Anything Model SAM for Robots [w/Code]

The Segment Anything Model (SAM) provides a general-purpose image segmentation capability that can generate object masks from visual prompts rather than depending entirely on a fixed semantic class vocabulary. In robotic perception, this makes SAM particularly useful after an open-vocabulary detector identifies an object of interest. Instead of requiring a separately trained segmentation model for every object category, the robot can use a detected region, point, or other visual cue to obtain a precise pixel-level representation.

SAM is based on a large-scale vision encoder, a prompt encoder, and a mask decoder. The image encoder converts the complete image into a reusable visual representation, while the prompt encoder represents information such as points, bounding boxes, or other segmentation cues. The mask decoder then combines these representations to generate one or more candidate masks. This separation allows the expensive image representation to be reused for multiple segmentation requests within the same scene.

The prompt-driven design is particularly important for robotics because the desired segmentation target may change according to the current task. A robot may first receive a bounding box from an open-vocabulary detector and provide that box to SAM as a prompt. Alternatively, a selected image point can indicate the object to segment. The resulting mask can then be associated with the semantic identity provided by the preceding perception module, creating a flexible connection between language-guided detection and precise visual segmentation.

In a typical open-vocabulary perception pipeline, Grounding DINO and SAM can operate as complementary components. Grounding DINO determines which image region corresponds to a language query such as "red emergency box," while SAM refines that region into a detailed object mask. This combination separates semantic discovery from boundary extraction. The detector supplies the meaning and approximate location, while the segmentation model provides a more accurate representation of the visible object extent.

The quality of the resulting mask depends strongly on the quality of the prompt and the visual conditions. Accurate bounding boxes or informative points generally provide stronger guidance than poorly localized prompts. Occlusion, reflections, transparent surfaces, low illumination, motion blur, clutter, and objects with weak visual boundaries can make segmentation more difficult. Robotic systems should therefore treat the generated mask as an estimated perception result rather than an automatically guaranteed physical boundary.

SAM can also produce multiple candidate masks when the prompt is ambiguous. This capability is useful when a point or region may correspond to more than one plausible object interpretation. A downstream robot perception system can evaluate the alternatives using confidence, object appearance, depth consistency, temporal stability, or task context. Such processing allows segmentation to remain flexible while preventing an uncertain mask from being immediately converted into a physical action.

For robotics, two-dimensional segmentation becomes considerably more useful when combined with depth and geometric information. A mask obtained from an RGB image can be associated with RGB-D depth, stereo reconstruction, or LiDAR-camera correspondence. Pixels inside the mask can then be projected into three-dimensional space to estimate an object's position, occupied volume, surface geometry, or approximate dimensions. This provides a bridge from visual segmentation to spatial reasoning and physical interaction.

The resulting three-dimensional representation can support manipulation tasks. After SAM identifies the visible extent of a requested object, depth information can be used to construct a point cloud for that region. Subsequent pose estimation or grasp-planning modules can analyze the object's geometry and determine feasible interaction points. In this architecture, SAM does not perform grasp planning itself; it supplies a more precise visual region that downstream geometric and manipulation algorithms can exploit.

SAM is also valuable for mobile robots because segmentation can be maintained across changing viewpoints. A robot moving through an environment may observe the same object under different scales, orientations, and partial occlusions. When combined with object tracking, masks from consecutive frames can be associated with a persistent object identity. Temporal consistency checks can then reject unstable masks and improve the reliability of the semantic object representation used by navigation or mapping systems.

Semantic mapping provides another important application. Instead of recording only a bounding box or a coarse object center, a robot can associate segmentation masks with spatial map elements. When combined with camera calibration, depth, localization, and three-dimensional reconstruction, segmented regions can contribute to semantic point clouds, object maps, or occupancy representations. A language-grounded object can therefore become a persistent spatial entity that later navigation or task reasoning modules can query.

SAM can support navigation by providing accurate regions for objects that act as semantic landmarks or task targets. For example, after a language query identifies an emergency device, the segmentation mask can help distinguish the target from nearby visually similar structures. Combined with depth and robot localization, the target can be converted into a spatial representation for navigation. However, segmentation alone does not determine whether a location is safe or reachable; planning and safety layers must independently evaluate traversability, collision risk, and other constraints.

Edge deployment introduces practical computational considerations. The image encoder can represent a substantial portion of the inference cost, particularly when high-resolution images are processed. A robotic implementation may therefore reuse image features, restrict segmentation to relevant frames or regions, reduce input resolution when acceptable, or execute SAM less frequently than lightweight tracking and control components. The appropriate strategy depends on the robot's GPU capability, camera resolution, required latency, and operational environment.

SAM should therefore be regarded as a general visual segmentation component within a larger robotic perception architecture rather than as a complete perception system. Open-vocabulary detection can determine what the robot should look for, SAM can determine the corresponding pixel region, depth and LiDAR can provide three-dimensional structure, tracking can maintain temporal identity, and task-level reasoning can determine how the information should be used. This modular organization allows each component to perform a clearly defined function while supporting flexible object understanding.

For safety-critical applications, segmentation results should also be independently validated before influencing physical behavior. A visually plausible mask may include background pixels, exclude partially occluded object regions, or incorrectly merge adjacent objects. Geometric consistency, temporal stability, sensor agreement, and confidence thresholds can provide additional evidence. When the available evidence is insufficient, the system should retain an unknown or uncertain state instead of forcing the segmentation result into a deterministic action.

The importance of SAM for robotic perception ultimately comes from its ability to transform relatively simple visual prompts into detailed object regions without requiring a dedicated segmentation class for every possible object. When integrated with open-vocabulary models, SAM enables a perception chain in which language specifies the target, detection identifies its approximate location, segmentation extracts its visible extent, and geometric processing converts the result into spatial information. This makes SAM an important component for open-vocabulary navigation, manipulation, semantic mapping, and broader Physical AI perception.

11.05. Open Vocabulary 3D Segmentation OpenScene [w/Code]

Open-vocabulary 3D segmentation extends language-grounded perception from individual images into three-dimensional environments. The provided structure places this topic within Open-Vocabulary Perception, following CLIP-based feature alignment, Grounding DINO detection, and SAM segmentation, and before language-guided navigation and novel-object manipulation. The central objective is to assign semantic meanings to 3D points or regions using flexible natural-language concepts rather than a fixed set of predefined 3D classes.

OpenScene represents an important approach to this problem by connecting pretrained vision-language representations with three-dimensional scene geometry. A conventional 3D segmentation model usually learns a closed vocabulary from annotated point clouds, which limits recognition to categories represented during training. An open-vocabulary system instead transfers semantic knowledge from large-scale image-language models into 3D space, allowing queries such as "red emergency box," "office chair," or "charging station" to be evaluated against spatial scene representations.

The basic processing pipeline begins with RGB images and a reconstructed or continuously observed 3D scene. A vision-language model provides semantic features for the camera images, while camera calibration and geometric reconstruction establish correspondence between image pixels and 3D points. Image-level semantic information can then be projected into the 3D representation. By aggregating observations from multiple viewpoints, individual 3D points can obtain richer and more stable semantic features than would be possible from a single camera frame.

A critical component is the alignment between 2D visual features and 3D geometric elements. Each visible 3D point may appear in several camera images, potentially under different viewpoints, lighting conditions, scales, and occlusion states. Features associated with these observations can be aggregated to create a persistent 3D semantic representation. The resulting point-level features remain connected to the shared vision-language embedding space, allowing them to be compared with text embeddings after the original image processing stage.

This approach changes the role of 3D segmentation from simple category assignment to semantic querying of physical space. Instead of asking only whether a point belongs to a class such as chair or table, the robot can compare its 3D representation with a language description. A query such as "objects that can be used for seating" or "red safety equipment" can therefore retrieve spatial regions whose learned representations are semantically compatible with the description, even when those concepts were not explicit training classes.

Multi-view feature aggregation is particularly important for mobile robots because a single observation is rarely sufficient to describe an object completely. As the robot moves, previously hidden surfaces may become visible and ambiguous observations can be reinforced by additional views. Aggregating features across time and viewpoint can increase semantic stability while reducing the influence of temporary occlusion, blur, reflections, or unfavorable viewing angles.

The resulting open-vocabulary 3D representation can support point-level, region-level, and object-level reasoning. Point-level features provide fine spatial resolution, while connected regions can be grouped into candidate objects or semantic areas. Geometric information such as surface continuity, spatial proximity, depth, and object boundaries can help separate neighboring objects that have similar visual semantics. This combination allows language-based semantic information to remain grounded in actual physical geometry.

Open-vocabulary 3D segmentation becomes especially valuable when integrated with semantic mapping. Instead of storing only fixed labels in a map, a robot can maintain spatial features that remain searchable through language. A previously mapped area can therefore be queried later with concepts that were not explicitly specified during mapping. This creates a more flexible semantic memory in which spatial information and language-compatible representations coexist.

The connection to navigation is direct because 3D segmentation provides semantic targets together with their physical locations. A language instruction can identify a desired object or region, while the 3D representation determines where that target exists relative to the robot. Navigation can then use the retrieved position together with occupancy, traversability, localization, and obstacle information. Semantic recognition therefore becomes an input to navigation rather than an isolated classification result.

Manipulation systems can use the same representation to discover novel objects in physical space. Once a language query identifies a candidate object, its associated 3D points can provide information about position, approximate shape, visible surfaces, and spatial extent. Subsequent pose estimation, grasp generation, and collision checking can operate on this geometrically grounded region. The open-vocabulary component identifies what the robot should consider, while downstream geometric modules determine how it can physically interact with the object.

However, semantic similarity in 3D space does not guarantee correct physical identification. Sparse observations, incorrect camera calibration, reconstruction errors, reflective surfaces, occlusion, motion, and accumulated feature noise can produce unstable or spatially incorrect semantic regions. A robust system should therefore combine language similarity with geometric consistency, multi-view agreement, temporal stability, depth information, and confidence estimation. Unknown or ambiguous regions should remain explicitly represented rather than being forced into a predefined class.

Real-time deployment introduces additional challenges because maintaining dense 3D semantic features can require significant memory, GPU computation, and data movement. A practical robot may update semantic features selectively, process only regions relevant to active language queries, reuse previously computed image features, or maintain different update rates for geometry and semantics. This allows high-rate geometric mapping and navigation to continue while computationally expensive open-vocabulary semantic updates occur less frequently.

OpenScene-style open-vocabulary 3D segmentation therefore forms a bridge between vision-language foundation models and spatial intelligence. Image-language models provide broad semantic knowledge, multi-view projection transfers that knowledge into physical space, and 3D aggregation creates persistent spatial representations that can be queried using natural language. The resulting architecture supports a progression from visual perception to semantic mapping, language-guided navigation, novel-object manipulation, and broader Physical AI systems in which robots must understand not only what they see, but where meaningful objects and concepts exist in the world.

11.06. Language Guided Navigation with OV Perception [w/Code]

Language-guided navigation with open-vocabulary perception extends conventional robot navigation by allowing navigation goals to be specified through natural language rather than only through coordinates, predefined object IDs, or fixed semantic classes. The approach builds upon the preceding open-vocabulary pipeline, in which vision-language alignment, language-conditioned detection, image segmentation, and 3D semantic representations provide flexible descriptions of objects and regions in the physical environment. :chatgpt-content-reference{index="0"}

A conventional navigation system generally receives a geometric goal such as a pose, waypoint, or map coordinate. Language-guided navigation introduces a semantic goal such as "go to the red emergency box," "find the charging station," or "move to the chair near the entrance." The robot must first interpret the language, identify the corresponding physical entity using open-vocabulary perception, estimate its spatial relationship to the robot, and then convert the semantic target into a navigation-compatible goal.

The first stage is language understanding and goal decomposition. A natural-language instruction may contain an object, attribute, spatial relationship, destination, or action condition. The system extracts meaningful concepts such as "red," "emergency box," and "near the entrance," then converts them into perception queries. These queries can be passed to open-vocabulary detectors or vision-language models, allowing the navigation system to search for concepts that were not necessarily included as fixed classes during conventional perception training.

Open-vocabulary detection provides the initial connection between the language goal and visual observations. A model such as Grounding DINO can receive the textual object description and identify candidate image regions corresponding to the query. If the robot searches for a "red emergency box," the detector can produce candidate bounding boxes and semantic correspondence scores. These detections provide approximate image locations, but they must subsequently be connected to depth, localization, and three-dimensional geometry before they can become navigation targets.

Segmentation can refine the detected region and provide a more precise representation of the target. A model such as SAM can use the detected bounding box or another visual prompt to generate a pixel-level mask. Combining this mask with RGB-D, stereo, or LiDAR information allows the system to estimate which three-dimensional points belong to the requested object. The resulting spatial representation can be used to determine the object's location, approximate extent, and relationship to nearby obstacles or navigable regions.

Open-vocabulary 3D perception provides a more persistent representation for navigation. Instead of repeatedly interpreting every camera frame independently, the robot can associate language-compatible features with 3D points, regions, or objects in a semantic map. Multi-view observations can reinforce semantic evidence as the robot moves through the environment. A language query can then be matched against the accumulated spatial representation, allowing the robot to search previously observed areas using concepts that were not necessarily specified when the map was created.

The semantic target must then be transformed into a navigation goal. An object detected in an image is not automatically a valid position for the robot to occupy. The navigation system may instead compute an approach pose, a nearby reachable position, or a safe observation point. For example, identifying a fire extinguisher does not mean that the robot should drive directly onto its location. The target must be related to the robot's footprint, obstacle boundaries, free space, orientation requirements, and task-specific approach constraints.

Spatial relationships make language-guided navigation more expressive but also more demanding. A command such as "go to the chair near the window" requires the system to identify both the chair and the window and then evaluate their spatial relationship. Similarly, "find the red box next to the charging station" requires multiple semantic entities and a relational constraint. Open-vocabulary perception supplies candidate objects, while the navigation reasoning layer evaluates relationships and selects the candidate that best satisfies the complete instruction.

The navigation stack can then use conventional planning methods once a semantic target has been converted into a geometric representation. Global planning can determine a collision-free route through the map, while local planning reacts to dynamic obstacles and changing environmental conditions. The semantic perception layer therefore does not replace localization, path planning, obstacle avoidance, or motion control. Instead, it provides a flexible mechanism for converting human-oriented semantic goals into the geometric targets required by those navigation components.

Continuous perception is important because semantic targets may move, become occluded, or be incorrectly localized. A robot approaching a requested object should continue observing the target and updating its estimated position. Tracking can maintain object identity across frames, while new observations can correct earlier localization errors. If the target disappears or confidence falls below an acceptable threshold, the navigation system should be able to pause, search, re-observe, or return to a safe state rather than blindly continuing toward an outdated position.

Uncertainty handling is particularly important for open-vocabulary navigation. Language similarity does not guarantee that a detected region corresponds to the intended physical object. Multiple visually similar candidates may exist, or the requested object may not be present at all. The system should therefore maintain confidence values and candidate hypotheses and combine semantic evidence with geometric consistency, depth information, temporal stability, and spatial relationships. An explicit unknown or not-found state is preferable to forcing an uncertain detection into a navigation decision.

Semantic maps can provide memory that reduces repeated perception effort. Once an object has been detected and geometrically localized, its semantic identity and spatial representation can be stored in the map. Future language queries can retrieve this information without requiring the robot to rediscover the same object from the beginning. However, the stored representation should include temporal information because objects can move, disappear, or change their environment. Persistent semantic memory therefore needs mechanisms for validation and updating.

Language-guided navigation also benefits from hierarchical reasoning. A high-level instruction can be interpreted as a semantic objective, an intermediate module can identify and localize the requested entity, and a conventional navigation stack can generate and execute the physical trajectory. This separation makes the system easier to validate because semantic reasoning does not directly control motors. The final motion remains constrained by localization, obstacle avoidance, vehicle dynamics, safety rules, and the navigation controller.

The approach can support a wide range of robot platforms because the semantic layer is relatively independent of the underlying mobility system. An indoor AMR may search for a shelf, pallet, workstation, or charging station, while an outdoor robot may search for a gate, vehicle, inspection target, or infrastructure component. A quadruped may use terrain-related semantic concepts, while a humanoid or mobile manipulator may use language to locate objects that later become manipulation targets.

Edge deployment introduces practical computational constraints. Open-vocabulary models can be considerably more expensive than conventional lightweight detectors, particularly when high-resolution images, multiple cameras, dense segmentation, or 3D feature processing are involved. A practical system can therefore use lightweight perception continuously and invoke expensive language-grounded perception only when a semantic query is active, when uncertainty increases, or when a new target must be discovered. Cached features and persistent semantic maps can further reduce repeated computation.

Safety requires a clear separation between semantic interpretation and physical action. A language model or open-vocabulary detector should not directly command the robot to move based solely on semantic similarity. The detected target must pass through geometric validation, localization, obstacle assessment, traversability analysis, and navigation safety constraints. This layered structure allows open-vocabulary perception to increase the flexibility of navigation while preserving deterministic safeguards for physical movement.

Language-guided navigation therefore forms a direct bridge between human-level task descriptions and conventional robot navigation. Natural language specifies what the robot should find, open-vocabulary perception identifies the corresponding physical entities, 3D perception determines where those entities exist, semantic reasoning resolves relationships, and the navigation stack converts the result into a safe geometric trajectory. This architecture transforms navigation from a system that primarily follows predefined coordinates into one capable of responding to dynamically specified semantic goals while retaining the established foundations of localization, planning, obstacle avoidance, and control.

11.07. OV Perception for Novel Object Manipulation [w/Code]

Open-vocabulary perception for novel-object manipulation extends robotic manipulation beyond objects that were explicitly included in the training vocabulary. The topic follows open-vocabulary 3D segmentation and language-guided navigation in the provided structure, and precedes zero-shot affordance estimation and edge deployment optimization. The central objective is to allow a robot to identify previously unseen or dynamically specified objects and connect their semantic descriptions to the geometric information required for physical interaction.

A conventional manipulation pipeline often assumes that the target object belongs to a predefined set of categories. Detection, pose estimation, grasp generation, and control models can consequently be optimized for known objects. Novel-object manipulation removes this assumption by allowing a language description, visual query, or task instruction to specify an object that may not have a dedicated detector or manipulation model. Open-vocabulary perception therefore becomes the entry point for discovering what the robot should manipulate.

The process can begin with a natural-language instruction such as "pick up the blue tool," "grasp the red container," or "move the damaged connector." A vision-language model converts the textual description into a semantic representation, while an open-vocabulary detector searches the current scene for visually compatible regions. Grounding DINO can provide candidate bounding boxes, while CLIP-style features can provide semantic correspondence between the language query and observed image content.

Segmentation can then refine the detected target into a more accurate visual region. A model such as SAM can use the detected bounding box or another visual prompt to produce a pixel-level mask. The mask is important because a manipulation system needs more than the approximate center of an object. It must distinguish the target from neighboring objects, estimate its visible boundaries, and determine which visual evidence belongs to the object that the language instruction actually refers to.

The next step is to connect the segmented object with three-dimensional geometry. RGB-D cameras, stereo systems, or calibrated LiDAR-camera fusion can associate the segmentation mask with depth measurements and reconstruct a target point cloud. This representation can provide the object's approximate position, dimensions, visible surfaces, and spatial extent. Camera calibration and robot kinematics are then required to transform the object representation into the coordinate system used by the manipulator.

Novel-object manipulation requires careful separation between semantic identification and physical interaction. Open-vocabulary perception can determine that a region is likely to correspond to a "blue tool," but it does not automatically determine how the robot should grasp it. Pose estimation, surface analysis, collision checking, grasp generation, and inverse kinematics must use the perception result to calculate physically feasible actions. The semantic layer identifies the target, while geometric and control layers determine the interaction.

The object's shape may be unknown or substantially different from previously encountered objects. Consequently, grasp planning should not depend entirely on a fixed object model. The segmented point cloud can instead be analyzed for candidate surfaces, edges, handles, cavities, or stable contact regions. A grasp generator can evaluate these geometric structures according to the capabilities of the robot hand or gripper, producing candidate grasp poses without requiring a complete CAD model of the novel object.

Language can also specify attributes that influence manipulation. Instructions such as "pick up the small red container," "grasp the handle of the black tool," or "move the damaged part without touching the cable" contain semantic constraints in addition to object identity. Open-vocabulary perception can provide candidate regions, while language reasoning identifies relevant attributes and relationships. These constraints can then be passed to grasp planning and motion planning as task-specific conditions.

Three-dimensional semantic representations become particularly valuable when multiple candidate objects are visually similar. If several containers are present, the system can combine color, shape, spatial location, surrounding objects, and language relationships to determine which candidate best matches the instruction. A semantic map or persistent object representation can provide additional context when the target was observed earlier. This allows manipulation to use information accumulated across multiple viewpoints rather than relying on a single image.

Temporal perception is also important during manipulation because the target and surrounding objects may move. Tracking can maintain the identity of the selected object as the robot approaches it. Repeated observations can update the estimated position and compensate for changes caused by viewpoint, occlusion, or motion. If the target becomes ambiguous or disappears, the manipulation system should be able to pause and request additional perception rather than continuing with an outdated object pose.

Uncertainty must be explicitly considered because open-vocabulary similarity does not guarantee correct object identification. Visually similar objects, unusual appearances, transparent materials, reflective surfaces, partial occlusion, and unfamiliar object configurations can produce incorrect detections or masks. A robust system should combine semantic confidence with geometric consistency, depth quality, multi-view agreement, temporal stability, and task constraints. When evidence is insufficient, the system should maintain an uncertain or unknown state instead of executing an irreversible grasp.

After target identification, the system can construct a manipulation scene representation containing the target object, nearby obstacles, robot configuration, and candidate interaction regions. Collision geometry is particularly important because the segmented object may be surrounded by other objects or located inside a constrained workspace. The manipulation planner must ensure that the selected end-effector trajectory remains compatible with the robot's kinematics and avoids unintended contact.

The perception-to-action interface should therefore expose structured information rather than only a class label. Useful outputs can include the semantic identity or query correspondence, segmentation mask, three-dimensional point cloud, object pose estimate, confidence, candidate grasp regions, and temporal tracking state. Such a representation allows downstream modules to decide whether the available evidence is sufficient for grasp planning and execution.

After a grasp is generated, visual feedback can continue to support closed-loop manipulation. The robot can observe whether the gripper approaches the expected location, whether the object moves as predicted, and whether the grasp remains stable after contact. Updated perception can correct errors before or during subsequent motion. This closed-loop structure is especially important for novel objects because their geometry and physical properties may differ from assumptions made during initial planning.

The same architecture can support a range of manipulation behaviors beyond simple pick-and-place. A robot may be instructed to identify and inspect an unfamiliar component, move an object to a specified location, select one item from a group, or manipulate an object according to a semantic attribute. Open-vocabulary perception provides the flexible object-discovery mechanism, while task reasoning, affordance estimation, grasp planning, and motion control determine the appropriate physical behavior.

Computational constraints become significant when high-resolution open-vocabulary perception is combined with depth processing, segmentation, tracking, and manipulation planning. A practical robot can use lightweight object tracking and geometric processing continuously while invoking expensive vision-language inference when a new target must be identified or when confidence decreases. Cached visual features, persistent 3D representations, and region-based processing can further reduce redundant computation on edge hardware.

Safety requires that semantic perception never become an uncontrolled direct path to actuator commands. Before physical interaction, the system should validate target identity, spatial location, collision geometry, grasp feasibility, robot configuration, and environmental conditions. For unfamiliar objects, additional checks may be required because the object's weight, flexibility, fragility, or physical response may not be known from visual appearance alone. The manipulation controller should therefore maintain conservative behavior when perception or physical assumptions are uncertain.

Open-vocabulary perception ultimately provides a semantic front end for manipulation of objects beyond a fixed training vocabulary. Language specifies the intended target, vision-language models establish semantic correspondence, open-vocabulary detection localizes candidates, segmentation isolates the object, and 3D perception provides geometric structure. Grasp planning, affordance reasoning, kinematic validation, and closed-loop control then transform that information into physical interaction. This architecture creates a practical pathway from flexible semantic object discovery toward manipulation of novel objects while preserving the geometric and safety requirements of real robotic systems.

11.08. Zero Shot Affordance Estimation [w/Code]

Zero-shot affordance estimation extends open-vocabulary perception from identifying what an object is to estimating how an object can be physically used. In the chapter structure, this topic follows novel-object manipulation and precedes edge deployment optimization, placing affordance reasoning as a bridge between semantic object understanding and executable robot behavior. The central idea is to infer actionable regions or possible interactions without requiring a separate affordance model trained for every object category.

An affordance describes an action possibility provided by an object or part of an environment relative to a particular agent. A handle may afford grasping, a flat surface may afford placing, a button may afford pressing, and an opening may afford insertion. Traditional affordance estimation can be trained on predefined objects and annotated interaction regions, but this approach becomes restrictive when robots encounter unfamiliar objects. Zero-shot estimation instead attempts to transfer semantic and functional knowledge to objects and interaction regions not explicitly represented during training.

Language provides a natural interface for specifying the desired affordance. A robot may receive instructions such as "find where to grasp the tool," "locate a surface for placing the object," or "find the button to press." A vision-language model can represent the relationship between the language description and observed visual features, while open-vocabulary perception identifies candidate objects and regions. The system can therefore search for an interaction possibility rather than merely assigning a category label to the complete object.

The first stage is semantic object grounding. Grounding DINO or a similar open-vocabulary detector can identify the object associated with a language query, while SAM can refine the detected region into a detailed mask. Open-vocabulary 3D perception can then connect the object to physical geometry. This sequence is important because affordance estimation generally needs to reason about specific parts of an object, and those parts must first be associated with the correct physical entity.

Affordance reasoning then focuses attention on functional regions rather than treating the entire object as equally actionable. For a tool, the handle may be suitable for grasping while the working end should not be used as the primary grasp region. For a container, an opening may afford insertion while another surface may provide a stable placement region. The system therefore needs to combine semantic understanding with local geometry, object structure, and the capabilities of the robot end effector.

Zero-shot affordance estimation can use visual-language similarity to associate image regions with action concepts. Text descriptions such as "graspable part," "pushable surface," "placeable surface," or "insertable opening" can be embedded into a shared representation space and compared with local visual features. Regions with stronger correspondence become candidates for further geometric and physical evaluation. This allows functional concepts to be specified dynamically instead of encoding every affordance as a fixed output class.

Three-dimensional information is essential because an affordance is ultimately related to physical interaction. A visually suitable region may be inaccessible, incorrectly oriented, too small for the gripper, or located behind another object. Depth, point clouds, surface normals, curvature, free space, and robot pose can therefore be combined with semantic affordance predictions. The resulting representation can estimate not only where an interaction may occur, but also whether the robot can physically approach and execute that interaction.

The relationship between affordance estimation and grasp planning should remain modular. Affordance estimation identifies regions that may support a particular action, while grasp generation determines candidate end-effector poses and contact configurations. Motion planning then evaluates whether those poses can be reached without collision. This separation allows the perception system to remain flexible when encountering novel objects while preserving established geometric and kinematic constraints in the manipulation pipeline.

Language can provide additional constraints that make affordance estimation more task-specific. "Grasp the handle," "place the object on the flat surface," and "press the red button" specify both an interaction type and a semantic target. The system can combine object identity, attributes, spatial relationships, and action concepts to rank candidate regions. This creates a task-conditioned affordance representation in which the same physical region may be useful for one instruction but irrelevant for another.

Multiple candidate affordances may exist simultaneously, so confidence and ranking are important. A robot may identify several possible grasp points on an unfamiliar object, multiple surfaces suitable for placement, or more than one button that visually resembles the requested target. Candidate regions should therefore be evaluated using semantic similarity, geometric feasibility, accessibility, collision constraints, and consistency across viewpoints. When no candidate satisfies the required evidence, the system should retain an uncertain or unavailable state rather than forcing an unsafe interaction.

Temporal perception can improve affordance estimation as the robot changes its viewpoint. A region that initially appears unsuitable may become clearly identifiable after the robot moves, while partial occlusion may hide an important handle or opening. Tracking the object and accumulating observations allows the system to update affordance hypotheses over time. This is particularly useful for mobile manipulators that must approach objects before obtaining sufficiently detailed visual and geometric evidence.

Zero-shot affordance estimation also supports manipulation of objects for which complete CAD models or dedicated training data are unavailable. Instead of requiring an exact object identity before planning an action, the robot can reason about functional properties visible in the current scene. A novel object may therefore be manipulated according to its graspable surfaces, openings, supports, or interaction points. This capability complements open-vocabulary object recognition by adding a functional layer between semantic identification and physical action.

Uncertainty is especially important because visual similarity to an affordance concept does not prove that an interaction will succeed. A region that appears graspable may be fragile, slippery, obstructed, or mechanically unsuitable. Similarly, a surface that appears suitable for placement may not support the object's weight. Affordance predictions should therefore be treated as hypotheses that require geometric, kinematic, and task-level validation before execution.

For practical robotic systems, the affordance representation should expose structured information to downstream modules. Useful outputs can include the semantic object identity, affordance type, candidate interaction region, three-dimensional location, surface orientation, confidence, and estimated feasibility. Grasp planning, motion planning, and control can then consume these outputs without requiring the affordance model itself to solve the complete manipulation problem.

Zero-shot affordance estimation therefore represents a functional extension of open-vocabulary perception. Language specifies the desired interaction, vision-language models provide semantic correspondence, open-vocabulary perception identifies the relevant object, segmentation isolates candidate regions, and 3D geometry provides physical context. Affordance reasoning then proposes where an action may be possible, while grasp planning, kinematic validation, collision checking, and closed-loop control determine whether and how the robot should execute it. This architecture connects flexible semantic perception with generalizable physical interaction while maintaining the geometric and safety requirements of robotic manipulation.

11.09. OV Perception Edge Deployment Optimization [w/Code]

Open-vocabulary perception introduces substantially higher computational demands than conventional closed-set perception because vision-language models, open-vocabulary detectors, segmentation models, and 3D semantic representations may operate together. For a physical robot, these models must execute within strict limits on latency, GPU memory, power consumption, and thermal capacity. Edge deployment optimization therefore focuses on preserving the semantic flexibility of open-vocabulary perception while adapting the complete pipeline to the computational resources available on the robot.

A practical edge architecture should separate perception functions according to their computational importance and required update frequency. Safety-critical detection, obstacle perception, and motion-related sensing may require continuous high-rate processing, whereas open-vocabulary semantic queries do not necessarily need to run at the same frequency. A lightweight perception layer can operate continuously while more expensive vision-language inference is activated when a semantic target is requested, when uncertainty increases, or when a new object must be identified.

Model selection is therefore an important part of deployment optimization. Large vision-language encoders may provide strong semantic representations but can exceed the memory and latency limits of embedded hardware. Smaller encoders, compact open-vocabulary detectors, reduced image resolutions, and region-based inference can lower computational cost. The appropriate model should be selected according to the required semantic capability, target object scale, camera configuration, available GPU memory, and maximum acceptable perception latency.

Image resolution has a direct influence on both accuracy and computational cost. High resolution is valuable for small objects, fine boundaries, and detailed affordance estimation, but processing the entire image at high resolution can become expensive. A practical system can use lower-resolution images for global semantic search and selectively process high-resolution regions after candidate objects have been identified. This coarse-to-fine strategy concentrates computation where additional visual detail provides the greatest benefit.

Feature reuse is another important optimization mechanism. Vision-language systems often perform expensive image encoding before comparing the resulting representation with multiple text queries. If the camera image or scene has not changed significantly, previously computed visual features can be cached and reused. Multiple language queries can then be evaluated against the same image representation without repeatedly executing the complete visual backbone, reducing redundant computation during interactive robot operation.

Open-vocabulary perception can also be scheduled asynchronously with the main navigation and control loops. A robot may maintain high-frequency localization, obstacle avoidance, and motion control while a lower-frequency semantic process searches for language-specified objects. Once a semantic target is detected, its position can be transferred to the navigation or manipulation stack. Tracking and geometric propagation can then maintain the target state between expensive open-vocabulary updates.

Quantization and optimized inference runtimes can further reduce the computational burden. Neural networks may be converted to lower-precision representations when the resulting accuracy loss is acceptable. Deployment frameworks can optimize graph execution, memory movement, kernel selection, and hardware acceleration. TensorRT-style optimization, mixed precision, and carefully selected numerical formats can be particularly useful when the robot must execute multiple perception models on a shared embedded GPU.

Memory management is as important as raw inference speed. Open-vocabulary 3D perception may require image features, depth information, point clouds, semantic embeddings, and temporal observations to coexist in memory. Persistent storage of every dense feature can become impractical on an edge platform. A system can therefore maintain compact representations, update semantic features selectively, discard obsolete observations, and preserve high-resolution information only for objects or regions relevant to current tasks.

Multi-camera robots require additional resource allocation because multiple streams can multiply preprocessing and inference costs. Running a large open-vocabulary model independently on every camera may be unnecessary. The system can select the most informative camera, perform hierarchical camera activation, or fuse lower-cost observations before invoking expensive semantic inference. Camera selection can also consider target visibility, viewpoint quality, occlusion, and the spatial relationship between the camera and the requested object.

For real-time robotic operation, latency should be evaluated as an end-to-end property rather than only as neural-network inference time. Image acquisition, preprocessing, model execution, post-processing, coordinate transformation, communication, tracking, and delivery of the result to navigation or manipulation all contribute to the effective response time. A model with fast isolated inference may therefore provide limited system-level benefit if data transfer or downstream processing introduces substantial delay.

Accuracy must also be evaluated under the operating conditions of the robot rather than only on a general benchmark. Open-vocabulary models can behave differently under low illumination, motion blur, outdoor environments, reflective surfaces, unusual viewpoints, small targets, and domain-specific industrial equipment. Edge optimization should therefore measure the trade-off among semantic accuracy, localization accuracy, latency, memory usage, power consumption, and robustness. Excessive compression or aggressive frame skipping may reduce the reliability of the complete perception-to-action chain.

A robust architecture should also support adaptive computation. When the scene is simple and the semantic target is clearly identified, the system can reduce expensive processing. When multiple candidates are visually similar, the target is partially occluded, or confidence falls, additional viewpoints, higher resolution, segmentation, or 3D verification can be activated. Computation is consequently allocated according to uncertainty and task importance rather than being applied uniformly to every frame and every object.

The final edge deployment architecture should preserve a clear separation between semantic perception and safety-critical control. Open-vocabulary models can identify flexible targets, generate semantic regions, and provide information for navigation or manipulation, but their outputs should pass through geometric validation and safety constraints before physical action. This is particularly important because semantic similarity can produce plausible but incorrect detections. Unknown-state handling, confidence thresholds, temporal consistency, depth verification, and collision constraints should remain active even when computational resources are limited.

Within the open-vocabulary perception pipeline, edge optimization therefore does not mean simply making every model smaller. It means allocating computation according to semantic importance, temporal requirements, uncertainty, hardware capability, and physical task constraints. Vision-language features can be cached, expensive models can be invoked selectively, lightweight tracking can maintain intermediate states, and geometric verification can refine semantic predictions. This approach allows open-vocabulary perception to operate as a practical component of an edge robot while maintaining the flexibility required for language-guided navigation, novel-object manipulation, semantic mapping, and zero-shot affordance estimation.

11.10. Open Vocabulary Perception Production Case

A production open-vocabulary perception system must transform the capabilities developed in earlier stages into a reliable and maintainable robotic subsystem. The chapter structure places this production case after language-guided navigation, novel-object manipulation, zero-shot affordance estimation, and edge deployment optimization. This sequence implies that production deployment should integrate semantic recognition, spatial grounding, task interaction, computational efficiency, and operational safety rather than treating open-vocabulary perception as an isolated neural-network model.

The production architecture can begin with synchronized RGB, depth, LiDAR, and robot-state inputs, followed by a staged perception pipeline. Vision-language features provide semantic representations, open-vocabulary detection identifies candidate objects, segmentation refines their visible regions, and three-dimensional processing associates those regions with physical space. The system should preserve clear interfaces between these stages so that individual models can be replaced, optimized, or upgraded without requiring a complete redesign of the robot perception stack.

A practical system should distinguish continuous perception from query-driven perception. Conventional safety and navigation sensors can operate continuously at high frequency, while computationally expensive open-vocabulary models can be activated when a semantic task requires them. A language query such as "find the charging station" or "locate the damaged container" can trigger semantic processing, after which lightweight tracking and geometric propagation maintain the target state. This architecture prevents expensive foundation-model inference from becoming an unnecessary constant computational burden.

Production operation also requires a structured representation of perception results. A detected object should not be passed downstream merely as a class name or bounding box. Its representation can include semantic identity, query correspondence, confidence, image region, segmentation mask, three-dimensional position, estimated extent, temporal state, and associated uncertainty. Such a representation allows navigation, manipulation, mapping, and task reasoning modules to consume the same perception output while applying their own geometric and operational constraints.

The system should also support persistent semantic memory. When an object has been detected and localized, its representation can be associated with the robot's semantic map or spatial memory. Later queries can retrieve previously observed objects without requiring the entire recognition process to start again. Because real environments change, however, stored observations should carry temporal information and confidence so that objects can be revalidated, updated, or removed when subsequent observations contradict the previous state.

Production robustness depends strongly on uncertainty management. An open-vocabulary model may produce semantically plausible detections even when the requested object is absent, partially occluded, visually ambiguous, or outside the model's reliable operating conditions. The production system should therefore support explicit unknown, uncertain, and not-found states. Semantic similarity should be combined with depth consistency, geometric plausibility, temporal stability, multi-view agreement, and task context before a detection is promoted to an actionable target.

The same principle applies to language-guided navigation and manipulation. A semantic detection should not directly command vehicle motion or an actuator. Navigation should validate the target against localization, occupancy, traversability, and collision constraints, while manipulation should validate object geometry, grasp feasibility, robot kinematics, and surrounding obstacles. This separation creates a controlled perception-to-action interface in which open-vocabulary models provide flexible semantic information without bypassing established safety mechanisms.

Production systems must also accommodate environmental variation. Indoor facilities, warehouses, hospitals, outdoor sites, and industrial workspaces can contain different lighting conditions, object appearances, camera viewpoints, background structures, and sensor configurations. The system should therefore avoid assuming that benchmark performance automatically represents field performance. Deployment evaluation should use representative operational data and measure semantic accuracy together with localization quality, latency, robustness, false detections, missed targets, and downstream task success.

Model lifecycle management becomes important once open-vocabulary perception is deployed across multiple robots. Models, prompts, thresholds, calibration parameters, and runtime configurations should be versioned and traceable. A change to a vision-language encoder or detection threshold can alter downstream behavior even if the robot software itself has not changed. Production engineering therefore benefits from explicit configuration management, reproducible deployment packages, controlled updates, and rollback procedures.

Edge optimization should be integrated into this lifecycle rather than treated as a final compression step. Model quantization, feature caching, region-based inference, asynchronous processing, and selective model activation can reduce computational requirements. At the same time, optimization should be validated against semantic accuracy and task performance. A smaller or faster model is not necessarily an improvement if it increases false detections or causes the robot to lose critical targets during navigation or manipulation.

Multi-camera and multi-sensor operation also requires deliberate resource management. A production robot may have several RGB cameras, depth sensors, LiDAR units, and auxiliary sensors, but running every expensive model on every stream may be unnecessary. Camera selection, hierarchical activation, sensor fusion, and query-dependent processing can concentrate computation on the most informative observations. Sensor timestamps, calibration, and coordinate transformations must remain consistent so that semantic information is correctly associated with physical space.

Monitoring and diagnostics are essential after deployment. The production system should record sufficient information to determine why a semantic query succeeded, failed, or became uncertain without requiring unrestricted storage of every raw sensor stream. Useful diagnostic information includes model versions, query text, confidence values, processing latency, selected camera, detection history, geometric validation results, and downstream task outcomes. These records provide evidence for performance analysis and targeted improvement.

A production perception system should also define operational fallback behavior. If an open-vocabulary model becomes unavailable, exceeds its latency budget, or produces insufficient confidence, the robot should continue using appropriate conventional perception functions where possible. Navigation may fall back to geometric obstacle avoidance, while manipulation may pause until a reliable target is reacquired. The objective is not to guarantee that open-vocabulary perception always succeeds, but to ensure that its failure does not automatically become an unsafe physical action.

The final production architecture therefore treats open-vocabulary perception as a coordinated subsystem rather than a single model. Language and vision provide semantic discovery, detection and segmentation establish visual grounding, 3D perception connects semantics to physical space, and navigation or manipulation converts validated information into task-specific behavior. Edge optimization, uncertainty handling, persistent memory, configuration management, monitoring, and safety validation complete the operational framework. This provides a practical path from experimental open-vocabulary capabilities to a maintainable robotic perception service suitable for repeated field operation.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Original GitHub manuscript: https://github.com/Statera-Library/08_Vol_14_Perception_and_Sensor_Fusion/blob/main/Chapter_11_Open_Vocabulary_Perception/Chapter_11_Open_Vocabulary_Perception_EN.md

Top comments (0)