Viewpoint Generation Using Feature-Based Constrained Spaces for Robot Vision Systems

Abstract

The efficient computation of viewpoints while considering various system and process constraints is a common challenge that any robot vision system is confronted with when trying to execute a vision task. Although fundamental research has provided solid and sound solutions for tackling this problem, a holistic framework that poses its formal description, considers the heterogeneity of robot vision systems, and offers an integrated solution remains unaddressed. Hence, this publication outlines the generation of viewpoints as a geometrical problem and introduces a generalized theoretical framework based on Feature-Based Constrained Spaces ($\mathcal{C}$-spaces) as the backbone for solving it. A $\mathcal{C}$-space can be understood as the topological space that a viewpoint constraint spans, where the sensor can be positioned for acquiring a feature while fulfilling the constraint. The present study demonstrates that many viewpoint constraints can be efficiently formulated as $\mathcal{C}$-spaces, providing geometric, deterministic, and closed solutions. The introduced $\mathcal{C}$-spaces are characterized based on generic domain and viewpoint constraints models to ease the transferability of the present framework to different applications and robot vision systems. The effectiveness and efficiency of the concepts introduced are verified on a simulation-based scenario and validated on a real robot vision system comprising two different sensors.

1. Introduction

The increasing performance of 2D and 3D image processing algorithms and the falling prices of electronic components (processors and optical sensors) over the last two decades have motivated not only researchers but also the industry to investigate and automate different machine vision tasks using robot vision systems (RVSs) consisting of a manipulator and a 2D or 3D sensor [1,2]. Whether programmed offline or online, RVSs demand multiple planning modules to execute motion and vision tasks efficiently and robustly. For instance, the efficient and effective planning of valid viewpoints to fulfill a vision task considering different constraints, known as the view(point) planning problem (VPP), still represents an open planning problem within diverse applications [2,3], e.g., camera surveillance, scene exploration, object detection, visual servoing, object reconstruction, image-based inspection, robot calibration, and mobile navigation [2,4].

1.1. Viewpoint Generation Problem Solved Using $\mathcal{C}$-Spaces

To tackle the VPP, we first re-examine its reformulation and propose its modularization. Then, this study focuses on the most fundamental sub-problem of the VPP, i.e., the Viewpoint Generation Problem (VGP). The VGP addresses the calculation of valid viewpoints to acquire a single feature while considering the fulfillment of different viewpoint constraints.

With this in mind, this paper outlines the VGP as a purely geometrical problem that can be solved in the special Euclidean group denoted as $SE\left(3\right)$ (6D spatial space), based on the concept of Feature-Based Constrained Spaces ($\mathcal{C}\text{-}\mathrm{space}$s). $\mathcal{C}\text{-}\mathrm{space}$s represent the spatial solution space of up to 6D of each viewpoint constraint $c_i\in \tilde{C}$, denoted as ${\mathcal{C}}_i^{}$, that comprises all valid sensor poses ${\mathit{p}}_s^{}$ to acquire a feature f. In other words, it can be assumed that any sensor pose lying within an i $\mathcal{C}\text{-}\mathrm{space}$ fulfills the corresponding i viewpoint constraint. Hence, this solution space can be interpreted as an analytical [5], geometrical solution with an infinite set of valid viewpoints to satisfy the regarded viewpoint constraint. Moreover, the integration of multiple $\mathcal{C}\text{-}\mathrm{space}$s spans the jointed $\mathcal{C}\text{-}\mathrm{space}$ denoted as ${\mathcal{C}}_{}^{}$, where all viewpoint constraints are simultaneously fulfilled. Figure 1 depicts a simplified representation of the VGP, $\mathcal{C}\text{-}\mathrm{space}$s, and overview of the most relevant components.

In this context, the most significant challenge behind the conceptualization of $\mathcal{C}\text{-}\mathrm{space}$s lies in the generic and geometric formulation and characterization of diverse viewpoint constraints. Throughout this paper, we will use the term formulationto refer to the formal, mathematical definition of an individual $\mathcal{C}\text{-}\mathrm{space}$; however, note that the characterization addresses the concrete implementation or computation of a $\mathcal{C}\text{-}\mathrm{space}$ using a specific algorithm or method. This publication introduces nine $\mathcal{C}\text{-}\mathrm{space}$s corresponding to different viewpoint constraints (i.e., sensor imaging parameters, feature geometry, kinematic errors, sensor accuracy, occlusion, multisensors, multi-features, and robot workspace) aligned to a consistent modeling framework to ensure their consistent integration.

1.2. Related Work

Our study treats the VGP as a sub-problem of the VPP. Since most authors do not explicitly consider such a problem separation, this section provides an overview of related research that addresses the VPP in general. In a broader sense, the VPP can be even categorized as a sub-problem of a more popular challenge within robotics, that is, the coverage path planning problem [6].

Over the last three decades, the VPP has been investigated within a wide range of vision tasks that integrate an imaging device, but not necessarily a robot, and require the computation of generalized viewpoints. For a vast overview of the overall progress, challenges, and applications of the VPP, we refer to the various surveys [2,4,7,8,9,10] that have been published.

The approaches for viewpoint planning can be classified depending on the knowledge required a priori about the RVS to compute a valid viewpoint. Thus, a rough distinction can be made between model-based and non-model-based approaches [11].

1.2.1. Model-Based

Most of the model-based viewpoint planning methods can be roughly differentiated between synthesis and sampling-based (related terms: generate and test) modeling approaches [5]. While synthesis approaches use analytical relationships to first characterize a continuous or discrete solution space before searching for an optimal viewpoint, sampling techniques are more optimization-oriented and compute valid viewpoints using a set of objective functions.

Since the present study seeks to characterize a solution space, i.e., a $\mathcal{C}\text{-}\mathrm{space}$, for each individual viewpoint constraint using analytical and geometrical relationships, the mathematical foundation of our framework can be classified as a model-based method following a synthesis approach. Hence, this section focuses mainly on the related literature following a similar approach.

Synthesis

Many of the reviewed publications considering model-based approaches have built their theoretical framework based on set theory to formulate either a continuous or discrete search space in a first step. Then, in a second step, optimization algorithms are used to find valid viewpoints within these search spaces and assess the satisfiability of the remaining constraints that were not explicitly considered.

The concept of characterizing such topological search spaces, in our work addressed as $\mathcal{C}\text{-}\mathrm{space}$s (related terms: viewpoint space, visibility map, visibility matrix, visibility volumes, imaging space, scannability frustum, configuration space, visual hull, search space), has been proposed since the first studies addressing the VPP. Such a formulation has the advantage of providing a straightforward comprehension and spatial interpretation of the general problem.

One of the first and seminal studies that considered the characterization of a continuous solution space in ${\mathbb{R}}^3$ can be attributed to the publication of [12]. In their work, the authors introduced a model-based method for 2D sensors, which synthesized analytical relationships to geometrically characterize a handful of constraints in terms of resolution, focus, field of view, visibility, view angle, occluding regions, and in later works [13], even constraints on the placement of a lighting source.

Based on the analytical findings provided by the previous work of [13], Ref. [7] introduced a model-based sensor planning system called Machine Vision Planner (MVP). On one hand, the MVP can be seen as a synthesis approach that characterizes a feature-based occlusion-free region using surface model decomposition [14,15]. On the other hand, the authors posed the problem in the context of an optimization setting using objective functions to find valid viewpoints within the occlusion-free space that meet imaging constraints.

The MVP was extended by [16] for its use with an industrial robot and moving objects. Their study addressed the drawbacks (non-linearity and convergence guarantee) of the optimization algorithms and opted to characterize 3D search spaces for the sensor’s resolution, field of view, and workspace of the robot. Although the authors could not synthesize every constraint in the Euclidean space, they confirmed the benefits of solving the problem in ${\mathbb{R}}^3$ instead of optimizing equations for finding suitable viewpoints. Furthermore, in a series of publications Reed et al. [17,18] extended some of the models introduced in the MVP and addressed the characterization of a search space in ${\mathbb{R}}^3$ for range sensors, which integrates imaging, occlusion, and workspace constraints. Their study also proposed the synthesis of an imaging space based on extrusion techniques of the surface models in combination with the imaging parameters of the sensor.

Another line of research within the context of model-based approaches follow the works of Tarbox and Gottschlich, which proposed the synthesis of a discretized search space using visibility matrices to map the visibility between the solution space and the surface space of the object. In combination with an efficient volumetric representation of the object of interest using octrees, Refs. [19,20] presented different algorithms based on the concept of visibility matrices to perform automated inspection tasks. The visibility matrices consider a discretized view space with all viewpoints lying on a tessellated sphere with a fixed camera distance. Analogously, under the consideration of further constraints, Refs. [11,21] introduced the measurability matrix extending the visibility matrix of Tarbox and Gottschlich to three dimensions. Within his work, Scott considered further sensor parameters, e.g., the shadow effect, measurement precision, and the incident angle, which many others have neglected. More recent works [3,22,23,24] confirmed the benefits of such an approach and used the concept of visibility matrices for encoding information between a surface point and a set of valid viewpoints.

In the context of space discretization and feature-driven approaches, further publications [20,25,26] suggested the characterization of the positioning space for the sensor using tessellated spheres to reduce the 6D sensor positioning problem to a 2D orientation optimization problem. Similarly, Refs. [27,28] introduced the concept of visibility maps to encode feature visibility mapped to a visibility sphere. Years later, Refs. [29,30] considered variations of this approach for their viewpoint planning systems. The major shortcomings of techniques considering a problem reduction is that most of them require a fixed working distance, which limits their applicability for other sensors and reduces their efficiency for the computation of multi-feature acquisition.

In the context of laser scanners, other relevant works, such as Refs. [31,32,33], also considered solutions to first synthesize a search space before searching for feasible solutions. Additionally, the publication of [34] needs to be mentioned, since, besides [21], this is one of the few studies that considered viewpoint visibility for multisensor systems using a lookup table.

Sampling-Based

Many other works [35,36,37,38,39] do not rely on the explicit characterization of a search space and assess the satisfiability of each viewpoint constraint individually by sampling the search space using metaheuristic optimization algorithms, e.g., simulated annealing or evolutionary algorithms. Such approaches concentrate on the adequate and efficient formulation of objective functions to satisfy the viewpoint constraints and find reasonable solutions.

1.2.2. Non-Model Based

In contrast, non-model based approaches require no a priori knowledge; the object can be utterly unknown to the planning system. In this case, online exploratory techniques based on the captured data are used to compute the next best viewpoint [26,40,41,42,43]. Most of these works focus on reconstruction tasks and address the problem as the next-best-view planning problem. Since our work is considered a feature-driven approach requiring a priori knowledge of the system, this line of research will be not further discussed.

1.2.3. Comparison and Need for Action

Although over the last three decades, many works presented well-grounded solutions to tackle the VPP for individual applications, a generic approach for solving the VPP has not been established yet for commercial or industrial applications nor research. Hence, we are convinced that a well-founded framework, comprising a consistent formulation of viewpoints constraints combined with a model-based synthesis approach, while also considering a continuous solution space, has the greatest potential to search for viewpoints that efficiently satisfy different viewpoint constraints.

Synthesis vs. Sampling: Within the related works, we have found recent publications following an explicit synthesis and sampling techniques of solution spaces for the same applications. Hence, a clear trend towards any of these model-based approaches could not be identified. On the one hand, sampling methods can be especially advantageous and computationally efficient within simple scenarios while considering few constraints. On the other hand, model uncertainties and nonlinear constraints are more difficult to model using objective functions and, within multi-feature scenarios, the computation efficiency can be severely affected. Therefore, we believe that this problem can be solved more efficiently based on $\mathcal{C}\text{-}\mathrm{space}$s composed of explicit models of all regarded viewpoint constraints within applications comprising robot systems and partially known environments with modeling uncertainties.

Continuous vs. Discrete Space: Most of the latest research has followed a synthesis approach based on visibility matrices or visibility maps to encode the surface space and viewpoint space together for a handful of applications and systems [3,22,23,24,30]. Although these works have demonstrated the use of discrete spaces to be practical and efficient, from our point of view, its major weakness lies in the intrinsic limited storage capacity and processing times associated with matrices. This limitation directly affects the synthesis of the solution space considering just a fixed distance between sensor and object. Moreover, in the context of RVS, limiting the robot’s working space seems to be in conflict with the inherent and most appreciated positioning flexibility of robots. Due to these drawbacks and taking into account that the fields of view and working distances of sensors have increased and will continue to do so, we believe that the discretization of the solution space could become inefficient for certain applications at some point.

Problem Formulation: Many of the revised works considering synthesized model-based approaches posed the VGP formulation on the fundamentals of set theory. However, our research suggests that a consistent mathematical framework, which promotes a generic formulation and integration of viewpoint constraints, has not been appropriately placed. Hence, we consider that an exhaustive domain modeling and consistent theoretical mathematical framework are key elements to provide a solid base for a holistic and generic formulation of the VGP.

1.3. Outline

After providing an overview of the related work that has addressed the VPP and VGP in Section 1.2, first, Section 2 presents the domain models of generic RVSs used through the present study. Then, Section 3 poses the formulation VGP from another perspective and introduces the concept of $\mathcal{C}\text{-}\mathrm{space}$s to tackle this problem. Section 4 exploits the introduced formulation, and different viewpoint constraints are formulated, characterized, and verified. Then, Section 5 assesses the validity of the proposed formulations and characterization of $\mathcal{C}\text{-}\mathrm{space}$s and demonstrate its potential and generalization using a real RVS. Finally, Section 6 presents a summary and conclusions of this paper. In addition, a comprehensive set of Supplemental Material (e.g., simulation video, $\mathcal{C}\text{-}\mathrm{space}$s meshes, renders) is digitally attached to this publication. An overview of the paper outline is presented in Figure 2.

1.4. Contributions

Our publication presents the fundamental concepts of a generic framework comprising innovative and efficient formulations to compute valid viewpoints based on $\mathcal{C}\text{-}\mathrm{space}$s to solve the fundamental sub-problem of the VPP, i.e., the VGP. The key contributions of this paper are summarized as follows:

• Mathematical, model-based, and modular framework to formulate the VGP based on $\mathcal{C}\text{-}\mathrm{space}$s and generic domain models.
• Formulation of nine viewpoint constraints using linear algebra, trigonometry, geometric analysis, and Constructive Solid Geometry (CSG) Boolean operations, in particular:

  -

  Efficient and simple characterization of $\mathcal{C}\text{-}\mathrm{space}$ based on sensor frustum, feature position, and feature geometry.

  -

  Generic characterization of $\mathcal{C}\text{-}\mathrm{space}$s to consider bi-static nature of range sensors extendable to multisensor systems.

• Exhaustive supporting material (surface models, manifolds of computed $\mathcal{C}\text{-}\mathrm{space}$s, rendering results) to encourage benchmark and further development (see Supplementary Materials).

Additionally, we consider the following principal advantages associated with the formulation of $\mathcal{C}\text{-}\mathrm{space}$s:

• Determinism, efficiency, and simplicity: $\mathcal{C}\text{-}\mathrm{space}$s can be efficiently characterized using geometrical analysis, linear algebra, and CSG Boolean techniques.
• Generalization, transferability, and modularity: $\mathcal{C}\text{-}\mathrm{space}$s can be seamlessly used and adapted for different vision tasks and RVSs, including different sensor imaging sensors (e.g., stereo, active light sensors) or even multiple range sensor systems.
• Robustness against model uncertainties: Known model uncertainties (e.g., kinematic model, sensor, or robot inaccuracies) can be explicitly modeled and integrated while characterizing $\mathcal{C}\text{-}\mathrm{space}$s. If unknown model uncertainties affect a chosen viewpoint, alternative solutions guaranteeing constraint satisfiability can be found seamlessly within $\mathcal{C}\text{-}\mathrm{space}$s.

In combination with a suitable strategy, $\mathcal{C}\text{-}\mathrm{space}$s can be straightforwardly integrated into a holistic approach for entirely solving the VPP. The use of $\mathcal{C}\text{-}\mathrm{space}$s within an adequate strategy represents the second sub-problem of the VPP, which falls outside the scope of this paper and will be handled within a future publication.

2. Domain Models of a Robot Vision System

This section outlines the generic domain models and minimal necessary parameters of an RVS, including assumptions and limitations, required to characterize the individual $\mathcal{C}\text{-}\mathrm{space}$s in Section 4.

We consider an RVS, a complex mechatronical system that comprises the following domains: a range sensor (s) that is positioned by a robot (r) to capture a feature (f) of an object of interest (o) enclosed within an environment (e). Figure 3 provides an overview of the RVS domains and some parameters described within this section.

2.1. General Notes

• General Requirements This paper follows a systematic and exhaustive formulation of the VGP, the domains of an RVS, and the viewpoint constraints to characterize $\mathcal{C}\text{-}\mathrm{space}$s in a generic, simple, and scalable way. To achieve this, and similar to previous studies [11,33,36], throughout our framework the following general requirements (GR) are considered: generalization, computational efficiency, determinism, modularity and scalability, and limited a priori knowledge. The given order does not consider any prioritization of the requirements. A more detailed description of the requirements can be found in

  Table A1. Description of general requirements.

  General Requirement	Description
  1. Generalization	The models and approaches used should be abstracted and generalized at the best possible level so that they can be used for different components of an RVS and can be applied to solve a broad range of vision tasks.
  2. Computational Efficiency	The methods and techniques used should strive towards a low level of computational complexity. Whenever possible, analytical, and linear models should be preferred over complex techniques, such as stochastic and heuristic algorithms. Nevertheless, when considering offline scenarios, the trade-off between computing a good enough solution within an acceptable amount of time should be individually assessed.
  3. Determinism	Due to traceability and safety issues within industrial applications deterministic approaches should be prioritized.
  4. Modularity and Scalability	The approaches and models should consider in general a modular structure and promote their scalability.
  5. Limited a priori Knowledge	The parameters required to implement the models and approaches should be easily accessible for the end-users. Neither in-depth optics nor robotic knowledge should be required.

  .

• Terminology Based on our literature research, we have found that a common terminology has not been established yet. The employed terms and concepts depend on the related applications and hardware. To better understand the relation of our terminology to the related work and in an attempt towards standardization, whenever possible, synonyms or related concepts are provided. Please note that in some cases, the generality of some terms is prioritized over their precision. This may lead to some terms not corresponding entirely to our definition; therefore, we urge the reader to study these differences before treating them as exact synonyms.

• Notation Our publication considers many variables to describe the RVS domains comprehensively. To ease the identification and readability of variables, parameters, vectors, frames, and transformations, we use the index notation given in

  Table A2. Overview of index notations for variables.

  Notation	Index Description
  ${{}_b{}^{r}x}_d^n$	x = variable, parameter, vector, frame, or transformation d = RVS domain, i.e., (s)ensor, (r)obot, (f)eature, (o)bject, (e)nvironment, or d element of a list or set n = related domain, additional notation, or depending variable r = base frame of the coordinate system $B_r$ or space of feature f b = origin frame of the coordinate system $B_b$
  Notes	The indices r and b just apply for pose vectors, frames, and transformations.
  Example	The index notation can be better understood consider following examples: d: Let the geometry of a feature be described by a surface point ${\mathit{g}}_f^{}\in {\mathbb{R}}^3$. d: If the feature comprises more surface points, then let the point with the index 2 be denoted by ${\mathit{g}}_{f,2}^{}\in {\mathbb{R}}^3$. r: Assuming that the position of a surface point ${\mathit{g}}_f^{}$ is described in the coordinate system of the feature, $B_f$, then it follows: ${{}^f\mathit{g}}_f^{}$. In case that the base coordinate frame has the same notation as the domain itself, i.e., $r=d$, then just the index for the domain is given: ${\mathit{g}}_f^{}={{}^f\mathit{g}}_f^{}$. b: In case that the frame of the surface point is given in the coordinate reference system of the object $B_o$, then following notation applies: ${{}^o\mathit{g}}_f^{}={{}_f{}^o\mathit{g}}_f^{}$.

  . Moreover, all topological spaces are given in calligraphic fonts, e.g., $\mathcal{V},\mathcal{P},\mathcal{I},\mathcal{C}$, while vectors, matrices, and rigid transformations are bold.

  Table A3. List of the most common symbols.

  Symbol	Description
  	General
  c	Viewpoint constraint
  $\tilde{C}$	Set of viewpoint constraints
  f	Feature
  F	Set of features
  $s_t$	t imaging device of sensor s
  v	Viewpoint
  ${\mathit{p}}_s^{}$	Sensor pose in $SE\left(3\right)$
  	Spatial Dimensions
  B	Frame
  ${\mathit{t}}_{}^{}$	Translation vector in ${\mathbb{R}}^3$
  ${\mathit{r}}_{}^{}$	Orientation matrix in ${\mathbb{R}}^{3x3}$
  $\mathit{V}$	Manifold vertex in ${\mathbb{R}}^3$
  	Topological Spaces
  ${\mathcal{C}}_{}^{}$	$\mathcal{C}\text{-}\mathrm{space}$ for a set of viewpoint constraints $\tilde{C}$
  ${\mathcal{C}}_i^{}$	i $\mathcal{C}\text{-}\mathrm{space}$ of the viewpoint constraint $c_i$
  ${\mathcal{I}}_s^{}$	Frustum space

  provides an overview of the most frequently used symbols.

2.2. General Models

• Kinematic model Each domain comprises a Kinematics subsection to describe its kinematic relationships. In particular, all necessary rigid transformations (given in the right-handed system) are introduced to calculate the sensor pose. The pose ${\mathit{p}}_{}^{}$ of any element is given by its translation ${\mathit{t}}_{}^{}\in {\mathbb{R}}^3$ and a rotation component that can be given as a rotation matrix ${\mathit{R}}_{}^{}\in {\mathbb{R}}^{3x3}$, Z-Y-X Euler angles ${\mathit{r}}_{}^{}={({\alpha }^z,{\beta }^{{}^y},{\gamma }^x)}^T$ or a quaternion ${\mathit{q}}_{}^{}\in \mathbb{H}$. For readability and simplicity purposes, we use mostly the Euler angle representation throughout this paper. Considering the special orthogonal $SO\left(3\right)\subset {\mathbb{R}}^{3x3}$, the pose $p\in SE\left(3\right)$ is given in the special Euclidean group $SE\left(3\right)={\mathbb{R}}^3\times SO\left(3\right)$ [44]). ${{}^f\mathit{p}}_s^{}\in SE\left(3\right)$ in the feature’s coordinate system $B_f$:

  $$\begin{array}{cc}\hfill {{}^f\mathit{p}}_s^{}=& {{}_s{}^f\mathit{T}}_{}^{}\hfill \\ \hfill =& {{}_o{}^f\mathit{T}}_f^{}·{{}_w{}^o\mathit{T}}_e^{}·{{}_r{}^w\mathit{T}}_e^{}·{{}_s{}^r\mathit{T}}_r^{}.\hfill \end{array}$$

  (1)

It is assumed that all introduced transformations are roughly known and given in the world (w) coordinate system or any other reference system. Moreover, we also consider a summed alignment error ${ϵ}_e$ in the kinematic chain to quantify the sensor’s positioning inaccuracy relative to a feature.

• Surface model A set of 3D surface models $\kappa \in K$ characterizes the volumetric occupancy of all rigid bodies in the environment. The surface models are not always explicitly mentioned within the domains. Nevertheless, we assume that the surface model of any rigid body is required if this collides with the robot or sensor or impedes the sensor’s sight to a feature.

2.3. Object

This domain considers an object (o) (related terms: object of interest, workpiece, artifact, measurement object, inspection object, or test object) that contains the features to be acquired.

• Kinematics The origin coordinate system of o is located at frame $B_o$. The transformation to the reference coordinate system is given in the world coordinate system $B_w$ by ${{}_o{}^w\mathit{T}}_e^{}$.

• Surface Model Since our approach does not focus on the object but rather on its features, the object may have an arbitrary topology.

2.4. Feature

A feature (f) (related terms: region, point or area of interest, inspection feature, key point, entity, artifact) can be fully specified considering its kinematic and geometric parameters, i.e., frame $B_f$ and set of surface points $G_f\left(L_f\right)$, which depend on a set of geometric dimensions $L_f$:

$$f:=(B_f,G_f\left(L_f\right)).$$

(2)

• Kinematics We assume that the translation ${{}^o\mathit{t}}_f^{}$ and orientation ${{}^o\mathit{r}}_f^{}$ of the feature’s origin is given in the object’s coordinate system $B_o$. In the case that the feature’s orientation is given by its minimal expression, i.e., just the feature’s surface normal vector ${{}^o\mathit{n}}_f^{}$. The full orientation is calculated by letting the feature’s normal to be the basis z-vector ${{}^o\mathit{e}}_f^z={{}^o\mathit{n}}_f^{}$ and considering the rest basis vectors ${{}^o\mathit{e}}_f^x$ and ${{}^o\mathit{e}}_f^y$ to be mutually orthonormal. The feature’s frame is given as follows:

  $$B_f={{}^o\mathit{T}}_f^{}({{}^o\mathit{t}}_f^{},{{}^o\mathit{r}}_f^{}).$$

  (3)

• Geometry While a feature can be sufficiently described by its position and normal vector, a broader formulation is required within many applications. For example, dimensional metrology tasks deal with a more comprehensive catalog of geometries, e.g., edges, pockets, holes, slots, and spheres.
  Thus, the present study explicitly considers the geometrical topology of a feature and a more extensive model of it [15,28]. Let the feature topology be described by a set of geometric parameters, denoted by $L_f$, such as the radius of a hole or a sphere or the lengths of a square.

• Generalization and Simplification Moreover, we consider a discretized geometry model of a feature comprising a finite set of surface points corresponding to a feature ${{\mathit{g}}_{\mathit{f}}}_{}^{}\in G_f$ with ${{\mathit{g}}_{\mathit{f}}}_{}^{}\in {\mathbb{R}}^3$. Since our work primarily focuses on 2D features, it is assumed that all surface points lie on the same plane, which is orthogonal to the feature’s normal vector ${{}^o\mathit{n}}_f^{}$ and co-linear to the z-axis of the feature’s frame $B_f$.
  Towards providing a more generic feature model, the topology of all features is approximated using a square feature with a unique side length of $\left\{l_f\right\}\in L_F$ and five surface points ${{\mathit{g}}_{\mathit{f}}}_{,c}^{},c=\{0,1,2,3,4\}$ at the center and at the four corners of the square. Figure 4 visualizes this simplification to generalize diverse feature geometries.

2.5. Sensor

We consider a sensor (s) (related terms: range camera sensor, 3D sensor, imaging system) a self-contained acquisition device comprising at least two imaging devices $\{s_1,s_2\}\in \tilde{S}$ (e.g., two cameras or a camera and a lighting source) capable of computing a range image containing depth information. Such sensors can be classified by the principles used to acquire this type of depth information, e.g., triangulation, intensity, or time of flight [45]. The present work does not explicitly distinguish between these acquisition principles. Moreover, this subsection outlines a generic and minimal sensor model that is in line with our framework. Note that even though the present report focuses primarily on range sensors, the models can also be considered for single imaging devices.

• Kinematics The sensor’s kinematic model considers the following relevant frames: $B_s^{TCP}$, $B_s^{s_1}$, and $B_s^{s_2}$. Taking into account the established notation for end effectors within the robotics field, we consider that the frame $B_s^{TCP}$ lies at the sensor’s tool center point (TCP). We assume that the frame of the TCP is located at the geometric center of the frustum space and that the rigid transformation ${{}_{TCP}{}^{ref}\mathit{T}}_s^{}$ to a reference frame such as the sensor’s mounting point is known.
  Additionally, we consider that frame $B_s^{s_1}$ lies at the reference frame of the first imaging device that corresponds to the imaging parameters $I_s$. We assume that the rigid transformation ${{}_{s_1}{}^{ref}\mathit{T}}_s^{}$ between the sensor lens and a known reference frame is also known. ${{}_{s_2}{}^{ref}\mathit{T}}_s^{}$ provides the transformation of the second imaging device at the frame $B_s^{s_2}$. The second imaging device $s_2$ might be a second camera considering a stereo sensor or the light source origin in an active sensor system.

• Frustum space The frustum space $\mathcal{I}\text{-}\mathrm{space}$ (related terms: visibility frustum, measurement volume, field-of-view space, sensor workspace) is described by a set of different sensor imaging parameters $I_s$, such as the depth of field $d_s$ and the horizontal and vertical field of view (FOV) angles ${\theta }_s^x$ and ${\psi }_s^y$. Alternatively, some sensor manufacturers may also provide the dimensions and locations of the near $h_s^{near}$, middle $h_s^{middle}$, and far $h_s^{far}$ viewing planes of the sensor. The sensor parameters $I_s$ allow only the topology of the $\mathcal{I}\text{-}\mathrm{space}$ to be described. To fully characterize the topological space in the special Euclidean, the sensor pose ${\mathit{p}}_s^{}$ must be considered:

  $$\begin{array}{cc}\hfill {\mathcal{I}}_s^{}:={\mathcal{I}}_s^{}({\mathit{p}}_s^{},I_s)=& \{{\mathit{p}}_s^{}\in SE\left(3\right),\hfill \\ \hfill & (d_s,h_s^{near},h_s^{far},{\theta }_s^x,{\psi }_s^y)\in I_s\}\hfill \end{array}.$$

  (4)
  The $\mathcal{I}\text{-}\mathrm{space}$ can be straightforwardly calculated based on the kinematic relationships of the sensor and the imaging parameters. The resulting 3D manifold ${\mathcal{I}}_s^{}$ is described by its vertices ${\mathit{V}}_k^{{\mathcal{I}}_s^{}}:={\mathit{V}}_k^{}\left({\mathcal{I}}_s^{}\right)={(V_k^x,V_k^y,V_k^z)}^T$ with $k=1,\dots ,l$ and corresponding edges and faces. We assume that the origin of the frustum space is located at the TCP frame, i.e., $B_s^{{\mathcal{I}}_s}=B_s^{TCP}$. The resulting shape of the $\mathcal{I}\text{-}\mathrm{space}$ usually has the form of a square frustum. Figure 5 visualizes the frustum shape and the geometrical relationships of the $\mathcal{I}\text{-}\mathrm{space}$.

• Range Image A range image (related terms: 3D measurement, 3D image, depth image, depth maps, point cloud) refers to the generated output of the sensor after triggering a measurement action. A range image is described as a collection of 3D points denoted by ${\mathit{g}}_s^{}\in {\mathbb{R}}^3$, where each point corresponds to a surface point of the measured object.

• Measurement accuracy The measurement accuracy depends on various sensor parameters and external factors and may vary within the frustum space [21]. If these influences are quantifiable an accuracy model can be considered within the computation of the $\mathcal{C}\text{-}\mathrm{space}$. For example, ref. [34] proposed a method based on a Look-Up Table to specify quality disparities within a frustum.

• Sensor Orientation When choosing the sensor pose for measuring an object’s surface point or a feature, additional constraints must be fulfilled regarding its orientation. One fundamental requirement that must be satisfied to guarantee the acquisition of a surface point is the consideration of the incidence angle ${{}^f\phi }_s^{}$ (related terms: inclination, acceptance, view, or tilt angle). This angle is expressed as the angle between the feature’s normal ${\mathit{n}}_f^{}$ and the sensor’s optical axis (z-axis) ${\mathit{e}}_s^z$ and can be calculated as follows:

  $${{}^f\phi }_s^{max}>\left|{{}^f\phi }_s^{}\right|,{{}^f\phi }_s^{}=arccos\left({\displaystyle \frac{{\mathit{n}}_f^{}·{\mathit{e}}_s^z}{\left|{\mathit{n}}_f^{}\right|·\left|{\mathit{e}}_s^z\right|}}\right).$$

  (5)
  The maximal incidence angle ${{}^f\phi }_s^{max}$ is normally provided by the sensor’s manufacturer. If the maximal angle is not given in the sensor specifications, some works have suggested empirical values for different systems. For example, ref. [46] propose a maximum angle of $60{}^{\circ }$ [47] suggests $45{}^{\circ }$, while [48] propose a tilt angle of $30{}^{\circ }$ to $50{}^{\circ }$. The incidence angle can also be expressed on the basis of the Euler angles (pan, tilt) around the x- and y-axes: ${{}^f\phi }_s^{}({{}^f\beta }_s^y,{{}^f\gamma }_s^x)$.
  Furthermore, the rotation of the sensor around the optical axis is given by the Euler angle ${\alpha }_s^z$ (related terms: swing, twist). Normally, this angle does not directly influence the acquisition quality of the range image and can be chosen arbitrarily. Nevertheless, depending on the lighting conditions or the position of the light source while considering active systems, this angle might be more relevant and influence the acquisition parameters of the sensor, e.g., the exposure time. Additionally, if the shape of the frustum is asymmetrical, the optimization of ${\alpha }_s^z$ should be considered.

2.6. Robot

The robot (related terms: manipulator, industrial robot, positioning device) has the main task of positioning the sensor to acquire a range image.

• Kinematics The robot base coordinate frame is placed at $B_r^{}$. We assume that the rigid transformations between the robot basis and the robot flange, ${{}_{f_r}{}^r\mathit{T}}_r^{}$, and between the robot flange and the sensor, ${{}_s{}^{f_r}\mathit{T}}_r^{}$, are known. We also assume that the Denavit-Hartenberg (DH) parameters are known and that the rigid transformation ${{}_{f_r}{}^r\mathit{T}}_r^{}\left(DH\right)$ can be calculated using an inverse kinematic model. The sensor pose in the robot’s coordinate system is given by

  $${{}^r\mathit{p}}_s^{}={{}_s{}^r\mathit{T}}_r^{}={{}_{f_r}{}^r\mathit{T}}_r^{}·{{}_s{}^{f_r}\mathit{T}}_r^{}.$$

  (6)
  The robot workspace is considered to be a subset in the special Euclidean, thus ${\mathcal{W}}_r\subseteq SE\left(3\right)$. This topological space comprises all reachable robot poses to position the sensor ${{}^r\mathit{p}}_s^{}\in {\mathcal{W}}_r$.

• Robot Absolute Position Accuracy It is assumed that the robot has a maximal absolute pose accuracy error of ${ϵ}_r$ in its workspace and that the robot repeatability is much smaller than the absolute accuracy; hence, it is not further considered.

2.7. Environment

The environment domain comprises models of remaining components that were not explicitly included by other domains. Particularly, we consider all other rigid bodies that may collide with the robot or affect the visibility of the sensor, e.g., fixtures, external axes, robot cell components. Thus, if the environment domain comprises rigid bodies, these must be included in the set of surface models ${\kappa }_e^{}\in K$.

2.8. General Assumptions and Limitations

The previous subsections have introduced the formal models and parameters to characterize an RVS. Hereby, we present some general assumptions and limitations considered within our work.

• Sensor compatibility with feature geometry: Our approach assumes that a feature and its entire geometry can be captured with a single range image.
• Range Image Quality: The sensor can acquire a range image of sufficient quality. Effects that may compromise the range image quality and have not been previously regarded are neglected include measurement repeatability, lighting conditions, reflection effects, and random sensor noise.
• Sensor Acquisition Parameters: Our work does not consider the optimization of acquisition sensor parameters such as exposure time, gain, and image resolution, among others.
• Robot Model: Since we assumed that a range image can just be statically acquired, a robot dynamics model is not contemplated. Hence, constraints regarding velocity, acceleration, jerk, or torque limits are not considered within the scope of our work.

3. Problem Formulation

This section first introduces the concept of generalized viewpoints and briefly describes the viewpoint constraints considered within the scope of our work. Then, the modularization of the VPP and formulation of the VGP as a geometric problem are introduced to understand the placement of the present study. In Section 3.5, $\mathcal{C}\text{-}\mathrm{space}$s are introduced within the context of Configuration Spaces as a practical and simple approach for solving the VGP. Moreover, considering that various viewpoint constraints must be satisfied to calculate a valid viewpoint, we outline the reformulation of the VGP based on $\mathcal{C}\text{-}\mathrm{space}$s within the framework of Constraint Satisfaction Problems.

3.1. Viewpoint and $\mathcal{V}$-Space

Although the concept of generalized viewpoints has been introduced by some of the related works (the concept of a generalized viewpoint was first introduced by [7] and was later used by [11,36]), there seems to be no clear definition of a viewpoint v. Hence, in this study, while considering a feature-centered formulation, we define a viewpoint as being a triple of the following elements: a sensor pose ${\mathit{p}}_s^{}\in SE\left(3\right)$ to acquire a feature $f\in F$ considering a set of viewpoint constraints $\tilde{C}$ from any domain of the RVS:

$$v:=(f,{\mathit{p}}_s^{},\tilde{C}).$$

Additionally, we consider that any viewpoint that satisfies all constraints is an element of the viewpoint space ($\mathcal{V}\text{-}\mathrm{space}$):

$$v\in {\mathcal{V}}_{}^{}.$$

(7)

Hence, the $\mathcal{V}\text{-}\mathrm{space}$ can be formally defined as a tuple comprising a feature space denoted by a feature set F and the $\mathcal{C}\text{-}\mathrm{space}$ ${{}^F\mathcal{C}}_{}^{}\left(\tilde{C}\right)$, which satisfies all spatial viewpoint constraints to position the sensor:

$${\mathcal{V}}_{}^{}:=(F,{{}^F\mathcal{C}}_{}^{}\left(\tilde{C}\right)).$$

Note that within this publication, we only consider spatially viewpoint constraints affecting the placement of the sensor. As given by the limitations of our work in Section 2.8, additional sensor setting parameters are not explicitly addressed. Nevertheless, for purposes of correctness and completeness, let these constraints be denoted by ${\tilde{C}}_s$, then Equation (7) can be extended as follows:

$$(f,{\mathit{p}}_s^{},\tilde{C},{\tilde{C}}_s)\in {\mathcal{V}}_{}^{}.$$

3.2. Viewpoint Constraints

To provide a comprehensive model of a generalized viewpoint and assess its validity, it is necessary to formulate a series of viewpoint constraints. Hence, we propose an abstract formulation of the viewpoint constraints needed to acquire a feature successfully. The set of viewpoint constraints $c_i\in \tilde{C},i=1,\dots ,j$ comprises all constraints $c_i$ affecting the positioning of the sensor; hence, the validity of a viewpoint candidate. Every constraint $c_i$ can be regarded as a collection of domain variables of the RVS under consideration, e.g., the imaging parameters $I_s$, the feature geometry length $l_f$, the maximal incidence angle ${{}^f\phi }_s^{max}$.

This subsection provides a general description of the constraints; a more comprehensive formulation and characterization are handled individually within Section 4. An overview of the viewpoint constraints considered in our work is given in

Table A4. Overview and description of the viewpoint constraints considered in our work.

Viewpoint Constraint	Description
1. Frustum Space	The most restrictive and fundamental constraint is given by the imaging capabilities of the sensor. This constraint is fulfilled if at least the feature’s origin lies within the frustum space (cf. Section 2.5).
2. Sensor Orientation	Due to specific sensor limitations, it is necessary to ensure that the maximal permitted incidence angle between the optical axis and the feature normal lies within an specified range; see Equation (5).
3. Feature Geometry	This constraint can be considered an extension of the first viewpoint constraint and is fulfilled if all surface points of a feature can be acquired by a single viewpoint, hence lying within the image space.
4. Kinematic Error	Within the context of real applications, model uncertainties affecting the nominal sensor pose compromise a viewpoint’s validity. Hence, any factor, e.g., kinematic alignment, robot’s pose accuracy, which affects the overall kinematic chain of the RVS must be considered (see Section 2.6).
5. Sensor Accuracy	Acknowledging that the sensor accuracy may vary within the sensor image space (see Section 2.5), we consider that a valid viewpoint must ensure that a feature must be acquired within a sufficient quality.
6. Feature Occlusion	A viewpoint can be considered valid if a free line of sight exists from the sensor to the feature. More specifically, it must be assured that no rigid bodies are blocking the view between the sensor and the feature.
7. Bistatic Sensor and Multisensor	Recalling the bistatic nature of range sensors, we consider that all viewpoint constraints must be valid for all lenses or active sources. Furthermore, we also extend this constraint for considering a multisensor RVS comprising more than one range sensor.
8. Robot Workspace	The workspace of the whole RVS is limited primarily by the robot’s workspace. Thus, we assume that a valid viewpoint exists if the sensor pose lies within the robot workspace.
9. Multi-Feature	Considering a multi-feature scenario, where more than one feature can be acquired from the same sensor pose, we assume that all viewpoint constraints for each feature must be satisfied within the same viewpoint.

.

Although some related studies [11,12,20,36] also considered similar constraints, the main differences in our formulations are found in their explicit characterization and integration with other constraints. While some of these works assess a viewpoint’s validity in a reduced 2D space or sampled space, our work focuses on characterizing each constraint explicitly in a higher dimensional and continuous space.

3.3. Modularization of the Viewpoint Planning Problem

The necessity to break down the VPP into two sub-problems can be better understood by considering the following minimal problem formulation based on [20]:

Problem 1.

How many viewpoints are necessary to acquire a given set of features?

We believe that considering a multi-stage solution to tackle the VPP can reduce its complexity and contribute to a more efficient solution. Thus, in the first step, we consider the modularization of the VPP and address its two fundamental problems separately: the VGP and the Set Covering Problem (SCP).

First, we attribute to the VGP the computation of valid viewpoints to acquire a single feature considering a set of various viewpoint constraints. Moreover, in the context of multi-feature scenarios and presuming that all features cannot be acquired using a single viewpoint, the efficient selection of more viewpoints becomes necessary to complete the vision task, with which arises a new problem, that is, the SCP.

This paper concentrates on the comprehensively formulation of the VGP; thus, this problem is discussed more extensively in the following sections. Although our work focuses on a feature-based approach, the concept of features can be also be extended to surface points or areas of interest.

3.4. The Viewpoint Generation Problem

The VGP (related terms: optical camera placement, camera planning) and concept of viewpoints can be better understood considering a proper formulation:

Problem 2.

Which is a valid viewpoint v to acquire a feature f considering a set of viewpoint constraints $\tilde{C}$?

A viewpoint v exists, if there is at least one sensor pose ${\mathit{p}}_s^{}$ that can capture a feature f and only if all j viewpoint constraints $\tilde{C}$ are fulfilled. The most straightforward way to find a valid viewpoint for Problem 2 is to assume an ideal sensor pose ${\mathit{p}}_s^0$ and assess its satisfiability against each constraint using a binary function $h_i:({\mathit{p}}_s^0,c_i)\to true$. If the sensor pose fulfills all j constraints, the viewpoint is valid. Otherwise, another sensor pose must be chosen and the process must be repeated until a valid viewpoint is found. The mathematical formulation of such conditions is expressed as follows:

$$\begin{array}{cc}\hfill v=& \{(f,{\mathit{p}}_s^0,\tilde{C})\in {\mathcal{V}}_{}^{}\hfill \\ \hfill & \mid f\in F,{\mathit{p}}_s^0\in SE\left(3\right),\hfill \\ \hfill & \bigcap _{C_i\in \tilde{C}}{h}_i:({\mathit{p}}_s^0,c_i)\to true\}.\hfill \end{array}$$

(8)

The formulation of a generalized viewpoint as given by Equation (8) can be considered one of the most straightforward formulations to solve the VGP, if for each viewpoint constraint, a Boolean condition can be expressed. For instance, by introducing such cost functions for different viewpoint constraints, several works [9,21,36,37,49,50,51] demonstrated that optimization algorithms (e.g., greedy, genetic, or even reinforcement learning algorithms) can be used to find local and global optimal solutions within polynomial times.

3.5. VGP as a Geometrical Problem in the Context of Configuration Spaces

Although the generalized viewpoint model as given by Equation (8) yields a sufficient and generic formulation to solve the VGP, this formulation is inefficient considering real applications with model uncertainties. System modeling inevitably involves discrepancies between virtual and real-world models, particularly within dynamically changing environments. Due to such model inconsistencies, considering optimal viewpoints to acquire a feature could be regarded as ineffective and inefficient in some applications. Hence, in our opinion, it is more reasonable to treat the VGP as a multi-dimensional problem and to consider multiple valid solutions throughout its formulation.

A sound solution for the VGP will require characterizing a continuous topological space comprising multiple solutions that allow deviation from an optimal solution and efficient choice of an alternative viewpoint. This challenge embodies the core motivation of our work to formulate and characterize $\mathcal{C}\text{-}\mathrm{space}$s.

If the VGP can be handled as a spatial problem that can be solved geometrically, we refer to the use of configuration spaces, as introduced by [52,53] and exhaustively studied by [54] in the well-studied motion planning field for solving geometrical path planning problems.

“Once the configuration space is clearly understood, many motion planning problems that appear different in terms of geometry and kinematics can be solved by the same planning algorithms. This level of abstraction is therefore very important.”

[54]

In our work, we use the general concepts of configuration spaces based on the formulation of topological spaces to characterize the manifold spanned by a viewpoint constraint—the $\mathcal{C}\text{-}\mathrm{space}$. A $\mathcal{C}\text{-}\mathrm{space}$ should not be confused with the configuration space (C-Space) used within the motion planning field to characterize the robot’s joint configuration space.

3.6. VGP with Ideal $\mathcal{C}$-Spaces

Following the notation and concepts behind the modeling of configuration spaces, we first consider a modified formulation of Problem 2 and assume an ideal system (i.e., sensor with an infinite field of view, without occlusions and neglecting any other constraint) for introducing some general concepts.

Problem 3.

Which is the ideal $\mathcal{C}\text{-}space$ ${\mathcal{C}}_{}^{*}$ to acquire a feature f?

Sticking to the notation established within the motion planning research field, let us first consider an ideal, unconstrained space denoted as ${\mathcal{C}}_{}^{*}\subseteq SE\left(3\right)$

$$\begin{array}{cc}\hfill {\mathcal{C}}_{}^{*}=& {\mathbb{R}}^3\times SO\left(3\right)\hfill \\ \hfill =& \{{\mathit{p}}_s^{}\in {\mathcal{C}}_{}^{*},{\mathit{p}}_s^{}\in SE\left(3\right)\}\hfill \\ \hfill =& \{{\mathit{p}}_s^{}({\mathit{t}}_s^{},{\mathit{r}}_s^{})\in {\mathcal{C}}_{}^{*}\hfill \\ \hfill & \mid {\mathit{t}}_s^{}\in {\mathbb{R}}^3,{\mathit{r}}_s^{}\in SO\left(3\right)\},\hfill \end{array}$$

(9)

which is spanned by the Euclidean Space ${\mathbb{R}}^3$ and the special orthogonal group $SO\left(3\right)$, and holds all valid sensor poses ${\mathit{p}}_s^{}$ to acquire a feature f. An abstract representation of the unconstrained $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_{}^{*}$ is visualized in Figure 6.

3.7. VGP with $\mathcal{C}$-Spaces

The ideal $\mathcal{C}\text{-}\mathrm{space}$ as given by Equation (9) considers a sufficient generic model that spans an ideal solution space to solve the VGP. Assuming a non-ideal RVS where a viewpoint must satisfy a handful of requirements, an extended formulation of the $\mathcal{C}\text{-}\mathrm{space}$ admitting viewpoint constraints is introduced within this subsection.

3.7.1. Motivation

The VGP recalls the formulation of decision problems, a class of computational problems, which has been widely researched within different applications. Inspired by other research fields dealing with artificial intelligence and optimization of multi-domain applications, we observed that decision problems, including multiple constraints, can be well formulated under the framework of Constraint Satisfaction Problems (CSP) [55]. This category of problems does not consider an explicit technique to formulate the constraints under consideration. Moreover, a consistent, declarative, and simple representation of the domain’s constraints can be decisive for their efficient resolution [56].

3.7.2. Formulation

Formulating the VGP as a CSP requires a proper formulation to consider viewpoint constraints in the first step; hence, let Problem Formulation 2 be extended by the following:

Problem 4.

Which is the $\mathcal{C}\text{-}\mathit{space}$ ${\mathcal{C}}_{}^{}$ spanned by a set of viewpoint constraints $\tilde{C}$ to acquire a feature f?

The $\mathcal{C}\text{-}\mathrm{space}$ denoted as

$${\mathcal{C}}_{}^{}:={\mathcal{C}}_{}^{}(f,\tilde{C})$$

(10)

can be understood as the topological space that all viewpoint constraints from the set $\tilde{C}$ span in the special Euclidean group so that the sensor can capture a feature f fulfilling all of these constraints. The $\mathcal{C}\text{-}\mathrm{space}$ that a single viewpoint constraint $c_i\in \tilde{C}$ spans is analogously given:

$${\mathcal{C}}_i^{}:={\mathcal{C}}_i^{}(f,c_i).$$

To guarantee a consistent formulation and integration of various viewpoint constraints, within our framework we consider the following characteristics for the formulation of $\mathcal{C}\text{-}\mathrm{space}$s:

• If an i constraint, $c_i$, can be spatially modeled, there exists a topological space denoted as ${\mathcal{C}}_i^{}$, which can be ideally formulated as a proper subset of the special Euclidean:

  $${\mathcal{C}}_i^{}\subseteq SE\left(3\right).$$
  In a broader definition, we consider that the topological space for each constraint is spanned by a subset of the Euclidean Space denoted as ${\mathcal{T}}_s^{}\subseteq {\mathbb{R}}^3$ and a special orthogonal group subset given by ${\mathcal{R}}_s^{}\subseteq SO\left(3\right)$. Hence, the topological space of a viewpoint constraint is given as follows:

  $$\begin{array}{cc}\hfill {\mathcal{C}}_i^{}=& {\mathcal{T}}_s^{}\times {\mathcal{R}}_s^{}\hfill \\ \hfill =& \{{\mathit{p}}_s^{}\in {\mathcal{C}}_i^{},{\mathit{p}}_s^{}\in SE\left(3\right)\}\hfill \\ \hfill =& \{{\mathit{p}}_s^{}({\mathit{t}}_s^{},{\mathit{r}}_s^{})\in {\mathcal{C}}_i^{}\hfill \\ \hfill & \mid {\mathit{t}}_s^{}\in {\mathcal{T}}_s^{},{\mathit{r}}_s^{}\in {\mathcal{R}}_s^{}\}.\hfill \end{array}$$

  (11)
• If there exists at least one sensor pose in the i $\mathcal{C}\text{-}\mathrm{space}$ $\exists {\mathit{p}}_s^{}\in {\mathcal{C}}_i^{}$, then this sensor pose fulfills the viewpoint constraint $c_i$ to acquire feature f; hence, a valid viewpoint exists $(f,{\mathit{p}}_s^{},c_i)\in {\mathcal{V}}_{}^{}$.
• If there exists a topological space, ${\mathcal{C}}_i^{}$, for each constraint $\forall c_i\in \tilde{C}$ then the intersection of all individual constrained spaces constitutes the joint $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_{}^{}\subseteq SE\left(3\right)$:

  $$\mathcal{C}\left(\tilde{C}\right)=\bigcap _{c_i\in \tilde{C}}{\mathcal{C}}_i\left(c_i\right).$$

  (12)
• If the joint constrained space is a non-empty set, i.e., $\mathcal{C}\ne \varnothing$, then there exists at least one sensor pose $\exists {\mathit{p}}_s\in \mathcal{C}$, and consequently a viewpoint $(f,{\mathit{p}}_s,\tilde{C})\in \mathcal{V}$ that fulfills all viewpoint constraints.

An abstract representation of the $\mathcal{C}\text{-}\mathrm{space}$s and the resulting topological space ${\mathcal{C}}_{}^{}$ intersected by various viewpoint constraints is depicted in Figure 6. It is worth mentioning that although the framework considers an independent formulation of each viewpoint constraint, the real challenge consists of characterizing each constraint individually to maintain a high generalization and flexibility of the framework (cf. Table A1).

4. Methods: Formulation, Characterization, and Verification of $\mathcal{C}$-Spaces

This section outlines the formulation and characterization of the $\mathcal{C}\text{-}\mathrm{space}$s of all viewpoint constraints considered in the scope of this paper (see Table A4). First, Section 4.1 introduces the core constraint, which is built up based on the imaging parameters of the sensor, which are characterized by the $\mathcal{I}\text{-}\mathrm{space}$. Then, Section 4.2 shows how $\mathcal{C}\text{-}\mathrm{space}$s can be combined to span a topological space in the special Euclidean group using different sensor orientations. We systematically analyze how the sensor frustum space can be used to describe a $\mathcal{C}\text{-}\mathrm{space}$ in $SE\left(3\right)$ and introduce simple and generic formulations for its computation.

While Section 4.1 and Section 4.2 introduced the core constraints for characterizing $\mathcal{C}\text{-}\mathrm{space}$s, the following Section 4.3, Section 4.4, Section 4.5, Section 4.6, Section 4.7 and Section 4.8 deal with the characterization of the remaining viewpoint constraints. Moreover, Section 4.9 presents one possible strategy to integrate all $\mathcal{C}\text{-}\mathrm{space}$s, demonstrating the advantages of a consistent and modular characterization.

The formulations presented in this section are motivated by the general requirements (cf. Table A1) that aim to deliver a high generalization of the models to facilitate their use with different RVSs and vision tasks. Hence, whenever possible, the characterization of some constraints using simple scalar arithmetic is prioritized over more complex techniques, and simplifications are introduced for the benefit of pragmatism, efficiency, and generalization of the approaches considered.

4.1. Frustum Space, Feature Position, and Fixed Sensor Orientation

This section shows how the feature position and the frustum space $\mathcal{I}\text{-}\mathrm{space}$ can be directly employed to characterize a $\mathcal{C}\text{-}\mathrm{space}$, which fulfills the first viewpoint constraint for a fixed sensor orientation. For the benefit of the comprehension of the concepts introduced within this subsection, we consider a feature as just a single surface point with a normal vector.

4.1.1. Formulation

Base Constraint Formulation In the first step, we introduce a simple condition for the first constraint, $c_1:=c_1({{\mathit{g}}_{\mathit{f}}}_{,0}^{},{\mathit{p}}_s^{},{\mathcal{I}}_s^{})$, which considers the feature (minimally represented by a surface point), and the frustum space, which is characterized by all imaging parameters and the sensor pose. Let $c_1$ be fulfilled for all sensor poses $\forall {\mathit{p}}_s^{}\in SE\left(3\right)$, if and only if the feature surface point lies within the corresponding frustum space at the regarded sensor pose:

$$c_1\iff {{\mathit{g}}_{\mathit{f}}}_{,0}^{}\in {\mathcal{I}}_s^{}\left({\mathit{p}}_s^{}\right).$$

(13)

Problem Simplification with fixed Sensor Orientation Due to the limitations of some sensors regarding their orientation, it is a common practice within many applications to define and optimize the sensor orientation beforehand. Then, the VGP can be reduced to an optimization of the sensor position ${\mathit{t}}_s^{}$. Hence, let condition (13) be reformulated to consider a fixed sensor orientation ${\mathit{r}}_s^{fix}\in SO\left(3\right)$ and to be true for all sensor positions $\forall {\mathit{t}}_s^{}\in {\mathbb{R}}^3$ that fulfill following condition:

$$c_1\iff {{\mathit{g}}_{\mathit{f}}}_{,0}^{}\in {\mathcal{I}}_s^{}({\mathit{p}}_s^{}({\mathit{t}}_s^{},{\mathit{r}}_s^{}={\mathit{r}}_s^{fix})).$$

(14)

Constraint Reformulation based on Constrained Spaces Recalling the idea of geometrically characterizing any viewpoint constraint (see Section 3.7), we find that the viewpoint constraint formulation of Equation (14) to be unsatisfactory. We believe that this problem can be solved efficiently using geometric analysis and assume there exists a topological space denoted by ${\mathcal{C}}_1^{}:={\mathcal{C}}_1^{}\left(c_1\right)$, which can be characterized based on the $\mathcal{I}\text{-}\mathrm{space}$ considering a fixed sensor’s orientation. If such space exists, then all sensor positions within it fulfill the viewpoint constraint given by Equation (14).

Combining the formulation for $\mathcal{C}\text{-}\mathrm{space}$s given by Equation (11) and the viewpoint constraint condition from Equation (14), the formal definition of the topological space ${\mathcal{C}}_1^{}$ is given:

$$\begin{array}{cc}\hfill {\mathcal{C}}_1^{}& =\{\forall {\mathit{p}}_s^{}({\mathit{t}}_s^{},{\mathit{r}}_s^{}={\mathit{r}}_s^{fix})\in {\mathcal{C}}_1^{}\hfill \\ \hfill & |\forall {\mathit{t}}_s^{}\in {\mathcal{T}}_s^{},{\mathit{r}}_s^{fix}\in {\mathcal{R}}_s^{},{{\mathit{g}}_{\mathit{f}}}_{,0}^{}\in {\mathcal{I}}_s^{}\}.\hfill \end{array}$$

(15)

4.1.2. Characterization

Within the framework of our research, we found out that the manifold of ${\mathcal{C}}_1^{}$ can be characterized in different ways. This subsection presents two possible solutions to characterize the $\mathcal{C}\text{-}\mathrm{space}$ as given by Equation (15) using analytic geometry. The manifolds of the computed $\mathcal{C}\text{-}\mathrm{space}$s and additional supporting material from this and the following subsections are found in the digital appendix of this publication.

• Extreme Viewpoints Interpretation The simplest way to understand and visualize the topological space ${\mathcal{C}}_1^{}$ is to consider all possible extreme viewpoints to acquire a feature f. These viewpoints can be easily found by positioning the sensor so that each vertex (corner) of the $\mathcal{I}\text{-}\mathrm{space}$, ${\mathit{V}}_k^{{\mathcal{I}}_s^{}}$, lies at the feature’s origin $B_f$, which corresponds to the position of the surface point ${{\mathit{g}}_{\mathit{f}}}_{,0}^{}$. The position of such an extreme viewpoint corresponds to the k vertex ${\mathit{V}}_k^{{\mathcal{C}}_1^{}}\in V_{}^{{\mathcal{C}}_1^{}}$ of the manifold ${\mathcal{C}}_1^{}$. Depending on the positioning frame of the sensor $B_s^{TCP}$ or $B_s^{s_1}$, the space can be computed for the TCP(${{}_{TCP}\mathcal{C}}_1^{}$) or the sensor lens (${{}_{s_1}\mathcal{C}}_1^{}$). The vertices can be straightforwardly computed following the steps given in Algorithm A1. The left Figure 7 illustrates the geometric relations for computing ${\mathcal{C}}_1^{}$ and a simplified representation of the resulting manifolds in ${\mathbb{R}}^2$ for the sensor TCP $B_s^{TCP}$ and lens $B_s^{s_1}$.
• Homeomorphism Formulation Note that the manifold ${{}_{ref}\mathcal{C}}_1^{}$ illustrated in Figure 7a has the same topology as the $\mathcal{I}\text{-}\mathrm{space}$. Thus, it can be assumed there exists a homeomorphism between both spaces such that $h:{\mathcal{I}}_s^{}\to {\mathcal{C}}_1^{}$. Letting the function h correspond to a point reflection over the geometric center of the frustum space, the vertices of the manifold $V_{}^{{{}_{ref}\mathcal{C}}_1^{}}$ can be straightforwardly estimated following the steps described in the Algorithm 1. The resulting manifold for the TCP frame is shown in Figure 7b.

Algorithm 1 Homeomorphism Characterization of the Constrained Space ${\mathcal{C}}_1^{}$

(a) Consider a constant sensor orientation ${{}_{ref}\mathit{r}}_s^{fix}$ to acquire a feature f. (b) Position the sensor reference frame at the feature’s surface point origin ${{}_{ref}\mathit{p}}_s^f({{}_{ref}\mathit{t}}_s^{}=B_f)$. (c) For each k vertex of the frustum space, compute its reflection transformation h across the reference pivot frame $B_s^{ref}$ $${{}_{ref}\mathit{V}}_k^{{{}_{ref}\mathcal{C}}_1^{}}=h({\mathit{V}}_k^{{\mathcal{I}}_s^{}}({{}_{ref}\mathit{t}}_s^{}=B_f),B_s^{ref}).$$ (d) Connect all vertices from $V_{}^{{{}_{ref}\mathcal{C}}_1^{}}$ analogously to the vertices of the frustum space $V_{}^{{\mathcal{I}}_s^{}}$ to obtain the ${{}_{ref}\mathcal{C}}_1^{}$ manifold.

General Notes We considered the steps described in Algorithm 1 to be the most traceable strategy using a homeomorphism to compute the constrained space ${{}_{ref}\mathcal{C}}_1^{}$. Nevertheless, we do not refuse any alternative approach for its characterization. For instance, the same manifold of ${{}_{ref}\mathcal{C}}_1^{}$ for any reference frame can also be obtained by first computing the reflection model of the frustum space ${\mathcal{I}}_s^{*}$ over its geometric center at $B_s^{{\mathcal{I}}_s^{}}$:

$$h:(B_s^{{\mathcal{I}}_s^{}},{\mathcal{I}}_s^{})\to {\mathcal{I}}_s^{*}.$$

(16)

The manifold of ${\mathcal{I}}_s^{*}$ can then be simply translated to the desired reference frame so that $B_s^{{\mathcal{I}}_s^{*}}=B_s^{ref}$, considering that the TCP must be positioned at the origin of the feature ${{}_{TCP}\mathit{p}}_s^f({{}_{TCP}\mathit{t}}_s^{}=B_f,{{}_{ref}\mathit{r}}_s^{fix})$.

Moreover, our approach considers that the topological space spanned by ${\mathcal{C}}_1^{}$ exists if the following conditions hold:

• the frames of all vertices of the frustum space ${\mathit{V}}_{s,i}^{}\left({\mathcal{I}}_s^{}\right),i=1..j$ are known,
• the frustum space is a watertight manifold,
• and the space between connected vertices of the frustum space is linear; hence, adjacent vertices are connected only by straight edges.

Throughout this paper, we characterize most of the $\mathcal{C}\text{-}\mathrm{space}$s considering just the reference frame for the sensor lens $s_1$; hence, if not stated otherwise, consider ${\mathcal{C}}_1^{}:={{}_{s_1}\mathcal{C}}_1^{}$.

4.1.3. Verification

Any of the two formulations presented in this subsection can be straightforwardly extended to characterize the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}$ in $SE\left(3\right)$. However, we found the homeomorphism formulation to be the most practical way to compute the ${\mathcal{C}}_1^{}$ manifold. Hence, to verify the characterization of ${\mathcal{C}}_1^{}$ using this approach, we first defined a sensor orientation in $SE\left(3\right)$ denoted as ${{}^{f_0}\mathit{r}}_s^{fix}$ to acquire the feature $f_0$. We then computed the $\mathcal{I}\text{-}\mathrm{space}$ manifold using a total of $j=8$ vertices with the imaging parameters of $s_1$ from

Table A6. Imaging parameters of the sensors $s_1$ and $s_2$.

Range Sensor	1		2
Manufacturer	Carl Zeiss Optotechnik GmbH, Neubeuern, Germany		Roboception, Munich, Germany
Model	COMET Pro AE		rc_visard 65
3D Acquisition Method	Digital Fringe Projection		Stereo Vision
Imaging Device $s_t$	Monochrome Camera: $s_1$	Blue Light LED-Fringe Projector: $s_2$	Two monochrome cameras: $s_3,$ $s_4$
Field of view	${\theta }_s^x=51.5{}^{\circ }$, ${\psi }_s^y=35.5{}^{\circ }$	${\theta }_s^x=70.8{}^{\circ }$, ${\psi }_s^y=43.6{}^{\circ }$	${\theta }_s^x=62.0{}^{\circ }$, ${\psi }_s^y=48.0{}^{\circ }$
Working distances and near, middle, and far planes relative to imaging devices lens.	$\begin{array}{cc}\hfill @400\mathrm{mm}:& (396\times 266){\mathrm{mm}}^2\hfill \\ \hfill @600\mathrm{mm}:& (588\times 392){\mathrm{mm}}^2\hfill \\ \hfill @800\mathrm{mm}:& (780\times 520){\mathrm{mm}}^2\hfill \end{array}$	$\begin{array}{c}@200\mathrm{mm}:(284\times 160){\mathrm{mm}}^2\hfill \\ @600\mathrm{mm}:(853\times 480){\mathrm{mm}}^2\hfill \\ @1000\mathrm{mm}:(1422\times 800){\mathrm{mm}}^2\hfill \end{array}$	$\begin{array}{c}@200\mathrm{mm}:(118\times 178){\mathrm{mm}}^2\hfill \\ @600\mathrm{mm}:(706\times 534){\mathrm{mm}}^2\hfill \\ @1000\mathrm{mm}:(1178\times 890){\mathrm{mm}}^2\hfill \end{array}$
Transformation between sensor lens and TCP ${{}_{s_t}{}^{TCP}\mathit{T}}_{}^{}$	${{}_{s_1}{}^{TCP}\mathit{t}}_{}^{}:(0,0,-602)\mathrm{mm}$ ${{}_{s_1}{}^{TCP}\mathit{r}}_{}^{}:(0,0,0){}^{\circ }$	${{}_{s_2}{}^{TCP}\mathit{t}}_{}^{}:(0,0,-600)\mathrm{mm}$ ${{}_{s_2}{}^{TCP}\mathit{r}}_{}^{}:(0,0,0){}^{\circ }$	${{}_{s_{3,4}}{}^{TCP}\mathit{t}}_{}^{}:(0,0,-600)\mathrm{mm}$${{}_{s_{3,4}}{}^{TCP}\mathit{r}}_{}^{}:(0,0,0){}^{\circ }$
Transformation between imaging devices of each sensor ${{}_{s_2}{}^{s_1}\mathit{T}}_{}^{}$,${{}_{s_4}{}^{s_3}\mathit{T}}_{}^{}$	${{}_{s_2}{}^{s_1}\mathit{t}}_{}^{}:(217.0,0,8.0)\mathrm{mm}$ ${{}_{s_2}{}^{s_1}\mathit{r}}_{}^{}:(0,-20.0,0){}^{\circ }$		${{}_{s_4}{}^{s_3}\mathit{t}}_{}^{}:(65.0,0,0)\mathrm{mm}$ ${{}_{s_4}{}^{s_3}\mathit{r}}_{}^{}:(0,0,0){}^{\circ }$
Transformation between both sensors ${{}_{s_3}{}^{s_1}\mathit{T}}_{}^{}$	${{}_{s_3}{}^{s_1}\mathit{t}}_{}^{}:(348.0,-81.0,42.0)\mathrm{mm}$, ${{}_{s_3}{}^{s_1}\mathit{r}}_{}^{}:(0.52,0.56,0.34){}^{\circ }$

and computed the reflected manifold of the $\mathcal{I}\text{-}\mathrm{space}$ ${\mathcal{I}}_s^{*}$ as proposed by Equation (16). Some sensors might consider a variation of the FOV between the near-middle planes and the middle-far planes. Hence, in such cases 12 vertices would be necessary to characterize the frustum space correctly. If the slope between the near and far planes is constant, eight vertices are sufficient. In the next step, we transformed ${\mathcal{I}}_s^{*}$ using the rigid transformation

$${{}_{s_1}{}^f\mathit{T}}_{}^{}={{}_{TCP}\mathit{p}}_s^f({{}_{TCP}\mathit{t}}_s^{}=B_{f_o},{{}_{TCP}\mathit{r}}_s^{}={{}^{f_0}\mathit{r}}_s^{fix})$$

to obtain the $\mathcal{C}\text{-}\mathrm{space}$ manifold ${{}_{TCP}\mathcal{C}}_1^{}$ for the sensor TCP frame and the transformation ${{}_{s_1}{}^f\mathit{T}}_{}^{}={{}_{TCP}{}^f\mathit{T}}_{}^{}·{{}_{s_1}{}^{TCP}\mathit{T}}_{}^{}$ to characterize the manifold ${{}_{s_1}\mathcal{C}}_1^{}$ for the sensor lens frame.

Figure 8 shows the resulting ${\mathcal{C}}_1^{}$ manifolds considering different sensor orientations. The left Figure 8a visualizes the ${{}_{s_1}\mathcal{C}}_1^{}$ and ${{}_{TCP}\mathcal{C}}_1^{}$ manifolds considering the following orientation in $SE\left(3\right)$: ${{}^{f_0}\mathit{r}}_s^{fix}({\alpha }_s^z=170{}^{\circ },{\beta }_s^y=5{}^{\circ },{\gamma }_s^x=45{}^{\circ })$. On the other hand, Figure 8b visualizes the $\mathcal{C}\text{-}\mathrm{space}$s just for the sensor lens considering two different sensor orientations.

To assess the validity of the characterized $\mathcal{C}\text{-}\mathrm{space}$s, we selected eight extreme sensor poses lying at the vertices of each $\mathcal{C}\text{-}\mathrm{space}$ manifold

$$\begin{array}{c}\hfill \{{\mathit{p}}_{s,1}^{}({\mathit{t}}_s^{}={\mathit{V}}_1^{{\mathcal{C}}_1^{}},{\mathit{r}}_s^{}={{}^{f_0}\mathit{r}}_s^{fix}),\dots ,\\ \hfill {\mathit{p}}_{s,8}^{}({\mathit{t}}_s^{}={\mathit{V}}_8^{{\mathcal{C}}_1^{}},{\mathit{r}}_s^{}={{}^{f_0}\mathit{r}}_s^{fix})\}\in {\mathcal{C}}_1^{}\end{array}$$

and computed their corresponding frustum spaces ${\mathcal{I}}_s^{}\left({\mathit{p}}_{s,1}^{}\right),\dots ,{\mathcal{I}}_s^{}\left({\mathit{p}}_{s,8}^{}\right)$. Our simulations confirmed that the feature $f_0$ lay within the frustum space for all extreme sensor poses, hence satisfying the core viewpoint condition (14). Some exemplary extreme sensor poses and their corresponding $\mathcal{I}\text{-}\mathrm{space}$ are shown in the Figure 8. The rest of the renders, manifolds of the $\mathcal{C}\text{-}\mathrm{space}$s, frames, and object meshes for this example and following examples can be found in the digital Supplement Material of this paper.

As expected, the computational efficiency for characterizing one $\mathcal{C}\text{-}\mathrm{space}$ showed a good performance with an average computation time (30 repetitions) of $4.1$ ms and a standard deviation of $\sigma =2.4$ ms. The computation steps comprise a read operation of the vertices (hard-coded) of the frustum space as well as the required reflection and transformation operations of a manifold with eight vertices.

4.1.4. Summary

This subsection outlines the formulation and characterization of the fundamental $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}$, which is characterized based on the sensor imaging parameters, the feature position, and a fixed sensor orientation. Using an academic example, we demonstrated that any sensor pose (fix orientation) within ${\mathcal{C}}_1^{}$ was valid to acquire the regarded feature satisfying the imaging sensor constraints. Moreover, two different strategies were proposed to efficiently characterize such a topological space based on fundamental geometric analysis.

The formulations and characterization strategies introduced in this subsection are considered the backbone of our framework. The potential and benefits of the core $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}$ are exploited within the following subsections to consider the integration of additional viewpoint constraints.

4.2. Range of Orientations

In the previous subsection, the formulation of $\mathcal{C}\text{-}\mathrm{space}$s for a fixed sensor orientation was introduced. Based on this formulation, this subsection outlines the formulation of a topological space in the special Euclidean group $SE\left(3\right)$, which allows a variation of the sensor orientation.

Within the scope of our work, and taking into account applications that comprise an a priori model of the object and its features as well as the problem simplification addressed in Section 4.1, we consider it unreasonable and inefficient to span a configuration space that comprises all orientations in ${\mathcal{R}}_s^{}\subseteq SO\left(3\right)$. This assumption can be confirmed by observing Figure 8b, which demonstrates that the topological space, which will allow sensor rotations with an incidence angle of $-25{}^{\circ }$ and $30{}^{\circ }$, does not exist.

For this reason, we consider it more practical to span a configuration space, which comprises a minimal and maximal sensor orientation range ${\mathit{r}}_s^{min}\le {\mathit{r}}_s^{}\le {\mathit{r}}_s^{max}\in {\mathcal{R}}_s^f$ instead of an unlimited space with all possible sensor orientations. The minimal and maximal orientation values can be defined considering the sensor limitations given by the second viewpoint constraint.

4.2.1. Formulation

First, consider the range of sensor orientations with

$${\mathit{r}}_{s,m}^{}\in R_s,{\mathit{r}}_s^{min}\le {\mathit{r}}_{s,m}^{}\le {\mathit{r}}_s^{max},m=1,\dots ,n,$$

and let the $\mathcal{C}\text{-}\mathrm{space}$ for a single orientation as given by Equation (15) be extended as follows:

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{C}}_2^{}\left(R_s\right)=& \{{\mathit{p}}_s^{}({\mathit{t}}_s^{},{\mathit{r}}_{s,m}^{})\in {\mathcal{C}}_2^{}\left(R_s\right)\hfill \\ \hfill & \mid {\mathit{t}}_s^{}\in {\mathcal{T}}_s^{},{{\mathit{g}}_{\mathit{f}}}_{,0}^{}\in {\mathcal{I}}_s^{},\hfill \\ \hfill & {\mathit{r}}_{s,m}^{}\in R_s,R_s\subseteq {\mathcal{R}}_s^f\}.\hfill \end{array}\end{array}$$

(17)

The topological space, which considers a range of sensor orientations, and denoted by ${\mathcal{C}}_2^{}\left(R_s\right)$, can be seamlessly computed by intersecting the individual configuration spaces for each m orientation:

$${\mathcal{C}}_2^{}\left(R_s\right)=\bigcap _{r_{s,m}\in R_s,}^n{\mathcal{C}}_1^{}({\mathit{r}}_{s,m}^{},{\mathcal{I}}_s^{},B_f).$$

(18)

4.2.2. Characterization

The $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_2^{}\left(R_s\right)$ as given by Equation (18) can be seamlessly computed using CSG Boolean Intersection operations. Considering that each intersection operation yields a new manifold with more vertices and edges, it is well known that the computation complexity of CSG operations increases with the number of vertices and edges. Thus, to compute ${\mathcal{C}}_2^{}\left(R_s\right)$ in a feasible time, a discretization of the orientation range must be first considered.

One simple and pragmatic solution is to consider different sensor orientations comprising the maximal and minimal allowed sensor orientations, e.g., $({\mathit{r}}_s^{min},{\mathit{r}}_{s,}^{ideal},{\mathit{r}}_s^{max})\in R_s$. Figure 9a illustrates the 2D ${\mathcal{C}}_1^{}$ manifolds of five different sensor orientations $\{-{20}^{{}^{\circ }},-{10}^{{}^{\circ }},0^{{}^{\circ }},{10}^{{}^{\circ }},{20}^{{}^{\circ }}\}\in {\beta }_s^y$ using a discretization step of $r_s^d=10{}^{\circ }$ for the positioning frames TCP ${{}_{TCP}\mathcal{C}}_2^{}\left(R_s\right)$ and sensor lens frame ${{}_{s_1}\mathcal{C}}_2^{}\left(R_s\right)$. The ${\mathcal{C}}_2^{}$ manifolds are characterized by intersecting all individual spaces ${\mathcal{C}}_1^{}$ as given by Equation (18).

As can be observed from Figure 9b,c, it should be noted that the manifolds ${{}_{s_1}\mathcal{C}}_2^{}$ and ${{}_{TCP}\mathcal{C}}_2^{}$ span different topological spaces (here in $SE\left(1\right)$) depending on the selected positioning frame. Contrary to the $\mathcal{C}\text{-}\mathrm{space}$s ${{}_{s_1}\mathcal{C}}_1^{}$ and ${{}_{TCP}\mathcal{C}}_1^{}$ considering a fixed sensor position, the selection of the reference positioning frame should be considered before computing ${\mathcal{C}}_2^{}$ and taking into account the explicit requirements and constraints of the vision task.

• Discretization without Interpolation Note that ${\mathcal{C}}_2^{}\left(R_s\right)$ spans a topological space that is just valid for the sensor orientations in $R_s$ and that the sensor orientation ${\mathit{r}}_s^{}$ cannot be arbitrary chosen within the range ${\mathit{r}}_s^{min}<{\mathit{r}}_s^{}<{\mathit{r}}_s^{max}$. This characteristic can be more easily understood by comparing the volume form, $Vol$, of the $\mathcal{C}\text{-}\mathrm{space}$s ${\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_s^{max})$ and ${\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_{s,}^{ideal},{\mathit{r}}_s^{max})$, which would show that the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_s^{max})$ is less restrictive:

  $$Vol\left({\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_s^{max})\right)>Vol\left({\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_{s,}^{ideal},{\mathit{r}}_s^{max})\right).$$

  (19)
  This characteristic can particularly be appreciated in the top of the ${{}_{s_1}\mathcal{C}}_2^{}({\mathit{r}}_s^{min},{\mathit{r}}_s^{max})$ manifold in Figure 9a. Thus, it should be kept in mind that the constrained space ${\mathcal{C}}_2^{}\left(R_s\right)$ does not allow an explicit interpolation within the orientations of $R_s$.

• Approximation of ${\mathcal{C}}_2^{}$ However, as can be observed from Figure 9, the topological space spanned while considering a step size of $10{}^{\circ }$, ${\mathcal{C}}_2^{}\left(R_s(r_s^d=10{}^{\circ })\right)$, is almost identical to the space if we would relax the step size to $20{}^{\circ }$, ${\mathcal{C}}_2^{}\left(R_s(r_s^d=20{}^{\circ })\right)$. Hence, it can be assumed for this case that the $\mathcal{C}\text{-}\mathrm{space}$s are almost identical and the following condition will hold:

  $${\mathcal{C}}_2^{}\left(R_s(r_s^d=10{}^{\circ })\right)\approx {\mathcal{C}}_2^{}\left(R_s(r_s^d=20{}^{\circ })\right).$$

  (20)

4.2.3. Verification

For verification purposes, we consider an academic example with the following sensor orientation ranges: ${\gamma }_s^x=\{-5,0,5\}{}^{\circ }$, ${\beta }_s^y=\{-5,0,5\}{}^{\circ }$, and ${\alpha }_s^z=\{-10,0,10\}{}^{\circ }$. The resulting $\mathcal{C}\text{-}\mathrm{space}$, ${\mathcal{C}}_2^{}\left(R_s\right)$, was computed by intersecting the constrained space ${\mathcal{C}}_1^{}(f_0,{\mathcal{I}}_s^{},{{}^f\mathit{r}}_s^{})$ for each possible sensor orientation combination, i.e., $3^3=27$ sensor orientations (cf. Section 4.1). The computation time correspond to $t\left({\mathcal{C}}_2^{}\left(R_s\right)\right)=15$ s. Figure A1 visualizes the 6D manifold of the $\mathcal{C}\text{-}\mathrm{space}$ obtained through Boolean intersection operations. Additionally, the ideal constrained space considering a null rotation is also displayed to show a qualitative comparison of the reduction of the $\mathcal{C}\text{-}\mathrm{space}$ considering a rotation space in $SE\left(3\right)$.

For verifying the validity of the computed manifold, four extreme viewpoints and their corresponding frustum spaces are displayed in Figure A1 considering the following random orientations:

$$\begin{array}{cc}\hfill {{}^f\mathit{r}}_{s,1}^{}& ({\gamma }_s^x=5{}^{\circ },{\beta }_s^y=0{}^{\circ },{\alpha }_s^z=10{}^{\circ }),\hfill \\ \hfill {{}^f\mathit{r}}_{s,2}^{}& ({\gamma }_s^x=0{}^{\circ },{\beta }_s^y=-5{}^{\circ },{\alpha }_s^z=-10{}^{\circ }),\hfill \\ \hfill {{}^f\mathit{r}}_{s,3}^{}& ({\gamma }_s^x=-4{}^{\circ },{\beta }_s^y=-5{}^{\circ },{\alpha }_s^z=-8{}^{\circ }),\hfill \\ \hfill {{}^f\mathit{r}}_{s,4}^{}& ({\gamma }_s^x=3{}^{\circ },{\beta }_s^y=-3{}^{\circ },{\alpha }_s^z=-9{}^{\circ }).\hfill \end{array}$$

Note that while the first two viewpoints consider an explicit orientation within the given orientation range $\{{{}^f\mathit{r}}_{s,1}^{},{{}^f\mathit{r}}_{s,2}^{}\}\in R_s$, the sensor orientation of the third and fourth viewpoints are not elements of $\{{{}^f\mathit{r}}_{s,3}^{},{{}^f\mathit{r}}_{s,4}^{}\}\notin R_s$; however, they lie within the interpolation range. The frustum spaces prove that all viewpoints can capture f satisfactorily. Although the sensor poses ${\mathit{p}}_{s,3}^{}$ and ${\mathit{p}}_{s,4}^{}$ as valid in this case, this assumption cannot be guaranteed for any other arbitrary orientation. Nevertheless, this confirms that the approximation condition as given by Equation (20) holds to some extent.

To provide a more quantifiable evaluation of this approximation, the constrained space, ${\mathcal{C}}_2^{}$, considering finer discretization steps $r_s^d$ of $2.5{}^{\circ }$ and $1{}^{\circ }$ for ${\gamma }_s^x$ and ${\beta }_s^y$ was computed. The total number of computed manifolds corresponds to $5\times 5\times 3=75$ $\mathcal{C}\text{-}\mathrm{space}$s with $t\left({\mathcal{C}}_2^{}\left(2.5{}^{\circ }\right)\right)=40s$ for $r_s^d=2.5{}^{\circ }$ and $11\times 11\times 3=363$ $\mathcal{C}\text{-}\mathrm{space}$s with $t\left({\mathcal{C}}_2^{}\left(1{}^{\circ }\right)\right)=720$ s for $r_s^d=2.5{}^{\circ }$. The relative volumetric ratio between the computed spaces is given as follows: $Vol\left({\mathcal{C}}_2^{}\left(1{}^{\circ }\right)\right)/Vol\left({\mathcal{C}}_2^{}\left(2.5{}^{\circ }\right)\right)=0.9999$ and $Vol\left({\mathcal{C}}_2^{}\left(1{}^{\circ }\right)\right)/Vol\left({\mathcal{C}}_2^{}\left(5{}^{\circ }\right)\right)=0.9995$. These experiments show that the differences between the manifold volume ratios for the selected steps are insignificant and that the approximation with a step of $r_s^d=2.5{}^{\circ }$ holds for this case.

General Notes It is important to note that the validity of the approximation introduced by Equation (20) must be individually assessed for each individual application, imaging parameters, and other constraints. Some preliminary experiments showed that when considering further viewpoint constraints that depend on the sensor orientation, e.g., the feature geometry (see Section 4.3), the differences between the spaces using different discretization steps may be more considerable. A more comprehensive analysis falls outside the scope of this study and remains to be further investigated. We urge the reader to perform some empirical experiments for choosing an adequate discretization step and a good trade-off between accuracy and efficiency.

4.2.4. Summary

Contrary to the previously introduced $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}$ which is limited to a fixed sensor orientation, this subsection outlined the formulation and characterization of the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_2^{}$ in $SE\left(3\right)$ that satisfies the sensor imaging parameters for different sensor orientations. We demonstrated that the manifold ${\mathcal{C}}_2^{}$ is straightforwardly characterized by intersecting multiple $\mathcal{C}\text{-}\mathrm{space}$s with different sensor orientations.

4.3. Feature Geometry

In many applications, the feature geometry is a fundamental viewpoint constraint that may considerably limit the space in $SE\left(3\right)$ for positioning the sensor. This subsection shows that the required $\mathcal{C}\text{-}\mathrm{space}$ affected by the feature geometry can be efficiently and explicitly characterized using trigonometric relationships that depend on the feature geometry, the sensor’s FOV angles, and the sensor orientation.

4.3.1. Formulation

Taking into account the third viewpoint constraint (see Section 2.8), it can be assumed that all feature surface points $G_f\left(L_f\right)$ must be acquired simultaneously. The $\mathcal{C}\text{-}\mathrm{space}$ that fulfills this requirement can be easily formulated by extending the base constraint of ${\mathcal{C}}_1^{}$ as given by Equation (15), considering that all surface points must lie within the frustum space:

$$\begin{array}{cc}\hfill {\mathcal{C}}_3^{}=& \{\forall {\mathit{p}}_s^{}({\mathit{t}}_s^{},{{}^{{}_{}}\mathit{r}}_s^{fix})\in {\mathcal{C}}_3^{}({\mathcal{C}}_1^{},G_f(L_f),{{}^{{}_{}}\mathit{r}}_s^{fix},{\mathcal{I}}_s^{})\hfill \\ \hfill & |\forall {\mathit{t}}_s^{}\in {\mathcal{T}}_s^{},{{}^{{}_{}}\mathit{r}}_s^{fix}\in {\mathcal{R}}_s^{},G_f(L_f)\subseteq {\mathcal{I}}_s^{}\}.\hfill \end{array}$$

(21)

4.3.2. Generic Characterization

Taking into account the generic formulation of the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_3^{}$ from Equation (21), in the simplest case, it can be assumed that ${\mathcal{C}}_3^{}$ could be obtained by scaling ${\mathcal{C}}_1^{}$. Let the required scaling vector be denoted by ${\Delta }_{}^{}({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)$ and depend on the feature geometry, the sensor rotation, and the FOV angles of the sensor. The generic characterization of the ${\mathcal{C}}_3^{}$ manifold can then be expressed as follows

$${\mathcal{C}}_3^{}={\mathcal{C}}_1^{}({\Delta }_{}^{}({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)).$$

Recalling that the ${\mathcal{C}}_1^{}$ manifold is not symmetrical in all planes, hence, the rotation variation assumes that the ${\mathcal{C}}_3^{}$ manifold cannot be correctly scaled regarding the same scaling vector. Thus, a more generic and flexible approach can then be considered by letting each k vertex of $\forall {\mathit{V}}_{v,k}^{{\mathcal{C}}_1^{}}\in V_v^{{\mathcal{C}}_1^{}}$ be individually scaled. Hence, each vertex of ${\mathcal{C}}_3^{}$ can be computed with

$${\mathit{V}}_k^{{\mathcal{C}}_3^{}}={\mathit{V}}_k^{{\mathcal{C}}_1^{}}-{\Delta }_k^{}({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y),$$

(22)

considering the following generalized vector:

$${\Delta }_k^{}({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)=\left(\begin{array}{c}{\Delta }_k^x({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)\\ {\Delta }_k^y({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)\\ {\Delta }_k^z({{}^{{}_{}}\mathit{r}}_s^{fix},L_f,{\theta }_s^x,{\psi }_s^y)\end{array}\right).$$

(23)

The explicit characterization of the scaling vector from Equation (23) requires an individual and comprehensive trigonometric analysis of each k vertex of ${\mathcal{C}}_3^{}$, which depends on the chosen sensor orientation. Moreover, since the scaling vector ${\Delta }_k^{}$ depends also on the feature geometrical properties, from now on we will assume the generalization of the feature geometry as introduced in Section 2.4 to characterize any feature by a square of the length $l_f$. This simplification contributes to a higher generalization of our models for different topologies and facilitates the comprehension of the trigonometric relationships introduced in the following subsections.

4.3.3. Characterization of the $\mathcal{C}$-Space with Null Rotation

The most straightforward scenario to quantify the influence of the feature geometry on the constrained space is first to consider a null rotation, ${{}^f\mathit{r}}_s^0$, of the sensor relative to the feature, i.e., the feature’s plane is parallel to the $xy$-plane of the TCP and the rotation around the optical axis equals zero, ${{}^f\mathit{r}}_s^0({\gamma }_s^x={\beta }_s^y={\alpha }_s^z=0)$.

First, span the core constrained space, ${\mathcal{C}}_1^{}$, considering the feature position and the null rotation of the sensor. Then, parting from one vertex of ${\mathcal{C}}_1^{}$, let the $\mathcal{I}\text{-}\mathrm{space}$ be translated in one direction until ${\mathcal{I}}_s^{}$ entirely encloses the whole feature. This step is exemplary, shown in the $x\text{-}z$ plane in Figure 10 at the third vertex of ${\mathcal{C}}_1^{}$, and can be interpreted as an analogy to the Extreme Viewpoints Interpretation (cf. Section 1) used to span ${\mathcal{C}}_1^{}$. Then, it is easily understood that to characterize ${\mathcal{C}}_3^{}$ all vertices $\forall {\mathit{V}}_{v,k}^{{\mathcal{C}}_1^{}}\in V_v^{{\mathcal{C}}_1^{}}$ of ${\mathcal{C}}_1^{}$ must be shifted in the x and y directions by a factor of $0.5·l_f$:

$$\begin{array}{cc}{\Delta }_k^x({\mathit{r}}_s^0,l_f)=& 0.5·l_f\\ {\Delta }_k^y({\mathit{r}}_s^0,l_f)=& 0.5·l_f\\ {\Delta }_k^z({\mathit{r}}_s^0,l_f)=& 0.\end{array}$$

(24)

4.3.4. Rotation around One Axis

Any other sensor orientation different from the null orientation requires an individual analysis of the exact trigonometric relationships for each vertex. To break down the complexity of this problem, within this subsection, we first provide the geometrical relationships needed to characterize the constrained space ${\mathcal{C}}_3^{}$ while considering an individual rotation around each axis. The characterization of the constrained spaces follows the same approach described in the previous subsection, which requires first the characterization of the base constraint, ${\mathcal{C}}_1^{}$, and then the derivation of the scaling vectors.

• Rotation around z-axis ${\alpha }_s^z\ne 0$ Assuming a sensor rotation around the optical axis, ${{}^f\mathit{r}}_s^z({\alpha }_s^z\ne 0,{\phi }_s({\beta }_s^y,{\gamma }_s^x)=0)$ (see Figure 11), the $\mathcal{C}\text{-}\mathrm{space}$ is scaled just along the vertical and horizontal axes, using the following scaling factors:

  $$\begin{array}{ccc}{\Delta }_k^x({\mathit{r}}_s^z,l_f)& =& \frac{l_f}{2}·(cos(|{\alpha }_s^z\left|\right)+sin\left(\right|{\alpha }_s^z\left|\right))\\ {\Delta }_k^y({\mathit{r}}_s^z,l_f)& =& \frac{l_f}{2}·(cos(|{\alpha }_s^z\left|\right)+sin\left(\right|{\alpha }_s^z\left|\right))\\ {\Delta }_k^z({\mathit{r}}_s^z,l_f)& =& 0.\end{array}$$

  (25)
  The derivation of the trigonometric relationships from Equation (25) can be better understood by looking at Figure A2.
• Rotation around x-axis or y-axis (${\gamma }_s^x\ne 0⊻{\beta }_s^y\ne 0$): A rotation of the sensor around the x-axis, ${\mathit{r}}_s^{}({\gamma }_s^x\ne 0,{\alpha }_s^z={\beta }_s^y=0)$, or y-axis, ${\mathit{r}}_s^{}({\beta }_s^y\ne 0,{\alpha }_s^z={\gamma }_s^x=0)$, requires deriving individual trigonometric relationships for each vertex of ${\mathcal{C}}_3^{}$. Besides the feature length, other parameters such as the FOV angles (${\theta }_s^x,{\psi }_s^y$) and the direction of the rotation must be considered.
  The scaling factors for the eight vertices of the $\mathcal{C}\text{-}\mathrm{space}$ while considering a rotation around the x-axis or y-axis can be found in

  Table A5. Scaling factors for the vertices of the constrained space ${\mathit{V}}_k^{{\mathcal{C}}_3^{}}$, considering a sensor rotation around the x-axis or y-axis relative to the feature frame.

  k Vertex of ${\mathit{V}}_{\mathit{k}}^{{\mathcal{C}}_3^{}}$	Rotation around y-axis				Rotation around x-axis
  	${{}^{\mathit{f}}\mathit{r}}_{\mathit{s}}^{}({\mathsf{\alpha }}_{\mathit{s}}^{\mathit{z}}={\mathsf{\gamma }}_{\mathit{s}}^{\mathit{x}}=0,{\mathsf{\beta }}_{\mathit{s}}^{\mathit{y}}\ne 0)$				${{}^{\mathit{f}}\mathit{r}}_{\mathit{s}}^{}({\mathsf{\alpha }}_{\mathit{s}}^{\mathit{z}}={\mathsf{\beta }}_{\mathit{s}}^{\mathit{y}}=0,{\mathsf{\gamma }}_{\mathit{s}}^{\mathit{x}}\ne 0)$
  	${\Delta }_{\mathit{k}}^{\mathit{x}}$		${\Delta }_{\mathit{k}}^{\mathit{y}}$	${\Delta }_{\mathit{k}}^{\mathit{z}}$	${\Delta }_{\mathit{k}}^{\mathit{x}}$	${\Delta }_{\mathit{k}}^{\mathit{y}}$		${\Delta }_{\mathit{k}}^{\mathit{z}}$
  	${\mathsf{\beta }}_{\mathit{s}}^{\mathit{y}}<\mathbf{0}$	${\mathsf{\beta }}_{\mathit{s}}^{\mathit{y}}>\mathbf{0}$				${\mathsf{\gamma }}_{\mathit{s}}^{\mathit{x}}<\mathbf{0}$	${\mathsf{\gamma }}_{\mathit{s}}^{\mathit{x}}>\mathbf{0}$
  1	${\lambda }^x$	${\rho }^x$	$\frac{l_f}{2}$	${\rho }^{z,y}$	$\frac{l_f}{2}$	${\lambda }^y$	${\rho }^y$	${\rho }^{z,x}$
  2	${\rho }^x$	${\lambda }^x$	$\frac{l_f}{2}$		$\frac{l_f}{2}$	${\lambda }^y$	${\rho }^y$
  3	${\sigma }^x$	${\rho }^x$	$\frac{l_f}{2}+{\mathsf{\varsigma }}^{x,y}$		$\frac{l_f}{2}+{\mathsf{\varsigma }}^{y,x}$	${\rho }^y$	${\sigma }^y$
  4	${\rho }^x$	${\sigma }^x$	$\frac{l_f}{2}+{\mathsf{\varsigma }}^{x,y}$		$\frac{l_f}{2}+{\mathsf{\varsigma }}^{y,x}$	${\rho }^y$	${\sigma }^y$
  5	${\lambda }^x$	${\rho }^x$	$\frac{l_f}{2}$		$\frac{l_f}{2}$	${\rho }^y$	${\lambda }^y$
  6	${\rho }^x$	${\lambda }^x$	$\frac{l_f}{2}$		$\frac{l_f}{2}$	${\rho }^y$	${\lambda }^y$
  7	${\sigma }^x$	${\rho }^x$	$\frac{l_f}{2}+{\mathsf{\varsigma }}^{x,y}$		$\frac{l_f}{2}+{\mathsf{\varsigma }}^{y,x}$	${\sigma }^y$	${\rho }^y$
  8	${\rho }^x$	${\sigma }^x$	$\frac{l_f}{2}+{\mathsf{\varsigma }}^{x,y}$		$\frac{l_f}{2}+{\mathsf{\varsigma }}^{y,x}$	${\sigma }^y$	${\rho }^y$

  regarding the following general auxiliary lengths for ${{}^f\mathit{r}}_s^{}({\alpha }_s^z={\gamma }_s^x=0,{\beta }_s^y\ne 0)$ (left) and for ${{}^f\mathit{r}}_s^{}({\alpha }_s^z={\beta }_s^y=0,{\gamma }_s^x\ne 0)$ (right):

$$\begin{array}{cccc}\hfill {\rho }^{z,y}=& \frac{l_f}{2}·sin\left(\right|{\beta }_s^y\left|\right)\hfill & \hfill & {\rho }^{z,x}=\frac{l_f}{2}·sin\left(\right|{\gamma }_s^x\left|\right)\hfill \\ \hfill {\rho }^x=& \frac{l_f}{2}·cos\left(\right|{\beta }_s^y\left|\right)\hfill & \hfill & {\rho }^y=\frac{l_f}{2}·cos\left(\right|{\gamma }_s^x\left|\right)\hfill \\ \hfill {\mathsf{\varsigma }}^x=& 2·{\rho }^{z,y}·tan\left(0.5·{\theta }_s^x\right)\hfill & \hfill & {\mathsf{\varsigma }}^y=2·{\rho }^{z,x}·tan\left(0.5·{\psi }_s^y\right)\hfill \\ \hfill {\mathsf{\varsigma }}^{x,y}=& 2·{\rho }^{z,y}·tan\left(0.5·{\psi }_s^y\right)\hfill & \hfill & {\mathsf{\varsigma }}^{y,x}=2·{\rho }^{z,x}·tan\left(0.5·{\theta }_s^x\right)\hfill \\ \hfill {\lambda }^x=& {\rho }^x-{\mathsf{\varsigma }}^x\hfill & \hfill & {\lambda }^y={\rho }^y-{\mathsf{\varsigma }}^y\hfill \\ \hfill {\sigma }^x=& {\rho }^x+{\mathsf{\varsigma }}^x\hfill & \hfill & {\sigma }^y={\rho }^y+{\mathsf{\varsigma }}^y.\hfill \end{array}$$

The derivation of the trigonometric relationships can be better understood using an exemplary case. First, assume a rotation of the sensor around the y-axis of ${\beta }_s^y>0\wedge {\gamma }_s^x=0$, as illustrated in Figure 12. The trigonometric relationships can then be derived for each vertex following the Extreme Viewpoints Interpretation as exemplary and depicted in Figure A3).

4.3.5. Generalization to 3D Features

Although our approach contemplates primary 2D features, the constrained space ${\mathcal{C}}_3^{}$ can be seamlessly extended to acquire 3D features considering a feature height $h_f\in L_f$.

This paper only considers the characterization of the scaling vectors for concave (e.g., pocket, slot) and convex (e.g., cube, half-sphere) features with a null rotation of the sensor, ${{}^f\mathit{r}}_s^0$. For instance, the back vertices $(k=1,2,5,6)$ of ${\mathcal{C}}_3^{}$ to capture a concave feature, as shown in Figure 13, must be scaled using the following factors:

$$\begin{array}{cc}\hfill {\Delta }_k^x=& h_f·tan\left(0.5·{\theta }_s^x\right)+0.5·l_f\hfill \\ \hfill {\Delta }_k^y=& h_f·tan\left(0.5·{\psi }_s^y\right)+0.5·l_f\hfill \\ \hfill {\Delta }_k^z=& h_f.\hfill \end{array}$$

(26)

The front vertices are scaled using the same factors as for a 2D feature as given by Equation (24). For convex features, let all vertices be scaled with the factors given by Equation (26), except for the depth delta factor of the back vertices, which follows ${\Delta }_k^z=0$.

Note that the characterization of the $\mathcal{C}\text{-}\mathrm{space}$, ${\mathcal{C}}_3^{}$, for considering 3D features just guarantees that the entire feature surface lies within the frustum space. We neglect any further visibility constraints that may influence the viewpoint’s validity, such as the maximal angles for the interiors of a concave feature. Moreover, it should be noted that the scaling factors given within this subsection hold for just a null rotation. The characterization of the scaling factors for other sensors orientations can also be derived by extending the previously introduced relationships for 2D features in Section 4.3.4.

4.3.6. Verification

The verification of the geometrical relationships introduced within this subsection was performed based on an academic example to acquire a square feature $f_1$ and a 3D pocket feature $f_1^{*}$. The $\mathcal{C}\text{-}\mathrm{space}$s for $f_1$, i.e., ${\mathcal{C}}_{3,1}^{}\left({{}^{f_1}\mathit{r}}_{s,1}^{}\right)$ and ${\mathcal{C}}_{3,2}^{}\left({{}^{f_1}\mathit{r}}_{s,2}^{}\right)$, consider a sensor orientation of ${{}^{f_1}\mathit{r}}_{s,1}^{}({\alpha }_s^z={\beta }_s^y=0,{\gamma }_s^x=30{}^{\circ })$ and ${{}^{f_1}\mathit{r}}_{s,2}^{}({\gamma }_s^x={\beta }_s^y=0,{\alpha }_s^z=15{}^{\circ })$, while the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_{3,3}^{}\left({{}^{f_1^{*}}\mathit{r}}_{s,3}^{}\right)$ for $f_1^{*}$ considers a null sensor orientation ${{}^{f_1^{*}}\mathit{r}}_{s,3}^{}({\alpha }_s^z={\beta }_s^y=,{\gamma }_s^x=0{}^{\circ })$. All constrained spaces were computed using the imaging parameters of sensor $s_1$ (cf. Table A6) and geometric parameters of the features from

Table A7. Overview of features and occlusion objects used for verification steps and simulation-based analysis.

Feature	$f_0$	$f_1$,$f_1^{*}$	$f_2$	$f_3$	${\kappa }_1$	${\kappa }_2$
Topology	Point	Slot	Circle	Half-Sphere	Icosahedron	Octaeder
Generalized Topology	-	Square	Square	Cube	-	-
Dimensions in mm	$l_{f_o}=0$	$l_{f_1}=50$, $h_{f_1^{*}}=30$	$l_{f_2}=20$	$l_{f_2}=40$, $h_{f_2}=40$	edge length: $\approx 14.0$	edge length: $\approx 20.0$
Translation vector in object’s frame $t_o^{}={(x_o,y_o,z_o)}^T$ in mm	$\left(\begin{array}{c}0.0\\ 0.0\\ 0.0\end{array}\right)$	$\left(\begin{array}{c}0.0\\ 0.0\\ 0.0\end{array}\right)$	$\left(\begin{array}{c}75.0\\ 150.0\\ 20.0\end{array}\right)$	$\left(\begin{array}{c}120.0\\ 30.0\\ 0.0\end{array}\right)$	$\left(\begin{array}{c}-67.5\\ 0.0\\ 240.0\end{array}\right)$	$\left(\begin{array}{c}117.5\\ 100.0\\ 445.0\end{array}\right)$
Rotation in Euler Angles in object’s frame $r_o^{}({\gamma }_s^x,{\beta }_s^y,{\alpha }_s^z)$ in ${}^{\circ }$	$(0,0,0)$	$(0,0,0)$	$(0,20,0)$	$(0,0,0)$	$(0,0,0)$	$(0,0,0)$

.

Figure 14 visualizes the scene comprising the $\mathcal{C}\text{-}\mathrm{spaces}$ for acquiring $f_1$ and $f_1^{*}$. All ${\mathcal{C}}_{3,1-3}^{}$ manifolds were computed by scaling first the manifold of the frustum space, considering the scaling factors addressed within the past subsections, and then by reflecting and transforming the manifold with the corresponding sensor orientation (for ${\mathcal{C}}_{3,1}^{}$ see Table A5, for ${\mathcal{C}}_{3,2}^{}$ see Equation (25), and for ${\mathcal{C}}_{3,3}^{}$ see Equation (26)).

To verify the geometrical relationships introduced within this subsection, a virtual camera using the trimesh Library [57] and the imaging parameters of $s_1$ were created. Then, the depth images and their corresponding point clouds at eight extreme viewpoints, i.e., the manifold vertices, were rendered to verify that the features could be acquired from each viewpoint. The images and point clouds of all extreme viewpoints confirm that the features lie at the border of the frustum space and can be entirely captured. Figure A4a–c demonstrate this empirically and show the depth images and point clouds at the selected extreme viewpoints (${{}^{f_1}\mathit{p}}_{s,1}^{},{{}^{f_1}\mathit{p}}_{s,2}^{},{{}^{f_1^{*}}\mathit{p}}_{s,3}^{}$) from Figure 14.

Our approach provides an analytical and straightforward solution for efficiently characterizing the $\mathcal{C}\text{-}\mathrm{space}$ which is limited by the frustum space and feature geometry. Since the delta factors can be applied directly to the vertices of the frustum space, the computational cost is similarly efficient to the computation time of ${\mathcal{C}}_1^{}$ with an average computation time of $5.8$ ms and $\sigma =3.2$ ms.

4.3.7. Summary

This subsection extended the formulation of the core $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}$ (see Section 4) to a $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_3^{}$, taking into account the feature’s geometry dimensions. Using an exhaustive trigonometric analysis, our study introduced the exact relationships to characterize the vertices of the required multi-dimensional manifold, considering a generalized model of the feature geometry, the sensor’s orientation, and its FOV angles. Our findings show that the characterization of the $\mathcal{C}\text{-}\mathrm{space}$ constrained by the feature geometry can be computed with high efficiency using an analytical approach.

The trigonometric relationships introduced in this section are sufficient to characterize the $\mathcal{C}\text{-}\mathrm{space}$ manifold while taking into account the rotation of the sensor in one axis. Characterizing the explicit relationships for a simultaneous rotation of the sensor in all axes is beyond the scope of this paper. However, the trigonometric relationships and general approach presented in this subsection can be used as the basis for their derivation.

4.4. Constrained Spaces Using Scaling Vectors

If a viewpoint constraint can be formulated using scaling vectors, as suggested for the feature geometry in Section 4.3, then the same approach can be equally applied to characterize the constrained space of different viewpoint constraints. This subsection introduces a generic formulation for integrating such viewpoint constraints and proposes characterizing kinematic errors and the sensor accuracy following this approach.

4.4.1. Formulation

If the influence of an i viewpoint constraint $c_i\in \tilde{C}$ can be characterized by a scaling vector $\Delta \left(c_i\right)$ to span its corresponding constrained space, ${\mathcal{C}}_i^{}$, then its vertices ${\mathit{V}}_{}^{{\mathcal{C}}_i^{}}$ can be scaled using a generalized formulation of Equation (22):

$${\mathit{V}}_k^{{\mathcal{C}}_i^{}}={\mathit{V}}_k^{{\mathcal{C}}_1^{}}-\Delta \left(c_i\right).$$

(27)

• Integrating Multiple Constraints The characterization of a jointed constrained space, which integrates several viewpoint constraints, can be computed using different approaches. On one hand, the constrained spaces can be first computed and intersected iteratively using CSG operations, as originally proposed in Equation (12). However, if the space spanned by such viewpoint constraints can be formulated according to Equation (27), the characterization of the constrained space ${\mathcal{C}}_{}^{}$ can be more efficiently calculated by simply adding all scaling vectors:

  $${\mathit{V}}_k^{{\mathcal{C}}_{}^{}}\left(\tilde{C}\right)={\mathit{V}}_k^{{\mathcal{C}}_1^{}}-\sum _{\begin{array}{c}\hfill c_i\in \tilde{C},i\ne \end{array}1}\Delta \left(c_i\right).$$

  (28)
  While the computational cost of CSG operations is at least proportional to the number of vertices between two surface models, note that the complexity of the sum of Equation (28) is just proportional to the number of viewpoint constraints.

• Compatible Constraints Within this subsection, we propose further possible viewpoint constraints that can be characterized according to the scaling formulation introduced by Equation (28).
  • Kinematic errors: Considering the fourth viewpoint constraint and the assumptions addressed in Section 2.8, the maximal kinematic error $ϵ$ is given by the sum of the alignment error ${ϵ}_e$, the modeling error of the sensor imaging parameters ${ϵ}_s$, and the absolute position accuracy of the robot ${ϵ}_r$:

    $$\left|ϵ\right|=\left|{ϵ}_e\right|+\left|{ϵ}_s\right|+\left|{ϵ}_r\right|.$$
    Assuming that the total kinematic error has the same magnitude in all directions, all vertices can be equally scaled. The vertices of the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_4^{}\left(ϵ\right)$ are computed using the scaling vector ${\Delta }_{}^{}\left(ϵ\right)$:

    $${\mathit{V}}_k^{{\mathcal{C}}_4^{}}\left({\Delta }_{}^{}\left(ϵ\right)\right)={\mathit{V}}_k^{{\mathcal{C}}_1^{}}-{\Delta }_{}^{}\left(ϵ\right)).$$
  • Sensor Accuracy: If the accuracy of the sensor $a_s$ can be quantified within the sensor frustum, then similarly to the kinematic error, the manifold of the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_5^{}\left(a_s\right)$ can be characterized using a scaling vector $\Delta \left(a_s\right)$:

    $${\mathit{V}}_k^{{\mathcal{C}}_5^{}}\left({\Delta }_{}^{}\left(a_s\right)\right)={\mathit{V}}_k^{{\mathcal{C}}_1^{}}-{\Delta }_{}^{}\left(a_s\right)).$$

Figure A5 visualizes an exemplary and more complex scenario comprising an individual scaling of each vertex. This example shows the high flexibility and adaptability of this approach for synthesizing $\mathcal{C}\text{-}\mathrm{space}$s for different viewpoint constraints according to the particular necessities of the application in consideration.

4.4.2. Summary

The use of scaling vectors is an efficient and flexible approach to characterizing the $\mathcal{C}\text{-}\mathrm{space}$ spanned by any viewpoint constraint. Within this section, we considered a few viewpoint constraints that can be modeled aligned to this formulation. However, it should be noted that this approach requires an explicit characterization of the individual scaling vectors and the overall characterization may be limited by the number of vertices of the base $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_1^{}\left({\mathcal{I}}_s^{}\right)$. For instance, our model of the $\mathcal{I}\text{-}\mathrm{space}$ considers just eight vertices. Thus, any viewpoint constraint that requires a more complex geometrical representation could be limited by this.

Moreover, note that if the regarded viewpoint constraint can be explicitly characterized as a manifold in $SE\left(3\right)$, this $\mathcal{C}\text{-}\mathrm{space}$ can then be directly intersected with the rest of the constraints as originally suggested by Equation (12); such an example is given for characterizing the robot workspace (see Section 4.8.1). However, note that such an approach is generally computationally more expensive; hence, this study recommends that the characterization of viewpoint constraints using scaling vectors be prioritized whenever possible.

4.5. Occlusion Space

We consider the occlusion-free view of a feature (related terms: shadow or bistatic effect) a non-negligible requirement that must be individually assessed for each viewpoint. In the context of our framework, this subsection outlines the formulation of a negative topological space—the occlusion space ${\mathcal{C}}_6^{occl}$—to ensure the free visibility of each viewpoint within the $\mathcal{C}\text{-}\mathrm{space}$. Although other authors [15,18,26] have already suggested the characterization of such spaces, the present study proposes a new formulation of such a space aligned to our framework. Our approach strives for an efficient characterization of ${\mathcal{C}}_6^{occl}$ using simplifications about the feature’s geometry and the occlusion bodies.

4.5.1. Formulation

If a feature f is not visible from at least one sensor pose within the $\mathcal{C}\text{-}\mathrm{space}$, it can be assumed that at least one rigid body of the RVS is impeding its visibility. Thus, an occlusion space for f denoted as ${\mathcal{C}}_6^{occl}\subset SE\left(3\right)$ exists and a valid sensor pose cannot be an element of it ${\mathit{p}}_s^{}\notin {\mathcal{C}}_6^{occl}$. However, it is well known that the characterization of such occlusion spaces can generally be computationally expensive. Therefore, it seems inefficient to formulate ${\mathcal{C}}_6^{occl}$ in the special Euclidean group for all possible sensor orientations. For this reason, and by exploiting the available $\mathcal{C}\text{-}\mathrm{space}$ spanned by other viewpoint constraints, ${\mathcal{C}}_6^{occl}$ can be formulated based on the previously generated $\mathcal{C}\text{-}\mathrm{space}$, a given sensor orientation ${{}^{{}_{}}\mathit{r}}_s^{fix}$, and the surface models of the rigid bodies $\kappa \in K$:

$${\mathcal{C}}_6^{occl}:={\mathcal{C}}_6^{occl}(f,{\mathcal{C}}_{}^{},{{}^{{}_{}}\mathit{r}}_s^{fix},K).$$

(29)

Contrary to all other viewpoint constraint formulations (see Equation (11)), it must be assumed that the occlusion space is not a subset of the $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_6^{occl}⊈{\mathcal{C}}_{}^{}$. Hence, let Equation (12) be reformulated as the set difference of the $\mathcal{C}\text{-}\mathrm{space}$s of other viewpoint constraints and the occlusion space:

$$\mathcal{C}=\left(\bigcap _{i=1,i\ne 6}^j{\mathcal{C}}_i\right)\setminus {\mathcal{C}}_6^{occl}.$$

(30)

If the resulting constrained space results in being a non-empty set, ${\mathcal{C}}_{}^{}\ne \varnothing$, there exists at least one valid sensor pose for the selected sensor orientation with occlusion-free visibility to the feature f.

4.5.2. Characterization

The present work proposes a strategy to compute ${\mathcal{C}}_6^{occl}$, which is broken down into the following general steps. In the first step, the smallest possible number of view rays is computed for detecting potential occlusions. In the second step, by means of ray-casting techniques, view rays are tested for occlusion against all rigid bodies of the RVS. For the interested reader, we refer to [58,59] for a comprehensive overview of ray-casting techniques. Then, the occlusion space is characterized using a simple surface reconstruction method using the colliding points of the rigid bodies and some further auxiliary points. In the last step, the occlusion space is integrated with the $\mathcal{C}\text{-}\mathrm{space}$ spanned by other viewpoint constraints, as given by Equation (30).

Algorithm A2 describes more comprehensively all the steps to characterize the occlusion space ${\mathcal{C}}_6^{occl}$, and Figure 15 provides an overview of the workflow and visualization of the expected results of the most significant steps, considering an exemplary occluding object ${\kappa }_1$. A more comprehensive description of the characterization of the view rays while using the $\mathcal{C}\text{-}\mathrm{space}$ characterized by other viewpoint constraints can be found in the Table A6.

4.5.3. Verification

For verification purposes, we consider an academic example, which comprises an icosahedron occluding the sight of feature $f_1$. The dimensions and location of the occluding object ${\kappa }_1$ are described in Table A7. Figure A6 displays the related scene and the manifolds of the computed occlusion space and corresponding occlusion-free space. In the first step, the $\mathcal{C}\text{-}\mathrm{space}$ for feature $f_1$, considering its geometry, i.e., ${\mathcal{C}}_3^{}$, and the following sensor orientation ${{}^{f_1}\mathit{r}}_s^{}({\alpha }_s^z={\gamma }_s^x=0,{\beta }_s^y=15{}^{\circ })$ was characterized. In the second step, the manifold of the occlusion space, ${\mathcal{C}}_6^{occl}$, was synthesized following the steps described in Algorithm A2. A discretization step of $d^{\mathsf{\varsigma }}=0.5{}^{\circ }$ was selected for computing the view rays ${\mathsf{\varsigma }}_{g_{f,c}}^{}$.

Figure 16 shows the rendered point cloud and range image at one extreme viewpoint within the resulting occlusion-free space. The rendered point cloud and image confirm that although the occluding body lies within the frustum space of the viewpoint, the feature and its entire geometry can still be completely captured. As expected, the computation of the collision points using ray casting was the most computationally expensive step with a total time of $t({\mathit{q}}_{f,c}^{occl,\kappa })=0.7$ s, and the total time required for characterizing the occlusion-free space corresponded to $t({\mathcal{C}}_6^{occl,\kappa })=1.32$ s.

4.5.4. Summary

Within this subsection, a strategy that combines ray-casting and CSG Boolean techniques to compute an occlusion space ${\mathcal{C}}_6^{occl}$ was introduced. The present study showed that the ${\mathcal{C}}_6^{occl}$ manifold can be thoroughly integrated with the $\mathcal{C}\text{-}\mathrm{space}$ spanned by other viewpoint constraints, complying with the framework proposed within this publication. Moreover, to enhance the efficiency of the proposed strategy, we considered a simplification of the feature geometry, discretization of the viewpoint space, and the use of a previous computed $\mathcal{C}\text{-}\mathrm{space}$.

Due to the aforementioned simplifications, it should be kept in mind that, contrary to the other $\mathcal{C}\text{-}\mathrm{space}$s, the ${\mathcal{C}}_6^{occl}$ manifold does not represent the explicit and real occlusion space and should be treated as an approximation of it. For example, a significant source of error regarding the accurate identification of all occluding points is the chosen discretizing step size for the computations of the view rays. This effect is known within ray-casting applications and can also be observed in Figure 15, where the right corner point of the colliding body is missed. Within the context of the present study, comprising robot systems, we assume that such simplifications can be safely taken into consideration if the absolute position accuracy of the robot is considered for the selection of the step size $d^{\mathsf{\varsigma }}$. For example, assuming a conservative absolute accuracy of a robot of $1\mathrm{mm}$ and a minimum working distance of $200\mathrm{mm}$, it is reasonable to choose a step size of $d^{\mathsf{\varsigma }}\approx 0.3{}^{\circ }$, using the arc length formula $(1\mathrm{mm}/200\mathrm{mm})=0.005$ rad. Alternatively, more robust solutions can be achieved by scaling the occlusion space with a safety factor.

Moreover, special attention must be given when computing the occlusion space as suggested in Step 5 of Algorithm A2 for rigid bodies with hollow cavities, e.g., a torus. It should be expected that the result of the occlusion space will be more conservative. A more precise characterization of the occlusion space falls outside the scope of this paper and could be achieved using more sophisticated surface reconstruction algorithms [60,61].

Finally, it is important to mention that the characterization of the occlusion space may lead to a non-watertight manifold, which may complicate the further processing of the jointed $\mathcal{C}\text{-}\mathrm{space}$. Thus, we recommend computing the occlusion space as the last viewpoint constraint.

4.6. Multisensor

Considering the intrinsic nature of a range sensor, in its minimal configuration, two imaging devices (two cameras or one camera and one active projector) are necessary to acquire a range image. Therefore, a range sensor can be regarded as a multisensor system. Up to this point, it had been assumed that the $\mathcal{C}\text{-}\mathrm{space}$ of a range sensor could be characterized using just one frustum space ${\mathcal{I}}_s^{}$ and that the resulting $\mathcal{C}\text{-}\mathrm{space}$ integrates the imaging parameters and any other viewpoint constraints of all imaging devices of a range sensor. On one hand, some of the previous introduced formulations and most of the related work demonstrated that this simplification is in many cases sufficient for computing valid viewpoints. On the other hand, this assumption could also be regarded as restrictive and invalid for some viewpoint constraints. For example, the characterization of the occlusion-free space as described in Section 4.5 will not guarantee free sight to both imaging devices of a range sensor.

For this reason, our study assumes that each imaging device can have individual and independent viewpoint constraints. As a result, we consider that an individual $\mathcal{C}\text{-}\mathrm{space}$ can be spanned for each imaging device. Furthermore, this section outlines a generic strategy to merge the individual $\mathcal{C}\text{-}\mathrm{space}$s of multiple imaging devices to span a common $\mathcal{C}\text{-}\mathrm{space}$ that satisfies all viewpoint constraints simultaneously.

To the best of our knowledge, none of the related work has considered viewpoint constraints for the individual imaging devices of a range sensor or for multisensor systems. In Section 5, the scalability and generality of our approach while considering a multisensor system is demonstrated with two different range sensors.

4.6.1. Formulation

Our formulation is based on the idea that each imaging device can be modeled independently and that all devices must simultaneously fulfill all viewpoint constraints. First, considering the most straightforward configuration of a sensor with a set of two imaging devices $\{s_1,s_2\}\in \tilde{S}$, we can assume that we have two different sensors with two different frustum spaces ${\mathcal{I}}_{s_1}^{}={\mathcal{I}}_0({{}_{s_1}\mathit{p}}_s^{},I_{s_1})$ and ${\mathcal{I}}_{s_2}^{}={\mathcal{I}}_0({{}_{s_2}\mathit{p}}_s^{},I_{s_2})$ (cf. Section 2.5). Thus, aligned to the formulation from Section 4.1, the core $\mathcal{C}\text{-}\mathrm{space}$ for each sensor can be expressed as follows:

$${\mathcal{C}}_1^{s_1}({{}_{s_1}\mathit{r}}_s^{fix},{\mathcal{I}}_{s_1}^{},B_f)\mathrm{and}{\mathcal{C}}_1^{s_2}({{}_{s_2}\mathit{r}}_s^{fix},{\mathcal{I}}_{s_2}^{},B_f).$$

In a more generic definition that considers all viewpoint constraints of $s_1$ or $s_2$, the $\mathcal{C}\text{-}\mathrm{space}$ is denoted as ${\mathcal{C}}_{}^{s_1}:={\mathcal{C}}_1^{s_1}(f,{\tilde{C}}^{s_1})$ or ${\mathcal{C}}_{}^{s_2}:={\mathcal{C}}_1^{s_2}(f,{\tilde{C}}^{s_2})$.

Considering that the rigid transformation ${{}_{s_2}{}^{s_1}\mathit{T}}_s^{}$ between the two imaging devices of a sensor is known, it can be assumed that a valid sensor pose for the first imaging device exists, if and only if the second imaging device also lies within in its corresponding $\mathcal{C}\text{-}\mathrm{space}$ simultaneously. The formulation of such condition follows

$${{}_{s_1}\mathit{p}}_s^{}\in {\mathcal{C}}_{}^{s_1}\iff {{}_{s_2}\mathit{p}}_s^{}({{}_{s_1}{}^{s_2}\mathit{T}}_s^{},{\mathit{p}}_{s_1}^{})\in {\mathcal{C}}_{}^{s_2},$$

(31)

supposing that the reference positioning frame for the sensor is the first imaging device $s_1$.

If the condition from Equation (31) is valid, there must exist a $\mathcal{C}\text{-}\mathrm{space}$ for $s_1$ that integrates the viewpoint constraints of both lenses being denoted as ${\mathcal{C}}_7^{s_{1.2}}$, which formulation follows:

$$\begin{array}{cc}\hfill {\mathcal{C}}_7^{s_{1,2}}={\mathcal{C}}_7^{s_1}({\mathcal{C}}_{}^{s_2})=& \{{{}_{s_1}\mathit{p}}_s^{}\in {\mathcal{C}}_7^{s_{1,2}}\hfill \\ & |{{}_{s_1}\mathit{r}}_s^{fix}\in {\mathcal{R}}_s^f,{{}_{s_2}\mathit{p}}_s^{}({{}_{s_1}{}^{s_2}\mathit{T}}_s^{},{{}_{s_2}\mathit{p}}_s^{})\in {\mathcal{C}}_{}^{s_2}\}.\hfill \end{array}$$

The joint space can then be integrated with the $\mathcal{C}\text{-}\mathrm{space}$ for $s_1$ employing a Boolean intersection:

$${\mathcal{C}}_{}^{s_{1,2}}={\mathcal{C}}_{}^{s_1}\bigcap ^{}{\mathcal{C}}_7^{s_{1,2}}.$$

A more generic formulation of the $\mathcal{C}\text{-}\mathrm{space}$ of $s_1$ being constrained by all imaging devices $s_t\in \tilde{S}$ is given:

$${\mathcal{C}}_{}^{{\tilde{S}}_1}={\mathcal{C}}_{}^{s_1}\bigcap _{s_t\in \tilde{S}}{\mathcal{C}}_7^{s_1,s_t}.$$

Analogously, ${\mathcal{C}}_{}^{{\tilde{S}}_2}$ denotes the space for positioning the imaging device $s_2$ including the constraints of $s_1$.

4.6.2. Characterization

The $\mathcal{C}\text{-}\mathrm{space}$ ${\mathcal{C}}_{}^{{\tilde{S}}_1}$, which comprises all constraints of all imaging devices $s_t\in \tilde{S}$, can be straightforwardly characterized following the five simple steps given in Algorithm A3. Figure 17 visualizes the interim manifolds at each step to ultimately characterize the manifold ${\mathcal{C}}_{}^{{\tilde{S}}_1}$. Finally, Figure 17f shows an exemplary extreme viewpoint, where both imaging devices frames are within their respective $\mathcal{C}\text{-}\mathrm{space}$s ${{}_{s_1}\mathit{p}}_s^{}\in {\mathcal{C}}_{}^{{\tilde{S}}_1}$ and ${{}_{s_2}\mathit{p}}_s^{}\in {\mathcal{C}}_{}^{{\tilde{S}}_2}$ and the feature geometry lies within both frustum spaces.

The space ${\mathcal{C}}_{}^{{\tilde{S}}_2}$ for the second imaging device can be computed analogously following the same steps. However, the topology of the resulting ${\mathcal{C}}_{}^{{\tilde{S}}_2}$ will be identical to ${\mathcal{C}}_{}^{{\tilde{S}}_1}$. Hence, instead of repeating the steps described in Algorithm A3 for a second imaging device, a more efficient alternative is to translate the manifold of ${\mathcal{C}}_{}^{{\tilde{S}}_1}$ to the position of $s_2$ at ${{}_{s_2}\mathit{p}}_s^{}({{}_{s_2}\mathit{t}}_s^{}=B_f)$ using the rigid translation

$${\mathcal{C}}_{}^{{\tilde{S}}_2}=\mathit{translation}({\mathcal{C}}_{}^{{\tilde{S}}_1},{{}_{s_1}{}^{s_2}\mathit{T}}_s^{}).$$

Note that if two imaging devices are the same orientation, the resulting $\mathcal{C}\text{-}\mathrm{space}$s are identical, hence ${\mathcal{C}}_{}^{{\tilde{S}}_1}={\mathcal{C}}_{}^{{\tilde{S}}_2}$.

4.6.3. Verification

The joint constrained spaces ${\mathcal{C}}_{}^{{\tilde{S}}_1}$ and ${\mathcal{C}}_{}^{{\tilde{S}}_2}$ for $s_1$ and $s_2$ were computed according to the steps provided in Algorithm 1 for acquiring feature $f_1$. The imaging parameters of both sensors and rigid transformation between them are given in Table A6.

First, the $\mathcal{C}\text{-}\mathrm{space}$s of both sensors were spanned considering their imaging parameters and a null orientation of the first sensor, i.e., ${{}^f\mathit{r}}_{s,1}^{}({\alpha }_s^z={\beta }_s^y={\gamma }_s^x=0{}^{\circ })$. The individual constrained spaces were computed for each sensor while considering the constrained space affected by the feature geometry, i.e., ${\mathcal{C}}_3^{s_1}$ and ${\mathcal{C}}_3^{s_2}$. Since the frustum space of the second sensor always lies within the first one, we additionally considered a fictitious accuracy constraint for the depth of the second sensor, ${\mathcal{C}}_5^{s_2}\left(a_{s_2}(z_{min}=500\mathrm{mm},z_{max}=700\mathrm{mm})\right)$, to limit its working distance.

Figure 18 visualizes the described scene and resulting manifolds of the constrained spaces. Figure 19 displays the frustum spaces, rendered depth images of both sensors, and the resulting point cloud at an extreme viewpoint. The rendered images provide a visual verification of our approach, demonstrating that $f_1$ is visible from both sensors. Note that the second device represents an active structured light projector (see in Table A6).

The total computation time of the constrained space was estimated at 200 ms. The computation time depended mainly on the intersection from Step 5, which corresponded to 100 ms. The computation results only apply to this case. Such analyses are difficult to generalize since the complexity of Boolean operations depends decisively on the number of vertices of the manifolds of the occluding objects.

4.6.4. Summary

This subsection introduced the formulation and characterization of a $\mathcal{C}\text{-}\mathrm{space}$, which considers the intrinsic configuration of range sensors comprising at least two imaging devices. Our approach enables the combination of individual viewpoint constraints for each device and the characterization of a $\mathcal{C}\text{-}\mathrm{space}$, which fulfills simultaneously all viewpoint constraints from all imaging devices.

The strategy proposed in Algorithm A3 was demonstrated to be valid and efficient in characterizing the manifolds of such a $\mathcal{C}\text{-}\mathrm{space}$. Nevertheless, we do not dispose alternative approaches for its characterization. For instance, if the frustum spaces are intersected initially, the resulting frustum space can be used as the base for spanning the rest of the constraints. However, we consider the steps proposed within this subsection more traceable, modular, and extendable to consider further constraints, multisensor systems, or even transferable to similar problems. For example, a variation of the algorithm could be applied to maximize or guarantee the registration space between two different viewpoints, which represents a fundamental challenge within many vision applications [62].

4.7. Robot Workspace

This section outlines the formulation of the robot workspace as a further viewpoint constraint to be seamlessly and consistently integrated with the other $\mathcal{C}\text{-}\mathrm{space}$s.

4.7.1. Formulation

A viewpoint can be considered valid if a sensor pose is reachable by the robot; hence, it lies within the robot workspace ${\mathit{p}}_s^{}\in {\mathcal{W}}_r$. The formulation of a constraint can then be straightforwardly formulated as follows:

$${\mathcal{C}}_8^{}={\mathcal{W}}_r=\{{\mathit{p}}_s^{}\in {\mathcal{C}}_{,8}^{}\mid {\mathit{p}}_s^{}\in {\mathcal{W}}_r\}.$$

(32)

4.7.2. Characterization and Verification

In our work, we assume that the robot workspace is known and can be characterized by a manifold in the special Euclidean group $SE\left(3\right)$. Considering this assumption, the constrained space ${\mathcal{C}}_8^{}$ can be seamlessly intersected with the rest of the viewpoint constraints. Figure 20 shows an exemplary scene for acquiring feature $f_1$ and the resulting constrained manifold ${\mathcal{C}}_3^{}\bigcap ^{}{\mathcal{C}}_8^{}$, which considers a robot with a workspace of a half-sphere and a working distance of $1000\mathrm{mm}$–$1800\mathrm{mm}$ and the $\mathcal{C}\text{-}\mathrm{space}$ manifold ${\mathcal{C}}_3^{}$ spanned by the feature geometry.

4.7.3. Summary

A more comprehensive formulation and characterization of the robot workspace to consider singularities requires more detailed modeling of the robot kinematics. Furthermore, our study has not considered the explicit characterization of the collision-free space of a robot, which had been the focus of exhaustive research in the last three decades. We assume that an explicit proof for collision must be performed in the last step for a selected sensor pose within the $\mathcal{C}\text{-}\mathrm{space}$. Nevertheless, we consider that our approach contributes substantially to a significant problem simplification by delimiting the search space to compute collision-free robot joint configurations more efficiently.

4.8. Multi-Feature Spaces

Up to this point, our work has outlined the formulation and characterization of $\mathcal{C}\text{-}\mathrm{space}$s to acquire just one feature. Within this subsection we briefly outline the characterization of a $\mathcal{C}\text{-}\mathrm{space}$,${{}^F\mathcal{C}}_{}^{}$, which allows for the capture of a set of features F and the simultaneous fulfillment of all viewpoint constraints from all features $f_m\in F$ with $m=1,\dots ,n$.

4.8.1. Characterization

The characterization of ${{}^F\mathcal{C}}_{}^{}$ can be seamlessly achieved according to the two steps described in Algorithm A4. In the first step, the $\mathcal{C}\text{-}\mathrm{space}$ for all n features, ${{}^{f_m}\mathcal{C}}_{}^{}$, are characterized considering a fixed sensor orientation ${{}^{{}_{}}\mathit{r}}_s^{fix}$ and the individual constraints $\tilde{C}\left(f_m\right)$. Then, the constrained space ${{}^F\mathcal{C}}_{}^{}$ is synthesized by intersecting all individual constrained spaces. Figure 21 shows the characterization of such a space for the acquisition of two features.

4.8.2. Verification

To verify the proposed characterization of a constrained space for acquiring multiple features, we outlined an exemplary use case comprising two features $\{f_1,f_{2,}\}\in F$ (see Table A7). We computed the space, ${{}^F\mathcal{C}}_{}^{}$, according to the steps of Algorithm A4, considering an orientation of ${{}^{f_1}\mathit{r}}_s^{}({\alpha }_s^z={\beta }_s^y=0{}^{\circ },{\gamma }_s^x=-10{}^{\circ },)$ relative to the feature $f_1$. Figure A7 shows the described scene and visualizes the resulting joint space. The rendered scene and range images of Figure 22 confirm the validity of the ${{}^F\mathcal{C}}_{}^{}$, demonstrating that both features can be simultaneously acquired at two extreme viewpoints within this space.

4.8.3. Summary

Within this subsection, we demonstrated that $\mathcal{C}\text{-}\mathrm{space}$s from different features can be seamlessly combined to span a topological space that guarantees the acquisition of these features, simultaneously satisfying the individual feature viewpoint constraints.

The current study assumes that the sensor orientation can be arbitrarily chosen and that the features can be acquired jointly by the sensor. In most applications, such assumptions cannot always be met and the following fundamental questions arise: which features can be acquired simultaneously and which is an adequate sensor orientation? These questions fall outside the scope of this paper and yield the motivation of our ongoing research, which addresses the efficient combination of $\mathcal{C}\text{-}\mathrm{space}$s to tackle the superordinated VPP.

4.9. Constraints Integration Strategy

The integration of viewpoint constraints can be considered to be commutative, i.e., the order of computation and integration of the constraints do not affect the characterization of the final constrained space. However, due to the diverse computation techniques that our our framework considers, a well-thought-out strategy may contribute towards increasing the computational efficiency of the overall process. In this publication, we outline one possible and simple strategy described in Algorithm A5 to integrate all viewpoint constraints into a single $\mathcal{C}\text{-}\mathrm{space}$.

The optimal integration of constraints falls outside the scope of this publication. Moreover, we consider that an optimal and efficient strategy can be tailored just by considering the individual application and its specific constraints.

5. Results

Within this section, a comprehensive evaluation of the constraint formulations and their integration is undertaken. First, Section 5.2 verifies the formulations of all regarded viewpoint constraints of our work based on an academic example. In Section 5.3, the framework’s broad generality and applicability is evaluated within an industrial RVS comprising two different sensors.

5.1. Technical Setup

This section provides an overview of the hardware and software used for the characterization of the $\mathcal{C}\text{-}\mathrm{space}$s within the following sections. We briefly introduce the considered domains, parameters, and specifications that were employed to verify the individual formulations of $\mathcal{C}\text{-}\mathrm{space}$s presented in Section 4.

5.1.1. Domain Models

• Sensors: We used two different range sensors for the individual verification of the $\mathcal{C}\text{-}\mathrm{space}$s and the simulation-based and experimental analyses. The imaging parameters (cf. Section 2.5) and kinematic relations of both sensors are given in Table A6. The parameters of the lighting source of the ZEISS Comet PRO AE sensor are conservatively estimated values, which guarantee that the frustum of the sensor lies completely within the field of view of the fringe projector. A more comprehensive description of the hardware is provided in Section 5.
• Object, features, and occlusion bodies: For verification purposes, we designed an academic object comprising three features and two occlusion objects with the characteristics given in Table A7 in the Appendix A.
• Robot: We used a Fanuc M-20ia six-axis industrial robot and respective kinematic model to compute the final viewpoints to position the sensor.

5.1.2. Software

The backbone of our framework was developed based on the Robot Operating System (ROS) (Distribution: Noetic Ninjemys) [63]. The framework was built upon a knowledge-based, service-oriented architecture. A more detailed overview of the general conceptualization of the architecture and knowledge-base is provided in our previous works [64,65].

Most of our algorithms consist of generating and manipulating 3D manifolds. Hence, based on empirical studies, we evaluated different open-source Python 3 libraries and used them according to their best performance for diverse computation tasks. For example, the PyMesh Library from [66] demonstrated the best computational performance for Boolean operations. On the contrary, we used the trimesh Library [57] for verification purposes and for performing ray-casting operations, due to its integration of the performance-oriented Embree Library [67]. Additionally, for further verification, visualization, and user interaction purposes, we coupled the ROS kinematic simulation to Unity [68] using the ROS# library [69].

All operations were performed on a portable workstation Lenovo W530 running Ubuntu 20.04 with the following specifications: Processor Intel Core i7-4810MQ @2.80 GHz, GPU Nvidia 3000 KM, and 32 GB Ram.

5.2. Academic Simulation-Based Analysis

This subsection presents a simple but thorough academic use case that considers all introduced viewpoint constraints to perform an exhaustive evaluation of the presented formulations. As stated in the introduction, with this present scenario we aim to provide a first draft of a much-needed benchmark for other researchers that can be used as basis for further development, reproducibility, and comparison. The surface models, the resulting manifolds of the computed $\mathcal{C}\text{-}\mathrm{space}$s, and the frustum spaces can be found attached in the additional material of our publication.

5.2.1. Use Case Description

The exemplary case regards an RVS with two sensors and an object of interest containing three different features with different sizes and geometries.

Table A8. Overview and description of the viewpoint constraints considered for the simulation-based analysis.

Viewpoint Constraint	Description	Approach
1	Two sensors($s^1$, $s^2$) with two imaging devices each: $\{s_1^1,s_2^1,s_3^2,s_4^2\}\in \tilde{S}$. The imaging parameters of all devices are specified in Table A6.	Linear algebra and geometry
2	Relative orientation to the object’s frame: ${{}^o\mathit{r}}_s^{}({\alpha }_s^z={\gamma }_s^x=0,{\beta }_s^y=6.64{}^{\circ })$.	Linear algebra and geometry
3	A planar rectangular object with three different features $\{f_1,f_2,f_3\}\in F$ (see Table A7).	Linear algebra, geometry, and trigonometry
4–5	The workspace of the second imaging device is restricted in the z-axis to the following working distance $z_{s_2}>450\mathrm{mm}$.	Linear algebra and geometry
6	Two objects with the form of an icosahedron (${\kappa }_1$) and a octahedron (${\kappa }_2$) occlude the visibility of the features.	Linear algebra, ray-casting, and CSG Boolean Operations
7	All constraints must be satisfied by all four imaging devices simultaneously.	Linear algebra and CSG Boolean Operations
8	Both sensors are attached to a six-axis industrial robot. The robot has a workspace of a half-sphere with a working distance of $1000\mathrm{mm}$–$1800\mathrm{mm}$.	CSG Boolean Operation
9	All features from the set G must be captured simultaneously.	CSG Boolean Operation

gives a detailed overview of the considered constraints. In addition to the imaging parameters of both range sensors, all other parameters can be assumed to be fictitious though realistic. The kinematic and imaging models corresponds to the real RVS, which is described in Section 5.3.1.

5.2.2. Results

Following the strategy described in Algorithm A5, the joint $\mathcal{C}\text{-}\mathrm{space}$s of the four imaging devices, i.e., ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_1}$, ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_2}$, ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_3}$, and ${{}^{F,}\mathcal{C}}_{}^{{\tilde{S}}_4}$, were computed for acquiring all features and considering all viewpoint constraints from Table A8. Figure 23 shows the complexity of the described case comprising three features and some of the resulting $\mathcal{C}\text{-}\mathrm{space}$s. The blue manifold ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_1}$ represents the final constrained space of the first imaging device $s_1$. It can be appreciated that ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_1}$ is characterized by the intersection of all other $\mathcal{C}\text{-}\mathrm{space}$s. Moreover, Figure 23 shows that the ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_1}$ manifold is mainly constrained by the $\mathcal{C}\text{-}\mathrm{space}$ corresponding to the second range sensor, i.e., ${{}^F\mathcal{C}}_7^{s_1,s_3,s_4}$.

To verify the validity of the computed $\mathcal{C}\text{-}\mathrm{space}$s, the depth images and point clouds for all imaging devices at eight extreme viewpoints were rendered. Figure 24 shows the corresponding rendered scene and resulting depth images for each imaging device at one extreme viewpoint ${{}_{s_1}\mathit{p}}_s^{}\in {{}^F\mathcal{C}}_{}^{{\tilde{S}}_1}$. The depth images demonstrate that all imaging devices can successfully acquire all features without occlusion simultaneously.

The total computation time for characterizing all $\mathcal{C}\text{-}\mathrm{space}$s corresponded to $t({{}^F\mathcal{C}}_{}^{{\tilde{S}}_1},{{}^F\mathcal{C}}_{}^{{\tilde{S}}_2},$ ${{}^F\mathcal{C}}_{}^{{\tilde{S}}_3},{{}^F\mathcal{C}}_{}^{{\tilde{S}}_4})\approx 50$ s. However, this time comprises other computation steps (e.g., frames transformation and inverse kinematic operations using ROS-Services) which distort the effective computation time of the $\mathcal{C}\text{-}\mathrm{space}$s. A proper analysis of the computational efficiency of the whole strategy remains to be further investigated.

5.2.3. Summary

Despite the complexity of the use case, the framework (models, methods, and integration strategy) presented within this paper demonstrated its effectiveness in characterizing a continuous topological space in the special Euclidean, where all defined viewpoint constraints could be fulfilled. The simulated depth images and point clouds confirmed that all selected viewpoints within the characterized $\mathcal{C}\text{-}\mathrm{space}$ satisfied all regarded constraints. Moreover, the proposed academic example effectively outlines a simple but sufficient complex scenario to benchmark our and future viewpoint planning strategies.

5.3. Real Experimental Analysis

To assess the usability and validity of our framework within real applications, the framework presented in this study was utilized to generate automatically valid viewpoints for capturing different features of a car door using real RVS, i.e., the AIBox from ZEISS. The AIBox is an industrial measurement cell used to automate different vision-based quality inspection tasks such as dimensional metrology and digitization, among others.

5.3.1. System Description

The AIBox is an integrated industrial RVS, equipped with a structured light sensor (ZEISS COMET PRO AE), a six-axis industrial robot (Fanuc M-20ia), and a rotary table for mounting an inspection object. Moreover, to evaluate the use of our approach considering a multisensor system, we additionally attached a stereo sensor (rc_visard 65, Roboception) to the structured light sensor. The imaging parameters of both sensors are given in Table A6. Figure 25 provides an overview of the reconfigured AIBox with the stereo sensor. We assume that the inspection object is roughly aligned, e.g., in [70] we presented a CNN fully trained on synthetic data to automate this task using the RVS sensor.

5.3.2. Vision Task Definition

The validation of our framework was performed on the basis of two vision tasks while considering diverse viewpoint constraints. For the first task, we considered just the ZEISS sensor’s ability to acquire the features $\{f_1,f_2\}\in F_1$, which lie on the outside of the door and can be potentially occluded by the door fixture. For the second task, we considered both sensors and the acquisition of two features $\{f_3,f_4,f_5\}\in F_2$ on the inside of the door. The incidence angle for the first case corresponded to a sensor orientation of ${{}^o\mathit{r}}_{s_2}^{}({\alpha }_s^z={\gamma }_s^x=0{}^{\circ },{\beta }_s^y=-15{}^{\circ })$ and for the second of ${{}^o\mathit{r}}_{s_1}^{}({\alpha }_s^z={\beta }_s^y={\gamma }_s^x=0{}^{\circ })$. To compensate for any kinematic modeling uncertainties, we consider an overall kinematic error of ${ϵ}_{s_1}^{x,y,z}=(70.0,70.0,50.0)\mathrm{mm}$ for $s_1$ and of ${ϵ}_{s_{3,4}}^{x,y,z}=(30.0,30.0,30.0)\mathrm{mm}$ for $s_3$ and $s_4$.

5.3.3. Results

For both vision tasks, we computed the necessary $\mathcal{C}\text{-}\mathrm{space}$s aligned to the strategy presented by Algorithm A5. The $\mathcal{C}\text{-}\mathrm{space}$s of the first inspection scenario for the camera ${{}^{F_1}\mathcal{C}}_{}^{{\tilde{S}}_1}$ and projector ${{}^{F_1}\mathcal{C}}_{}^{{\tilde{S}}_2}$ of the Comet Pro AE and its corresponding occlusion spaces are displayed on the left image of Figure 26. To assess the validity of the characterized $\mathcal{C}\text{-}\mathrm{space}$s, we chose diverse extreme viewpoints at the vertices of the ${{}^{F_1}\mathcal{C}}_{}^{{\tilde{S}}_1}$ manifold and performed real measurements. On the right side of Figure 26, the real monochrome images of the camera and the resulting point clouds at two validating viewpoints are displayed. The 2D images and point clouds prove that both features can be successfully acquired from both of these viewpoints, which confirms the free sight for the sensor and the illumination of both features without shadows.

Moreover, on the left of Figure 27, the constrained spaces of the first imaging device of each sensor, i.e., ${{}^{F_2}\mathcal{C}}_{}^{{\tilde{S}}_1}$ and ${{}^{F_2}\mathcal{C}}_{}^{{\tilde{S}}_3}$, are visualized for the second inspection scenario. Analogously to the first scenario, two extreme viewpoints at the vertices of the manifolds were selected to assess the validity of the computed $\mathcal{C}\text{-}\mathrm{space}$s. As expected, the real 2D images of all imaging devices and the resulting point clouds of both sensors at two exemplary extreme viewpoints (shown on the right of Figure 26) demonstrate that all features can be successfully acquired by the four imaging devices of both sensors.

5.3.4. Summary

Using an industrial RVS, and regarding real viewpoint constraints, we were able to validate the formulations, characterization, and application of $\mathcal{C}\text{-}\mathrm{space}$s for inspection tasks in an industrial context. These experiments show the suitability of our framework for an industrial application on a real RVS with multiple range sensors.

Furthermore, our strategy for merging individual $\mathcal{C}\text{-}\mathrm{space}$s to capture more than one feature proved to be effective for the vision tasks under consideration. However, a more complex task such as the inspection of all door features requires a more complex strategy, which considers the search of features that can be acquired together. This question recalls the overall VPP, which falls outside the scope of this publication and we intend to address in our future work.

6. Conclusions

6.1. Summary

The computation of valid viewpoints considering different system constraints, named VGP in this publication, is considered a complex and unsolved challenge that lacks a generic and holistic framework for its proper formulation and resolution. In this paper, we outline the VGP as a geometric problem that can be solved explicitly in the special Euclidean group $SE\left(3\right)$ using suitable and explicit models of all related domains of an RVS and viewpoint constraints. Within this context, much of our effort was devoted to the comprehensive and systematic formulation of the VGP and the exhaustive characterization of domains and viewpoint constraints aligned to the formulation of geometric problems.

The core result of this study is the characterization of $\mathcal{C}\text{-}\mathrm{space}$s, which can be understood as topological manifolds that span a space with infinite viewpoint solutions to acquire one feature or a group of features, while considering various viewpoint constraints and modeling uncertainties. Our approach focuses on providing rather infinite valid solutions instead of optimal ones. If the entire a priori knowledge of the RVS can be formalized and integrated into the $\mathcal{C}\text{-}\mathrm{space}$, then it we can assume that any viewpoint within it is a local optimum. Our work shows that a handful of viewpoint constraints can be efficiently and simply modeled geometrically and integrated in a common framework to span such constrained spaces. Finally, based on a comprehensive academic example and a real application, we demonstrate the usability of such a framework.

6.2. Limitations and Chances

We are aware that the framework proposed in the present study may have some limitations that may prevent its straightforward application for other RVS or use cases. First, it must be acknowledged that our framework can be classified under the category of model-based approaches. Therefore, in a first step, a priori information to model the components of the considered RVS must be examined. We consider that an exhaustive and explicit modeling of the necessary domains is required to deliver solutions that offer a higher generalization for other applications and systems. For the benefit of generalization, complexity reduction, and computational efficiency, we addressed various simplifications which could affect the accuracy of some models and might yield more conservative, though more robust, solutions.

We firmly believe that the VGP can be efficiently solved geometrically. We demonstrated that many constraints can be explicitly and efficiently characterized by combining several techniques, including linear algebra, trigonometry, and geometrical analysis. In the scope of our experiments, we confirmed that the computation of the $\mathcal{C}\text{-}\mathrm{space}$s manifolds based on these approaches ran efficiently in linear times. However, we also noted that algorithms comprising CSG Boolean operations are more computationally expensive, especially for calculations considering multiple Boolean operations on the same manifold. Although this limitation can be minimized by filtering and smoothing algorithms for decimating manifolds, this characteristic could still be considered insufficient for some users and applications. Although the shortcomings of CSG Boolean techniques regarding their computational efficiency have been mentioned in some prior works, we also believe that the present available computational performance and paralleling capabilities of CPUs and GPUs require a new reevaluation of their overall performance. Additionally, our work also suggests that combined with efficient imaging processing libraries, approaches requiring heavy use of CSG operations can be efficiently used within many applications. Nevertheless, a comprehensive computational efficiency analysis to find a break-even point between our approach and others remains to be further investigated.

Moreover, we also see room for improvement to increase the efficiency of some of the algorithms presented. For instance, the computation of the occlusion space and integration of constraints could also be improved using an alternative strategy and more efficient algorithms implemented in low-level programming languages. Additionally, we also see potential for improving the efficiency of some algorithms. For instance, the performance of many algorithms could enormously benefit of computational optimization techniques such as parallelization and GPU computation. For replication purposes of our work, we encourage the reader to make a thorough evaluation of the performance of the state-of-the-art libraries available at the present time, according to their application needs and system requirements.

6.3. Outlook

We consider the use of $\mathcal{C}\text{-}\mathrm{space}$s appropriate, but not limited to vision tasks that rely on features. For example, we showed how our concept could be extended to applications that generally would not consider features and demonstrate its application for an object detection problem with a certain level of spatial uncertainty. Our ongoing work concentrates on assessing further applications or systems that may benefit from our approach, e.g., feature-based robot calibration or adaption to laser sensors. Further studies should still be undertaken in this direction to verify the usability and explore the limitations of our framework within other applications and RVSs.

Recalling that we neglected any sensor parameters that may directly constrain the $\mathcal{C}\text{-}\mathrm{space}$, e.g., exposure time, gain, and others, we consider some other lines of research that integrate such a parameter space in the $\mathcal{V}\text{-}\mathrm{space}$. For instance, our ongoing study investigates the combination of a data-based approach to optimize the exposure times and the use of $\mathcal{C}\text{-}\mathrm{space}$s for finding optimized sensor poses.

The most promising future research should be devoted to the overall problem of the VPP, which could be reformulated based on the present study and its findings. Further research that will exploit this and comprises a holistic strategy for its resolution is already in progress.

We believe that our work will serve as a solid base and guideline for further studies to adapt and extend our framework according to the individual requirements of their concrete applications and RVS.