Improved polarimetric analysis of human skin through stitching: advantages, limitations, and applications in dermatology

Mueller matrix polarimetry involves representing the polarization changing properties of a sample using a 4 × 4 matrix called the Mueller matrix $M$ [11].

$$\begin{eqnarray}&&{\left(\begin{array}{c}{S}_{1}\\ {S}_{2}\\ \begin{array}{c}{S}_{3}\\ {S}_{4}\end{array}\end{array}\right)}_{{out}}=\left[\begin{array}{cc}\begin{array}{cc}{M}_{1,1} & {M}_{1,2}\\ {M}_{2,1} & {M}_{2,2}\end{array} & \begin{array}{cc}{M}_{1,3} & {M}_{1,4}\\ {M}_{2,3} & {M}_{2,4}\end{array}\\ \begin{array}{cc}{M}_{3,1} & {M}_{3,2}\\ {M}_{4,1} & {M}_{4,2}\end{array} & \begin{array}{cc}{M}_{3,3} & {M}_{3,4}\\ {M}_{4,3} & {M}_{4,4}\end{array}\end{array}\right]{\left(\begin{array}{c}{S}_{1}\\ {S}_{2}\\ \begin{array}{c}{S}_{3}\\ {S}_{4}\end{array}\end{array}\right)}_{{in}}\end{eqnarray} \tag{ 1 }$$

Here, ${S}_{{out}}$ and ${S}_{{in}}$ denote the Stokes vectors of outgoing and incoming light, respectively. A Stokes vector, composed of four parameters characterizes the polarization state of a light wave. The Mueller matrix can be used to describe the effects of more complex polarization-modifying samples, such as those that introduce changes in phase or intensity of the light [12]. It can be used to describe a wide range of polarization phenomena, including depolarization power, retardance, diattenuation and polarizance [13]. Depolarization power refers to a mediums's ability to randomize the polarization state of light. Retardance is the phase difference created by a material between two orthogoanl polarization components of light. Diattenuation describes the polarization-dependet absoprtion of light. Finally, polarizance is the differential transmission of orthogonal polarization states through an optical element.

Figure 1 shows a schematic of an exemplary measurement with the employed polarimeter in diffuse reflection mode.

Zoom In Zoom Out Reset image size

For our experiments involving the 532 nm laser, we utilized a power of 1 mW. For those involving the 633 nm laser, the power was 10 mW. The dual-wavelength configuration facilitates the investigation of wavelength-dependent polarization characteristics of skin tissues, attributed to their differential penetration depths and scattering properties. For the current study, data was predominantly acquired using the 532 nm laser wavelength, chosen due to its efficacy in balancing polarimetric contrast in skin samples.

To perform a MMP measurement, a laser light source is used to illuminate the sample under study, and the polarization state of the transmitted or reflected light is measured [14]. Figure 2 illustrates the data acquisition process.

Zoom In Zoom Out Reset image size

Once the detector intensities for the individual polarization settings of the light have been measured, the Mueller matrix can be calculated using mathematical algorithms, which can be found in the literature [15] and figure 6. Equation (2) shows the exemplary calculation of ${{\rm{M}}}_{1,1}$ from the data acquistion illustrated in figure 2.

$$\begin{eqnarray}&&{{\rm{M}}}_{1,1}={\rm{HH}}+{\rm{HV}}+{\rm{VH}}+{\rm{VV}}\end{eqnarray} \tag{ 2 }$$

The elements of the Mueller matrix can then be analyzed to extract information about the polarization properties of the sample, such as its degree of linear or circular polarization, its diattenuation and retardance, and its degree of anisotropy [16]. The ${M}_{\mathrm{1,1}}$ entry is utilized for the normalization of all MM entries.

Our system's precision in polarization imaging is ensured by calibration using the Compain method [17] and regular tests with known optical samples. We have implemented various measures to address signal-to-noise improvement, including a controlled environment that minimizes ambient light and consistent calibration checks. Any deviations observed in the experimentally acquired MMs of samples with known polarization properties are within acceptable limits for our applications.

Image stitching is the process of combining multiple images into a single, larger image. It is a technique that is commonly used in photography and computer vision to create panoramic images or to extend the field of view [18].

Figure 3 displays the effect of the random patient movement (due to shaking, breathing, etc) on the data acquisition for the case of MMP.

Zoom In Zoom Out Reset image size

Patient motion moves the FoV for every acquisition and results in a small region of skin overlap. The complete specification of the Mueller matrix is only possible within the intersection of area 1 and area 2, shown in figure 3. In the residual regions, the description of the Mueller matrix remains partially incomplete, with only certain entries being fully defined. Figure 6 also shows the raw data requiring overlap for the computation of the MM entry. Strong patient motion could potentially result in insufficient overlap between these raw data images, impeding the calculation of the MM entry consequently.

The overlapping region is critical in our work with Mueller matrix polarimetry, particularly for aligning images of diverse samples like porcine skin, human skin from arms and fingers, and synthetically created sample data. Without a sufficient overlap, it becomes challenging to precisely correlate the same areas of the skin across the 36 images capturing similar, but not identical, skin regions. The overlap enables the algorithm to identify a substantial number of corresponding key points, which greatly aid in the accurate alignment of the images. In the absence of an overlap region, the process of manually finding matching pairs becomes a significant hurdle. This not only proves time-consuming but may also be practically impossible to achieve with high accuracy. Consequently, the overlap plays a key role in enhancing the precision and efficiency of image stitching in our study and facilitates the robust analysis of large skin patches. Figure 4 illustrates the two settings that can be utilized for data acquisition, particularly when trying to measure a larger field of view.

Zoom In Zoom Out Reset image size

For the data acquisition, either the patient or the system has to be moved. System movement might lead to a reduction in random patient movement as the skin region under study can be fixed slightly. Additionally, it could be integrated into total body scanning systems. While patient movement is the more cost-effective and simpler to execute option, it introduces complexity in terms of ensuring reproducibility and control.

There are several techniques that can be used for image stitching. Feature-based stitching involves detecting distinctive features in each image and using these traits to align and stitch the images [19]. Homography-based stitching methods use a mathematical model known as homography to transform and align the images, taking into consideration any perspective shifts that may have happened [20]. Direct techniques compute the best transformation directly from the images, without intermediate processes such as feature extraction or homography estimation [19]. The choice of the stitching approach will depend on the specific requirements and constraints of the application, such as the type and quality of the images, the desired level of accuracy, and the computational processing resources available [21].

In this study, we propose a robust feature-based image stitching technique for polarimetric data for in vivo MMP. Additionally, the method provides an alignment functionality. Further, the method is capable of automatic and manual polarimetric data stitching. Figure 5 illustrates the developed stitching process.

Zoom In Zoom Out Reset image size

The algorithm consists of the three main steps: stitching, aligning and cropping. Firstly, the raw data of at least two measurements is acquired. The algorithm will detect and extract the image features before matching them. With a sufficient number of matching feature pairs, the two images will be stitched. Otherwise, the operator can manually choose at least two matching pairs of features. This is continued for all 36 polarimetric input images recorded. A high number of matching feature pairs ensures high stitching quality. We integrate automatic feature-based registration with manual control point mapping. The feature extraction and description in this system are performed using KAZE [22], which has already been succesfully employed for the registration of in vivo polarimetric data [23]. Using this technique, feature detection identifies the regions of an image that have distinctive information, such as blobs [24]. The feature detection locates potential key points that are not always related to physical structures. It is important to identify features that are locally invariant so they can be recognized even in the presence of rotation or scaling [25].

The stitched images are aligned to one another in the second step. To align all stitched images, we employed a similar approach like the stitching method, using feature-based and control point mapping tools. The technique provides the option to select a stitched image as a reference, before aligning all other stitched images to that reference image.

The MM algorithm requires all stitched images to have the same size in order to calculate the MM after stitching and aligning the stitched images. To solve this problem, the algorithm can crop the stitched and aligned images to the area of interest chosen by the user, the maximum reference overlap zone of all images, or add zeros in the missing areas, which is the standard configuration.

Each full measurement cycle comprising the acquisition of 36 raw data images typically takes about 20 seconds. This duration can be subject to slight variations depending on the reflective properties of the particular skin sample being examined. On the other hand, patient motion, especially in a clinical setting for in vivo measurements, tends to occur on a much shorter timescale. Thus, despite the relatively brief duration of each measurement, the potential for patient movement during this period underscores the importance of stabilizing measures and careful timing to maintain the integrity of the data collection process. Figure 6 visualizes the data loss resulting from random patient motion.

Zoom In Zoom Out Reset image size

The colored patches represent a single image, e.g., the violet patch shows the HH image. For visualization purposes, we only show six of the 36 acquired images. The dashed line rectangle marks the skin area in which the MM entries ${{\rm{M}}}_{1,1},$ ${{\rm{M}}}_{1,2},$ ${{\rm{M}}}_{2,1}$ and ${{\rm{M}}}_{2,2}$ are defined, as the images HH, HV, VH and VV have a common overlap. The solid line rectangle shows the skin area in which the additional entries ${{\rm{M}}}_{3,1}$ and ${{\rm{M}}}_{3,2}$ are defined, i.e. where the images HH, HV, VH, VV, HP and VP have a common overlap. The MM can be calculated either from 16 or 36 images, where the latter improves the overall measurement accuracy. Stitching can counteract the reduction of the field of view which is reduced by random patient motion.

Blending is a commonly used technique in image stitching where the pixels of two or more images are combined to create a seamless view. We employ three specific feature-based stitching techniques and consider their accuracy, their efficiency, and their wide application in related fields [26, 27]. Firstly, compared to other techniques such as intensity-based methods, feature-based techniques tend to yield higher accuracy in feature matching, a crucial factor in our work. In fact, preliminary attempts using intensity-based methods produced less satisfactory results in terms of alignment precision [23]. Feature-based methods are also particularly fast and straightforward to implement [28]. This efficiency becomes an important consideration when dealing with numerous images that need to be processed in a reasonable timeframe.

The blending process can be accomplished in various ways, one being alpha blending [29]. In this research, three blending approaches are compared in the context of polarimetry.

A fundamental method for stitching images is called overwrite blending, which involves swapping out a specific area of one image for a comparable area of another image. According to the following description, the mathematical model for overwrite blending.

Suppose that, A and B stand for two images of the size M × N. In order to accomplish overwrite blending, we can build a binary matrix W of size M × N, where W(i,j) = 1 if pixel (i,j) in image A should be replaced by the matching pixel in image B, and W(i,j) = 0 otherwise. The final blended image, C, can be understood as by:

$$\begin{eqnarray}&&C(i,j)=A(i,j)* (1-W(i,j))+B(i,j)* W(i,j)\end{eqnarray} \tag{ 3 }$$

Depending on the value of W at that position, equation (3) calculates the blended pixel value at each location (i, j) as a weighted sum of the corresponding pixel values from A and B. The pixel value from B is applied if W(i, j) = 1; otherwise, if W(i, j) = 0, the pixel value from A is used.

A method of overlaying and blending, which is used to stitch images, involves layering one image on top of the other and modifying the opacity of the top image to achieve a transition that is smooth. The mathematical model for overlaying blending can now be described as follows:

Suppose again that, A and B are two images with dimensions M × N. The blending function F(x,y), which mixes the pixel value x from image A with the equivalent pixel value y from image B to create the final pixel value F, can be used to realize overlaying blending. A definition of the blending function is:

$$\begin{eqnarray}&&F(x,y)={ax}+{by}\end{eqnarray} \tag{ 4 }$$

where the blending coefficients a and b regulate the opacity of the two images. Typically, a + b = 1, so that the total opacity of the blended image is 1.

The process of mixing two or more images in a way that creates a smooth and seamless transition between them is known as seamless blending.

This variant of blending involves the application of a weight function. We can create a blending function, G(x,y), that takes the pixel value x from image A and interacts it with the equivalent pixel value y from image B to create the final pixel value G. This allows us to execute flawless blending. One definition of the blending function is:

$$\begin{eqnarray}&&G(x,y)=(1-w(x))* x+w(x)* y\end{eqnarray} \tag{ 5 }$$

where w(x) is a blending weight function that establishes exactly how the two images are blended. This method typically computes the blending weights by examining the gradients, colors, or frequencies of the overlapping portions of the images.

In figure 7, the working principles of the stitching methods are visualized.

Zoom In Zoom Out Reset image size

With the help of pixel visualization, figure 7 depicts the three different stitching techniques. It shows that overwrite blending uses either the pixel value from image A or image B for the fused image.

The pixel values in the overlapped regions are changed in overlaying blending and seamless blending, respectively, to produce a seamless transition between the images. This can be seen in figure 8 along with a comparison of stitching methods using raw data from pork skin.

Zoom In Zoom Out Reset image size

Figure 8 also shows that the stitching technique can have a strong influence on the relative distribution of the pixel intensities in the fused image.

Figure 9 presents the MMs derived from synthetically generated data. The synthetic dataset, created with MATLAB, consists of 36 distinct images. Each image features a polygon randomly positioned within a 500 × 500 pixel grid. Notably, the greyscale intensities of the foreground (the polygon) and the background exhibit random variations within a predetermined range.

Zoom In Zoom Out Reset image size

The findings from the stitching process applied to the synthetic raw Mueller matrix data, particularly under conditions involving motion, reveal an intriguing characteristic: the resultant field of view (FoV) differs across each Mueller matrix entry post-stitching and computation. This suggests that the areas determining each Mueller matrix entry are not universally applicable or consistent throughout. In contrast, upon examining the original images from which specific Mueller matrix images are computed, it becomes clear that the defining areas for each Mueller matrix entry are distinctly unique. This discrepancy underscores the complexity and variability inherent in the process of stitching polarimetric raw data.

Since overwrite blending does not modify the pixel values, it ensures consistency in the MM calculation across all images and avoids any potential inconsistencies that may arise from using different blending techniques. On the other hand, overlaying and seamless blending change the pixel values which affects the MM calculation and introduces an error in the analysis. By its inherent nature, overwriting does not have the potential to mitigate the error generated by noise. To demonstrate this, we create synthetic polarimetric raw data, which allows us to illustrate how various stitching techniques are capable of reducing the error stemming from random or speckle noise. The use of synthetic data thus provides an effective platform to investigate and compare the noise reduction capabilities of different stitching methodologies. Figure 10 illustrates the outcomes of the stitching process, specifically highlighting noise reduction in the context of synthetic raw data.

Zoom In Zoom Out Reset image size

In figure 10, we present the resulting entries of the Mueller matrices after the application of various stitching techniques. A visual examination of the data reveals a notable reduction in noise between the yellow marked Gaussian Noise data and the stitched data, contingent upon the employment of suitable stitching methodologies. This observation underlines the effectiveness of strategic stitching in attenuating noise interference, thus enhancing the reliability of the Mueller matrix outputs.

Figure 11 shows the standard deviation for all entries within the stitched Mueller matrices in the horizontal direction. This is demonstrated across different stitching techniques.

Zoom In Zoom Out Reset image size

Figure 11 presents the computed standard deviation for each entry in the horizontally-oriented stitched Mueller matrices across different stitching techniques. The black arrows show that the standard deviation is reduced by a half for all stitching methods indicating a noise reduction. The absense of significant variation in standard deviation values - as marked with the black ellipses - showcases the potential impact of the stitching process on the accuracy of the resultant matrices. However, no specific stitching technique seems to be particularly more effective than the others. This may be due to the following: In the process of image alignment, specifically during the geometric transformation and warping stages, the images may undergo subtle modifications. However, these alterations are not inherently aimed at blurring or denoising of the images. The perceived blurring or denoising might be a by-product of the transformation and warping process. Notably, during the warping phase, the images undergo interpolation, which, in many cases, might lead to a semblance of blurring. This effect appears to be more pronounced than the noise reduction associated with the stitching technique. The premise for this observation is that the overlapped region of the stitched images consistently retained its noise levels, irrespective of the stitching technique employed.

These observations suggest that the transformation and warping procedures inherent to image alignment might inadvertently induce some blurring effects, thereby reducing the noise levels in the stitched images. The blurring effect introduced by these procedures supersedes the potential noise reduction that could be achieved through the various stitching techniques, as indicated by the invariant noise present within the overlapped regions of the stitched images.

In figure 12, we present the raw data of HH images corresponding to both the synthetic and stitched data sets. The figure also includes the corresponding histograms for these two data sets. These histograms provide a detailed illustration of the pixel intensity distributions, further accentuating the contrasts and similarities between the synthetic and stitched data. The data representations facilitate the analysis of image transformation effects on the pixel distribution, thereby aiding in understanding the fundamental changes occurring due to the stitching process.

Zoom In Zoom Out Reset image size

The top-left panel shows the original synthetic data with notable noise levels. In contrast, the other panels illustrate the resultant image post-stitching, where the noise appears to be noticeably reduced, indicating the potential blurring effect introduced by the transformation process. The comparative analysis of these images visibly demonstrates the blurring that is inherent to the image transformation during the stitching procedure.

The potential for noise reduction in Mueller matrix polarimetry via image stitching could depend upon the noise level and noise type; further exploration in this realm could be beneficial in future studies. However, in the context of the current setup, this differentiation is not significant. Nevertheless, caution must be taken when implementing stitching techniques. The transformation processes involved in image stitching can inadvertently manipulate the Mueller matrix data, potentially introducing artefacts or biases. Therefore, the careful management and interpretation of these data transformations are crucial in the generation of accurate and meaningful polarimetric analyses.

In the controlled experimental setup, we used a printed symbol on paper as a phantom. The generation of phantom data involved the motion of the phantom during the acquisition of raw data. Subsequent image processing divided the image into two sections, labeled as the left and right portions. These portions were designed to share a significant overlap of 50%, meaning half of the content from each image could be located in its counterpart. This technique provides a reliable means of showing the alignment and image quality.

Figure 13 shows the exemplary raw data of the phantom (HH, HM, HP, HR and HV) and the corresponding stitching result. Additionally, the resulting MMs for the different stitching techniques are displayed.

Zoom In Zoom Out Reset image size

The stitching technique overwrite demonstrated excellent performance in recreating the accurate overall polarimetric picture. Nevertheless, the application of both alpha blending techniques—fused and seamless—can result in the generation of unwanted polarimetric artefacts. These could potentially distract dermatologists, particularly if Mueller Matrix Polarimetry (MMP) were to be implemented as an imaging technique in clinical practice, i.e. for in vivo measurements. The presence of these artefacts might inadvertently complicate the interpretation of images, thus potentially hindering the efficiency and accuracy of diagnostic procedures.

Firstly, we show that stitching is possible for a biological sample in a controlled setting without motion by stitching polarimetric data of a cold cut of chicken meat and gelatine.

Figure 14 shows the resulting MMs after stitching the raw data with three different techniques for the cold cut.

Zoom In Zoom Out Reset image size

Figure 14 illustrates the effectiveness of image stitching techniques in significantly extending the field of view, thereby enabling an integrated, complete visualization of the sample. This expanded view considerably aids in discerning and distinguishing the various components of the sample. It's important to note that without these stitching techniques, single measurements were unable to capture the entire cold cut slice, complicating the task of comparing and differentiating the various parts of the sample. Figure 15 on the other side displays a comparison of the MMs for ex vivo pork skin after stitching with three different stitching techniques.

Zoom In Zoom Out Reset image size

Despite the sample not displaying pronounced polarimetric contrast, it illustrates the potential for improperly chosen stitching techniques to introduce extraneous polarimetric features or artefacts. These unwanted intrusions can create unnecessary distractions and potentially confound accurate interpretation, thus emphasizing the importance of careful and precise execution of stitching techniques in polarimetric imaging.

Figure 16 shows the MM result after stitching of raw data obtained from an ex vivo scar on porcine skin.

Zoom In Zoom Out Reset image size

The scar on the ex vivo sample of porcine skin exhibits minimal polarimetric contrast. Only with the employment of the overwriting technique is it possible to ascertain the position of the scar. The other stitching procedures introduce artefacts, which render the scar indiscernible, thereby illustrating the critical role of appropriate imaging and stitching techniques in accurately capturing and representing skin features.

However, in the clinical setting, where patient motion can occur randomly, the algorithm's effectiveness can be affected. Thus, it is essential to investigate the algorithm's robustness to patient motion to evaluate its practicality in real-world scenarios. In the following we study the stitching of polarimetric data for an in vivo measurement of two human fingers to provide insight into the algorithm's capabilities in handling the negative impact of a relatively small degree of random patient motion. The motion is minimized by pressing the fingers onto the optical table during the measurement.

Zoom In Zoom Out Reset image size

Our findings indicate that, in the context of stitching polarimetric images, overwrite blending may prove more effective than both overlaying blending and seamless blending. We demonstrate this through a comparative analysis of Mueller matrices derived from stitched polarimetric data of two human fingers, processed through different stitching techniques. Both alpha blending techniques were observed to generate significant polarimetric artefacts, compromising the clarity of the resulting image. Remarkably, the individual fingers were discernible exclusively in the overwrite blending case. The significant artefact production associated with other blending techniques masks the true number of fingers present in the measurement, highlighting the superior performance of overwrite blending in this scenario.

Figure 18 presents the results of a polarimetric analysis following the implementation of a stitching process on a pattern drawn on human skin, specifically an arm. This pattern serves as an in vivo phantom, demonstrating the practical application and feasibility of the stitching method in a realistic context. The intricacy of the drawing provides an abundance of key points, facilitating the ease and efficiency of the stitching process. Additionally, figure 18 underscores the variable outcomes achieved when different stitching techniques are employed, showing that the choice of technique significantly influences the final result.

Zoom In Zoom Out Reset image size

The application of stitching to in vivo polarimetric data is carried out on a phantom for human skin. The application of the polarimetric stitching method significantly expands the field of view and allows for a comprehensive analysis of large skin patches, which was previously not possible due to the inherent limitations of traditional polarimetric setups. The results of the in vivo measurements show the algorithm's effectiveness in the clinical setting, where patient motion is inevitable. During our examination procedure, the initial measurement obtains a particular FoV. With each additional measurement, we incorporate an additional area of about 37% of the original FoV, as depicted in figure 18. Thus, the total FoV grows substantially in the course of the five measurements.

In figures 13–18, we demonstrate that the Mueller matrices calculated for the overwrite images show far fewer polarimetric artefacts than with other overlaying and seamless blending. Therefore, overwrite blended raw data is ideal for these types of MMP applications.