A study of deep single sketch-based modeling: View/style invariance, sparsity and latent space disentanglement

Highlights

• Sketch-based modeling robust to view and style change.

• A new formulation of the regression loss fitted to work with SDFs.

• Latent space disentanglement with regression loss increases reconstruction accuracy.

• Challenges inherent to sketch input in the context of deep-reconstruction methods.

• Auxiliary network learns to predict foreground mask supportting user sparse labels.

Abstract

Deep image-based modeling has received a lot of attention in recent years. Sketch-based modeling in particular has gained popularity given the ubiquitous nature of touchscreen devices. In this paper, we (i) study and compare diverse single-image reconstruction methods on sketch input, comparing the different 3D shape representations: multi-view, voxel- and point-cloud-based, mesh-based and implicit ones; and (ii) analyze the main challenges and requirements of sketch-based modeling systems. We introduce the regression loss and provide two variants of its formulation for the two most promising 3D shape representations: point clouds and signed distance functions. We show that this loss can increase general reconstruction accuracy, and the view- and style-robustness of the reconstruction methods. Moreover, we demonstrate that this loss can benefit the disentanglement of latent space to view-invariant and view-specific information, resulting in further improved performance. To address the figure-ground ambiguity typical for sparse freehand sketches, we propose a two-branch architecture that exploits sparse user labeling. We hope that our work will inform future research on sketch-based modeling.

Introduction

The challenge of being able to obtain a 3D model from a single sketch has intrigued researchers for decades. Typically, proposed methods make assumptions on the type of the input [1], [2] or restrict users to a specific user interface [3], [4], [5]. Being an under-constrained problem, for which it is hard to devise a reliable set of heuristics, it naturally asks for deep learning-based methods.

In light of the recent surge of image-based reconstruction, deep sketch-based modeling is gaining popularity [6], [7], [8], [9]. In this work, we study and compare the state-of-the-art deep single RGB image 3D shape reconstruction methods on sketch inputs. This allows us to identify the main challenges of working with sketch input: style variance between humans, imprecise perspective, and sparsity. We propose targeted solutions to increase the robustness of existing methods on a sketch input.

The first challenge comes from style differences, i.e., each person carries a unique sketching style. To address this problem, we use for training three synthetic datasets: naive, stylized, and one where the style is unified by an additional image processing network, as we proposed in the conference version of this paper [10]. The naive dataset represents rendering style with a uniform line width, commonly used in sketch-based reconstruction papers, where the lines are obtained from 2D images or via non-photorealistic rendering from 3D models. In this work, we rely on the latter. The stylized dataset is designed to capture the diversity of freehand sketching styles. The strategy of the style-unifying image translation network in the context of sketch-based modeling was first proposed in [6] and proved to be efficient on doodle sketches. We aim at more detailed sketches and show that if the sketching style is within an expected variance on line widths and over-sketching then training on the proposed stylized dataset results in more accurate reconstructions compared to a style-unifying network.

Second, we address the robustness of reconstruction methods with respect to viewpoint and style. It is common for deep single image methods to train and test their models on a predefined set of viewpoints [11]. Nevertheless, it was observed by Gryaditskaya et al. [12] that even professional designers, when asked to sketch from a given viewpoint, produce sketches with large angular deviations from the set viewpoint. Therefore, we create a dataset by generating for each shape 48 viewpoints, where 8 viewpoints are fixed and 5 additional viewpoints for each viewpoint are randomly sampled from a normal distribution with the mean matching the parameters of one of the fixed viewpoints. To explicitly account for the variation of styles and viewpoints, we aim at learning style- and viewpoint-invariant shape representation by proposing a regression loss that encourages the correlation of distances between 3D shapes and distances in a latent feature space. We extend [10] by proposing a new formulation of the regression loss, enabling the representation of 3D shapes as signed distance functions in the context of the regression loss. We show that this representation achieves the most accurate reconstruction results. We carefully study the choice of the parameter used in this loss, sampling and training strategies. Moreover, we show the advantage of this loss on the task of feature space disentanglement to view-invariant and view-specific components, allowing to further improve the accuracy of the reconstruction results.

The final challenge in deep sketch-based reconstruction comes from the difficulties in distinguishing the foreground from the background, due to the sparsity of sketch lines. To alleviate this problem we suggest a simple framework, where the user can provide a few sparse labels, and the network propagates these labels to robustly predict shape foreground binary mask, which is then passed to a 3D shape reconstruction network alongside the input sketch. We apply the strategy proposed in [10] and show its efficiency in the context of SDFs.

In summary, we propose the following contributions:

• We discuss the key challenges inherent to sketch input in the context of deep-reconstruction methods.

• We compare alternative strategies to handle freehand sketching styles variations.

• We adopt the regression loss to learn style- and viewpoint-invariant sketch embedding, and provide two formulations of the regression loss targeted two 3D shape representations: point clouds and signed distance functions.

• We study the effectiveness of the regression loss in neighborhood relations preservation between the reconstruction results, its style and view invariance properties.

• Finally, we propose to use an auxiliary network that learns to predict foreground mask from the input sketch and supports user sparse labels when necessary. We demonstrate how such mask can be incorporated as an input to a reconstruction network and allows to better account for the sparsity of input sketches.