General Virtual Sketching Framework for Vector Line Art
Haoran Mo1,  Edgar Simo-Serra2,  Chengying Gao*1,  Changqing Zou3,  Ruomei Wang1
1Sun Yat-sen University,  2Waseda University, 
3Huawei Technologies Canada

Accepted by ACM SIGGRAPH 2021
[Paper]  [Code]


Given clean line drawings, rough sketches or photographs of arbitrary resolution as input, our framework generates the corresponding vector line drawings directly. As shown in (b), the framework models a virtual pen surrounded by a dynamic window (red boxes), which moves while drawing the strokes. It learns to move around by scaling the window and sliding to an undrawn area for restarting the drawing (bottom example; sliding trajectory in blue arrow). With our proposed stroke regularization mechanism, the framework is able to enlarge the window and draw long strokes for simplicity (top example).
Abstract
Vector line art plays an important role in graphic design, however, it is tedious to manually create. We introduce a general framework to produce line drawings from a wide variety of images, by learning a mapping from raster image space to vector image space. Our approach is based on a recurrent neural network that draws the lines one by one. A differentiable rasterization module allows for training with only supervised raster data. We use a dynamic window around a virtual pen while drawing lines, implemented with a proposed aligned cropping and differentiable pasting modules. Furthermore, we develop a stroke regularization loss that encourages the model to use fewer and longer strokes to simplify the resulting vector image. Ablation studies and comparisons with existing methods corroborate the efficiency of our approach which is able to generate visually better results in less computation time, while generalizing better to a diversity of images and applications.
[Paper]    [Paper (ACM)]    [Supplementary]    [Code]    [All Precomputed Results]
Method

Framework Overview



Our framework generates the parametrized strokes step by step in a recurrent manner. It uses a dynamic window (dashed red boxes) around a virtual pen to draw the strokes, and can both move and change the size of the window. (a) Four main modules at each time step: aligned cropping, stroke generation, differentiable rendering and differentiable pasting. (b) Architecture of the stroke generation module. (c) Structural strokes predicted at each step; movement only is illustrated by blue arrows during which no stroke is drawn on the canvas.

Overall Introduction

(Or watch on Bilibili)
👇

Results
Our framework is applicable to a diversity of image types, such as clean line drawing images, rough sketches and photographs.

Vectorization


Rough sketch simplification


Photograph to line drawing


More Results

(Or watch on Bilibili)
👇


[Download Our Precomputed Output Results (7MB)]
Presentations

3-5 minute presentation

(Or watch on Bilibili)
👇


👉 15-20 minute presentation: [YouTube] [Bilibili]
👉 30-second fast forward: [YouTube] [Bilibili]
BibTeX
@article{mo2021virtualsketching,
    title   = {General Virtual Sketching Framework for Vector Line Art},
    author  = {Mo, Haoran and Simo-Serra, Edgar and Gao, Chengying and Zou, Changqing and Wang, Ruomei},
    journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH 2021)},
    year    = {2021},
    volume  = {40},
    number  = {4},
    pages   = {51:1--51:14}
}

Related Work
Jean-Dominique Favreau, Florent Lafarge and Adrien Bousseau. Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization. SIGGRAPH 2016. [Paper] [Webpage]

Mikhail Bessmeltsev and Justin Solomon. Vectorization of Line Drawings via PolyVector Fields. SIGGRAPH 2019. [Paper] [Code]

Edgar Simo-Serra, Satoshi Iizuka and Hiroshi Ishikawa. Mastering Sketching: Adversarial Augmentation for Structured Prediction. SIGGRAPH 2018. [Paper] [Webpage] [Code]

Zhewei Huang, Wen Heng and Shuchang Zhou. Learning to Paint With Model-based Deep Reinforcement Learning. ICCV 2019. [Paper] [Code]