Yuki Koyama
Daisuke Sakamoto
Takeo Igarashi
ABSTRACT
Color enhancement is a very important aspect of photo editing. Even when photographers have tens of or hundreds of photographs, they must enhance each photo one by one by manually tweaking sliders in software such as brightness and contrast, because automatic color enhancement is not always satisfactory for them. To support this repetitive manual task, we present self-reinforcing color enhancement, where the system implicitly and progressively learns the user's preferences by training on their photo editing history. The more photos the user enhances, the more effectively the system supports the user. We present a working prototype system called SelPh, and then describe the algorithms used to perform the self-reinforcement. We conduct a user study to investigate how photographers would use a self-reinforcing system to enhance a collection of photos. The results indicate that the participants were satisfied with the proposed system and strongly agreed that the self-reinforcing approach is preferable to the traditional workflow.
Supplemental Material
- Adobe Systems Inc. 2015a. Photoshop Elements Help | Auto Smart Tone. (2015). Retrieved September 23, 2015 from https://helpx.adobe.com/ photoshop-elements/using/auto-smart-tone.html.Google Scholar
- Adobe Systems Inc. 2015b. Photoshop Help | Creating actions. (2015). Retrieved September 23, 2015 from https://helpx.adobe.com/photoshop/using/ creating-actions.html.Google Scholar
- Luca Benedetti, Holger Winnemöller, Massimiliano Corsini, and Roberto Scopigno. 2014. Painting with Bob: Assisted Creativity for Novices. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, NY, NY, USA, 419--428. DOI: http://dx.doi.org/10.1145/2642918.2647415 Google ScholarDigital Library
- Yoshua Bengio and Pascal Vincent. 2004. Locally Weighted Full Covariance Gaussian Density Estimation. Cirano working papers. CIRANO.Google Scholar
- Floraine Berthouzoz, Wilmot Li, Mira Dontcheva, and Maneesh Agrawala. 2011. A Framework for Content-adaptive Photo Manipulation Macros: Application to Face, Landscape, and Global Manipulations. ACM Trans. Graph. 30, 5, Article 120 (Oct. 2011), 14 pages. DOI: http://dx.doi.org/10.1145/2019627.2019639 Google ScholarDigital Library
- V. Bychkovsky, S. Paris, E. Chan, and F. Durand. 2011. Learning Photographic Global Tonal Adjustment with a Database of Input/Output Image Pairs. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '11). IEEE Computer Society, Washington, DC, USA, 97--104. DOI: http://dx.doi.org/10.1109/CVPR.2011.5995413 Google ScholarDigital Library
- J.C. Caicedo, A. Kapoor, and Sing Bing Kang. 2011. Collaborative personalization of image enhancement. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. 249--256. DOI: http://dx.doi.org/10.1109/CVPR.2011.5995439 Google ScholarDigital Library
- Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based Photo Recoloring. ACM Trans. Graph. 34, 4, Article 139 (July 2015), 11 pages. DOI: http://dx.doi.org/10.1145/2766978 Google ScholarDigital Library
- Fanny Chevalier, Pierre Dragicevic, and Christophe Hurter. 2012. Histomages: Fully Synchronized Views for Image Editing. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, NY, NY, USA, 281--286. DOI: http://dx.doi.org/10.1145/2380116.2380152 Google ScholarDigital Library
- Michael A. A. Cox and Trevor F. Cox. 2008. Multidimensional Scaling. In Handbook of Data Visualization. Springer Berlin Heidelberg, 315--347. DOI: http://dx.doi.org/10.1007/978-3-540-33037-0_14Google Scholar
- Allen Cypher. 1991. EAGER: Programming Repetitive Tasks by Example. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91). ACM, NY, NY, USA, 33--39. DOI: http://dx.doi.org/10.1145/108844.108850 Google ScholarDigital Library
- Ritendra Datta, Dhiraj Joshi, Jia Li, and James Z. Wang. 2006. Studying Aesthetics in Photographic Images Using a Computational Approach. In Proceedings of the 9th European Conference on Computer Vision - Volume Part III (ECCV'06). Springer-Verlag, Berlin, Heidelberg, 288--301. DOI: http://dx.doi.org/10.1007/11744078_23 Google ScholarDigital Library
- Leah Findlater and Jacob Wobbrock. 2012. Personalized Input: Improving Ten-finger Touchscreen Typing Through Automatic Adaptation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 815--824. DOI: http://dx.doi.org/10.1145/2207676.2208520 Google ScholarDigital Library
- Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the Design Space for Adaptive Graphical User Interfaces. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '06). ACM, NY, NY, USA, 201--208. DOI: http://dx.doi.org/10.1145/1133265.1133306 Google ScholarDigital Library
- Floraine Grabler, Maneesh Agrawala, Wilmot Li, Mira Dontcheva, and Takeo Igarashi. 2009. Generating Photo Manipulation Tutorials by Demonstration. ACM Trans. Graph. 28, 3, Article 66 (July 2009), 9 pages. DOI: http://dx.doi.org/10.1145/1531326.1531372 Google ScholarDigital Library
- Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid Dense Correspondence with Applications for Image Enhancement. ACM Trans. Graph. 30, 4, Article 70 (July 2011), 10 pages. DOI: http://dx.doi.org/10.1145/2010324.1964965 Google ScholarDigital Library
- Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2013. Optimizing Color Consistency in Photo Collections. ACM Trans. Graph. 32, 4, Article 38 (July 2013), 10 pages. DOI: http://dx.doi.org/10.1145/2461912.2461997 Google ScholarDigital Library
- R. Jaroensri, S. Paris, A. Hertzmann, V. Bychkovsky, and F. Durand. 2015. Predicting Range of Acceptable Photographic Tonal Adjustments. In Proceedings of the 2015 IEEE International Conference on Computational Photography (ICCP '15). IEEE Computer Society, Washington, DC, USA, 1--9. DOI: http://dx.doi.org/10.1109/ICCPHOT.2015.7168372Google ScholarCross Ref
- Steven G. Johnson. 2015. The NLopt nonlinear-optimization package. (2015). http://ab-initio.mit.edu/nloptGoogle Scholar
- Sing Bing Kang, A. Kapoor, and D. Lischinski. 2010. Personalization of image enhancement. In Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '10). IEEE Computer Society, Washington, DC, USA, 1799--1806. DOI: http://dx.doi.org/10.1109/CVPR.2010.5539850Google Scholar
- Ashish Kapoor, Juan C. Caicedo, Dani Lischinski, and Sing Bing Kang. 2014. Collaborative Personalization of Image Enhancement. Int. J. Comput. Vision 108, 1--2 (May 2014), 148--164. DOI: http://dx.doi.org/10.1007/s11263-013-0675--3 Google ScholarDigital Library
- Yan Ke, Xiaoou Tang, and Feng Jing. 2006. The Design of High-Level Features for Photo Quality Assessment. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1 (CVPR '06). IEEE Computer Society, Washington, DC, USA, 419--426. DOI: http://dx.doi.org/10.1109/CVPR.2006.303 Google ScholarDigital Library
- Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered Parameter Analysis for Visual Design Exploration. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, NY, NY, USA, 65--74. DOI: http://dx.doi.org/10.1145/2642918.2647386 Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25. Curran Associates, Inc., 1097--1105.Google ScholarDigital Library
- Brian Kulis. 2013. Metric Learning: A Survey. Foundations and Trends® in Machine Learning 5, 4 (2013), 287--364. DOI: http://dx.doi.org/10.1561/2200000019Google ScholarCross Ref
- Tessa Lau, Lawrence Bergman, Vittorio Castelli, and Daniel Oblinger. 2004. Sheepdog: Learning Procedures for Technical Support. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, NY, NY, USA, 109--116. DOI: http://dx.doi.org/10.1145/964442.964464 Google ScholarDigital Library
- Congcong Li, Alexander C. Loui, and Tsuhan Chen. 2010. Towards Aesthetics: A Photo Quality Assessment and Photo Selection System. In Proceedings of the International Conference on Multimedia (MM '10). ACM, NY, NY, USA, 827--830. DOI: http://dx.doi.org/10.1145/1873951.1874089 Google ScholarDigital Library
- Greg Little, Tessa A. Lau, Allen Cypher, James Lin, Eben M. Haber, and Eser Kandogan. 2007. Koala: Capture, Share, Automate, Personalize Business Processes on the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, NY, NY, USA, 943--946. DOI: http://dx.doi.org/10.1145/1240624.1240767 Google ScholarDigital Library
- D. C. Liu and J. Nocedal. 1989. On the Limited Memory BFGS Method for Large Scale Optimization. Math. Program. 45, 3 (Dec. 1989), 503--528. DOI: http://dx.doi.org/10.1007/BF01589116Google ScholarCross Ref
- Yiwen Luo and Xiaoou Tang. 2008. Photo and Video Quality Evaluation: Focusing on the Subject. In Proceedings of the 10th European Conference on Computer Vision: Part III (ECCV '08). Springer-Verlag, Berlin, Heidelberg, 386--399. DOI: http://dx.doi.org/10.1007/978-3-540-88690-7_29 Google ScholarDigital Library
- Luca Marchesotti, Florent Perronnin, Diane Larlus, and Gabriela Csurka. 2011. Assessing the Aesthetic Quality of Photographs Using Generic Image Descriptors. In Proceedings of the 2011 International Conference on Computer Vision (ICCV '11). IEEE Computer Society, Washington, DC, USA, 1784--1791. DOI: http://dx.doi.org/10.1109/ICCV.2011.6126444 Google ScholarDigital Library
- J. Marks, B. Andalman, P. A. Beardsley, W. Freeman, S. Gibson, J. Hodgins, T. Kang, B. Mirtich, H. Pfister, W. Ruml, K. Ryall, J. Seims, and S. Shieber. 1997. Design Galleries: A General Approach to Setting Parameters for Computer Graphics and Animation. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., NY, NY, USA, 389--400. DOI: http://dx.doi.org/10.1145/258734.258887 Google ScholarDigital Library
- Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2011. Color Compatibility from Large Datasets. ACM Trans. Graph. 30, 4, Article 63 (July 2011), 12 pages. DOI: http://dx.doi.org/10.1145/2010324.1964958 Google ScholarDigital Library
- Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning Layouts for Single-Page Graphic Designs. IEEE Transactions on Visualization and Computer Graphics 20, 8 (Aug. 2014), 1200--1213. DOI:http://dx.doi.org/10.1109/TVCG.2014.48 Google ScholarDigital Library
- Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 1221--1224. DOI: http://dx.doi.org/10.1145/2702123.2702149 Google ScholarDigital Library
- Katharina Reinecke, Tom Yeh, Luke Miratrix, Rahmatri Mardiko, Yuechen Zhao, Jenny Liu, and Krzysztof Z. Gajos. 2013. Predicting Users' First Impressions of Website Aesthetics with a Quantification of Perceived Visual Complexity and Colorfulness. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 2049--2058. DOI: http://dx.doi.org/10.1145/2470654.2481281 Google ScholarDigital Library
- Juliane Schäfer and Korbinian Strimmer. 2005. A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics. Statistical Applications in Genetics and Molecular Biology 4, 1 (November 2005). DOI: http://dx.doi.org/10.2202/1544--6115.1175Google ScholarCross Ref
- Adrian Secord, Jingwan Lu, Adam Finkelstein, Manish Singh, and Andrew Nealen. 2011. Perceptual Models of Viewpoint Preference. ACM Trans. Graph. 30, 5, Article 109 (2011), 12 pages. DOI: http://dx.doi.org/10.1145/2019627.2019628 Google ScholarDigital Library
- Burr Settles. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648. University of Wisconsin-Madison, Madison, WI, USA.Google Scholar
- L. Shapira, A. Shamir, and D. Cohen-Or. 2009. Image Appearance Exploration by Model-Based Navigation. Comput. Graph. Forum 28, 2 (2009), 629--638. DOI: http: //dx.doi.org/10.1111/j.1467--8659.2009.01403.xGoogle ScholarCross Ref
- Jerry O. Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory Modeling with Collaborative Design Spaces. ACM Trans. Graph. 28, 5, Article 167 (2009), 10 pages. DOI: http://dx.doi.org/10.1145/1618452.1618513 Google ScholarDigital Library
- Joshua B. Tenenbaum, Vin de Silva, and John C. Langford. 2000. A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 290, 5500 (2000), 2319--2323. DOI: http://dx.doi.org/10.1126/science.290.5500.2319Google ScholarCross Ref
- Michael Terry and Elizabeth D. Mynatt. 2002. Side Views: Persistent, On-demand Previews for Open-ended Tasks. In Proceedings of the 15th Annual ACM Symposium on User Interface Software and Technology (UIST '02). ACM, NY, NY, USA, 71--80. DOI: http://dx.doi.org/10.1145/571985.571996 Google ScholarDigital Library
Index Terms
- SelPh: Progressive Learning and Support of Manual Photo Color Enhancement
Recommendations
Sequential line search for efficient visual design optimization by crowds
Parameter tweaking is a common task in various design scenarios. For example, in color enhancement of photographs, designers tweak multiple parameters such as "brightness" and "contrast" to obtain the best visual impression. Adjusting one parameter is ...
Sequential gallery for interactive visual design optimization
Visual design tasks often involve tuning many design parameters. For example, color grading of a photograph involves many parameters, some of which non-expert users might be unfamiliar with. We propose a novel user-in-the-loop optimization method that ...
Crowd-powered parameter analysis for visual design exploration
UIST '14: Proceedings of the 27th annual ACM symposium on User interface software and technologyParameter tweaking is one of the fundamental tasks in the editing of visual digital contents, such as correcting photo color or executing blendshape facial expression control. A problem with parameter tweaking is that it often requires much time and ...
Comments