Carmine Vassallo
Harald C. Gall
ABSTRACT
An effective and efficient application of Continuous Integration (CI) and Delivery (CD) requires software projects to follow certain principles and good practices. Configuring such a CI/CD pipeline is challenging and error-prone. Therefore, automated linters have been proposed to detect errors in the pipeline. While existing linters identify syntactic errors, detect security vulnerabilities or misuse of the features provided by build servers, they do not support developers that want to prevent common misconfigurations of a CD pipeline that potentially violate CD principles (“CD smells”). To this end, we propose CD-Linter, a semantic linter that can automatically identify four different smells in pipeline configuration files. We have evaluated our approach through a large-scale and long-term study that consists of (i) monitoring 145 issues (opened in as many open-source projects) over a period of 6 months, (ii) manually validating the detection precision and recall on a representative sample of issues, and (iii) assessing the magnitude of the observed smells on 5,312 open-source projects on GitLab. Our results show that CD smells are accepted and fixed by most of the developers and our linter achieves a precision of 87% and a recall of 94%. Those smells can be frequently observed in the wild, as 31% of projects with long configurations are affected by at least one smell.
Supplemental Material
- Checkstyle Team. 2020. Checkstyle. Retrieved September 10, 2020 from http://checkstyle.sourceforge.netGoogle Scholar
- Coala Team. 2020. Coala-Linting and fixing for all languages. Retrieved September 10, 2020 from https://coala.io/Google Scholar
- Jacob Cohen. 1960. A Coeficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 ( 1960 ), 37-46.Google Scholar
- Cesar Couto, João Eduardo Montandon, Christofer Silva, and Marco Tulio Valente. 2011. Static correspondence and correlation between field defects and warnings reported by a bug finding tool. Software Quality Journal 21 ( 2011 ), 241-257.Google Scholar
- P.M. Duvall, S. Matyas, and A. Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk. Pearson Education.Google ScholarDigital Library
- FindBugs Team. 2020. FindBugs. Retrieved September 10, 2020 from http://findbugs.sourceforge.net/Google Scholar
- Forrester Team. 2019. The 2019 Forrester Wave Report. Retrieved September 10, 2020 from https://about.gitlab.com/analysts/forrester-cloudci19/Google Scholar
- Keheliya Gallaba, Christian Macho, Martin Pinzger, and Shane McIntosh. 2018. Noise and heterogeneity in historical build data: an empirical study of Travis CI. In ASE. ACM, 87-97.Google Scholar
- Keheliya Gallaba and Shane McIntosh. 2020. Use and Misuse of Continuous Integration Features: An Empirical Study of Projects That (Mis)Use Travis CI. IEEE Trans. Software Eng. 46, 1 ( 2020 ), 33-50.Google ScholarCross Ref
- T. A. Ghaleb, D. Alencar da Costa, Y. Zou, and A. E. Hassan. 2019. Studying the Impact of Noises in Build Breakage Data. IEEE Transactions on Software Engineering ( 2019 ), 1-1.Google Scholar
- GitLab Team. 2020. GitLab. Retrieved September 10, 2020 from https://about. gitlab.comGoogle Scholar
- GitLab Team. 2020. GitLab-CI Linter. Retrieved September 10, 2020 from https://docs.gitlab.com/ce/ci/yaml/README.html #validate-the-gitlab-ciymlGoogle Scholar
- GitLab Team. 2020. GitLab CI/CD Pipeline Configuration Reference. Retrieved September 10, 2020 from https://docs.gitlab.com/ee/ci/yaml/Google Scholar
- GitLab Team. 2020. GitLab DAST Template. Retrieved September 10, 2020 from https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/ Security/DAST.gitlab-ci.ymlGoogle Scholar
- GitLab Team. 2020. GitLab Review Apps. Retrieved September 10, 2020 from https://docs.gitlab.com/ee/ci/review_apps/Google Scholar
- GitLab Team. 2020. GitLab SAST Template. Retrieved September 10, 2020 from https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/ Security/SAST.gitlab-ci.ymlGoogle Scholar
- GitLab Team. 2020. GitLab Triage Template. Retrieved September 10, 2020 from https://gitlab.com/gitlab-org/gitlab-triage/blob/master/.gitlab-ci.ymlGoogle Scholar
- Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-ofs in continuous integration: assurance, security, and flexibility. In ESEC/SIGSOFT FSE. ACM, 197-207.Google Scholar
- Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In ASE. ACM, 426-437.Google Scholar
- Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Addison-Wesley Professional.Google ScholarDigital Library
- John Micco. 2016. Flaky tests at Google and how we mitigate them. Retrieved September 10, 2020 from https://testing.googleblog.com/ 2016 /05/flaky-tests-atgoogle-and-how-we.htmlGoogle Scholar
- JUnit Team. 2020. JUnit. Retrieved September 10, 2020 from https://junit.org/ junit5/Google Scholar
- Sunghun Kim and Michael D. Ernst. 2007. Which warnings should I fix first?. In ESEC/SIGSOFT FSE. ACM, 45-54.Google Scholar
- Carlene Lebeuf, Margaret-Anne D. Storey, and Alexey Zagalsky. 2018. Software Bots. IEEE Software 35, 1 ( 2018 ), 18-23.Google Scholar
- Carlene Lebeuf, Alexey Zagalsky, Matthieu Foucault, and Margaret-Anne D. Storey. 2019. Defining and classifying software bots: a faceted taxonomy. In BotSE@ICSE. IEEE / ACM, 1-6.Google Scholar
- Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In SIGSOFT FSE. ACM, 643-653.Google ScholarDigital Library
- Christian Macho, Shane McIntosh, and Martin Pinzger. 2018. Automatically repairing dependency-related build breakage. In SANER. IEEE Computer Society, 106-117.Google Scholar
- Paul M. Duvall. 2010. Continuous Integration. Patterns and Antipatterns. Retrieved September 10, 2020 from https://dzone.com/refcardz/continuousintegration?chapter= 1Google Scholar
- Paul M. Duvall. 2011. Continuous Delivery: Patterns and Antipatterns in the Software Life Cycle. Retrieved September 10, 2020 from https://dzone.com/ refcardz/continuous-delivery-patternsGoogle Scholar
- Pip. 2020. Pip. Retrieved September 10, 2020 from https://pypi.org/project/pip/Google Scholar
- Pip Team. 2020. Pipenv: Python Development Workflow for Humans. Retrieved September 10, 2020 from https://docs.pipenv.org/Google Scholar
- PMD Team. 2020. PMD. Retrieved September 10, 2020 from https://pmd.github.io/Google Scholar
- Pylint Team. 2020. Pylint. Retrieved September 10, 2020 from https://www. pylint.org/Google Scholar
- Pytest Team. 2020. Pytest. Retrieved September 10, 2020 from http://pytest.org/Google Scholar
- Python Wheel Team. 2020. Python Wheel. Retrieved September 10, 2020 from https://pypi.org/project/wheel/Google Scholar
- Akond Rahman, Chris Parnin, and Laurie Williams. 2019. The Seven Sins: Security Smells in Infrastructure As Code Scripts. In Proceedings of the 41st International Conference on Software Engineering (Montreal, Quebec, Canada) ( ICSE '19). IEEE Press, Piscataway, NJ, USA, 164-175.Google ScholarDigital Library
- Tony Savor, Mitchell Douglas, Michael Gentili, Laurie A. Williams, Kent L. Beck, and Michael Stumm. 2016. Continuous deployment at Facebook and OANDA. In ICSE (Companion Volume). ACM, 21-30.Google Scholar
- Tushar Sharma, Marios Fragkoulis, and Diomidis Spinellis. 2016. Does your configuration code smell?. In MSR. ACM, 189-200.Google Scholar
- D. Spencer and J.J. Garrett. 2009. Card Sorting: Designing Usable Categories. ( 2009 ).Google Scholar
- Sphinx Team. 2020. Spinx Python Documentation Generator. Retrieved September 10, 2020 from http://www.sphinx-doc.org/Google Scholar
- Spring Boot Team. 2020. Dependency Management in Spring Boot. Retrieved September 10, 2020 from https://docs.spring.io/spring-boot/docs/current/reference/ html/using-spring-boot.html#using-boot-dependency-managementGoogle Scholar
- Simon Urli, Zhongxing Yu, Lionel Seinturier, and Martin Monperrus. 2018. How to Design a Program Repair Bot?: Insights from the Repairnator Project. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (Gothenburg, Sweden) (ICSE-SEIP '18). ACM, 10.Google ScholarDigital Library
- Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar T. Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In ESEC/SIGSOFT FSE. ACM, 805-816.Google Scholar
- Carmine Vassallo, Sebastian Proksch, Harald C. Gall, and Massimiliano Di Penta. 2019. Automated reporting of anti-patterns and decay in continuous integration. In ICSE. IEEE / ACM, 105-115.Google Scholar
- Carmine Vassallo, Sebastian Proksch, Anna Jancso, Harald C. Gall, and Massimiliano Di Penta. 2020. Replication Package for “Configuration Smells in Continuous Delivery Pipelines: A Linter and a Six-Month Study on GitLab”. https://doi.org/10.5281/zenodo.3861003. Google ScholarCross Ref
- Carmine Vassallo, Sebastian Proksch, Timothy Zemp, and Harald C. Gall. 2020. Every build you break: developer-oriented assistance for build failure resolution. Empirical Software Engineering 25, 3 ( 2020 ), 2218-2257.Google Scholar
- Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella. 2017. A Tale of CI Build Failures: An Open Source and a Financial Organization Perspective. In ICSME. IEEE Computer Society, 183-193.Google Scholar
- Fadi Wedyan, Dalal Alrmuny, and James M. Bieman. 2009. The Efectiveness of Automated Static Analysis Tools for Fault Detection and Refactoring Prediction. In ICST. IEEE Computer Society, 141-150.Google Scholar
- Fiorella Zampetti, Carmine Vassallo, Sebastiano Panichella, Gerardo Canfora, Harald C. Gall, and Massimiliano Di Penta. 2020. An empirical characterization of bad practices in continuous integration. Empirical Software Engineering 25, 2 ( 2020 ), 1095-1135.Google Scholar
Index Terms
- Configuration smells in continuous delivery pipelines: a linter and a six-month study on GitLab
Recommendations
Do Developers Fix Continuous Integration Smells?
PROMISE 2023: Proceedings of the 19th International Conference on Predictive Models and Data Analytics in Software EngineeringContinuous Integration (CI) is a common software engineering practice in which the code changes are frequently merged into a software project repository after automated builds and tests have been successfully run. CI enables developers to quickly ...
Are architectural smells independent from code smells? An empirical study
Highlights- Case study analyzing the correlations among code smells, groups of code smells and architectural smells.
AbstractBackground. Architectural smells and code smells are symptoms of bad code or design that can cause different quality problems, such as faults, technical debt, or difficulties with maintenance and evolution. Some studies ...
On the effects of continuous delivery on code quality: A case study in industry
Highlights- A case study highlighting the effects of the adoption of continuous delivery on the code and product quality in a Brazilian company.
AbstractContinuous delivery has been adopted by organizations to make software available to their users at any time. The transition from traditional software delivery methodologies to continuous delivery can impact on the results generated by ...
Comments