Open Source Development Course Continuous integration and deployment (CI/CD) Vojtˇech Trefn´y vtrefny@redhat.com 23. 3. 2022 twitter.com/vojtechtrefny github.com/vojtechtrefny ɭ gitlab.com/vtrefny Pipeline CI/CD Pipeline • Steps that need to be performed to test and deliver a new version of the software. • Defines what needs to be done: when, how and in what order. • Steps can vary for every project. • Multiple pipelines or steps can run in parallel. 1 / 30 CI/CD Pipeline 1. Testing environment Preparation of the environment to run the tests: deploying containers, starting VMs... 2. Static Analysis Finding defects by analyzing the code without running it. 3. Code style Checking for violations of the language or project style guides. 4. Build Building the project from source. 5. Tests Running project test suite or test suites. 6. Packaging and Deployment Building source archives, packages or container images. 2 / 30 Testing Environment Testing Environment 1. Preparation of VMs/containers to run the tests We might want to run tests in different environments on multiple different distributions or architectures. 2. Installation of the test dependencies Test dependencies are usually not covered by the project dependencies. 3. Getting the code Clone the PR or get the latest code from the master branch. 3 / 30 Static Analysis Static Analysis • Tools that can identify potential bugs by analyzing the code without running it. • Can detect problems not covered by the test suite – corner cases, error paths etc. • Coverity (C/C++, Java, Python, Go. . . )1 • Cppcheck (C/C++)2 • Pylint (Python)3 • RuboCop (Ruby)4 1 https://scan.coverity.com 2 http://cppcheck.sourceforge.net/ 3 https://www.pylint.org 4 https://docs.rubocop.org 4 / 30 Coverity Error: USE AFTER FREE (CWE-825): libblockdev-2.13/src/plugins/lvm-dbus.c:1163: freed_arg: "g_free" frees "output". libblockdev-2.13/src/plugins/lvm-dbus.c:1165: pass_freed_arg: Passing freed pointer "output" as an argument to "g_set_error". # 1163| g_free (output); # 1164| if (ret == 0) { # 1165|-> g_set_error (error, BD_LVM_ERROR, BD_LVM_ERROR_PARSE, # 1166| "Failed to parse number from output: ’%s’", # 1167| output); 5 / 30 LGTM https://lgtm.com/projects/g/storaged-project/blivet-gui/ 6 / 30 Runtime Analysis • Tools that can identify bugs during runtime. • Needs the code to actually run, either through manual testing or when running the test suite. • Valgrind – memory management and threading bugs5 • ASan – AddressSanitizer – memory management bugs (buffer overflow, dangling pointers...). Part of the LLVM Analyzers project, integrated into gcc and clang6 5 https://valgrind.org/ 6 https://github.com/google/sanitizers/wiki/AddressSanitizer 7 / 30 Code Style Code style and style guides • Coding conventions – naming, code lay-out, comment style. . . • Language specific (PEP 87), project specific (Linux kernel coding style8) or library/toolkit specific (GTK coding style9). • Automatic checks using specific tools (pycodestyle) or (partially) by the static analysis tools. 7 https://www.python.org/dev/peps/pep-0008/ 8 https://www.kernel.org/doc/html/v5.11/process/coding-style.html 9 https://developer.gnome.org/programming-guidelines/stable/c-coding-style.html.en 8 / 30 Linux kernel coding style https://www.kernel.org/doc/html/v5.11/process/coding-style.html 9 / 30 Python and PEP 8 • Automatic code style checking tools exist for the Python PEP 8 style code. • pycodestyle10(formerly pep8) is used for checking/enforcing PEP 8 in many Python applications. • black11can be used to automatically format Python code in a PEP 8 compliant way. • Static analysis tools like pylint or pyflakes also check for some PEP 8 style violations. 10 https://github.com/PyCQA/pycodestyle 11 https://github.com/psf/black 10 / 30 Python and PEP 8 $ pycodestyle-3 blivetgui/blivetgui.py blivetgui/blivetgui.py:23:80: E501 line too long (80 > 79 characters) blivetgui/blivetgui.py:30:1: E402 module level import not at top of file 11 / 30 Documentation style • Documentation might be checked in the same way code is. • Similar style documents and tools for checking documentations exist (for example PEP 25712and pydocstyle13for Python). • In some cases wrong or missing documentation (docstrings in the code) can lead to a broken build or missing features. 12 https://www.python.org/dev/peps/pep-0257/ 13 http://www.pydocstyle.org 12 / 30 Build Build • Building the project, a preparation to run the test suite. • Depends on language – mostly no-op for interpreted languages, more complicated for compiled ones. • Build in the CI environment can detect issues with dependencies. • Builds on different architectures can help detect issues related to endianness or data types sizes. 13 / 30 Tests Tests • Running tests that are part of the project. • New tests should be part of every change to the codebase. • New features require new unit and integration tests. • Bug fixes should come with a regression test. • For some project (like libraries) running test suites of their users might be an option. 14 / 30 Coverage • Code coverage (or Test coverage) represents how much of the code is covered by the test suite. • Usually percentual value that shows how many lines of the code were “visited” by the test. • Generally a check that all functions and branches are covered by the suite. • Used as a measure of the test suite “quality”. 15 / 30 Coverage Name Stmts Miss Branch BrPart Cover Missing -------------------------------------------------------------------------------- a.py 487 9 178 11 97% 206, 268, 377->376, 393->392, 418, 448, 452, 460, 660, 729, 746, 889->891, 891->894 b.py 220 8 74 8 95% 81, 173, 193->197, 279, 340, 342, 346 c.py 19 9 8 2 44% 35-36, 50->48, 53, 60-70 d.py 5 0 6 1 91% 31->exit e.py 46 0 4 0 100% ... -------------------------------------------------------------------------------- TOTAL 3600 1477 1381 100 56% 16 / 30 Coverage • Automated coverage tests might be part of the CI. • Decrease in coverage can be viewed as a reason to reject contribution to the project. 17 / 30 Delivery and Deployment Packaging and publishing • Delivery – releasing new changes quickly and regularly (daily, weekly...). • Deployment – delivery with automated push to production, without human interaction. • Usually after merging the changes, not for the PRs. • Building packages, container images, ISO images. . . • Built packages can be used for further testing (manually by the Quality Assurance or in another CI infrastructure) or directly pushed to production or included in testing/nightly builds of the project. 18 / 30 CI Tools Demo GitHub Actions • Automation framework integrated into GitHub. • Does not cover only CI but also CD (publishing packages on various services and deploying on many public clouds) and project and issue management. • Free for all public repositories, limited and paid options for private projects. • https://github.com/features/actions 19 / 30 GitHub Actions 20 / 30 GitHub Actions 21 / 30 GitLab CI • CI/CD automation integrated into GitLab. • Configuration is done with a YAML file in the repository. • Pipleines/tests can run either on infastructure provided by GitLab or on custom runners. • https://docs.gitlab.com/ee/ci/ 22 / 30 GitLab CI 23 / 30 Jenkins • Automation system, not a “true” CI/CD tool. • Can automatically run given tasks on a node or set of nodes. • Tasks can be started on time basis or triggered by an external event (like a new commit or PR on GitHub). • https://jenkins.io/ 24 / 30 Fedora CI • Complex CI system with the task to deliver an “Always Ready Operating System”. • Packages are tested after every change and gated if the CI pipeline fails. • The goal is to prevent breaking the distribution. CI will stop the broken package before it can affect the distribution. 25 / 30 Fedora CI 26 / 30 Packit • Tool for integrating upstream projects to Fedora. • RPM packages are automatically built on every pull request. • New releases can be automatically built and pushed to Fedora. 27 / 30 Packit 28 / 30 Travis CI • Used to be the most popular CI service for open source products. • Can be integrated into your projects on GitHub and GitLab. • Configured using .travis.yml file in the project • Unfortunately Travis drastically limited free plans for open source projects in 202014. • https://travis-ci.com 12 https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing 29 / 30 Questions Questions Thank you for your attention. https://github.com/crocs-muni/open-source-development-course 30 / 30