1.2. CI/CD Ecosystem
1.2.1. Agility
Pair Programming in PyCharm: https://www.jetbrains.com/help/pycharm/code-with-me.html
Further Reading: https://dev.astrotech.io/agile/index.html
1.2.2. Ecosystem
Further Reading: https://dev.astrotech.io/summary/index.html
1.2.3. Version Control System
Git
Git Flow
Github Flow
Github
Bitbucket
GitLab
GitOps: https://www.gitops.tech
FluxCD: https://github.com/fluxcd/flux
Further Reading: https://dev.astrotech.io/git/index.html
1.2.4. Virtualization
Docker
LXC - Linux Containers
OCI - Open Container Initiative
Kubernetes
Containerd
OpenShift
Open Stack
Amazon EKS, ECS
Further Reading: https://dev.astrotech.io/docker/index.html
1.2.5. Continuous Integration / Delivery
Jenkins
Github Actions
Bitbucket Pipelines
CircleCI
Travis
GitLab
Further Reading: https://dev.astrotech.io/jenkins/index.html
1.2.6. Quality Assurance
SonarQube
SonarLint [1]
SonarScanner
SonarCloud
Coverage
PEP-8
PyLint
Black
Further Reading: https://dev.astrotech.io/sonarqube/index.html
Further Reading: https://python3.info/devops/ci-cd/tools.html#static-analysis
Further Reading: https://python3.info/devops/ci-cd/code-style.html
Further Reading: https://python3.info/devops/ci-cd/coverage.html
Further Reading: https://python3.info/devops/ci-cd/static-analysis.html
1.2.7. Issue Tracker
Jira
Gitlab
Github issues
Jira Integration: https://jira.astrotech.io/end-user/automation.html
Further Reading: https://dev.astrotech.io/jira/index.html
1.2.8. SSH
Further Reading: https://dev.astrotech.io/linux/index.html
1.2.9. Testing
Further Reading: https://test.astrotech.io
1.2.10. Mutation Testing
mutmut (actively maintained) https://github.com/boxed/mutmut
mutpy (last commit 2019) https://github.com/mutpy/mutpy
https://hackernoon.com/mutmut-a-python-mutation-testing-system-9b9639356c78
https://www.geeksforgeeks.org/software-testing-mutation-testing/
Following code:
>>> def is_adult(age: int) -> bool:
... if age >= 18: # original line
... return True
... else:
... return False
Will be modified to:
>>> def is_adult(age: int) -> bool:
... if age > 18: # mutated line
... return True
... else:
... return False
And then all tests will be executed to check if you have good tests to cover for such change
mutmut
2.0 creates the following mutants
(source):
Operator mutations: About 30 different patterns like replacing + by - , * by ** and similar, but also > by >= .
Keyword mutations: Replacing True by False , in by not in and similar.
Number mutations: You can write things like 0b100 which is the same as 4, 0o100, which is 64, 0x100 which is 256, .12 which is 0.12 and similar. The number mutations try to capture mistakes in this area. mutmut simply adds 1 to the number.
Name mutations: The name mutations capture copy vs. deepcopy and "" vs. None .
Argument mutations: Replaces keyword arguments one by one from dict(a=b) to dict(aXXX=b).
or_test and and_test: and ↔ or
String mutation: Adding XX to the string.
Those can be grouped into three very different kinds of mutations:
value mutations (string mutation, number mutation),
decision mutations (switch if-else blocks, e.g. the or_test / and_test and the keyword mutations),
statement mutations (removing or changing a line of code).
$ ln -s /usr/bin/python3 /home/ubuntu/bin/python
$ pip install mutmut
$ mutmut --help
$ mutmut run
$ mutmut results
$ mutmut show 2
$ mutmut show 22
$ mutmut html
$ python3 -m http.server 8080 --directory html
$ mutmut junitxml
$ rm -fr .mutmut-cache
[mutmut]
paths_to_mutate=src/
backup=False
runner=python -m hammett -x
tests_dir=tests/
dict_synonyms=Struct, NamedStruct
some_code_here() # pragma: no mutate
In order to better integrate with CI/CD systems, mutmut supports the generation of a JUnit XML report (using https://pypi.org/project/junit-xml/). This option is available by calling mutmut junitxml. In order to define how to deal with suspicious and untested mutants, you can use
$ mutmut junitxml --suspicious-policy=ignore --untested-policy=ignore
The possible values for these policies are:
ignore: Do not include the results on the report at all
skipped: Include the mutant on the report as "skipped"
error: Include the mutant on the report as "error"
failure: Include the mutant on the report as "failure"
If a failed mutant is included in the report, then the unified diff of the mutant will also be included for debugging purposes.
1.2.11. BDD Testing
Lettuce: http://lettuce.it/index.html
Cucumber: https://cucumber.io
Behave: https://behave.readthedocs.io/en/stable/tutorial.html
Example:
Feature: showing off behave Scenario: run a simple test Given we have behave installed When we implement a test Then behave will test it for us!from behave import * @given('we have behave installed') def step_impl(context): pass @when('we implement a test') def step_impl(context): assert True is not False @then('behave will test it for us!') def step_impl(context): assert context.failed is False$ behave Feature: showing off behave # features/tutorial.feature:1 Scenario: run a simple test # features/tutorial.feature:3 Given we have behave installed # features/steps/tutorial.py:3 When we implement a test # features/steps/tutorial.py:7 Then behave will test it for us! # features/steps/tutorial.py:11 1 feature passed, 0 failed, 0 skipped 1 scenario passed, 0 failed, 0 skipped 3 steps passed, 0 failed, 0 skipped, 0 undefined
Parameters:
Scenario: look up a book
Given I search for a valid book
Then the result page will include "success"
Scenario: look up an invalid book
Given I search for a invalid book
Then the result page will include "failure"
@then('the result page will include "{text}"')
def step_impl(context, text):
if text not in context.response:
fail('%r not in %r' % (text, context.response))
Step Data:
Scenario Outline: Blenders
Given I put <thing> in a blender,
when I switch the blender on
then it should transform into <other thing>
Examples: Amphibians
| thing | other thing |
| Red Tree Frog | mush |
Examples: Consumer Electronics
| thing | other thing |
| iPhone | toxic waste |
| Galaxy Nexus | toxic waste |
1.2.12. Load Testing
Gatling: https://gatling.io
Locust: https://locust.io/
JMeter: https://jmeter.apache.org
package computerdatabase // 1
import scala.concurrent.duration._
// 2
import io.gatling.core.Predef._
import io.gatling.http.Predef._
class BasicSimulation extends Simulation { // 3
val httpProtocol = http // 4
.baseUrl("http://computer-database.gatling.io") // 5
.acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8") // 6
.doNotTrackHeader("1")
.acceptLanguageHeader("en-US,en;q=0.5")
.acceptEncodingHeader("gzip, deflate")
.userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
val scn = scenario("BasicSimulation") // 7
.exec(
http("request_1") // 8
.get("/")
) // 9
.pause(5) // 10
setUp( // 11
scn.inject(atOnceUsers(1)) // 12
).protocols(httpProtocol) // 13
}
What does it mean?
The optional package.
The required imports.
The class declaration. Note that it extends Simulation.
The common configuration to all HTTP requests.
The baseUrl that will be prepended to all relative urls.
Common HTTP headers that will be sent with all the requests.
The scenario definition.
An HTTP request, named request_1. This name will be displayed in the final reports.
The url this request targets with the GET method.
Some pause/think time.
1.2.13. Testing UI
Selenium: https://www.selenium.dev
from selenium import webdriver
from selenium.webdriver.common.by import By
def test_eight_components():
driver = webdriver.Chrome()
driver.get("https://google.com")
title = driver.title
assert title == "Google"
driver.implicitly_wait(0.5)
search_box = driver.find_element(by=By.NAME, value="q")
search_button = driver.find_element(by=By.NAME, value="btnK")
search_box.send_keys("Selenium")
search_button.click()
search_box = driver.find_element(by=By.NAME, value="q")
value = search_box.get_attribute("value")
assert value == "Selenium"
driver.quit()
1.2.14. Testing microservices
Further Reading: https://arch.astrotech.io
Source: https://martinfowler.com/articles/microservice-testing/
1.2.15. Provisioning
Ansible
Puppet
Chef
Salt, SaltStack
Vagrant
Further Reading: https://dev.astrotech.io/puppet/index.html
Further Reading: https://dev.astrotech.io/ansible/index.html
Further Reading: https://dev.astrotech.io/vagrant/index.html
1.2.16. Setup
git clone https://github.com/sages-pl/src-python /home/ubuntu/src
sudo apt update
sudo apt install -y uidmap
curl https://get.docker.com/rootless |sh
echo 'export PATH=/home/ubuntu/.local/bin:$PATH' >> ~/.profile
echo 'export DOCKER_HOST=unix:///run/user/1000/docker.sock' >> ~/.profile
echo 'export IP=$(curl -s ipecho.net/plain)' >> ~/.profile
source ~/.profile
docker network create ecosystem
Gitea:
cat > /home/ubuntu/bin/run-gitea << EOF docker run \\ --name gitea \\ --detach \\ --restart always \\ --env USER_UID=1000 \\ --env USER_GID=1000 \\ --env GITEA__server__ROOT_URL=http://$IP:3000/ \\ --env GITEA__database__DB_TYPE=sqlite3 \\ --env GITEA__database__PATH=/var/lib/gitea/data/gitea.db \\ --env GITEA__database__HOST=... \\ --env GITEA__database__NAME=... \\ --env GITEA__database__USER=... \\ --env GITEA__database__PASSWD=... \\ --dns 8.8.8.8 \\ --network ecosystem \\ --publish 3000:3000 \\ --publish 2222:22 \\ --volume gitea_data:/var/lib/gitea \\ --volume gitea_config:/etc/gitea \\ --volume /etc/timezone:/etc/timezone:ro \\ --volume /etc/localtime:/etc/localtime:ro \\ gitea/gitea:latest-rootless echo "Gitea running on: http://$IP:3000/" EOF chmod +x /home/ubuntu/bin/run-gitea run-gitea
Jenkins:
cat > /home/ubuntu/bin/run-jenkins << EOF chmod o+rw /run/user/1000/docker.sock sudo ln -s /usr/bin/python3 /usr/bin/python sudo ln -s /home/ubuntu/.local/share/docker/volumes/jenkins_data/_data/ /var/jenkins_home docker run \\ --name jenkins \\ --detach \\ --restart always \\ --network ecosystem \\ --publish 8080:8080 \\ --volume jenkins_data:/var/jenkins_home \\ --volume /run/user/1000/docker.sock:/var/run/docker.sock \\ jenkinsci/blueocean:latest docker exec -u root jenkins apk add python3 py3-pip echo "Jenkins running on: http://$IP:8080/" EOF chmod +x /home/ubuntu/bin/run-jenkins run-jenkins
SonarQube:
cat > /home/ubuntu/bin/run-sonarqube << EOF docker run \\ --name sonarqube \\ --detach \\ --restart always \\ --network ecosystem \\ --publish 9000:9000 \\ --volume sonarqube_data:/opt/sonarqube/data \\ --volume sonarqube_logs:/opt/sonarqube/logs \\ --volume sonarqube_extensions:/opt/sonarqube/extensions \\ sonarqube echo "SonarQube running on: http://$IP:9000/" EOF chmod +x /home/ubuntu/bin/run-sonarqube run-sonarqube
SonarScanner:
sonar-project.properties
Further Reading: https://dev.astrotech.io/sonarqube/sonarscanner.html
Further Reading: https://python3.info/devops/ci-cd/static-analysis.html
docker pull sonarsource/sonar-scanner-cli
Docker Registry:
cat > /home/ubuntu/bin/run-registry << EOF docker run \\ --detach \\ --restart always \\ --name registry \\ --net ecosystem \\ --publish 5000:5000 \\ --volume registry_data:/var/lib/registry \\ registry:2 echo "Registry running on: http://$IP:5000/" EOF chmod +x /home/ubuntu/bin/run-registry run-registry
Registry UI:
cat > /home/ubuntu/registry-ui.yml << EOF listen_addr: 0.0.0.0:8888 base_path: / registry_url: http://registry:5000 verify_tls: true # registry_username: user # registry_password: pass # The same one should be configured on Docker registry as Authorization Bearer token. event_listener_token: token event_retention_days: 7 event_database_driver: sqlite3 event_database_location: data/registry_events.db # event_database_driver: mysql # event_database_location: user:password@tcp(localhost:3306)/docker_events cache_refresh_interval: 10 # If users can delete tags. # If set to False, then only admins listed below. anyone_can_delete: false # Users allowed to delete tags. # This should be sent via X-WEBAUTH-USER header from your proxy. admins: [] # Debug mode. Affects only templates. debug: true # How many days to keep tags but also keep the minimal count provided no matter how old. purge_tags_keep_days: 90 purge_tags_keep_count: 2 EOFcat > /home/ubuntu/bin/run-registry-ui << EOF docker run \\ --name registry-ui \\ --detach \\ --restart always \\ --network ecosystem \\ --publish 8888:8888 \\ --volume /home/ubuntu/registry-ui.yml:/opt/config.yml:ro \\ quiq/docker-registry-ui echo "Registry UI running on: http://$IP:8888/" EOF chmod +x /home/ubuntu/bin/run-registry-ui run-registry-ui
Files:
cat > /home/ubuntu/src/Dockerfile << EOF FROM python:3.10 COPY game.pyz /game.pyz CMD python3 /game.pyz EOFcat > /home/ubuntu/src/sonar-project.properties << EOF ## Sonar Server sonar.host.url=http://sonarqube:9000/ sonar.login=TOKEN ## Software Configuration Management sonar.scm.enabled=true sonar.scm.provider=git ## SonarScanner Config sonar.sourceEncoding=UTF-8 sonar.verbose=false sonar.log.level=INFO sonar.showProfiling=false sonar.projectBaseDir=/usr/src/ sonar.working.directory=/tmp/ ## Quality Gates sonar.qualitygate.wait=true sonar.qualitygate.timeout=300 ## About Project sonar.projectKey=mypythonproject sonar.projectName=MyPythonProject ## Python sonar.language=py sonar.python.version=3.10 sonar.sources=src sonar.tests=test sonar.inclusions=**/*.py sonar.exclusions=**/migrations/**,**/*.pyc,**/__pycache__/** sonar.python.xunit.skipDetails=false sonar.python.xunit.reportPath=.tmp/xunit.xml sonar.python.coverage.reportPaths=.tmp/coverage.xml,./cobertura.xml sonar.python.bandit.reportPaths=.tmp/bandit.json sonar.python.pylint.reportPaths=.tmp/pylint.txt sonar.python.flake8.reportPaths=.tmp/flake8.txt EOFcat > /home/ubuntu/src/Jenkinsfile << EOF pipeline { agent any triggers { pollSCM('* * * * *') } stages { stage('Env Prepare') { steps { sh 'run/env-prepare' }} stage('Env Setup') { steps { sh 'run/env-setup' }} stage('Env Debug') { steps { sh 'run/env-debug' }} stage('Test') { parallel { stage('Test Code Style') { steps { sh 'run/test-codestyle' }} stage('Test Functional') { steps { sh 'run/test-functional' }} stage('Test Integration') { steps { sh 'run/test-integration' }} stage('Test Lint') { steps { sh 'run/test-lint' }} stage('Test Load') { steps { sh 'run/test-load' }} stage('Test Mutation') { steps { sh 'run/test-mutation' }} stage('Test Regression') { steps { sh 'run/test-regression' }} stage('Test Security') { steps { sh 'run/test-security' }} stage('Test Smoke') { steps { sh 'run/test-smoke' }} stage('Test Static') { steps { sh 'run/test-static' }} stage('Test UI') { steps { sh 'run/test-ui' }} stage('Test Unit') { steps { sh 'run/test-unit' }} }} stage('Test Report') { steps { sh 'run/test-report' }} stage('Artifact Prepare') { steps { sh 'run/artifact-prepare' }} stage('Artifact Build') { steps { sh 'run/artifact-create' }} stage('Artifact Publish') { steps { sh 'run/artifact-publish' }} stage('Artifact Cleanup') { steps { sh 'run/artifact-cleanup' }} stage('Deploy Dev') { steps { sh 'run/deploy-dev' }} stage('Deploy Test') { steps { sh 'run/deploy-test' }} stage('Deploy Preprod') { steps { sh 'run/deploy-preprod' }} stage('Deploy Prod') { steps { sh 'run/deploy-prod' }} } } // To run all: // grep -Po "^[^/].*sh '\K.+(?=')" Jenkinsfile |sh -x EOF
Tests:
cd /home/ubuntu/src mkdir -p run/ touch run/test-codestyle touch run/test-coverage touch run/test-functional touch run/test-integration touch run/test-lint touch run/test-load touch run/test-mutation touch run/test-regression touch run/test-report touch run/test-security touch run/test-smoke touch run/test-static touch run/test-ui touch run/test-unit touch run/artifact-prepare touch run/artifact-create touch run/artifact-publish touch run/artifact-cleanup touch run/deploy-dev touch run/deploy-test touch run/deploy-preprod touch run/deploy-prod chmod +x run/*cat > run/env-prepare << EOF env |sort EOFcat > run/env-setup << EOF python3 -m pip install --upgrade -r requirements.dev EOFcat > run/env-debug << EOF which python3 python3 --version python3 -m pip freeze EOFcat > run/test-codestyle << EOF export PYTHONPATH=src python3 -m flake8 --exit-zero --doctest --output-file=.tmp/flake8.txt src EOFcat > run/test-coverage << EOF export PYTHONPATH=src python3 -m coverage run src python3 -m coverage xml -o .tmp/coverage.xml EOFcat > run/test-functional << EOF echo 'Not Implemented' EOFcat > run/test-integration << EOF export PYTHONPATH=src python3 -m doctest -v test/*.py EOFcat > run/test-lint << EOF export PYTHONPATH=src python3 -m pylama --verbose --async src || true python3 -m pylint --exit-zero --msg-template="{path}:{line}: [{msg_id}({symbol}), {obj}] {msg}" --output=.tmp/pylint.txt --disable=C0114,C0115,C0116,E0401,C0103 src EOFcat > run/test-load << EOF echo 'Not Implemented' EOFcat > run/test-mutation << EOF rm -fr .mutmut-cache mutmut run || true mutmut results mutmut junitxml --suspicious-policy=ignore --untested-policy=ignore > .tmp/xunit.xml EOFcat > run/test-regression << EOF echo 'Not Implemented' EOFcat > run/test-report << EOF docker run --rm --net ecosystem -v $(pwd):/usr/src sonarsource/sonar-scanner-cli EOFcat > run/test-security << EOF export PYTHONPATH=src python3 -m bandit --format json --output=.tmp/bandit.json --recursive src EOFcat > run/test-smoke << EOF echo 'Not Implemented' EOFcat > run/test-static << EOF export PYTHONPATH=src python3 -m mypy --ignore-missing-imports --cobertura-xml-report=.tmp src || test EOFcat > run/test-ui << EOF echo 'Not Implemented' EOFcat > run/test-unit << EOF export PYTHONPATH=src python3 -m unittest discover -v test EOFcat > run/artifact-prepare << EOF python3 -m pip install --upgrade --no-cache-dir -r requirements.prod --target src rm -fr src/*.dist-info python3 -m compileall -f src # find src -name '*.py' -not -name '__main__.py' -not -name '__init__.py' -delete # not working for now python3 -m zipapp --python="/usr/bin/env python3" --output=game.pyz src EOFcat > run/artifact-create << EOF docker build . -t localhost:5000/myapp:$(git log -1 --format='$h') EOFcat > run/artifact-publish << EOF docker push localhost:5000/myapp:$(git log -1 --format='$h') EOFcat > run/artifact-cleanup << EOF docker rmi localhost:5000/myapp:$(git log -1 --format='$h') EOFcat > run/deploy-dev << EOF echo 'Not Implemented' EOFcat > run/deploy-test << EOF echo 'Not Implemented' EOFcat > run/deploy-preprod << EOF echo 'Not Implemented' EOFcat > run/deploy-prod << EOF echo 'Not Implemented' EOF