PyParis2018 - Python tooling for continuous deployment

PyParis2018 - Python tooling for continuous deployment

• 1.
  Python tooling forcontinuousPython tooling for continuous deploymentdeployment PyParis Novembre 2018PyParis Novembre 2018 Arthur Lutz - LogilabArthur Lutz - Logilab 1
• 2.
  Introduction /meIntroduction /me ArthurLutz / @arthurlutz @[email protected] Logilab Python Nantes Paris Salt Meetup 2
• 3.
  IntroductionIntroduction How we benefitedfrom a rich Python ecosystem to move from sprint-based delivery to continuous delivery. 3
• 4.
  Continous deliveryContinous delivery Checkin Check in Check in Trigger Trigger Trigger Trigger Trigger Feedback Feedback Feedback Feedback Feedback Feedback Approval Approval Delivery team Version control Build & unit tests Automated acceptance tests User acceptance tests Release Source: wikipedia 4
• 5.
  Python StackPython Stack (ignorethe order) View it on StackShare 5
• 6.
  Reproducible environmentReproducible environment usingvolumes for dev parts (equivalent to pip install -e) to set up postgres, redis, celery, etc. using docker- compose.override.yml and then .env same docker images for acceptance testing, pre-production and production with configuration via environment variables docker docker-compose 6
• 7.
  Test often, break,test againTest often, break, test again - reproducible test envs - test runner / - functionnal testing (launch daemons in pytest fixtures) Tools used in dev, CI/test and supervision. tox pytest flake8 isort pifpaf 7
• 8.
  Ramping up CIRampingup CI 1. draft-based 2. parametric (manual) 3. public changeset (hg phases) + user-centric 4. component-centric 5. feature-centric jenkins (not python) with phabricator + jenkins = and JenkinsFile jenkins-job-builder Differential jenkins job 8
• 9.
  Acceptance test often,break,Acceptance test often, break, test againtest again BDD with /behave robber selenium python bindings 9
• 10.
  Push, decrease quality,pushPush, decrease quality, push againagain Jenkins test metrics and coverage SonarQube (not python) with Peer review with + (amend) + (phabsend) Continuous Quality Control SonarPython mercurial evolve phabricator mercurial extension 10
• 11.
  Ship often, break,ship againShip often, break, ship again rundeck (not python) & for shared deploy logs generate unique version information tag for validation the shipped Jira tickets & for continuous collection of errors rundeckrun sentry raven-python 11
• 12.
  Ship often, measure,ship againShip often, measure, ship again collect metrics with and serve with collecting custom metrics see carbon graphite-api cfmgmtcamp: Use Saltstack to deploy a full monitoring and supervision stack 12
• 13.
  Agile infrastructureAgile infrastructure infrastructureas code, deploy multiple environments portable cloud scaling for in-house cloud hosting used for example to restore production database snapshots in pre-production saltstack salt-cloud openstack awscli 13
• 14.
  Agile infrastructure -Agileinfrastructure - supervision and metricssupervision and metrics sensu (not python) supervision using , ease of deploy of new checks netdata supervision and metrics with to test out complex scenarios testinfra sensu- formula python.d.plugin testinfra 14
• 15.
  Dashboards everywhereDashboards everywhere toextract data from Jira to extract data from RunDeck Badges everywhere! (not python) grafana (not python) serving data from Jira python bindings requests https://shields.io graphite 15
• 16.
  Project impactsProject impacts shortertickets, split them into tiny chunks unstable environments can be OK - work with the users feature flags to encourage early integration of code more peer review, collective appropriation no more semantic versionning 16
• 17.
  FutureFuture Merge request /feature stack centric generated environments Automation of data migration between environments Use of version control in sentry and regression identification OpenShift / OKD / Kubernetes type continuous deployment Dev environments could be less necessary 17
• 18.
  Logilab is hiring!Logilab is hiring ! 18
• 19.
  Conclusion / QuestionsConclusion/ Questions Thanks for your attention Any questions ? Slides (full of links !) : ,html pdf 19