Continuous Integration and Continuous Deployment ================================================ Overview -------- Each git commit to the `master` branch in GitHub triggers both the TravisCI automated tests as well as the AWS CodeBuild building. Each git tag applied to a git commit triggers a CodeBuild build. Travis CI --------- Travis CI runs tests on the MozDef code base with each commit to `master`. The results can be seen on the `Travis CI MozDef dashboard `_ The Test Sequence _________________ * Travis CI creates webhooks when first setup which allow commits to the MozDef GitHub repo to trigger Travis. * When a commit is made to MozDef, Travis CI follows the instructions in the `.travis.yml `_ file. * `.travis.yml` installs `docker-compose` in the `before_install` phase. * In the `install` phase, Travis runs the `build-tests `_ make target which calls `docker-compose build` on the `docker/compose/docker-compose-tests.yml`_ file which builds a few docker containers to use for testing. * In the `script` phase, Travis runs the `tests `_ make target which * calls the `build-tests` make target which again runs `docker-compose build` on the `docker/compose/docker-compose-tests.yml`_ file. * calls the `run-tests `_ make target which. * calls the `run-tests-resources `_ make target which starts the docker containers listed in `docker/compose/docker-compose-tests.yml`_. * runs `flake8` with the `.flake8 `_ config file to check code style. * runs `py.test tests` which runs all the test cases. AWS CodeBuild ------------- Enabling GitHub AWS CodeBuild Integration _________________________________________ Onetime Manual Step ******************* The steps to establish a GitHub CodeBuild integration unfortunately require a onetime manual step be done before using CloudFormation to configure the integration. This onetime manual step **need only happen a single time for a given AWS Account + Region**. It need **not be performed with each new CodeBuild project or each new GitHub repo** 1. Manually enable the GitHub integration in AWS CodeBuild using the dedicated, AWS account specific, GitHub service user. 1. A service user is needed as AWS CodeBuild can only integrate with GitHub from one AWS account in one region with a single GitHub user. Technically you could use different users for each region in a single AWS account, but for simplicity limit yourself to only one GitHub user per AWS account (instead of one GitHub user per AWS account per region) 2. To do the one time step of integrating the entire AWS account in that region with the GitHub service user 1. Browse to `CodeBuild`_\  in AWS and click Create Project 2. Navigate down to ``Source`` and set ``Source Provider`` to ``GitHub`` 3. For ``Repository`` select ``Connect with a GitHub personal access token`` 4. Enter the persona access token for the GitHub service user. If you haven't created one do so and grant it ``repo`` and ``admin:repo_hook`` 5. Click ``Save Token`` 6. Abort the project setup process by clicking the ``Build Projects`` breadcrumb at the top. This “Save Token” step was the only thing you needed to do in that process Grant the GitHub service user access to the GitHub repository ************************************************************* 1. As an admin of the GitHub repository go to that repositories settings, select Collaborators and Teams, and add the GitHub service user to the repository 2. Set their access level to ``Admin`` 3. Copy the invite link, login as the service user and accept the invitation Deploy CloudFormation stack creating CodeBuild project ****************************************************** Deploy the ``mozdef-cicd-codebuild.yml`` CloudFormation template to create the CodeBuild project and IAM Role .. _CodeBuild: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/ The Build Sequence __________________ * A branch is merged into `master` in the GitHub repo or a version git tag is applied to a commit. * GitHub emits a webhook event to AWS CodeBuild indicating this. * AWS CodeBuild considers the Filter Groups configured to decide if the tag or branch warrants triggering a build. These Filter Groups are defined in the ``mozdef-cicd-codebuild.yml`` CloudFormation template. Assuming the tag or branch are acceptable, CodeBuild continues. * AWS CodeBuild reads the `buildspec.yml `_ file to know what to do. * The `install` phase of the `buildspec.yml` fetches `packer `_ and unzips it. * `packer` is a tool that spawns an ec2 instance, provisions it, and renders an AWS Machine Image (AMI) from it. * The `build` phase of the `buildspec.yml` runs the `cloudy_mozdef/ci/deploy `_ script in the AWS CodeBuild Ubuntu 14.04 environment. * The `deploy` script calls the `build-from-cwd `_ target of the `Makefile` which calls `docker-compose build` on the `docker-compose.yml `_ file, building the docker images in the AWS CodeBuild environment. These are built both so they can be consumed later in the build by packer and also for use by developers and the community. * `deploy` then calls the `docker-push-tagged `_ make target which calls * the tag-images_ make target which calls the `cloudy_mozdef/ci/docker_tag_or_push tag `_ script which applies a docker image tag to the local image that was just built by AWS CodeBuild. * the `hub-tagged `_ make target which calls the `cloudy_mozdef/ci/docker_tag_or_push push `_ script which * Uploads the local image that was just built by AWS CodeBuild to DockerHub. If the branch being built is `master` then the image is uploaded both with a tag of `master` as well as with a tag of `latest`. * If the branch being built is from a version tag (e.g. `v1.2.3`) then the image is uploaded with only that version tag applied. * The `deploy` script next calls the `packer-build-github `_ make target in the `cloudy_mozdef/Makefile `_ which calls the `ci/pack_and_copy `_ script which does the following steps. * Calls packer which launches an ec2 instance, executing a bunch of steps and and producing an AMI * Shares the resulting AMI with the AWS Marketplace account * Copies the resulting AMI to a list of additional AWS regions * Copies the tags from the original AMI to these copied AMIs in other regions * Shares the AMIs in these other regions with the AWS Marketplace account * Creates a blob of YAML which contains the AMI IDs. This blob will be used in the CloudFormation templates * When `ci/pack_and_copy` calls packer, packer launches an ec2 instance based on the configuration in `cloudy_mozdef/packer/packer.json `_ * Within this ec2 instance, packer `clones the MozDef GitHub repo and checks out the branch that triggered this build `_. * Packer replaces all instances of the word `latest` in the `docker-compose-cloudy-mozdef.yml `_ file with either the branch `master` or the version tag (e.g. `v1.2.3`). * Packer runs `docker-compose pull` on the `docker-compose-cloudy-mozdef.yml `_ file to pull down both the docker images that were just built by AWS CodeBuild and uploaded to Dockerhub as well as other non MozDef docker images. * After packer completes executing the steps laid out in `packer.json` inside the ec2 instance, it generates an AMI from that instance and continues with the copying, tagging and sharing steps described above. * Now back in the AWS CodeBuild environment, the `deploy` script continues by calling the `publish-versioned-templates `_ make target which runs the `ci/publish_versioned_templates `_ script which * injects the AMI map yaml blob produced earlier into the `mozdef-parent.yml `_ CloudFormation template so that the template knows the AMI IDs of that specific branch of code. * uploads the CloudFormation templates to S3 in a directory either called `master` or the tag version that was built (e.g. `v1.2.3`). .. _docker/compose/docker-compose-tests.yml: https://github.com/mozilla/MozDef/blob/master/docker/compose/docker-compose-tests.yml .. _tag-images: https://github.com/mozilla/MozDef/blob/cfeafb77f9d4d4d8df02117a0ffca0ec9379a7d5/Makefile#L109-L110