Two of my side projects are ASP.Net Core applications that run in Azure with Azure SQL Databases. These days, setting up a deployment pipeline is something I do with every project I start that will be deployed somewhere.
Everything I'm outlining in this post (and sample code I'm pointing to) is just shy of production level. It's not a drop-in solution, but there shouldn't be that many gotchas if you borrowed heavily from these as a starting point.
Table of Contents
Outline of the delivery pipeline
This is a basic starting pipeline for my projects. The intent is to do initial verifications of changes and then safely deliver the changes to "production".
Planning
My first step is to plan out what the minimum looks like, which typically means:
- What is going to be running in production
- How does it safely get to production
- What basic verifications can I run to fail fast if something is broken
Here are the details for the sample application I'll be linking to:
- ASP.Net Core 5 web app
- SPA front-ends, using React, TypeScript, SCSS, and Parcel
- MS SQL Server database with scripted migrations
- Hosting is Azure Web Apps and Azure SQL
- Notifications to private channels on a Discord server
I will have two workflows:
- CI for every commit on any branch
- CI -> Continuous Deployment for my
mainbranch
The deployment process is going to be a blue/green deployment, and it will work like this:
- Deploy
{new version}to a slot behind the scenes - Run the database migrations against the DB
- Verify a healthchecks endpoint on the staged slot
- Swap
{new version}and{old version} - Shut down the
{old version}slot
I'll also be including some firewall rule updates, keeping my web app inaccessible to the wider internet and only opening it to access from my CD pipeline for a very brief window.
Building the Pipeline
The full CircleCI config is located here: github
Now to walk through the major sections of the process.
Two Stages: Build->Package, Package Deploy
There are two builds in my pipeline, the CI stage and the Deploy stage. Let's skip to the end of the config to see that:
# ...
workflows:
build_and_deploy:
jobs:
- build-application
- deploy-application:
requires:
- build-application
filters:
branches:
only: main
The build-application job is responsible for running CI and producing (1) test results, (2) a deployable web package, and (3) the SQL migration tool that can run in the second job. It runs on every single commit to any branch.
The deploy-application job runs only on commits to main branch, and has a dependency on build-application to have been successful.
Building the Application
This build is going to do most of the heavy lifting for he build, prepping for the later deploy stage.
Microsoft provides a number of images for the dotnet sdk, you can find them here: DockerHub: Microsoft DotNet SDK.
That solves a big dependency, but this project also needs a few more, critically nodejs, yarn and zip, so the build starts w/ that setup work:
Image and Dependencies:
build-application:
docker:
- image: mcr.microsoft.com/dotnet/sdk:5.0
environment:
VERSION_NUMBER: 0.0.0.<< pipeline.number >>
JEST_JUNIT_OUTPUT_DIR: "../../../reports/"
JEST_JUNIT_OUTPUT_NAME: "frontend.xml"
JEST_JUNIT_ANCESTOR_SEPARATOR: " > "
JEST_JUNIT_SUITE_NAME: "{filename}"
JEST_JUNIT_CLASSNAME: "{classname}"
JEST_JUNIT_TITLE: "{classname} > {title}"
steps:
- checkout
- run:
name: Install Build/System Dependencies
command:
|
apt-get update -yq \
&& apt-get install curl gnupg -yq \
&& curl -sL https://deb.nodesource.com/setup_14.x | bash \
&& apt-get install nodejs -yq
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
apt update && apt install yarn -yq
apt install zip -yqQuick note on the environment variables: One of these is to allow us to easily stamp the version on the C# assemblies later so they reference back to my CircleCI build number (super useful when debugging captured errors), the other is me trying to beat jest-junit into submission.
Front-end CI: Next I need to get the JS dependencies, run a variety of front-end CI commands to lint and test the code, and then perform a production build of the code so it can be bundled in the Web App package:
- run:
name: Front-end Test
command:
|
cd frontend/react-parcel-ts
yarn install
yarn run ci:lint --output-file ~/reports/eslint.xml
yarn run ci:test
- run:
name: Front-end Build
command:
|
cd frontend/react-parcel-ts
yarn run buildNote: definitely be careful with this piece, one of the commands in the first set is not returning exist codes the way you would expect and will not break the build and I've been distracted instead of fixing it.
The test results are all sent to the ~/results folder to be picked up later in the configuration.
Back-end CI
With the back-end, we need to break things into a few additional stages because dotnet test doesn't output JUnit format, which I tend to standardize on since all build services tend to understand it.
To plug this in, we install an extra tool to convert from TRX format to JUnit and we configure this step to run always, because it's especially important to get these results when the tests fail.
- run:
name: Back-end Test
command:
|
cd backend
cp ./ELA.App.Tests/appsettings.none.json ./ELA.App.Tests/appsettings.json
dotnet test --filter "TestCategory!=Database-Tests" -l trx -l console -v m -r ~/reports/
- run:
name: Back-end Test Results
when: always
command:
|
dotnet tool install -g trx2junit
export PATH="$PATH:/root/.dotnet/tools"
trx2junit ~/reports/*.trxNote that I'm filtering out my integration tests in the first step. I still intend to come back and look into a docker image that includes both SQL Server and DotNet SDK for scenarios like this, but haven't gotten to it yet. A shorter-term solution would be to set up a $5/month database in Azure for integration tests, which is an option I have running for production projects in the meantime.
Make the Packages
Finally, assuming everything was successful to this point, we need to build a web app package that can run in Azure and a Migration Tool package to run in a later build step. I could embed the migration in the Web App and have it run automatically on startup, ala Entity Framework, but prefer to have it separate for future testing purposes.
- run:
name: Build for Release
command:
|
cd backend
dotnet publish ./ELA.App/ELA.App.csproj -c Release /property:Version=$VERSION_NUMBER -o ../app-publish --runtime win-x64
dotnet publish ./ELA.Tools.DatabaseMigration/ELA.Tools.DatabaseMigration.csproj -c Release /property:Version=$VERSION_NUMBER -o ../app-migrate --self-contained -r linux-musl-x64
cd ../app-publish
zip -r ../app-publish.zip *
cd ../app-migrate
chmod 755 ELA.Tools.DatabaseMigration
- persist_to_workspace: # store the built files into the workspace for other jobs.
root: ./
paths:
- app-publish.zip
- app-migrate
- tools/PollHealthcheckI'm running my Azure Web App on windows (I need to change that), so I build the Web App for windows and the Migration tool for linux and make sure it's executable. Those two results plus a local script I wrote to poll the healthchecks are persisted for the next build stage, so that it won't have to do any type of build, dependency downloads, etc.
Everything is broken
Sorry, there's actually one more step, which is to announce problems if the build fails.
- discord/status:
fail_only: true
failure_message: "Application: **$CIRCLE_JOB** job has failed!"
webhook: "${DISCORD_STATUS_WEBHOOK}"
mentions: "@employees"You could easily hook this to slack or another option, in this case I happen to have a private Discord already set up as a central feed "dashboard" for one of my projects. I create the integration in Discord for the channel, grab the webhook URL and add it as an environment variable in CircleCI, and done.
Deploying the Application
The second job is the deployment. For this one, I'm using an image with Azure CLI already built in, microsoft/azure-cli.
Similar to the first job, I also need to add some extra dependencies, namely some dependencies the migration tool requires to run properly via the dotnet command and nodejs. I'm also attaching /tmp/workspace to have access to the persisted packages from the first job.
deploy-application:
docker:
- image: microsoft/azure-cli:latest
environment:
VERSION_NUMBER: 0.0.0.<< pipeline.number >>
steps:
- attach_workspace:
at: /tmp/workspace
- run:
name: Install Dependencies
command:
|
# dependencies to run dotnet db migration tool
apk add libc6-compat
ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2
# dependencies for monitoring health checkout
apk add nodejs Now we can start the deployment, using the Azure CLI to deploy the web package to a staging slot. Once it's deployed, I have a node script to check the healthcheck endpoint so I can ensure everything the configuration is correct before moving to the next step.
- run:
name: Azure Staging Deploy
command:
|
cd /tmp/workspace
# Deploy to staging slot
az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
az webapp deployment source config-zip -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging --src app-publish.zip
az webapp start -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging
# Poll health endpoint, 500ms intervals, 15s timeout
cd /tmp/workspace/tools/PollHealthcheck
node index.js ${AzureStagingUrl}/health 500 15000If the healthcheck works, I know the site is running, can connect to the local database and other dependencies, and is ready to go.
Next, I ping an outside URL to find out the public address I'm running behind, then use the Azure CLI to add a temporary rule so I can run the DB migration against the database. I'm using an encrypted connection, account with specific permissions, etc. but it's still good to keep the door closed as much as possible (and this may remain an experimental option vs running locally within Azure or through a VPN tunnel).
- run:
name: Database Migration
command:
|
ip=$(curl -s https://api.ipify.org)
echo "Adding firewall rule for: $ip"
az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
az sql server firewall-rule create --subscription ${AzureSubscription} -s ${AzureSqlServerName} -g ${AzureResourceGroup} -n CircleCI-Job-$CIRCLE_JOB --start-ip-address $ip --end-ip-address $ip
cd /tmp/workspace/app-migrate
./GDB.Tools.DatabaseMigration --ConnectionString "${AzureDatabaseConnectionString}"
- run:
name: Database Migration - Firewall Cleanup
command:
|
ip=$(curl -s https://api.ipify.org)
echo "Removing firewall rule for: $ip"
az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
az sql server firewall-rule delete --subscription ${AzureSubscription} -s ${AzureSqlServerName} -g ${AzureResourceGroup} -n CircleCI-Job-$CIRCLE_JOB
when: alwaysI make sure to immediately delete the firewall rule again, and this is set for always to ensure it runs even if the earlier step failed.
Last step for the deploy is to double check the health point again, and then make it live.
- run:
name: Azure Production Swap
command:
|
# Poll health endpoint, 500ms intervals, 15s timeout
cd /tmp/workspace/tools/PollHealthcheck
node index.js ${AzureStagingUrl}/health 500 15000
# Swap into production
az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
az webapp deployment slot swap -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging --target-slot production
# Shutdown old version
az webapp stop -g ${AzureResourceGroup} -n ${AzureWebApp} -s stagingOnce the new instance is live and receiving all of the traffic, I turn off the old one and we're done.
The last, last step is reporting. Like the first build, I'm reporting to Discord and, in this case, I'm reporting both Failures and Successes.
- discord/status:
fail_only: true
failure_message: "Application: **$CIRCLE_JOB** deploy has failed!"
webhook: "${DISCORD_STATUS_WEBHOOK}"
mentions: "@employees"
- discord/status:
success_only: true
success_message: "Application: **$CIRCLE_JOB** deployed successfully: $VERSION_NUMBER"
webhook: "${DISCORD_STATUS_WEBHOOK}"And there we are, living in the future. An ASP.Net Core website built and deployed from linux images, including running migrations against MS SQL Server.
Once again, the full config is here, along with the larger project: github
Everything is broken, until it isn't
Of course, the first builds don't go successfully even when we're starting with a config from a parallel working project:
This is normal, in my experience, and I'd probably be worried if it didn't occur.
From here, the next step is "YAML Bashing": a series of 10-30 mini-commits until the config + CI are happy.
See:
But it eventually pays off. We have a working pipeline we can flesh out further as we go and won't have a major project in X months to try and glue CD on at the end.
You may also be interested in these related posts:
