Eli Weinstock-Herman

Deploying an ASP.Net website to Azure via CircleCI

January 30, 2021 ▪ technical posts

Two of my side projects are ASP.Net Core applications that run in Azure with Azure SQL Databases. These days, setting up a deployment pipeline is something I do with every project I start that will be deployed somewhere.

Everything I'm outlining in this post (and sample code I'm pointing to) is just shy of production level. It's not a drop-in solution, but there shouldn't be that many gotchas if you borrowed heavily from these as a starting point.


Table of Contents


Outline of the delivery pipeline

This is a basic starting pipeline for my projects. The intent is to do initial verifications of changes and then safely deliver the changes to "production".

Planning

My first step is to plan out what the minimum looks like, which typically means:

  1. What is going to be running in production
  2. How does it safely get to production
  3. What basic verifications can I run to fail fast if something is broken

Here are the details for the sample application I'll be linking to:

  • ASP.Net Core 5 web app
  • SPA front-ends, using React, TypeScript, SCSS, and Parcel
  • MS SQL Server database with scripted migrations
  • Hosting is Azure Web Apps and Azure SQL
  • Notifications to private channels on a Discord server

Project folder: circleci, frontend, backend, and database

I will have two workflows:

  • CI for every commit on any branch
  • CI -> Continuous Deployment for my main branch

The deployment process is going to be a blue/green deployment, and it will work like this:

  • Deploy {new version} to a slot behind the scenes
  • Run the database migrations against the DB
  • Verify a healthchecks endpoint on the staged slot
  • Swap {new version} and {old version}
  • Shut down the {old version} slot

A diagram for the two builds and their steps

I'll also be including some firewall rule updates, keeping my web app inacciessble to the wider internet and only opening it to access from my CD pipeline for a very brief window.

Building the Pipeline

The full CircleCI config is located here: github

Now to walk through the major sections of the process.

Two Stages: Build->Package, Package Deploy

There are two builds in my pipeline, the CI stage and the Deploy stage. Let's skip to the end of the config to see that:

# ...
workflows:
  build_and_deploy:
    jobs:
      - build-application
      - deploy-application:
          requires:
            - build-application
          filters: 
            branches:
              only: main

The build-application job is responsible for running CI and producing (1) test results, (2) a deployable web package, and (3) the SQL migration tool that can run in the second job. It runs on every single commit to any branch.

The deploy-application job runs only on commits to main branch, and has a dependency on build-application to have been successful.

Building the Application

This build is going to do most of the heavy lifting for he build, prepping for the later deploy stage.

A diagram for this build and the steps

Microsoft provides a number of images for the dotnet sdk, you can find them here: DockerHub: Microsoft DotNet SDK.

That solves a big dependency, but this project also needs a few more, critically nodejs, yarn and zip, so the build starts w/ that setup work:

Image and Dependencies:

build-application:
    docker: 
      - image: mcr.microsoft.com/dotnet/sdk:5.0
    environment:
      VERSION_NUMBER: 0.0.0.<< pipeline.number >>
      JEST_JUNIT_OUTPUT_DIR: "../../../reports/"
      JEST_JUNIT_OUTPUT_NAME: "frontend.xml"
      JEST_JUNIT_ANCESTOR_SEPARATOR: " > "
      JEST_JUNIT_SUITE_NAME: "{filename}"
      JEST_JUNIT_CLASSNAME: "{classname}"
      JEST_JUNIT_TITLE: "{classname} > {title}"
    steps:
      - checkout
      - run: 
          name: Install Build/System Dependencies
          command:
            |
            apt-get update -yq \
              && apt-get install curl gnupg -yq \
              && curl -sL https://deb.nodesource.com/setup_14.x | bash \
              && apt-get install nodejs -yq
            curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
            echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
            apt update && apt install yarn -yq
            apt install zip -yq

Quick note on the environment variables: One of these is to allow us to easily stamp the version on the C# assemblies later so they reference back to my CircleCI build number (super useful when debugging captured errors), the other is me trying to beat jest-junit into submission.

Front-end CI: Next I need to get the JS dependencies, run a variety of front-end CI commands to lint and test the code, and then perform a production build of the code so it can be bundled in the Web App package:

- run: 
    name: Front-end Test
    command: 
      |
      cd frontend/react-parcel-ts
      yarn install
      yarn run ci:lint --output-file ~/reports/eslint.xml
      yarn run ci:test

- run: 
    name: Front-end Build
    command: 
      |
      cd frontend/react-parcel-ts
      yarn run build

Note: definitely be careful with this piece, one of the commands in the first set is not returning exist codes the way you would expect and will not break the build and I've been distracted instead of fixing it.

The test results are all sent to the ~/results folder to be picked up later in the configuration.

Back-end CI

With the back-end, we need to break things into a few additional stages because dotnet test doesn't output JUnit format, which I tend to standardize on since all build services tend to understand it.

To plug this in, we install an extra tool to convert from TRX format to JUnit and we configure this step to run always, because it's especially important to get these results when the tests fail.

- run:
    name: Back-end Test
    command:
    |
    cd backend
    cp ./ELA.App.Tests/appsettings.none.json ./ELA.App.Tests/appsettings.json
    dotnet test --filter "TestCategory!=Database-Tests" -l trx -l console -v m -r ~/reports/

- run:
    name: Back-end Test Results
    when: always
    command:
    |
    dotnet tool install -g trx2junit
    export PATH="$PATH:/root/.dotnet/tools"
    trx2junit ~/reports/*.trx

Note that I'm filtering out my integration tests in the first step. I still intend to come back and look into a docker image that includes both SQL Server and DotNet SDK for scenarios like this, but haven't gotten to it yet. A shorter-term solution would be to set up a $5/month database in Azure for integration tests, which is an option I have running for production projects in the meantime.

Make the Packages

Finally, assuming everything was successful to this point, we need to build a web app package that can run in Azure and a Migration Tool package to run in a later build step. I could embed the migration in the Web App and have it run automatically on startup, ala Entity Framework, but prefer to have it separate for future testing purposes.

- run:
    name: Build for Release
    command: 
    |
    cd backend
    dotnet publish ./ELA.App/ELA.App.csproj -c Release /property:Version=$VERSION_NUMBER -o ../app-publish --runtime win-x64
    dotnet publish ./ELA.Tools.DatabaseMigration/ELA.Tools.DatabaseMigration.csproj -c Release /property:Version=$VERSION_NUMBER -o ../app-migrate --self-contained -r linux-musl-x64
    cd ../app-publish
    zip -r ../app-publish.zip *
    cd ../app-migrate
    chmod 755 ELA.Tools.DatabaseMigration

- persist_to_workspace: # store the built files into the workspace for other jobs.
    root: ./
    paths:
    - app-publish.zip
    - app-migrate
    - tools/PollHealthcheck

I'm running my Azure Web App on windows (I need to change that), so I build the Web App for windows and the Migration tool for linux and make sure it's executable. Those two results plus a local script I wrote to poll the healthchecks are persisted for the next build stage, so that it won't have to do any type of build, dependency downloads, etc.

Everything is broken

Sorry, there's actually one more step, which is to announce problems if the build fails.

- discord/status:
    fail_only: true
    failure_message: "Application: **$CIRCLE_JOB** job has failed!"
    webhook: "${DISCORD_STATUS_WEBHOOK}"
    mentions: "@employees"

You could easily hook this to slack or another option, in this case I happen to have a private Discord already set up as a central feed "dashboard" for one of my projects. I create the integration in Discord for the channel, grab the webhook URL and add it as an environment variable in CircleCI, and done.

Deploying the Application

The second job is the deployment. For this one, I'm using an image with Azure CLI already built in, microsoft/azure-cli.

A diagram for this build and the steps

Similar to the first job, I also need to add some extra dependencies, namely some dependencies the migration tool requires to run properly via the dotnet command and nodejs. I'm also attaching /tmp/workspace to have access to the persisted packages from the first job.

deploy-application:
    docker:
      - image: microsoft/azure-cli:latest
    environment:
      VERSION_NUMBER: 0.0.0.<< pipeline.number >>
    steps:
      - attach_workspace:
          at: /tmp/workspace

      - run: 
          name: Install Dependencies
          command:
            |
            # dependencies to run dotnet db migration tool
            apk add libc6-compat
            ln -s /lib/libc.musl-x86_64.so.1 /lib/ld-linux-x86-64.so.2
            # dependencies for monitoring health checkout
            apk add nodejs

Now we can start the deployment, using the Azure CLI to deploy the web package to a staging slot. Once it's deployed, I have a node script to check the healthcheck endpoint so I can ensure everything the configuration is correct before moving to the next step.

- run: 
    name: Azure Staging Deploy
    command:
        |
        cd /tmp/workspace
        # Deploy to staging slot
        az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
        az webapp deployment source config-zip -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging --src app-publish.zip
        az webapp start -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging
        # Poll health endpoint, 500ms intervals, 15s timeout
        cd /tmp/workspace/tools/PollHealthcheck
        node index.js ${AzureStagingUrl}/health 500 15000

If the healthcheck works, I know the site is running, can connect to the local database and other dependencies, and is ready to go.

Next, I ping an outside URL to find out the public address I'm running behind, then use the Azure CLI to add a temporary rule so I can run the DB migration against the database. I'm using an encrypted connection, account with specific permissions, etc. but it's still good to keep the door closed as much as possible (and this may remain an experimental option vs running locally within Azure or through a VPN tunnel).

- run:
    name: Database Migration
    command: 
    |
    ip=$(curl -s https://api.ipify.org)
    echo "Adding firewall rule for: $ip"
    az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
    az sql server firewall-rule create --subscription ${AzureSubscription} -s ${AzureSqlServerName} -g ${AzureResourceGroup} -n CircleCI-Job-$CIRCLE_JOB --start-ip-address $ip --end-ip-address $ip
    cd /tmp/workspace/app-migrate
    ./GDB.Tools.DatabaseMigration --ConnectionString "${AzureDatabaseConnectionString}"

- run:
    name: Database Migration - Firewall Cleanup
    command: 
    |
    ip=$(curl -s https://api.ipify.org)
    echo "Removing firewall rule for: $ip"
    az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
    az sql server firewall-rule delete --subscription ${AzureSubscription} -s ${AzureSqlServerName} -g ${AzureResourceGroup} -n CircleCI-Job-$CIRCLE_JOB
    when: always

I make sure to immediately delete the firewall rule again, and this is set for always to ensure it runs even if the earlier step failed.

Last step for the deploy is to double check the health point again, and then make it live.

- run: 
    name: Azure Production Swap
    command:
    |
    # Poll health endpoint, 500ms intervals, 15s timeout
    cd /tmp/workspace/tools/PollHealthcheck
    node index.js ${AzureStagingUrl}/health 500 15000
    # Swap into production
    az login --service-principal -u http://${AzureServicePrincipal} -p ${AzurePassword} --tenant ${AzureTenant}
    az webapp deployment slot swap -g ${AzureResourceGroup} -n  ${AzureWebApp} -s staging --target-slot production
    # Shutdown old version
    az webapp stop -g ${AzureResourceGroup} -n ${AzureWebApp} -s staging

Once the new instance is live and receiving all of the traffic, I turn off the old one and we're done.

The last, last step is reporting. Like the first build, I'm reporting to Discord and, in this case, I'm reporting both Failures and Successes.

- discord/status:
    fail_only: true
    failure_message: "Application: **$CIRCLE_JOB** deploy has failed!"
    webhook: "${DISCORD_STATUS_WEBHOOK}"
    mentions: "@employees"

- discord/status:
    success_only: true
    success_message: "Application: **$CIRCLE_JOB** deployed successfully: $VERSION_NUMBER"
    webhook: "${DISCORD_STATUS_WEBHOOK}"

And there we are, living in the future. An ASP.Net Core website built and deployed from linux images, including running migrations against MS SQL Server.

Once again, the full config is here, along with the larger project: github

Everything is broken, until it isn't

Of course, the first builds don't go successfully even when we're starting with a config from a parallel working project:

Discord build failure from CircleCI

This is normal, in my experience, and I'd probably be worried if it didn't occur.

From here, the next step is "YAML Bashing": a series of 10-30 mini-commits until the config + CI are happy.

See:

YAML Bashing my CircleCI build - lots of git commits

But it eventually pays off. We have a working pipeline we can flesh out further as we go and won't have a major project in X months to try and glue CD on at the end.

YAML Bashing my CircleCI build - lots of git commits


You may also be interested in these related posts:


Share: