How to secure your state in azurerm backend

by Julian Weins

Another week, another dive into an interesting issue for working with Terraform. This time I will introduce you into my set of best practises for remote backends — a crucial, often overlooked part in managing your state files. After a brief intro into the security features of remote backends, I will show you how to secure your state file in Azure Blob Storage using the azurerm backend. By the end, you will be able to secure your state file with minimal effort and understand the accompanying terraform script. In the end, I will provide you with a link to my GitHub repository, where you can find the code for this post.

❗ DISCLAIMER: Although I am trying to provide you with the good solution, I am not a security expert. Adjust this configuration to your specific needs & review it with your security team, if necessary.

❗ Please do not hesitate to reach out to me if you have any suggestions on how to improve this project.

Background:

As you might know, Terraform enables you to flexibly choose between different backends to store your state. This is a great feature, as it allows you to choose the backend that best fits your needs. Besides local and Terraform Cloud backend, every hyperscaler (AWS, Azure, GCP) can serve as a backend. However, this flexibility comes with a price. As you store your state file in a remote location, you have to make sure to limit access to authorized users only. This is especially important when you are working in a team. In this post, I will introduce you to the features of this backend and show you how to secure your state with minimal effort. I will use Azure Blob Storage using the azurerm backend.

Important preassumptions:

  • Team: I assume that you are an enterprise user of Terraform. This means you are working in a team and have to ensure that your state file is secure.

  • Network: I assume that you have a virtual network in place, and at least a P2S-VPN connection. This is important, since you most likely do not want to store your state file in a public blob storage container or configure extended security without a hybrid cloud setup.

  • Azure: The most obvious, but I will only refer to Azure services. However, the information are directly transferable to AWS / GCP.

Key concepts:

When we opt for a remote backend, Terraform will store the state file in a remote location. This is a great feature, as it allows you to share the state file with your team. While Terraform Cloud will abstract the security layer away from you, you have to take care of it yourself when using a hyperscaler as a backend.

Here are the options you should consider when using a hyperscaler as a backend:

  • Access control: Ensure access is restricted to authorized individuals only. Employ the innate access control capabilities of the cloud platform. For instance, leverage Azure RBAC to grant specific users or groups access privileges.

  • Encryption: Prioritize securing your state file while it’s at rest. Leverage the integrated encryption functionalities of the cloud provider. For instance, utilize Azure Storage Service Encryption to encrypt your state file while it’s stored.

  • Versioning: Embrace the practice of enabling versioning for your state file. Leverage the inherent versioning features offered by the cloud platform. For instance, employ Azure Blob Versioning to activate versioning for your state file.

  • Backup: Safeguard your state file through consistent backup measures. Leverage the inherent backup capabilities provided by the cloud service. For instance, adopt Azure Blob Backup to routinely back up your state file.

Azure integration:

So let’s dive into how we might integrate these features in Azure. As mentioned above, I will use Azure Blob Storage using the azurerm backend. This is the most common setup, as it is easy to use and provides all the features you need. Azurerm is a recommended / officially supported backend, so it complies with the requirements of Terraform. (thinking of state locking, etc.)

Azure RBAC - Protect your state file from lurking eyes

In the realm of digital security, the concept extends far beyond mere physical barriers, encompassing the meticulous allocation of appropriate permissions to individuals. At the heart of your digital infrastructure lies your state file - a pivotal asset warranting the utmost protection. Through the strategic utilization of Azure’s Role-Based Access Control (RBAC), you are empowered to intricately tailor access privileges, akin to a surgeon’s precision. This formidable instrument affords the capability to meticulously designate access rights to distinct users or specialized groups. As a result, the veil of confidentiality enshrouding your state file is reserved solely for those deemed custodians of its privileged insights.

In the context of your state file, you should consider the following best practices:

  • Least Privilege: Grant the least amount of privileges necessary to perform a task. This is especially important for your state file, as it contains all kind of sensitive data. For instance, you should only grant read access to your state file to your CI/CD pipeline & service principal (Terraform Agent) This way you can ensure that your state file is not modified by unauthorized individuals.

  • Separation of Duties: Ensure that no single individual and service principal has access to all privileges. It is good practise to have multiple service principals in growing teams and terraform configurations. For instance, you should create a separate service principal for your CI/CD pipeline and your Terraform Agent.

  • Rotation: Rotate your service principal credentials on a regular basis. A rule of thumb is to rotate your credentials every 90 days.

These roles are used to grant access to your state file:

  • Storage Blob Data Reader: Read access to your state file (use it as supervisor)
  • Storage Blob Data Contributor: Read & write access to your state file (use it for Terraform Agent)

These roles are used to grant access to your storage account:

  • Storage Account Contributor: Full access to your storage account (not recommended)

Azure Storage Service Encryption - Protect your state file at rest

While the concept of encryption is not new, its significance is often overlooked. Encryption is the process of encoding information in a manner that renders it unreadable to unauthorized individuals. In the context of your state file, encryption is a crucial safeguard to ensure that your state file is protected while it’s at rest. Your state file contains all kind of sensitive data in plain (JSON-encoded) text. Therefore, it is necessary to encrypt your storage account. For this reason you should go for a customer-managed key (CMK) instead of a microsoft-managed key (MMK). MMK provide a good level of security, but you have no control over the encryption key. In contrast, CMK allow you to manage the encryption key yourself.

These are some of the benefits of using a CMK:

  • Full control over the encryption key (especially important for compliance in high security environments)
  • Rotation of the encryption key (e.g. every 90 days)
  • Key Strength (e.g. 2048bit RSA)

Azure Blob Versioning - In case it happens: Roll back to a previous state

In the event of a state file corruption, you will probably want to roll back to a previous state. You might already be familiar with the fact that Terraform creates a backup of your state file. However, this backup is only created with local backend. While Terraform Cloud will once again keeps a backup of your state file, you have to take care of it yourself when using a hyperscaler backend.

Therefore, you have to enable versioning for your state file. This way you can ensure your state file can be rolled back or restored in case of any unexpected issues.

Azure Blob Backup - When a zone goes down: Restore your state file

As your terraform configurations grow in size and complexity, so does the importance of reliable access to your state files. In the event of a zone going down, you will probably want to keep your state file accessible. In analogy to your app being deployed in multiple zones, you should also consider storing your state file in multiple zones. This way you can ensure your CI/CD pipelines are not interrupted by a zone going down.

Consider implementing your state storage with GRS. This way you can ensure your state file is stored and kept synced in multiple zones.

Hands-on examples:

In the following section I will provide you with an example that I used in a recent project. I will show you how to implement the concepts mentioned above in Azure using Terraform:

Prerequisites:

I assume that you have a resource group, key vault and a Key in Place. Furthermore, I will use a Managed Identity specifically for encryption. This is not a requirement, but I recommend it for security reasons. (Separation of concerns)

In the following example I will iteratively build up the configuration for a secure storage account. I will start with the most basic configuration and then subsequently add the features mentioned above.

1. Create a storage Account

For our purpose, we will create a storage account that should have the following properties:

  • Replication: GRS
  • Encryption: CMK
  • Versioning: Enabled
  • Access: Private (no public access, deny all access from Portal, CLI, etc. unless explicitly allowed)
resource "azurerm_storage_account" "storage_account" {
  name                     = "unique-storage-account-name"
  resource_group_name             = "my-resource-group"
  location                        = "westeurope"
  account_tier                    = "Standard"
  account_kind                    = "StorageV2"
  account_replication_type        = "GRS"
  min_tls_version                 = "TLS1_2"
  allow_nested_items_to_be_public = false
  enable_https_traffic_only = true
  tags = {
    environment = "dev"
    occasion    = "terraform tuesday"
  }
}

With this configuration, we have created a storage account with GRS replication, TLS 1.2 and no public access. However, we still need to enable versioning and encryption.

2. Enable encryption

For encryption, we will use a customer-managed key (CMK) from our key vault. For this purpose, we will use a previously created managed identity. We reference the managed identity as well as the key used for our storage account encryption.

Note: I shortened the code for the sake of simplicity.

resource "azurerm_storage_account" "storage_account" {
  name                     = "unique-storage-account-name"
  resource_group_name      = "my-resource-group"

  # ...Step 1

  identity {
    type = "UserAssigned"
    identity_ids = [
      # Reference to the managed identity
      # Skip this with type "SystemAssigned"
      data.azurerm_user_assigned_identity.mi.id 
    ]
  }

  # Enable encryption at rest using a customer managed key
  customer_managed_key {
    key_vault_key_id          = data.azurerm_key_vault_key.cmk.id
    user_assigned_identity_id = data.azurerm_user_assigned_identity.mi.id
  }
}

3. Enable versioning

For versioning, we will dive into the blob_properties. This is where we can enable versioning for our storage account. We will also enable soft delete, which allows us to restore deleted blobs.

Note: The retention period must be less than the deletion period. (e.g. 7 days retention, 8 days deletion) Otherwise, you will get an error from the Azure API. Makes sense, right? 😉

resource "azurerm_storage_account" "storage_account" {
  name                     = "unique-storage-account-name"
  resource_group_name      = "my-resource-group"

  # ...Step 1

  # ...Step 2

  # Enable soft delete and versioning
  blob_properties {
    delete_retention_policy {
      days    = 8
    }
    restore_policy {
      days    = 7
    }
    versioning_enabled = true
    change_feed_enabled      = true
  }
  

}

4. Additional Fine-tuning

Even though it might not comply with the IaC principle, I would like to add a lifecycle block to our storage account. This way we are able to adjust the periods for soft delete and versioning, enabling us to save costs. In my opinion, this is a acceptable tradeoff, since estimating the amount of changes to your state files can not be estimated upfront. It is pretty straightforward, so I will not go into detail here.

resource "azurerm_storage_account" "storage_account" {
  name                     = "unique-storage-account-name"
  resource_group_name      = "my-resource-group"

  # ...Step 1

  # ...Step 2

  # ...Step 3

  # Additional fine-tuning
  lifecycle {
    prevent_destroy = true
    ignore_changes = [
      blob_properties[0].delete_retention_policy[0].days,
      blob_properties[0].restore_policy[0].days
    ]
  }
}

Or just simply ignore all changes to the blob_properties block:

resource "azurerm_storage_account" "storage_account" {
  name                     = "unique-storage-account-name"
  resource_group_name      = "my-resource-group"

  # ...Step 1

  # ...Step 2

  # ...Step 3

  # Additional fine-tuning
  lifecycle {
    prevent_destroy = true
    ignore_changes = [
      tags,
      blob_properties
    ]
  }
}

5. Create a Storage Container

Now that we have created our storage account, we can create a storage container, to host our state file(s). With the right access level (here private), we can use previously listed RBAC to prevent exposure through the Azure Portal, CLI, etc.

resource "azurerm_storage_container" "storage_container" {
  name                  = "unique-storage-container-name"
  storage_account_name  = azurerm_storage_account.storage_account.name
  container_access_type = "private"
}

Conclusion:

In conclusion, securing your Terraform state file when using remote backends, such as Azure Blob Storage with the azurerm backend, is a critical aspect of maintaining the integrity and confidentiality of your infrastructure’s configuration. By following the best practices outlined in this guide, you can ensure that your state file remains protected, even in a team-based and dynamic cloud environment. Let’s recap the key takeaways:

  1. Access Control: Utilize Azure’s Role-Based Access Control (RBAC) to carefully allocate access privileges to authorized individuals or groups. Adhere to the principles of least privilege, separation of duties, and regular rotation of credentials to minimize the risk of unauthorized access.

  2. Encryption: Prioritize the encryption of your state file at rest. Opt for customer-managed keys (CMKs) to maintain control over the encryption keys, enhance security, and enable key rotation. This safeguards sensitive information within your state file from potential threats.

  3. Versioning: Enable versioning for your state file using Azure Blob Versioning. This ensures the ability to roll back or restore your state file in case of corruption or unintended changes, providing a safety net for your infrastructure’s configuration.

  4. Backup and Redundancy: Implement Geo-Redundant Storage (GRS) to store your state file in multiple zones, mitigating the impact of zone outages. This strategy ensures the accessibility of your state file, critical for maintaining the continuity of your CI/CD pipelines and operations.

By integrating these practices into your setup using the azurerm backend in Terraform, you’re taking a proactive approach to safeguarding your infrastructure’s state file. This not only enhances the security of your configurations but also contributes to the overall resilience of your cloud-based systems.

As you continue to manage and evolve your infrastructure, a well-secured and well-maintained state file will serve as a foundation for successful and confident cloud operations. 🤓

Like always, I hope you enjoyed this article and learned something new. If you have any questions, feel free to reach out to me on LinkedIn. I am always happy to help! 😊

Code

Github Repository

References

AzureRM Provider Documentation Azure Storage Account Documentation