About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
⚠️⚠️⚠️ STG SRG Portal ⚠️⚠️⚠️/Technical article/Use cases of OPA in Terraform

Use cases of OPA in Terraform

Ishikawa Kumo, Service Reliability Group (SRG), Media Management Division@ishikawa_kumo)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is a past post on QiitaUse cases of OPA in TerraformThis is an enhanced version of the article.
 

overview

Open Policy Agent (OPA) has become widely used in recent years for JSON and YAML-based Infrastructure as Code (IaC) management. In particular, the introduction of OPA and GateKeeper to Kubernetes manifests is becoming standardized.
On the other hand, integration with tools like Terraform, which excel at describing complex cloud infrastructure, remains challenging, and there are few actual examples of such integration. While there are use cases in combination with Terraform Cloud and Styra, the use of OPA in a pure Terraform environment (for example, when the backend is S3) is still not very common.
This article will focus on OPA use cases in Terraform environments and practical ways to utilize them.

About OPA and Rego

OPA is a centralized engine for managing Rego policies. It enables the application of consistent policies across multiple systems to input data in JSON format. Rego is a domain-specific language for OPA and is used to express policies on data.
As a declarative programming language, Rego is not well-suited for complex logical processing or handling dynamic JSON structures. Furthermore, developing Rego policies differs significantly from procedural programming languages, sometimes requiring policies to be written from a mathematical logic perspective.

Existing Rego policies and related tools

Starting Rego development without a proper development system in place can lead to significant maintenance burdens. We strongly recommend using Rego projects already created by the community whenever possible.
  • Rego LibraryExamples of policies shared within the OPA community
For best practices with Rego, it's best to refer to the information provided by the developers of OPA.
Furthermore, it is an excellent linter.regalIt's also available, so please feel free to use it.

Points to note about Rego

Below is a summary of common problems and recommendations encountered in Rego policy development.
  1. The variable cannot be reassigned.
      2. name
      sample
  1. Safety Principles
      2. Safety: every variable appearing in the head or in a builtin or inside a negation must appear in a non-negated, non-builtin expression in the body of the rule.
      In short, all variables appearing in the red box below require a clear, non-negative definition.
      not p[x] == 0
      sample
      true
      x
      rego_unsafe_var_error: var x is unsafe
      x
  1. Setting default values
      2. Undefined

Selecting an Input

Since OPA validation is JSON-based, each validation target in Terraform needs to be output as JSON.
Terraform validation generally covers the following types of targets:
  • state JSON
      • tfstate
  • terraform HCL
      • hcl2json

Selection of a Policy Evaluation Engine

tfsec・trivy

The tfsec cli has the ability to implement custom policies written in Rego.
For example, if you implement a simple Rego project on the following S3 instance, you will get the following result.
Terraform
Rego
tfsec --print-rego-input | jq '.google'
From February 2023 onwards,tfsec has become part of the Trivy project.Therefore, it would be better to use Trivy directly in the future. According to the documentation, Trivy also supports custom policies.A more refined formatThis can be defined in the metadata.
v0.49.1

tflint

tflint has a Custom Plugin feature. Development in Go is required. The following repository is a plugin for evaluating OPA Rego via a Go plugin.
0.5.0
Preparation
  1. .tflint.hcl
    1. Policy preparation
        2. .tflint.d/policies/bucket.rego
        policies
    1. Overall directory structure
      execution
      When executing this, please pay close attention to the following environment variables.
      • TFLINT_OPA_POLICY_DIR
        • policy file
      • TFLINT_OPA_TRACE
        • OPA Tracing
      • TFLINT_OPA_TEST
        • OPA Test Mode
      • TFLINT_LOG=debug
        • TFLINT's own logs and OPA Print debugging are also available.

      conftest

      conftest has advantages such as being able to directly read HCL, and it is easier to set up and configure than tfsec or tflint.
      conftest test *.tf -p policies/
      Furthermore, conftest can directly utilize policies that are publicly available on GitHub.
      However, currently Conftest only supports rego's trace mode, which may make developing OPA policies difficult.

      Use cases for Terraform OPA integration

      Defining and applying organization-specific policies

      Organization-specific policies, such as tagging for statistics, specifying multi-environment compatible formats, and assigning IAM to specific resources, can be flexibly configured using OPA. These policies should be carefully considered in line with the organization's actual requirements.
      For example, consider the following:
      All Variables are structured in a map that can be distinguished by each Terrafrom Workspace.
      sample

      Consistency check

      Validation
      Of course, in most cases, I think the problem can be solved by redesigning the variable structure and combining it with validation blocks and pre/postcondition blocks. Rego is well-suited for writing code that controls complex conditions.
      Here are some examples:
      Setting conditions that affect each other between two Variables
      sample
      Additionally, logic that you previously had to verify visually can now be written in Rego.
      • Check if the AWS Network Firewall subnet and RouteTable settings are correct.
      • Does the AWS WAF Rule implement the required rules?

      Part of the automation flow

      While using it as a JSON parser might seem excessive, utilizing OPA as a JSON parser in addition to implementing company-specific policies can be effective.
      However, at present, OPA's output usage is still limited to debugging and error messages. Therefore, we recommend evaluating policies and rules for specific compliance within your CI flow and using only boolean (True/False) results.
      Here are some examples:
      sample
      If a resource is scheduled to be deleted, this policy will generate an appropriate warning message and prompt the user to reconfirm whether the action is intentional.

      In conclusion

      Integrating OPA with the Rego language has great potential to streamline cloud infrastructure management with Terraform. However, Rego development requires considerable effort, which is one of the barriers to adopting OPA in a Terraform environment. This article should have helped you understand the importance of integrating Terraform and OPA and the best practices for doing so. Currently, there are still few examples of Terraform and OPA working together, but the future development of this field is very promising.
       
      SRG is looking for new team members. If you are interested, please contact us here.
       
      SRG runs a podcast where we chat about the latest hot IT technologies and books. We hope you'll enjoy listening to it while you work.