Hi, founder of Anyshift here. We've build a solution for handling issues with Terraform downstream dependencies without additional frameworks (mono or multirepos), and wanted to explain how we've done it.
1.First of all, the key problems we wanted to tackle:
Handling hardcoded values
Handling remote state dependencies
Handling intricate modules (public + private)
we knew that it was possible to do it without adding additional frameworks, by going through the Terraform State Files.
2.Key Assumptions:
Your infra is a graph. To model the infrastructure accurately, we used Neo4j to capture relationships between resources, states, and modules.
All the information you need is within your cloud and code: By parsing both, we could recreate the chain of dependencies and insights without additional overhead.
Our goal was to build a digital twin of the infrastructure. Encompassing code, state, and cloud information to surface and prevent issues early.
3.Our solution:
To handle downstream dependencies we are :
Creating a digital twin of the infra with all the dependencies between IaC code and cloud
For each PR, querying this graph with Cypher (Neo4J query language) to retrieve those dependencies
-> Build an up-to-date Cloud-to-Code graph
i - Understanding Terraform Stat Files
Terraform state files are super rich in term of information, way more than the files. They hold the exact state of deployed resources, including:
Resource types
Unique identifiers
Relationships between modules and their resources
By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities. By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities.
ii- Building this graph using Neo4J
Neo4j allowed us to model complex relationships natively. Unlike relational databases, graph databases are better suited for interconnected data like infrastructure resources.
We modeled infrastructure as nodes (e.g., EC2 instances, VPCs) and relationships (e.g., "CONNECTED_TO," "IN_REGION"). For example:
Nodes: Represent resources like an EC2 instance or a Security Group.
Relationships: Define how resources interact, such as an EC2 instance being attached to a Security Group.
iii- Extracting and Reconciling Data
We developed services to parse state files from multiple repositories, extracting relevant data like resource definitions, unique IDs, and relationships. Once extracted, we reconciled:
Resources from code with resources in the cloud.
Dependencies across repositories, resolving naming conflicts and overlaps.
We also labeled nodes to differentiate between sources (e.g., TF_CODE, TF_STATE) for a clear picture of infrastructure intent vs. reality.
-> Query this graph to retrieve the dependencies before a change
Once the graph is built, we use Cypher, Neo4j's query language, to answer questions about the infrastructure downstream dependencies.
Step 1 : Make a change
We make a change on resource or a module. For instance expanding an IP range in a VPC CIDR.
Step 2 : Cypher query
We're going query the graph of dependencies through different cypher queries to see which downstream dependencies will be affected by this change, potentially in other IaC repositories. For instance this change can affect 2 ECS and 1 security group.
Step 3 : Give back the info in the PR
4. Current limitations:
To handle all the use cases, we are limited by the Cypher queries we define. We want to make it as generic as possible.
It only works with Terraform, and not other IaC frameworks (could work with Pulumi though)
1.First of all, the key problems we wanted to tackle:
Handling hardcoded values
Handling remote state dependencies
Handling intricate modules (public + private)
we knew that it was possible to do it without adding additional frameworks, by going through the Terraform State Files.
2.Key Assumptions:
Your infra is a graph. To model the infrastructure accurately, we used Neo4j to capture relationships between resources, states, and modules.
All the information you need is within your cloud and code: By parsing both, we could recreate the chain of dependencies and insights without additional overhead.
Our goal was to build a digital twin of the infrastructure. Encompassing code, state, and cloud information to surface and prevent issues early.
3.Our solution:
To handle downstream dependencies we are :
Creating a digital twin of the infra with all the dependencies between IaC code and cloud
For each PR, querying this graph with Cypher (Neo4J query language) to retrieve those dependencies
-> Build an up-to-date Cloud-to-Code graph
i - Understanding Terraform Stat Files
Terraform state files are super rich in term of information, way more than the files. They hold the exact state of deployed resources, including:
Resource types
Unique identifiers
Relationships between modules and their resources
By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities. By parsing these state files, we could unify insights across multiple repositories and environments. They acted as a bridge between code-defined intentions and cloud-deployed realities.
ii- Building this graph using Neo4J
Neo4j allowed us to model complex relationships natively. Unlike relational databases, graph databases are better suited for interconnected data like infrastructure resources.
We modeled infrastructure as nodes (e.g., EC2 instances, VPCs) and relationships (e.g., "CONNECTED_TO," "IN_REGION"). For example:
Nodes: Represent resources like an EC2 instance or a Security Group.
Relationships: Define how resources interact, such as an EC2 instance being attached to a Security Group.
iii- Extracting and Reconciling Data
We developed services to parse state files from multiple repositories, extracting relevant data like resource definitions, unique IDs, and relationships. Once extracted, we reconciled:
Resources from code with resources in the cloud.
Dependencies across repositories, resolving naming conflicts and overlaps.
We also labeled nodes to differentiate between sources (e.g., TF_CODE, TF_STATE) for a clear picture of infrastructure intent vs. reality.
-> Query this graph to retrieve the dependencies before a change
Once the graph is built, we use Cypher, Neo4j's query language, to answer questions about the infrastructure downstream dependencies.
Step 1 : Make a change
We make a change on resource or a module. For instance expanding an IP range in a VPC CIDR.
Step 2 : Cypher query
We're going query the graph of dependencies through different cypher queries to see which downstream dependencies will be affected by this change, potentially in other IaC repositories. For instance this change can affect 2 ECS and 1 security group.
Step 3 : Give back the info in the PR
4. Current limitations:
To handle all the use cases, we are limited by the Cypher queries we define. We want to make it as generic as possible.
It only works with Terraform, and not other IaC frameworks (could work with Pulumi though)