March was busy as we added major new features and had Y Combinator's demo day, where Hassan (our CEO) delivered an awesome 60 second pitch on a Zoom call with hundreds of investors!

You can upgrade to the latest version (v0.8.3) to pickup the new features. If you are using v0.7 (or older) please follow the v0.8 migration guide.

🗒️ See diffs in CLI#

A highly requested feature was the ability to see the difference in cost between the current state and the planned state of Terraform projects in the CLI (we already have this feature in CI/CD integrations). Check it out by running infracost diff --help. We have also updated the CI output to make it easier to read.

⚙️ Automated usage-based resource definitions#

Usage-based resources, such as AWS Lambda or Google Cloud Storage, require estimated usage data so Infracost can show costs in the output. You can define these in a YAML file, called a usage file, and use that to get cost estimates for such resources.

Previously you had to create this file manually. You can now use the --sync-usage-file option to generate a new usage file or update an existing one from your Terraform project. This option is a safe sync: it adds any missing resources (with zeros for the usage estimates), it does not overwrite any lines that you have changed in the YAML, and it deletes any resources that are not used in the Terraform project.

😃 Simplified inputs, outputs and config file#

We like it when things are made easy:

• Inputs: a new path flag has been introduced to replace the various methods of running Infracost. You can now simply point Infracost to the path of a Terraform directory, plan binary file, or plan JSON file and it'll just work. This lays some of the groundwork for supporting other IaC tools in the future.
• Outputs: the dashes (-) in the output have been replaced with price descriptions such as Cost depends on usage: $0.20 per 1M requests so you can understand the pricing structure of usage-based resources such as AWS Lambda or Google Cloud Storage.
• Config file: the config file has been updated to support infra-as-code repos that have multiple workspaces and projects. This enables you to combine projects into the same breakdown or diff output. So if a Terraform module or variable is used across workspaces/projects, you can quickly see the cost impact of changing it.

🚀 New Pull request comment options#

We've updated the CI/CD integrations to add a new post_condition option so you can decide when you'd like cost comments to be shown in pull requests. Options include: always leave a cost comment, only comment when there is a change to the cost, or only comment when a percentage threshold has been reached (e.g. more than 5% increase or decrease in costs).

⛅ New cloud resources#

We are working on adding Microsoft Azure to Infracost. This has two steps: the first is to add the prices to the Cloud Pricing API, then to add the resources to the CLI. We completed adding around 300,000 prices from Microsoft Azure to the Cloud Pricing API (step one), and now we're looking for volunteers to add resources to the CLI (step two) before the initial release. Please email ali@infracost.io if you are an Azure user and would like to contribute (basic golang knowledge is required).

We also added support for the following cloud resources:

• AWS: Elastic File System (EFS), EBS GP3 volumes, DX Connection, Route53 Health checks, RDS Serverless
• Google: Memorystore Redis, Cloud Monitoring and Logging, Compute Images and Snapshots

Thanks for being part of the community! We are always looking forward to your feedback.

Recently we released a new infracost diff command inspired by git diff. This shows a diff of monthly cloud cost estimates between the current and planned state of Terraform projects. At a high-level this might seems like a simple exercise of subtracting the current state's cost estimate from the planned state, but cloud costs are rarely that simple to deal with. Let's take a look at the following screenshot to understand some of the nuances.

1. The aws_instance is being changed, which reduces the cost by $125/month (from $743 to $618).

2. AWS EC2 has many different cost components, so to explain what caused the above change, we also flag the sub-resource ebs_block_device[0] that changed (the first attached block device). Underneath it, we show the cost component that caused the actual cost diff, Provisioned IOPS SSD Storage (io1); i.e. reducing the size of that volume can save $1500/year. For those who have done this in production, they know it's not a one-click change as you need to create a new EBS volume and copy over the data. What Infracost enables you to do is to quickly tell how much such a change would save you, then decide if it's worth it.

3. A new aws_lambda_function is being added. Since we don't know how much it's going to be used, we can't show a cost estimate. But we can still show you the prices you'll be charged for: $0.20 per 1M requests and a tiny amount per GB-second. This is a usage-based resource, so if you like you can create a yaml file to provide usage estimates and get a cost estimate. It's hard to think in GB-seconds, so we enable you to input the average request duration and we'll do the math to map that to GB-seconds based on the memory_size of your function and any rounding rules that AWS applies.

   version: 0.1

   resource_usage:

   aws_lambda_function.hello_world:

   monthly_requests: 100000000 # Monthly number of requests.

   request_duration_ms: 250 # Average duration of each request in milliseconds.

   Copy

4. Finally we show a summary at the bottom: the EC2 instance change reduces the cost by 17%, and you can use the above yaml file to do simple what-if analysis on the Lambda costs.

The new infracost diff command is used by our CI/CD integrations and is open source alongside the rest of Infracost. We look forward to hearing what you do with it via GitHub issues or our community Slack!

What is "shift left"?#

"Shift left" has become a popular buzzword for both Software Engineering and DevOps. It means introducing processes earlier in the software development cycle.

The "shift left" principle started with testing. In a traditional waterfall model testing is performed just before release. Shift left testing started it earlier by introducing practices such as Test-Driven Development (TDD) and Behaviour-Driven Development (BDD).

We are now seeing the "shift left" principle applied to other disciplines. Continuous delivery platforms allow engineering teams to deploy frequently and integrate a suite of tools into their cycle. The term DevSecOps has been coined. The idea behind it is to introduce security as early as possible in the software development cycle. This has given rise to a whole ecosystem of tools to help implement this practice. Companies like Snyk and Anchore integrate automated security scanning into DevOps workflows so teams can proactively find and fix vulnerabilities.

Can you shift too far left?#

There's an argument that "shifting left" gives too much work and responsibility to engineering teams. This can be the case if the right tooling is not available.

"Shift left" isn't about performing one-off tasks earlier in the cycle, and this is where the name causes some confusion. It's about introducing processes and automation earlier and performing them continuously throughout development.

When a company introduces a "shift left" mentality it is important that it doesn't impact developer velocity. Tools that help here should fit into developers' workflows and show them the right level of information at the right time.

Will cloud cost shift left?#

Currently cloud costs aren't discussed until they become a problem. A common story is when cloud costs become a problem, companies will set a top-down directive and form a team to reduce their spend by X%. They manage to fix the immediate pain but after six to twelve months the problem returns.

Cloud costs aren't a one-off problem that can be solved. That's why it's inevitable that cloud costs will "shift left". Building a cost-aware engineering team is crucial to keep cloud bills under control.

That's why we built Infracost. We help engineering teams implement this culture of cost-awareness without impacting their velocity. You can integrate Infracost directly into your existing workflow to see cost information throughout your DevOps process. Check out our integrations for instructions.

Here's what we released in February - big thanks to the community contributors! You can upgrade to the latest version (v0.7.20) to pickup these goodies:

🚀 Speed improvements#

The CLI now only runs terraform init if required since Terraform commands aren't the fastest in the world (init usually takes 20+ secs for me, but it depends on how many plugins you have). Furthermore, calls to the Cloud Pricing API have been switched from sequential to parallel. Infracost should run much faster than before.

⚙️ Config file#

Depending on your Terraform workflow, you'll run Infracost with different options. Things can get complicated when you have multiple projects in a repo, each requiring their own Terraform variables. For example, if you have two workspaces and want to see their total cost estimate, you would run something like this:

You can now create an infracost.yml config file in your repo to describe your setup, then just run infracost --config-file infracost.yml.

🌎 Atlantis integration#

Infracost now integrates with Atlantis, which is a popular CI/CD tool that enables Terraform pull request automation.

🗒️ Diff functionality in JSON output#

You can now get the monthly cost diff from the Infracost JSON output, e.g. the following shows the monthly cost is going to be increased by $1530 if the Terraform plan is applied. You can also get totalHourlyCost, or add --no-color=true --log-level=warn if you don't want the spinners/logs/color.

⛅ New cloud resources#

We also shipped support for the following cloud resources:

• AWS: Config, ECS on EC2, EventBridge, Route 53 Resolver, CodeBuild
• Google: Key Management Service (KMS), Google Cloud Functions
• Azure: great progress is being made, stay tuned for exciting news soon

The usage file params for Google Cloud Functions are pretty cool; as shown below you can define 3 simple params and we'll estimate the cost for you, no need for you to decode how function memory maps to GHz-seconds and rounding.

As always, looking forward to your feedback (hello@infracost.io).

Over the last 10 years we’ve seen a lot of shadow IT with AWS being used as the infrastructure provider. The AWS bills were put on company credit cards and expensed. Individual business units could spin up the resources they needed and work in an incredibly agile manner. Unfortunately, for the enterprise as a whole this resulted in shadow spend, less control and security issues. To address these issues, central IT departments built “Self Service” portals with single sign-on (SSO). Business units could still spin up resources as needed, and the enterprise gained some visibility and control.

This caused a new issue. When successful shadow IT projects started consuming more resources, bills went up until the company credit card limits were reached. Then someone would have to go up the ranks to figure out what rules the business unit broke and how to solve it going forward. Self service portals addressed the credit card limit issue by enabling usage, and adding showback and chargeback via spend reports at the end of the month. Optimization was left to the end user to figure out after getting the bill.

A great way to help the end user optimize cloud costs before the showback/chargeback report is to let them see how much resources cost before they are launched. Multiple enterprises have achieved this with Infracost, and I want to share how:

1. Show costs in your self service portal to the end user#

The first step is to show cost estimates in your self service portal. If you are using Terraform to launch resources, this is easy to do by integrating with Infracost. Read our API documentation for more information. Infracost is free and open source.

2. Set expectations in management and finance#

The next step is to set expectations by giving visibility of the cost estimates to management and finance. You can use our HTML output reports to achieve this.

3. Monitor ongoing costs#

Finally, set up alerts and reports for ongoing costs. There are many companies who can help with this, including the cloud providers themselves.

Our users have connected their self service portal in a matter of hours. Their end users immediately begin seeing cost estimates. You can start using Infracost now, or email us for a deeper engagement with our team (hello@infracost.io).

Most cloud providers enable users to set budget alerts on their actual cloud spend. This is a critical safety net as usage-based resources could incur a lot of cost (e.g. data transfer). There is also another safety net that companies should set up, and that is catching significant cost changes to their infrastructure before going live. For example, finding out how much increasing the RAM for a Lambda function costs before putting the new function into production. Usage estimates can also be considered during cost estimation.

Infracost is an open source tool that can be put into CI/CD pipelines (GitHub, GitLab, CircleCI, Bitbucket and Atlantis…) and will leave a comment with the cloud cost implications of changes to your infrastructure-as-code: "this change to your terraform file will increase your cloud bill by 25% next month".

In some cases, you may only want an Infracost comment when a threshold is reached. For example, if the cost implications of the change are minor (e.g. under 3%), then no comment is needed. We have now added this ability into Infracost - from our CI/CD integration docs, select your CI system, and set the percentage_threshold flag.

We hope Infracost can help your enterprise become more cost-aware when it comes to cloud spend, and maybe we can help reduce that 30% cloud waste! ^[1]

As always, if you have any feedback, add an issue to our GitHub repo, join our community Slack channel, or reach out to me directly on Twitter: @hassankhosseni.

[1] Flexera 2020 State of the Cloud Report

Last week I wrote about giving cloud cost estimates to DevOps teams via pull requests as they make changes to infrastructure components. The hope is to create a “Prius Effect” for cloud costs: it was observed that many Prius drivers would drive more efficiently simply because they were presented with immediate feedback on the Prius dashboard.

In this blog, I'd like to answer a question that a user posed to us:

We need to know when significant budget changes are expected so we can plan for them before the money is spent. Billing alerts help us react to unexpected changes but they still can be a surprise. My DevOps now have costs in pull requests. What about my team leads and managers? They can’t go through every pull request to see what the cost changes are.

When you have a single engineer with access to change infrastructure a simple discussion about significant cost changes will suffice. Once your team grows to multiple engineers or multiple teams a process can really help.

Infracost now has a new infracost report command which generates a table or HTML report from multiple Infracost JSON files. The output shows a breakdown of all the cost information in a straightforward format alongside tags. You can then upload these reports to AWS S3 and share them with management and team leads.

Example command:

This is the output you'd get in HTML format. Notice that the filename and all tags are shown:

This is our first solution to this use-case. I'd love to hear your feedback so we can iterate on it and improve it! This is available now. Please see our Infracost Docs: Report section for usage instructions.

If you have any feedback add an issue to our GitHub repo, join our Community Slack channel, or reach out to me directly on Twitter: @hassankhosseni.

Over the last 10 years we have seen more enterprises give direct cloud access to their developers and DevOps teams. There are many ways this has been achieved, from giving admin logins to the cloud console, to having devs create request tickets to the central IT team, to creating a self-service catalogue in which business units can select what they need from a list of pre-approved resources or environments.

What ever model enterprises have chosen, one common thing happens. The cloud bill becomes less predictable. Someone didn't know what they were launching, they didn't turn it off, they didn't consider that when you create one resource, it will auto-create a bunch of other resources. As we shift left with Infrastructure as Code, and further with Serverless, in which code decisions have a direct impact on the cloud bill, we need a better answer than just policing what developers have access to and doing post-bill analysis.

Myself and my co-founders have been working in the cloud cost space for 10 years, and we now want to address this issue. Our theory is that if we give developers and DevOps engineers a clear picture of costs of their infrastructure before they launch, then they will be more mindful about what they launch. Think of it like The Prius Effect in which it was observed and documented that a large subset of Prius drivers would respond to the data on the dashboard by driving in a manner that decreased fuel consumption, but for cloud costs.

We launched infracost.io, an open source tool which looks at a terraform project and creates a cost estimate for the resources directly in the CLI:

You might say “developers don't care about cloud costs since they don't pay for it”. I would say two things:

1. The traction we have seen with Infracost indicates to us that this assumption might be false and that developers do care. They are just unaware of cost implications due to the complexities of cloud pricing models.
2. If a developer doesn't care about cloud costs, what if we put Infracost in the CI pipeline so the cost implications can also be peer reviewed in the pull request? Will a single person in a team pull the cost efficiency up?

This is what Infracost looks like if you put it into the CI pipeline, a comment summarizes the percentage change in the cloud cost due to the changes in the terraform project, as well as a detailed breakdown:

Our Theory#

We are working on validating this theory: can we get better cloud cost efficiency if we give developers clear visibility into potential cloud costs directly in their workflow with no added effort. I'd like to invite you to join this journey, and let's see if this theory is right or not.

1. Install Infracost - it's free and open source.
2. Write a blog, or a tweet about the impacts you see in the short, medium and long term. Do developers talk more about costs? Do they change their code? Are there comments in peer review process about costs? Do others suggest different resources to be used? What else are you seeing?
3. Share the blogs with me, tag me in tweets etc, and I will collate all the results and write another blog with a summary and link to your results.

Cheers!

Infracost helps developers and DevOps engineers get cost estimates from their IaC (Infrastructure as Code). Here's an example of it running:

The complexity of cloud costs keeps increasing - When we were building PlanForCloud in 2012 AWS had just hit 10,000 different pricing points for their services - now there are over 300,000.

We found existing tools fit in too late in the process and are not aimed at the people in control of the infrastructure. It's difficult to get cost estimations when you are building and deploying your services, which often leads to bill shock and no easy way to track down these costs. So we wanted to build a CLI tool that can plug into your existing development and operations processes and bring cost visibility to the engineers.

Currently Infracost supports AWS and Terraform, but we will add support for more cloud vendors (GCP, Azure) and other IaC tools (Pulumi).

Update 25 Nov 2020 : We have now added initial support for Google cloud.

We also want to go beyond just the baseline costs of infrastructure - data transfer costs and other usage-based costs can often be a significant portion of a cloud bill, and are also the hardest to predict and track down. If you have any ideas about the best way to handle these then please reach out to me on Twitter.

Infracost is open source, you can check it out at https://github.com/infracost/infracost.