Data Exports

Data exports are required for Infracost to see individual billing line items, cost optimization recommendations, and S3 usage metrics — complete the AWS CLI setup first, then follow this guide to create them, or bring your own existing exports instead. This sets up the same exports as the Terraform module and CloudFormation template.

Everything in this guide is performed on the AWS organization's management account, in the us-east-1 region — BCM Data Exports only exists in us-east-1. The data it exports still covers every region you use; only the export configuration and S3 buckets themselves must live in us-east-1. Pass --region us-east-1 to every command below (or set AWS_DEFAULT_REGION=us-east-1).

note

AWS can take up to 24 hours to populate the buckets with the first data exports after you complete this setup.

Prerequisites

Enable Cost Optimization Hub and Compute Optimizer

The Cost Optimization Hub recommendations export requires your AWS Organization's management account to be opted in to both AWS Compute Optimizer and AWS Cost Optimization Hub, org-wide.

aws compute-optimizer update-enrollment-status \
  --status Active \
  --include-member-accounts \
  --region us-east-1

aws cost-optimization-hub update-enrollment-status \
  --status Active \
  --include-member-accounts \
  --region us-east-1

AWS documents that enrollment can take up to 24 hours to take effect, though it is often faster.

Enable trusted access for S3 Storage Lens

aws organizations enable-aws-service-access --service-principal storage-lens.s3.amazonaws.com

This is required for cross-account visibility — the Storage Lens configuration created below won't be able to see member-account buckets without it.

Get your Organization ARN and trusted service principals

aws organizations describe-organization --query 'Organization.Arn' --output text

aws organizations list-aws-service-access-for-organization \
  --query 'EnabledServicePrincipals[].ServicePrincipal' --output text

The second command's output must include storage-lens.s3.amazonaws.com — if it's missing, that means the previous step wasn't applied yet (or hasn't propagated). Keep the Organization ARN handy for Step 4.

Step 1: Create the export S3 buckets

Create two private S3 buckets in us-east-1 — one for BCM Data Exports, one for S3 Storage Lens exports. Substitute your account ID for 111111111111 throughout this guide.

aws s3api create-bucket \
  --bucket infracost-bcm-exports-111111111111 \
  --region us-east-1

aws s3api create-bucket \
  --bucket infracost-storagelens-exports-111111111111 \
  --region us-east-1

Block public access on both:

for BUCKET in infracost-bcm-exports-111111111111 infracost-storagelens-exports-111111111111; do
  aws s3api put-public-access-block \
    --bucket "$BUCKET" \
    --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
done

Leave default encryption as SSE-S3 (the default) — or see Optional: encrypt with a KMS key below to use your own key instead.

Add a lifecycle rule to each bucket

Save as lifecycle.json:

{
  "Rules": [
    {
      "ID": "intelligent-tiering",
      "Status": "Enabled",
      "Filter": {},
      "Transitions": [
        { "Days": 0, "StorageClass": "INTELLIGENT_TIERING" }
      ],
      "Expiration": { "Days": 365 },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

for BUCKET in infracost-bcm-exports-111111111111 infracost-storagelens-exports-111111111111; do
  aws s3api put-bucket-lifecycle-configuration \
    --bucket "$BUCKET" \
    --lifecycle-configuration file://lifecycle.json
done

Add a bucket policy to each bucket

Save as bcm-bucket-policy.json, substituting your account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowBCMDataExportsWrite",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "bcm-data-exports.amazonaws.com",
          "billingreports.amazonaws.com"
        ]
      },
      "Action": ["s3:GetBucketPolicy", "s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::infracost-bcm-exports-111111111111",
        "arn:aws:s3:::infracost-bcm-exports-111111111111/*"
      ],
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "111111111111" }
      }
    },
    {
      "Sid": "DenyNonSSLRequests",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::infracost-bcm-exports-111111111111",
        "arn:aws:s3:::infracost-bcm-exports-111111111111/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    }
  ]
}

Save as storagelens-bucket-policy.json, substituting your account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStorageLensDataExportsWrite",
      "Effect": "Allow",
      "Principal": {
        "Service": "storage-lens.s3.amazonaws.com"
      },
      "Action": ["s3:PutObject", "s3:PutObjectAcl"],
      "Resource": [
        "arn:aws:s3:::infracost-storagelens-exports-111111111111",
        "arn:aws:s3:::infracost-storagelens-exports-111111111111/*"
      ],
      "Condition": {
        "StringEquals": { "aws:SourceAccount": "111111111111" }
      }
    },
    {
      "Sid": "DenyNonSSLRequests",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::infracost-storagelens-exports-111111111111",
        "arn:aws:s3:::infracost-storagelens-exports-111111111111/*"
      ],
      "Condition": {
        "Bool": { "aws:SecureTransport": "false" }
      }
    }
  ]
}

aws s3api put-bucket-policy --bucket infracost-bcm-exports-111111111111 --policy file://bcm-bucket-policy.json
aws s3api put-bucket-policy --bucket infracost-storagelens-exports-111111111111 --policy file://storagelens-bucket-policy.json

Step 2: Create the BCM Data Exports service-linked role

Creating the exports below requires the AWSServiceRoleForBCMDataExports service-linked role to exist in the management account:

aws iam create-service-linked-role --aws-service-name bcm-data-exports.amazonaws.com

caution

If your account has previously used BCM Data Exports, this role may already exist — this command returns an InvalidInput error saying so, which is safe to ignore.

Step 3: Create the FOCUS and Cost Optimization Hub exports

Save as focus-export.json, substituting your bucket name:

{
  "Name": "InfracostDailyFocusExport",
  "DataQuery": {
    "QueryStatement": "SELECT AvailabilityZone, BilledCost, BillingAccountId, BillingAccountName, BillingAccountType, BillingCurrency, BillingPeriodEnd, BillingPeriodStart, CapacityReservationId, CapacityReservationStatus, ChargeCategory, ChargeClass, ChargeDescription, ChargeFrequency, ChargePeriodEnd, ChargePeriodStart, CommitmentDiscountCategory, CommitmentDiscountId, CommitmentDiscountName, CommitmentDiscountQuantity, CommitmentDiscountType, CommitmentDiscountStatus, CommitmentDiscountUnit, ConsumedQuantity, ConsumedUnit, ContractedCost, ContractedUnitPrice, EffectiveCost, InvoiceId, InvoiceIssuerName, ListCost, ListUnitPrice, PricingCategory, PricingCurrency, PricingCurrencyContractedUnitPrice, PricingCurrencyEffectiveCost, PricingCurrencyListUnitPrice, PricingQuantity, PricingUnit, ProviderName, PublisherName, RegionId, RegionName, ResourceId, ResourceName, ResourceType, ServiceCategory, ServiceName, ServiceSubcategory, SkuId, SkuPriceDetails, SkuPriceId, SkuMeter, SubAccountId, SubAccountName, SubAccountType, Tags, x_Discounts, x_Operation, x_ServiceCode FROM FOCUS_1_2_AWS",
    "TableConfigurations": {
      "FOCUS_1_2_AWS": { "TIME_GRANULARITY": "DAILY" }
    }
  },
  "DestinationConfigurations": {
    "S3Destination": {
      "S3Bucket": "infracost-bcm-exports-111111111111",
      "S3Prefix": "",
      "S3Region": "us-east-1",
      "S3OutputConfigurations": {
        "OutputType": "CUSTOM",
        "Format": "PARQUET",
        "Compression": "PARQUET",
        "Overwrite": "OVERWRITE_REPORT"
      }
    }
  },
  "RefreshCadence": { "Frequency": "SYNCHRONOUS" }
}

aws bcm-data-exports create-export --export file://focus-export.json --region us-east-1 --no-cli-pager

Save as cost-optimization-export.json, substituting your bucket name:

{
  "Name": "InfracostDailyCostOptimizationExport",
  "DataQuery": {
    "QueryStatement": "SELECT account_id, account_name, recommendation_id, resource_arn, region, current_resource_type, recommended_resource_type, action_type, current_resource_summary, recommended_resource_summary, estimated_monthly_cost_before_discount, estimated_monthly_cost_after_discount, estimated_monthly_savings_before_discount, estimated_monthly_savings_after_discount, estimated_savings_percentage_before_discount, estimated_savings_percentage_after_discount, currency_code, restart_needed, rollback_possible, implementation_effort, tags, current_resource_details, recommended_resource_details, recommendation_source, recommendation_lookback_period_in_days, last_refresh_timestamp FROM COST_OPTIMIZATION_RECOMMENDATIONS",
    "TableConfigurations": {
      "COST_OPTIMIZATION_RECOMMENDATIONS": {
        "INCLUDE_ALL_RECOMMENDATIONS": "TRUE",
        "FILTER": "{}"
      }
    }
  },
  "DestinationConfigurations": {
    "S3Destination": {
      "S3Bucket": "infracost-bcm-exports-111111111111",
      "S3Prefix": "",
      "S3Region": "us-east-1",
      "S3OutputConfigurations": {
        "OutputType": "CUSTOM",
        "Format": "PARQUET",
        "Compression": "PARQUET",
        "Overwrite": "OVERWRITE_REPORT"
      }
    }
  },
  "RefreshCadence": { "Frequency": "SYNCHRONOUS" }
}

aws bcm-data-exports create-export --export file://cost-optimization-export.json --region us-east-1 --no-cli-pager

Exact settings reference

Setting	FOCUS export	Cost Optimization Hub export
Export type	FOCUS 1.2	Cost Optimization Hub recommendations
Columns	Include all columns	Include all recommendations
Compression type and file format	Parquet - Parquet	Parquet - Parquet
Time granularity	Daily	N/A

Step 4: Create the S3 Storage Lens configuration

Save as storage-lens-config.json, substituting your account ID and Organization ARN from the prerequisites:

{
  "Id": "InfracostStorageLensExport",
  "AccountLevel": {
    "ActivityMetrics": { "IsEnabled": true },
    "AdvancedCostOptimizationMetrics": { "IsEnabled": true },
    "BucketLevel": {
      "ActivityMetrics": { "IsEnabled": true },
      "AdvancedCostOptimizationMetrics": { "IsEnabled": true }
    }
  },
  "AwsOrg": {
    "Arn": "arn:aws:organizations::111111111111:organization/o-abc123xyz"
  },
  "DataExport": {
    "S3BucketDestination": {
      "AccountId": "111111111111",
      "Arn": "arn:aws:s3:::infracost-storagelens-exports-111111111111",
      "Format": "Parquet",
      "OutputSchemaVersion": "V_1"
    }
  },
  "IsEnabled": true
}

aws s3control put-storage-lens-configuration \
  --account-id 111111111111 \
  --config-id InfracostStorageLensExport \
  --storage-lens-configuration file://storage-lens-config.json \
  --region us-east-1

Exact settings reference

Setting	Value
Status	Enable
Bucket coverage	AWS Organization
Metrics selection	Advanced metrics and recommendations — Activity metrics + Advanced cost-optimization metrics, account level and bucket level
Metrics export format	Parquet

Data ownership and lifecycle

Your AWS account retains ownership of the data. Infracost is granted solely read-only access to the export buckets — you granted this via the infracost-management-account-readonly policy when you set up the cross-account role, plus the S3 access policy below.

Each data exports bucket has a 365-day object lifecycle policy (configured in Step 1) with intelligent tiering applied to reduce storage costs.

note

The data exports are read periodically by Infracost, which may incur minor AWS query costs (~$0.0004 per 1,000 S3 requests). S3 storage costs also apply.

Step 5: Grant Infracost's role access to the export buckets

Save as s3-access-policy.json, substituting your account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InfracostS3BucketAccessExports",
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::infracost-bcm-exports-111111111111",
        "arn:aws:s3:::infracost-storagelens-exports-111111111111"
      ]
    },
    {
      "Sid": "InfracostS3ObjectAccessExports",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:HeadObject"],
      "Resource": [
        "arn:aws:s3:::infracost-bcm-exports-111111111111/*",
        "arn:aws:s3:::infracost-storagelens-exports-111111111111/*"
      ]
    }
  ]
}

aws iam put-role-policy \
  --role-name infracost-readonly \
  --policy-name infracost-s3-access \
  --policy-document file://s3-access-policy.json \
  --no-cli-pager

Optional: provide custom data exports instead

If you prefer to manage your own data exports, or want to provide additional data to Infracost, you can grant access to your own bucket(s) instead of (or alongside) the ones above — add statements for each extra bucket to the same infracost-s3-access policy, following the same GetBucketLocation/ListBucket/GetObject pattern. Contact support@infracost.io to confirm your use case is supported.

If you are creating the FOCUS billing data export yourself (as in Step 3 above) but managing it outside of this guide's naming conventions, use these settings so Infracost can still ingest it:

Manual FOCUS settings

Setting	Value
Export type	FOCUS 1.2
Columns	Include all columns
Compression type and file format	Parquet - Parquet
Time granularity	Daily

Provide the following to Infracost so it can be added to your integration:

Field	Example
AWS S3 bucket ARN	arn:aws:s3:::my-custom-export

Optional: encrypt export buckets with a KMS key

To use a customer-managed KMS key instead of the default SSE-S3 (AES-256) encryption:

for BUCKET in infracost-bcm-exports-111111111111 infracost-storagelens-exports-111111111111; do
  aws s3api put-bucket-encryption \
    --bucket "$BUCKET" \
    --server-side-encryption-configuration '{
      "Rules": [{
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:us-east-1:111111111111:key/mrk-abc123"
        },
        "BucketKeyEnabled": true
      }]
    }'
done

Grant the Infracost cross-account role decrypt access — save as kms-decrypt-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "InfracostKMSDecrypt",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:DescribeKey"],
      "Resource": "arn:aws:kms:us-east-1:111111111111:key/mrk-abc123"
    }
  ]
}

aws iam put-role-policy \
  --role-name infracost-readonly \
  --policy-name infracost-kms-decrypt \
  --policy-document file://kms-decrypt-policy.json \
  --no-cli-pager

Update the KMS key's own key policy to grant two sets of principals access. Save as kms-key-policy.json, replacing 111111111111 with your management account ID:

{
  "Statement": [
    {
      "Sid": "AllowExportServicePrincipals",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "bcm-data-exports.amazonaws.com",
          "billingreports.amazonaws.com",
          "storage-lens.s3.amazonaws.com"
        ]
      },
      "Action": [
        "kms:GenerateDataKey",
        "kms:Decrypt"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "111111111111"
        }
      }
    },
    {
      "Sid": "AllowInfracostRoleDecrypt",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111111111111:role/infracost-readonly"
      },
      "Action": [
        "kms:Decrypt",
        "kms:DescribeKey"
      ],
      "Resource": "*"
    }
  ]
}

This is a merge, not a replace — fetch the key's existing policy first with aws kms get-key-policy --key-id YOUR_KEY_ID --policy-name default, add these statements to its Statement array, then apply it:

aws kms put-key-policy \
  --key-id YOUR_KEY_ID \
  --policy-name default \
  --policy file://kms-key-policy.json

Removing

FOCUS_ARN=$(aws bcm-data-exports list-exports --query "Exports[?ExportName=='InfracostDailyFocusExport'].ExportArn" --output text)
COH_ARN=$(aws bcm-data-exports list-exports --query "Exports[?ExportName=='InfracostDailyCostOptimizationExport'].ExportArn" --output text)
aws bcm-data-exports delete-export --export-arn "$FOCUS_ARN" --region us-east-1
aws bcm-data-exports delete-export --export-arn "$COH_ARN" --region us-east-1

aws s3control delete-storage-lens-configuration \
  --account-id 111111111111 \
  --config-id InfracostStorageLensExport \
  --region us-east-1

aws s3 rm s3://infracost-bcm-exports-111111111111 --recursive
aws s3 rm s3://infracost-storagelens-exports-111111111111 --recursive
aws s3api delete-bucket --bucket infracost-bcm-exports-111111111111
aws s3api delete-bucket --bucket infracost-storagelens-exports-111111111111

aws iam delete-role-policy --role-name infracost-readonly --policy-name infracost-s3-access --no-cli-pager
aws iam delete-role-policy --role-name infracost-readonly --policy-name infracost-kms-decrypt --no-cli-pager 2>/dev/null || true

The AWSServiceRoleForBCMDataExports service-linked role is shared across a service's features, not owned by any one consumer — leave it in place unless you're certain nothing else in the account uses BCM Data Exports.

Troubleshooting

create-export fails with a ValidationException

This typically happens for one of the following reasons:

• Management account is not yet opted in to Compute Optimizer and Cost Optimization Hub — see Prerequisites above, then retry.
• You're not working in the management/payer account.
• IAM billing access is not enabled. If this setting is deactivated, IAM users and roles in the account can't access the Billing and Cost Management console pages, even with administrator access.
  • As the root user, open the account menu in the upper-right corner and select Account.
  • Scroll to IAM user and role access to Billing information, click Edit, and check Activate IAM Access.
• Your IAM user lacks cur:PutReportDefinition or bcm-data-exports:CreateExport permissions.

Storage Lens configuration doesn't show data from member accounts

Confirm trusted access for S3 Storage Lens is enabled (see Prerequisites), and that the Organization ARN you used in Step 4 is correct.