Vantage Launches Advanced Analytics: See the Cost of Individual AWS Resources(vantage.sh)
vantage.sh
Vantage Launches Advanced Analytics: See the Cost of Individual AWS Resources
https://www.vantage.sh/blog/vantage-launches-advanced-analytics
10 comments
Why is cost transparency so challenging for the CSPs? Is it planned obfuscation, or is it just that dedicating engineering time to billing isn't as interesting as building more impactful products?
I think the problem is two fold.
1) a CSP can't make money off of cost tools and anything that doesn't generate revenue will always be de-prioritized. You could make an argument that having good cost tooling natively will prevent churn (which is probably true) but still this only retains revenue that other products generate rather than generate new revenue itself. I don't think there is a world where customers wouldn't be enraged if AWS started charging for Cost Explorer even if it meant a better version of CE.
2) All of the product teams that generate the cost data are not incentivized to make the cost data easy to understand or sensible from a billing perspective. They are incentivized to generate more revenue. It feels as if AWS billing is meant to be easy for the product teams to bill based on rather than for the customer to grok the bill.
1) a CSP can't make money off of cost tools and anything that doesn't generate revenue will always be de-prioritized. You could make an argument that having good cost tooling natively will prevent churn (which is probably true) but still this only retains revenue that other products generate rather than generate new revenue itself. I don't think there is a world where customers wouldn't be enraged if AWS started charging for Cost Explorer even if it meant a better version of CE.
2) All of the product teams that generate the cost data are not incentivized to make the cost data easy to understand or sensible from a billing perspective. They are incentivized to generate more revenue. It feels as if AWS billing is meant to be easy for the product teams to bill based on rather than for the customer to grok the bill.
Not to challenge it, but how does it differ from Cost Explorer in AWS?
Cost Explorer does not give per resource cost analytics outside of EC2. So you are unable to see specific resource costs in AWS Cost Explorer without making a very, very large investment in tagging each specific resource with something different.
Vantage Advanced Analytics will now show all customers not only total accrued costs per AWS resource but give you day-by-day trends broken down by editorialized category (i.e. Data Transfer) as well as specific AWS billing-code subcategory ("i.e. EU Egress Charges"). Additionally Vantage will show cost trends for each resource so you can plot month-over-month how specific resources like S3 Buckets or SQS Queues are trending in cost.
AWS Cost Explorer does not show any of this information.
Vantage Advanced Analytics will now show all customers not only total accrued costs per AWS resource but give you day-by-day trends broken down by editorialized category (i.e. Data Transfer) as well as specific AWS billing-code subcategory ("i.e. EU Egress Charges"). Additionally Vantage will show cost trends for each resource so you can plot month-over-month how specific resources like S3 Buckets or SQS Queues are trending in cost.
AWS Cost Explorer does not show any of this information.
Tagging is required yes, but its a very simple practice. I worked in both small (3 accounts) and huge (200 accounts) orgs, and tagging, at least by name was never a problem regardless of tens of thousands of resources we manage.
Cost Explorer does all of what you mentioned. UI allows 1 level group by, API does 2. You can filter & group by API Operation (S3 put object), Service (S3), Usage Type (S3 Standard Storage GB/mo), (predefined) Tags and it works brilliantly over even 1 year of data, fast.
I'm really not convinced on what you can do additionally, since you work on the same data exactly (detailed dumps on S3) apart from nicer charts.
Cost Explorer does all of what you mentioned. UI allows 1 level group by, API does 2. You can filter & group by API Operation (S3 put object), Service (S3), Usage Type (S3 Standard Storage GB/mo), (predefined) Tags and it works brilliantly over even 1 year of data, fast.
I'm really not convinced on what you can do additionally, since you work on the same data exactly (detailed dumps on S3) apart from nicer charts.
Just so I understand this correctly....you have individually tagged each individual resource to do this for an AWS org with over 200 member accounts? Then enabled the presumably hundreds of thousands of individual tags for Cost Explorer to pick up?
From what we've heard from our customers no one is doing this (or wants to do this) and would prefer to just a flip a switch and have Vantage handle it on their behalf.
From what we've heard from our customers no one is doing this (or wants to do this) and would prefer to just a flip a switch and have Vantage handle it on their behalf.
I agree AWS should tag all resources with their names already for them to show up in the Cost Explorer. However, we already tag by cost center or teams to identify costs. It's required as a policy and baked in most of our automations. It's also a best practice recommended by even most basic tutorials. If you have a TAM, they would tell you to do this as well when your finance team or person is asking about why these bills are large.
Also, even if they are untagged, it's not like we need to tag them by hand, AWS Resource Tagging API works nice enough, and not every resource is costly that we need to tag them by resource-level.
I support what you are doing however there are too many companies working CUR data in S3 and none of them is good enough and neccessarily faster or more useful than Cost Explorer. Just wanted to understand what you can add on top of it, apart from automatic tagging? part.
Also, even if they are untagged, it's not like we need to tag them by hand, AWS Resource Tagging API works nice enough, and not every resource is costly that we need to tag them by resource-level.
I support what you are doing however there are too many companies working CUR data in S3 and none of them is good enough and neccessarily faster or more useful than Cost Explorer. Just wanted to understand what you can add on top of it, apart from automatic tagging? part.
Got it - so what I'm inferring is that for your use-cases you don't need to see per-resource cost granularity out of the box which is completely fair.
We've heard from customers that some top issues they have with cost explorer include (1) lack of visibility into K8S workloads (2) more complex filtering around creating cost reports and (3) more effective tools for chargebacks/showbacks. All of these are on our roadmap - in addition to supporting other clouds (Azure, GCP) and other cloud services.
Out of curiosity, what would you want from a "better cost explorer" service for your organization?
We've heard from customers that some top issues they have with cost explorer include (1) lack of visibility into K8S workloads (2) more complex filtering around creating cost reports and (3) more effective tools for chargebacks/showbacks. All of these are on our roadmap - in addition to supporting other clouds (Azure, GCP) and other cloud services.
Out of curiosity, what would you want from a "better cost explorer" service for your organization?
K8s integration sounds useful, there are also many startups in this area, and it's trivial to implement, however please consider that just a workload is requesting 2 CPUs, does not mean you can divide the underlying EC2's price per vCPU or Memory. There is significant overhead running K8s, log collectors, agents, wasted space, and spot instance pricing etc. please be more insightful for that, not just divide by request/limits. Combining these with overall AWS costs would make more sense. However as I said, we tag by service or cost centers, so I would want to associate my Pods and its SQS queue cost together.
Chargebacks in AWS is very complex, i.e in cost explorer net amortized cost vs unblended cost is hard to understand for an average user. It'd be nice if you can have some clarifications on it, what does even amortized mean etc.
We use Cost Explorer API to identify anomalies. Even basic statistics works sometimes. But there is also new native feature that is good enough for us, so we removed that. But it was a very sensitive topic, i.e our S3 usage is steadily increasing %.x per month. Trend & seasonality analysis is a must.
Sorry for my quick and dirty English. This is a hot topic for me and I get excited.
Chargebacks in AWS is very complex, i.e in cost explorer net amortized cost vs unblended cost is hard to understand for an average user. It'd be nice if you can have some clarifications on it, what does even amortized mean etc.
We use Cost Explorer API to identify anomalies. Even basic statistics works sometimes. But there is also new native feature that is good enough for us, so we removed that. But it was a very sensitive topic, i.e our S3 usage is steadily increasing %.x per month. Trend & seasonality analysis is a must.
Sorry for my quick and dirty English. This is a hot topic for me and I get excited.
[Disclosure] My name is Ben Schaechter and I'm Co-Founder and CEO at https://www.vantage.sh/ - before Vantage I worked at both AWS and DigitalOcean in a technical product management capacity.
Advanced Analytics is our biggest feature update since the initial launch of Vantage and gives you the ability to see costs for each individual AWS resource broken down day-by-day by category (i.e. Data Transfer) as well as subcategory ("i.e. EU Egress Charges"). So imagine seeing the cost of pretty much any resource including things like S3 Buckets, Lambda Functions, SQS Queues, SNS Topics and doesn't require any tags/changes to your infrastructure.
We believe this provides a richer set of analytics than AWS Cost Explorer provides. I'd be happy to answer any questions if folks have them - our original launch happened on HackerNews and we got a great set of questions.
Advanced Analytics is our biggest feature update since the initial launch of Vantage and gives you the ability to see costs for each individual AWS resource broken down day-by-day by category (i.e. Data Transfer) as well as subcategory ("i.e. EU Egress Charges"). So imagine seeing the cost of pretty much any resource including things like S3 Buckets, Lambda Functions, SQS Queues, SNS Topics and doesn't require any tags/changes to your infrastructure.
We believe this provides a richer set of analytics than AWS Cost Explorer provides. I'd be happy to answer any questions if folks have them - our original launch happened on HackerNews and we got a great set of questions.