> The cost for something that can be replicated free and open source is absurd.
open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money.
oh you want it maintained?
the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without.
yes, yes, you just have to explain yourself. just help management understand. justification is part of the role. how long does that take, and how much effort? (the answer is subjective and contextual.)
to be blunt, this kind of advice is barely more than “do better.” it ignores the situational example of using snowflake cheaply. it acts like most devs aren’t just going to go fire up a postgres rds - you said open source! it oversimplifies all problems, implicitly, to you.
> Pay to make a problem disappear.
no, pay to change the parameters of the problem. this is a fundamental misunderstanding of how to get things done in a constrained environment. it isn’t either/or, and every saas comes with problems. you pay to trade problems. otherwise it wouldn’t be worth a blog post name dropping and shitting on databricks and snowflake costs. in the hands of an untrained user or in a sufficiently constrained environment, they cost a lot - that’s one of the problems you buy. cost management.
> a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations
again, hope you’ve taken the time to build these bridges. the author does indicate this is most important.
and it has a cost. someone is building that bridge, and likely someone from each side.
you sometimes pay vendors to deal with technical problems so you can deal with non-technical ones.
most importantly, don’t assume a random stranger on the internet has sufficient context to give you worthwhile recommendations.
That's what the Databricks salespeople say. My consistent experience has been that the experts "learned" it over the weekend by reading over the brochure and now I not only need to get a deep understanding of it myself I also have to waste my time explaining it to them because they have to actually do it in the closed environment the Databricks is overcharging us for.
have yet to meet a vendor-recommended solutions provider that felt like more than a cash grab for anyone too dazzled by the initial onboarding.
the math of outsourcing implementation to consulting firms always seems off - send the problem elsewhere, usually for a premium, and hope the internal folks can digest whatever the actual consulting dev writes after the sales dev talks big game on hypotheticals.
hesitant to say my sample size is too small. feels like a trap.
I don't need a degree in Individual Contributorism with a citation in Consultantology to know that Snowflake and Databricks are ripping people off.
I'm not sure why anyone would be especially passionate about figuring out the details how though. If you want to level a criticism at the author, it's spending time on blog posts instead of ostensibly making the super cheap, better data analytics platform he's advocating for.
I think that's fair. I also think it's fair to count on-prem versus hosted as a metric for your org that should be evaluated.
I think that what is missing from the article is that it's not a matter of free versus paid, but rather integrated solution versus dispersed solution with custom glue code. Depending on your organization and your constraints, you should define your budget, wants and needs, build metrics to represent those, evaluate all systems against your particular metrics, and come to a dispassionate result.
[Disclaimer: I work at ClickHouse] I see a lot of the responses here focus on the age-old ‘build v. buy’ debate.
It’s also worth considering the comparative cost of Snowflake against other saas warehouses or databases, depending on your needs. For instance, we’ve heard from users that ClickHouse Cloud can be much more cost-effective for many use cases when compared to Snowflake - real-time analytics is a great example. For those who can’t build (or run OSS themselves), this is another interesting (and important) dimension. What's great about ClickHouse is that it's open source, too :).
Big corps are risk management companies. The happy path is that you will have a working solution at a fraction of the cost. Mid to worst case scenario is that you end up over your head, the costs are higher than you anticipated and the end result is mediocre at best. Big corps are willing to pay a high premium to guarantuee a certain level of performance with penalties if necessary, especially if the solution is pretty far removed from their core business.
>open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money. oh you want it maintained?
Funny. I heard this same kind of argument used against replacing oracle with postgres. It reminds me of Microsoft's "TCO" PR offensive back when they were public about how much they loathed open source competition.
Thing is that it wasnt just a straw man (nobody that needs to be convinced is under any illusions that software has to be maintained), Oracle was also way more expensive to maintain ON top of being expensive to run.
They had their hooks into that organization pretty good though and once that happens technology choices become highly political - which experts are you going to fire and which ones are you going to hire?
The oracle lot obviously didnt want it to be them that get managed out.
you’re calling it an organization, that alone typically indicates a larger scale, in which supporting something internal might be feasible. or at least it is a term used by those that have been there. you’re also referencing politics, which, again, that’s highly suggestive of a specific type of experience. experience within scale if not at scale.
please don’t mistake my intent to be that open source never makes sense. with the right plan and personnel, it can work better.
with the right scale and support, open source projects are started, though they aren’t always open from day one.
I think you are vastly under-estimating the amount of cost/manpower needed for snowflake and cloud based solutions (they are anything but turn-key and have a lot of churn). There are cases for both but it's not as simple as open source = hire and cloud = no people needed.
With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketing, and management behind it. But on top of all that you probably need to hire people to implement and maintain a solution utilizing it for your organization.
> With Snowflake you are paying for not only the development of the product, the hosting of it, the engineers to run it, but also the sales, marketing, and management behind it.
and you can run the numbers to determine if the cost of ownership offsets headcount. what your dollar goes to isn’t part of that formula. only what your dollar gets you.
> But on top of all that you probably need to hire people to implement and maintain a solution utilizing it for your organization.
that is also a given. you would need at least as many people to implement analytics on top of the homebrew data warehouse, in addition to headcount to run the warehouse itself.
you seem to want to make this out to be vastly different spend amounts. it can be. sometimes it is, sometimes it is not, and the direction can vary.
We aren't talking about a homebrew data warehouse though, we are talking about open source software. Designing the schema and/or data flows on top of the warehouse is needed regardless of whether you use open source or Snowflake.
if it isn’t already running, yes, we are. open source is rarely if ever “off the shelf.”
if you’re suggesting vendor-hosted open source, that’s a consideration in vendor selection, and as much as it may seem obvious to suggest alternatives now, it wasn’t when they didn’t exist.
and yes, you correctly gathered the point on schemas and the like being the same everywhere.
Big corps are risk management companies. The happy path is that you will have a working solution at a fraction of the cost. Mid to worst case scenario is that you end up over your head, the costs are higher than you anticipated and the end result is mediocre at best. Big corps are willing to pay a high premium to guarantuee a certain level of performance with penalties if necessary.
open source it may be. free it is not. paying an expert to correctly deploy an open source solution takes time and money.
oh you want it maintained?
the three recommendations sound like those of a consultant. they work great with exec buy-in, and are a joke without.
yes, yes, you just have to explain yourself. just help management understand. justification is part of the role. how long does that take, and how much effort? (the answer is subjective and contextual.)
to be blunt, this kind of advice is barely more than “do better.” it ignores the situational example of using snowflake cheaply. it acts like most devs aren’t just going to go fire up a postgres rds - you said open source! it oversimplifies all problems, implicitly, to you.
> Pay to make a problem disappear.
no, pay to change the parameters of the problem. this is a fundamental misunderstanding of how to get things done in a constrained environment. it isn’t either/or, and every saas comes with problems. you pay to trade problems. otherwise it wouldn’t be worth a blog post name dropping and shitting on databricks and snowflake costs. in the hands of an untrained user or in a sufficiently constrained environment, they cost a lot - that’s one of the problems you buy. cost management.
> a talk with your local Dev Ops Engineer/ manager to discuss how to secure your implementations
again, hope you’ve taken the time to build these bridges. the author does indicate this is most important.
and it has a cost. someone is building that bridge, and likely someone from each side.
you sometimes pay vendors to deal with technical problems so you can deal with non-technical ones.
most importantly, don’t assume a random stranger on the internet has sufficient context to give you worthwhile recommendations.