Splunk IP suit against Cribl(splunk.com)
splunk.com
Splunk IP suit against Cribl
https://www.splunk.com/en_us/blog/bulletins/splunk-files-intellectual-property-complaint-against-cribl.html
106 comments
I can add from the other side of the fence. I worked for a startup that was acquired by Splunk. They are everything listed here and worse on the inside.
My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.
I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.
Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.
My first few weeks at Splunk were very odd. They try to indoctrinate new hires with a barrage of "A-players" that continuously talked about how awesome Splunk was. Except... When I started Splunk was getting their ass kicked by cloud-first players that had recently come to market. Splunk's monolithic architecture wasn't well suited to be run as SaaS at the time and Splunk was burning cash and losing money on every customer that they suckered into moving away from their perpetual licenses into subscription hell. I left money on the table when I ran out the door less than 6 months later.
I'm curious what Splunk's long game is with this because they just told every F2000 that their bottom line is being chipped away by Cribl and friends. So if I'm an enterprising procurement department I'd be tossing Cribl or Rudderstack or whatever other data transformation preprocessor on the table alongside my renewal. Expand opportunity? If you put your ear to the tracks you can almost hear all of the account managers digging out missed quota excuses.
Splunk isn't innovative and hasn't been for a long time. Most of the employees saw the writing on the wall and went to Snowflake as soon as the opportunity presented itself. Splunk tried to capitalize on the security market by, basically, double charging customers for ES. Instead of delivering value it seems to be Splunk is just looking for ways to squeeze a few last drops of lemonade.
Sounds like a good move... From their about page:
Splunkers have received over 1,020 patents to date
I am one of those people. There was a bonus for every patent granted. They were telling us that we need to big patent arsenal to fend off against IBM. It turned out that Splunk is IBM now.
“You either die a hero or you live long enough to see yourself become the villain”
I am against software patents and choose to ignore all pleadings from my employers regarding patent filings. IMO, the bonuses (~$1-2K) are not worth going against my views.
You could have made the same choice, but did not.
You could have made the same choice, but did not.
Your last sentence is rather dismissive and seems unnecessary to make your point. You're assuming that they share your views on software patents. They might not. Or at that time not realise the issue with software patents in the first place.
Hear hear. I know what you mean, I've lived through the same in a different Fortune 100.
Interesting, I helped manage a splunk install at a fortune 200 about a decade ago. At the time the recommendation was to use syslog-ng to filter incoming logs before indexing. I just heard of cribl 2 weeks ago because the fortune 20 I currently work for is planning on switching to it. I didn't realize it was a massive shift like that, I just thought it was the corporation switching things just because they do that sometimes.
There have been a few other recommendations over the years, including putting a separate tier of forwarders first in line to perform transforms and such. There were always plenty of options for on-prem/DIY/Enterprise especially when using syslog instead of directly via HEC.
Their SaaS offering used to have said inline tier called IDM (Inputs Data Manager) where we were directed to configure filters during our POC… a key requirement for moving from Enterprise to SaaS because conf files aren’t managed the same. One month (to the day!) after we moved, they randomly decided to migrate us to a new “Victoria experience” where that tier suddenly disappeared without explanation. We filed support tickets asking 1) what happened? and 2) how do we filter things out now? and were directed to hire professional services because that was outside the scope of standard support!
The whole point of moving to SaaS was to not have to babysit our own clusters (small shop at the time), so spinning up a ton of infra in front of the freshly greenlit SaaS setup would have negated the productivity gains and financial pivot.
Ultimately, the entropy of hundreds of applications logging in disparate formats and namespaces outweighed our ability to sanitize each app within a reasonable amount of time, leading to unwanted data being indexed, ergo overages. Overages that our sales engineer originally assured us we could address by filtering things out with the snap of a finger. Bait and switch.
Ingest Actions were not available at the time, and were not functional (even in beta) until 10 months later.
Their SaaS offering used to have said inline tier called IDM (Inputs Data Manager) where we were directed to configure filters during our POC… a key requirement for moving from Enterprise to SaaS because conf files aren’t managed the same. One month (to the day!) after we moved, they randomly decided to migrate us to a new “Victoria experience” where that tier suddenly disappeared without explanation. We filed support tickets asking 1) what happened? and 2) how do we filter things out now? and were directed to hire professional services because that was outside the scope of standard support!
The whole point of moving to SaaS was to not have to babysit our own clusters (small shop at the time), so spinning up a ton of infra in front of the freshly greenlit SaaS setup would have negated the productivity gains and financial pivot.
Ultimately, the entropy of hundreds of applications logging in disparate formats and namespaces outweighed our ability to sanitize each app within a reasonable amount of time, leading to unwanted data being indexed, ergo overages. Overages that our sales engineer originally assured us we could address by filtering things out with the snap of a finger. Bait and switch.
Ingest Actions were not available at the time, and were not functional (even in beta) until 10 months later.
[deleted]
No comment about the company, but want to make clear as a buyer you understand the procurement and legal parts i.e. MFN or MFC.
If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.
If they do discount, even 5%, then it ripples across their accounts as a legal matter, esp at your scale. I was a buyer for some big companies, 8 digit, and the procurement office would only do a deal with MFN/MFC clause. They would also audit the supplier from time to time.
I totally understand that ripple effect and am very familiar with Most Favored X when it comes to unit pricing of a tangible good (e.g. xx,xxx physical servers with a particular SKU), but in this case we were talking about a SaaS product where overages were disputed. Nearly every vendor would jump at the chance to discount additional commitments or support at the ‘expense’ of waiving some past overages.
Thanks for the response, been there on the overages per SAAS’s. Now running a startup, they scare me even more.
[deleted]
Last used splunk around 2010, and we ran a bunch of scripts to truncate and reformat logs before they got anywhere near our splunk data load license
What are you planning to move to?
Sounds crazy, but Datadog. I’ve been hammering their product teams for years with specific use cases for the sole purpose of replacing Splunk. They recently migrated search technologies and are rapidly closing the gap. Plus, their exclusion features are instant and fantastic, and their C-suite replies to me when I escalate.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
Elasticsearch simply couldn’t handle key collisions. We have hundreds of various apps across 5-10 different languages and frameworks where a key name may be reused as either a string or a hash or an integer or an array. If we can’t freeform search (which Splunk is EXCELLENT at), we just need to be able to transform the data beforehand. Datadog plans to do so with their recent acquisition of Vector.
Sad. Splunk should be more fantastic. They have done the heavy lifting of taking streams of data at high volume, which should be the basis to build a log search product, metrics And alerting, and observability.
Instead, each of these systems have their own collectors and correlating from one to the other is hard. A canonical log line is so much more valuable than a metric collected every 60 seconds, and the former can derive the latter: https://stripe.com/blog/canonical-log-lines
Splunk should have been the lynchpin.
Instead, each of these systems have their own collectors and correlating from one to the other is hard. A canonical log line is so much more valuable than a metric collected every 60 seconds, and the former can derive the latter: https://stripe.com/blog/canonical-log-lines
Splunk should have been the lynchpin.
Absolutely agree. It’s tragic.
Shoving SignalFx down our mouths and trying to get us to create “metric” indices was the straw that broke my back.
Shoving SignalFx down our mouths and trying to get us to create “metric” indices was the straw that broke my back.
I built a PCI compliance solution for a customer back in 2008 for ~$200k all-in when the closest competitor's bid was five times that. The product was amazing at runtime but of course had some idiosyncrasies in how it was configured and whatnot. I've been a user (only) of Splunk heavily ever since and just last year got pulled into a project to migrate a huge install to a cloud platform. It felt like I got into a time machine...there were seemingly zero administrative or architectural improvements to make the product more manageable or supportable in the 10+ years since I had last looked at it from an ops perspective.
I'm sure that's not 100% true but it felt like it. Trying to build Splunk on top of a modern IaC deployment methodology is a huuuuge lift.
I'm sure that's not 100% true but it felt like it. Trying to build Splunk on top of a modern IaC deployment methodology is a huuuuge lift.
Yeah, team decided K8s for Splunk would be too much work and ended up needing to use vanilla VM’s with block storage on an open stack env on prem.
Pretty lame not cloud native.
Pretty lame not cloud native.
Would love to know more about specific use-cases you've been talking to Datadog about. I'm starting a company (log-store.com) that I pitch to people as 75% of the features of Splunk at 50% the price. Right now that 75% is probably more like 25%, and the 50% is _actually_ 0%... it's FREE! Any and all feedback is greatly appreciated!
Sounds crazy indeed. I worked at Splunk for many years and was a DataDog customer later. The costs of either are not something I care to deal with.
You might also want to review NewRelic. Their new plans allow for lots of data ingestion at low price including logs.
Exactly.
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
I mean EVERYTHING.
This is the question. If you’re looking for APM well you’ve got great options but for those using Splunk in the security space (SIEM & SOAR) you’re screwed.
There’s no better SIEM alternative that deals with logs at scale.
Splunk recently screwed a friends Fortune 50 company. They didn't pay a bill on time (renewal negotiations) and Splunk without even contacting them just left all the logs from one of their instances on the floor. They lost everything for literally an entire country.
I mean EVERYTHING.
Lol, this is exactly what I was referring to! We were negotiating a massive renewal (50x budget IN EARNEST!) and out of nowhere they threatened to cut us off with less than 24 hours notice because we were a week behind their schedule, despite signed agreements. Thankfully our VAR (a longtime partner) jumped in and cut a PO to vouch for us and our word.
NONE of our other vendors have EVER done that shit in my entire career. EVER. My word means nothing to them; they act like a pure private equity player now.
NONE of our other vendors have EVER done that shit in my entire career. EVER. My word means nothing to them; they act like a pure private equity player now.
Lol, I could have written exactly the same comments you did about Splunk. I'm the one who decided to start using splunk for our company 8 years ago, we're working on migrating away and I will never under any circumstances consider recommending splunk for any future companies I work for.
I can truly empathize. I’ve never wasted so much of my time on a vendor. It’s infuriating and certainly not worth the personal stress and agony. It shouldn’t be this hard!
Can echo the licensing sentiments across 2 companies now. It's a shame because I grew to really like SPL and how Splunk handled web based distributed search. What are you moving towards now, if I may ask?
It's pretty ridiculous this can happen. With building Matano (https://github.com/matanolabs/matano), an open source security lake, one of our core decisions was to store all data in S3 in Apache Iceberg tables (an open table format that lets you query data from any supporting tool i.e Spark, Presto, Athena, Snowflake, more). This let's you own your data without it being held hostage on some vendors instances in a proprietary format.
-https://github.com/matanolabs/matano
- https://iceberg.apache.org
-https://github.com/matanolabs/matano
- https://iceberg.apache.org
> They didn't pay a bill on time (renewal negotiations)
I don't side with Splunk on their actions but I can't stand customers who withhold payment as a renewal tactic and think that it's equally wrong.
I don't side with Splunk on their actions but I can't stand customers who withhold payment as a renewal tactic and think that it's equally wrong.
Wait, what? When a license expires my understanding is indexing continues, you just can't search, you can get a key to unlock search by contacting Splunk. I assume you mean the SaaS Splunk Cloud, then, because the renewal period would be far in advance of the license expiring, Splunk doesn't just "stop". That behavior makes no sense and would lose of a lot of clients, you can't just backfill data gaps.
>There’s no better SIEM alternative that deals with logs at scale.
I think folks that use Splunk for basic search just don't fully comprehend how capable the product is for hunt-type operations when someone fluent in SPL is at the helm.
I think folks that use Splunk for basic search just don't fully comprehend how capable the product is for hunt-type operations when someone fluent in SPL is at the helm.
I can't agree more. I've used every main 'competitor' now and nothing can compare to splunk for hunting across massive logging pools. It genuinely feels like magic with advanced SPL and solid regex.
My frustrations with Splunk have been around their certification and training changes over the years. Used to be able to get a solid tool certificate and decent training materials all for free. It only hurts Splunk though as less people have experience with the tool it lessens their advantage. Makes me disappointed as I really do like the tools itself but literally everything else is terrible. I'd much rather deal with Elastic or go open source with Security Onion.
My frustrations with Splunk have been around their certification and training changes over the years. Used to be able to get a solid tool certificate and decent training materials all for free. It only hurts Splunk though as less people have experience with the tool it lessens their advantage. Makes me disappointed as I really do like the tools itself but literally everything else is terrible. I'd much rather deal with Elastic or go open source with Security Onion.
Ex-splunker here. I just started working at FeatureBase and would say, if your data is in Kafka, FeatureBase might be something to consider. It’s a crazy fast binary index built on Roaring bitmaps.
On the one hand, taking code from your employer and posting it to GitHub with the copyright notices removed is about as clear-cut a case of copyright infringement as you can get -- if they have evidence. Should be easy to confirm or deny by looking at version control history.
(This seems to be the repository in question, but it's been taken down: https://web.archive.org/web/20210104032001/https://github.co...)
On the other hand, the patent claims referenced in the lawsuit seem to me like great examples of software patents that ought to be struck down for being uselessly over-broad. For example, I would love to hear an argument as to how the "'433 Patent" wouldn't be infringed by running Wireshark in a Kubernetes pod. That meets every single one of the claimed elements that Splunk is claiming Cribl is infringing.
(This seems to be the repository in question, but it's been taken down: https://web.archive.org/web/20210104032001/https://github.co...)
On the other hand, the patent claims referenced in the lawsuit seem to me like great examples of software patents that ought to be struck down for being uselessly over-broad. For example, I would love to hear an argument as to how the "'433 Patent" wouldn't be infringed by running Wireshark in a Kubernetes pod. That meets every single one of the claimed elements that Splunk is claiming Cribl is infringing.
I'm not sure I understand the business impact of a protocol?
Presumably anyone with Wireshark could reverse it, so does it impart a significant advantage? Or is it just about control?
Presumably anyone with Wireshark could reverse it, so does it impart a significant advantage? Or is it just about control?
A more complete archive:
https://archive.ph/HKAF2
From the lawsuit looks like the most clear cut evidence they have is:
- Founder publishing a private protocol definition to help in building for it
- Sales staff sending account and prospect info to their new cribl email addresses before leaving Splunk
- Engineers leaving Splunk with technical specifications, such as their newer S2S protocol versions
The patent stuff is kind of whatever, but all three of those items would be enough to establish some very clear damages. Cribls an exciting new player but they can't take shortcuts like this, if the allegations are founded.
- Founder publishing a private protocol definition to help in building for it
- Sales staff sending account and prospect info to their new cribl email addresses before leaving Splunk
- Engineers leaving Splunk with technical specifications, such as their newer S2S protocol versions
The patent stuff is kind of whatever, but all three of those items would be enough to establish some very clear damages. Cribls an exciting new player but they can't take shortcuts like this, if the allegations are founded.
Honest question, where is the line here? Obviously we all retain knowledge from previous jobs so what's the line between that and exactly copying a spec?
I think the line here is pretty straightforward: the contents of your mind are all yours. Anything beyond that (documents, source code, lists of prospects) is not.
Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.
Non-compete clauses will try to limit the usefulness of the "in your mind" knowledge by restricting the domains in which you can work post-departure. It's my understanding that such clauses are generally held to be unenforceable except in an acquisition scenario.
I think the line is drawn at actual stealing. In this case, they're not redesigning the protocol from memory or black box testing. Allegedly they took several specs of the protocol from former employees. Even then, apparently the founders were involved in some of the Patent filings from splunk that they are accused of violating. You cant claim IP for a company in the form of a patent and then turn around and re-implement that IP. You clearly believed it was patent-able since you patented it. There's ways of doing this (clean room dev) if you wanted to do that without infringement. (I do feel a lot of the patent claims in the lawsuit are typical generic weak software patents)
Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.
Really egregious is taking the sales data. Business analytics around leads, customer satisfaction, pricing, etc are not the same as retaining general knowledge. If you left and remember the point of contact you had at a customer, that's allowed (barring non-solicitation agreements). If you leave and you take a list of customers, data that the business has generated about them, etc, that was never yours and it's not your knowledge. It's clearly the business's and there's usually dozens of people involved in the creation. That's clearly theft, especially since it was never yours to begin with.
honestly, I've never moved from a role in one company to a role in a new company that directly competes with the role I had in my old company.
While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.
So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.
While I have jumped to competitors, I moved to roles that weren't in any form competition to my former team/role. That makes it easy, even if I would accidentally take things with me, I wouldn't be tempted to look at it, as there would be no point.
So yes, I take all my growth, knowledge and experience, but nothing that is really unique (say trade secrets) to old company would directly apply to my new role, so there has never been any problem. Once one is willing to jump to a competitor in a manner where you trade secret knowledge would benefit your role directly, one is creating a problem.
To me, unless there is a legal document you signed with your employer, there is no line. Even, IMO, IP is not `property` so that it cannot be used against. But that is another discussion.
Splunk is the best at what it does with no close competition.
I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.
Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.
Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.
Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.
I've been looking into Cribl and it seems their product has surpassed their competition as well but not in search, more in data summarization and log reduction, possibly before you ship it off to a more proper place like Splunk.
Splunk's cost makes it inaccessible to most people or companies. I mean, I work in infosec and I highly caution against Splunk because it is so amazing you will hate anything else but in security you need tons of otherwise rubbish data collected centrally sometimes and it will force you into a corner where you will say you can't afford to store that log you really should be storing. Better a crappy tool that can be used to find the logs you need than a nice tool that can only retain so much.
Cribl is supposed to help people reduce what they put i Splunk so they can keep using Splunk, it would have been nice if they partnered instead.
Graylog is another nice tool I like that is somewhat but only slightly similar to Cribl that was founded by a former Splunker out of frustration.
Is Splunk fast now?
Last time I used it was almost a decade ago and it was rubbish, queries took 10-40 minutes to complete.
Last time I used it was almost a decade ago and it was rubbish, queries took 10-40 minutes to complete.
It has always been the Cadillac of search, and moreso with unstructured indexing (e.g. key collisions with different data structures. Foo = string vs foo = integer vs foo = array).
Your queries or infrastructure were not optimized. It’s very fast when optimized.
Your queries or infrastructure were not optimized. It’s very fast when optimized.
Interesting,
It was Splunk managed and configured, so I would have thought it optimized, but I guess they made more money from it not being optimized.
If I remember right then we were throwing about 200+ GB at it a day.
It was Splunk managed and configured, so I would have thought it optimized, but I guess they made more money from it not being optimized.
If I remember right then we were throwing about 200+ GB at it a day.
Splunk worked with us to optimize our configs but we always managed it ourselves.
Nope, sports betting. I was on the operations side of things and Splunk was something the corporate side organised and championed, but it just couldn't be used to troubleshoot issues in the time frames we needed.
[deleted]
Very very fast, I can do an all time search on terabytes of data in seconds.
But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.
But you have to learn to use it, if you don't give it an index and a sourcetype that will slow it down, and like ES leading wildcards slow things down. The fastest searches are simple terms like a word or an IP.
Interesting,
From the general responses it sounds like we got unlucky with a dud implementation.
From the general responses it sounds like we got unlucky with a dud implementation.
40 minutes sounds exceptionally bad, but 5-10 minutes with splunk was totally common when I worked at Apple almost a decade ago, and I could never figure out why because I only ever used it for O(grep on a log file on disk) level operations. I was probably holding it wrong or maybe the infra team had misconfigured it, idk.
I personally brought Splunk to Apple in 2010 alongside a small handful of people (Hi, Sean and Ariel!). There is a massive difference between real-time searches where latency beyond a few seconds is unacceptable, and historical searches which can take a bit longer. I can assure you that it did its job spectacularly, much to my chagrin that there are few competitors to this day.
Cool! That makes sense. I only really used it as part of the (now defunct I think) "orchard" internal hosting platform that was very much beta when I was using it, for tiny internal apps running on like 4 instances at most, and missed being able to just grep log files; my wild, uneducated guess from what you said is that there was some kind of pooling of our meager logs with other people's from orchard, or we were otherwise off the happy path.
I was part of Orchard and miss it dearly. It had a lot of potential but it was launched as a proof of concept built on pooled resources freed up from optimizing legacy workloads (namely, moving Siri from VMware to Mesos).
It never got the love it deserved and I could absolutely believe that its Splunk cluster suffered as a result. RIP
It never got the love it deserved and I could absolutely believe that its Splunk cluster suffered as a result. RIP
100% agree, the idea of an internal Heroku was a great one, it just didn't seem to work with how Apple was designed organizationally or something and seemed under resourced.
We recently transitioned to it at Notion and it’s been very fast, outperforming the previous log vendor substantially while offering better search and UX. If you used the on-prem version, the cloud version is quite a different experience.
I don't know which version we used, just that it was managed and configured by Splunk.
We were only sending a small subset of our logs to it so about 200+ GB a day. Our Linux box with spinning disks could grep the full set of logs much faster than querying Splunk, so I don't think anyone really used it.
We were only sending a small subset of our logs to it so about 200+ GB a day. Our Linux box with spinning disks could grep the full set of logs much faster than querying Splunk, so I don't think anyone really used it.
Yeah but you had clue. Those who don’t use Splunk.
Yes BUT it needs tuning. Splunk is complicated and takes continuous maintenance to optimize speed.
I work as a Splunk integrator and here's what I often see:
1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.
2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.
3. Customer chooses between outside help or DIY. DIY rarely works.
4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.
Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.
I work as a Splunk integrator and here's what I often see:
1. Customer installs Splunk with a qualified Splunk or third-party architect team. The deployment works well.
2. Customer adds infrastructure to the deployment. Splunk slows down. License costs go up.
3. Customer chooses between outside help or DIY. DIY rarely works.
4. Customer now needs outside help. Now Splunk is very slow and expensive, and now it will cost a lot to tune it.
Splunk, the company, is in a tough spot for several reasons: rotating c-level cast, unpopular changes to license model, bad acquisitions. The product is still best in class but tough to keep optimized.
So basically what you are saying is this.
A firm with a competent IT team is unable to get splunk to work and only "outside help" can make the product work?
Given splunks license costs are tied to data ingested, how do you integrate new infrastructure to the deployment and not have license costs go up?
Way to sell us on Splunk?
A firm with a competent IT team is unable to get splunk to work and only "outside help" can make the product work?
Given splunks license costs are tied to data ingested, how do you integrate new infrastructure to the deployment and not have license costs go up?
Way to sell us on Splunk?
Anecdotal - I took over a small, ill maintained Splunk installation at $JOB-2 and reworked it following Splunks current best-practices and it ran like a top as of when I left that place. Having done that process I'm fully convinced that if you're going to run Splunk on-prem you need a dedicated sysadmin for it that knows Splunk's stack. And that kind of person isn't cheap to hire or keep in that role.
we had an on-prem splunk implementation and it was SOO SLOW.. it was built/managed by splunk and its consultants.
We finally got rid of it a few years later, but for the entire time we had it, it was a constant "round hole square peg" problems. Each time the consultants assured us Splunk could do what we needed, each time it could not.
We finally got rid of it a few years later, but for the entire time we had it, it was a constant "round hole square peg" problems. Each time the consultants assured us Splunk could do what we needed, each time it could not.
I wonder if Splunk has a QA problem with their consultants or if there are certain edge cases they simply don't do well with.
Just that it looks like most people here had a good experience and we had a bad one for some reason.
Just that it looks like most people here had a good experience and we had a bad one for some reason.
Just coming back around to this, we also used Splunk consultants for their SIEM solution and the first one we got wasn't very good, but the second was amazing (I wish we could have hired her directly).
The guy we had help us tune our clusters after I rebuilt them all was also very good. Fortunately I'd done most everything by the books and we overkilled the nodes with hardware (we had some older hypervisor nodes lying around I stole for Splunk).
The guy we had help us tune our clusters after I rebuilt them all was also very good. Fortunately I'd done most everything by the books and we overkilled the nodes with hardware (we had some older hypervisor nodes lying around I stole for Splunk).
On modern NVMe storage it is INCREDIBLY fast.
> Splunk is the best at what it does with no close competition.
I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.
I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.
I'm with you. Splunk core - the indexing, automatic parsing, HA architecture, is unsurpassed. You can rebuild/duplicate parts of it but it's not going to come close to what Splunk can do, effortlessly, out of the box. I'm frustrated at the crud that Splunk has acquired which doesn't solve their customer's core problems. Splunk isn't well-rep in the network space. In my past I've worked for a huge tech company that was the darling of its day and Splunk business trajectory reminds me of that; we're within the start of the descent.
I read through the complaints in this thread, how it's slow, behemoth, hard to manage, copmlexities grow ... I've never experienced this problem. I've built and managed 3 Splunk clustered installations, in the 10sTB/day, and I will never use anything else. Sadly, that makes me only able to work for people able to afford the license :nervous laugh: So if you're made of money and want black car white glove data service, buy Splunk and hire people like me.
Humio is faster and cheaper than Splunk and while I haven't compared the two products feature by feature, does pretty much the same.
As an end user having used both to manage logs on a few dozen distributed applications I would never choose Splunk over Humio.
As an end user having used both to manage logs on a few dozen distributed applications I would never choose Splunk over Humio.
There's also Gravwell (https://www.gravwell.io) that competes head-on with Splunk and doesn't punish you for storing/ indexing more data. I'm on their board and knew the founders before I joined so I'm a bit biased but it's basically what if you wrote Splunk from scratch using modern tech.
[deleted]
Splunk is a great tool but expensive. I like splunk's aggregation feature very much. If it is server logs, it can aggregate and tell me how many http 500 errors I have, how many requests resulted in 404 etc. It can tell me top IP addresses where I am getting requests from, etc.
I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent.
Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime.
Thx!
I want to take a CSV file and provide same functionality. Eg. Give user information on how many times each field occurs. For example, if it is a CSV file with cities, countries, continents, I want to aggregate and tell how many cities are in each country and how many countries are in each continent.
Is there an open source version of splunk I can modify? I tried logstash but it is not straight forward to work with. It still needs me to define schema everytime.
Thx!
> Is there an open source version of splunk I can modify?
https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.
https://github.com/grafana/loki might work for you. It’s not a drop in replacement for Splunk, FWIW.
What you're describing sounds like Loki (Grafana's Prometheus inspired logging tool, which is super fast and cheap/easy, even though it sacrifices some flexibility to get there) Metric Queries: https://grafana.com/docs/loki/latest/logql/metric_queries/
We're building Matano (https://github.com/matanolabs/matano), an open source security lake platform. It's a different approach since we normalize logs from JSON, csv, etc, and ingest them into Apache Iceberg tables, but it allows for massive scale and joins, aggregations, etc using SQL.
Hope Splunk loses. Trying to kill a good player that makes Splunk less expensive.
Dang! Back in early days of AppDynamics, the founder who started AppDynamics after working at CA got hit by CA lawsuit, which lasted for a while but eventually got settled. Similar allegations. it was highly unpleasant and detrimental to the IPO preps. Some of my colleagues from there went to Cribl and sure hope they aren't going to be impacted, but they likely will will.
Our alerting solution, "OpterVics", was bought by Splunk. Since then it's been a shitshow - the service is running, but it's almost impossible to get a response from support.
They sent us an invoice for renewal in early August. I replied back (5 separate times) asking for the original contract (our ops department is tightening up on vendor management, didn't have it on file already); and we've heard nothing. Our service has continued to work despite not having paid (or signed a renewal), but we're switching to opsgenie.
They sent us an invoice for renewal in early August. I replied back (5 separate times) asking for the original contract (our ops department is tightening up on vendor management, didn't have it on file already); and we've heard nothing. Our service has continued to work despite not having paid (or signed a renewal), but we're switching to opsgenie.
After reading the full lawsuit, I think Cribl has a real threat on their hands. They've been playing fast and loose with the rules for a long time. Exports of leads from departing Splunkers, using licenses they're not entitled to use, and yes, using proprietary code that was gathered through less than fully kosher means. While this doesn't look great for Splunk, they wouldn't have filed the suit if they thought they would lose.
Pretty happy with opentelemetry collector that allows to receive traces/metrics/logs etc in different formats and then cleanup the data and push it to aggregators like Splunk or Datadog. Makes it easy to switch when the tool I am using now gets a bit expensive for tracing
[deleted]
Splunk is/was a damn fine tool, but I had to stop using it 5+ years ago because they priced themselves out of the stratosphere.
Another funny tidbit:
> On March 24, 2017, a few months after his initial copying of Splunk’s source code, Mr. Sharp resigned from Splunk to co-found Cribl with Dritan Bitincka and Ledion Bitincka— both former software architects at Splunk.
Except that they didn't because initially the had created a company called diag.io that was focused on troubleshooting fault configurations.
> On March 24, 2017, a few months after his initial copying of Splunk’s source code, Mr. Sharp resigned from Splunk to co-found Cribl with Dritan Bitincka and Ledion Bitincka— both former software architects at Splunk.
Except that they didn't because initially the had created a company called diag.io that was focused on troubleshooting fault configurations.
Splunk now an IP bully?
Go Clint & Ledio!
Go Clint & Ledio!
I’m currently working with a company in this space (axiom.co) and this shit scares me because it’s splunk scared. Maybe cribl did this? But the press release reads like a self-Pat on the back.
Splunk is great software.
With no details, hard to read this suit. Would need to know what evidence Splunk has that Clint Sharp stole source code. All the rest seems superfluous.
With no details, hard to read this suit. Would need to know what evidence Splunk has that Clint Sharp stole source code. All the rest seems superfluous.
The lawsuit is linked in the post. I didn’t read the whole thing, but skimming, the smoking gun seems to be that an implementation of Splunk’s S2S protocol was posted to his personal GitHub while he was still an employee. They claim that the header files just had the Splunk copyright notices removed, but just being a re-implementation of the objects wouldn’t surprise me. Depending on the jurisdiction that might matter.
There are also various copyright claims on things in manuals, plus claims that they infringed numerous patents.
All in all, it sounds pretty bad, but lawsuits almost always do. I would wait to read the responses before coming to any conclusions.
There are also various copyright claims on things in manuals, plus claims that they infringed numerous patents.
All in all, it sounds pretty bad, but lawsuits almost always do. I would wait to read the responses before coming to any conclusions.
Pretty poor that S2S is a proprietary protocol to begin with.
From the lawsuit:
Although Splunk provides HEC for third parties to use, Splunk maintains other aspects of its software as proprietary. One example of such proprietary software is the “S2S” protocol. S2S stands for “Splunk-to-Splunk,” and this is software that Splunk itself uses to send data to, or receive data from, Splunk Enterprise and other Splunk software and technologies. Splunk does not support use of S2S by third parties, does not publish S2S’s source code, and does not document S2S in a manner that facilitates third-party use of this protocol.
Although Splunk provides HEC for third parties to use, Splunk maintains other aspects of its software as proprietary. One example of such proprietary software is the “S2S” protocol. S2S stands for “Splunk-to-Splunk,” and this is software that Splunk itself uses to send data to, or receive data from, Splunk Enterprise and other Splunk software and technologies. Splunk does not support use of S2S by third parties, does not publish S2S’s source code, and does not document S2S in a manner that facilitates third-party use of this protocol.
I still think that is poor form.
I don't think Splunk claims will hold in court. As said running Wireguard would too just fine.
Mr. Sharp posted a derivation of Splunk’s proprietary and confidential S2S source code to his personal github webpage (a publicly accessible website for sharing source code). Mr. Sharp named this derived code “go-S2S.”
Mr. Sharp posted a derivation of Splunk’s proprietary and confidential S2S source code to his personal github webpage (a publicly accessible website for sharing source code). Mr. Sharp named this derived code “go-S2S.”
can anyone explain to a developer what splunk does?
Conveniently (and expensively) provide a destination for log data streams and enable realtime ad-hoc querying against that data at scale, with time series features and a Unix shell pipeline-like query syntax. As the comments tell, though, you must tweak it to actually see performance. And, having seen its use in ecommerce with eg logged transaction details such as credit card numbers and other PII, the prospect of easily and loosely logging all the things is limited by data security and privacy concerns, with some practices a recipe for big-time breaches and lawsuits. Aand, per seat licensing is expensive, such that more often than not IME you have a Splunk guy/gal to whom you must address your data reporting needs, questioning any benefits that ad-hoc querying your logs may have.
It does Enterprise level logging better than any other tool I've ever used.
And it drains your bank account.
And it drains your bank account.
I would love to use splunk on some of my side projects. Does anyone know of a decent alternative for non-enterprise customers?
I posted above, but starting log-store.com so I'm trying to promote it in threads without being too pushy :-)
It's in beta and free. My plan is honestly to have my pricing be free for small amounts of data, and then 50% the price of Splunk for larger data sets. Just show me an invoice, and you'll pay half!
It's in beta and free. My plan is honestly to have my pricing be free for small amounts of data, and then 50% the price of Splunk for larger data sets. Just show me an invoice, and you'll pay half!
(co-founder here) Try axiom.co - we support splunk-like query syntax, dashboards, monitors, unlimited sources/hosts/etc, and you get 500GB/mo ingest + 30 days retention on the free plan.
OpsVerse allows you to bring up Grafana Loki for logs.
There's also Logz.io or you can use Elastic for an ES backend.
There's also Logz.io or you can use Elastic for an ES backend.
You can't start Cribl without some Crib.
PMs leave and start competing products. This stuff happens all the time.
Unless splunk has a smoking gun it’s hard to really take their side here.
Unless splunk has a smoking gun it’s hard to really take their side here.
They are mad because Cribl is good at transforming data before it ingested by Splunk, so as to reduce the amount of data that is indexed. Period.
Splunk ONLY RECENTLY released “Ingest Actions” to filter data post-ingest (to avoid indexing) for their SaaS product — something that has always been a mainstay of their on-premise “Enterprise” product. Their ONLY suggestion to filter data that we didn’t care to index in early 2021? Cribl. There’s literally no other reason for us to use Cribl.
I’ve been paying for Splunk since 2008 and can’t wait to get away from them. Their sales teams have decayed into unethical slimebags and I am trying everything in my power to not renew our contracts with them. This just sealed the deal.
Source: I cut checks to Splunk for $x,xxx,xxx yearly