An Open Letter from the CEO of Puppet: Puppet and Perforce(puppet.com)
puppet.com
An Open Letter from the CEO of Puppet: Puppet and Perforce
https://puppet.com/blog/an-open-letter-from-the-ceo-of-puppet//
153 comments
I think VMs are going to start coming back in vogue when companies now that are highly dependent on cloud services start to realize the cost swing back to owning bare metal machines at scale is actually cheaper once your general work loads stabilize.
I think Puppet and Ansible still have a future, its just a "down time" for their popularity right now.
I think Puppet and Ansible still have a future, its just a "down time" for their popularity right now.
Even if it's more cost effective to own the host machines (whether it be bare metal or VM), I'd guess you still save yourself considerable management headaches if you continue to put containers on them. There is still some stuff to manage with something like puppet and ansible, but I think if most of your configuration is in containers, it gets a lot simpler and these tools become less critical/complicated.
As noted elsewhere it was likely called and Open Letter because the acquisition is probably understood internally to be not what they wanted to happen, but they were out of options.
Agree! I felt like puppet missed an opportunity to be an IaaC for cloud providers/provisioning, especially with containers. From my experience, developers switch to other tools (terraform, ansible, etc) to fill the gap.
If that was the case we wouldn't have KataContainers or isolated Windows containers.
Apparently there are some benefits to VMs.
Apparently there are some benefits to VMs.
TL; DR: Puppet acquired by Perforce (yes, the version control software).
> (yes, the version control software)
Oh, didn't you see?
> What is Perforce?
> Perforce solutions power the technology that drives the digital economy. The company’s comprehensive DevOps portfolio and deep domain expertise drives quality, security, compliance, collaboration, and speed – across the technology lifecycle.
> Perforce has a 25 year-plus history of developing and supporting developer productivity tools.
So Perforce is actually, er, whatever that word salad means. It seems version control is not one of the things Perforce want associated with their name - which is also how i would feel if i'd developed Perforce.
Oh, didn't you see?
> What is Perforce?
> Perforce solutions power the technology that drives the digital economy. The company’s comprehensive DevOps portfolio and deep domain expertise drives quality, security, compliance, collaboration, and speed – across the technology lifecycle.
> Perforce has a 25 year-plus history of developing and supporting developer productivity tools.
So Perforce is actually, er, whatever that word salad means. It seems version control is not one of the things Perforce want associated with their name - which is also how i would feel if i'd developed Perforce.
This is a weird "open letter" but okay.
I feel like Puppet, for whatever reason, has lost out on things and Ansible won, which is quite unfortunate as Ansible is just not as good as Puppet. I wonder if Preforce will be able to revive the interest in Puppet.
Anyway, if someone plans to move away from Puppet (I imagine someone will, because someone always disagrees with a buyout), I've recently found mgmt[0] which has Puppet-lang compatibility. A big plus (to me anyway) is that it's not Ruby based, and is instead written in Go.
[0] https://github.com/purpleidea/mgmt
I feel like Puppet, for whatever reason, has lost out on things and Ansible won, which is quite unfortunate as Ansible is just not as good as Puppet. I wonder if Preforce will be able to revive the interest in Puppet.
Anyway, if someone plans to move away from Puppet (I imagine someone will, because someone always disagrees with a buyout), I've recently found mgmt[0] which has Puppet-lang compatibility. A big plus (to me anyway) is that it's not Ruby based, and is instead written in Go.
[0] https://github.com/purpleidea/mgmt
Why would it not being Ruby-based be a plus? Go is a big improvement over Ruby in some use cases, like where performance or concurrency is important, but this is not one of those cases.
Personal opinion, I'm sure many will disagree, but I've never been a fan of Ruby. It tends to be slow and, at least last time I used Puppet, there was a lot of dependency issues (from downloading them, to ensuring you run the right version of Ruby among other things). Whereas with Go downloading a single binary takes a whole lot of headaches away. Sure concurrency/performance won't matter much in the case of Puppet like software but depending on a single binary vs lots of dependency is a huge bonus.
Rails is often slow. Ruby's plenty fast though. Puppet was trying to do a lot with the way that it was designed and I think it's probably a side effect.
Ansible can be plenty slow at time too.
Ansible can be plenty slow at time too.
Rubygems is slow, the way that it requires files from libraries is by scanning through every gem, for every file, every time. Which is particularly brutal on Windows. It probably doesn't need to be that way via either extending the syntax to require from a specific gem and/or to have a file manifest database (and lose some manual hack-it-up-itude)
> Ansible can be plenty slow at time too.
Yep, I've never been a fan of Ansible. Beyond that slowness of it, the fact that it's duct tape of YAML and shell with randomness and a sprinkle of YAML craziness make it really hard for me to like it.
Yep, I've never been a fan of Ansible. Beyond that slowness of it, the fact that it's duct tape of YAML and shell with randomness and a sprinkle of YAML craziness make it really hard for me to like it.
Fwiw, despite all that I still prefer it to the alternatives because it doesn't require an agent and is easy to use anywhere you have SSH access. So whether it's officially part of the agreed upon tools or not, I can still use it to make my own life easier vs raw SSH.
Ansible won out because Puppet had scaling issues. OpenStack infra at one point was using Ansible to orchestrate puppet pulls, until everything could be re-written in Ansible
I'm genuinely interested in what the infrastructure was that couldn't support orchestration of Puppet clients. I hear this from people sometimes and it usually ends up being related to poorly architected Puppet infrastructure for their environment.
A properly architected Puppet environment should have no problems dealing with thousands of clients.
A properly architected Puppet environment should have no problems dealing with thousands of clients.
Depends what you want to do. If you're happy for changes to dribble in over time then size your puppetmaster pool to how many hosts you want to be able to run puppet simultaneously and stagger client execution to avoid a stampeding herd. Accept that sometimes there will be individual failures due to load and clients will just have to wait till the next time.
Alternatively, in real life, many teams have Change Management to consider and Maintenance windows. If there's a need to update thousands of systems on a saturday morning then expect teams to start puppet runs manually. You'd better have a seriously big pool of puppetmasters ready and waiting to manage the load, and don't forget Puppet DB, that has to be scaled up too to avoid lock ups. Even then, if teams start too many puppet runs at once, you'll get flattened.
We ended up scrapping all the puppetmasters in individual DCs and consolidating them in an AWS EC2 Autoscaling group. The number of puppetmasters started at 70 and just went up. That came with problems of its own. e.g. ensuring that all puppetmasters share the same copy of role versions at the same time. Being able to spin up new puppetmasters fast enough to meet spikes in demand. Various other corner case tuning issues.
It's taken a dedicated team years to get to grips with puppet, tame it and master it. Very glad I'm not involved in that any more.
Alternatively, in real life, many teams have Change Management to consider and Maintenance windows. If there's a need to update thousands of systems on a saturday morning then expect teams to start puppet runs manually. You'd better have a seriously big pool of puppetmasters ready and waiting to manage the load, and don't forget Puppet DB, that has to be scaled up too to avoid lock ups. Even then, if teams start too many puppet runs at once, you'll get flattened.
We ended up scrapping all the puppetmasters in individual DCs and consolidating them in an AWS EC2 Autoscaling group. The number of puppetmasters started at 70 and just went up. That came with problems of its own. e.g. ensuring that all puppetmasters share the same copy of role versions at the same time. Being able to spin up new puppetmasters fast enough to meet spikes in demand. Various other corner case tuning issues.
It's taken a dedicated team years to get to grips with puppet, tame it and master it. Very glad I'm not involved in that any more.
The non-deterministic run order was a nightmare to debug so everyone moved to ansible
That's exactly it. Trying to fight to get things to run in a proper order was a nightmare.
This leaves CFEngine as the last OG independent configuration management vendor:
Saltstack - acquired by VMWare
Ansible - acquired by Red Hat
Chef - acquired by Progress software
Saltstack - acquired by VMWare
Ansible - acquired by Red Hat
Chef - acquired by Progress software
What does that say about CFEngine?
That’s an honest question for all. Is it considered a good or bad thing?
That’s an honest question for all. Is it considered a good or bad thing?
> That’s an honest question for all. Is it considered a good or bad thing?
When we last experimented with CFEngine, we've concluded that "Friends don't let friends use CFEngine".
It's a state-forcing engine by default, and is very heavy handed at that.
Then I found SaltStack, and loving it to be honest.
When we last experimented with CFEngine, we've concluded that "Friends don't let friends use CFEngine".
It's a state-forcing engine by default, and is very heavy handed at that.
Then I found SaltStack, and loving it to be honest.
I haven't used CFEngine for about 5 years, but I don't think anybody would want to buy it or pick it for a new project. It feels like an unfinished experiment in the style of the 1990's.
It felt very rough around the edges a decade ago, and I have yet to see any real change since.
This blogpost is as unnecessarily complicated as Puppet codebase.
I hate working on Puppet codebase, it is so fragile and too easy to write spaghetti.
I hate working on Puppet codebase, it is so fragile and too easy to write spaghetti.
God this is just awful.
If your Open letter needs a FAQ at the end, you might as well start with the FAQ.
I am now reading this letter the third time. I dont hold anything against Puppet or Perforce. But this letter, there is something about this letter, and dare I say whether it was the PR or its CEO who wrote it, really irritates me.
If your Open letter needs a FAQ at the end, you might as well start with the FAQ.
I am now reading this letter the third time. I dont hold anything against Puppet or Perforce. But this letter, there is something about this letter, and dare I say whether it was the PR or its CEO who wrote it, really irritates me.
> there is something about this letter
It's the self-importance of a company which I only just realised isn't just a GitHub repo somehow thinking they need to write a letter longer than Kurt Cobain's suicide note to tell me that they got acquired by some other bland-as-fuck open-core company.
All we needed to hear was:
> Hi, I'm the CEO of Puppet. Yeah, that thing you use to do stuff in Chrome with code. Yeah, yeah, we're a company, yup, somehow. Anyway: we're getting acquired by this other company. They're called Perforce. No, you don't know them either. Yeah, they also have some indecipherable enterprise website with some Gartner Quadrant bollocks on it. No, you don't really need to know any of this.
Edit: Fuck, that's Puppeteer. OK, I just have no idea what this company does. Oh well.
It's the self-importance of a company which I only just realised isn't just a GitHub repo somehow thinking they need to write a letter longer than Kurt Cobain's suicide note to tell me that they got acquired by some other bland-as-fuck open-core company.
All we needed to hear was:
> Hi, I'm the CEO of Puppet. Yeah, that thing you use to do stuff in Chrome with code. Yeah, yeah, we're a company, yup, somehow. Anyway: we're getting acquired by this other company. They're called Perforce. No, you don't know them either. Yeah, they also have some indecipherable enterprise website with some Gartner Quadrant bollocks on it. No, you don't really need to know any of this.
Edit: Fuck, that's Puppeteer. OK, I just have no idea what this company does. Oh well.
Puppet is one of the DevOps configuration languages. IIRC it had mostly "won" against Chef, and subsequently "lost" against Ansible.
Perforce is basically Git, but shittier and proprietary. I think it's mostly popular in the games industry.
Perforce is basically Git, but shittier and proprietary. I think it's mostly popular in the games industry.
Interesting. My recollection is that Chef and Ansible both completely ran roughshod over Puppet, and that Puppet is now in the "antiquated tools that used to be relevant" bucket.
Anecdotal, yeah. But DevOps is my daily work, and I like to think I keep an eye on the tools ecosystem.
Anecdotal, yeah. But DevOps is my daily work, and I like to think I keep an eye on the tools ecosystem.
That was my sense too back than, Puppet had early traction, but lost out to Chef, and then Ansible turned out to be the marathoner of the 3.
Back in the day (~10 years ago?) when the company I worked for was looking at replacing our internal custom provisioning and desired state configuration tooling it was Ansible that won out. We were running a mixed fleet of Windows and Linux hosts - the company was an ISP and shared hoster.
At that point in time both Puppet and Chef needed agents to be installed on your servers and we really didn't want to have to manage another background service running on hundreds to thousands of boxes. Ansible didn't have this requirement.
Also Windows support was pretty sketchy on Chef and Puppet back then, to the point of being almost studiously ignored, Ansible provided remote PowerShell support. And that was primarily the deciding factor.
At that point in time both Puppet and Chef needed agents to be installed on your servers and we really didn't want to have to manage another background service running on hundreds to thousands of boxes. Ansible didn't have this requirement.
Also Windows support was pretty sketchy on Chef and Puppet back then, to the point of being almost studiously ignored, Ansible provided remote PowerShell support. And that was primarily the deciding factor.
I used Puppet around 2012. I remember looking at Ansible, and immediately seeing that its agentless, SSH-based approach was highly preferable. But every time i discussed it with the devops experts in and contracted to our company, they assured me that agentless was terrible, and had too many disadvantages, and couldn't scale.
And yet, here we are. Puppet grew an agentless mode, and then lost anyway. My former company's deep investment in integrating Puppet and MCollective into our infrastructure tooling must look increasingly like a millstone. Oh well, it was all interesting to work on.
And yet, here we are. Puppet grew an agentless mode, and then lost anyway. My former company's deep investment in integrating Puppet and MCollective into our infrastructure tooling must look increasingly like a millstone. Oh well, it was all interesting to work on.
Seems like doing $100M+ in ARR (as hinted in the article) is a win. Even more because Chef was acquired for $220M. I think Perforce will pay a bit more for Puppet.
Their last round in 2020 ($40M from Black Round) was debt financing, not investment. That to me is generally a sign of a struggling company. And when the CEO of such a company claims they were looking for other companies to acquire, it raises a number of questions.
The shorthand at the time--don't know how accurate--was that sysadmins tended to prefer Puppet and devs tended to prefer Chef. But, yes, I remember from attending Config mgmt camp a few times during the period that Ansible was really spiking in popularity. From what I saw, Ansible was easier to get started with and probably more appropriate for more general IT automation tasks.
Perforce is probably more widespread than people realise, and not just games industry although they've focused more on the benefits they bring to that field in recent years. Perforce were offering a good quality version control system for years before the initial release of git, as a result it can still be found in many older companies who have projects that are older than git itself and haven't migrated.
Git has obviously won the VCS battle now in most fields, but version control used to be a much more diverse and interesting space; I'd used Perforce, SVN and Mercurial professionally before git just won out most places.
Git has obviously won the VCS battle now in most fields, but version control used to be a much more diverse and interesting space; I'd used Perforce, SVN and Mercurial professionally before git just won out most places.
Perforce is "Git for people who don't want to merge a 200MB binary FBX asset". Their core value proposition is more like SVN or Microsoft's Version Control in that people check out a file and then only they are allowed to modify it until they check it back in. That way, merges can be avoided.
And yes, it's very popular in the games industry, because even Git+LFS is very painful if you have merge conflicts.
And yes, it's very popular in the games industry, because even Git+LFS is very painful if you have merge conflicts.
It's popular in the gaming industry because it handles large individual assets better than anything out there.
Hence the invention of things like Git LFS:
https://git-lfs.github.com/
https://git-lfs.github.com/
Thanks for the no-bullshit explanation - much appreciated! That letter makes me think approximately the same thing I think to myself when someone tells me their sister got married: like, "hmm, I'm not really sure why you're telling me this, but good for them I guess."
> Perforce is basically Git, but shittier and proprietary. I think it's mostly popular in the games industry.
I think Google also uses Perforce. It's understandable because Google started before git.
I think Google also uses Perforce. It's understandable because Google started before git.
They don't, though they used to long ago.
https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
https://m-cacm.acm.org/magazines/2016/7/204032-why-google-st...
Great summary! This should have been in the FAQ
p4 was known to handle large binary blobs well, vs most of the other vcss.
While I agree with you, Perforce is not open-core at all, in fact, they're are fully closed-source[1]
[1] https://en.wikipedia.org/wiki/Perforce#Helix_Core
[1] https://en.wikipedia.org/wiki/Perforce#Helix_Core
While the Helix Core Server itself is not Open Source, much of the surrounding tooling is. E.g., see the code here: https://swarm.workshop.perforce.com/user/perforce_software
I don't think it'd be fair to say it's "fully" closed-source.
Ah, fair enough. I suppose I should offer an explicit disclaimer: my descriptions of both companies aren't necessarily 100% accurate, haha.
How has nobody heard of Perforce. It was the SCM of choice for a lot of companies until git came along. Doesn't Google still use it?
BTW, Perforce's diffing/merging tools [0] are fantastic.
[0] https://www.perforce.com/products/helix-core-apps/merge-diff...
BTW, Perforce's diffing/merging tools [0] are fantastic.
[0] https://www.perforce.com/products/helix-core-apps/merge-diff...
Instead of ranting about how much I am annoyed by this problem, let me just ask: Are there any good diff tools/algorithms that don't consistently freak out when you change the indentation of a block of code? Yes, diff tools like to go by unique/common lines, and yes, those two } are technically the only common line in this chunk, but that is beyond useless when all you did was wrap something in an if/else.
(And yes, P4V tools have "ignore all white space" - which is a great way to produce a completely useless merge result, in my experience. If you can tune out of nonsensical indentation in one side of the comparison, it's at least passable for diffing.)
(And yes, P4V tools have "ignore all white space" - which is a great way to produce a completely useless merge result, in my experience. If you can tune out of nonsensical indentation in one side of the comparison, it's at least passable for diffing.)
There are starting to be more code-aware diff tools.
https://github.com/Wilfred/difftastic
I'm currently trying this out, but I'm not sure if I'll keep using it.
https://github.com/Wilfred/difftastic
I'm currently trying this out, but I'm not sure if I'll keep using it.
Rational only stored the diffs between two decorated abstract syntax trees. So you got as-you-like it pretty printing, no conflicts on formatting, just everything you ever dreamed of. In 1987.
Last time I used Perforce, more than a decade ago, the visual diff app was fantastic and all other aspects were total shit. Would rather use Subversion if someone insisted on not using git.
Google has something that acts sort of like it with a bit of modernisation, but it's not actually Perforce any more (it's called Piper, there's some info out there). They out-scaled Perforce some years back.
i thought google used mercurial (hg) ?
They were on Perforce for a long time.
See e.g. https://info.perforce.com/rs/perforce/images/GoogleWhitePape... and https://news.ycombinator.com/item?id=15889958.
See e.g. https://info.perforce.com/rs/perforce/images/GoogleWhitePape... and https://news.ycombinator.com/item?id=15889958.
I don't think Perforce is open core. It's open plugin, aka Embrace Extend, Extinguish.
Anyway, emotions aside, it's an important company with a product used by important companies.
Anyway, emotions aside, it's an important company with a product used by important companies.
Could you give an example of where it has Embraced->Extinguished?
> Yeah, that thing you use to do stuff in Chrome with code.
In where?
In where?
I'm not sure I can parse this question successfully... But, in case this happens to answer your question, it was Puppeteer I had in mind (like I mentioned in the edit).
I meant why Chrome? When I used Puppet, it was in terminal and in git to edit stuff. Parsing is easy, use keyword in.
Puppeteer is an automation library to remote control Chrome.
The parent poster had confused Puppet (the deployment automation software) and Puppeteer (the chrome remote control software).
The parent poster had confused Puppet (the deployment automation software) and Puppeteer (the chrome remote control software).
Yup, in answer to the guy above: this is correct. Thanks for clarifying for me! (Some day HN will figure out notifications and I’ll be able to reply to comments myself within 24 hours...)
[deleted]
I had to read that a lot more carefully than I wanted in order to figure out that it wasn't about something that was terribly interesting.
(I'm not impressed with Puppet, having had to wade through its innards from time to time. Perforce is a fine centralized code repo. In any event: Shrug, CxO needs an editor).
(I'm not impressed with Puppet, having had to wade through its innards from time to time. Perforce is a fine centralized code repo. In any event: Shrug, CxO needs an editor).
I think my problem is the writing style is that of a Twitter thread or LinkedIn post.
It's really atrocious in so many ways.
[deleted]
"Perforce" is not mentioned until the fifth paragraph.
When writing a document like this, open with your most important statement. For example: Title: "Perforce Acquires Puppet." Then, "We are announcing that Perforce acquired Puppet. When I started at Puppet three years ago..."
There. Was that too hard?
When writing a document like this, open with your most important statement. For example: Title: "Perforce Acquires Puppet." Then, "We are announcing that Perforce acquired Puppet. When I started at Puppet three years ago..."
There. Was that too hard?
No, no, you don't understand.
Announcing an acquisition from the perspective of the acquired, is typically considered Bad News. So the exec rulebook imposes that you will try to soften the blow by talking up how great things are anyway and how marvellously his tenure has been. That is the most important bit, from his perspective. His target is to get you to read paragraph after paragraph of spin, so that when the Bad News comes, you'll be (if not persuaded) at least willing to consider it might not be too bad after all.
It's like supermarkets placing daily essentials (bread, milk, meat) at the far end of the shop, so that you'll be forced to wander through and likely buy other stuff.
Announcing an acquisition from the perspective of the acquired, is typically considered Bad News. So the exec rulebook imposes that you will try to soften the blow by talking up how great things are anyway and how marvellously his tenure has been. That is the most important bit, from his perspective. His target is to get you to read paragraph after paragraph of spin, so that when the Bad News comes, you'll be (if not persuaded) at least willing to consider it might not be too bad after all.
It's like supermarkets placing daily essentials (bread, milk, meat) at the far end of the shop, so that you'll be forced to wander through and likely buy other stuff.
This right here is the salient point, which explains among other things why she called it an open letter. It's not good news, and the implication is the letter is addressing Puppet employees, cc general public.
Although she's desperately spinning it as positive, the fact that the announcement came from the acquiree's side, not the acquirer implies this was an end of the line option for them, maybe even a fire sale.
And the CEO's writing style here does little to dissuade one from drawing the conclusion Puppet has been suffering through a lack of anything resembling leadership from her for the past 3 years.
Although she's desperately spinning it as positive, the fact that the announcement came from the acquiree's side, not the acquirer implies this was an end of the line option for them, maybe even a fire sale.
And the CEO's writing style here does little to dissuade one from drawing the conclusion Puppet has been suffering through a lack of anything resembling leadership from her for the past 3 years.
FYI, absolutely all the exec / management courses that I have taken specifically advise against dragging a "bad" news after a sea of explanation.
All the "exec rulebook" tell the same thing: announce the news first in simple - unambiguous terms terms - and their immediate, impactful consequences. Then provide the "why", then provide a prospect for the future.
Yvonne Wassenaar did not apply the exec rulebook, she should have.
All the "exec rulebook" tell the same thing: announce the news first in simple - unambiguous terms terms - and their immediate, impactful consequences. Then provide the "why", then provide a prospect for the future.
Yvonne Wassenaar did not apply the exec rulebook, she should have.
That's the sort of thing they teach you in school because few actually do it, like using "were" for hypotheticals.
In my experience, most managers will try their hardest to sandwich bad news in a sea of platitudes.
In my experience, most managers will try their hardest to sandwich bad news in a sea of platitudes.
*Her
But that's not fearless and brave!
[deleted]
Perforce is owned by two private equity companies. The ownership is split 50-50 between Clearlake Capital and Francisco Partners. It looks like PEs invest in Perforce and use them to acquire smaller companies in the dev tools area. Perforce owns Selenium and RogueWave Software. I wouldn't be surprised if they become a public company at some point so the PE can cash on their investment.
If they use the standard private equity playbook all the good engineering staff will be gone in a year and all new feature development will stop while at the same time the sales/legal staff will grow to exploit the maintenance needs of enterprise Puppet users.
I’m not a cloudy/devops guy. I work on the realms that are mostly not cloudy. I figured puppet was some sort of enterprisey thing, but just know it “as one of those names.”
Anyway, I read through the first 3 paragraphs and still had no clue what puppet actually does/provides. But I knew that they had a leader, and that leader could use lots of big businessy words.
Curious why this is uptrending on HN. Do lots of people here use this? Usually ally this crowd goes more for the tech talk stuff.
Anyway, I read through the first 3 paragraphs and still had no clue what puppet actually does/provides. But I knew that they had a leader, and that leader could use lots of big businessy words.
Curious why this is uptrending on HN. Do lots of people here use this? Usually ally this crowd goes more for the tech talk stuff.
Puppet was a pretty important company in the configuration management and DevOps (the two things being only somewhat related) space at one point. But configuration management moved on to a large degree from mutable server instances/VMs so automation tools have either shifted their focus or they've become less relevant.
Given this announcement I feel like I kinda have my answer.
But I have to wonder how popular Puppet is now? I used it extensively... 6 years ago? somewhere in that range. I don't remember for sure. I went to training and though it worked pretty well. I remember there being a lot of excitement around Puppet.
But since then I have moved more into AWS, with docker and serverless. Now I know as much as AWS and Docker love to claim otherwise... there is a large portion of companies that have not moved to this type of infrastructure.
So I am legitimately curious where they stand. Now I have to imagine if I was ever in a situation where I needed to build a machine/image I would use Packer or EC2 Image Builder (I think that's the name).
But I have to wonder how popular Puppet is now? I used it extensively... 6 years ago? somewhere in that range. I don't remember for sure. I went to training and though it worked pretty well. I remember there being a lot of excitement around Puppet.
But since then I have moved more into AWS, with docker and serverless. Now I know as much as AWS and Docker love to claim otherwise... there is a large portion of companies that have not moved to this type of infrastructure.
So I am legitimately curious where they stand. Now I have to imagine if I was ever in a situation where I needed to build a machine/image I would use Packer or EC2 Image Builder (I think that's the name).
Places that run tens or hundreds of thousands of machines generally really need something managing the mutable space between the base image and the container to make sure things get updated more or less continuously for security patches, user access, etc. This space gets a lot thinner with containers, but doesn't disappear completely, in my experience. This is where something like Puppet comes in. I know there's the immutable infra movement, but I don't know how practical it really is to shoot every single server in the head and recycle it when a critical vulnerability comes out. Maybe others have seen this, but I haven't.
It doesn't matter if it's AWS or an application.conf, you still want to manage the state of your infrastructure.
That management might be Terraform and Helm, but it could easily be argued that a module for Puppet or Ansible would have several advantages.
Unfortunately, instead of nourishing their open source roots they went corporate and began making enterprise products instead. It is an easy way to profitability but at the same time will make yourself less relevant for your user base. Compare with Docker Inc. and Elastic NV.
They had the misfortune of doing this at the same time as Python was edging out Ruby's mindshare, also visible in the slow shift from Rails to Django over the decade, thereby doubling down on the road to slow irrelevancy.
This is all pretty unfortunate since the sysadmin tooling in the cloud native era leaves a lot to be desired, but it is what it is.
That management might be Terraform and Helm, but it could easily be argued that a module for Puppet or Ansible would have several advantages.
Unfortunately, instead of nourishing their open source roots they went corporate and began making enterprise products instead. It is an easy way to profitability but at the same time will make yourself less relevant for your user base. Compare with Docker Inc. and Elastic NV.
They had the misfortune of doing this at the same time as Python was edging out Ruby's mindshare, also visible in the slow shift from Rails to Django over the decade, thereby doubling down on the road to slow irrelevancy.
This is all pretty unfortunate since the sysadmin tooling in the cloud native era leaves a lot to be desired, but it is what it is.
Textbook burying of the lede here.
Reading this was almost as unnecessarily complex as using Puppet for DevOps.
I thought it was only who didn't understand this letter. As an open letter it's confusing, as an announcement it doesn't look like so, well it's just not well written.
Puppet is one of those tools I refuse to use unless there's absolutely no other way to get something done. Mutable state is an anti pattern. Immutable versioned artifacts are the way.
In theory yes. In practice, no.
While immutable assets (I imagine you mean container or VM images) are easy to audit, they don't solve all problems. How do you spin up a database and add database-users over time? Do you just just rebuild your container-VM from scratch and loose your data? With Ansible, I can just enforce a new database user as I run a new service against this database.
Also, how do you spin up the host which runs your static VM and container images?
There are tons of problems which can't be solved by immutable artifacts. Yes, immutable containers/VMs for application code. But it doesn't work well when dealing with databases, caching systems and object storages. People deploying Postgres, Redis and/or Minio on k8s or openshift are facing a totally different set of problems which are already solved by mutable state.
While immutable assets (I imagine you mean container or VM images) are easy to audit, they don't solve all problems. How do you spin up a database and add database-users over time? Do you just just rebuild your container-VM from scratch and loose your data? With Ansible, I can just enforce a new database user as I run a new service against this database.
Also, how do you spin up the host which runs your static VM and container images?
There are tons of problems which can't be solved by immutable artifacts. Yes, immutable containers/VMs for application code. But it doesn't work well when dealing with databases, caching systems and object storages. People deploying Postgres, Redis and/or Minio on k8s or openshift are facing a totally different set of problems which are already solved by mutable state.
In every environment I've worked in, the content of a DB is not considered a build artifact. The container that actually runs e.g. PostgreSQL is, but it will use volume mounts.
I think what OP is referring to is the fact that the container itself is stateless. If you (to continue the PostgreSQL example) needed to upgrade the version of a dependency in the container, you'd publish a new artifact and use the same volume mount, performing a data migration as you go. Rather than performing upgrades to an already running instance. ("containers are cattle, not pets").
I think what OP is referring to is the fact that the container itself is stateless. If you (to continue the PostgreSQL example) needed to upgrade the version of a dependency in the container, you'd publish a new artifact and use the same volume mount, performing a data migration as you go. Rather than performing upgrades to an already running instance. ("containers are cattle, not pets").
I think the point of GP is that you really can't completely take the position that "mutable state is an anti-pattern" since that volume you've mounted is itself mutable state.
GP here, exactly. Also how does one manages these mutable states? And these volumes?
Most of these hosts are managed with Ansible, which the Top-Poster was criticising so heavily as "an anti-pattern".
Most of these hosts are managed with Ansible, which the Top-Poster was criticising so heavily as "an anti-pattern".
I was referring to your comment about having to recreate build images every time you want to insert data (users) into the DB. I don't see how _any_ deployment solution could be stateless, if you want a database. I might be misinterpreting you though.
> If you (to continue the PostgreSQL example) needed to upgrade the version of a dependency in the container, you'd publish a new artifact and use the same volume mount, performing a data migration as you go.
You're just proving the point I made: "People deploying Postgres, Redis and/or Minio on k8s or openshift are facing a totally different set of problems which are already solved by mutable state."
When one does what you describe, they now have to deal with downtime, etc...
You're just proving the point I made: "People deploying Postgres, Redis and/or Minio on k8s or openshift are facing a totally different set of problems which are already solved by mutable state."
When one does what you describe, they now have to deal with downtime, etc...
I think we're talking past each other because containerization solutions aren't designed to solve the problem you're talking about. It's just not the target problem domain. That isn't a shortcoming of containerization. If you need zero downtime, there are ways to make that work using additional tooling. I think we may actually agree on that point.
In that spirit, Perforce is a great match. Nobody willingly uses Perforce for version control.
Having immutable ideas is considered harmful, though.
Versioned immutable artifacts help reducing incidents and increase traceability, but they're not silver bullets, and many teams were working well before these are popularized.
Taking this immutability extreme creates a lot of headaches and problems, too.
Versioned immutable artifacts help reducing incidents and increase traceability, but they're not silver bullets, and many teams were working well before these are popularized.
Taking this immutability extreme creates a lot of headaches and problems, too.
I would love software it is wasn't for mutable state!
I would also love to run a business if it wasn't for all those pesky customers!
I would also love to run a business if it wasn't for all those pesky customers!
What does it mean that it took 4 paragraphs and 400 words before it got to the point? Seems really weird.
So long Puppet, have some fond memories. Good luck with the OSS thing at Perforce.
Could you give an example of where Perforce is not OSS-friendly? Sure, the product core is not OSS, but parts of the periphery are, and there's at least one section of the company whose entire business model is supporting OSS:
https://www.openlogic.com/ https://www.openlogic.com/about-openlogic-perforce
https://www.openlogic.com/ https://www.openlogic.com/about-openlogic-perforce
Ugh Perforce...
Poor Perforce. Once upon a time it was a very useful and interesting product. For those of you who have only known a world of git, there was a time where a product like perforce was just awesome. Fast, functional, reasonably priced, and supported a variety of platforms (VMS!). But that was a long time ago.
Once git arrived, they didn't have the same place in the world and now its... a very different company.
Once git arrived, they didn't have the same place in the world and now its... a very different company.
The only advantage Perforce has, and the reason why giants such as Ubisoft have committed to it, is that "it handles large files better". That's it.
> The only advantage Perforce has,
Perforce is also ripping fast for updating/status checking, (run git status on the Unreal Engine repo to see the difference), is centralized (a feature, not a bug), has full support for partial and shallow checkouts. It also offers support for _running_ perforce, not just their hosted offering.
> and the reason why giants such as Ubisoft have committed to it
The reason companies like Ubisoft, Epic, etc use it is because it predates git.
> is that "it handles large files better". That's it.
Lets' not pretend that "better" is a good comparison. Until 5-6 years ago, git didn't handle large binary files at all. Even now, it's a "pick your poison".
Perforce is also ripping fast for updating/status checking, (run git status on the Unreal Engine repo to see the difference), is centralized (a feature, not a bug), has full support for partial and shallow checkouts. It also offers support for _running_ perforce, not just their hosted offering.
> and the reason why giants such as Ubisoft have committed to it
The reason companies like Ubisoft, Epic, etc use it is because it predates git.
> is that "it handles large files better". That's it.
Lets' not pretend that "better" is a good comparison. Until 5-6 years ago, git didn't handle large binary files at all. Even now, it's a "pick your poison".
Did they ever get over the idea that a version control system should try to prevent two people from modifying (or "checking out") a file at the same time? That always drove me insane. Every time I got on a plane I'd have to do whatever trick that was that let me edit files even though I never told the server I was going to, and deal with the consequence later. I don't exactly remember the details or how bad the consequences were, but it always seemed ridiculous.
Oh, and cloning a repo so you have a second copy of it for an experiment took ages because you couldn't just 'cp -a' the directory, you had to "enlist" in all the files with the server... I do not recall p4 fondly.
Oh, and cloning a repo so you have a second copy of it for an experiment took ages because you couldn't just 'cp -a' the directory, you had to "enlist" in all the files with the server... I do not recall p4 fondly.
> Did they ever get over the idea that a version control system should try to prevent two people from modifying (or "checking out") a file at the same time?
Checking out is still a fundamental concept of perforce, and it's the reason it's so fast. Not all files need exclusive locks, that's usually reserved for binary assets (and is a selling point of perforce as it prevents an unsolvable merge conflict later on). When I started using perforce 13 years ago, every editor I used had either a perforce plugin, or an option to "checkout on edit or save",
> Oh, and cloning a repo so you have a second copy of it for an experiment took ages because you couldn't just 'cp -a' the directory, you had to "enlist" in all the files with the server...
I don't really see the problem. doing `cp -a` on a large git repo can be an extra 30GB if not more on disk. The tradeoff of a very uncommon operation (duplicating a repo for an experiment sounds like a great use for shelves in p4, or branches in git) taking longer for performance on the golden path seems like a really good one to me.
Checking out is still a fundamental concept of perforce, and it's the reason it's so fast. Not all files need exclusive locks, that's usually reserved for binary assets (and is a selling point of perforce as it prevents an unsolvable merge conflict later on). When I started using perforce 13 years ago, every editor I used had either a perforce plugin, or an option to "checkout on edit or save",
> Oh, and cloning a repo so you have a second copy of it for an experiment took ages because you couldn't just 'cp -a' the directory, you had to "enlist" in all the files with the server...
I don't really see the problem. doing `cp -a` on a large git repo can be an extra 30GB if not more on disk. The tradeoff of a very uncommon operation (duplicating a repo for an experiment sounds like a great use for shelves in p4, or branches in git) taking longer for performance on the golden path seems like a really good one to me.
For me, duplicating a repo usually means i want the two different things side by side at the same time, rather than a single directory switching between branches.
It also offers granular ACLs where git doesn't.
Also has partial checkouts, important for huge repos.
Ive done partial checkouts in Git before as well.
Its called a "sparse checkout".
https://www.git-scm.com/docs/git-sparse-checkout
Its called a "sparse checkout".
https://www.git-scm.com/docs/git-sparse-checkout
Yes, but (sorry for yelling, that's in the source)
> THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN THE FUTURE.
> THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN THE FUTURE.
[deleted]
Are any notable companies still using Puppet? Does Google still use it to manage their Macs?
Google migrated off Perforce years ago, but now they are Perforce customers again, lol.
[deleted]
So, the founder is ashamed of the acquisition and wrote this corpspeak BS to... assuage their feelings?
[deleted]
tl;dr: Perforce Acquires Puppet
They seem to have given their blog post a pretty weird title. An open letter is usually a complaint, with an invitation for the general public to join in. This seems to be more of an announcement that they were acquired.
The whole time I was reading it, I was expecting a "but this acquisition is actually bad because <reasons that indicate something generally toxic about the industry>."
https://www.merriam-webster.com/dictionary/open%20letter
The whole time I was reading it, I was expecting a "but this acquisition is actually bad because <reasons that indicate something generally toxic about the industry>."
https://www.merriam-webster.com/dictionary/open%20letter
I think your source is overstating it. OED says "a letter addressed to a particular person or persons but intended for a more public readership, as by deliberate publication in a newspaper or journal.", which sounds about right to me.
This, however, is not obviously addressing a particular person or persons. It's just a blog post.
This, however, is not obviously addressing a particular person or persons. It's just a blog post.
It might just be that my summary was off, Merriam seems pretty close to yours:
"a published letter of protest or appeal usually addressed to an individual but intended for the general public."
I guess I added the implication that the public is "invited to join in."
"a published letter of protest or appeal usually addressed to an individual but intended for the general public."
I guess I added the implication that the public is "invited to join in."
Yep I was confused reading through it. Only at the end in the FAQ was it clear what they were saying.
It would have been clearer to have the title say this is about their acquisition by Perforce. :)
It would have been clearer to have the title say this is about their acquisition by Perforce. :)
[deleted]
[deleted]
Puppet has to be one of the worst peices of software I've ever used. The language is full of terrible ideas (lets get rid of loops since those confuse sysadmins and make the only replacement require combinators via the goofy custom functions) and the documentation is full of broken links. Hira data always leaves you guessing at what a value actually is. It's like JS, you can make it work but it requires a ton of discipline and effort and is just ridiculous.
If you need to write unit tests for your deployment automation then you might be over complicating it.
If you need to write unit tests for your deployment automation then you might be over complicating it.
One of the worst classes of software is the software whose architecture starts out with "Turing completeness is bad and hard and we'll just ban it", but it turns out the problem space needs it, whether you like it or not, and it ends up creeping back in one step at a time anyhow.
Terraform kinda took a similar path, although the reigning champion of this style of software has to be HTML templating solutions. "Templates shouldn't have code in them! Ever! Any! Totally broken bad bad bad, and that's why I started this new template language that has no code constructs in it at all! Although, uh, we do kinda need to be able to loop through things like table rows from databases... and, well, we need if statements to highlight a particular result... and, well, we need full arithmetic support so we can use the modulo operator on those rows... and we need functions to factor out reusable functionality at the template layer... and oh crap that's all you need for Turing Completeness, isn't it? Well... uhh..." "Hi, I'm another developer and I'm starting a new template language because that previous one is broken and busted and written by dum dums because it has Turing completeness in it! Templates shouldn't have any code in them, ever! Although, uh, I do kinda need for loops for table rows from databases..."
Terraform kinda took a similar path, although the reigning champion of this style of software has to be HTML templating solutions. "Templates shouldn't have code in them! Ever! Any! Totally broken bad bad bad, and that's why I started this new template language that has no code constructs in it at all! Although, uh, we do kinda need to be able to loop through things like table rows from databases... and, well, we need if statements to highlight a particular result... and, well, we need full arithmetic support so we can use the modulo operator on those rows... and we need functions to factor out reusable functionality at the template layer... and oh crap that's all you need for Turing Completeness, isn't it? Well... uhh..." "Hi, I'm another developer and I'm starting a new template language because that previous one is broken and busted and written by dum dums because it has Turing completeness in it! Templates shouldn't have any code in them, ever! Although, uh, I do kinda need for loops for table rows from databases..."
Doesn't React somewhat solve this, when used correctly? (Though it's entirely possible to make an even bigger mess with React.)
Isn't React Turing-complete?
I think they mean it doesnt reinvent for/if like angular and vue do..
You just use the js primitives.
You just use the js primitives.
I wasn't 100% sure enough about React to be sure, but based on my limited experience, that sounds correct, so I don't think it fits my example.
I'm looking more at things like the Django template library, handlebars, Go HTML templates, etc. etc. And most especially things written with the idea that code shouldn't be in templates. Template languages that were always capable of running code, and were simply separate languages so designers don't have to learn a general-purpose programming language but can learn just this focused thing they need are obviously not what I mean.
I've drunk from the "templates shouldn't have logic in them" well too, but I think as you scale up it just becomes impractical. And in that case, you're better off with a template that always knew it was going to have code in rather than a template language that was designed from the beginning to not have it, but had code sneak in through the back door anyhow. The first one will at least be designed. The second one... well....
I'm looking more at things like the Django template library, handlebars, Go HTML templates, etc. etc. And most especially things written with the idea that code shouldn't be in templates. Template languages that were always capable of running code, and were simply separate languages so designers don't have to learn a general-purpose programming language but can learn just this focused thing they need are obviously not what I mean.
I've drunk from the "templates shouldn't have logic in them" well too, but I think as you scale up it just becomes impractical. And in that case, you're better off with a template that always knew it was going to have code in rather than a template language that was designed from the beginning to not have it, but had code sneak in through the back door anyhow. The first one will at least be designed. The second one... well....
Yes, exactly.
Exactly! And once you bring back recursion in the new language you're without the tools and practices everyone has developed over the past half century. Everything is just so much harder.
Really what we should be doing is generating data structures in <whatever> language rather than creating new languages.
Really what we should be doing is generating data structures in <whatever> language rather than creating new languages.
*cough* Chef *cough* Ruby *cough*
As someone that has built and maintained several Puppet based infrastructure environments, I can get the frustration of Puppet when it is not configured correctly or left to its own devices. You absolutely need to keep Puppet up to date in your environment or it will steamroll you over time with pain/upgrades.
However, I do need to correct you that Puppet does support loops and various other ways to munge data for quite a few years/versions now. In addition, _Hiera_ is not that bad once you understand the overall hierarchy of your infrastructure and it gets easier if you shift the default data store to something like HashiCorp Vault for storing secret data.
We get away with not writing tests (we do heavy linting/checks though) in our Puppet infrastructure currently because everything has been documented for our development teams on best practices, trainings, and just communicating expectations well. There are some things that can sneak thru sure, but overall we've had #greatsuccess with just being open about we expect in our repos and being approachable to new committers for onboarding reasons.
Puppet can be a beast to keep up to date I will say, but if you have a good plan in place it's really a wonderful tool for everyone involved.
However, I do need to correct you that Puppet does support loops and various other ways to munge data for quite a few years/versions now. In addition, _Hiera_ is not that bad once you understand the overall hierarchy of your infrastructure and it gets easier if you shift the default data store to something like HashiCorp Vault for storing secret data.
We get away with not writing tests (we do heavy linting/checks though) in our Puppet infrastructure currently because everything has been documented for our development teams on best practices, trainings, and just communicating expectations well. There are some things that can sneak thru sure, but overall we've had #greatsuccess with just being open about we expect in our repos and being approachable to new committers for onboarding reasons.
Puppet can be a beast to keep up to date I will say, but if you have a good plan in place it's really a wonderful tool for everyone involved.
Unless they changed something significant since I used it last, the puppet language is Ruby. They didn't get rid of loops, they used functional iteration (recursion) so you don't have side effects; it's declarative. Ansible uses YAML to do the same, and it's really the only way I'm aware to get idempotent results.
Your last sentence is true only for the most trivial of deployments, at which point, why are you using deployment automation in the first place, because it's a huge tech cost to invest in. If your deployments are that simple, why ass complexity by using a deployment automation and not just do a bin dump?
Your last sentence is true only for the most trivial of deployments, at which point, why are you using deployment automation in the first place, because it's a huge tech cost to invest in. If your deployments are that simple, why ass complexity by using a deployment automation and not just do a bin dump?
Perforce has to be one of the worst pieces of software I've ever encountered. Every perforce shop I go to treats perforce like a disk or san with little to do with versioning or software development. Find me a perforce instance that doesn't have gigabytes or terabytes of binary data. You can't... it's become a catch all for everything.
One shop I was at that's fairly well known had multiple fedora and centos RPM repos checked in. Compiled versions of Linux from version 2 to 5, silverlight exe's, pdf's, hundreds of excel files, what I had a hard time finding was actual source code.
I couldn't think of two better pieces of software to go die together.
One shop I was at that's fairly well known had multiple fedora and centos RPM repos checked in. Compiled versions of Linux from version 2 to 5, silverlight exe's, pdf's, hundreds of excel files, what I had a hard time finding was actual source code.
I couldn't think of two better pieces of software to go die together.
[deleted]
But the time of VMs has passed for most companies and by now there are better solutions available, so it makes sense to sell this off as long as it's worth something.
Not sure why this announcement was called an Open Letter?