"We ran out of columns"(jimmyhmiller.github.io)
jimmyhmiller.github.io
"We ran out of columns"
https://jimmyhmiller.github.io/ugliest-beautiful-codebase
587 コメント
This is amazing. I can so well imagine a bright young hire joining that team, helpfully offering to "setup this thing called git" only to be laughed out of the meeting by all the "senior" staff.
Astonishingly, It took a long time for revision control to become widespread.
Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later).
Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there).
Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes.
A few years later a preventable crisis was how revision control first became client-server.
Around 1991 when Cygnus had 6-7 employees and was based in the apartment complex where I lived, none of the GNU codebase was hosted in any sort of revision control. Everything was FTPed around as obscurely named tarballs. We had gathered something like 27 different forks of gdb floating around the net, for example. This was back when forking was generally considered a tragedy, something I managed to change five or six years later).
Rich Pixley came and said “all of our code should be in revision control, and I want to use this newfangled thing called CVS.” Michael was OK with it but John and I were steadfastly opposed. We agreed to an experiment, grudgingly, subject to a whole whiteboard of absurd conditions (“must be transparently integrated into emacs so we don’t have to know it’s there).
Pixley agreed to all of that and then ignored all of it completely. It was immediately such a win that everybody adopted it without complaint, including us two obstreperous a-holes.
A few years later a preventable crisis was how revision control first became client-server.
Why is it astonishingly?
Git is not a natural development at all. Obviously, it is a standard right now.
But I as a hobby coder at my teens I started out with FileZilla and copying over index2.php, index3.php, index_final.php, index_final_2.php and all of it worked well enough when at that point.
I took a little break from that hobby, and it still took me a lot of time to build intuition around Git when I started out professionally.
Obviously now that I have over a decade of professional experience I think it's a no brainer, but I don't think it's natural to understand it as it's suggested to you. I at least kind of had to go along with it and trust that it's good. It felt frustrating at first as many other things are. Git as a thing was FAR more difficult for me to understand in my 20s than PHP and how to do anything with it was for me at teens. Coding is raw logic, Git is about imagining teamwork at scale, which is a whole other thing.
The funny thing is that I feel like I was overall still more productive when I was at my teens building stuff solo with FileZilla and PHP, compared to a corporate environment now with all the processes and 1000s of engineers.
Git is not a natural development at all. Obviously, it is a standard right now.
But I as a hobby coder at my teens I started out with FileZilla and copying over index2.php, index3.php, index_final.php, index_final_2.php and all of it worked well enough when at that point.
I took a little break from that hobby, and it still took me a lot of time to build intuition around Git when I started out professionally.
Obviously now that I have over a decade of professional experience I think it's a no brainer, but I don't think it's natural to understand it as it's suggested to you. I at least kind of had to go along with it and trust that it's good. It felt frustrating at first as many other things are. Git as a thing was FAR more difficult for me to understand in my 20s than PHP and how to do anything with it was for me at teens. Coding is raw logic, Git is about imagining teamwork at scale, which is a whole other thing.
The funny thing is that I feel like I was overall still more productive when I was at my teens building stuff solo with FileZilla and PHP, compared to a corporate environment now with all the processes and 1000s of engineers.
> Why is it astonishingly?
Git is still a baby as these things go. Software version control systems go back to the early 1960s and descended from manual processes used to manage documents in large organizations, blueprints, and such from the early 20th century. Don’t imagine the Manhattan Project functioned by just passing around bundles of paper! There were staffs of people whose job was to manage document control! And I don’t mean “control” in the sense of secrecy, but making sure everybody was on the the same revision at the same time.
And for a field so steeped in building its own tools and automating human effort in development it is indeed astonishing revision control took so long to be accepted.
Git is still a baby as these things go. Software version control systems go back to the early 1960s and descended from manual processes used to manage documents in large organizations, blueprints, and such from the early 20th century. Don’t imagine the Manhattan Project functioned by just passing around bundles of paper! There were staffs of people whose job was to manage document control! And I don’t mean “control” in the sense of secrecy, but making sure everybody was on the the same revision at the same time.
And for a field so steeped in building its own tools and automating human effort in development it is indeed astonishing revision control took so long to be accepted.
> Git is not a natural development at all. Obviously, it is a standard right now.
Most standard tools we use today are not obvious to beginners nor would they be the first thing that beginners reach for.
But senior developers can understand the problems that they claim to address, and why they are important and common problems.
Most standard tools we use today are not obvious to beginners nor would they be the first thing that beginners reach for.
But senior developers can understand the problems that they claim to address, and why they are important and common problems.
You wouldn't believe the amount of crap I take whenever I introduce very basic version control at the various 3 to 6 man shops I find work at these days.
I'm 100% sure that once I left that the devs went back to remote server crash and burn FTP development...they couldn't be bothered with the "hassle" and unneeded headaches of git.
I'm 100% sure that once I left that the devs went back to remote server crash and burn FTP development...they couldn't be bothered with the "hassle" and unneeded headaches of git.
> I'm 100% sure that once I left that the devs went back to remote server crash and burn FTP development...they couldn't be bothered with the "hassle" and unneeded headaches of git.
Have you considered introducing Mercurial or even Subversion?
While Git may be a kind of an industry 'standard', if you're starting from zero, some of its concepts may be a bit mind-bending for folks, and it has often been commented that Hg seems to have a more beginner-friendly interface.
And if branching isn't going to be used (a large strength of git/hg), then Subversion may have an even simpler mental model (svn of course does branching, but the others are more optimized for it).
If folks are doing FTP-push deployment, then moving to 'just' SVN-push (commit) deployment can be an improvement.
Have you considered introducing Mercurial or even Subversion?
While Git may be a kind of an industry 'standard', if you're starting from zero, some of its concepts may be a bit mind-bending for folks, and it has often been commented that Hg seems to have a more beginner-friendly interface.
And if branching isn't going to be used (a large strength of git/hg), then Subversion may have an even simpler mental model (svn of course does branching, but the others are more optimized for it).
If folks are doing FTP-push deployment, then moving to 'just' SVN-push (commit) deployment can be an improvement.
There was a new hire who was told to make a small change in a piece of server software, found the repo, read the docs, made the change, ran it in testing, and pushed to prod. Cue the 'senior engineer' screaming bloody blue murder because he'd been directly monkeypatching the servers for 3 years with no vc and no backups.
That’s… not a senior engineer. Whatever his title may have been.
This could have been written and 2014 and 2004 (hi, it's me). There will always be people who don't use it and others who won't remember a time when they hadn't used it :P
Add 1984 to that list of dates.
We used PVCS on MSDOS
By 1994 I had moved to other companies and a NeXT equivalent to svn with a good GUI - and then ClearCase.
By 1994 I had moved to other companies and a NeXT equivalent to svn with a good GUI - and then ClearCase.
Hello, fellow old person. I was just remembering PVCS (Polytron Version Control System) since it was the first I worked with back in the 80s. Now I see that it's still out there, with the latest release in 2021. Which is insane.
>But senior developers can understand the problems that they claim to address, and why they are important and common problems.
Senior developers have simply grown accustomed to the warts, and accepted their fate - some even celebrating their mastery of the warts as necessary knowledge.
The only pragmatic reason not to change things is the compatibility chaos that will ensue and the resources required - not that problems don't exist.
Senior developers have simply grown accustomed to the warts, and accepted their fate - some even celebrating their mastery of the warts as necessary knowledge.
The only pragmatic reason not to change things is the compatibility chaos that will ensue and the resources required - not that problems don't exist.
> some even celebrating their mastery of the warts as necessary knowledge.
Hi git fans!
Hi git fans!
I don’t know if I agree that it’s unnatural.
It only takes doing index73.php and index_old3.php for a few months and then eventually screwing up (unless you’re perfect) to realize how dumb putting numbers and _old4 at the end of names is. Then at that point, you naturally go look if there’s a better way.
It only takes doing index73.php and index_old3.php for a few months and then eventually screwing up (unless you’re perfect) to realize how dumb putting numbers and _old4 at the end of names is. Then at that point, you naturally go look if there’s a better way.
> Then at that point, you naturally go look if there’s a better way.
That might be true now but it wasn’t always. Plenty of shops didn’t have any version control at all. And the build was whatever came off your local dev box. And “local dev box” wasn’t even a term…
That might be true now but it wasn’t always. Plenty of shops didn’t have any version control at all. And the build was whatever came off your local dev box. And “local dev box” wasn’t even a term…
Yes, so you invent CVS, not git.
You invent VMS versioned files, and RCS. :)
>Git is not a natural development at all.
It's a false dichotomy. Before git there were other version managements systems which would've fit your use case much better than git. Subversion is the easiest perhaps.
It's a false dichotomy. Before git there were other version managements systems which would've fit your use case much better than git. Subversion is the easiest perhaps.
Git is pretty unnatural, but CVS? That is much closer to the "copy a shared file back and forth between people", except with nice things such as "keep track of who is editing what", "know what the latest edition is", and "keep track of the history".
That said, if I was going to start out teaching someone coding today, version control would be timestamped .zip files.
That said, if I was going to start out teaching someone coding today, version control would be timestamped .zip files.
Ouch, please don't. Anything else like bzr or hg or svn will be easier to grasp than git, can work locally too (IIRC for Subversion) and not much harder than zip files — don't get them used to the wrong mental model and bad habits.
>That said, if I was going to start out teaching someone coding today, version control would be timestamped .zip files.
Oh no, we're in a different time zone.
>We'll put the timezone in.
Oh no, it's daylight savings time.
Oh no, we're in a different time zone.
>We'll put the timezone in.
Oh no, it's daylight savings time.
To be fair that's easy, just always use UTC for everyone. Or just the unix timestamp in seconds or millis if you want to be safer.
Entirely besides the point I was trying to make. I wasn’t trying to say that it’s a good idea for a team, but that git is complicated. Too complicated for beginners.
But someone works in Arizona!
Well yeah for hobby use you're right that it is "normal" that it took a long time to get there. But professional use is completely different. That's the part they are referring to I'd say.
> Git is not a natural development at all. Obviously, it is a standard right now.
Git is actually an unnatural development. Its UI is atrocious. And it worked like crap on Windows for forever.
Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Then Github won due to VC cash dumping and foisted Git on all of us. Source control was no longer fine.
Thankfully, there is now Jujutsu(jj) which works on top of the Git storage layer so you can forget about the craptastic UI of Git. Source control is now fine again.
Git is actually an unnatural development. Its UI is atrocious. And it worked like crap on Windows for forever.
Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Then Github won due to VC cash dumping and foisted Git on all of us. Source control was no longer fine.
Thankfully, there is now Jujutsu(jj) which works on top of the Git storage layer so you can forget about the craptastic UI of Git. Source control is now fine again.
GitHub won because it was good.
SourceForge had a messy interface and frankly I hated it. Google Code was like frozen in time, like most Google projects. Microsoft’s thing — I don’t even remember the name — felt half assed and like they were going to abandon it… and they abandoned it. There were also others… none of which I even think were serious enough.
Also Git won because SVN and CVS were centralized VCSes and you needed a server. I can just make Git repos before I even know if a project is serious.
There were other distributed VCSes of course, like Hg, but they either cost $ or wasn’t backed by a major user… like Linux. I admittedly just waited this one out and chose Git because it was more popular.
SourceForge had a messy interface and frankly I hated it. Google Code was like frozen in time, like most Google projects. Microsoft’s thing — I don’t even remember the name — felt half assed and like they were going to abandon it… and they abandoned it. There were also others… none of which I even think were serious enough.
Also Git won because SVN and CVS were centralized VCSes and you needed a server. I can just make Git repos before I even know if a project is serious.
There were other distributed VCSes of course, like Hg, but they either cost $ or wasn’t backed by a major user… like Linux. I admittedly just waited this one out and chose Git because it was more popular.
You either meant Visual SourceSafe or Team Foundation Server for the Microsoft product.
SourceSafe did get abandoned... because your source wasn't actually safe in it.
Its replacement, Team Foundation Server got rebranded as Azure DevOps; but the product's still around, 19 years later.
SourceSafe did get abandoned... because your source wasn't actually safe in it.
Its replacement, Team Foundation Server got rebranded as Azure DevOps; but the product's still around, 19 years later.
Yes, but it's source control shifted to git years ago (I think you can still use TFS, but it's strongly legacy ) and git is mich better than TFS ever was.
True on both counts. (They’re never going to be able to kill TFVC entirely, but it’s disabled for new projects by default, and they’re going to take away the switch to reenable it.)
This discussion was about git, not GitHub. There's a difference.
If git doesn't need a server, then why do you need Github?
GitHub won because it was free.
GitHub won because they built a great product. It was better than all the alternatives and people flocked to it for that reason.
Git itself is mediocre, and some other DVCS like Mercurial could have won out, although HgHub really doesn't have the same ring to it. The halo effect from the Linux kernel was also a factor. But the VC cash dump and Microsoft buyout came later, people used GitHub because it was better than the rest.
Git itself is mediocre, and some other DVCS like Mercurial could have won out, although HgHub really doesn't have the same ring to it. The halo effect from the Linux kernel was also a factor. But the VC cash dump and Microsoft buyout came later, people used GitHub because it was better than the rest.
It's sad that this discussion about git being the standard turned into "Why GitHub won" -- shows that people conflate the two. Who said GitHub had anything to do with git becoming the standard in the first place? (I only even heard of GitHub long after we'd adopted git at a previous workplace.)
git didn't win because of GitHub. Sure, GitHub helped a lot. But actually, it's because its a DVCS, and it does branching and merging way better than anything I've seen so far.
You can thank BitKeeper for all of this, and Andrew Tridgwell for forcing Linus Torvalds into creating git.
You can thank BitKeeper for all of this, and Andrew Tridgwell for forcing Linus Torvalds into creating git.
You don’t think if the company behind GitHub went all-in on mercurial that might have emerged as the winner? There were plenty of companies using either. Git wasn’t the only distributed game in town. I definitely think GitHub had a lot to do with it.
Before there was GitHub there was Sourceforge. Sourceforge supported several different protocols and git still won.
Gits internals can be confusing at first, but once you understand them there's really nothing you can't do. Being distributed by design also helps.
Gits internals can be confusing at first, but once you understand them there's really nothing you can't do. Being distributed by design also helps.
Sourceforge had a flat namespace for projects and required you to manually apply for each project. It was a PITA and that added a huge barrier to entry.
Plus it didn’t have a mechanism to send contributions. I think GitHub “won” because of its web-based PR system.
Plus it didn’t have a mechanism to send contributions. I think GitHub “won” because of its web-based PR system.
Sourceforge was complete garbage though. I hated, hated when projects were hosted on it. It was slow, full of ads, impossible to find what you need to download..
GitHub is to sourceforge what Facebook was to MySpace. MySpace was first but it was buggy as hell.
GitHub is to sourceforge what Facebook was to MySpace. MySpace was first but it was buggy as hell.
You're kind of disproving the point you're trying to make, IMO: Saying "GitHub made git the standard!" is kind of like saying "Facebook invented social media!"
Nope, MySpace did. Or blogs, for that matter. Facebook just co-opted the phenomenon, and has for many become synonymous with it. Just like GitHub has managed to do with git.
Nope, MySpace did. Or blogs, for that matter. Facebook just co-opted the phenomenon, and has for many become synonymous with it. Just like GitHub has managed to do with git.
SF started to be filled with ads only in a second phase. By memory I would say around 2010, and checking Wikipedia it says it changes ownership in 2012. But when it was the de facto "central repository" for Linux softwares codebases, I don't remember it being full of ads.
That comparison is pretty harsh and really underemphasizes how awful sourceforge was. Myspace was mostly fine, death by feature creep. Sourceforge was a flaming pile of garbage that was poorly designed, ad laden, AND silently bundled in ad/spyware to normal downloads.
A more apt comparison would be comparing Facebook to a hypothetical social media site that when you click on a thumbnail of a user's image, you get a fullsize image of something like goatse...which thankfully doesn't exist(yet).
A more apt comparison would be comparing Facebook to a hypothetical social media site that when you click on a thumbnail of a user's image, you get a fullsize image of something like goatse...which thankfully doesn't exist(yet).
You mean the comparison is too mild? https://www.merriam-webster.com/dictionary/harsh
Sourceforge wasn't the only offering at the time.
Lots of companies were doing Git and Mercurial "forges" at the time. Many of them were better than Github.
Everything was evolving nicely (including Git and Github) until Github used VC money to offer everything for free to blow everybody out in order to lock people into their product (for example--export of anything other than code is still hit or miss on Github).
At which point, everything in the source control space completely collapsed.
Lots of companies were doing Git and Mercurial "forges" at the time. Many of them were better than Github.
Everything was evolving nicely (including Git and Github) until Github used VC money to offer everything for free to blow everybody out in order to lock people into their product (for example--export of anything other than code is still hit or miss on Github).
At which point, everything in the source control space completely collapsed.
That's such a weird rewriting of history. I know blaming VC is very fun and all, but Bitbucket, originally centered around mercurial, had just as much resources as GitHub and even more. Tons of early contributors to git and the ecosystem were from google and Microsoft. Microsoft started using and shifting towards gif when GitHub was still a baby, etc.
> That's such a weird rewriting of history. I know blaming VC is very fun and all, but Bitbucket, originally centered around mercurial, had just as much resources as GitHub and even more.
You might want to go review your history before accusing someone else of that.
Github took $100 million from VCs in 2012. It then took another $250 million from VCs in 2015. Who the hell else was even within an order of magnitude of that in the same timeframe? Nobody. (Gitlab took almost the same amounts but did so somewhere between 5-8 years later depending upon how you count).
Bitbucket got bought by Atlassian in 2012. Atlassian was bootstrapped until it took $60 million in VC in 2010 and had revenues (not profits) of about $100 million in that time frame. It had nowhere near the resources to be able to drop the equivalent of $350 million on Bitbucket between 2012-2015.
You might want to go review your history before accusing someone else of that.
Github took $100 million from VCs in 2012. It then took another $250 million from VCs in 2015. Who the hell else was even within an order of magnitude of that in the same timeframe? Nobody. (Gitlab took almost the same amounts but did so somewhere between 5-8 years later depending upon how you count).
Bitbucket got bought by Atlassian in 2012. Atlassian was bootstrapped until it took $60 million in VC in 2010 and had revenues (not profits) of about $100 million in that time frame. It had nowhere near the resources to be able to drop the equivalent of $350 million on Bitbucket between 2012-2015.
Minor nit.
> Bitbucket got bought by Atlassian in 2012.
29th September 2010[0] (just over a month after Atlassian raised the $60M.) ~18 months before Github took any VC money. If the VC money was key to Github's success, why did Atlassian/Bitbucket's 18 month head start not get them anywhere?
[0] https://web.archive.org/web/20160303204710/http://www.itwire...
> Bitbucket got bought by Atlassian in 2012.
29th September 2010[0] (just over a month after Atlassian raised the $60M.) ~18 months before Github took any VC money. If the VC money was key to Github's success, why did Atlassian/Bitbucket's 18 month head start not get them anywhere?
[0] https://web.archive.org/web/20160303204710/http://www.itwire...
By 2012 the writing was already on the wall. This was already well into the first PaaS era with Heroku, Engine Yard, etc. Github was bootstrapped, and using git was a grassroots movement. It was just better than what most people had been using. I never looked back after first switching from SVN to git in 2009.
VCs didn't make git successful, developers did.
VCs didn't make git successful, developers did.
Sure, but 2010 to 2012 might as well be two different eras in the context of VCS adoption. Things changed very quickly.
In any case,that doesn't really matter considering that git had big players adopting before GitHub got any sizeable investment. And I'm not just talking about Linux. Rails migrated towards it when GitHub was still in beta.
In any case,that doesn't really matter considering that git had big players adopting before GitHub got any sizeable investment. And I'm not just talking about Linux. Rails migrated towards it when GitHub was still in beta.
Sourceforge isn't GitHub. What's the relevance?
If you think SF+CVS is equivalent to GitHub+Git then you never used SF+CVS. Git won because of GitHub, specifically, not because generically it could be hosted on the Internet.
If you think SF+CVS is equivalent to GitHub+Git then you never used SF+CVS. Git won because of GitHub, specifically, not because generically it could be hosted on the Internet.
No, GitHub won because of git, lots of VC (later MS) money, and brilliant marketing that makes people think "Git won because of GitHub".
To be fair, by the time git came around, sourceforge had been enshitifying itself for 10 years or so. Git would be popular without github, github just makes things easier.
> Git would be popular without github,
Yep.
> github just makes things easier.
Which is the transition between steps 1 and 2 of Embrace, Extend, Extinguish. Though the last step is nowadays perhaps unnecessary, depending on how you look at it: Having co-opted FOSS software like git, what you need to extinguish is not that itself, just any other way to use it than your own. The main example of step 2 was of course "Pull Requests".
Yep.
> github just makes things easier.
Which is the transition between steps 1 and 2 of Embrace, Extend, Extinguish. Though the last step is nowadays perhaps unnecessary, depending on how you look at it: Having co-opted FOSS software like git, what you need to extinguish is not that itself, just any other way to use it than your own. The main example of step 2 was of course "Pull Requests".
yeah, git is just the right layer of indirection to support everyone's weird take on how the workflow should go, so you can just use it however you want.
> I definitely think GitHub had a lot to do with it.
And I definitely think you're wrong. GitHub has managed to get itself conflated with git, and has benefitting hugely from this confusion.
And I definitely think you're wrong. GitHub has managed to get itself conflated with git, and has benefitting hugely from this confusion.
Yes, git has solved a big problem with version control: it has made transactional atomic changesets (commits in the git parlance) mainstream. A git repository is a tree of atomic changesets that group changes into meaningful chunks as opposed to a versioned tree of files where changes to each file are harder to trace back to the intent, i.e. whether or not they are related.
Atomic commits can also easily be moved around (since the repository is a tree of commits with the commits being leaves), and they also make merging simpler in many scenarios.
Larry McVoy was famously wont to engage in trolling upon the Linux Kernel Mailing List, whereupon he did boast of his BitKeeper, which possessed atomic changesets. Concurrently, he did deride Subversion for its lack of the same. Thus, a great drama did ensue, one which ultimately bestowed upon us the creation of git.
git has also succeeded as a DVCS where others have faltered, for various reasons. For there have been monotone, darcs and other such systems; yet, it is chiefly git that has endured.
Atomic commits can also easily be moved around (since the repository is a tree of commits with the commits being leaves), and they also make merging simpler in many scenarios.
Larry McVoy was famously wont to engage in trolling upon the Linux Kernel Mailing List, whereupon he did boast of his BitKeeper, which possessed atomic changesets. Concurrently, he did deride Subversion for its lack of the same. Thus, a great drama did ensue, one which ultimately bestowed upon us the creation of git.
git has also succeeded as a DVCS where others have faltered, for various reasons. For there have been monotone, darcs and other such systems; yet, it is chiefly git that has endured.
> Larry McVoy was famously wont to engage in trolling upon the Linux Kernel Mailing List, whereupon he did boast of his BitKeeper, which possessed atomic changesets. Concurrently, he did deride Subversion for its lack of the same
Subversion definitely has atomic commits. That was it's major advance over CVS.
The major difference between svn and git/hg is svn assumes a centralised repository, whereas git/hg are designed to work as distributed repositories that exchange change sets.
Turns out that you can build sophisticated work flows on top of features they added to support that distributed model - things like PR's. These things are useful even if you have a single centralised monorepositiry of the style svn insists on.
But atomic commits aren't part distributed feature set. You need them in a monorepositiry. After you've experienced your CVS repository being turned into a mess by two people doing a commit at the same time that becomes obvious.
Subversion definitely has atomic commits. That was it's major advance over CVS.
The major difference between svn and git/hg is svn assumes a centralised repository, whereas git/hg are designed to work as distributed repositories that exchange change sets.
Turns out that you can build sophisticated work flows on top of features they added to support that distributed model - things like PR's. These things are useful even if you have a single centralised monorepositiry of the style svn insists on.
But atomic commits aren't part distributed feature set. You need them in a monorepositiry. After you've experienced your CVS repository being turned into a mess by two people doing a commit at the same time that becomes obvious.
> it has made transactional atomic changesets (commits in the git parlance) mainstream
Nope. Subversion made them mainstream:
"An svn commit operation publishes changes to any number of files and directories as a single atomic transaction. In your working copy, you can change files' contents; create, delete, rename, and copy files and directories; and then commit a complete set of changes as an atomic transaction.
By atomic transaction, we mean simply this: either all of the changes happen in the repository, or none of them happens. Subversion tries to retain this atomicity in the face of program crashes, system crashes, network problems, and other users' actions." [0]
[0] https://svnbook.red-bean.com/en/1.5/svn.basic.in-action.html
Nope. Subversion made them mainstream:
"An svn commit operation publishes changes to any number of files and directories as a single atomic transaction. In your working copy, you can change files' contents; create, delete, rename, and copy files and directories; and then commit a complete set of changes as an atomic transaction.
By atomic transaction, we mean simply this: either all of the changes happen in the repository, or none of them happens. Subversion tries to retain this atomicity in the face of program crashes, system crashes, network problems, and other users' actions." [0]
[0] https://svnbook.red-bean.com/en/1.5/svn.basic.in-action.html
git isn't foisted on anyone, although I will acknowledge that if you're working anywhere remotely web-y, you'll be required to use it.
But the vast majority of people I personally see who complain about git simply don't know how to use it, and apparently don't care to learn. They say it's hard, and then they joke about never learning more than three git commands. Can't we do better? Is that where the bar is?
But the vast majority of people I personally see who complain about git simply don't know how to use it, and apparently don't care to learn. They say it's hard, and then they joke about never learning more than three git commands. Can't we do better? Is that where the bar is?
It's not that the basics of git are hard. It's that there are 30 footguns you're forced to deal with every time you want to do anything, just because someone somewhere once had to use one of them -- probably caused by another one of them.
Git lovers mostly use it as a file copy service.
Git lovers mostly use it as a file copy service.
> Git[‘s] UI is atrocious.
Well, that’s in the eye of the beholder. Yes, I hate the ‘program command’ syntax (“git add”, “git commit” etc) but I just call git-add etc and for me those commands are pretty clear.
But I understand how git works. I imagine most people treat it as a black box and then its behavior probably is rather obscure. I don’t think it was intended for that use case.
Well, that’s in the eye of the beholder. Yes, I hate the ‘program command’ syntax (“git add”, “git commit” etc) but I just call git-add etc and for me those commands are pretty clear.
But I understand how git works. I imagine most people treat it as a black box and then its behavior probably is rather obscure. I don’t think it was intended for that use case.
> Well, that’s in the eye of the beholder.
I acknowledge that Git is really good on a technical level, but don't like the CLI experience regardless.
Instead, I much prefer to use tools like GitKraken (paid), SourceTree (free, limited platform support), Git Cola (free, lightweight, a bit basic) and know that most of my colleagues just use the Git integration in their IDE of choice.
Working that way, it's a decidedly pleasant experience, not unlike using visual merge tools like Meld, where you can also stage individual lines/code blocks and operate with branches and tags in a very visual and obvious manner.
That said, sometimes the support for Git LFS and things like submodules is all over the place and I've had cases where not adding node_modules or something like that to .gitignore has made the UI unresponsive with how much stuff initially shows up in the working copy, so sometimes you still have to drop down to the CLI and do a fix herer and there.
I acknowledge that Git is really good on a technical level, but don't like the CLI experience regardless.
Instead, I much prefer to use tools like GitKraken (paid), SourceTree (free, limited platform support), Git Cola (free, lightweight, a bit basic) and know that most of my colleagues just use the Git integration in their IDE of choice.
Working that way, it's a decidedly pleasant experience, not unlike using visual merge tools like Meld, where you can also stage individual lines/code blocks and operate with branches and tags in a very visual and obvious manner.
That said, sometimes the support for Git LFS and things like submodules is all over the place and I've had cases where not adding node_modules or something like that to .gitignore has made the UI unresponsive with how much stuff initially shows up in the working copy, so sometimes you still have to drop down to the CLI and do a fix herer and there.
git has plenty I like and lots of things I would rather not do, but I find most GUIs too obfuscating. I do patch adds, I prefer resolving conflicts in the raw file, and as many plugins as I’ve tried for vim to make git “pretty”, they just don’t hit for me. I used Kraken for a while but ultimately it just felt like unnecessary fluff - knowing the commands I’m using makes me feel like I know what’s going to happen, and GUIs make me feel like I’m not sure. I’m happy for my coworkers that use them, and I’m also usually the first person to get a ping when something beyond what makes sense in the GUI comes up.
I'm consistently baffled by the people who don't understand how git works who then complain that it's hard. Yes, of course something you don't know is hard. Version control is a hard problem; I'm not sure why so many expect it to be unicorns and rainbows.
>> Git[‘s] UI is atrocious. […] Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial.
> Well, that’s in the eye of the beholder. […]
I would say it is not.
Usability testing is a thing, and products can get better and worse scores when trying to do the same task / reach the same result. I'm not aware of any published studies on the topic, but I would not be surprised if Git got lower scores than Hg on UI/UX, even though they can basically do the same thing (distributed development).
Given the GP trained people on multiple version control systems, including another distributed one (hg), and people mostly had a problem with Git, I would say Git is the problem.
> Well, that’s in the eye of the beholder. […]
I would say it is not.
Usability testing is a thing, and products can get better and worse scores when trying to do the same task / reach the same result. I'm not aware of any published studies on the topic, but I would not be surprised if Git got lower scores than Hg on UI/UX, even though they can basically do the same thing (distributed development).
Given the GP trained people on multiple version control systems, including another distributed one (hg), and people mostly had a problem with Git, I would say Git is the problem.
I am unable to do anything with Photoshop beyond starting it and exiting. I have no idea how or even what it does beyond somehow modifying photos. That’s not photoshop’s problem, it’s that I don’t know what I’m doing. So I just don’t use it.
I only hear git complaints from people who don’t have an internal model that reflects what git is doing (I’ll agree that it uses a couple — but only a couple — of terms in a way that might violate some people’s intuitive model). But if you know what you want to accomplish the command line is pretty unremarkable.
But when you don’t know how the tool works you easily get snarled. And when you blindly resort to various “cheat sheet” sites it’s a crap shoot whether you “fix” your problem (how would you know?) or make things worse. That’s like me with photoshop.
I used to see this with CVS too
I only hear git complaints from people who don’t have an internal model that reflects what git is doing (I’ll agree that it uses a couple — but only a couple — of terms in a way that might violate some people’s intuitive model). But if you know what you want to accomplish the command line is pretty unremarkable.
But when you don’t know how the tool works you easily get snarled. And when you blindly resort to various “cheat sheet” sites it’s a crap shoot whether you “fix” your problem (how would you know?) or make things worse. That’s like me with photoshop.
I used to see this with CVS too
> I am unable to do anything with Photoshop beyond starting it and exiting. I have no idea how or even what it does beyond somehow modifying photos. That’s not photoshop’s problem, it’s that I don’t know what I’m doing. So I just don’t use it.
As stated up-thread:
> Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Seems that things only got hard(er) teaching people with Git.
> I only hear git complaints from people who don’t have an internal model that reflects what git is doing
Yeah, every time someone says that:
> git gets easier once you get the basic idea that branches are homeomorphic endofunctors mapping submanifolds of a Hilbert space.
* Isaac Wolkerstorfer, https://twitter.com/agnoster/status/44636629423497217
In the past I have used RCS, CVS, and Subversion. I never had to learn any kind of internal model for any of them.
As stated up-thread:
> Over time, I taught non-computer people who used Windows all of CVS, Subversion, and Mercurial. They got each one and why things were better. The first time they had to recover something, they really got it. Source control got out of their way and was generally fine.
Seems that things only got hard(er) teaching people with Git.
> I only hear git complaints from people who don’t have an internal model that reflects what git is doing
Yeah, every time someone says that:
> git gets easier once you get the basic idea that branches are homeomorphic endofunctors mapping submanifolds of a Hilbert space.
* Isaac Wolkerstorfer, https://twitter.com/agnoster/status/44636629423497217
In the past I have used RCS, CVS, and Subversion. I never had to learn any kind of internal model for any of them.
Linux was managed with tarballs and patch emails until a little over 20 years ago when they started using bitkeeper, then Linus wrote git.
> Pixley agreed to all of that and then ignored all of it completely.
Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.
Hahaha that's brilliant, and an important lesson for junior developers. Sometimes this is the best way forward. High risk of course. But the reward can be great.
We had the right corporate culture to make this attitude successful. By the time we were a couple of hundred people, though, and had hired some MBAs, that culture was mostly gone.
It requires a level of trust and a level of performance, that I now believe doesn’t scale. Note that we had only a single junior developer, an intern, who wrote our bug tracking system but didn’t otherwise interact with the code base. The rest of the team consisted entirely of experienced senior developers, including management. I think at up to about 25 employees, the shortest working experience on the team (apart from the intern) might have been 10 years of shipping code.
But I really respect Rich for taking this approach which is why I referred to him by name.
It requires a level of trust and a level of performance, that I now believe doesn’t scale. Note that we had only a single junior developer, an intern, who wrote our bug tracking system but didn’t otherwise interact with the code base. The rest of the team consisted entirely of experienced senior developers, including management. I think at up to about 25 employees, the shortest working experience on the team (apart from the intern) might have been 10 years of shipping code.
But I really respect Rich for taking this approach which is why I referred to him by name.
I don't think you'd even necessarily need to ignore.
Roll it out in phases. You aren't going to have to deliver the final finished solution all at once.
Some elements are inevitably going to end up being de-prioritized, and pushed further into the future. Features that do end up having a lot of demand could remain a priority.
I don't think this is even a case of "ask for forgiveness, not permission" (assuming you do intend to actually work on w/e particular demands if they end up actually continuing to demand it), but a natural product of triage.
Some elements are inevitably going to end up being de-prioritized, and pushed further into the future. Features that do end up having a lot of demand could remain a priority.
I don't think this is even a case of "ask for forgiveness, not permission" (assuming you do intend to actually work on w/e particular demands if they end up actually continuing to demand it), but a natural product of triage.
> Some elements are inevitably going to end up being de-prioritized, and pushed further into the future. Features that do end up having a lot of demand could remain a priority.
Why, that sounds positively... Agile. In the genuine original sense.
Why, that sounds positively... Agile. In the genuine original sense.
I remember working somewhere where revision control actually created a mess.
I believe the problem was a combination of xenix (with limited length filenames) and sccs (which used 's.<filename>' to store each file).
Anyway, long filenames like longestpossiblename.c got checked into revision control as s.longestpossiblename where the s. prefix truncated the .c at the end
Later builds would fail since the checked out file longestpossiblename was no longer a c file and wouldn't compile.
I believe the problem was a combination of xenix (with limited length filenames) and sccs (which used 's.<filename>' to store each file).
Anyway, long filenames like longestpossiblename.c got checked into revision control as s.longestpossiblename where the s. prefix truncated the .c at the end
Later builds would fail since the checked out file longestpossiblename was no longer a c file and wouldn't compile.
My first job out of college was on a university administrative system where all the source files were managed by RCS. I came on in '97, but the system had been operation in at least that state to 1995. Obnoxiously, the vendor layered some secret sauce on top of it in Makefiles for the actual checkins and outs, so you did something like `make co F=foo.c` which totally messed up filename completion. I had it aliased in tcsh within minutes to preserve my own sanity.
When my consultant contract finally came to an end in 2003 they were still using it.
When my consultant contract finally came to an end in 2003 they were still using it.
[deleted]
> none of the GNU codebase was hosted in any sort of revision control.
That isn't true, RCS was already in heavy use by 1991 in the GNU project.
That isn't true, RCS was already in heavy use by 1991 in the GNU project.
I was one of those once. Tried to get CVS in a project.
Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow.
I left.
Then some other dev committed 9MB of tabs 0x09 at the end of a file. Then the site was "slow" (cause the homepage was 10MB). And the blame went to...CVS somehow.
I left.
> And the blame went to...CVS somehow.
CVS was notorious for doing "text/binary" conversions (CR/LF line endings to CR and vice versa), sometimes inappropriately. More than once, it resulted in files where every other line was empty. I can very well see this happening several times, resulting in exponential growth of whitespaces.
CVS was notorious for doing "text/binary" conversions (CR/LF line endings to CR and vice versa), sometimes inappropriately. More than once, it resulted in files where every other line was empty. I can very well see this happening several times, resulting in exponential growth of whitespaces.
Yep, that bit me/my teams more than once. But this case we had the commit from CVS showing it was just a fat-finger.
But the seniors on the team pointed to CVS. Not the individual, not the "test" process, not the deploy checks.
Nobody noticed it slow cause everything ran on LAN. Only noticed the issue on 56k
But the seniors on the team pointed to CVS. Not the individual, not the "test" process, not the deploy checks.
Nobody noticed it slow cause everything ran on LAN. Only noticed the issue on 56k
That's a really fat finger
Assuming a key repeat rate of 15Hz (a number I admittedly just pulled out of thin air), they would have had to lean on the Tab key for almost exactly 1 week.
The guy ate, slept, possibly got married while his finger was still accidentally holding down that key.
The guy ate, slept, possibly got married while his finger was still accidentally holding down that key.
Then they probably fired the guy who deleted 2MM lines of tabs for not meeting LoC metrics.
Many years later there was a scene in Silicon Valley where their VC set a tequila bottle on the delete key and caused havoc.
That's when I figured out what happened in 1998.
That's when I figured out what happened in 1998.
I have had a website taken down by (pretty sure) a person setting something on their F5 key.
Let's hear it!
The symptom was the website wasn’t loading, just spinning forever.
Looking at the http logs, around 5pm an ip address started spamming a whole bunch of requests to the same url. The same ip was doing pretty normal stuff on the website about an hour earlier.
My theory is holding down F5 key would cause the page to reload about 30 times a second. The website was not able to handle that many requests per second and it effectively became a denial of service attack.
This was around 2007-2010 and I think by now browsers have stopped repeating a reload if the F5 key is held down.
Looking at the http logs, around 5pm an ip address started spamming a whole bunch of requests to the same url. The same ip was doing pretty normal stuff on the website about an hour earlier.
My theory is holding down F5 key would cause the page to reload about 30 times a second. The website was not able to handle that many requests per second and it effectively became a denial of service attack.
This was around 2007-2010 and I think by now browsers have stopped repeating a reload if the F5 key is held down.
Nice.
I worked briefly on a site that had this rankings page for users, and it was done by going player by player, and pulling the players table each loop to compare the current player to all the others. For things like "results against women"
Anyway you could DDoS the site by requesting that page. You could actually watch the page fill in as it computed, I want to say it was about a ten second load time.
I worked briefly on a site that had this rankings page for users, and it was done by going player by player, and pulling the players table each loop to compare the current player to all the others. For things like "results against women"
Anyway you could DDoS the site by requesting that page. You could actually watch the page fill in as it computed, I want to say it was about a ten second load time.
Yeah, I think it's surprising how websites don't get DoS'd more often.
DoS has become more of an issue now that all of the GPTBots are generating a lot more traffic. https://news.ycombinator.com/item?id=40865627
DoS has become more of an issue now that all of the GPTBots are generating a lot more traffic. https://news.ycombinator.com/item?id=40865627
I had a visceral reaction to this comment! I once joined a company doing ETL with Apache camel and a half dozen underpowered pet machines. Ingesting their entire dataset and running a suite of NLP models took 3-6 months (estimated; it was so slow nobody ever reprocessed the data to fix bugs or release improvements). I drew up a simple architecture using Kafka, hbase, and MapReduce to implement a lambda architecture. The CTO very patronizingly told me that just because something is shiny and new it doesn't mean we need to implement it. This was 2017 :laugh-cry:.
But maybe this isn't really what they felt that they needed at the time? I don't mean to defend bad practices, but your comment makes it sound like nobody had tasked you with re-architecting the business, and you took it upon yourself to show them how it should be done (in your opinion), without having earned the necessary trust. This might have also come across as patronizing, or at least antagonistic, and in any case unbeneficial. Not saying that's the case as I obviously wasn't there, just something to think about.
Fair comment. And I'm usually suspicious of young engineers wanting to implement the new hotness and I'm also a fan of "if it ain't broken don't fix it". In this case, though, the system was in very rough shape. Our customers were complaining about data problems which we had no way to fix (short of manually editing the prod db, which was the SOP). I definitely took it upon myself to do something that nobody had asked for, but it was because the people in charge were entirely asleep at the wheel! They did not last long in their positions.
It's a shame the CTO was patronizing. I've generally found this to be the attitude of many IT workers in similar positions. I would recommend trying to allocate (work) time to prototype and get important back of the envelope metrics that they think are valuable along with those that you think are valuable.
At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)
At least that's what I would ask of anyone who was trying to improve a system (And not just the developers circumstance which I think is perhaps what they CTO is cautious of)
I was in this position before but would point out that there is a tactical approach when you know that others will not follow. I set up a cron job (on Windows, not really cron) to check scan the network location for updated source files. The git repo was on my drive and on the corporate GitHub account, safe from those who should have been using it. Whenever files changed it would just auto commit on the main branch with the username included in the message. I could do whatever I wanted on my own branches, keep track of what others were doing, and essentially wield git. You don’t have to try to be a hero inflicting proper source control upon your teams (their perspective) to still occasionally appear like a wizard to save them from inevitable, oft-occurring peril.
I never had to deal with “we don’t use source control”, luckily.
One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess.
I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow.
In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way.
But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in.
But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server.
Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere?
I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business.
I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.
One company I joined was cleaning up the last vestiges of “customize it for every customer by letting people edit their copy on server,” which predictably turned into a mess. They were all very small customizations to styles or images but enough to make upgrades a total mess.
I did work at a company where despite having full source control they didn’t actually know of they could ever deploy the server component again. Edits got made to the live server but then made again in source control, or vice versa. There was one more senior person who couldn’t be talked out of their old workflow.
In theory everything matched.Eventually they even checked and got it all under control where they were positive it was the same and kept it that way.
But it had only ever been deployed from scratch… once. And for like 15 years it lived there and kept getting upgraded. It would all be moved when new hardware was brought in.
But it wasn’t installed from scratch. We truly did not know if we were capable of doing that. It is possible if that server was destroyed and we couldn’t restore from a back up it would take us an unknown amount of time. Even though in theory deploying should be as simple copying the files and starting the web server.
Were there odd configurations that had been made eight years ago that kept it running? Some strange permission changed somewhere?
I wasn’t on that team. But it always made me nervous. That was absolutely a core application of the business.
I really like small shops sometimes. You get a lot of freedom and get to have your hands in a lot of disciplines. You learn a lot of things, including things that’s should never be duplicated.
That's not an exception, I think? Most online service businesses that I've worked with wouldn't be able to run their operations from a "cold start". That takes a lot of effort to engineer and in practice it doesn't happen enough, so that's a risk that most are willing to run with.
Really? Everywhere else I’ve worked has been able to.
I’m not saying all the data in the database. Pretend your DB is fine or you have a full up to date backup. I know losing that KILLS businesses. But in my example just the web server immolated and the backup tapes sitting next to it melted.
Can you set up a new web server?
As a Java developer it’s usually install OS, add Apache + Tomcat, set a few basic config parameters, copy WARs.
PHP or Python is even easier.
They didn’t know they could successfully do that.
I’m not saying all the data in the database. Pretend your DB is fine or you have a full up to date backup. I know losing that KILLS businesses. But in my example just the web server immolated and the backup tapes sitting next to it melted.
Can you set up a new web server?
As a Java developer it’s usually install OS, add Apache + Tomcat, set a few basic config parameters, copy WARs.
PHP or Python is even easier.
They didn’t know they could successfully do that.
With a single server sure, but anything complex than that, most companies don't have a well-rehearsed cold start procedure.
Been there. There was this old fashioned developer in one of the companies I worked for a decade ago who never understood nor embraced version control (we were talking of SVN at the time, not even git). Luckily that wasn't the case for all the others developers in the company. But when it came to the projects he owned, I witnessed several scenes along the lines of "hey, customer X has an issue with your component Y, what version do they have?"
He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it.
I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.
He had a spreadsheet where he kept track of the versions used by every customer. Once identified the version, he would open (no joke) a drawer in his desk and pick the right USB stick with that version on it.
I've always wondered whether this overhead was a worth price to pay for not wanting to learn a couple of SVN commands.
My first job out of college, way back in the 90's, I had to "train" an entire department on how to use CVS. This wasn't a startup. This was a billion dollar, established company with 1000's of employees.
He was in essence doing manual version and release control - just.. like you would in 1990 with floppy disks.
Perhaps he was more concerned with job control, and saw centralized code as a detriment to his job security
I have a confession to make. I was working in a company where the main development center was located in a different country. The main development center wanted to centralize all code on a single SVN server and move the build process to Jenkins
I said we already had source control (CVS) and a a working build process (a script on a server that pulled from CVS, built a jar file and deployed it to X servers). We were too "busy" at the moment but would look into it in the future. This never happened.
The real reason was that I was concerned that the central development center would take over the project, as they had tried to do so in the past. Looking back, I should probably have let them take over as there was more than enough work for everyone.
I said we already had source control (CVS) and a a working build process (a script on a server that pulled from CVS, built a jar file and deployed it to X servers). We were too "busy" at the moment but would look into it in the future. This never happened.
The real reason was that I was concerned that the central development center would take over the project, as they had tried to do so in the past. Looking back, I should probably have let them take over as there was more than enough work for everyone.
At this level of dysfunction, installing git won't do anything. You need a holistic change in thinking which starts with convincing people there's a problem.
I’ve yet to see people actually change their mind at this level. Firing is often the only way.
Yeah, this level of dysfunction takes years to cultivate.
You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way
You need the “Dead Sea effect” to be in effect long enough that not only have the good people left, but for them to have been gone long enough that people rising into management have never even worked with somebody competent so they don’t know there’s a better way
I am occasionally on firefighting projects for customers.
It's super hard to get out of this mess.
The code base has gone to shit, customers are often passed enough that they want functioning code NOW, the reaming developers are bottom of the barrel, management is either cause of the problems or has now clue.
Getting out of that is difficult especially if management doesn't understand it has to change first.
Years, and a special kind of myopia in leadership to not see the seeds sprouting. It doesn’t happen quickly, but once established is very, very expensive to fix.
Oh man that description immediately made me think of a government IT job a friend used to work in
I've been that person a few times.
1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS.
2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master.
The list goes on!
1. The only developer on the team with Github and put forward the idea of the company not hosting their own source code with TFS.
2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master.
The list goes on!
Given the various accidental leaks dues to people not realising deletion still has a history with git when publishing (not to mention git has no equivalent to mercurial's "censor" operation), or people confusing public and private repos on their github accounts, or even the story just last week here on HN with counterintuitive behaviour of forking in terms of privacy (https://news.ycombinator.com/item?id=41060102), I can totally understand a company being opposed to being on github.
Perhaps you should have proposed anything-but-TFS (git/mercurial/even svn - no need for distributed in a small company... https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs...) but internal.
Might be a relatively easy sell if you just setup a local migration of their data to repository with a decent web gui for viewing the commits and associating with tickets, given my experiences with TFS in terms of slowness, and crashes/data corruption.
Perhaps you should have proposed anything-but-TFS (git/mercurial/even svn - no need for distributed in a small company... https://www.bitquabit.com/post/unorthodocs-abandon-your-dvcs...) but internal.
Might be a relatively easy sell if you just setup a local migration of their data to repository with a decent web gui for viewing the commits and associating with tickets, given my experiences with TFS in terms of slowness, and crashes/data corruption.
> not to mention git has no equivalent to mercurial's "censor" operation
Haven't followed that for a while, but it used to be the case that it was Mercurial who was principled/fanatic about never rewriting history, ever, while git happily let you mess up with hard deletes.
Haven't followed that for a while, but it used to be the case that it was Mercurial who was principled/fanatic about never rewriting history, ever, while git happily let you mess up with hard deletes.
Which is still the case. They use the hidden phase to avoid that. You can use hg absorb (also awesome) locally of course to simplify matters.
What censor is for is the case where something absolutely has to be removed from the repo, for legal or security. It allows you to do it in a clean standard way, even replacing it with, say, a key signed statement, without forcing regeneration of the whole repository or altering the tree in any way.
Just remove the contents while keeping the hash.
https://wiki.mercurial-scm.org/CensorPlan
What censor is for is the case where something absolutely has to be removed from the repo, for legal or security. It allows you to do it in a clean standard way, even replacing it with, say, a key signed statement, without forcing regeneration of the whole repository or altering the tree in any way.
Just remove the contents while keeping the hash.
https://wiki.mercurial-scm.org/CensorPlan
... and gotta say. mercurial will let you do hard deletes. They just discourage it and try to offer tooling that allows you to do what you want without destroying history / changing hashes / generally complicating life for those using the repo.
They also do have the tools to completely rewrite history if you really really want to.
So, "principled fanatic" is not quite accurate I feel. They just have a lot more options than git (which also applies to their commandline tooling I feel, although of course there's a lot of 3rd party stuff out there for git to mimic much of what mercurial has out of the box these days).
They also do have the tools to completely rewrite history if you really really want to.
So, "principled fanatic" is not quite accurate I feel. They just have a lot more options than git (which also applies to their commandline tooling I feel, although of course there's a lot of 3rd party stuff out there for git to mimic much of what mercurial has out of the box these days).
Here’s a few of my horror stories where I was a consultant at various companies:
1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins.
2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month)
3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too
4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed
Any many more!
This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now,
Good times!
1. Previous consultants left no documentation or anything, and a running Hadoop cluster handling (live!) 300 credit card transactions a second. Management hired 8 junior sysadmins - who were all windows sysadmins, had never used Linux before, and were expected to take over running this Linux cluster immediately. They all looked at me white as ghosts when I brought up SSH prompt, that’s the point where I learned they were all windows sysadmins.
2. Another company: all Java and MySQL developers who were trying to use Spark on Hadoop, refusing to learn anything new they ended up coding a Java app that sat on a single node, with a mysql database on the same node, that “shelled out” to a single trivial hello-world type function running in spark, then did the rest of the computation in Java on the single node, management celebrated a huge success of their team now using “modern cluster computing” even though the 20 node cluster did basically nothing and was 99.99% idle. (And burning huge $ a month)
3. Another company: setup a cluster then was so desperate to use the cluster for everything installed monitoring on the cluster, so when the cluster went down, monitoring and all observability went down too
4. A Cassandra cluster run by junior sys-admins and queried by junior data scientists had this funny arms race where the data scientists did what was effectively “select * from *” for every query and the sysadmins noticing the cluster was slow, kept adding more nodes, rather than talk to each other things just oscillated back and forwards with costs spiralling out of control as more and more nodes were deployed
Any many more!
This might sound like I’m ragging on juniors a bit but that’s definitely not the case - most of these problems were caused by bad management being cheap and throwing these poor kids into the deep end with no guidance. I did my best to upskill them rapidly and I’m still friends with many of them today, even though it’s nearly 10 years later now,
Good times!
"Senior" holds no weight with me. I've had plenty dumb founding conversations with "seniors".
My favorite was at the company that was self hosting their code. The senior team lead wanted me to help him find a memory leak that plagued the product for months. Customers were told to restart the application every few weeks (this was a C++ application).
I sat down with the senior and looked at the code. I spotted the error.
I was like, "You know when you do new[] you need to use delete[]?" as all of his deletions were without [].
The look on his face was the best.
My favorite was at the company that was self hosting their code. The senior team lead wanted me to help him find a memory leak that plagued the product for months. Customers were told to restart the application every few weeks (this was a C++ application).
I sat down with the senior and looked at the code. I spotted the error.
I was like, "You know when you do new[] you need to use delete[]?" as all of his deletions were without [].
The look on his face was the best.
> I was like, "You know when you do new[] you need to use delete[]?" as all of his deletions were without [].
This seems like a pretty major lack of a specific piece of knowledge on the senior developers part, yes, but it seems like a much more unforgivable miss on the part of the code reviewers. Was the team stuck in a rut where only a single person (with coincidentally the same blind spot) was reviewing his code, or did multiple reviewers somehow miss this?
This seems like a pretty major lack of a specific piece of knowledge on the senior developers part, yes, but it seems like a much more unforgivable miss on the part of the code reviewers. Was the team stuck in a rut where only a single person (with coincidentally the same blind spot) was reviewing his code, or did multiple reviewers somehow miss this?
Code review? There was no code review. We did get our own offices though!
No rank means much. Not even degrees like a PhD. In fact I swear there’s negative correlation.
I've worked with a consultancy that prouded itself by exclusively hiring top engineers from top universities and I swear, I can put my hand in boiling hot oil and tell you, they were some of the worst coders I've ever seen, I have no doubts I've met way more brilliant people coming from boot camps.
The fact that people study for exams has absolutely no correlation with how much they will remember or care for what they studied, none.
The fact that people study for exams has absolutely no correlation with how much they will remember or care for what they studied, none.
Yeah it reminds me of that old saying “Some developers have 20 years experience and some have 1 year experience 20 times” haha
Titles really mean very little. The company I work at recently hired a DevOps specialist to help configure Docker services and assist with development. He was a decent developer but had no idea how to securely configure server side services. Still stuck with his mess two years later :)
> 2. The only developer using branches with git when the co-founder asked (demanded) everyone to only use master.
I’d argue everything else becomes expensive quickly.
…but it is also imperative to remember any modification on checked out code is a branch, technically, regardless of the version control system used. This becomes important if your testing is expensive.
I’d argue everything else becomes expensive quickly.
…but it is also imperative to remember any modification on checked out code is a branch, technically, regardless of the version control system used. This becomes important if your testing is expensive.
There are advantages to trunk based development, not exactly only using master, but close enough that you could probably spin it to comply.
I'm sure I'm not alone in actually having lived such an experience.
I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.
I joined a dynamic DNS provider once that had been around since 1999. Their tech, sadly, had not progressed much beyond that point. Showing the higher ups version control was like showing cavemen fire. Of course once the higher ups arranged to have training sessions led by the new hire for the entire dev team the VP of Engineering couldn't handle it and had me fired. Fun times.
I started in 2008. This is what I did eventually. Over the years I introduced the small company to Linux, git, defensive programming, linting, continuous integration, Scrum..., but only for the new projects and stayed 13 years there.
That old project though was never fixed though, probably still running that way now.
That old project though was never fixed though, probably still running that way now.
Same story here. Introduced static Python to our team’s entire codebase. Best place I’ve ever worked.
I think of Scrum as a crime against humanity
[deleted]
Anecdote seems long before git creation, so Visual SourceSafe maybe. Which did not work well over a WAN. Needed other tools to replicate and synchronize VSS.
I was working at a game development company in the mid 90s that used Visual Source Safe and to avoid file corruption due to concurrent commits, we had a shiny silver baseball cap which you had to find and physically possess to commit.
After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.
After we had off-site devs, the physical cap wouldn't work. So the project had a "silvercap.txt" file and you had to exclusively check out . And of course people forgot to release that and work ground to a halt.
You can remove the "over a WAN" part: VSS had been designed as a local VCS, so until the addition of a proper server in the mid aught using it over a network share was the only way to actually use it. And it really wasn't good.
I don't know if that made it better, I assume not much, VSS was really trash.
I don't know if that made it better, I assume not much, VSS was really trash.
I did this at my first, learned quick oldheads would get flustered and feel challenged if not eased into things a certain way.
Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.
Ultimately by the time I left I tried to introduce redbeanphp (orm), git for source control, and CakePHP for some structure. Nothing stuck. When I left it was still raw sql string queries, .zip files when they remembered for backups, and 400,000 line php files with everything caked on there.
Yes, there is a good lesson here. If you walk onto a dev team still using stone tools, there is likely a culture that distrusts of new technology, so tread lightly or you may be perceived as a heretic.
Have been that person before. As an intern, and they even listened! In the days of SVN, just before git, so I ran a server in my laptop and my manager somehow decided we needed a big Red hat server or something, IIRC. In a 20 ppl company.
Setting up git is the easy part. We all used it. Except the owner of the company who would fix bugs in prod and not tell anyone. Then next release we'd unintentionally un-fix those bugs because the fixes never made it back to source control.
Software Configuration Management has existed as a discipline and with proper tooling for at least 50 years. Mainframe and VAX machines had tooling in the early 80s.
For VB Sourcesafe was the go to tool if memory serves.
This is not a case of new vs old, rather incompetence vs competence.
For VB Sourcesafe was the go to tool if memory serves.
This is not a case of new vs old, rather incompetence vs competence.
Some of these stories sound a bit far fetched, especially those that involve Unix systems. RCS was released in 1982 and CVS in 1990 so Unix systems have had version control available for over forty years.
I can assure you they are true. Version control was still “controversial” in a lot of shops for quite some time. Plenty of places had the classic “v1, v2.3, v2_next_debug_cruffle_duffle” way of managing versions for a very, very long time.
Well into the subversion era…
Well into the subversion era…
I know places that still do it that way. Not necessarily tiny places either.
There is a lag between availability and mindshare to apply a technology.
I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it
git simply didn't have the mind share jet, thus I went with what I knew
I myself introduced SVN as versioning solution to a company in 2007, years after git was available I and I was aware of it
git simply didn't have the mind share jet, thus I went with what I knew
> I myself introduced SVN as versioning solution to a company in 2007
In 2013, I was tasked with writing wrapper scripts around SVN to make it look like SCCS to avoid confusing the development people who only knew how to use SCCS whilst the company migrated to SVN. Fun but traumatising.
In 2013, I was tasked with writing wrapper scripts around SVN to make it look like SCCS to avoid confusing the development people who only knew how to use SCCS whilst the company migrated to SVN. Fun but traumatising.
Hear, hear
Git was too weird for a team, but svn with its linear timeline was a hit. They are still with svn, and for a small shop it's ok
Git was too weird for a team, but svn with its linear timeline was a hit. They are still with svn, and for a small shop it's ok
Version control was virtually unknown outside UNIX systems, though, and, in lieu of it, mainframe/PC/Mac developers resorted to barbaric practices which included file tree renaming (v2, v1_John_is_an_idiot), countless zip files with similarly meaningful names with snapshots of entire projects and stuff like that. Then commercial version control systems started popping up, and they were very expensive, usually buggy af, and had no feature parity across themselves, i.e. knowledge of each was not portable.
Whereas nearly every UNIX installation included version control systems for free (SCCS in AT&T UNIX SVR1-4, RCS in BSD UNIX or both) that worked exactly the same everywhere.
Whereas nearly every UNIX installation included version control systems for free (SCCS in AT&T UNIX SVR1-4, RCS in BSD UNIX or both) that worked exactly the same everywhere.
In the late 2000s, I worked at $MAJOR_TELCO where management steadfastly refused to implement version control. Upgrades in production were executed by individually SSHing into each physical machine in the prod cluster and typing in commands by hand.
My attempt to introduce a "multissh" tool that automatically executed the same commands in each node at once was regarded with the highest suspicion and shot down. Shortly after I left, they had a multi-week outage caused by somebody fat-fingering the permissions on a network interface.
My attempt to introduce a "multissh" tool that automatically executed the same commands in each node at once was regarded with the highest suspicion and shot down. Shortly after I left, they had a multi-week outage caused by somebody fat-fingering the permissions on a network interface.
At least as late as 1998, I recall version control was thought of in some circles the way people think of C vs memory-safe languages today.
Some thought version control was an obvious improvement to make better, more bug-free software. Others had a fetish for doing things the hard way with little justification beyond emotional resistance.
Some thought version control was an obvious improvement to make better, more bug-free software. Others had a fetish for doing things the hard way with little justification beyond emotional resistance.
SCCS was released in 1977 and it hasn't even turned up in these comments at all. "Not evenly distributed". (I introduced RCS as a replacement for "just editing files" at a job in 1990; "there's a man page and some binaries on the system" really doesn't contribute to adoption. CVS at least propagated through things like USENIX, because solbourne made cool toys and talked about the challenges.)
It's maybe better than to take the pain to set up git only to see people use it in the same way, setting up a gazillion branches called V1, v2_customer_fix, v3_final, etc...
I'm assuming this story predates git and the Internet, but I'm not OP. Things were different back in those days.
Wait until you work in an industry where customers are extremely reluctant to upgrade and each one lands on a different release.
Then you have to start tracking the limitations and issues in 2-3 year old releases and the solution to issues is never “upgrade to the latest”.
At that point, source control starts feeling worthless.
Then you have to start tracking the limitations and issues in 2-3 year old releases and the solution to issues is never “upgrade to the latest”.
At that point, source control starts feeling worthless.
I think you misspelled "Visual SourceSafe" - and depending on when this VB was written it might actually predate VSS as a Microsoft product.
Source code management back in the days of VB (long before VB.Net) was not the same as what you see today.
Source code management back in the days of VB (long before VB.Net) was not the same as what you see today.
I don't know how they convinced the users to do this and how they managed with the license but it was done
Enterprise and business users will wade through endless swamps of crap if the software provides enough value. This is a lesson in why "it must be perfect before we release it" is such nonsense - that just says the value your app provides is so low that users barely care and they'll abandon it at the first problem. If that's the case you're not providing anything worth paying for.
Enterprise and business users will wade through endless swamps of crap if the software provides enough value. This is a lesson in why "it must be perfect before we release it" is such nonsense - that just says the value your app provides is so low that users barely care and they'll abandon it at the first problem. If that's the case you're not providing anything worth paying for.
As much as this stuff is nuts to think of today and there's tons to hate, I am kinda nostalgic for some aspects of my experience of working at a place where software is maybe needed and/or valued but isn't a core competency. Or maybe a time when software was a new fangled thing that hadn't fully been integrated into corporate structure yet:
- No one having any preconception of how you're /supposed to/ do things or whether you'd even be the type of person to know, so you just kinda figure it out yourself. You spend a lot of time on reading and learning skills. Version control? Wow, cool, what's git, let's try that! A new graphing library? Let's give that a shot, maybe it'll make things easier! You want XYZ? Let me go read about that for a day.
- No one having any idea what's even possible: being treated like a wizard for introducing the tiniest piece of automation or improvement that makes someone's day easier or doing something they never thought was possible. Lots of appreciation and excitement for showing and teaching people new things (... and I know this is somewhat selfish and ego driven, but who doesn't like being appreciated?)
- Similarly people having no idea how long those things should take which, tbh, can be a nightmare if you're not trusted and respected enough to be consulted but also great if people believe you when you say it's gonna take 3 months.
- Beyond the basics just being mostly kinda left alone to do your job however: no standups or tickets or the 30 other kinds of daily (micro)management that is probably necessary but ends up feeling tiresome and stifling at an individual level
- Not being part of software company "culture": no performance review driven development and promo packet madness, no weird rating and ranking systems, no OGPs or KPIs. No ladder. Your bosses think you did what was required of you, so then you're good, and if it's a good year you get a raise, and that's that. I do recognize that with a bad boss this can be a terrible and unfair spot to be in - but again, subjectively with a decent enough boss it felt like a lot less weight on my shoulders at the time.
- No hacker ninja pirate segway mini quadcopter you're the smartest people in the world and we're the best company to work for sort of b.s.
- Socializing with people who are good at and love to talk about stuff other than software
Reading over that, I'm thinking maybe I lucked out a lot and that wasn't most people's experience from that era. And there's some level of rose tinted glasses going on. And/or maybe my years in the rat race are starting to show :-)
- No one having any preconception of how you're /supposed to/ do things or whether you'd even be the type of person to know, so you just kinda figure it out yourself. You spend a lot of time on reading and learning skills. Version control? Wow, cool, what's git, let's try that! A new graphing library? Let's give that a shot, maybe it'll make things easier! You want XYZ? Let me go read about that for a day.
- No one having any idea what's even possible: being treated like a wizard for introducing the tiniest piece of automation or improvement that makes someone's day easier or doing something they never thought was possible. Lots of appreciation and excitement for showing and teaching people new things (... and I know this is somewhat selfish and ego driven, but who doesn't like being appreciated?)
- Similarly people having no idea how long those things should take which, tbh, can be a nightmare if you're not trusted and respected enough to be consulted but also great if people believe you when you say it's gonna take 3 months.
- Beyond the basics just being mostly kinda left alone to do your job however: no standups or tickets or the 30 other kinds of daily (micro)management that is probably necessary but ends up feeling tiresome and stifling at an individual level
- Not being part of software company "culture": no performance review driven development and promo packet madness, no weird rating and ranking systems, no OGPs or KPIs. No ladder. Your bosses think you did what was required of you, so then you're good, and if it's a good year you get a raise, and that's that. I do recognize that with a bad boss this can be a terrible and unfair spot to be in - but again, subjectively with a decent enough boss it felt like a lot less weight on my shoulders at the time.
- No hacker ninja pirate segway mini quadcopter you're the smartest people in the world and we're the best company to work for sort of b.s.
- Socializing with people who are good at and love to talk about stuff other than software
Reading over that, I'm thinking maybe I lucked out a lot and that wasn't most people's experience from that era. And there's some level of rose tinted glasses going on. And/or maybe my years in the rat race are starting to show :-)
Don’t think so. My first job was kind of like that. I don’t even know how they thought that little old me just out of university could be left alone to successfully build applications on my own, but I think people trusted a lot more during that era because eternal september hadn’t arrived yet.
Working directly for the users without any weird BA/PM/TA shit in between is glorious, both because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue), and in a place like that you can likely deploy it in the middle of the day and immediately improve their workflow.
It still amuses me that IT was located together with finance, because we did reports xD
Working directly for the users without any weird BA/PM/TA shit in between is glorious, both because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue), and in a place like that you can likely deploy it in the middle of the day and immediately improve their workflow.
It still amuses me that IT was located together with finance, because we did reports xD
> I don’t even know how they thought that little old me just out of university could be left alone to successfully build applications on my own, but I think people trusted a lot more during that era
A similar feeling on my end too :-) That might be it - trust sounds like a big part of it for me. Taking a chance on someone who you might eventually end up being good, rather than interviewing and evaluating them 7 ways till sunday. I understand the impulse. I wouldn't want to be a new engineer out of college today though - seems rough.
I did get paid less then than some new grads seem to be now so that might have been a factor in taking the pressure off.
> because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue)
Oh absolutely!
> It still amuses me that IT was located together with finance, because we did reports xD
It was communications for me, because the software tool we built was free to use on the web, and websites are communications, obviously :D
A similar feeling on my end too :-) That might be it - trust sounds like a big part of it for me. Taking a chance on someone who you might eventually end up being good, rather than interviewing and evaluating them 7 ways till sunday. I understand the impulse. I wouldn't want to be a new engineer out of college today though - seems rough.
I did get paid less then than some new grads seem to be now so that might have been a factor in taking the pressure off.
> because you can always walk up to get immediate feedback (people generally like to see you are actively working on their issue)
Oh absolutely!
> It still amuses me that IT was located together with finance, because we did reports xD
It was communications for me, because the software tool we built was free to use on the web, and websites are communications, obviously :D
You’ve got a point! There was a special moment there for a while. Your description perfectly captures my experience interning on a small IT team around 2000. This was in England so the secretaries would snigger whenever I said “debugger”. The downside was that the management had absolutely no clue about software as they’d jumped from some other career and the field was advancing quickly.
> There was a LOT of global variables (seemingly random 4 uppercase letters) controlling everything.
I once ran across a (c) program that had 26 variables, each one letter long, one for each letter of the alphabet. They were all global variables. And many of them were re-used for completely unrelated things.
I once ran across a (c) program that had 26 variables, each one letter long, one for each letter of the alphabet. They were all global variables. And many of them were re-used for completely unrelated things.
Ah, the "user-defined registers" paradigm.
I inherited a control program from Risø National Laboratories. It had roughly 600 globals of the form A9$, three local variables, and one comment - "Midlertidig".
However on a more practical note, the "Java" used on Smartcards effectively requires that all variables be treated as constants, other than one array. You dynamically allocate the array when handing an event, and it only lasts for the duration of that event.
However on a more practical note, the "Java" used on Smartcards effectively requires that all variables be treated as constants, other than one array. You dynamically allocate the array when handing an event, and it only lasts for the duration of that event.
wow this is just evil
I want an obfuscator doing such a transformation:D
I want an obfuscator doing such a transformation:D
Dear god this is pretty much what I went through when I started taking over a company with a 35-40 year old codebase. Files spread everywhere, no consensus, and supporting customizations for thousands of customers who we didn’t know if they were even still using the system.
It took five years and the firing of the long-time “head” programmer until some meaningful change was made.
It took five years and the firing of the long-time “head” programmer until some meaningful change was made.
This is nightmare-fuel. How does this happen?
As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices??
Honest question. How does our profession root out intrinsically obvious bad practices?
As a glib answer, can I suggest, that without proper training, there were a lot of developers who had never trained under anyone or any company with proper practices??
Honest question. How does our profession root out intrinsically obvious bad practices?
It happens because it’s easier - until it’s impossible, anyway.
The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.
The training and best practices you’re talking about is learned experience about how to avoid it getting impossible. But that almost always involves expense that the business side considers ‘stupid’.
> At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio.
Holy smoke! That's actually the most creative solution (horrible, but creative) I've ever heard to fix a Heisenbug:
https://en.wikipedia.org/wiki/Heisenbug
Holy smoke! That's actually the most creative solution (horrible, but creative) I've ever heard to fix a Heisenbug:
https://en.wikipedia.org/wiki/Heisenbug
Often it’s enough to compile in debug mode and maybe set an env var! This sounds like a horrible approach by all measures I think.
These type of stories are a humbling reminder that PMF always beats engineering excellence
Well that depends entirely on what you consider to be the goal - as a software engineer, your role is entirely concerned with engineering excellence. As a member of a team, especially a team of extremely highly paid and highly educated individuals, it is your duty to spend your time (and thus, the company’s resources) efficiently by doing what you’re educated, qualified, and hired to do.
Engineering excellence is a means to an end. Great engineers solve problems - sometimes by writing software.
Few people agree that the goal of SWE is engineering excellence. It is to solve business problems. Engineering excellence is a means to a goal: to be able to solve _difficult_ problems _correctly_ and allow solving new problems _in the future_. All these things can be traded off, and sometimes aren’t even needed at all
There is "do things that don't scale" and the there is bending over backward to do things in the dumbest possible way. Not sure where this lands.
Can I ask roughly what year this was? I can't even imagine something like that today
The thing is, I encountered something very similar with a product that had maybe 20 customers… in 2017. All of them had slightly different versions of the codebase. Version control was used, but haphazardly.
You’d think this sort of thing would be a problem of the 90s or early 2000s, but I’d bet you there are any number of companies with similar situations today.
You’d think this sort of thing would be a problem of the 90s or early 2000s, but I’d bet you there are any number of companies with similar situations today.
Visual Basic was a smash hit released in 1991.
For reference, Perforce was released in 1995, Subversion in 2000, git in 2005. RCS existed, but only in the Unix world.
For reference, Perforce was released in 1995, Subversion in 2000, git in 2005. RCS existed, but only in the Unix world.
Don't forget Visual Sourcesafe, that came out around 1995 as well and was the standard source control package for most of the Windows shops in the 90s and even up to the mid 2000s (at least in my experience)
Until about 2014 or so in my department, when we started using SVN. We were more concerned about file locking and a "single version of the truth" of MS Word files stored centrally but accessed locally than we were of fine-grained version control, and Git didn't have the kind of file locking we needed (as I understood it).
Don't forget CVS, released in 1990. It was popular for about a decade before Subversion took over.
Did it run under DOS? I'm not even asking about a win16 native GUI client. (You can still enjoy CVS if you want to contribute to OpenBSD, for instance.)
Yes, and for Win95 users there was actually a very nice GUI plugin to explorer called TortoiseCVS.
The funny thing about CVS and Subversion is that they were last version control systems that non-programmers could actually use. TortoiseCVS/TortoiseSVN were easy enough that with a lot of training you could get technical writers or artists to use it too. Valuable for game projects. With git, forget about it. It can't even handle binary files well.
The funny thing about CVS and Subversion is that they were last version control systems that non-programmers could actually use. TortoiseCVS/TortoiseSVN were easy enough that with a lot of training you could get technical writers or artists to use it too. Valuable for game projects. With git, forget about it. It can't even handle binary files well.
Mercurial and TortoiseHg were fine, too.
Only Git had the brain damage to take the source control idea of "there are two areas: working or committed" and add the dumbass index/staging/stash areas that confuse the hell out of everything.
Only Git had the brain damage to take the source control idea of "there are two areas: working or committed" and add the dumbass index/staging/stash areas that confuse the hell out of everything.
I'm not sure. In the mid-90's, I worked at a Solaris and NT shop, which was where I first used it.
The documentation for Office 2000 [1] describes version control for Visual Basic using Visual SourceSafe integration, although I'm not sure if anyone used it.
[1] https://learn.microsoft.com/en-us/previous-versions/office/d...
[1] https://learn.microsoft.com/en-us/previous-versions/office/d...
2008
This is that meme when the pizza guy walks into a burning room. Also, this comment is worthy of The Daily WTF.
I was in a broker trader in 2016 where this was still the case.
I was brought in when an old sheet cost them $10m on a bad trade because the yahoo finance end point it was hitting stopped responding and it just used the last value it had gotten - three months before.
I was brought in when an old sheet cost them $10m on a bad trade because the yahoo finance end point it was hitting stopped responding and it just used the last value it had gotten - three months before.
This is entirely typical of especially VB scripts. When I was a software engineer for a Fortune-20 company, I spent more time debugging (and trying to normalize, though that met with mixed levels of resistance) VB applets than anything else.
That's why cloud solutions exist.
Now you only need to run the app in debug mode on your own server.
Now you only need to run the app in debug mode on your own server.
> with some still having bugs fixed years before for others
Hopefully they won't discover at some point that customers were in fact depending on those bugs. That's poison for the programmer's soul.
Hopefully they won't discover at some point that customers were in fact depending on those bugs. That's poison for the programmer's soul.
Yup! Hyrum's Law :)
Oh dear god no. The solution is not throw VS at it and run from the code, the next step is some combination of excessive logging (which to be fair may resolve the issue all by itself) and/or throwing in a ton of DoEvents because Visual Basic.
My first role was with a company who had hit a limit for VB6 variable names iirc. So they'd all been renamed to shorter names. This may be the same issue. They were in the process of rewriting in VB.net.
That sounds like they were doing something dumb. VB6 did not have a limit for variable names, unless it was 255 characters or something.
And yep, I just checked: The limit was 255 characters.
And yep, I just checked: The limit was 255 characters.
I think more like globally. Like there was a fixed amount of space for all variable names.
This sounds like what I see when an inexperienced filmmaker takes on a big project and hands me the “organized” drive to edit, but way way worse and more consequential lol
Oh man, that brings me back. My first tech job was an ecommerce company that a basic online cart with backed by incredibly in-depth set of industry catalogs. We also sold a marketing package as an add-on to the online store where we would proactivly contact our customers and get the info from them to replicate their monthly/weekly physical advertising on the web. This was back in '05-ish, lots of money to be made just helping people get their business online.
We had a group of talented designers/artists wrangling Photoshop and pumping out a few of these designs a day, each, and as we scaled up and gained a lot of repeat customers, tracking these PSDs became a big problem. The designers were graphically talented, not technically savvy. The PSDs were stored on a shared NAS drive that the whole company could see. The designers had a complex naming system to manage major revisions, but overall there was no "history" beyond the classic "_v2_2008_09_best.psd" naming technique.
Several times every week I had to fix "accidentally dragged the PSD folder somewhere and lost it" level problems. Getting IM'd from tech support because the underlying server was falling over trying to clone the multi-GB folder 3 times, logging into a workstation as Admin and searching for an updated PSD that the vacationing Designer hadn't synced back to the NAS before leaving for work, that kind of thing.
As soon as I was promoted to Supervisor I made my first big move. It took a lot of training, far more talking than I thought it should (back then I didn't know anything about politics), but I was able to get SVN implemented to replace the NAS share. I wrote quick-reference documents, in-depth guides, (this was before I knew that no one reads anything, ever, for any reason) and eventually just had to do one-on-one training with everyone just to explain the concept and basic useage.
One of the most satisfying feelings of my career continues to be watching attitudes change over the course of a summer. None of the design-y people liked the new set of hoops they had to jump through. Check-Out, Check-In, Lock, etc, it was "too much". Then, at a happy hour someone mentioned how we hadn't lost the PSD folder in a while. Later someone came to me panicking because a client wanted to re-run an ad from 2 months ago with a couple tweaks, and she didn't have the source PSD or the source material -- I did a live demo on how to get a historical version back, and that's when it really clicked with everyone. With internal political will behind the toolset, it now became an IT problem, as our SVN useage was nothing like Engineering's usage.
Of course file locking was a huge PITA, that feature replaced "forgot to copy the changed file back before vacation" as a problem category. But it also eliminated the problem where 2 people would open the same PSD directly from the NAS share, make their changes, and only the last one to save gets their work persisted. So, a toss-up I guess.
We had a group of talented designers/artists wrangling Photoshop and pumping out a few of these designs a day, each, and as we scaled up and gained a lot of repeat customers, tracking these PSDs became a big problem. The designers were graphically talented, not technically savvy. The PSDs were stored on a shared NAS drive that the whole company could see. The designers had a complex naming system to manage major revisions, but overall there was no "history" beyond the classic "_v2_2008_09_best.psd" naming technique.
Several times every week I had to fix "accidentally dragged the PSD folder somewhere and lost it" level problems. Getting IM'd from tech support because the underlying server was falling over trying to clone the multi-GB folder 3 times, logging into a workstation as Admin and searching for an updated PSD that the vacationing Designer hadn't synced back to the NAS before leaving for work, that kind of thing.
As soon as I was promoted to Supervisor I made my first big move. It took a lot of training, far more talking than I thought it should (back then I didn't know anything about politics), but I was able to get SVN implemented to replace the NAS share. I wrote quick-reference documents, in-depth guides, (this was before I knew that no one reads anything, ever, for any reason) and eventually just had to do one-on-one training with everyone just to explain the concept and basic useage.
One of the most satisfying feelings of my career continues to be watching attitudes change over the course of a summer. None of the design-y people liked the new set of hoops they had to jump through. Check-Out, Check-In, Lock, etc, it was "too much". Then, at a happy hour someone mentioned how we hadn't lost the PSD folder in a while. Later someone came to me panicking because a client wanted to re-run an ad from 2 months ago with a couple tweaks, and she didn't have the source PSD or the source material -- I did a live demo on how to get a historical version back, and that's when it really clicked with everyone. With internal political will behind the toolset, it now became an IT problem, as our SVN useage was nothing like Engineering's usage.
Of course file locking was a huge PITA, that feature replaced "forgot to copy the changed file back before vacation" as a problem category. But it also eliminated the problem where 2 people would open the same PSD directly from the NAS share, make their changes, and only the last one to save gets their work persisted. So, a toss-up I guess.
[deleted]
How long ago was this?
My worst codebase story:
In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.
Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).
Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.
This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.
Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.
It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.
In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files. This functionality was not part of our actual source control system, but was instead accomplished with a series of csh shell scripts you could run after ssh'ing into our development server.
Each of our customers had a 'master' jar file that represented the actual final compiled product (a jar file is really a zip file archive, which bundles together the resulting compiled java class files).
Once you had finished implementing your code changes, you ran another set of scripts which found the master jar file for each customer, unzips it, copies the compiled files from your local machine into it, and zips it back up again. Finally the source control lock is released.
This means, effectively, that the codebase was never compiled as a whole at any point in the process, instead, we just manually patched the jar file over time with individually compiled class files.
Over the years, small errors in the process allowed a huge amount of inconsistencies to creep into the codebase. Race conditions would allow two developers to lock the same file at once, or a developer would change a class that was a dependency of some other code that somebody else was changing. Sometimes code changes would make it into some of the customer jar files, but not others. Nobody knew why.
It took a small team two years to migrate the entire codebase to git with proper CI, and a huge chunk of that time was reproducing a version of the codebase that actually compiled properly as a whole. After the project was finished, I resigned.
So they had a problem, got 2 years of approved development effort of a small team to solve it property which they did successfully, and then you resigned? After they fixed the problem?
Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.
I presume there were other reasons for resigning, or you just like massive refactoring projects.
Of course where they started was just awful but a place that recognized it's problems, commits to fixing it, and has sufficient competency to actually fix it sounds rather nice to me. Many orgs get stuck at step 1.
I presume there were other reasons for resigning, or you just like massive refactoring projects.
It was a little tongue in cheek, but yes. I had large grievances with the software culture there, but after I got sign off on the project to modernise our build process, I couldn't bring myself to abandon ship in the middle of trying to fix it.
After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
After everything was finished up, I was feeling burnt out and realised that I'd held on for too long at a company with a fundamentally bad culture that wasn't going to change just because the tech did, so I moved on.
Thank you for the clarification. Because you said “it took a small team … and then I resigned”, it was unclear that you were part of that small team and instead made it sound like you left because the problem was fixed.
For what it’s worth, it wasn’t unclear when I read it.
I worked at a company recently where I couldn't get a team to do the fairly simple task of cleaning up our git ignore (so we wouldn't have to painstakingly add certain files to the index every time we changed something) so I take it as a massive accomplishment moving to git within two years.
If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.
If I know anything about work, I doubt this is all they did for two years. Business doesn't care you have an important project. They will bug you to do other things.
Yeah, maybe it's one of those reading between the lines things, but I think anyone familiar with burnout would understand.
There are few efforts as truly thankless in many organizations as fixing things.
There are few efforts as truly thankless in many organizations as fixing things.
Had a similar thing on a small team at a bigger company. We didn't have any type of version control so the team did all development on a shared dev server that we SSH'd into. We'd use vim to edit files one at a time and rely on vim's file locking to make sure that we weren't editing anything at the same time as anybody else.
Oddly, the project was one of the best I've ever worked on.
Oddly, the project was one of the best I've ever worked on.
I recall something similar from my first job, except the shared file locking was a full on feature in Macromedia dreamweaver.
CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.
CSS was just starting to get adopted and every project we worked on just had one “gobal.css” file. When someone else had global.css locked, you’d call dibs if you needed it next. Inevitably, everyday someone would leave the office and forget to unlock global.css and no one else could get anything done.
Macromedia flash / .fla files weren't particularly ideal for collaboration either, though I still feel a bit nostalgic for working with flash in general
It just did this by creating a sentinel file- so if you needed to, you could just delete the file manually.
SVN provided the ability to steal locks, and locks were opt-in so e.g. you wouldn't have made CSS need locks because it's a text file and merges fine. Mandatory locking was mostly for binary work files e.g. PSD and the like.
Good to know, though I’m optimistic this is knowledge I’ll never need again!
...and whoever did that "accidentally" when they had to leave around lunchtime on Friday was, presumably, celebrated as a local legend.
I think a lot of people who've only learned programming in the last 10 years or so don't realize that Git, and even more so the large-scale popularity of Git, is actually pretty new. A lot of programming happened before Git was created, and before it became mature and popular enough to be the default for everyone.
Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.
Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.
I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.
[0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
Many of these earlier systems sound terrible and hacky to us, but they were the best that was available at the time. Systems to "lock" files you were working on were pretty common because basically nobody did merges well. Most of them were based on having a server to manage the whole thing too, so they were only really common in corporations and larger and more mature hobbyist teams - this was also before you could spin up a cloud server for most things with a few keystrokes and $5 a month. It's low-effort to spin up a local Git repo to track your work on a tiny personal project, but nobody's going to set up a CVS server for that.
Anybody remember Joel On Software's 12 Steps [0], written in the year 2000, 5 years before the Git project was started? Where "use source control" is Step 1? There's a reason why that's there - at the time it was written, source control was clunky and a pain in the ass to set up, so a lot of smaller companies or ones with less mature development teams never got around to it.
I may be getting a little "old man yells at clouds" here, but be really thankful that Git is FOSS, ubiquitous, works great for 98% of projects, and is superlatively awesome compared to everything that came before it.
[0] https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
I use CVS, RCS and Subversion on my personal projects for several years before I learned git. I don’t remember any of them being a pain to setup for small projects and Sourceforge provided free CVS hosting for small projects.
I started with Rational Clearcase, which I interacted with via Eclipse, and I remember that you did have to lock a file to make changes to it... pretty sure there was no merge capability (but maybe I was just too junior to know stuff like that, it was definitely not something you would do as a matter of routine, like we do now with git). Someone leaving for vacation and forgetting to unlock a file could be a huge pain in the ass.
After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.
After that I used Subversion via TortoiseSVN for Windows, which was quite nice to use, though I don't really remember much of the details... I do remember asking myself why everyone was moving to git when "perfectly good" source control already existed, but after 10+ years on Git, I think I was wrong and we did need it, we just didn't know it yet.
Rational Clearcase is a disaster but you could do merges.
The fun/frustrating part was getting branching and "views" correct.
I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.
The fun/frustrating part was getting branching and "views" correct.
I worked at an organization on this side of the century where it was one person's full-time job to handle merging code from different teams. The same organization didn't actually use raw Clearcase either -- it was wrapped in hundreds-of-thousands-of-lines-of Perl scripts.
I've never actually set up CVS myself, so I suppose I can't accurately say how easy or not easy it is. The "nobody" might deserve an asterisk - I'm sure it's not literally not a single person, but it seems to be pretty rare in my impression. Nevertheless, AFAIK, the non-distributed nature seems pretty limiting.
In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?
In Git, I can just create a local repo for anything, and I don't have to decide where I want it. I could never push it anywhere, or push it to a private hosted repo to collaborate with a person or team, or push it to a public repo to open-source it, and change between any of those at any time. I don't think you can really do that with these hosted systems. You could spin up a local CVS server if you really wanted to, but if you later decided you wanted to open-source it on Sourceforge, don't you have to abandon the history and publish it there as-is?
I don’t think so, it’s been a while, but I think there were tools for copying history between cvs servers. I don’t remember CVS being particularly difficult to manage.
I set up SVN for myself before Git and IMO Git is definitely easier.
I’ve used Clearcasse in 2006 and it had all the branches that we needed. We didn’t use locks because that was annoying. It was at Deutsche Bank and we were 200.
When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).
When I moved to SVN, it was a regression. I really don’t understand why people describe SVN as cutting-edge (and why people didn’t like Clear Case).
I once worked at a corp that sold software. For the exact same software product, maybe 10-20% of customers received so much benefit we could have charged 2-3x the price and they would still be very happy. On the other end, for maybe 10-30+%, the exact same product ranged from painful to a disaster. Often because they would try to force it to work the way their pre-existing paradigms worked instead of taking the time to learn how it actually worked.
> source control was clunky and a pain in the ass to set up
I concur. Creating a CVS repository was not harder than creating a git repository now `cvs -d $dir init`, and using, typical for the time, shared network drive, gives server experience without setting up a specialized server. It was just not included with Windows/typical Windows dev tools, so you had to know it existed, and knowledge was slower and harder to get.
I concur. Creating a CVS repository was not harder than creating a git repository now `cvs -d $dir init`, and using, typical for the time, shared network drive, gives server experience without setting up a specialized server. It was just not included with Windows/typical Windows dev tools, so you had to know it existed, and knowledge was slower and harder to get.
We all take too much for granted how much git "just works", even when it doesn't.
Even those before git - what they're describing sounds a bit like RCS, the precursor to CVS, which came before SVN.
I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
I've never used RCS or CVS myself, but I remember the file locking thing in descriptions of it, and that it was why CVS was named "concurrent" - it fixed that limitation.
VSS6 had file checkout, which IIRC locked the file preventing other devs from editing it. It was popular with Visual C++ 6.0, circa the year 2000.
https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe
https://en.wikipedia.org/wiki/Microsoft_Visual_SourceSafe
[deleted]
This sounds so much like dealing with MS Access databases... Unfortunately, part of my responsibility in my current role is to manage a handful of Access applications... they are ancient, I am talking early 2000's. They are the most unruly thing I have ever had to work with, and I will not go into details but your story reminds me so much of having to work on these apps...
MSAccess is a tragedy, imo. Right-up until, say, Access 2007, it was a simple (in a good way!) RAD platform following those zombie 4GL predecessors, but it’s been left to stagnate without any real effort to modernise it, and it’s clear from how Microsoft’s been downplaying Access that it’s a product they’d really rather not have to support, but they know if they do kill it then it would weaken on-prem Office’s moat and everyone will probably just rush to Google Firebase.
Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.
Most of Access’ problem is how it’s inseparable from both VBA and COM. While Microsoft has tried to make COM sexy again with WinRT and C++/CX, VBA is quite senile now. Microsoft has killed VBA in Outlook, killed VBScript in Windows Server, and is now trying to kill VBA in Excel too. MS is pitching modern JS as a successor (which I’m not too mad about…) but I just don’t see how JS could unseat the sheer volume of Access VBA out there. Especially as “Office JS” is intentionally kneecapped for things like local computer files and Win32 so it has feature parity on web vs mobile app vs desktop - it’s going to be awful.
> It took a small team two years to migrate the entire codebase to git with proper CI
I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.
I'm amazed they even allowed people to spend company time and money doing this. Corporations tend to not see the value in improving such precarious situations as long as the software keeps working. They just want employees to cope and accept the messiness. People usually cope by quitting.
I’ve worked on “check out” code bases very similar to this. I mean. Nothing so insane as scripts patching class files in to jars Willy nilly, but it seems the “just check out a file so nobody else can work on it” thing is a “common” “solution” to this.
I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)
I am interested in how you managed it when someone had to hold code for years (as I’ve seen)? Basically, we also had a way to share control with one other person, and the person who was taking the second source were responsible for updating both versions at the same time manually (which never happened, so implementing a large project that touched hundreds of source files had to dedicate a couple weeks to manually hand comparing files, manually implementing the changes, and then manually retesting)
> In my first real job, I worked for a company that maintained a large legacy product programmed in a combination of COBOL and Java.
> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.
COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!
But we were using Subversion's lock system and had a proper CI/CD with Jenkins.
So you won, your project was worse!
> In order to work on the Java side of the product, you checked out individual files from source control to work on, which 'locked' the files and prevented other developers from checking out the same files.
COBOL + Java and file lock... I thought we worked on the same project and it brought back many nightmares!
But we were using Subversion's lock system and had a proper CI/CD with Jenkins.
So you won, your project was worse!
Visual SourceSafe would show that a file was checked out hinting to maybe stay away. Good times.
Did you know, theres a GitHub repo called [you-poor-bastard](https://github.com/SirkleZero/you-poor-bastard)? It converts a vss repo to git (not very well, but well enough), ignoring the VSS "passwords."
IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.
The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.
The admin user could override or unlock locked files. We had to do this if a developer left a file checked out after they left, or were on vacation. Nobody knew the admin password for the SourceSafe repository. That was OK though, all you had to do was make a local account on your PC named the same as a the source safe admin acccount, and you'd have admin access in SourceSafe.
> IIRC SourceSafe could be configured with strict locking or advisory locking? I might be wrong about that.
IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).
I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).
IIRC SourceSafe could be configured with either strict locking (you had to lock a file in order to edit it) or no locking à la SVN (the tool would check if your file was up to date when trying to commit it).
I recall that I spent a while suffering under strict locking at my first internship before a colleague discovered this mode existed and we were able to work more reasonably (there were no editable binary files so locking was never really needed).
We still use VSS for a couple codebases. We’re a small enough team that conflicts are rare, but the occasional Teams message “hey, let me know when you’re done with module.cpp” is not unheard of.
Fascinating. An anachronism worthy of steampunk novels!
I’m impressed VSS still works. I used it as part of my first professional software engineering job in 2008 and it felt old then.
I'm so sorry. We were using it in the late 90s, and on a monthly basis it would be down for a day or two while the admin plunged out the database. Good times.
It surprisingly works well. We do use a front end to it called VssConnect, but to be honest, I really don’t have any complaints.
That's just maddening enough to start a cult!
The Rube-Goldberg VCS-mutex is hilarious though.
The Rube-Goldberg VCS-mutex is hilarious though.
This is insane. Thanks for the post.
I bet now they deal with broken pipelines and dependency hell.
Tools do not fix bad design.
Tools do not fix bad design.
Probably some of the worst code I ever worked on was a 12k+ line single file Perl script for dealing with Human Genome Project data, at Bristol-Myers Squibb, in the late 1990s.
The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo.
By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.
The primary author of it didn't know about arrays. I'm not sure if he didn't know about them being something that had already been invented, or whether he just didn't know Perl supported them, but either way, he reimplemented them himself on top of scalars (strings), using $foo and $foo_offsets. For example, $foo might be "romemcintoshgranny smithdelicious" and $foo_offsets = "000004012024", where he assumes the offsets are 3 digits each. And then he loops through slices (how does he know about slices, but not arrays?) of $foo_offsets to get the locations for $foo.
By the time I was done refactoring that 12k+ was down to about 200 ... and it still passed all the tests and ran analyses identically.
We should use the Antarctic highlands as a prison colony for people found guilty of writing Stringly Typed Code. Siberia isn’t awful enough for them. I thought people who stuffed multiple independent values into a single database column were the worst and then I saw what people can accomplish without even touching a database.
Ha - we did that too at BMS. We were paying Oracle by the column or something like that, so people would shove entire CSV rows into a single value (because corporate said everything HAD to be in Oracle) and then parse them application-side.
> people would shove entire CSV rows into a single value
So you invented a “no-sql” document database in Oracle.
So you invented a “no-sql” document database in Oracle.
We got rid of most of the DBAs because they got overfond of saying 'no' and they became a major source of friction for doing reasonable architecture. People started figuring out ways to work around the DBAs like stuffing more shit into existing columns or adding secondary tables that established new relationships between the sacrosanct tables.
A lot of the worst sins happened on Oracle databases, mostly because they were the most popular at the time, and they were the only real DB cult, so the cottage industry of specializations that told other developers not to worry their pretty little heads was especially bad.
A lot of the worst sins happened on Oracle databases, mostly because they were the most popular at the time, and they were the only real DB cult, so the cottage industry of specializations that told other developers not to worry their pretty little heads was especially bad.
I have to work with a DBA who has decided that nothing new gets developed using Postgres and is confident that our use cases are best served with a document db… all without knowing any of our use cases, requirements or constraints.
Now I just don’t involve him in anything unless forced.
Now I just don’t involve him in anything unless forced.
Now it's JSON strings, so it's okay.
Sounds like they charged by the column but you actually paid with your souls.
Needful Databases
A friend once picked up a tiny contract from Uncle Sam, requiring some changes to a Perl script that stashed values as "this/that/thenext" in one column of an Oracle table. She had not previously dealt with Perl, so I explained to her about "split" and "join".
I don't know why the developers had chosen that method.
I don't know why the developers had chosen that method.
Ah yes, like calling collect and telling them your name is "mompickmeupfromthemoviesnowbye".
No we should reserve the antarctic highlands for type system fetischists who write abstract protocol adaptor factory repositories for every damn simple piece of functionality.
That sounds more like an OOP thing than a type system thing - functional languages make excellent use of type systems without mention of anything remotely sounding like a “Design Pattern (tm)”
No those people just get recycled.
People balk at the idea of licensure, but the bar is so low that all you have to do is ask "what is an array?", and you'll filter out vast swaths of people like this.
Goes to show that solutions don't need to be good to be impressive. Because damn, I'm impressed he did that
If I were being charitable (sometimes I try), I'd guess he was concerned that an array of scalars is going to have a high per scalar size overhead and perhaps worse cache locality, whereas the non-unicode perl string implementation at the time probably just stored the string as a large char* with a length prefix.
In a similar vein, an alternate explanation may just be that a lot of bioinformatics algorithms are described as string algorithms. It would be a pity if that was just taken too literally, but I've seen a lot of bioinformatics perl that works on strings in this way.
While trying to be charitable though, it's almost impossible to come up with a rationalization for the three character offsets string. Sometimes I think when people really go wrong is when they hear of some problem or are trying to be too clever and end up going beyond their abilities and stray into farce.
In a similar vein, an alternate explanation may just be that a lot of bioinformatics algorithms are described as string algorithms. It would be a pity if that was just taken too literally, but I've seen a lot of bioinformatics perl that works on strings in this way.
While trying to be charitable though, it's almost impossible to come up with a rationalization for the three character offsets string. Sometimes I think when people really go wrong is when they hear of some problem or are trying to be too clever and end up going beyond their abilities and stray into farce.
I’ve seen this “misunderstood genius” argument so many times I think there needs to be a law to refute it:
Never attribute to competence that which is adequately explained by incompetence.
Never attribute to competence that which is adequately explained by incompetence.
Being wrong with malicious intent and being wrong because you have an interstate-number IQ are indistinguishable; the outcome is still terrible. I don't care who's playing 4D chess, the pieces are still all over the floor and I have to clean them up.
That's a very impressive achievement, reducing that to around 200 lines. Congrats :)
Well the lines could very well be 1000s of bytes long. Perl oneliners used to be a thing in a company that I used to work for.
At least there were tests ;-).
I am somewhat surprised that a programmer who was unaware of arrays in Perl managed to have tests. But then again, he managed to implement their own version of arrays, maybe he came up with the concept of software testing by himself :-P
I am somewhat surprised that a programmer who was unaware of arrays in Perl managed to have tests. But then again, he managed to implement their own version of arrays, maybe he came up with the concept of software testing by himself :-P
It sounds more like the refactoring started with creating tests for the old code base. We do this with our legacy code base, too, sometimes. Of course, it’s always the high risk code of an application that‘s never been accompanied by automatic tests.
Nice. But I have to ask, considering it was Perl: Could an outsider understand it after you reduced it to 200 lines?
199 of the lines were comments.
As an outsider of any reasonably large application can you really ever expect to grasp 200 random lines of code in any language?
Maintenance programming is all about understanding the software‘s context and implicit design or a model thereof — even if it is a 20 year old patch work.
Young developers tend to be amazed when I find the source of an obscure bug in our 1M lines of Perl code application as a senior. But the thing is, I just happen to „get the application“ and developed intuition about its inner working. It’s rarely of knowing Perl particularly well. The thing could have been written in TCL or even Brainfuck — with a working model of your software in your head (not with the memory of the total code base, mind you) you will eventually find the problem in a piece of software written in any language.
Maintenance programming is all about understanding the software‘s context and implicit design or a model thereof — even if it is a 20 year old patch work.
Young developers tend to be amazed when I find the source of an obscure bug in our 1M lines of Perl code application as a senior. But the thing is, I just happen to „get the application“ and developed intuition about its inner working. It’s rarely of knowing Perl particularly well. The thing could have been written in TCL or even Brainfuck — with a working model of your software in your head (not with the memory of the total code base, mind you) you will eventually find the problem in a piece of software written in any language.
If you get a typically-structured Java/Spring application then often yes. By typically structured I mean domain objects, DTOs, service layer, controller layer, view layer. The idiomatic use of dependency injection frees the mind of the reader from understanding all the dependencies in code, just use an @Autowired class you want and someone part has the knowledge to configure it and does so.
Is code typically written for outsiders?
You were very lucky there were tests.
Admittedly I've somehow only worked in perl but the worst code I tried for fix felt similar. They know about arrays but every map and grep used perks default $_ and there was enough nesting the function was near 1k lines if I remember right.
Oh, the shipping! Now, from the customer’s point of view.
My youngest worked in a furniture chain over the summer. And they got sent a big, heavy furniture set from the central warehouse, which the store actually didn't want. So, they sent it back. The problem was that the system didn't allow them to say: please, don't send this again. And the non-natural intelligence at the central base decided to send this set again. When my youngest started working - they were loading this set back for the seventh time.
Why 'loading'? Because no one could find a way in the program to send this ill-fated set back on the same truck that brought it. No one, except my youngest, that is. He managed to find the combination of keys and checkboxes that allowed them not to unload the unwanted set and ship it back on the same truck - and he immediately got a raise.
I suspect the set is still traveling. But now they only load-unload it at the central warehouse.
My youngest worked in a furniture chain over the summer. And they got sent a big, heavy furniture set from the central warehouse, which the store actually didn't want. So, they sent it back. The problem was that the system didn't allow them to say: please, don't send this again. And the non-natural intelligence at the central base decided to send this set again. When my youngest started working - they were loading this set back for the seventh time.
Why 'loading'? Because no one could find a way in the program to send this ill-fated set back on the same truck that brought it. No one, except my youngest, that is. He managed to find the combination of keys and checkboxes that allowed them not to unload the unwanted set and ship it back on the same truck - and he immediately got a raise.
I suspect the set is still traveling. But now they only load-unload it at the central warehouse.
What an amazing read, funny, nostalgic, the duality of the perfect mess but still so much opportunity and ability to make progress, and somehow it all just chugging along.
I feel a lot of this is the difference between theory and practice. Sure each of these things are bad, but probably a lot might have been the right choice at the time, and in a way, most companies, even most projects running for many years, end with similar quirky processes, messes, and hacks.
It's the war stories of what used to be, often only told by the code and the database as the creators have long left.
When I look at our couple year old startup, we have some of these things. Prototypes that just keep functioning and only get more attention when they break, manual processed that sort of make sense but also don't, integrations with systems that were build in the early 2000's (it works, but it ain't pretty). We fix many, but I'm sure some will survive way too long.
As software engineers, especially online, we like to discuss the ideal way to do things, best practices, testing strategies, redundancies, the whole nine yards. When time is of the essence and stakes are high, it all goes out of the window and you gotta just make it work in whatever way is possible to fight another day.
I feel a lot of this is the difference between theory and practice. Sure each of these things are bad, but probably a lot might have been the right choice at the time, and in a way, most companies, even most projects running for many years, end with similar quirky processes, messes, and hacks.
It's the war stories of what used to be, often only told by the code and the database as the creators have long left.
When I look at our couple year old startup, we have some of these things. Prototypes that just keep functioning and only get more attention when they break, manual processed that sort of make sense but also don't, integrations with systems that were build in the early 2000's (it works, but it ain't pretty). We fix many, but I'm sure some will survive way too long.
As software engineers, especially online, we like to discuss the ideal way to do things, best practices, testing strategies, redundancies, the whole nine yards. When time is of the essence and stakes are high, it all goes out of the window and you gotta just make it work in whatever way is possible to fight another day.
This is the main takeaway for me. The decentralized way of software development in a large scale. It does echoes with microservices a lot, but this can be done with a more traditional stack as well. It's ultimately about how you empower teams to develop features in parallel, and only coordinate when patterns emerge.
The worst codebase I've ever worked on was the Telegram client for Android. I mean just look at this file, it's so huge GitHub gives up rendering it: https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/...
Or this one, it implements EVERYTHING for rendering and interacting with a message in a single class, all non-service message types, all drawn manually on a canvas "for performance", and all input processed manually too: https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/...
I even had the permission from the main developer to refactor some of it, but I never got around to actually doing it.
It's a miracle this app works at all.
Or this one, it implements EVERYTHING for rendering and interacting with a message in a single class, all non-service message types, all drawn manually on a canvas "for performance", and all input processed manually too: https://github.com/DrKLO/Telegram/blob/master/TMessagesProj/...
I even had the permission from the main developer to refactor some of it, but I never got around to actually doing it.
It's a miracle this app works at all.
This dude has absolutely no right to say anything about the quality of the source code of the android app, because:
1) His libtgvoip code, previously used for audio calls, is the worst code I have ever had the displesaure of touching, it caused all kinds of issues ranging from instability to segfaults, and thankfully it was completely replaced by webrtc.
2) The android app is literally the smoothest and most responsive android app I've ever used.
Grishka, please stop being so salty. VoIP is hard to do right, but not impossibly so. VoIP being hard is still not an excuse for writing garbage C/C++ code (and I don't mean to offend personally here, but the code was really a mess, please stick to Java development on your own personal projects).
1) His libtgvoip code, previously used for audio calls, is the worst code I have ever had the displesaure of touching, it caused all kinds of issues ranging from instability to segfaults, and thankfully it was completely replaced by webrtc.
2) The android app is literally the smoothest and most responsive android app I've ever used.
Grishka, please stop being so salty. VoIP is hard to do right, but not impossibly so. VoIP being hard is still not an excuse for writing garbage C/C++ code (and I don't mean to offend personally here, but the code was really a mess, please stick to Java development on your own personal projects).
Speaking as an outsider... Your comment reads as a blatant ad-hominem attack and does nothing to support your point. You should consider toning it down if your goal is to convince anyone.
Some of us have trod this path before, and recognize righteous anger when we see it.
That's just a cultural difference. I've seen Russian developers getting into physical fights over code more than once.
Thankfully, western (and especially american) corporate culture is much more conflict-awerse.
Thankfully, western (and especially american) corporate culture is much more conflict-awerse.
1) Which version are we talking about? Segfaults were exceedingly rare either way. Especially so when I stopped using raw pointers. But yes, to no one's surprise, early versions were a mess. I can agree with you on that. I can't agree about the same about the last versions though.
It was replaced by WebRTC that would sometimes just suddenly disconnect your calls, right.
> I managed to improve the situation after weeks of refactoring and testing
Did you submit any PRs with your improvements? I used to review and merge those, unlike the Android app devs.
2) It might provide the best UX in the entire universe, but that still doesn't justify having a two-megabyte Java file with over a hundred anonymous inner classes that gaslight you because it's often a `new FrameLayout(activity){` that overrides onLayout and does something completely different. So you see a FrameLayout, you assume it behaves like one, you add views to it, then it doesn't work the way you expect. You start questioning your sanity and only then do you realize that it's one of these stupid things.
Oh and did I mention that even just editing ChatActivity.java is an exercise in frustration? At the time, I had a 2012 MacBook Pro that worked fine for everything I did on it, except editing 2-megabyte Java sources in Android Studio. It would sometimes take a solid 10 seconds for keystrokes to register.
In other words, it might be the best Android app in the world, but it blows up in your face nearly every time you try to add a new feature.
It was replaced by WebRTC that would sometimes just suddenly disconnect your calls, right.
> I managed to improve the situation after weeks of refactoring and testing
Did you submit any PRs with your improvements? I used to review and merge those, unlike the Android app devs.
2) It might provide the best UX in the entire universe, but that still doesn't justify having a two-megabyte Java file with over a hundred anonymous inner classes that gaslight you because it's often a `new FrameLayout(activity){` that overrides onLayout and does something completely different. So you see a FrameLayout, you assume it behaves like one, you add views to it, then it doesn't work the way you expect. You start questioning your sanity and only then do you realize that it's one of these stupid things.
Oh and did I mention that even just editing ChatActivity.java is an exercise in frustration? At the time, I had a 2012 MacBook Pro that worked fine for everything I did on it, except editing 2-megabyte Java sources in Android Studio. It would sometimes take a solid 10 seconds for keystrokes to register.
In other words, it might be the best Android app in the world, but it blows up in your face nearly every time you try to add a new feature.
> but that still doesn't justify
No, that really justifies anything. Programs should be pleasant for users to use, not for developers to work on them
No, that really justifies anything. Programs should be pleasant for users to use, not for developers to work on them
One does not exclude the other.
And, honestly, these things are connected. It takes more needless work from the developer to maintain the user-visible pleasantness while adding new features. The code is much more fragile than it could've been which means bugs are much easier to introduce and much harder to diagnose and fix.
And I'm not asking for a complete rewrite. Just losslessly restructure the existing code, extract all those custom FrameLayouts and ViewGroups into separate classes, split those 2000-line if statements into separate methods, make some things (like ChatMessageCell) into sensible class hierarchies, introduce constants or enums to replace magic numbers, all that sort of stuff. This will not affect the UX, but it would make bugs harder to introduce and easier to deal with.
I'm generally not a fan of the developer-experience-focused approach to software development (Electron, React, and the web dev "trust me bro" attitude should not exist), but sensible code structure is where I draw the line.
And, honestly, these things are connected. It takes more needless work from the developer to maintain the user-visible pleasantness while adding new features. The code is much more fragile than it could've been which means bugs are much easier to introduce and much harder to diagnose and fix.
And I'm not asking for a complete rewrite. Just losslessly restructure the existing code, extract all those custom FrameLayouts and ViewGroups into separate classes, split those 2000-line if statements into separate methods, make some things (like ChatMessageCell) into sensible class hierarchies, introduce constants or enums to replace magic numbers, all that sort of stuff. This will not affect the UX, but it would make bugs harder to introduce and easier to deal with.
I'm generally not a fan of the developer-experience-focused approach to software development (Electron, React, and the web dev "trust me bro" attitude should not exist), but sensible code structure is where I draw the line.
Tell me you're not a professional software engineer without telling me you're not a professional software engineer.
Telegram happens to be one of the smoothest app experiences on Android, both in terms of perf and UX, so they must be doing something right.
Well, Telegram is faster/smoother than FB Messenger, Whatsapp, Discord (dog slow) etc...
So it is not only working but working well.
So it is not only working but working well.
It’s even more crazy because Telegram is by far the smoothest messenger app on Android.
BTW, would you mind sharing how you got that job?
BTW, would you mind sharing how you got that job?
> BTW, would you mind sharing how you got that job?
I started working with Pavel Durov in 2011 on the VKontakte Android app, after winning several of his developer contests, and it all just went from there. I was the last one of the "old" team to transfer to Telegram. My main job was actually the libtgvoip[1] library and its integration into the Android app (the other platforms were done by their corresponding devs).
I got fired in 2019 because the calls weren't working satisfactorily for Pavel. He got furious when he was unable to call some journalist, cursed at me in front of the entire team and sent me a termination notice. In hindsight, I should've definitely not taken building a VoIP library from scratch alone upon myself. It's a massive undertaking. But at least I learned a lot, and many people say that my implementation performed noticeably better than the current WebRTC-based one ¯\_(ツ)_/¯
I'm now working on two fediverse projects at the same time — Smithereen[2], a VK-like ActivityPub server, and the Mastodon Android app[3].
[1] https://github.com/grishka/libtgvoip
[2] https://github.com/grishka/Smithereen
[3] https://github.com/mastodon/mastodon-android
I started working with Pavel Durov in 2011 on the VKontakte Android app, after winning several of his developer contests, and it all just went from there. I was the last one of the "old" team to transfer to Telegram. My main job was actually the libtgvoip[1] library and its integration into the Android app (the other platforms were done by their corresponding devs).
I got fired in 2019 because the calls weren't working satisfactorily for Pavel. He got furious when he was unable to call some journalist, cursed at me in front of the entire team and sent me a termination notice. In hindsight, I should've definitely not taken building a VoIP library from scratch alone upon myself. It's a massive undertaking. But at least I learned a lot, and many people say that my implementation performed noticeably better than the current WebRTC-based one ¯\_(ツ)_/¯
I'm now working on two fediverse projects at the same time — Smithereen[2], a VK-like ActivityPub server, and the Mastodon Android app[3].
[1] https://github.com/grishka/libtgvoip
[2] https://github.com/grishka/Smithereen
[3] https://github.com/mastodon/mastodon-android
> He got furious when he was unable to call some journalist, cursed at me in front of the entire team and sent me a termination notice.
Honestly does not sound like a very healthy person to work for anyway.
Honestly does not sound like a very healthy person to work for anyway.
Does this seem like a healthy person to you?
https://www.instagram.com/durov/
https://www.instagram.com/durov/
Depends on what you mean by "healthy". Physically, yes, sure looks so. But that's just one definition.
I feel like those are missing a bear.
I guess the best thing that can be said is at least he’s confident enough in himself to post that.
I guess the best thing that can be said is at least he’s confident enough in himself to post that.
No, that's painfully cringe.
There are two sides to every story
There's no side where berating your subordinates in front of others is justified. That's unprofessional and childish. Always.
Sure, but I do not feel like it’d really affect the broad strokes of ‘criticized in front of the entire team’.
What was it like to work for telegram/durov? A part from the "getting fired" part that is ;)
Why did the call fail? What failed with your implementation/the code?
We'll never know. There are many things that could go wrong because VoIP over the public internet is inherently fragile. When I asked for logs so I could look into it, I got cursed at once again.
Grishka wrote very low quality code (an extremely buggy and segfaulty mess of C and C++), combined with little testing in not-so-edge case network conditions.
I managed to improve the situation after weeks of refactoring and testing for my library, and thankfully now it was completely replaced by webrtc.
I managed to improve the situation after weeks of refactoring and testing for my library, and thankfully now it was completely replaced by webrtc.
They didn't specify it was their job, the project is open source after all.
My first day at my first full-time programming gig, I was asked to look at some reporting job that had been failing to run for months. I logged in, found the error logs, found that it needed a bit more memory assigned to the script (just a tweak in the php.ini) and let the team lead know it should run fine that night. He was shocked, "Dude, if you just fixed that report you probably just got a promotion, no one has been able to figure that out for months." He was joking about the promotion, but my boss was just as shocked. I'd realize later that most the other people on the dev team didn't like linux and wanted to rewrite everything in .NET and move everything to Windows so no one even tried with anything related to any of the linux machines.
I know things have gotten somewhat better, but the amount of wasted time and latency of using RDP and Windows UI for development, testing and production maintenance is insane. Throw in some security requirements of RDP into host 1 to the RDP jump to host 2 and companies are just wasting money on latency. There is, often, not an appreciation of the administrative costs of the delivery. Not necessarily system admin costs, but developer and QA time associated with delivering and ongoing maintenance.
>Throw in some security requirements of RDP into host 1 to the RDP jump to host 2 and companies are just wasting money on latency.
So much this at my job, login to my laptop, login to vpn using password and 2-factor, check out my admin credentials using my login and 2-factor, login to the jump box using my admin creds and a different 2-factor, finally login to the system I need to be on using my admin creds. Multiply the last step by however many systems I need to connect to. Also, the clipboard and screen resolution are going to get messed up along the way no matter how much you mess with the settings.
So much this at my job, login to my laptop, login to vpn using password and 2-factor, check out my admin credentials using my login and 2-factor, login to the jump box using my admin creds and a different 2-factor, finally login to the system I need to be on using my admin creds. Multiply the last step by however many systems I need to connect to. Also, the clipboard and screen resolution are going to get messed up along the way no matter how much you mess with the settings.
Oh yeah, been there. I personally find Windows to be a terrible server OS. I'd rather be stuck with a fleet of AIX machines or something equally uncommon (but still nix-ish) than any Windows system.
The reverse also applies, where people that like Linux just totally ignore Windows maintenance. They’ll look down their noses at it and say that it’s insecure when they spend their time patching Linux and leaving Windows to fend for itself against attackers.
Someone must have made that "smart" decision to use servers with a proprietary OS, so that someone surely has paid the extra cost for MS support. No need to worry then.
Windows Updates are free.
I am not talking about Windows updates here. But even if I was, there are things wrong with it. One example, that frequently bites me on my gaming OS:
Windows 10 by itself decides: "Hey, I'm going to replace your installed graphics card driver with some other version, that I will find myself online!" Then next time I start this OS, I get crashing graphics card driver and need to go and look on AMD website for the correct drivers, installing them over whatever crap Windows installed. This has happened at least 3 times already and it is pissing me off. It would probably be better to deactivate automatic Windows updates completely and then try to have a manual selection of installed components. But even then I could not be sure, that it will not try to replace my installed drivers with crap.
Windows 10 by itself decides: "Hey, I'm going to replace your installed graphics card driver with some other version, that I will find myself online!" Then next time I start this OS, I get crashing graphics card driver and need to go and look on AMD website for the correct drivers, installing them over whatever crap Windows installed. This has happened at least 3 times already and it is pissing me off. It would probably be better to deactivate automatic Windows updates completely and then try to have a manual selection of installed components. But even then I could not be sure, that it will not try to replace my installed drivers with crap.
>and need to go and look on AMD website for the correct drivers
That's why nvidia has an app for that.
That's why nvidia has an app for that.
I think AMD has that too in its app, but if your graphics card driver is crashing in windows, your screen is freezing. Windows might try to restart the driver or so, getting the screen working briefly, but then it crashes again and ultimately it fails completely, you can only press the power button on your machine then.
I once had a project to turn a customer’s Excel VBA application into a real application (I used ASP.Net). He had been hacking on this Excel spreadsheet for like 15 years. Once I printed the VBA code because it was hard to navigate and it was like 250+ pages printed out rather compactly.
The worse part wasn’t the code itself (although it was bad), but the fact that there was so much abandoned paths in it and there would be three or more different versions of crucial functions with the same name (or sometimes a different name but doing the same thing) in different places and sometimes all being called by different paths in the workflow. Or not being called at all.
And it was very math heavy (calculating natural gas pressures and flow rates through different sized pipes and fittings and sizing regulators to meet certain loads). Think Excel formulas on cells that referenced 15-20 other cells, each of which was a formula on their own that referenced other pages and cells, some of which were filled by VBA. And that wasn’t even involving the VBA code full of brute force solvers for multi-variable equations that used heuristics he’d worked out by trial and error (if it’s a delta over 1.5, add 5 to this variable, otherwise subtract 3, but if the delta was less than 0.5, add 1 and so on - it eventually converged or found no solution, but a binary solved did the same thing, only faster and easier).
It took me and a junior developer several months, during which, of course, multiple change requests were going through to make it multiuser and secure.
Both my nightmare project and one that I’m very proud of once it was completed.
The worse part wasn’t the code itself (although it was bad), but the fact that there was so much abandoned paths in it and there would be three or more different versions of crucial functions with the same name (or sometimes a different name but doing the same thing) in different places and sometimes all being called by different paths in the workflow. Or not being called at all.
And it was very math heavy (calculating natural gas pressures and flow rates through different sized pipes and fittings and sizing regulators to meet certain loads). Think Excel formulas on cells that referenced 15-20 other cells, each of which was a formula on their own that referenced other pages and cells, some of which were filled by VBA. And that wasn’t even involving the VBA code full of brute force solvers for multi-variable equations that used heuristics he’d worked out by trial and error (if it’s a delta over 1.5, add 5 to this variable, otherwise subtract 3, but if the delta was less than 0.5, add 1 and so on - it eventually converged or found no solution, but a binary solved did the same thing, only faster and easier).
It took me and a junior developer several months, during which, of course, multiple change requests were going through to make it multiuser and secure.
Both my nightmare project and one that I’m very proud of once it was completed.
By the end of the first sentence I thought “that sounds like engineering” (the real kind). I know people still dealing with this kind of thing!
That's more adventurous than all the Excel projects I've been given. Many of the most satisfying solutions are the most simple that nobody's implemented yet.
Like, at a job where all the lead devs and, apparently, the whole internet agreed that you can't do proper source control for Excel because you can't export-and-import code for all modules (including sheets & ThisWorkbook) without copy-paste and because a running module can't replace itself. The solution ending up so simple, that I was embarrassed to hear it called "a stroke of genius". I still have that code somewhere.
Like, at a job where all the lead devs and, apparently, the whole internet agreed that you can't do proper source control for Excel because you can't export-and-import code for all modules (including sheets & ThisWorkbook) without copy-paste and because a running module can't replace itself. The solution ending up so simple, that I was embarrassed to hear it called "a stroke of genius". I still have that code somewhere.
The ending is pure gold. Some of the best times in my career were working on a codebase for an application serving folks I knew on a first name basis and had had lunch with.
I could talk through pain points they were having, we’d come up with a solution together, I’d hack up a quick prototype and launch it just to them to try out. We’d tweak it over a couple of weeks and when it was good I’d launch it to all customers.
Messy, ugly code base. But it worked well because it wasn’t over-managed. Just developers doing development. Amazing what happens when you get out of the way of smart, talented people and let them do their work.
I could talk through pain points they were having, we’d come up with a solution together, I’d hack up a quick prototype and launch it just to them to try out. We’d tweak it over a couple of weeks and when it was good I’d launch it to all customers.
Messy, ugly code base. But it worked well because it wasn’t over-managed. Just developers doing development. Amazing what happens when you get out of the way of smart, talented people and let them do their work.
The key here is to put the devs and users together.
... and get them talking directly to the user. I feel that's where the real magic happens.
Surely it is much more efficient for a PM to ask all the wrong questions and relay the answers using totally different words to the developers. As many many companies love hiring tons of PMs, this is surely the optimal system
When you actually understand the problem you are solving, and the users you solve it for you start to care about a solution and not just the technology.
So many “Oh, I just didn’t ask because I thought it would take months” saved!
[Deleted]
I built a system as horrible as this. One of my many terrible inventions was as follows:
Originally our company only did business in one marketplace (the UK). When we started doing business in multiple marketplaces, it came time to modify the system to cope with more than one. Our system assumed, _everywhere_, that there was only ever one marketplace. I had a plan: I would copy-paste the system, make an "international" version that supported many marketplaces, then transition over our original marketplace to the international version and shut down the old system. This way, everyone could keep working on the original marketplace like normal, they'd get the new marketplaces on the new system, and we'd do a clean cutover once ready.
It started out quite well. I got the international version working and onboarded our first new marketplace. The business was very happy, there was lots of opportunity in the new marketplaces. They asked that I delay the cutover from the old system and focus on developing things for these new marketplaces. After all, the old system still works for our original marketplace, we can move it over once our work is done on the new marketplaces. I said yes, of course!
It's now 5 years later, it turns out there's a lot to do in those other marketplaces. To say that the system is split-brained is an understatement. When training new hires, we teach them the difference between a "product" and an "international product". When either system does something, it double checks and cross-references with the other system via a colourful assortment of HTTP APIs. There are a variety of database tables that we "froze" in the old system and continued in the new system, so you could tell that an ID referred to something in the new system or the old system if it was above or below the magic number we froze it at. We have overarching BI dashboards that query both systems to produce results, and they have to heavily manipulate the old system's data to fit with the new multi-marketplace model, and any other changes we made. Both systems are extremely tightly coupled, the old one is a ball and chain to the new one, but the new one is so tightly integrated into it that there's no hope of separating them now.
I've learned to say no since then.
Originally our company only did business in one marketplace (the UK). When we started doing business in multiple marketplaces, it came time to modify the system to cope with more than one. Our system assumed, _everywhere_, that there was only ever one marketplace. I had a plan: I would copy-paste the system, make an "international" version that supported many marketplaces, then transition over our original marketplace to the international version and shut down the old system. This way, everyone could keep working on the original marketplace like normal, they'd get the new marketplaces on the new system, and we'd do a clean cutover once ready.
It started out quite well. I got the international version working and onboarded our first new marketplace. The business was very happy, there was lots of opportunity in the new marketplaces. They asked that I delay the cutover from the old system and focus on developing things for these new marketplaces. After all, the old system still works for our original marketplace, we can move it over once our work is done on the new marketplaces. I said yes, of course!
It's now 5 years later, it turns out there's a lot to do in those other marketplaces. To say that the system is split-brained is an understatement. When training new hires, we teach them the difference between a "product" and an "international product". When either system does something, it double checks and cross-references with the other system via a colourful assortment of HTTP APIs. There are a variety of database tables that we "froze" in the old system and continued in the new system, so you could tell that an ID referred to something in the new system or the old system if it was above or below the magic number we froze it at. We have overarching BI dashboards that query both systems to produce results, and they have to heavily manipulate the old system's data to fit with the new multi-marketplace model, and any other changes we made. Both systems are extremely tightly coupled, the old one is a ball and chain to the new one, but the new one is so tightly integrated into it that there's no hope of separating them now.
I've learned to say no since then.
> All that remained were ragtag interns and junior developers.
For many people, their first job in software engineering is the worst codebase they will deal with professionally for this reason. The first job hires lot of people with little/no experience. As soon as someone gains some experience than can move on to better paying jobs, where there are better developers with better standards.
For many people, their first job in software engineering is the worst codebase they will deal with professionally for this reason. The first job hires lot of people with little/no experience. As soon as someone gains some experience than can move on to better paying jobs, where there are better developers with better standards.
Worst code bases are often ones taken over from IT consultancies. They drive young, inexperienced developers working many hours to deliver functionality. While the project may start out “clean” using whatever is the current hotness in technology, at some point getting stuff developed and throwing over the wall to QA is the important part.
This is the exact boat I am in. I always say the best thing about our codebase is the worst thing: junior developers can do whatever they want.
Back about 15 years ago, I worked for a web hosting company that provided some sysadmin consultation services. Customer paid us, and I would take a look.
I had one customer who came back with the same request, slightly differently worded, every single month, and every single month I'd say the same thing. They had this site they were running that was essentially a Yellow Pages type site. They had a large set of companies with contact details, each with multiple business categories associated with it. You'd choose a category, and they'd return a list of matching companies.
The problem was the site was really slow. I took a quick look around, and saw that all the time was lost querying the database. Taking a quick look at the schema I discovered that their approach to categorisation was to have a TEXT column, with semicolon separated 4 character strings in it. Each 4 character string mapped to a business category.
So when someone wanted to load up, say, all pest control companies, it would check the category mapping table, get the 4 character string, and then go to the companies table and do:
I guess that's probably okay for the developer without real world scale data, and real world traffic counts to worry about. But they had lots of data in the database, and that category field could have dozens of categories against a company. As soon as they had more than about 4-5 simultaneous customers performance started tanking.
I could never get them to accept that they needed to rethink the database schema. One month they were bleating about how is it possible that Google can manage to do such a search across a much larger amount of data, much faster. They really didn't like my answer that amounted to "By having a sane database schema". All they were willing to do was pay over the odds for our most powerful server at the time, which had enough capacity to hold the entire database in memory.
I had one customer who came back with the same request, slightly differently worded, every single month, and every single month I'd say the same thing. They had this site they were running that was essentially a Yellow Pages type site. They had a large set of companies with contact details, each with multiple business categories associated with it. You'd choose a category, and they'd return a list of matching companies.
The problem was the site was really slow. I took a quick look around, and saw that all the time was lost querying the database. Taking a quick look at the schema I discovered that their approach to categorisation was to have a TEXT column, with semicolon separated 4 character strings in it. Each 4 character string mapped to a business category.
So when someone wanted to load up, say, all pest control companies, it would check the category mapping table, get the 4 character string, and then go to the companies table and do:
SELECT * FROM companies WHERE categories LIKE "%PEST%"
I guess that's probably okay for the developer without real world scale data, and real world traffic counts to worry about. But they had lots of data in the database, and that category field could have dozens of categories against a company. As soon as they had more than about 4-5 simultaneous customers performance started tanking.
I could never get them to accept that they needed to rethink the database schema. One month they were bleating about how is it possible that Google can manage to do such a search across a much larger amount of data, much faster. They really didn't like my answer that amounted to "By having a sane database schema". All they were willing to do was pay over the odds for our most powerful server at the time, which had enough capacity to hold the entire database in memory.
In case anyone is looking for a performant way to implement categories like that in Postgres: https://news.ycombinator.com/item?id=33251745
I stumbled across that comment a few years back and it changed the way I handle tags and categories so just sharing it here. If anyone has an equivalent for Sqlite, I’d love to hear it!
I stumbled across that comment a few years back and it changed the way I handle tags and categories so just sharing it here. If anyone has an equivalent for Sqlite, I’d love to hear it!
postgres also supports gin indexes, so could do int arrays & contains queries, for example.
For online Poker, one of the current database tools runs in to this issue - PokerTracker 4 (PT4).
These tracker databases are usually used to generate a HUD - numerical readouts surrounding each player shown on the table. PT allows for custom HUDs to be saved/exported/shared, and there is a cottage industry building and selling such HUDS. These HUDs can often bundle hundreds of custom stats - basically a subset of SQL queries, a simple example would be "times_raised / opportunities_to_raise WHERE position= 'button'", that sort of thing.
For performance reasons these custom stats are cached, which obviously makes some sense. However, each cached stat creates a new column in the custom_cache table of PT4's current database, a PostgreSQL 9.0 backend. If you play mostly Heads-Up SNG or the 3-handed Spins, it's actually quite easy to go over the (IIRC) 4096 column limit there by purchasing and importing one too many fancy HU HUD packages!
This completely borks PT4, it can no longer open - and players can no longer make money! In my previous work, I've supported many a player and fixed this "too many columns" error numerous times (by deleting enough custom HUDs from their account until the PT4 can start up correctly once more). Funny to see this pop up elsewhere!
These tracker databases are usually used to generate a HUD - numerical readouts surrounding each player shown on the table. PT allows for custom HUDs to be saved/exported/shared, and there is a cottage industry building and selling such HUDS. These HUDs can often bundle hundreds of custom stats - basically a subset of SQL queries, a simple example would be "times_raised / opportunities_to_raise WHERE position= 'button'", that sort of thing.
For performance reasons these custom stats are cached, which obviously makes some sense. However, each cached stat creates a new column in the custom_cache table of PT4's current database, a PostgreSQL 9.0 backend. If you play mostly Heads-Up SNG or the 3-handed Spins, it's actually quite easy to go over the (IIRC) 4096 column limit there by purchasing and importing one too many fancy HU HUD packages!
This completely borks PT4, it can no longer open - and players can no longer make money! In my previous work, I've supported many a player and fixed this "too many columns" error numerous times (by deleting enough custom HUDs from their account until the PT4 can start up correctly once more). Funny to see this pop up elsewhere!
Would be great to work in such a company as a Linux guru.
There are so many entangled services and machines that you feel like an Indiana Jones. You ssh into a machine and feel century-old dust beneath your footsteps. And you never know what will you find. Maybe a service which holds the company together. Maybe a CPU eating hog which didn't do anything useful last 3 years.
I don't enjoy writing new code much. But in such an environment even with my limited skills I can do decent improvements, especially from security point of view. Feels great
There are so many entangled services and machines that you feel like an Indiana Jones. You ssh into a machine and feel century-old dust beneath your footsteps. And you never know what will you find. Maybe a service which holds the company together. Maybe a CPU eating hog which didn't do anything useful last 3 years.
I don't enjoy writing new code much. But in such an environment even with my limited skills I can do decent improvements, especially from security point of view. Feels great
When I started my recent job the team kept referring to a box running "weird linux".
After getting on-boarded and accessing the box it turned out to be running a very old version of OpenBSD.
To this day I'm curious who had the wherewithall to install OpenBSD in prod but was seemingly ignorant of the 1yr support cycle.
I started reading and was hoping someone had installed one of the Linux distros that were popular at the turn of the millennium, like Mandrake, Slackware, etc. and it was still trucking along 25 years later.
In 2010 or 2011, deviantArt was running on several hundred slackware servers. One of my projects at that job was converting everything to run on Debian with config management in Puppet to replace all of the old manually configured services.
Thats a name I haven't heard for a while - I was obsessed with scouring deviantart for cool wallpapers during my very early Linux days of desktop tinkering (mostly fluxbox and gkrellm tweaking).
Oldest boxes running are RHEL5 and Win Server 2008.
I was trying to explain to the IT Director that both were released when I was in elementary school so I'd have to brush up on them.
I'm guilty of something like that. At my very first job I used OpenBSD to manage the corporate firewall and proxy. 6 months later quit to attend college, and I have no idea what they did to that box lol.
Sounds like me, I left a number of openbsd machines at a previous shop I worked at. I kept them updated, but based on the bewildering mishmash of linux distros and versions, there were a surprising number of sco boxes hanging around as well, their general philosophy was to set a box up for a task then never update it afterwards. So I expect all my obsd boxes are still there at exactly the same version I left them at.
Basically the entire application is loaded with low hanging fruits and things that look like fruits but are bombs that explode upon contact.
Well it sounded like there are exactly 0 Linux machines running there.. it's all windows .net / c# and a bunch of native windows apps, as I understood the article.
But maybe you can replace your statement with "windows guru" and SSH with "remote desktop" and perhaps that would be fun
But maybe you can replace your statement with "windows guru" and SSH with "remote desktop" and perhaps that would be fun
well, with linux at least I get get the source code of the kernel and of the MySQL when given a database server which hasn't been restarted for 7 years
This made me think of my first job. I was the sole developer on a project because the old developer left. Nothing was documented and nobody knew why things were designed the way they were.
We had no code reviews, no design docs, no tests, nothing. We made the changes the way we thought they were right and would git pull them onto the production server.
After I struggled to get productive for the first four months, my manager went on a four-week Christmas vacation. In a moment of frustration, I seized the opportunity and rewrote the whole project from scratch. I don’t remember if my manager ever noticed, but that was the moment I finally got productive.
We had no code reviews, no design docs, no tests, nothing. We made the changes the way we thought they were right and would git pull them onto the production server.
After I struggled to get productive for the first four months, my manager went on a four-week Christmas vacation. In a moment of frustration, I seized the opportunity and rewrote the whole project from scratch. I don’t remember if my manager ever noticed, but that was the moment I finally got productive.
> my manager went on a four-week Christmas vacation.
Is that kind of stuff common? People checking out on Black Friday and coming back for New Year's?
Is that kind of stuff common? People checking out on Black Friday and coming back for New Year's?
In Europe totally, and for example in Germany it's customary to get you six week vacation in the summer.
This also has the benefit that the workplace has to have real back-up person for all matters, as six weeks is too long to shove everything under the carpet waiting for your return.
This also has the benefit that the workplace has to have real back-up person for all matters, as six weeks is too long to shove everything under the carpet waiting for your return.
As we all should. Life is too short to be surprised about not working for a month.
>Is that kind of stuff common? People checking out on Black Friday and coming back for New Year's?
Happens the further up you get in management, especially if you are in an industry that already takes a long break around christmas/newyears.
Happens the further up you get in management, especially if you are in an industry that already takes a long break around christmas/newyears.
I bet that’s Europe, it’s not uncommon
Yess, we were the software team for a university library. There wasn't a lot of pressure and people on my team generally had a chill and comfortable life.
However, there wasn't a lot of room for ambition.
However, there wasn't a lot of room for ambition.
> I miss that direct connection. The fast feedback. The lack of making grand plans.
There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated?
But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do.
Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff.
These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...
There's no date on this article, but it feels "prior to the MongoDB-is-webscale memes" and thus slightly outdated?
But, hey, I get where they're coming from. Personally, I used to be very much schema-first, make sure the data makes sense before even thinking about coding. Carefully deciding whether to use an INT data type where a BYTE would do.
Then, it turned out that large swathes of my beautiful, perfect schemas remained unoccupied, while some clusters were heavily abused to store completely unrelated stuff.
These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Then, some indexes specified by `json_extract` expressions, some clever NULL coalescing in the consuming code, resulting in a generally-better experience than before...
> These days, my go-to solution is SQLite with two fields (well, three, if you count the implicit ROWID, which is invaluable for paging!): ID and Data, the latter being a JSONB blob.
Really!? Are you building applications by chance or something else? Are you doing raw sql mostly or an ORM/ORM-like library? This surprises me because my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks. SQLite's fluid type system haa been a nice middle ground for me personally. For reference my application layer is kysely/typescript.
Really!? Are you building applications by chance or something else? Are you doing raw sql mostly or an ORM/ORM-like library? This surprises me because my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks. SQLite's fluid type system haa been a nice middle ground for me personally. For reference my application layer is kysely/typescript.
> my experience dabbling in json fields for CRUD apps has been mostly trouble stemming from the lack of typechecks
Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
Practical example: I recently wrote my own SMTP server (bad idea!), mostly to be able to control spam (even worse idea! don't listen to me!). Initially, I thought I would be really interested in remote IPs, reverse DNS domains, and whatever was claimed in the (E)HLO.
So, I designed my initial database around those concepts. Turns out, after like half a million session records: I'm much more interested in things like the Azure tenant ID, the Google 'groups' ID, the HTML body tag fingerprint, and other data points.
Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent, but fortunately modern C# makes that easy as well.
And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
Practical example: I recently wrote my own SMTP server (bad idea!), mostly to be able to control spam (even worse idea! don't listen to me!). Initially, I thought I would be really interested in remote IPs, reverse DNS domains, and whatever was claimed in the (E)HLO.
So, I designed my initial database around those concepts. Turns out, after like half a million session records: I'm much more interested in things like the Azure tenant ID, the Google 'groups' ID, the HTML body tag fingerprint, and other data points.
Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent, but fortunately modern C# makes that easy as well.
And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
> Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations. And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
Our solution for a similar situation involving semi-structured data (in postgres) was to double it up: put all the json we send/receive with a vendor into a json field, then anything we actually need to work on gets extracted into regular table/columns. We get all the safety/performance guarantees the database would normally give us, plus historical data for debugging or to extract into a new column if we now need it. The one thing we had to monitor in code reviews was to never use the json field directly for functionality.
Our solution for a similar situation involving semi-structured data (in postgres) was to double it up: put all the json we send/receive with a vendor into a json field, then anything we actually need to work on gets extracted into regular table/columns. We get all the safety/performance guarantees the database would normally give us, plus historical data for debugging or to extract into a new column if we now need it. The one thing we had to monitor in code reviews was to never use the json field directly for functionality.
This is exactly what I've tried (and failed at) doing! Can I ask how you handle normalization from vendor data when it contains relationships and multilevel nesting? How do you know when to create a new child table, and which ones to create, and their relationships etc. I haven't found a good balance yet.
Basically what the other reply said - handle it the same as you would any complex data. You just don't need to handle all of the json immediately, only the parts you plan on using for the moment.
Odd-shaped miscellaneous data that you only need to retrieve is a good candidate for a JSON field. Once you're heavily using some piece of data, or if you need to index it (which means you are heavily using it), you should insert the data in the database "properly".
If some vendor is giving you a list of categories you don't care about, there's no need to make a vendor categories table and a many-to-many link table until you actually need them.
The point is that putting data properly in the database lets you use database features on it and get database performance.
If some vendor is giving you a list of categories you don't care about, there's no need to make a vendor categories table and a many-to-many link table until you actually need them.
The point is that putting data properly in the database lets you use database features on it and get database performance.
> Well, you move the type checks from the database to the app, effectively, which is not a new idea by any means (and a bad idea in many cases), but with JSON, it can actually work out nicely-ish, as long as there are no significant relationships between tables.
That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Also I don't understand why you're afraid of migrations, especially since you're the only developer on your own SMTP server.
That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Also I don't understand why you're afraid of migrations, especially since you're the only developer on your own SMTP server.
> Has it occurred to you that MongoDB exists?
My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""
So, care to take another guess? And, while we're here, does MongoDB run fully in-process these days? And/or allow easy pagination by ROWID?
My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""
So, care to take another guess? And, while we're here, does MongoDB run fully in-process these days? And/or allow easy pagination by ROWID?
> My original comment started with "but it feels "prior to the MongoDB-is-webscale memes""
Which feels off by six generations or so of memes. It feels prior to "memes" existing in the first place (at least this modern iteration, pics with captions in them; not Dawkins' original verion). I'd guess it is, ironically, chronologically closer to, well, your username here.
Which feels off by six generations or so of memes. It feels prior to "memes" existing in the first place (at least this modern iteration, pics with captions in them; not Dawkins' original verion). I'd guess it is, ironically, chronologically closer to, well, your username here.
> Has it occurred to you that MongoDB exists?
What gain would MongoDB offer here?
You certainly would lose a lot of things, like a well supported path to linking with to the database engine, and a straightforward way to start to introduce relational tables as the project matures. Nothing completely insurmountable, of course, but carry a lot of extra effort for what benefit?
What gain would MongoDB offer here?
You certainly would lose a lot of things, like a well supported path to linking with to the database engine, and a straightforward way to start to introduce relational tables as the project matures. Nothing completely insurmountable, of course, but carry a lot of extra effort for what benefit?
How does MongoDB handle someone pulling the power cord out of the server? Because that’s another reason to use something like SQLite, and it often gets used in embedded systems.
> That way you're throwing away 50% of the reason you use a relational database in the first place. Has it occurred to you that MongoDB exists?
Did you miss that he’s using sqlite? The dev experience with a sqlitedb is way better than running yet another service, especially for personal projects.
Sqlite is used just as much as an application file format as it is a relational database.
Did you miss that he’s using sqlite? The dev experience with a sqlitedb is way better than running yet another service, especially for personal projects.
Sqlite is used just as much as an application file format as it is a relational database.
> Fortunately, my session database is just 'JSON(B) in a single table', so I was able to add those additional fields without the need for any migrations.
> And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
That's a migration.
> Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent
That's an effect of not migrating - having to process null and absent fields instead of just null fields. After doing more of these, you'll run into the same thing that made people stop using NoSQL databases: with no schema, your code has to parse all previous versions of the data format and they probably aren't even well-documented. While an RDBMS can just set the new column to null in existing rows.
> And SQLite's `json_extract` makes adding indexes after-the-fact super-easy.
That's a migration.
> Of course, these additional fields need to be explicitly nullable, and I need to skip processing based on them if they're absent
That's an effect of not migrating - having to process null and absent fields instead of just null fields. After doing more of these, you'll run into the same thing that made people stop using NoSQL databases: with no schema, your code has to parse all previous versions of the data format and they probably aren't even well-documented. While an RDBMS can just set the new column to null in existing rows.
> And, no, no need for an ORM, except `JsonSerializer.Deserialize<T>`... (And yeah, all of this is just a horrible hack, but one that seems surprisingly resilient so far, but YMMV)
I do the same thing with serde_json in Rust for a desktop app sqlitedb and it works great so +1 on that technique.
In Rust you can also tell serde to ignore unknown fields and use individual view structs to deserialize part of the JSON instead of the whole thing and use string references to make it zero copy.
I do the same thing with serde_json in Rust for a desktop app sqlitedb and it works great so +1 on that technique.
In Rust you can also tell serde to ignore unknown fields and use individual view structs to deserialize part of the JSON instead of the whole thing and use string references to make it zero copy.
“Without the need for any migrations” seems like a weird one to me. Of all of the things I don’t like having to do, migrations are pretty low on my list - far below any scripting-in-prod or applying hacks or even just rudimentary business logic changes. Granted, I used to fear them a lot back when I was expected to apply them by hand rather than use CICD and testing to validate/apply them
> I recently wrote my own SMTP server (bad idea!)
Recently I’ve been tempted to make an SMTP server that translates emails into a web-hook. Please tell me more horror stories so that I might be convinced not do it.
Recently I’ve been tempted to make an SMTP server that translates emails into a web-hook. Please tell me more horror stories so that I might be convinced not do it.
[deleted]
In my current company, we're using a similar approach: just shove everything into a JSON blob. If you need a constraint or an index, you can create a computed column (in PostgreSQL) that pulls out a field from JSON.
For the data schema, we're using Protobufs with buf validate. This works surprisingly well, you can use the same types in the backend API and on the frontend. We even have a cron job that reads all the data periodically and verifies that the JSON blobs conform to the schema. Our next intern is going to write a PostgreSQL extension to do that on commit/update :)
One real advantage of this approach is that it's easy to do stuff like "search everywhere".
For the data schema, we're using Protobufs with buf validate. This works surprisingly well, you can use the same types in the backend API and on the frontend. We even have a cron job that reads all the data periodically and verifies that the JSON blobs conform to the schema. Our next intern is going to write a PostgreSQL extension to do that on commit/update :)
One real advantage of this approach is that it's easy to do stuff like "search everywhere".
Do you go to the trouble of updating individual values in objects using some kind of deep/partial updating function or do you just accept race conditions that come with updating full objects?
We use optimistic versioning, with a dedicated "version" field (that is actually always pulled out of the blob in all tables).
Classic fine-grained schemas are not that much different. A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.
In addition, our frontend apps also support offline mode. They can get all the relevant objects, and then operate on them locally. So our API was designed from the start to deal with conflicts.
Classic fine-grained schemas are not that much different. A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.
In addition, our frontend apps also support offline mode. They can get all the relevant objects, and then operate on them locally. So our API was designed from the start to deal with conflicts.
> We use optimistic versioning, with a dedicated "version" field (that is actually always pulled out of the blob in all tables).
All well and good but you do need to handle failures elegantly in code. The nice thing about flat DB tables and SQL is you don't really have to care if your goal is to update a single column authoritatively. The state of the other values in the table are, often, immaterial. It gets even more complicated reconciling deeply nested conflicts in open schemas.
Not knocking your approach, it's just a trade-off I guess.
All well and good but you do need to handle failures elegantly in code. The nice thing about flat DB tables and SQL is you don't really have to care if your goal is to update a single column authoritatively. The state of the other values in the table are, often, immaterial. It gets even more complicated reconciling deeply nested conflicts in open schemas.
Not knocking your approach, it's just a trade-off I guess.
> A lot of high-level ORM frameworks simply save all the objects' fields on update, without doing fine-grained diffs.
Perhaps that is true... I'm a Rails dev and ActiveRecord definitely tracks mutations on the object and selectively updates.
I would assume this is consistent with Django and I'd assume a policy is probably required to be specified by SQLAlchemy.
Perhaps that is true... I'm a Rails dev and ActiveRecord definitely tracks mutations on the object and selectively updates.
I would assume this is consistent with Django and I'd assume a policy is probably required to be specified by SQLAlchemy.
A lot of databases include functions for manipulating JSON which can be used for atomic updates. SQLite has JSON_PATCH(), for instance.
Yeah, makes sense. To write safe code though at that point don't you have to enforce rigid schemas?
An example would be updating Kubernetes resources, the admission controller will verify the correctness of the change you are trying to make and reject the patch if it's non-comformant. Nested values have... value... in terms of contextualizing and isolating/localizing data leaves but at the end of the day aren't you still dealing with strict schemas and, when those schemas change you have to reconcile schema migrations?
An example would be updating Kubernetes resources, the admission controller will verify the correctness of the change you are trying to make and reject the patch if it's non-comformant. Nested values have... value... in terms of contextualizing and isolating/localizing data leaves but at the end of the day aren't you still dealing with strict schemas and, when those schemas change you have to reconcile schema migrations?
> To write safe code though at that point don't you have to enforce rigid schemas?
Certainly. But you have to do that with SQLite anyway, because (by default, without strict mode [1] enabled) it won't stop you from putting the wrong type of data into a column.
[1]: https://www.sqlite.org/stricttables.html
Certainly. But you have to do that with SQLite anyway, because (by default, without strict mode [1] enabled) it won't stop you from putting the wrong type of data into a column.
[1]: https://www.sqlite.org/stricttables.html
I've worked on a system were a guy "migrated" a database from NoSQL to SQL as a proof-of-concept. Except he really didn't migrate anything. He basically created tables with "ID" (UUID) and "data" (json) columns everywhere. No indexes.
The actual "data" is a mess: different data types for the same JSON field. Imagine storing "price" as both a float and a string, depending on whatever buggy code happened to do the inserts.
It worked enough for a prototype and that was enough for management to believe the project was a success. I can't wait until we actually try and deploy it.
The actual "data" is a mess: different data types for the same JSON field. Imagine storing "price" as both a float and a string, depending on whatever buggy code happened to do the inserts.
It worked enough for a prototype and that was enough for management to believe the project was a success. I can't wait until we actually try and deploy it.
I’m still in the think hard about the schema camp. I like to rely on the database to enforce constraints.
Yeah, a good database is pretty damn handy.
Have you had the pleasure of blowing young minds by revealing that production-grade databases come with fully fledged authnz systems that you can just...use right out of the box?
Have you had the pleasure of blowing young minds by revealing that production-grade databases come with fully fledged authnz systems that you can just...use right out of the box?
Can you say more? I’m interested.
Databases have pretty robust access controls to limit (a sql user's) access to tables, schemas, etc. Basic controls like being able to read but not write, and more advanced situations like being able to access data through a view or stored procedure without having direct access to the underlying tables.
Those features aren't used often in modern app development where one app owns the database and any external access is routed through an API. They were much more commonly used in old school apps enterprise apps where many different teams and apps would all directly access a single db.
Those features aren't used often in modern app development where one app owns the database and any external access is routed through an API. They were much more commonly used in old school apps enterprise apps where many different teams and apps would all directly access a single db.
I think supabase leans quite heavily into this, although I haven’t used it myself.
Row level security has been wonderful for multi tenancy in my experience though. I would highly recommend it.
Row level security has been wonderful for multi tenancy in my experience though. I would highly recommend it.
I guess they mean something like Postgres' row-level security:
https://www.postgresql.org/docs/current/ddl-rowsecurity.html
https://www.postgresql.org/docs/current/ddl-rowsecurity.html
Here's a vendor, but the underlying functionality is shipped with Postgresql server
https://supabase.com/docs/guides/database/postgres/row-level...
https://supabase.com/docs/guides/database/postgres/row-level...
This is perfectly fine when you are driving some app that has a per-user experience that allows you to wrap up most of their experience in some blobs.
However I would still advise people to use a third normal form - they help you, constraints help you, and often other sets of tooling have poor support for constraints on JSON. Scanning and updating every value because you need to update some subset sucks.
You first point is super valid though - understanding the domain is very useful and you can get easily 10x the performance by designing with proper types involved, but importantly don't just build out the model before devs and customers have a use for anything, this is a classic mistake in my eyes (and then skipping cleanup when that is basically unused.)
If you want to figure out your data model in depth beforehand there's nothing wrong with that... but you will still make tons of mistakes mistakes, lack of planning will require last minute fixes, and the evolution of the product will have your original planning gather dust.
However I would still advise people to use a third normal form - they help you, constraints help you, and often other sets of tooling have poor support for constraints on JSON. Scanning and updating every value because you need to update some subset sucks.
You first point is super valid though - understanding the domain is very useful and you can get easily 10x the performance by designing with proper types involved, but importantly don't just build out the model before devs and customers have a use for anything, this is a classic mistake in my eyes (and then skipping cleanup when that is basically unused.)
If you want to figure out your data model in depth beforehand there's nothing wrong with that... but you will still make tons of mistakes mistakes, lack of planning will require last minute fixes, and the evolution of the product will have your original planning gather dust.
> Scanning and updating every value because you need to update some subset sucks.
Mirrors my experience exactly. Querying json can get complex to get info from the db. SQLite is kind of forgiving because sequences of queries (I mean query, modify in appliation code that fully supports json ie js, then query again) are less painful meaning it's less moprtant to do everytning in the database for performance reasons. But if you're trying to do everything in 1 query, I think you pay for it at application-writing time over and over.
Mirrors my experience exactly. Querying json can get complex to get info from the db. SQLite is kind of forgiving because sequences of queries (I mean query, modify in appliation code that fully supports json ie js, then query again) are less painful meaning it's less moprtant to do everytning in the database for performance reasons. But if you're trying to do everything in 1 query, I think you pay for it at application-writing time over and over.
Oh good question on date essay was written -- put dates on your things on the internet people!
Internet Archive has a crawl from today but no earlier; which doesn't mean it can't be earlier of course. My guess is it was written recently though.
Internet Archive has a crawl from today but no earlier; which doesn't mean it can't be earlier of course. My guess is it was written recently though.
created 14 hours ago https://github.com/jimmyhmiller/jimmyhmiller.github.io/commi...
But clearly in retrospect. It sounds like some of the things I was doing in 2009.
It includes a reference to Backbone and Knockout JS, which were released in 2010, so presumably it was around that era. The database, though, was probably much older...
Touche. It was "unnatural things to work around sql server's limitations" that rung a bell.
At least we know it wasn't written after today!
[deleted]
Byte vs. Int is premature optimization. But indexing, primary keys, join tables, normalization vs. denormalization, etc. are all important.
> Byte vs. Int is premature optimization
I think you can only judge that by knowing the context, like the domain and the experience of the designer/dev within that domain.
I looked at a DB once and thought "why are you spending effort to create these datatypes that use less storage, I'm used to just using an int and moving on."
Then I looked at the volumes of transactions they were dealing with and I understood why.
I think you can only judge that by knowing the context, like the domain and the experience of the designer/dev within that domain.
I looked at a DB once and thought "why are you spending effort to create these datatypes that use less storage, I'm used to just using an int and moving on."
Then I looked at the volumes of transactions they were dealing with and I understood why.
Deep down, the optimizer in me wants this to be true, but I'm having trouble seeing how this difference manifests in these days of super powerful devices and high bandwidth.
I guess I just answered my own question though. Supposing there's a system which is slow and connected with very slow connectivity and still sending lots of data around, I guess there's your answer. An embedded system on the Mars Rover or something.
I guess I just answered my own question though. Supposing there's a system which is slow and connected with very slow connectivity and still sending lots of data around, I guess there's your answer. An embedded system on the Mars Rover or something.
Absolutely
I actually love your approach and haven’t thought of that before.
My problem with relational databases often stems from the fact that remodeling data types and schemas (which you often do as you build an application, whether or not you thought of a great schema beforehand) often comes with a lot of migration effort.
Pairing your approach with a „version“ field where you can check which version of a schema this rows data is saved with would actually allow you to be incredibly flexible with saving your data while also being able to be (somewhat) sure that your fields schema matches what you’re expecting.
Pairing your approach with a „version“ field where you can check which version of a schema this rows data is saved with would actually allow you to be incredibly flexible with saving your data while also being able to be (somewhat) sure that your fields schema matches what you’re expecting.
Having to write and perform migrations for every small schema change is a bore, but it means your software doesn't have to worry about handling different versions of data. Going "schemaless" with version numbers means moving code from "write-and-forget" migrations to the main codebase, where it will live forever.
I think not doing database migrations only makes sense when you can make do without version numbers (or if you can't do atomic migrations due to performance constraints, but that's only a problem for a very small number of projects).
I think not doing database migrations only makes sense when you can make do without version numbers (or if you can't do atomic migrations due to performance constraints, but that's only a problem for a very small number of projects).
You’re correct there. I mostly work on CMSes with page builder functionality, which often bake the content schema into the database columns, which makes changing that schema (for new frontend features or reworking old ones) difficult and often prone to losing content, especially in dev environments.
Best case is obviously that you never have to version your changes, but I‘d prefer making a new schema and writing an adapter function in the codebase depending on the schemas version to spending a lot of time migrating old content. That might just be due to me not being too comfortable with SQL and databases generally.
> remodeling data types and schemas (which you often do as you build an application, whether or not you thought of a great schema beforehand)
This is not my experience, it only happens rarely. I’d like to see an analysis of what causes schema changes that require nontrivial migrations.
This is not my experience, it only happens rarely. I’d like to see an analysis of what causes schema changes that require nontrivial migrations.
Same here. If your entities are modelled mostly correctly you really don't have to worry about migrations that much. It's a bit of a red herring and convenient "problem" pushed by the NoSQL camp.
On a relatively neat and well modelled DB, large migrations are usually when relationships change. E.g. One to many becomes a many to many.
Really the biggest hurdle is managing the change control to ensure it aligns with you application. But that's a big problem with NoSQL DB deployments too.
At this point I don't even want to hear what kind of crazy magic and "weird default and fallback" behavior the schema less NoSQL crowd employs. My pessimistic take is they just expose the DB onto GraphQL and make it front ends problem.
On a relatively neat and well modelled DB, large migrations are usually when relationships change. E.g. One to many becomes a many to many.
Really the biggest hurdle is managing the change control to ensure it aligns with you application. But that's a big problem with NoSQL DB deployments too.
At this point I don't even want to hear what kind of crazy magic and "weird default and fallback" behavior the schema less NoSQL crowd employs. My pessimistic take is they just expose the DB onto GraphQL and make it front ends problem.
I agree. Migrations have never been a problem at my company and this codebase is 9 years old. Just regular old postgres with a few JSONB columns of things that don't need to be relational.
Same for the database that was about 8-9 years old at my last company. Migrations are fine. It's the relationship-changing that is painful.
Same for the database that was about 8-9 years old at my last company. Migrations are fine. It's the relationship-changing that is painful.
>> If your entities are modelled mostly correctly you really don't have to worry about migrations that much
I'm gonna take a wild guess here that you have never worked in unfamiliar domains (like lets say deep cargo shiping or subpremium loans) where your so called subject matter experts provided by client werent the sharpest people you could hope for and actually did not understand what they where doing for most of the time?
Because I on the other hand am very familiar with such projects and doing schema overhaul third time in a row for production system is bread and butter for me.
Schemaless systems is the only reason I'm still developer and not lumberjack.
I'm gonna take a wild guess here that you have never worked in unfamiliar domains (like lets say deep cargo shiping or subpremium loans) where your so called subject matter experts provided by client werent the sharpest people you could hope for and actually did not understand what they where doing for most of the time?
Because I on the other hand am very familiar with such projects and doing schema overhaul third time in a row for production system is bread and butter for me.
Schemaless systems is the only reason I'm still developer and not lumberjack.
I think they are referring to the fact that software development as a field has matured a lot and there are established practices and experienced developers all over who have been in those situations, so generally, these days, you don't see such code bases anymore.
That is how I read it.
Another possible reason you don't see those code bases anymore is the fact that such teams/companies don't have a competitive comp, so there are mostly junior devs or people who can't get a job at a more competent team that get hired in those places
That is how I read it.
Another possible reason you don't see those code bases anymore is the fact that such teams/companies don't have a competitive comp, so there are mostly junior devs or people who can't get a job at a more competent team that get hired in those places
So, you're basically running DynamoDB on top of a sql server?
This is essentially just a data warehousing style schema. I love me a narrow db table.
But I do try and get a schema that fits the business if I can.
But I do try and get a schema that fits the business if I can.
why a separate id and rowid? why not just rowid and data?
"The rowid is implicit and autoassigned, but we want developer-friendly IDs." maybe
But of course, the obvious solution is to have one table with just ROWID and data, and another table with the friendly IDs! If you time the insertions really well, then the ROWIDs in both tables with match and voilà.
But of course, the obvious solution is to have one table with just ROWID and data, and another table with the friendly IDs! If you time the insertions really well, then the ROWIDs in both tables with match and voilà.
It's the same thing actually
https://stackoverflow.com/questions/41928749/pros-cons-of-re...
https://stackoverflow.com/questions/41928749/pros-cons-of-re...
I’m seeing a lot of comments about terrible code bases, but I do think there’s something really beautiful here, something like “worse is better”:
> This may sound like a mess to you. But it was remarkably enjoyable to work in. Gone were the concerns of code duplication. Gone were the concerns of consistency. Gone were the concerns of extensibility. Code was written to serve a use, to touch as little of the area around it as possible, and to be easily replaceable. Our code was decoupled, because coupling it was simply harder.
> This may sound like a mess to you. But it was remarkably enjoyable to work in. Gone were the concerns of code duplication. Gone were the concerns of consistency. Gone were the concerns of extensibility. Code was written to serve a use, to touch as little of the area around it as possible, and to be easily replaceable. Our code was decoupled, because coupling it was simply harder.
Early on in my career, at a large company, I encountered someone who took “codebase” a little too literally. At the time every department had their own developers, sometimes just employees who had an aptitude for computers.
This one guy established himself by making an Access database for their core business, and when the web became a thing, built a customer site. But not on it—in it. He simply served ASP pages directly from the database, inserting dynamic content in queries. When I was asked to help improve their terrible design, I was forced to untangle that unholy mess of queries, ASP (new to me) and HTML. It was easiest to write all the HTML and insert their ASP right before I sent the files back (because I wasn’t given access to their DB/web server). Thinking “I could do better than this” got me into programming.
He was a Microsoft-everything head. Finally went too far when he presented a new web interface starring a Clippy-like parrot using Microsoft’s DirectX avatar API. The executives were unimpressed and then I noted that 20% of our customers couldn’t use his site. (I probably still had a “best viewed with IE” badge on the main site, lol)
This one guy established himself by making an Access database for their core business, and when the web became a thing, built a customer site. But not on it—in it. He simply served ASP pages directly from the database, inserting dynamic content in queries. When I was asked to help improve their terrible design, I was forced to untangle that unholy mess of queries, ASP (new to me) and HTML. It was easiest to write all the HTML and insert their ASP right before I sent the files back (because I wasn’t given access to their DB/web server). Thinking “I could do better than this” got me into programming.
He was a Microsoft-everything head. Finally went too far when he presented a new web interface starring a Clippy-like parrot using Microsoft’s DirectX avatar API. The executives were unimpressed and then I noted that 20% of our customers couldn’t use his site. (I probably still had a “best viewed with IE” badge on the main site, lol)
Wow, this is exactly how I felt with regard to my first job as well. This old codebase no one wants to touch but works somehow. The quite nice later-on additions. Even the "build a pipe-separated string using reflection and a 150 classes hierarchy" rings something.
The old codebase was an hairy ball of scala using a very outdated version of a (now) infamous actor framework. Before they figured out that untyped messages kinda left out one of the major selling point of Scala.
The code was readable, but the authors had this strange idea that every little piece of logic should be part of its own "Actor". An actor is pretty much equivalent to a class, and each one of them had their own little file. With this many classes, with very specialized purposes, you ended up with 90 character identifier names.
To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic. Generally, at the end, you find that most of the code is about passing around a value, and there is this one file where there is actual logic applied to the value.
It wasn't that awful. The thing was that it was impossible to change anything: no documentation, no test, no bug tracking, not even any PR process, laconic commit messages, no Wiki pages. And the actor framework made it very difficult to add tests. But later developers did manage it pretty well, they drew fences around the old services, with an HTTP API to communicate with it. And all later additions were part of different services that were very cleanly and consistently designed.
The old codebase was an hairy ball of scala using a very outdated version of a (now) infamous actor framework. Before they figured out that untyped messages kinda left out one of the major selling point of Scala.
The code was readable, but the authors had this strange idea that every little piece of logic should be part of its own "Actor". An actor is pretty much equivalent to a class, and each one of them had their own little file. With this many classes, with very specialized purposes, you ended up with 90 character identifier names.
To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic. Generally, at the end, you find that most of the code is about passing around a value, and there is this one file where there is actual logic applied to the value.
It wasn't that awful. The thing was that it was impossible to change anything: no documentation, no test, no bug tracking, not even any PR process, laconic commit messages, no Wiki pages. And the actor framework made it very difficult to add tests. But later developers did manage it pretty well, they drew fences around the old services, with an HTTP API to communicate with it. And all later additions were part of different services that were very cleanly and consistently designed.
> To understand what would be a single function in a normal code base, you would have to dive through half a dozen files through several repositories to piece together the logic.
I've experienced something like this, where the other devs preferred to breakecode down into the smallest units possible. I often saw a half-page procedure changed into multiple function calls, each with their own boilerplate and error handling (as required by the style-guide), such that you could not view the entire logic on one screen.
Every procedure name described the intent, they said, so you know what the parent did without having to worry about details like how it did it. I, meanwhile, trying to track down what the actual implementation was, would hold as many subfunctions as I could in windows with a tiny font, and hold the rest in my head…just to find out stuff like:
"Oh, so it's trying to use system feature foo.x, but I know foo.x is broken current release for a certain edgecase. Which case happens to be what our customer does all the time…"
I've experienced something like this, where the other devs preferred to breakecode down into the smallest units possible. I often saw a half-page procedure changed into multiple function calls, each with their own boilerplate and error handling (as required by the style-guide), such that you could not view the entire logic on one screen.
Every procedure name described the intent, they said, so you know what the parent did without having to worry about details like how it did it. I, meanwhile, trying to track down what the actual implementation was, would hold as many subfunctions as I could in windows with a tiny font, and hold the rest in my head…just to find out stuff like:
"Oh, so it's trying to use system feature foo.x, but I know foo.x is broken current release for a certain edgecase. Which case happens to be what our customer does all the time…"
It’s refreshing to hear that last paragraph. Honestly, sounds like the difference between hobby programming and professional engineering.
> Now the story I heard at the time was that once upon a time SQL Server didn't support auto-incrementing ids. This was the accepted, correct answer.
At a company that I used to work, they heard the same rumor, so instead of using identity columns or sequences, they kept a table with a number of ids "available" (one row per id). Whenever unique id was needed, the table would be locked, an id selected and marked as used. If there were no ids available, more ids would be added and then one used. A scheduled job would remove ids marked as used from time to time. Note that there was a single "sequence table", that was shared among all of the entities.
That was not even the weirdest part. That id was unique, but NOT the primary key of the entity, only part of it.
The structure of the database was fairly hierarchical, so you had for example a table CUSTOMER in 1-to-many relation with a USER table, with a 1-to-many relation with an ADDRESS table.
while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
At a company that I used to work, they heard the same rumor, so instead of using identity columns or sequences, they kept a table with a number of ids "available" (one row per id). Whenever unique id was needed, the table would be locked, an id selected and marked as used. If there were no ids available, more ids would be added and then one used. A scheduled job would remove ids marked as used from time to time. Note that there was a single "sequence table", that was shared among all of the entities.
That was not even the weirdest part. That id was unique, but NOT the primary key of the entity, only part of it.
The structure of the database was fairly hierarchical, so you had for example a table CUSTOMER in 1-to-many relation with a USER table, with a 1-to-many relation with an ADDRESS table.
while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
> while the primary key of the CUSTOMER table was a single CUSTOMER_ID column, the primary key of the USER table was (CUSTOMER_ID,USER_ID), and the primary key of the ADDRESS table was (CUSTOMER_ID,USER_ID,ADDRESS_ID). There were tables with 5 or 6 columns as a primary key.
Maybe they wanted to avoid maintaining a separate CUSTOMER_ADDRESS 1-to-many table or maybe it was done to make easy reverse lookups.
Maybe they wanted to avoid maintaining a separate CUSTOMER_ADDRESS 1-to-many table or maybe it was done to make easy reverse lookups.
I had dubious pleasure of working with similar codebases and devs. I'll remember one of those guys forever, because whenever he wanted to work on a new branch he would clone the repo, make changes to the master branch, and push code to a new repo numbered repo0001, repo002, ... He refused to change his ways, because "I have a PhD so you are wrong".
Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. That made porting code fun.
Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type. That made porting code fun.
> Another WTF moment was realisation that MS SQL Server does not support BOOLEAN type.
The standard does not have a boolean type. It's a postgres extension that the other open source databases adopted (because, yeah, it's obvious). But the proprietary ones insist on not having.
The official recommendation is using byte on MS SQL and char(1) on Oracle. Both are ridiculous.
The standard does not have a boolean type. It's a postgres extension that the other open source databases adopted (because, yeah, it's obvious). But the proprietary ones insist on not having.
The official recommendation is using byte on MS SQL and char(1) on Oracle. Both are ridiculous.
Pedantic, but the type is bit in MSSQL.
I don't think that's true. Even 2005 had a bit type and it's better optimized for space: https://learn.microsoft.com/en-us/sql/t-sql/data-types/bit-t...
C programmer joins the chat.
Hey we've had _Bool and <stdbool.h> since 1999!
Well, everything is a boolean in ANSI C. If I use `if (a)` I know the code will be executed only if `a` is nonzero.
Sounds awfully like my first job, with the addition of not having _any_ sort of test - functional, integration or unit. Nothing.
A few months in, when I approached the CTO and asked if I could start writing a test framework, he deemed it a waste of time and said "by the time you'd commit the test, it would go out of date and you'd need to rewrite it".
Naturally, the build would break about 5 times a week.
Boeing was a major customer of this system, so when shit hit the fan at Boeing a while ago, I wasn't surprised.
A few months in, when I approached the CTO and asked if I could start writing a test framework, he deemed it a waste of time and said "by the time you'd commit the test, it would go out of date and you'd need to rewrite it".
Naturally, the build would break about 5 times a week.
Boeing was a major customer of this system, so when shit hit the fan at Boeing a while ago, I wasn't surprised.
A couple of weeks ago I had a flat on the way to the airport. It was a total blowout, and my car doesn't include a spare. We were already over budget on our trip, so I had the car towed to the closest tire shop and had them put on the cheapest tire that could get me to the airport. I know I'll need to replace other tires, as it's an AWD, and I know it's not a tire I really want. I made a calculated choice to make that a problem for future me due to the time crunch I was under.
Programming is a lot like this.
Programming is a lot like this.
I worked at a startup where we had to make a lot of compromises. The (non technical) founders could not get enough funding at the beginning, so they had to constantly raise small amounts to survive the next few months.
We had to grow fast and prove that we had a market fit. This meant making compromises, but the technical debt was actually under control. We knew what to improve and how to do it once we had enough resources.
Eventually we proved ourselves and got decent funding and could hire more developers. We were looking forward to improving our system.
Unfortunately the new guys (many fresh out of college, lead by a few "FAANG"-ish seniors) decided that the technical debt was a proof of our incompetence and convinced the leadership that they needed to take over the technical development and rewrite everything. The result was a year where no major features were added and we lost our momentum. The funding dried up and the company had to let go 2/3 of the employees.
The worst part was that the new systems were more complicated to extend due to their clever "scalable design" (distributed monolith) with more micro services than developers. When the rewrite was done and it was time to add the new features, they had lost the trust of the leadership and just threw code at it in desperation to get things done as quickly as possible. I pithy the developers that inherited the result... By that time the rock stars had moved on and are inflicting damage elsewhere after touring conferences where they presented their greatness.
We had to grow fast and prove that we had a market fit. This meant making compromises, but the technical debt was actually under control. We knew what to improve and how to do it once we had enough resources.
Eventually we proved ourselves and got decent funding and could hire more developers. We were looking forward to improving our system.
Unfortunately the new guys (many fresh out of college, lead by a few "FAANG"-ish seniors) decided that the technical debt was a proof of our incompetence and convinced the leadership that they needed to take over the technical development and rewrite everything. The result was a year where no major features were added and we lost our momentum. The funding dried up and the company had to let go 2/3 of the employees.
The worst part was that the new systems were more complicated to extend due to their clever "scalable design" (distributed monolith) with more micro services than developers. When the rewrite was done and it was time to add the new features, they had lost the trust of the leadership and just threw code at it in desperation to get things done as quickly as possible. I pithy the developers that inherited the result... By that time the rock stars had moved on and are inflicting damage elsewhere after touring conferences where they presented their greatness.
One of the most important skills a person can have is to assess a situation, understand the actual context and constraints and build something based on this information. It sounds to me your team worked this way. You were playing within the given constraints.
Exactly the opposite of the one trick pony’s who just mindlessly apply what they learned or think is cool with a total disregard for context.
Exactly the opposite of the one trick pony’s who just mindlessly apply what they learned or think is cool with a total disregard for context.
A lot of "why TF would anyone you do this?" questions can be answered with variations like this. Borrowing from economics, the discount rate defines how much more valuable something is RIGHT NOW vs at some point in the future. If the system is down and we're losing money by the minute until it is up; if the company will be broke next week unless we make customer X happy; if you have to get to the airport in 2 hours - it is completely rational to make the quick fix. The costs of not doing so outweigh the costs of doing it again properly.
It really becomes a problem when the same short term calculation becomes the standard - you can quickly rack up costs that future you or future company will never pay down.
It really becomes a problem when the same short term calculation becomes the standard - you can quickly rack up costs that future you or future company will never pay down.
Aside - the fact that cars dont include spares these days is infuriating.
IMO its a classic example of real world enshittification
IMO its a classic example of real world enshittification
Supposedly there are reasons (for example, efficiency by avoiding the extra weight) and they give you a kit that's essentially a "fix a flat" product, but in the 2 times I've had a flat in a car outfitted like this, what they provided wasn't useful, usually due to the type of damage to the tire.
that's not enshittification!
it sucks, yes, but when coined, the term referred to a platform extracting value from its users by degrading their experience, forcing them to upgrade to a premium plan. your car not having a spare isn't an example of that!
https://en.wikipedia.org/wiki/Enshittification
it sucks, yes, but when coined, the term referred to a platform extracting value from its users by degrading their experience, forcing them to upgrade to a premium plan. your car not having a spare isn't an example of that!
https://en.wikipedia.org/wiki/Enshittification
Many of us grew up with cars having spare tyres as a matter of course. Now car manufacturers are extracting value from their customers by degrading our experience (of safety and expedience of repair), forcing us to pay extra for a spare tyre (if even available), or even upgrade to a better car.
Ordinary cars not having a spare tyre as standard equipment any more is a perfect example of enshittification.
Ordinary cars not having a spare tyre as standard equipment any more is a perfect example of enshittification.
Somebody at Toyota has read your comment and is now adding "spare tyres" to their subscription offering.
The first line really hits me hard. There’s something so incredibly freeing about being a kid and doing stuff like coding. There’s simply no expectations. Even the smallest project felt like such an achievement. But now I code professionally and I don’t know how to turn off engineering brain. I don’t know how to be okay doing something poorly, but on my terms.
This is something I've had to train myself to overcome. The first step for me was having a place where I clearly signaled that this wasn't my best work. Where the rules were allowed to be broken. That place is my junk drawer of code[1].
I have a zsh alias `alias changes='git add . && git commit -am "Changes" && git push'` that I use for my commits in the repo.
This all may feel silly, but it's what I needed to get back to that time of playing. Where I never worried about the code. But where I also didn't feel I was wasting my time working on code no one could see. I'd definitely recommend trying something like that if you are struggling with it.
[1]: https://github.com/jimmyhmiller/PlayGround
I have a zsh alias `alias changes='git add . && git commit -am "Changes" && git push'` that I use for my commits in the repo.
This all may feel silly, but it's what I needed to get back to that time of playing. Where I never worried about the code. But where I also didn't feel I was wasting my time working on code no one could see. I'd definitely recommend trying something like that if you are struggling with it.
[1]: https://github.com/jimmyhmiller/PlayGround
next step: not using git ;-)
i considered a few times to make an alias like this but i feel like git is so precious to get right that i would rather type out each command just to "feel" what i'm doing and be active in it.
i considered a few times to make an alias like this but i feel like git is so precious to get right that i would rather type out each command just to "feel" what i'm doing and be active in it.
Writers often say that your first N books will be crap, and the answer to that problem is to just write more than N books. I feel that’s true of software as well - the first years of the magical programming I did without any expectations meant those projects were utter crap yet functional.
I think you can add a mode to engineer brain, in which it's aware when it's kludging something, but kludging is the appropriate thing to do.
Might help to have enough experience that you have (well-placed) confidence in your gut feel for how much engineering a particular thing needs.
(More common is to not have an engineer mode, or to not have enough experience to do it well.)
If you don't have the kludge mode, try some things where kludges are outright necessary. One time I recall this was when writing Emacs extensions. For example, it used to be that you'd hit a point where normal programmatic navigation/parsing of the buffer was just too slow, and a well-placed regexp operation that worked 99.99% of the time (and the other 0.01% of the time it's OK enough, and the user understands) made the rest of the code viable.
Another example is personal open source projects, when you really want to experiment with an usual implementation approach, and you give yourself permission.
(Maybe don't do this when the open source is done only for resume purposes, where you're trying to demonstrate you follow all the current fashionable conventions. You're almost guaranteed that someday a new-grad on a public forum or not-very-technical hiring manager will stumble across that bit of code, and fixate on the one thing they think they recognize as a bad practice. Documenting the kludge and rationale, in, say, a code comment, is great practice, but, again, it will also draw the attention of the least-skilled. Much like, if you're relating an anecdote in a job interview, and implied are 20 things you did right and 1 thing you did brilliantly, and you note as an aside one mistake you made, most of that will whoosh over the head of the dimmest FAANG interviewer, but they'll latch onto that mistake you spelled out, as a wise insight they have, and that's what's going in their report. :)
Then, armed with an experienced multi-skilled brain, when your startup's imminent MVP launch is facing crazy constraints, you triage what bits need to be rock-solid so they'll absolutely work and not fail, what bits need creative kludging so you can hit your launch window for other key requirements, and what bits you need to mitigate any compromises. Ideally, you have the wisdom to know which is which, and can activate different skills for each kind of work.
That Turbo Encabulator needs to be well-machined, but the new sensor you just decided it needs for a test doesn't need a week to be drilled into the engine block, nor a mount to be designed and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or maybe wad of chewing gum will do.
Might help to have enough experience that you have (well-placed) confidence in your gut feel for how much engineering a particular thing needs.
(More common is to not have an engineer mode, or to not have enough experience to do it well.)
If you don't have the kludge mode, try some things where kludges are outright necessary. One time I recall this was when writing Emacs extensions. For example, it used to be that you'd hit a point where normal programmatic navigation/parsing of the buffer was just too slow, and a well-placed regexp operation that worked 99.99% of the time (and the other 0.01% of the time it's OK enough, and the user understands) made the rest of the code viable.
Another example is personal open source projects, when you really want to experiment with an usual implementation approach, and you give yourself permission.
(Maybe don't do this when the open source is done only for resume purposes, where you're trying to demonstrate you follow all the current fashionable conventions. You're almost guaranteed that someday a new-grad on a public forum or not-very-technical hiring manager will stumble across that bit of code, and fixate on the one thing they think they recognize as a bad practice. Documenting the kludge and rationale, in, say, a code comment, is great practice, but, again, it will also draw the attention of the least-skilled. Much like, if you're relating an anecdote in a job interview, and implied are 20 things you did right and 1 thing you did brilliantly, and you note as an aside one mistake you made, most of that will whoosh over the head of the dimmest FAANG interviewer, but they'll latch onto that mistake you spelled out, as a wise insight they have, and that's what's going in their report. :)
Then, armed with an experienced multi-skilled brain, when your startup's imminent MVP launch is facing crazy constraints, you triage what bits need to be rock-solid so they'll absolutely work and not fail, what bits need creative kludging so you can hit your launch window for other key requirements, and what bits you need to mitigate any compromises. Ideally, you have the wisdom to know which is which, and can activate different skills for each kind of work.
That Turbo Encabulator needs to be well-machined, but the new sensor you just decided it needs for a test doesn't need a week to be drilled into the engine block, nor a mount to be designed and CNC'd or 3D-printed, but the nearest Velcro or zip-tie or maybe wad of chewing gum will do.
All such great advice. The one time I’ve found success is when I figure out the entire scope of the little project up front. Then I find myself not even worrying about maintenance or scalability. I know I won’t need that extra class abstraction or modularity or whatever. I just don’t like planning at the start, but it’s probably, counterintuitively, part of the answer.
P.S. you opened a parenthesis and forgot to close it so now everything I read all day is part of your anecdote)
P.S. you opened a parenthesis and forgot to close it so now everything I read all day is part of your anecdote)
It's because you don't get to write apps from scratch enough. Everything is a POC. You get it working. Optimization is just dumb until the project is so big it doesn't fit in the developers brain anymore. Then you write tests for the core part so you can keep interating fast. You will see parts that clearly need refactorwd before you can move on.
Also, be aware of, but don't get hung up on, engineering best practices. Chances are that someone is going to see your POC and put it in front of a client or promise it to someone. It will enter service being less than perfect, if you are lucky.
When you pull this off people will tell stories about you and brag about you when you are not around. None of them will know anything about the code engineering.
Also, be aware of, but don't get hung up on, engineering best practices. Chances are that someone is going to see your POC and put it in front of a client or promise it to someone. It will enter service being less than perfect, if you are lucky.
When you pull this off people will tell stories about you and brag about you when you are not around. None of them will know anything about the code engineering.
> When you pull this off people will tell stories about you and brag about you when you are not around.
True!
> None of them will know anything about the code engineering.
Not true. In environments where it's very easy to fuck up (hyperconnected codebases, lots of legacy tech debt, fragile systems generally), programmers who create tools/libraries that let other engineers get work done without blowing their feet off or spending huge amounts of time on silly things do get praised. Maybe the praise comes from different people than in your example, but it definitely is rewarded.
Two caveats: this only happens in certain environments; some gigs are just culturally unable to care about quality tools (death march feature factories and the like). Also, you actually have to save time and reduce defect rates with your engineering work; starting an architecture-astronaut rewrite that isn't actually useful for other programmers' day-to-day work doesn't count. Making "one tool to rule them all" that requires a Ph. D or pairing session with the author doesn't count either.
True!
> None of them will know anything about the code engineering.
Not true. In environments where it's very easy to fuck up (hyperconnected codebases, lots of legacy tech debt, fragile systems generally), programmers who create tools/libraries that let other engineers get work done without blowing their feet off or spending huge amounts of time on silly things do get praised. Maybe the praise comes from different people than in your example, but it definitely is rewarded.
Two caveats: this only happens in certain environments; some gigs are just culturally unable to care about quality tools (death march feature factories and the like). Also, you actually have to save time and reduce defect rates with your engineering work; starting an architecture-astronaut rewrite that isn't actually useful for other programmers' day-to-day work doesn't count. Making "one tool to rule them all" that requires a Ph. D or pairing session with the author doesn't count either.
What ever small success I had outside work has been whenever I managed to turn off Enterprise brain.
It's hard for those who came into the discipline in the past twenty years to realize just how much things have changed around version control and building.
Joel Spolsky released the "Joel Test" for determining if the software team you were interviewing had good practices in 2000. One of the requirements was that they used version control. Not all that many teams actually did. Especially during the dotcom craze.
Today, passing the Joel Test is table stakes, and it's the rare shop that doesn't. But it took years for that sort of thing to become ubiquitous.
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
Joel Spolsky released the "Joel Test" for determining if the software team you were interviewing had good practices in 2000. One of the requirements was that they used version control. Not all that many teams actually did. Especially during the dotcom craze.
Today, passing the Joel Test is table stakes, and it's the rare shop that doesn't. But it took years for that sort of thing to become ubiquitous.
https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
Do most places pass the Joel Test? He says that if you answer no to two or more of these, "you have serious problems", and I count about seven that I would say no to at my current big-name (and well-reputed) tech company:
4. Do you have a bug database? ["an organized database listing all known bugs in the code"]
No, each team has one or several poorly organized Jira boards that contain some fraction of the known bugs.
5. Do you fix bugs before writing new code?
Lol? Does any major company do this?
6. Do you have an up-to-date schedule?
No, we have weekly sprints and the vaguest idea of when bigger projects will land.
7. Do you have a spec?
I haven't seen a real spec in years.
8. Do programmers have quiet working conditions?
Well, I work from home, but all the fancy offices are open-plan.
10. Do you have testers?
Not for any of the stuff I work on, and I don't know if we have them at all.
11. Do you do hallway usability testing?
Absolutely not.
So that's 5/12. How many companies you know would get 11 or 12?
4. Do you have a bug database? ["an organized database listing all known bugs in the code"]
No, each team has one or several poorly organized Jira boards that contain some fraction of the known bugs.
5. Do you fix bugs before writing new code?
Lol? Does any major company do this?
6. Do you have an up-to-date schedule?
No, we have weekly sprints and the vaguest idea of when bigger projects will land.
7. Do you have a spec?
I haven't seen a real spec in years.
8. Do programmers have quiet working conditions?
Well, I work from home, but all the fancy offices are open-plan.
10. Do you have testers?
Not for any of the stuff I work on, and I don't know if we have them at all.
11. Do you do hallway usability testing?
Absolutely not.
So that's 5/12. How many companies you know would get 11 or 12?
This reminds me of my first software job, an internship in college in 2012 building an application from scratch to facilitate evaluating teachers. It was basically an app to allow people to create a form for their school’s evaluation criteria and then submit those forms. Sounds super straightforward, right? It was. The catch was our team was 2 CS undergrad, a masters CS student, and a high school student. All with no professional experience. We knew nothing. Well kind of but more on that in a second. Our manager was absolutely non technical. In fact they were the second highest person in the company (fairly small company) and were managing our project and a bunch of other stuff at the company. And somehow with almost 0 oversight we built a functional Django application that the business was able to sell and make money from. My favorite highlights were 1) the codebase was initially shared over FTP (“Hey you’re not editing file X, right? Oh you are? Ah woops I just overwrote all your changes.”) till someone intelligently suggested “Uhhh Git?” 2) the actual best programmer amongst us was the high schooler. They suggested Django, picked the DB, they suggested using Celery to speed up async work, Redis for caching, and yes, “Uhh Git?” In retrospect the only reason we succeeded was because of them. They were like top 5 on the stack overflow Code Golf site IIRC. 3) My interview was basically showing my aforementioned manager who had never coded in his life a project I worked on at school and him being like “Yeah looks good. You’re hired.”
With 10 years of hindsight, I cringe thinking back to all the bad decisions I pushed for and the no-doubt terrible code I wrote. But I also marvel and look back fondly at being given a shot and being in an environment where I could just build something from the ground up and learn everything soup to nuts on the job. God bless whoever inherited that codebase.
With 10 years of hindsight, I cringe thinking back to all the bad decisions I pushed for and the no-doubt terrible code I wrote. But I also marvel and look back fondly at being given a shot and being in an environment where I could just build something from the ground up and learn everything soup to nuts on the job. God bless whoever inherited that codebase.
Man, I remember my Gilfoyle. We had an employee who would do one off programs for anyone with a use case - but ours wiped his computer before giving it back, so frequently we'd get tickets for software we'd never heard of that was somehow mission critical, and we'd have a few days to spin up a new version on the spot.
Probably some of the most fun I've ever had writing software was making Gilfoyle-2s.
Probably some of the most fun I've ever had writing software was making Gilfoyle-2s.
This hits close to home. At my current job we have a similar experience, We are building an Android app and the codebase is probably the same age as me (25yo). It started as a Windows Phone app that was later ported over to Android, you can easily find files and segments of the codebase that were auto generated by some code converter from C# to Java.
In the codebase itself you can see the evolution of code styles, older views and the core of the system is written in a very old and very stateful manner while newer parts of the application use modern architecture practices, we have two local databases that load configuration into the app for our different clients and their requirements, an global state is loaded at all times to check what is the correct business logic to follow.
Im a junior, few years into Android Programming and while sometimes its frustrating having to deal with some nasty bug because a random variable is updated for reasons only god knows, i think the experience its giving me its something im going to appreciate years down the road.
In the codebase itself you can see the evolution of code styles, older views and the core of the system is written in a very old and very stateful manner while newer parts of the application use modern architecture practices, we have two local databases that load configuration into the app for our different clients and their requirements, an global state is loaded at all times to check what is the correct business logic to follow.
Im a junior, few years into Android Programming and while sometimes its frustrating having to deal with some nasty bug because a random variable is updated for reasons only god knows, i think the experience its giving me its something im going to appreciate years down the road.
> Windows Phone - Wikipedia
> 1 week ago - It was first launched in October 2010 with Windows Phone 7.
25 years ago phones still had cords. Infact, C# itself is only just barely 19 or 20 years old at best. I know I felt like everything that came before me was ancient when I was young, but a decade is a long time in tech so i just felt the need to point out that if you had a codebase for a phone from 25 years ago if it was not written directly in assembly, it would have been running a very slim java version called Java ME (Micro Edition) for feature phones which was all the rage before android, or if you were unlucky you would be dealing with BREW which was a SDK for c/cpp development on feature handsets.
https://en.wikipedia.org/wiki/Java_Platform,_Micro_Edition https://en.wikipedia.org/wiki/Binary_Runtime_Environment_for...
C# Would have been beyond a luxury for a mobile application 25 years ago.
25 years ago phones still had cords. Infact, C# itself is only just barely 19 or 20 years old at best. I know I felt like everything that came before me was ancient when I was young, but a decade is a long time in tech so i just felt the need to point out that if you had a codebase for a phone from 25 years ago if it was not written directly in assembly, it would have been running a very slim java version called Java ME (Micro Edition) for feature phones which was all the rage before android, or if you were unlucky you would be dealing with BREW which was a SDK for c/cpp development on feature handsets.
https://en.wikipedia.org/wiki/Java_Platform,_Micro_Edition https://en.wikipedia.org/wiki/Binary_Runtime_Environment_for...
C# Would have been beyond a luxury for a mobile application 25 years ago.
When it comes to building things, the outcome is dictated by the constraints. But not the way most people realize.
Constraints affect outcomes in (at least) three ways:
There is an injured person on top of a mountain, and they need medical attention. You need to go up the mountain, get them, and bring them down.
You only have so much strength/endurance, so nothing you do will ever be faster than what your body is capable of. You need to get up and down in less than two hours. You need to carry an injured person. The mountain has two slopes: a long gradual one, and a short steep one.
Most people would climb the long gradual slope, because it's the path of least resistance. But it will take 4x as long to climb up and down it. Climbing straight up the steep slope would be incredibly tiring, and unsafe to bring someone down.
You can, however, zig-zag up and down the steep hill. It will take more time than going straight up, but faster than the long way, you will be less tired, and it's safer to bring someone down hill.
---
Constraints can be good and bad. Good use of constraints can allow you to get something done effectively. Bad use of constraints leads to failure. So it's important to have someone who can utilize those constraints effectively.
Vision, leadership, and wisdom is more important than the people, skills, materials, or time involved. The former determines the quality of the outcome more than the latter.
Constraints affect outcomes in (at least) three ways:
- by absolute limitation (you cannot violate these)
- by path of least resistance (the constraint makes X easier)
- by strategy (creative use of knowledge & skills leads to novel solutions)
There is an injured person on top of a mountain, and they need medical attention. You need to go up the mountain, get them, and bring them down.
You only have so much strength/endurance, so nothing you do will ever be faster than what your body is capable of. You need to get up and down in less than two hours. You need to carry an injured person. The mountain has two slopes: a long gradual one, and a short steep one.
Most people would climb the long gradual slope, because it's the path of least resistance. But it will take 4x as long to climb up and down it. Climbing straight up the steep slope would be incredibly tiring, and unsafe to bring someone down.
You can, however, zig-zag up and down the steep hill. It will take more time than going straight up, but faster than the long way, you will be less tired, and it's safer to bring someone down hill.
---
Constraints can be good and bad. Good use of constraints can allow you to get something done effectively. Bad use of constraints leads to failure. So it's important to have someone who can utilize those constraints effectively.
Vision, leadership, and wisdom is more important than the people, skills, materials, or time involved. The former determines the quality of the outcome more than the latter.
So to summarise:
„It depends“
Well, no... the summary is, you have to think hard in order to use your constraints as effectively as possible or the outcome will suck.
„You have to think hard“ is just „it depends“ worded differently.
If you don’t have to think hard then it also doesn’t depend on the details/circumstances and the outcome will be fine regardless of constraints.
If you don’t have to think hard then it also doesn’t depend on the details/circumstances and the outcome will be fine regardless of constraints.
This reminds me of my first job at a very small shop. Here’s two stories:
The calendar table struck a chord. We had one for holidays. One system needed to know when they were to calculate pay dates. Except once a year it would “run out” and someone would have to go add in the next year’s worth after a bad calculation was spotted.
The second time I was told to do it, I put in 10 years worth. The company didn’t survive long enough to need more.
My first “big” project was actually that pay date code. Every once in a while the server would hang, and people had deduced the date calculation was the problem. But no one knew why. And it wasn’t frequent enough to be worth taking time from the other two programmers. But I was new and thus “spare”.
After not being able to find any errors, I brute forced it. I ran that calculation for every day of the year for every pay type (weekly, monthly, bi-monthly, etc) and it quickly got locked in an infinite loop. Armed with the “bad” data that triggered it, it was easy to solve and provide tests to prove it would never happen again. I don’t remember what it was, exactly.
The calendar table struck a chord. We had one for holidays. One system needed to know when they were to calculate pay dates. Except once a year it would “run out” and someone would have to go add in the next year’s worth after a bad calculation was spotted.
The second time I was told to do it, I put in 10 years worth. The company didn’t survive long enough to need more.
My first “big” project was actually that pay date code. Every once in a while the server would hang, and people had deduced the date calculation was the problem. But no one knew why. And it wasn’t frequent enough to be worth taking time from the other two programmers. But I was new and thus “spare”.
After not being able to find any errors, I brute forced it. I ran that calculation for every day of the year for every pay type (weekly, monthly, bi-monthly, etc) and it quickly got locked in an infinite loop. Armed with the “bad” data that triggered it, it was easy to solve and provide tests to prove it would never happen again. I don’t remember what it was, exactly.
This reminds me of my first internship. The company I worked at backed everything up on CD:s. I was tasked with writing a new service for their intranet, indexing the content of all CD:s, so staff members could lookup which CD contained a certain file.
I wrote a little program to scan the CD:s as I inserted them into my computer, indexing the data into a database I had created, and then labelling the CD.
It wasn’t exactly exciting work but I still miss those days sometimes, everything was new and unexplored.
I wrote a little program to scan the CD:s as I inserted them into my computer, indexing the data into a database I had created, and then labelling the CD.
It wasn’t exactly exciting work but I still miss those days sometimes, everything was new and unexplored.
This codebase sounds like a haunted graveyard[1], where everyone just fixes their local corner of things and avoid the risk of untangling the existing mess.
Not needing to conform to some company-wide standard is probably really pleasant while it lasted, but every such effort adds to the haunted graveyard, and the lack of consistency will eventually come back to bite whoever is still around.
[1] https://www.usenix.org/sites/default/files/conference/protec...
Not needing to conform to some company-wide standard is probably really pleasant while it lasted, but every such effort adds to the haunted graveyard, and the lack of consistency will eventually come back to bite whoever is still around.
[1] https://www.usenix.org/sites/default/files/conference/protec...
It could maybe work if every team is responsible for their own area of the code base, where the code base starts to match the company tree
Im currently deep in rewriting a c# monolith thats 10+ years old that has thousands of lines of extra code that i was able to throw away because most of it was written before there were optional arguments so they made overloads for every permutation of arguments for every framework function
Boring but satisfying.
Oh, those kinds of columns. I thought we were talking text columns, and I was about to relate.
I work at a small business. Despite computer software being about the literal opposite of our business (plants), the founder built an entire suite of interconnected tools that runs off MS BASIC for Xenix, on a single HP machine running SCO OpenServer. The machine has so many customizations, self-scheduling cron/at jobs, odd nooks for files, weird tweaked programs, and special conventions that if a server with a dedicated hostname qualifies as a pet (as opposed to cattle), I'd be THIS THING'S pet.
The system handled EVERYTHING. Accounting, payroll, pesticide management, inventory, attendance, business contacts, shipping label printing... all out of a bunch of terminal menus (which are actually text files with control codes that get `cat`ed out).
But by God, the most horrifying part of it all are those BASIC files. They're IMPENETRABLE.
Firstly, I don't believe this version of BASIC supports named functions or subroutines. At all. But that's fine. MS BASIC being what it is, the interpreter only can deal with a certain number of characters per logical line, and that includes data definitions.
This version of BASIC (like so many others) includes its own serialization format and record/file access scheme. You declare the layout of the data file you want, open that file, and BASIC will handle (most of) the rest.
So when the founder started hitting the internal line limit while defining the data file's fields, he would cut the names of the fields down to fit more on that one line. Over time `30 AS EMPLOYEENAME` became `30ASEMPLNAME`, which became `30ASEMNAME` which became `30ASAF(1)`.
Every cent we transact, and every employee's timecards still flow through this old system, some even using bona fide Wyse terminals. To reiterate, this man was, first and foremost, a farmer. His contraption is terrifying, but commands immense respect. It's lasted 30-some years with continuous tweaking and refining, and we still have yet to replicate even half of its functionality. (Though there are other organizational issues that are making that difficult.)
On a personal note, aside from the calcified codebase and occasional spelling errors, it's a stellar business application. It's fast, mostly coherent, and keyboard-driven in such a way that experienced employees can navigate it faster than the terminal can refresh. We've been working for years to replace it, but at the same time, there's a lot our newfangled Angular+PHP+MySQL replacement could learn from it.
I work at a small business. Despite computer software being about the literal opposite of our business (plants), the founder built an entire suite of interconnected tools that runs off MS BASIC for Xenix, on a single HP machine running SCO OpenServer. The machine has so many customizations, self-scheduling cron/at jobs, odd nooks for files, weird tweaked programs, and special conventions that if a server with a dedicated hostname qualifies as a pet (as opposed to cattle), I'd be THIS THING'S pet.
The system handled EVERYTHING. Accounting, payroll, pesticide management, inventory, attendance, business contacts, shipping label printing... all out of a bunch of terminal menus (which are actually text files with control codes that get `cat`ed out).
But by God, the most horrifying part of it all are those BASIC files. They're IMPENETRABLE.
Firstly, I don't believe this version of BASIC supports named functions or subroutines. At all. But that's fine. MS BASIC being what it is, the interpreter only can deal with a certain number of characters per logical line, and that includes data definitions.
This version of BASIC (like so many others) includes its own serialization format and record/file access scheme. You declare the layout of the data file you want, open that file, and BASIC will handle (most of) the rest.
So when the founder started hitting the internal line limit while defining the data file's fields, he would cut the names of the fields down to fit more on that one line. Over time `30 AS EMPLOYEENAME` became `30ASEMPLNAME`, which became `30ASEMNAME` which became `30ASAF(1)`.
Every cent we transact, and every employee's timecards still flow through this old system, some even using bona fide Wyse terminals. To reiterate, this man was, first and foremost, a farmer. His contraption is terrifying, but commands immense respect. It's lasted 30-some years with continuous tweaking and refining, and we still have yet to replicate even half of its functionality. (Though there are other organizational issues that are making that difficult.)
On a personal note, aside from the calcified codebase and occasional spelling errors, it's a stellar business application. It's fast, mostly coherent, and keyboard-driven in such a way that experienced employees can navigate it faster than the terminal can refresh. We've been working for years to replace it, but at the same time, there's a lot our newfangled Angular+PHP+MySQL replacement could learn from it.
Why a webpage instead of another TUI?
In consumer banking, there is a pretty popular software called Finacle developed by Infosys. It was originally a TUI called Bancs2000 and was written in C and Pro*C. It could basically accept input as fast as one could type. With a few months experience, you would be blazing fast. Then they replaced it with a web based component and called it Finacle - and everyone hated it at first. The replacement wasn't particularly slow by standards of a web based component, just that the users were spoiled by the combination of muscle memory supported by TUI speed.
This reminds me of working with a company that provided market-wide pricing data for a particular commodity across the US. They were the de facto aggregator of pricing for the sector and at the time I worked for one of their larger customers. We requested they add another vendor’s pricing in a particular region and received a response along the lines of “Sure, as soon as we can figure out how to add another entry. We’re at the maximum row count on the system.”
Needless to say it gave my team a few days of flabbergast and speculation on what system they must have built on to hit a row limit at only 5 digits. And yet for a non-tech industry it was mostly working.
Needless to say it gave my team a few days of flabbergast and speculation on what system they must have built on to hit a row limit at only 5 digits. And yet for a non-tech industry it was mostly working.
This is glorious. I can imagine people could write versions of this for every industry out there and they would all be equally fun to read.
There were a couple times I was convinced that basic methods were as good or better than highfalutin ones, or got me thinking about solutions that hew closer to the basic method.
In 2014 or so, someone at a company told me that they just don’t do branches, and merge everything into trunk. I agree that long-lived branches are bad, but no branches at all? Sure enough, this was OK — meaning branches could be kept and synced by a few people outside the main repo, but forks are never pushed to the repo.
Using table columns for data, as long as you’re building for an actual specific business vertical and not just a general-purpose framework.
You never know when you’ll want to put an index on the data, and databases already have built-in mechanisms to select a subset of the fields, etc. You avoid all kinds of joins. If you need to extend the table, just start a new table with the same synthetic ID.
I would say, in tables where you don’t want to even have global locks (eg sharded tables), just don’t have an autoincrementing ID. Instead, try a random ID or a UUID, and insert it, then use it across tables.
In short, what was described here is actually good design.
In 2014 or so, someone at a company told me that they just don’t do branches, and merge everything into trunk. I agree that long-lived branches are bad, but no branches at all? Sure enough, this was OK — meaning branches could be kept and synced by a few people outside the main repo, but forks are never pushed to the repo.
Using table columns for data, as long as you’re building for an actual specific business vertical and not just a general-purpose framework.
You never know when you’ll want to put an index on the data, and databases already have built-in mechanisms to select a subset of the fields, etc. You avoid all kinds of joins. If you need to extend the table, just start a new table with the same synthetic ID.
I would say, in tables where you don’t want to even have global locks (eg sharded tables), just don’t have an autoincrementing ID. Instead, try a random ID or a UUID, and insert it, then use it across tables.
In short, what was described here is actually good design.
Honestly, I like to work on such systems because:
- there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it)
- it’s back to school again and everything goes. You can implement features in any way you want because the constraints the system imposes. Granted, sometimes it’s painful to add functionality
- there’s usually no room to subjective topics like clean code and architecture. The most important thing with these systems is correctness (and this is usually an objective topic)
- nobody can blame you for something that doesn’t work. It’s always the fault of the legacy system
I wouldn’t recommend working on such systems to junior engineers, though.
I don’t really like to work on “perfect” codebases where everyone follows the same pattern, with linters, where if something breaks is because of your shitty code (because the codebase is “clean”). It’s very frustrating and limiting.
- there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it)
- it’s back to school again and everything goes. You can implement features in any way you want because the constraints the system imposes. Granted, sometimes it’s painful to add functionality
- there’s usually no room to subjective topics like clean code and architecture. The most important thing with these systems is correctness (and this is usually an objective topic)
- nobody can blame you for something that doesn’t work. It’s always the fault of the legacy system
I wouldn’t recommend working on such systems to junior engineers, though.
I don’t really like to work on “perfect” codebases where everyone follows the same pattern, with linters, where if something breaks is because of your shitty code (because the codebase is “clean”). It’s very frustrating and limiting.
> there is so much stuff to improve. There’s nothing better than the feeling of improving things with code (or by removing it)
And there is nothing worse than a crappy codebase the company won't let you improve.
And there is nothing worse than a crappy codebase the company won't let you improve.
I mean, it is kind of nice to notice that something isn't going to work because you have the linters and tests. When your developer count goes up, chances that erroneous behavior is included also goes up.
I've created proof-of-concepts that worked perfectly but would make you cry if you looked at how they worked. Eventually they became that mess. Everything is a self-contained unit so it doesn't mess anything else up. Of course, there is always time to keep adding new stuff but never to refactor it into what it should be.
I prefer the way with linters and tests, it at least lessens the chances of whatever is put in being broken (or breaking something else). (Then again, somebody putting "return true" in a test _will_ surprise you sooner or later)
I've created proof-of-concepts that worked perfectly but would make you cry if you looked at how they worked. Eventually they became that mess. Everything is a self-contained unit so it doesn't mess anything else up. Of course, there is always time to keep adding new stuff but never to refactor it into what it should be.
I prefer the way with linters and tests, it at least lessens the chances of whatever is put in being broken (or breaking something else). (Then again, somebody putting "return true" in a test _will_ surprise you sooner or later)
You also get to see some genuinely creative stuff that works well, written by people who aren't indoctrinated in a particular approach.
And weird code can lead to crazy outages.
I put together a thread of all of the wild incidents I've seen over my many years of working as a FinTech SRE:
https://x.com/alexpotato/status/1215876962809339904
I put together a thread of all of the wild incidents I've seen over my many years of working as a FinTech SRE:
https://x.com/alexpotato/status/1215876962809339904
I've seen something very similar to the Sequence key table. But there wasn't just one of them, it was a common pattern. And it had 2 rows and 2 columns.
The reason we had it is we had master-master replication, but without requiring the peer to acknowledge befor committing a transaction. To avoid inconsistencies, we preferred one server for even ids and the other for odd ids. But when writing a new record, an autogenerating sequence would just give the next id without regard to what "side" the request was on. So we had a table to keep track of the next id to use for each side, where we incremented the next id by 2 each time a new id was allocated.
It was a little weird, but it worked fairly well. Although we eventually changed the architecture and removed the need for those tables.
The reason we had it is we had master-master replication, but without requiring the peer to acknowledge befor committing a transaction. To avoid inconsistencies, we preferred one server for even ids and the other for odd ids. But when writing a new record, an autogenerating sequence would just give the next id without regard to what "side" the request was on. So we had a table to keep track of the next id to use for each side, where we incremented the next id by 2 each time a new id was allocated.
It was a little weird, but it worked fairly well. Although we eventually changed the architecture and removed the need for those tables.
Oh man, this article reminds me of an article that was a parody of some horrid business logic.
Something like this: a "genius" programmer was somehow, for some reason using svn commits as a method dispatcher. Commit ids were sprinkled throughout the codebase. A new hire broke the entire system by adding comments, and comments weren't compatible the bespoke svn method dispatcher.
Does anybody remember this article? I really want to read it again.
Something like this: a "genius" programmer was somehow, for some reason using svn commits as a method dispatcher. Commit ids were sprinkled throughout the codebase. A new hire broke the entire system by adding comments, and comments weren't compatible the bespoke svn method dispatcher.
Does anybody remember this article? I really want to read it again.
Tom is a genius!
https://thedailywtf.com/articles/the-inner-json-effect
https://thedailywtf.com/articles/the-inner-json-effect
Yes!
Has anyone tried implementing it? If not I'm going to give it a shot. :)
Has anyone tried implementing it? If not I'm going to give it a shot. :)
It'd be way easier (and actually not completely insane, though still far from sane) in git: you'd store the functions in blobs instead of misusing svn commits. It'd probably be a lot faster too as Git is fairly good at giving you object content.
Admittedly the GC would be an issue, but I think that would be fixable: instead of a `functions` key in a json, link all the functions in a tree object, the link that tree object from a commit that way you have the author for free. And then the class name can be a ref' in a bespoke namespace.
Well that's the issue that it's not necessarily clear what commits go together to compose the system, so might actually be better to replace the ref' per class by a tree for the entire thing, with each class being a commit entry, and an other commit capping the entire thing. Of course git will not understand what's happening as commit entries in trees are normally for submodules but that's fine, git is just used as a data store here.
Admittedly the GC would be an issue, but I think that would be fixable: instead of a `functions` key in a json, link all the functions in a tree object, the link that tree object from a commit that way you have the author for free. And then the class name can be a ref' in a bespoke namespace.
Well that's the issue that it's not necessarily clear what commits go together to compose the system, so might actually be better to replace the ref' per class by a tree for the entire thing, with each class being a commit entry, and an other commit capping the entire thing. Of course git will not understand what's happening as commit entries in trees are normally for submodules but that's fine, git is just used as a data store here.
The Inner JSON Effect. Most recent HN discussion of it:
https://news.ycombinator.com/item?id=40923258
https://news.ycombinator.com/item?id=40923258
Thanks! Also:
The Inner JSON Effect (2016) - https://news.ycombinator.com/item?id=35964931 - May 2023 (11 comments)
The Inner Json Effect - https://news.ycombinator.com/item?id=12185727 - July 2016 (142 comments)
The Inner JSON Effect (2016) - https://news.ycombinator.com/item?id=35964931 - May 2023 (11 comments)
The Inner Json Effect - https://news.ycombinator.com/item?id=12185727 - July 2016 (142 comments)
I worked many years with an open source eCommerce platform called Magento[0] which, at the time, used something called the "Entity Attribute Value" (or EAV) system[1].
One particularity of the EAV system is that you end up having tables with hundreds (and growing) of columns. It made Magento itself extremely hard to work with and optimize. I hope they moved away from this model since.
To be fair, this was before nosql databases were a thing.
[0]: https://en.wikipedia.org/wiki/Magento
[1]: https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
One particularity of the EAV system is that you end up having tables with hundreds (and growing) of columns. It made Magento itself extremely hard to work with and optimize. I hope they moved away from this model since.
To be fair, this was before nosql databases were a thing.
[0]: https://en.wikipedia.org/wiki/Magento
[1]: https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...
I can second the sequence key table being because auto increment wasn't available on some databases. I ran into the same some years back at a company who's software dated back to the late 60's.
He acts like sequence key is odd, but that’s quite normal in database world.
https://www.postgresql.org/docs/current/sql-createsequence.h...
https://www.postgresql.org/docs/current/sql-createsequence.h...
Databases have built-in features for this now. What the author is talking about is a regular table.
In reality, that wasn't too unusual to see because frameworks would use that technique because it's a lowest common denominator across RDMS.
In reality, that wasn't too unusual to see because frameworks would use that technique because it's a lowest common denominator across RDMS.
Does SQLite have sequences yet?
Sqlite only has automatically assigned row ID. You can type a column as "integer primary key" to make it an alias for the row ID, or use ROWID directly (not recommended for compatibility).
I think the intriguing part was purposefully using the same sequence value for rows in multiple tables.
I've worked with globally unique (to our application) integer keys, and per table integer sequences (which obviously aren't globally unique), but I don't recall seeing anyone use a global sequence but purposefully reuse elements of the sequence before.
I've worked with globally unique (to our application) integer keys, and per table integer sequences (which obviously aren't globally unique), but I don't recall seeing anyone use a global sequence but purposefully reuse elements of the sequence before.
I've seen this fairly recently and I was surprised to also see it in this article because I think that makes it twice that I've seen it in ~25 years.
They may even be the same codebase!
It’s kind of like an idempotency key assigned either by the request or generated at the beginning of processing that request
If the caller/consumer isn't choosing the idempotence key then I don't think it would act as a particularly good one.
In this scenario the database is choosing the id during a transaction and so you'll only find out the value if the transaction successfully commits and the response makes it back to you.
Generally you'd want the client to provide the idempotency key such that whatever failure occurs you know the value and can provide it on a retry such that the server can use it to prevent double entry / etc
In this scenario the database is choosing the id during a transaction and so you'll only find out the value if the transaction successfully commits and the response makes it back to you.
Generally you'd want the client to provide the idempotency key such that whatever failure occurs you know the value and can provide it on a retry such that the server can use it to prevent double entry / etc
Yes, but both these have very different properties. He said (I don't know if its the case) that the db didnt have an autoincremental type. Postgres uses these sequence objects to implement autoincremental ids as he was referring to, they are implemented in-engine and are very fast and have already solved data races.
In the article, what he complains about is not about what a sequence is, but about implementing it manually with a table that is read, incremented and then saved. This us more expensive, and depending on how it was implemented you need to take care of the whole data flow so you are unable to allocate the same id twice. That's what he considers odd
In the article, what he complains about is not about what a sequence is, but about implementing it manually with a table that is read, incremented and then saved. This us more expensive, and depending on how it was implemented you need to take care of the whole data flow so you are unable to allocate the same id twice. That's what he considers odd
The scary thing to me about that setup is how the global value is updated. Every individual script must successfully increment the value to avoid duplicate keys.
Really hope they had a “get key” stored procedure to handle that.
Really hope they had a “get key” stored procedure to handle that.
It's possible they had some requirement for gapless ID numbers. You can't do that with a sequence.
I think you missed the crucial part: every related record across all tables would have the same sequence item (the same id). That's really not normal in the database world. It sounds a lot like ECS though.
yeah, I've seen tables with a single row and single columns a few times, for alright reasons. Sometimes you do just want to store a single global value!
Working on something like that would drive me absolutely batty. I am happy you were able to find your zen in the middle of that chaos. This post truly speaks to the human condition
Once not long enough, I'd worked on 4 projects which was literally copied from the first made and changed parts of the customers and internal users according to each use of it. So, the main problem is bugs found in one project was found on the other 3 and I'd to fix the same bug!
Codebases like this or from the OP is cool to learn how to not do certain things.
Codebases like this or from the OP is cool to learn how to not do certain things.
I'm glad OP was able to maintain a good sense of humor about it. Such discoveries as the nested classes of empty methods have sent me into teeth-gnashing fury followed by darkness and despair. One such experience is why I would rather change careers if my only option were to build on the Salesforce platform, for instance.
In my first job, I was part of an offshore ops team who maintained lot of code that no one (Onshore and offshore) wants to maintain. But like all ops. Code, it was business critical.
This included a project called IefParser, its job was to parse incoming data files and put the data into databases. It was a serious project. Like really serious. The software input format came with a full manual with a section highlighting the changes from previous versions. I have not seen that level of detail before or since.
And It was also very old. Written about a year or two after I was born.and never rewritten after. So the software side of things were less serious and more hacky.
Core code was written in C. Database used was oracle. And code was driven by Perl batch script triggered via cron job. And all that ran on IBM AIX (unix). Yes,not windows or Linux. It ran on unix.
The sheer difference in software requirements which were meticulously and the software was mind boggling.
Some fun facts:
- c code could not be compiled on windows . You need to login to dev server via putty and run makefile to do it.
- Perl code was not checked in to repository. Perl code also was different for each environment.
- unix version has a vi editor which for some reason didn’t tell if you were in edit mode or command mode. WTF! Yes, other editor didn’t exist. Apparently I was only one who bothered to learn vi in India. As no one else in India could reliably edit files
- cron job schedule was also no checked in. After a “great learning opportunity “, I decided indeed to check that in
- I once had the horror of fixing bugs in Perl and cron job. Apparently global variables are the state of art in Perl.
- cron job always failed on New Year’s Day because the batch script uses MMDD formatted folder for temp file storage, and was unable to understand 0101 is greater than 1231. I volunteered to track down the issue, and fix it for good. M3 shot it down saying, “we don’t to take risks on such a mission critical service”
This included a project called IefParser, its job was to parse incoming data files and put the data into databases. It was a serious project. Like really serious. The software input format came with a full manual with a section highlighting the changes from previous versions. I have not seen that level of detail before or since.
And It was also very old. Written about a year or two after I was born.and never rewritten after. So the software side of things were less serious and more hacky.
Core code was written in C. Database used was oracle. And code was driven by Perl batch script triggered via cron job. And all that ran on IBM AIX (unix). Yes,not windows or Linux. It ran on unix.
The sheer difference in software requirements which were meticulously and the software was mind boggling.
Some fun facts:
- c code could not be compiled on windows . You need to login to dev server via putty and run makefile to do it.
- Perl code was not checked in to repository. Perl code also was different for each environment.
- unix version has a vi editor which for some reason didn’t tell if you were in edit mode or command mode. WTF! Yes, other editor didn’t exist. Apparently I was only one who bothered to learn vi in India. As no one else in India could reliably edit files
- cron job schedule was also no checked in. After a “great learning opportunity “, I decided indeed to check that in
- I once had the horror of fixing bugs in Perl and cron job. Apparently global variables are the state of art in Perl.
- cron job always failed on New Year’s Day because the batch script uses MMDD formatted folder for temp file storage, and was unable to understand 0101 is greater than 1231. I volunteered to track down the issue, and fix it for good. M3 shot it down saying, “we don’t to take risks on such a mission critical service”
I really love these kinds of stories. Does anybody know if there's a collection of similar stories/code bases anywhere?
I guess there is dailywtf but that's mostly bugs. Probably good enough though
I guess there is dailywtf but that's mostly bugs. Probably good enough though
It’s a cute story, it made me laugh.
If this story reminds you of the codebase you work on, don’t let its romanticism deceive you into thinking that suffering is somehow actually good.
If this story reminds you of the codebase you work on, don’t let its romanticism deceive you into thinking that suffering is somehow actually good.
Two early databases I worked on.
The first contained monetary values. These were split over two columns, a decimal column holding the magnitude of the value, and a string column, containing an ISO currency code. Sounds good so far, right? Well, I learned much later (after, of course, having relied on the data) that the currency code column had only been added after expanding into Europe … but not before expanding into Canada. So when it had been added, there had been mixed USD/CAD values, but no currency code column to distinguish them. But when the column was added, they just defaulted it all to USD. So and USD value could be CAD — you "just" needed to parse the address column to find out.
Another one was a pair of Postgres DBs. To provide "redundancy" in case of an outage, there were two such databases. But no sort of Postgres replication strategy was used between them, rather, IIRC, the client did the replication. There was no formal specification of the consensus logic — if it could even be said to have such logic; I think it was just "try both, hope for the best". Effectively, this is a rather poorly described multi-master setup. They'd noticed some of the values hadn't replicated properly, and wanted to know how bad it was; could I find places where the databases disagreed?
I didn't know the term "split brain" at the time (that would have helped!), but that's what this setup was in. What made pairing data worse is that, while any column containing text was a varchar, IIRC the character set of the database was just "latin1". The client ran on Windows, and it was just shipping the values from the Windows API "A" functions directly to the database. So Windows has two sets of APIs for like … everything with a string, an "A" version, and a "W" version. "W" is supposed to be Unicode¹, but "A" is "the computer's locale", which is nearly never latin1. Worse, the company had some usage on machines that were set to like, the Russian locale is, or the Greek locale. So every string value in the database was, effectively, in a different character set, and nowhere was it specified which. The assumption is the same bytes would always get shipped back to the same client, or something? It wasn't always the case, and if you opened a client and poked around enough, you'd find mojibake easily enough. Now remember we're trying to find mismatched/unreplicated rows? Some rows were mismatched in character encoding only: the values on the two DBs were technically the same, just encoded differently. (Their machines' Python setup was also broken, because Python was ridiculously out of date. I'm talking 2.x where the x was too old, this was before the problems of Python 3 were relevant. Everything in the company was C++, so this didn't matter much to the older hands there, but … god a working Python would have made working with character set issues so much easier.)
¹IIRC, it's best described as "nearly UTF-16"
The first contained monetary values. These were split over two columns, a decimal column holding the magnitude of the value, and a string column, containing an ISO currency code. Sounds good so far, right? Well, I learned much later (after, of course, having relied on the data) that the currency code column had only been added after expanding into Europe … but not before expanding into Canada. So when it had been added, there had been mixed USD/CAD values, but no currency code column to distinguish them. But when the column was added, they just defaulted it all to USD. So and USD value could be CAD — you "just" needed to parse the address column to find out.
Another one was a pair of Postgres DBs. To provide "redundancy" in case of an outage, there were two such databases. But no sort of Postgres replication strategy was used between them, rather, IIRC, the client did the replication. There was no formal specification of the consensus logic — if it could even be said to have such logic; I think it was just "try both, hope for the best". Effectively, this is a rather poorly described multi-master setup. They'd noticed some of the values hadn't replicated properly, and wanted to know how bad it was; could I find places where the databases disagreed?
I didn't know the term "split brain" at the time (that would have helped!), but that's what this setup was in. What made pairing data worse is that, while any column containing text was a varchar, IIRC the character set of the database was just "latin1". The client ran on Windows, and it was just shipping the values from the Windows API "A" functions directly to the database. So Windows has two sets of APIs for like … everything with a string, an "A" version, and a "W" version. "W" is supposed to be Unicode¹, but "A" is "the computer's locale", which is nearly never latin1. Worse, the company had some usage on machines that were set to like, the Russian locale is, or the Greek locale. So every string value in the database was, effectively, in a different character set, and nowhere was it specified which. The assumption is the same bytes would always get shipped back to the same client, or something? It wasn't always the case, and if you opened a client and poked around enough, you'd find mojibake easily enough. Now remember we're trying to find mismatched/unreplicated rows? Some rows were mismatched in character encoding only: the values on the two DBs were technically the same, just encoded differently. (Their machines' Python setup was also broken, because Python was ridiculously out of date. I'm talking 2.x where the x was too old, this was before the problems of Python 3 were relevant. Everything in the company was C++, so this didn't matter much to the older hands there, but … god a working Python would have made working with character set issues so much easier.)
¹IIRC, it's best described as "nearly UTF-16"
> But when the column was added, they just defaulted it all to USD. So and USD value could be CAD — you "just" needed to parse the address column to find out.
I bet no one even considered the idea that someone in Canada might pay in USD dollars.
I had to... discuss... with someone the possibility that if we were recording money received from a bank, we might do well to record the currency type as well, because... banks can deal with multiple currencies. "No, banks don't do that. That's not possible". This US company was getting ready to expand their services over to Europe, and I couldn't understand why no one could understand that this might be a necessary item to record.
Someone using this system in France to help finance a project in New York, for example, might want to know whether Euros or USD were the subject of the project. This was sort of a PM tool to note the cost of a project and how much was left in a budget - it wasn't moving money directly. We had a system to choose specific banks and note that 'person X moved Y currency units to the project' but... no ability to select the currency units. If it was from a UK bank, it would be GBP. A US bank, USD, etc. I was voted down as not understanding how banks worked.
Months later, a demo was presented right before launch to the people who were financing the project, and they asked how someone would know the currency being used for each project. Back came a work ticket "high priority" because we were already late for launch and multiple screens now had to accommodate this "new" requirement.
I know at least some of this is how I present info, but... being 'nice', being 'inquisitive', raising issues up the chain of command, etc.. rarely work. These sorts of extremely obvious things get ignored until the actual end users bring up the same concern. Somehow I wasn't supposed to know this sort of info because "you're a developer, you're not some international banker". Insane...
I bet no one even considered the idea that someone in Canada might pay in USD dollars.
I had to... discuss... with someone the possibility that if we were recording money received from a bank, we might do well to record the currency type as well, because... banks can deal with multiple currencies. "No, banks don't do that. That's not possible". This US company was getting ready to expand their services over to Europe, and I couldn't understand why no one could understand that this might be a necessary item to record.
Someone using this system in France to help finance a project in New York, for example, might want to know whether Euros or USD were the subject of the project. This was sort of a PM tool to note the cost of a project and how much was left in a budget - it wasn't moving money directly. We had a system to choose specific banks and note that 'person X moved Y currency units to the project' but... no ability to select the currency units. If it was from a UK bank, it would be GBP. A US bank, USD, etc. I was voted down as not understanding how banks worked.
Months later, a demo was presented right before launch to the people who were financing the project, and they asked how someone would know the currency being used for each project. Back came a work ticket "high priority" because we were already late for launch and multiple screens now had to accommodate this "new" requirement.
I know at least some of this is how I present info, but... being 'nice', being 'inquisitive', raising issues up the chain of command, etc.. rarely work. These sorts of extremely obvious things get ignored until the actual end users bring up the same concern. Somehow I wasn't supposed to know this sort of info because "you're a developer, you're not some international banker". Insane...
For such told-you-so scenarios, when there is a disagreement and you are pretty sure you are correct, raise a ticket, describe the issue and close it "As Designed" with comment detailing the discussion (including the overruling party). Next time there is a ticket to do this with "high priority", tell them "we already have a ticket for this, let me reopen it" for everyone to see... risky though, saving face/burning bridges and all that.
Agreed. I usually document the disagreement somewhere - tickets, or email thread at very least.
> I bet no one even considered the idea that someone in Canada might pay in USD dollars.
Certainly not I, at the time! (In my defense, I was an intern, and was shocked to learn that rows labelled "USD" could not be assumed to be USD. I hope today I am less naïve… but even by today me's standards, that's pretty appalling data quality.)
> I was voted down as not understanding how banks worked.
Yep, I feel this. The best engineers I have worked with make it their job to become an expert in the subject matter they're designing/engineering software for. I mean … why wouldn't they?
Certainly not I, at the time! (In my defense, I was an intern, and was shocked to learn that rows labelled "USD" could not be assumed to be USD. I hope today I am less naïve… but even by today me's standards, that's pretty appalling data quality.)
> I was voted down as not understanding how banks worked.
Yep, I feel this. The best engineers I have worked with make it their job to become an expert in the subject matter they're designing/engineering software for. I mean … why wouldn't they?
Reminds me of a date column where half the dates were MM/DD/YYYY and the rest DD/MM/YYYY. I did manage to eventually find the point in time when it changed and normalise the database. For those wondering about DD/MM, Australia.
And most countries :)
https://en.wikipedia.org/wiki/List_of_date_formats_by_countr...
https://en.wikipedia.org/wiki/List_of_date_formats_by_countr...
Sure, just trying to explain it to US ;)
In one of my early career jobs, we had applications that were so buggy we had 1 or 2 developers dedicated to logging into customers databases and fixing data.
I just wrote an answer regarding data denormalization:
https://stackoverflow.com/a/78831591/467460
It is from the point of view of an app developer, not a DBA, but should be relevant to most people on HN.
https://stackoverflow.com/a/78831591/467460
It is from the point of view of an app developer, not a DBA, but should be relevant to most people on HN.
Slicing columns into multiple tables is fairly common type of sharding. Sort of SQL way to do columnary store.
But splitting into multiple tables because you hit the 1024 column limit is probably not a common type of sharding...
This is such an horrendous practice, I am yet to find a database were this makes sense. Maybe my brain is not wired for it
The problem is maybe not so much the splitting and putting extra columns in a separate table. It's that you even have a table that large that it necessitates such a thing. Worst case you have a main table and a detail table that has a one to one correlation to the main entity table.
Why is that worse than a couple of dozen joins?
Because that means your data is highly denormalized and has plenty of duplicates. But in all likelihood it means no one knows wtf this table actually represents and you should be firing people.
I've seen this play out. Usually the many columns is because everyone misuses the table and eventually their special little business scenario or "filter" needs to be a column. Bonus points is whoever has to reference this table, they have to copy over whatever the hell your PK seems to be, and the cycle repeats, this time a bit worse.
Last place I did a brief project in had this. Queue 1000 tables spread across 25 schemas, each table having wide PKs, 20 redundant indexes on each table, and despite all this the database performs poorly. No one can tell you what each table represents, the table names are meaningless and the same data is everywhere. In order to get anything done you have to ask a small cabal of priests that knows the processes that write between these tables. After about 10 years, a partial rewrite happens and you now have 1/3rd of the data on each side with plenty of duplicate and overlap because hey.
I feel torn, I really wanna name&shame this company as a warning to all developers thinking about working there.
I've seen this play out. Usually the many columns is because everyone misuses the table and eventually their special little business scenario or "filter" needs to be a column. Bonus points is whoever has to reference this table, they have to copy over whatever the hell your PK seems to be, and the cycle repeats, this time a bit worse.
Last place I did a brief project in had this. Queue 1000 tables spread across 25 schemas, each table having wide PKs, 20 redundant indexes on each table, and despite all this the database performs poorly. No one can tell you what each table represents, the table names are meaningless and the same data is everywhere. In order to get anything done you have to ask a small cabal of priests that knows the processes that write between these tables. After about 10 years, a partial rewrite happens and you now have 1/3rd of the data on each side with plenty of duplicate and overlap because hey.
I feel torn, I really wanna name&shame this company as a warning to all developers thinking about working there.
> But in all likelihood it means no one knows wtf this table actually represents and you should be firing people.
That's almost exactly the opposite of my experience, but then I've worked on smaller teams with long-term team members, so perhaps that's why.
Our tables are wide because law require we store the data for 10 years, and it's easier and faster to store things in a single row, rather than spread out over several dozen detail tables.
We don't denormalize in the sense of storing only invoice items, repeating the head data on each item. We do denormalize in the sense of storing full name and address for each party rather than storing that in a detail table. So minimal, if any, duplication of data. If we duplicate it's almost always due to the law, ie data might change in one table but shouldn't be changed in the other to preserve history.
For child tables we always have a unique ID, sequence or autoinc, and the FK to the parent, but we include the root-level ID as well. This way we can select all the child rows in a single select regardless of level, again without tons of joins.
This way our core data is just a handful of tables, and there's not much doubt about what goes where.
That's almost exactly the opposite of my experience, but then I've worked on smaller teams with long-term team members, so perhaps that's why.
Our tables are wide because law require we store the data for 10 years, and it's easier and faster to store things in a single row, rather than spread out over several dozen detail tables.
We don't denormalize in the sense of storing only invoice items, repeating the head data on each item. We do denormalize in the sense of storing full name and address for each party rather than storing that in a detail table. So minimal, if any, duplication of data. If we duplicate it's almost always due to the law, ie data might change in one table but shouldn't be changed in the other to preserve history.
For child tables we always have a unique ID, sequence or autoinc, and the FK to the parent, but we include the root-level ID as well. This way we can select all the child rows in a single select regardless of level, again without tons of joins.
This way our core data is just a handful of tables, and there's not much doubt about what goes where.
I feel you... I mean "...you should be firing people", this is my day to day way of thinking.
My thoughts are that, hyper-specialization, and grind breed this type of data structure. But so many companies are forced to choose, and generally tend to sacrifice on the database side of things. Then you end up with this type of unruly structure.
Database theory and practice should be a MUST on all software development courseware.
My thoughts are that, hyper-specialization, and grind breed this type of data structure. But so many companies are forced to choose, and generally tend to sacrifice on the database side of things. Then you end up with this type of unruly structure.
Database theory and practice should be a MUST on all software development courseware.
It's honestly relieving to read all these stories because I feel like I am in the middle of this right now. Our legacy product is a mishmash of perl and php that was probably started in the early 2000s. Which I wouldn't feel bad about supporting if it wasn't for the new system that has been bolted on top of it. The previous team that started the migration is all gone and now we are halfway between two two systems that we have to support. On top of that my manager is afraid to say no to our eccentric CEO so we are flooded with new feature requests but can never get anything done because we spend so much time chasing down problems in the old system. Is there any sane way out of this? I just feel like I am white-knuckling it through every day.
Get you, your manager and the CEO in one room, and tell them the facts. Once those are on the table, discuss solutions. Otherwise nobody wins.
I loved reading this article. It made me laugh and cry and rage scream all at the same time. It is a true miracle that most things work most of the time. Don't look behind the curtains fowks! It might give you nightmares.
Elasticsearch and OpenSearch have a similar issue. They have a soft limit on the number of "searchable" fields (called "mapping") a table can have (by the way, a table is called an "index" in their terminology. How confusing!), which is 1000. We ran into this problem because we tried to jam all the logs in every single microservice into a single table. Each microservice had a different log format, so after combining them all the number of different fields surged. We did it this way because the operations team wanted to maintain a single table, by the name of simplicity, which to this day is a reason I can't completely fathom. We were surprised because we thought Elasticsearch and OpenSearch were some kind of magic box that can somehow ingest all the data we put yet still are perfectly performant. After that incident, we introduced a common log format that applies to every microservice.
[deleted]
This is obviously a work of fiction. Not because there isn't bad code similae to this out there, but because the way it is told and many of the details don't add up and have all the hallmarks of fabrication.
My favorite, obligatory reference when we discuss horror of legacy codebase: https://news.ycombinator.com/item?id=18442941
sure it might be a mess, but at least it's purpose built. I love that kind of performance gains. Honestly most companies die before purpose built code like that becomes a problem.
Truly, the Tao was alive in that company.
https://www.mit.edu/~xela/tao.html
https://www.mit.edu/~xela/tao.html
You ship your org chart.
If you have a messy app, you have a messy organisation.
If you have a messy app, you have a messy organisation.
This codebase sold for $4.3 billion. That table was cursed.
Jira's now discontinued server version had a sequence table to stop you sharding it. It also made disaster recovery from a hard shutdown awful. I have nothing good to say about Atlassian.
Atlassian, and JIRA specifically, are responsible for so much wasted time and capital expenditure. If I could get ahold of metrics like "hours spent building, using, and maintaining JIRA" versus "value obtained from JIRA" for each of the companies I've worked at, I'm pretty sure I could generate a report so scathing that nobody would ever use it again.
Gatekeeping your work organization system from the people working on, and often MANAGING it is such a huge friction point I'm amazed they ever got clientele. I'm an Admin on a project right now and I can't change our status types without going through our assigned Atlassian rep.
And don't even get me started on the dumpster fire that is BitBucket. Ever tried to use the API? it's somehow even more worthless than the UI.
Gatekeeping your work organization system from the people working on, and often MANAGING it is such a huge friction point I'm amazed they ever got clientele. I'm an Admin on a project right now and I can't change our status types without going through our assigned Atlassian rep.
And don't even get me started on the dumpster fire that is BitBucket. Ever tried to use the API? it's somehow even more worthless than the UI.
Man, this post is gold. I think we all have our own version of the best worst codebase.
My take on this story was a backend "deal capture" system for an emissions trading desk about 25 years ago. The application was written in classic ASP, T-SQL and VB6 COM components. Oh, and VBScript. It only ran in Internet Explorer 6.
The heart of the machine was a massive, unholy stored procedure. It was about 35kb, and what it did was generate a very long string which was - you guessed it - a gigantic SQL statement with countless JOINs across many temporary tables.
The business logic was deemed sophisticated enough that it was decided we needed a workflow engine and some genius decided that we should use an expensive proprietary enterprise-y tool which used Microsoft Exchange Server as its datastore and operating environment. This was as terrible an idea as it sounds, because the "API" to talk to this engine was single-threaded and massively bottlenecked by whatever made Exchange Server suck. Most significantly, it also could only be properly accessed by a very specific version of a DLL, requiring that a server setup follow an agonizingly specific software installation procedure that was scientifically proven to be impossible for IT to execute because they would always jump to the end and install the latest version of the service packs and suddenly the DLL we needed would be impossible to access without wiping the machine and starting over. This is what people meant when you hear the term "DLL Hell".
The most interesting aspect of this system was the client UX. This was 1999, and Ajax was literally not a thing yet. However, the precursor to XmlHttpRequest was an IE-only thing called ActiveXObject. You could use VBScript to wire up form elements to be updated by an ActiveXObject in the way we'd consider familiar today, except that instead of FORM GET/POST, your ActiveXObject would talk to a VB6 COM+ DLL running inside of Microsoft Transaction Server. Looking back on it now, the whole thing was simultaneously an abomination and years ahead of its time. The security profile would likely give a modern developer night terrors; I'm pretty sure that if you were on a page that was being served by an IIS instance that had COM+ registered, your ActiveXObject could call methods on that object so long as the VBScript had the UUID that identified the COM+ object. End of story. Just wow.
Final detail that my memory retains is that the room full of chain smoking women who operated this system did not use mice. That is, they were entering data almost exclusively through the number pad and furiously slamming the tab key to move to the next field. They would develop intense muscle memory for the 200-300 fields on this form. They needed repeatability or it would literally break them. Thing is, there were many instances where they didn't want to fill fields in the order they are defined in the HTML, so over time about 90% of those form elements gained in-line event handlers, again, written in VBScript, which replaced the built-in tab key handling functionality of the element with whatever was deemed the correct next step in their insanely specific process. If we accidentally broke their expectation, there would be hell to pay.
That is, we would login to the live server with VNC and live edit the changes into the codebase while they were on the phone over the remote screen share connection.
There was no source control, and if there were backups, I'm dubious that they would have been restorable.
My take on this story was a backend "deal capture" system for an emissions trading desk about 25 years ago. The application was written in classic ASP, T-SQL and VB6 COM components. Oh, and VBScript. It only ran in Internet Explorer 6.
The heart of the machine was a massive, unholy stored procedure. It was about 35kb, and what it did was generate a very long string which was - you guessed it - a gigantic SQL statement with countless JOINs across many temporary tables.
The business logic was deemed sophisticated enough that it was decided we needed a workflow engine and some genius decided that we should use an expensive proprietary enterprise-y tool which used Microsoft Exchange Server as its datastore and operating environment. This was as terrible an idea as it sounds, because the "API" to talk to this engine was single-threaded and massively bottlenecked by whatever made Exchange Server suck. Most significantly, it also could only be properly accessed by a very specific version of a DLL, requiring that a server setup follow an agonizingly specific software installation procedure that was scientifically proven to be impossible for IT to execute because they would always jump to the end and install the latest version of the service packs and suddenly the DLL we needed would be impossible to access without wiping the machine and starting over. This is what people meant when you hear the term "DLL Hell".
The most interesting aspect of this system was the client UX. This was 1999, and Ajax was literally not a thing yet. However, the precursor to XmlHttpRequest was an IE-only thing called ActiveXObject. You could use VBScript to wire up form elements to be updated by an ActiveXObject in the way we'd consider familiar today, except that instead of FORM GET/POST, your ActiveXObject would talk to a VB6 COM+ DLL running inside of Microsoft Transaction Server. Looking back on it now, the whole thing was simultaneously an abomination and years ahead of its time. The security profile would likely give a modern developer night terrors; I'm pretty sure that if you were on a page that was being served by an IIS instance that had COM+ registered, your ActiveXObject could call methods on that object so long as the VBScript had the UUID that identified the COM+ object. End of story. Just wow.
Final detail that my memory retains is that the room full of chain smoking women who operated this system did not use mice. That is, they were entering data almost exclusively through the number pad and furiously slamming the tab key to move to the next field. They would develop intense muscle memory for the 200-300 fields on this form. They needed repeatability or it would literally break them. Thing is, there were many instances where they didn't want to fill fields in the order they are defined in the HTML, so over time about 90% of those form elements gained in-line event handlers, again, written in VBScript, which replaced the built-in tab key handling functionality of the element with whatever was deemed the correct next step in their insanely specific process. If we accidentally broke their expectation, there would be hell to pay.
That is, we would login to the live server with VNC and live edit the changes into the codebase while they were on the phone over the remote screen share connection.
There was no source control, and if there were backups, I'm dubious that they would have been restorable.
In SQL Server, you're likely to run into the 8000 byte row size limit before the column count limit. Ask me how I know.
I know about the limits on max column number per table. The other day I was thinking about the best table layout for AirTable like systems, I finally decided to use relational tables with json fields. E.g. row is a several fields (id, name, who fields) and one json field where I can put any number of fields. Hopefully, this is going to be the best from both worlds and work good from performance view.
That is why people these days tend to use a single JSON blob instead of multiple columns. And because it is so popular, SQLITE and other DBs are building better and better JSON support into the DB.
I wonder if better support of EAV tables would solve this issue better.
If one could do "SELECT price,color,year FROM cars! WHERE status='sold'" and the "!" would indicate that cars is an EAV table ...
I wonder if better support of EAV tables would solve this issue better.
If one could do "SELECT price,color,year FROM cars! WHERE status='sold'" and the "!" would indicate that cars is an EAV table ...
entity attribute value
1 price 20750
1 color red
1 year 2010
1 status sold
20750 red 2010
Have you looked at the "hstore" type in Postgres? It seems to cover that use case.
First real job, stayed for 10yrs, 5yrs too long, put VB6 COM objects into the asp.net codebase.
> went by the name Munch
How do you pronounce that?
Was it like the word munch in “munching on some snacks”?
Or like the name of the painter Edward Munch? https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 (note: this link is in Norwegian)
How do you pronounce that?
Was it like the word munch in “munching on some snacks”?
Or like the name of the painter Edward Munch? https://www.nrk.no/kultur/nrk-endrer-munch-uttale-1.539667 (note: this link is in Norwegian)
As in munching on snacks
Hey former colleague - just had to say hello on a comment where you might see.
I started reading this article and everything started feeling so familiar... as soon as you told me Munch was the resident shaman, everything clicked.
My favorite factoid for others was that when I was there, we used split-horizon DNS to squat on an in-use domain name for tons of internal services, including Github. I kept wondering what would happen if the owner realized & set up his own services to catch people who weren't on the VPN.
My favorite factoid for others was that when I was there, we used split-horizon DNS to squat on an in-use domain name for tons of internal services, including Github. I kept wondering what would happen if the owner realized & set up his own services to catch people who weren't on the VPN.
::Edvard Munch scream emoji::
What a beautiful, humorously written, bitter-sweet piece of writing!
[deleted]
[deleted]
When I was still in college studying Computer Science (~2001) I got a part time job at a Medical Claims clearing house doing VB and Java programming. My task was to re-write in Java the ancient and archaic system which would read in medical claims that were uploaded by providers and then translate them into the one of several formats, depending on which insurance provider we were sending them to. The inputs were big batches of data in one of two formats, "ASC X12"¹ or "NSF"² and the outputs were some ideocyncratic version of those two, or a 3rd option which was just ascii text, layed out (monospace) such that it would line up when you dump the file directly to a printer loaded with the pre-printed UB-92³ forms.
None of this is particularly interesting, unless you have a fascination with archaic file formats and/or an interest in historical and highly idiosyncratic government bureaucracy.
The really interesting (horrifying) thing about the job, though, was the state of the VB6 code which I was asked to translate into well structured and performant Java. There were some really hilarious and nightmare inducing subroutines like "BunchOfIfs" and "BigSelect", each of these monsters were several thousand lines long and consisted of exactly what you'd expect. Huge branching structures with absolutely no reasonable organization or design. I'm not even exaggerating to say it was just the accumulated cruft of 10 years of dirty hacks tacked on by the cheapest coders they could find where the only standards were if it works it ships. Literally the worst procedural code you can imagine with zero factorization, modularization, plenty of duplicated code copied and pasted then modified to do something slightly different than the 3 other versions of similar code elsewhere in the project.
Somehow, after a year of part-time work (20 hours a week, between classes) I managed to produce a working system to translate claims from any one of the weird formats into any other one of the weird formats, including 5 or 6 variants of said formats, each one which violated the spec in a unique way in order to satisfy the strange requirements of some particular insurance company. The Java version was less than 10% the size (lines of code) of the old system, ran 30x faster and produced correct output.
Still to this day it's objectively the most difficult, painstaking, excruciating, but also probably the best, most impressive work I've done. And it's the least I've ever been paid for writing code.
Oh and I forgot to mention, nothing was in source control and there were several variants of the codebase that had been modified slightly to do a similar but different task and then just continued to drift away from the codebase it was copied from.
1. https://en.wikipedia.org/wiki/ASC_X12 2. https://manuals.momed.com/edb_pdf/NSF%20(National%20Standard... 3. https://owcprx.dol.gov/static/UB-92.pdf
None of this is particularly interesting, unless you have a fascination with archaic file formats and/or an interest in historical and highly idiosyncratic government bureaucracy.
The really interesting (horrifying) thing about the job, though, was the state of the VB6 code which I was asked to translate into well structured and performant Java. There were some really hilarious and nightmare inducing subroutines like "BunchOfIfs" and "BigSelect", each of these monsters were several thousand lines long and consisted of exactly what you'd expect. Huge branching structures with absolutely no reasonable organization or design. I'm not even exaggerating to say it was just the accumulated cruft of 10 years of dirty hacks tacked on by the cheapest coders they could find where the only standards were if it works it ships. Literally the worst procedural code you can imagine with zero factorization, modularization, plenty of duplicated code copied and pasted then modified to do something slightly different than the 3 other versions of similar code elsewhere in the project.
Somehow, after a year of part-time work (20 hours a week, between classes) I managed to produce a working system to translate claims from any one of the weird formats into any other one of the weird formats, including 5 or 6 variants of said formats, each one which violated the spec in a unique way in order to satisfy the strange requirements of some particular insurance company. The Java version was less than 10% the size (lines of code) of the old system, ran 30x faster and produced correct output.
Still to this day it's objectively the most difficult, painstaking, excruciating, but also probably the best, most impressive work I've done. And it's the least I've ever been paid for writing code.
Oh and I forgot to mention, nothing was in source control and there were several variants of the codebase that had been modified slightly to do a similar but different task and then just continued to drift away from the codebase it was copied from.
1. https://en.wikipedia.org/wiki/ASC_X12 2. https://manuals.momed.com/edb_pdf/NSF%20(National%20Standard... 3. https://owcprx.dol.gov/static/UB-92.pdf
In my very first programming job, I got hired on right at the end of a major refactor that had been going on for the better part of a year. I was a hacker kid who started programming less than a year prior. My first task at this company was to merge and resolve all conflicts from the refactored branch back into mainline. No one checked what I did. As long as it compiled on my machine and passed the CI test (just one), it got merged and released into production.
Shockingly, this project was notorious for regressions. We were on a weekly(!) update cycle, and we constantly had bugs reappear that had been solved months prior.
This was 2010 or so, and we were using SVN because the project lead didn't trust or understand git. He also didn't understand SVN. The solution to our constant regressions was pretty simple. Instead of merging branches into trunk, we would delete the master branch every week and create a new one out of all of the finished develop branches.
Surprising absolutely nobody apart from that project manager, this plan was a spectacular failure.
He also stole code out of my personal git repos from before I worked there, and bragged about how smart he was for stealing my code. So, y'know, just a general idiot and asshat.
Shockingly, this project was notorious for regressions. We were on a weekly(!) update cycle, and we constantly had bugs reappear that had been solved months prior.
This was 2010 or so, and we were using SVN because the project lead didn't trust or understand git. He also didn't understand SVN. The solution to our constant regressions was pretty simple. Instead of merging branches into trunk, we would delete the master branch every week and create a new one out of all of the finished develop branches.
Surprising absolutely nobody apart from that project manager, this plan was a spectacular failure.
He also stole code out of my personal git repos from before I worked there, and bragged about how smart he was for stealing my code. So, y'know, just a general idiot and asshat.
In case anyone is curious about what the proper solution to some of these problems is in modern SQL Server:
1) Many columns can happen with per-customer customisations to a shared table. The common way is to have a "customcolumns" table with "ID,ColumnName,ColumnValue" linked to the base table that has an "ID" key, but SQL Server also supports this natively now with Sparse Columns: https://learn.microsoft.com/en-us/sql/relational-databases/t...
2) Shared identity or globally sequential numbers have a built-in schema object type now: https://learn.microsoft.com/en-us/sql/t-sql/statements/creat...
3) Manually populated calendar tables are actually a common schema design pattern, especially in manufacturing, shipping, and OLAP reporting systems. This is not that bizarre, it's just a bit weird that it would break logins! These tables can let you define all sorts of things such as international holidays, manufacturing calendars, tax years, finance reporting schedules, etc...
4) Dropping and recreating tables is also common, albeit usually done in a transaction. The fancy way to do this is with partition switching, where a new table is populated from whatever (external data, business logic, etc...) and then instantly swapped for the original without any long-running operations like truncate & insert would. See: https://pragmaticworks.com/blog/table-partitioning-in-sql-se...
5) Delayed reporting replicas ("here was a copy of the database. Data in this copy was about 10 minutes out of date.") is also a common pattern. At this point, the blog author is just complaining about the realities of business databases. Typically you'd have a bunch of read only replicas with different delays: Synchronous for HA failover, Asynchronous for DR failover and real-time reporting, and deliberately delayed on a schedule ETL copies for "point in time" consistent reporting. These would typically be done at 3am or something to minimise inconsistencies in the data. The modern way to do this is a SQL Server Always On readable secondary: https://learn.microsoft.com/en-us/sql/database-engine/availa...
6) "The main codebase I worked in was half VB, half C#." this is also surprisingly common. It's often not worth rewriting old code, there's not enough return on the invested money. These days there are automated conversion tools for VB to C#, such as: https://marketplace.visualstudio.com/items?itemName=SharpDev...
7) The Gilfoyle character is honestly the only thing that stands out here as an actual problem, not just a typical thing that happens in large enterprise bespoke software development.
1) Many columns can happen with per-customer customisations to a shared table. The common way is to have a "customcolumns" table with "ID,ColumnName,ColumnValue" linked to the base table that has an "ID" key, but SQL Server also supports this natively now with Sparse Columns: https://learn.microsoft.com/en-us/sql/relational-databases/t...
2) Shared identity or globally sequential numbers have a built-in schema object type now: https://learn.microsoft.com/en-us/sql/t-sql/statements/creat...
3) Manually populated calendar tables are actually a common schema design pattern, especially in manufacturing, shipping, and OLAP reporting systems. This is not that bizarre, it's just a bit weird that it would break logins! These tables can let you define all sorts of things such as international holidays, manufacturing calendars, tax years, finance reporting schedules, etc...
4) Dropping and recreating tables is also common, albeit usually done in a transaction. The fancy way to do this is with partition switching, where a new table is populated from whatever (external data, business logic, etc...) and then instantly swapped for the original without any long-running operations like truncate & insert would. See: https://pragmaticworks.com/blog/table-partitioning-in-sql-se...
5) Delayed reporting replicas ("here was a copy of the database. Data in this copy was about 10 minutes out of date.") is also a common pattern. At this point, the blog author is just complaining about the realities of business databases. Typically you'd have a bunch of read only replicas with different delays: Synchronous for HA failover, Asynchronous for DR failover and real-time reporting, and deliberately delayed on a schedule ETL copies for "point in time" consistent reporting. These would typically be done at 3am or something to minimise inconsistencies in the data. The modern way to do this is a SQL Server Always On readable secondary: https://learn.microsoft.com/en-us/sql/database-engine/availa...
6) "The main codebase I worked in was half VB, half C#." this is also surprisingly common. It's often not worth rewriting old code, there's not enough return on the invested money. These days there are automated conversion tools for VB to C#, such as: https://marketplace.visualstudio.com/items?itemName=SharpDev...
7) The Gilfoyle character is honestly the only thing that stands out here as an actual problem, not just a typical thing that happens in large enterprise bespoke software development.
"Merchants2 was the solution."
"When you can't tie a knot, tie a lot"
At some point, the application had some bugs which were not appearing when the application was run in debug mode in Visual Studio. The solution was obvious: installing Visual Studio for each customer on site and teaching the users to run the app in debug mode from Visual Studio. I don't know how they convinced the users to do this and how they managed with the license but it was done.
What happened next was even worse.
There was no version control of course, the code being available on a shared disk on the local network of the company with the code copied over in multiple folders each having its own version, with no particular logic to it either, V1, V2, V2.3, V2a, V2_customer_name, V2_customer_name_fix, ...
After that, when there was a problem for a customer, the programmer went there to debug and modified the code on site. If the bug/problem was impacting other customers, we had to dispatch some guys for each customer to go copy/edit the code for all of them. But if the problem was minor, it was just modified there, and probably saved on the shared folder in some new folder.
What happened next was to be expected: there was no consensus on what was the final version, each customer having slightly different versions, with some still having bugs fixed years before for others.