Ask HN: What costly inefficiency do you see at work?
I'm 95% curious, 5% hoping some HN reader gets a startup out of it.
66 comments
Countless AWS instances with less than 1% utilization costing millions of dollars a year. It's easier to raise more money than try to fix it, so it just keeps getting worse.
Focusing on pumping vanity 'growth' metrics when our product has negative gross margins. We have no idea if anyone likes using our product or if they're just using us to buy dollar bills for $0.97. However management is ego-driven (we're going to "change the world") and it's easier to raise another round than to make a profit.
Focusing on pumping vanity 'growth' metrics when our product has negative gross margins. We have no idea if anyone likes using our product or if they're just using us to buy dollar bills for $0.97. However management is ego-driven (we're going to "change the world") and it's easier to raise another round than to make a profit.
The first one is even valid on a personal level. (well at a smaller scale). There was a big thing on NPR a year ago about this, what they called "zombie money" or something like that. I know I have various AWS/DO/Heroku/Namecheap/GApps/Dropbox accounts that I've been paying for for years and haven't canceled "just in case". Granted I've cleaned up a lot as it used to be 2-3x that. I have to imagine I'm not the only one.
This is actually possibly a good problem for a real startup. Though somewhat technical, there's certainly quite a bit of tedium there, per http://www.paulgraham.com/schlep.html, so it's potentially really profitable.
This is actually possibly a good problem for a real startup. Though somewhat technical, there's certainly quite a bit of tedium there, per http://www.paulgraham.com/schlep.html, so it's potentially really profitable.
Do you see the work being converting standard EC2/Heroku/DO... servers to lambda/cloud functions?
> or if they're just using us to buy dollar bills for $0.97
This kind of thing confuses me. I've heard similar thoughts before, but as someone out of the loop, aren't investors still expecting a return on their investment? Sure VCs have high toleration of risk but they need at least one or two of their companies to make money eventually, right? If investors really are buying dollars for $.97 this seems like the best indication yet that we are in a huge tech bubble.
This kind of thing confuses me. I've heard similar thoughts before, but as someone out of the loop, aren't investors still expecting a return on their investment? Sure VCs have high toleration of risk but they need at least one or two of their companies to make money eventually, right? If investors really are buying dollars for $.97 this seems like the best indication yet that we are in a huge tech bubble.
It's a big company and there are lots of other products and opportunities that have positive gross margins. Just not the product my office primarily works on.
I'm not a backend dev, but it feels like something lambda is trying to partially address.
Government: money expires after two years. Which means that if you come in under budget, you have to find something else to spend the money on - which is sometimes needed infrastructure, like upgraded computers, but is more often whatever happens to be in stock at Supply. Or you dump it on a contract with a services contractor with whom you already have a contract vehicle in place. Or you lose it. But you can't save it. You can't hold onto it to do more work on the project after the two years is up.
You also can't extend the deadline for project completion in case of unforeseen events. If you run out of time, but not money, you just have to stop, because the money evaporates.
You also can't extend the deadline for project completion in case of unforeseen events. If you run out of time, but not money, you just have to stop, because the money evaporates.
A friend of mine was a radio tech and he was ordered to put in a 1.2mil PO for a new transmitter because a wire was frayed. They could have just replaced the wire for 20$, but then they would have lost that 1.2mil off the next year's budget.
[deleted]
It's not just the government. I was brought into building out an operations budget at a startup a few years back, and found out that it was happening there too.
Yes, the startup is still going.
Yes, the startup is still going.
Corporate IT - which is now it's own behemoth in most modern organizations. IMHO it's actively preventing any work getting done. There are tons of startups that are meeting this need (generally referred to as "dark IT" or "shadow IT").
More power to you if you find yourself actually meeting business needs over the company's own internal programmers and systems folks.
I am, of course, trying to effect change from within the org I am employed by.
More power to you if you find yourself actually meeting business needs over the company's own internal programmers and systems folks.
I am, of course, trying to effect change from within the org I am employed by.
> Corporate IT - which is now it's own behemoth in most modern organizations. IMHO it's actively preventing any work getting done
I will co-sign to this every day of the week and twice on sundays. In our org chart, developers are not "IT" and getting nearly anything done that requires involvement from IT borders on impossible.
I honestly will probably never work for this sort of Microsoft/Oracle/IBM shop ever again unless I get a blood promise I can use a Mac or Linux workstation, get on-demand development VM creation, and have absolutely 0 Oracle product I'm required to integrate with.
The org I work at is so old that it still has groups with "Wintel" in their names, but the UNIX guys (the majority of our non-windoze stuff is AIX) aren't any better. It literally feels like I'm in a time warp back in the 1990s.
Some horror stories:
- having a systems engineer tell me to put in a ticket for the purpose of making a phone call to discuss something that had no pending action needed on his part
- having a DBA tell me the same thing
- having a DBA not know how to turn on query logging
- getting told we had to put together a hardware acquisition plan to do an in-place upgrade of database modules
- having a ticket to get log rotation turned on ignored for months and then told (when logs filled up the disk) that they'd have to turn off the service
- tediously going through change control to alter permissions, elevation rights, etc., only to have "implemented" changes mysteriously disappear the next time security patches were applied
- requesting and getting access to some very nice and expensive commercial applications only to be told that our use case was unsupported
- a half-dozen vendors' antivirus clients on windows machines
- unplanned outages during non-brownout/storm conditions corrupting databases
Ironically enough, some of the PMs (who are generally non-technical) we've talked to in IT literally complain about the exact same sort of problems.
The upside is I can take two or three hour lunch breaks whenever I have to wait on these guys, which is a couple times a week.
I will co-sign to this every day of the week and twice on sundays. In our org chart, developers are not "IT" and getting nearly anything done that requires involvement from IT borders on impossible.
I honestly will probably never work for this sort of Microsoft/Oracle/IBM shop ever again unless I get a blood promise I can use a Mac or Linux workstation, get on-demand development VM creation, and have absolutely 0 Oracle product I'm required to integrate with.
The org I work at is so old that it still has groups with "Wintel" in their names, but the UNIX guys (the majority of our non-windoze stuff is AIX) aren't any better. It literally feels like I'm in a time warp back in the 1990s.
Some horror stories:
- having a systems engineer tell me to put in a ticket for the purpose of making a phone call to discuss something that had no pending action needed on his part
- having a DBA tell me the same thing
- having a DBA not know how to turn on query logging
- getting told we had to put together a hardware acquisition plan to do an in-place upgrade of database modules
- having a ticket to get log rotation turned on ignored for months and then told (when logs filled up the disk) that they'd have to turn off the service
- tediously going through change control to alter permissions, elevation rights, etc., only to have "implemented" changes mysteriously disappear the next time security patches were applied
- requesting and getting access to some very nice and expensive commercial applications only to be told that our use case was unsupported
- a half-dozen vendors' antivirus clients on windows machines
- unplanned outages during non-brownout/storm conditions corrupting databases
Ironically enough, some of the PMs (who are generally non-technical) we've talked to in IT literally complain about the exact same sort of problems.
The upside is I can take two or three hour lunch breaks whenever I have to wait on these guys, which is a couple times a week.
Somewhat related. A fair amount of the cloud initiatives I've seen within Fortune 500's aren't really about the cloud at all. It's solely to get to a data center where the provider does what you ask in a reasonable amount of time without overzealous and slow approval flows. "Shadow Data Center" I guess.
This applies to pretty much any institutional IT (non-profit, education, etc). We waste an incredible amount of time and money badly maintaining systems just for hosting websites, for example. And not for any good reason, mind you--our systems are less convenient, less secure, and far more expensive than any competent third party host would be.
The time it takes to get a host set up is absolutely mind-boggling. There's talk about making changes, but it takes years to overcome years of bad leadership and apathy, and there's institutional inertia that actively works against modernizing.
The time it takes to get a host set up is absolutely mind-boggling. There's talk about making changes, but it takes years to overcome years of bad leadership and apathy, and there's institutional inertia that actively works against modernizing.
Corporate IT is a disaster. So many people responsible for only one thing, causing them not to think outside their box. Going from one conference call to the other without any dedicated focus on the project. Instead of getting an application live in several days, it costs months. Slow communication, cheap outsourcing so in the end you need to do constant hand-holding to complete installation of an web application.
Does this sound familiar to anyone?
Does this sound familiar to anyone?
Absolutely, the more layers the worse it gets. It's a version of Chinese whispers/Telephone where not only is confusion added at each stage but huge amounts of time delay.
This is spot on.
It's bad enough that money is saved within IT departments that costs multiples of wasted staff wages in the business, but there are battles between sub sections of IT that try to save money in their area by pushing it on to others (again at net cost).
It's bad enough that money is saved within IT departments that costs multiples of wasted staff wages in the business, but there are battles between sub sections of IT that try to save money in their area by pushing it on to others (again at net cost).
Spending more money on AWS services than we'd spend hosting them ourselves, spending more time beating our head against insufficient documentation of proprietary Amazon services and wrappers compared to their corresponding open source technologies, spending more time debugging and on general uncertainty, spending more time waiting for responses from another company, having to stop what we're doing and handle backward incompatibility on Amazon's schedule instead of our own, spending more developer hours and fewer sysadmin hours on managing and debugging services instead of more sysadmin hours and fewer developer hours on our own services. (Developer hours are more costly and scarce.)
I'm seeing this too. We end up spending a lot of developer hours trying to figure out AWS specific quirks instead of just using generic and well proven open source tools. And this doesn't count the insane monthly AWS costs.
"How do I authenticate a Github webhook in the API Gateway?" "How much should I provision for DynamoDB?" "Why is this IAM role not working?"
"How do I authenticate a Github webhook in the API Gateway?" "How much should I provision for DynamoDB?" "Why is this IAM role not working?"
I hear the AWS argument quite a bit, but am not quite sure if the costs of AWS vs. independent infrastructure have been appropriately analyzed by the person making the accusation, in many cases. Independent infrastructure ages and requires updating as time goes on. Amazon normalizes those costs and provides the billed entity with semi-predictable costs on an ongoing basis. Keep in mind the costs of power, cooling, and bandwidth as well.
Good accountants can amortize the costs of servers (etc.) just as well as AWS can.
> Keep in mind the costs of power, cooling, and bandwidth as well.
And it will still probably be cheaper to host your own, with one caveat: if you have wildly varying traffic loads, AWS' autoscaling is hard to beat. The reality is, though, most companies don't have traffic loads which vary enough to justify the extra costs.
> Keep in mind the costs of power, cooling, and bandwidth as well.
And it will still probably be cheaper to host your own, with one caveat: if you have wildly varying traffic loads, AWS' autoscaling is hard to beat. The reality is, though, most companies don't have traffic loads which vary enough to justify the extra costs.
> I hear the AWS argument quite a bit, but am not quite sure if the costs of AWS vs. independent infrastructure have been appropriately analyzed by the person making the accusation, in many cases.
I would pay around 8x times more if I would use AWS instead of dedicated servers.
> in many cases. Independent infrastructure ages and requires updating as time goes on.
That's why you use dedicated servers unless you are really huge (then you go with your hardware) or have specific requirements. In 99% other cases you use dedicated servers which are refreshed by data centers with newer hardware then you just switch (if you really need it...).
I would pay around 8x times more if I would use AWS instead of dedicated servers.
> in many cases. Independent infrastructure ages and requires updating as time goes on.
That's why you use dedicated servers unless you are really huge (then you go with your hardware) or have specific requirements. In 99% other cases you use dedicated servers which are refreshed by data centers with newer hardware then you just switch (if you really need it...).
Seen it at previous work places. Focusing too much the agile boards and not enough on engineering. Then hire expensive
Consultants because you're hitting the wall.
I've seen this many times. I think it usually comes down to whoever designed the initial architecture. If you have a bad foundation there's no saving you. Many times the architect had political influence and overrode general censuses.
I worked a project with a public company that had a 1:1 dev / scrum master ratio. Then they brought on consultants for a straight week for full time training to fix the lack of progress. You can't make these things up..
I worked a project with a public company that had a 1:1 dev / scrum master ratio. Then they brought on consultants for a straight week for full time training to fix the lack of progress. You can't make these things up..
Putting product in charge of engineering instead of making is a partnership.
Then not holding product accountable for an inspired roadmap that is delivered 2 months into the half.
My team is working way below peak efficiency for 4 months a year as a result. The engineers are super unhappy about this but there is "nothing to be done" according to my boss and his bosses boss.
Then not holding product accountable for an inspired roadmap that is delivered 2 months into the half.
My team is working way below peak efficiency for 4 months a year as a result. The engineers are super unhappy about this but there is "nothing to be done" according to my boss and his bosses boss.
Do we work together?? /s
But yes, the amount of reversions, refactors and plain jane "we gotta rebuild this" has lead to some real fatigue at least from me.
But yes, the amount of reversions, refactors and plain jane "we gotta rebuild this" has lead to some real fatigue at least from me.
Our department gets a few thousand projects a year (work in large format printing). The amount of repetitive steps that each project share is simply mind boggling. By writing 10 or so scripts, was able to shave department labor hours down by 14%.
Excel... how people use excel and the amount of waste that happens just because they don't properly know how to use 'vlookup' or 'if' simply just boils my blood.
Excel... how people use excel and the amount of waste that happens just because they don't properly know how to use 'vlookup' or 'if' simply just boils my blood.
Excel is the universal format for the enterprise.
Need to share a screen shot of your error? Paste it into excel.
Project management? Excel.
Database of assets? Excel.
Transactional information? Excel.
Incredibly complex, multi page calculations? Excel.
Kill me.
Need to share a screen shot of your error? Paste it into excel.
Project management? Excel.
Database of assets? Excel.
Transactional information? Excel.
Incredibly complex, multi page calculations? Excel.
Kill me.
I once saw a very detailed Gantt chart in MS Project detailing 6 months worth of lies about various projects would deliver.
My colleague nudged me out of a stupor as I stared at the overhead projection. "Its Excel!" He whispered. "What are you on about?" I said. He was clearly mad. "The chart! He's done it in Excel". To my horror I began to see. This wasn't MS Project at all. It was subtle because the cells were so small they were almost no bigger than a pixel. I gasped and almost took the presentation off course.
It was like a mosaic. A feat. I've never seen anything like it since. Still baffles me how much effort he must have put in and the pain he must've felt every time that crappy plan inevitably moved to the right.
My colleague nudged me out of a stupor as I stared at the overhead projection. "Its Excel!" He whispered. "What are you on about?" I said. He was clearly mad. "The chart! He's done it in Excel". To my horror I began to see. This wasn't MS Project at all. It was subtle because the cells were so small they were almost no bigger than a pixel. I gasped and almost took the presentation off course.
It was like a mosaic. A feat. I've never seen anything like it since. Still baffles me how much effort he must have put in and the pain he must've felt every time that crappy plan inevitably moved to the right.
I'm not sure if we should commend, or condemn Microsoft for making a application that is so easily used by less than savvy end users.
I'm working for a big nonprofit that uses convio for web donations and raisersedge for data management but the two don't link up so I have to downland the csv from convio clean it up in excel and import into raisersedge. I run a large macro and do some cleaning manually too but would love to offer them a better solution. ..or I want to suggest better software because raisersedge is atrocious
I act as sort of a clearing house / accidental DB admin within my department. The amount of time I have to spend cleaning up people's Excel input data is absurd.
Internal transaction costs. My company has dozens of different offices who each invoice each other, sometimes in circles. We need to make internal markets easier to operate
Disposable utensils, cups, etc. in the break room and kitchen.
But i'm kind of a dirty hippie.
But i'm kind of a dirty hippie.
My office just got rid of these and I've been grumpy about it. I didn't consider the environmental angle though, assumed it was just corporate greed.
The only downside is that, at least once, some hapless intern is guaranteed to load the dishwasher with hand soap. I don't know why, but there's always one.
"Buy don't build" becoming such an embedded piece of wisdom that there are literally a half-dozen COTS appliances that do the same thing, but not inventorying them, to the point that you might have just one or more likely zero people using them. Literally no one watches our DLP tool, but we keep paying for it.
It's basically a field day for consultants and for appliance vendors. I'm not a huge fan of NIH but it's really demoralizing to see money get spent on toys that immediately get locked up in the attic.
It's basically a field day for consultants and for appliance vendors. I'm not a huge fan of NIH but it's really demoralizing to see money get spent on toys that immediately get locked up in the attic.
Meetings - I think anytime someone mentions they won't be able to get things done because of meetings, they should reassess if the meeting is needed at all.
The most costly inefficiencies I've seen are companies / organizations that are wrongly constituted to begin with.
For example, when you have a division of larger company that really should be its own company. But, no one's allowed to talk about that because "that's been talked to death already" and the show must go on, etc. Or because no on really cares (or bothers to think about things on that level) basically.
For example, when you have a division of larger company that really should be its own company. But, no one's allowed to talk about that because "that's been talked to death already" and the show must go on, etc. Or because no on really cares (or bothers to think about things on that level) basically.
This is a sign that you're working too far down the hierarchy. Problems like this can always be solved, if indeed it's a problem, if you go high enough up in the org chart.
One of the major causes of inefficiency is bad management--the cause behind many of the responses given here.
Incompetent managers up the chain don't have a coherent vision that moves us forward. They spend most of their time playing at politics, trying to look good, maintaining the status quo, and/or attempting to coalesce power/status. Even good managers get caught up in this game. It seems to be the nature of the system.
On my team, for example, we have too many projects, projects taken on for the wrong reasons, and projects that don't fit our team's skillset. Most of those projects aren't managed well or even managed at all.
Old projects molder with no real plan for fixing them, so we spend inordinate amounts of time on every change. New projects are announced with someone claiming that they'll be "easy", but there's no plan and things don't get done or they get done badly. Sometimes I sit here in pure bewilderment wondering what the point of it all is.
Is there some way a startup could tackle organizational/social issues? That seems like an interesting idea, but I can't think of how it might work.
Incompetent managers up the chain don't have a coherent vision that moves us forward. They spend most of their time playing at politics, trying to look good, maintaining the status quo, and/or attempting to coalesce power/status. Even good managers get caught up in this game. It seems to be the nature of the system.
On my team, for example, we have too many projects, projects taken on for the wrong reasons, and projects that don't fit our team's skillset. Most of those projects aren't managed well or even managed at all.
Old projects molder with no real plan for fixing them, so we spend inordinate amounts of time on every change. New projects are announced with someone claiming that they'll be "easy", but there's no plan and things don't get done or they get done badly. Sometimes I sit here in pure bewilderment wondering what the point of it all is.
Is there some way a startup could tackle organizational/social issues? That seems like an interesting idea, but I can't think of how it might work.
I believe the future of companies will be internal startups vying for investment dollars. With disruption challenging even the most traditional and stable industries, we'll think back and muse to ourselves as to how political and beauracratic we were.
That being said politics will always exist, but not the debilitating level (IMO) to which we've seen in the past.
That being said politics will always exist, but not the debilitating level (IMO) to which we've seen in the past.
Politics.
Using the too expensive AWS and Google Cloud where DO would have been totally fine.
People who can't admit their mistakes or even blame others for theirs.
Bad architectural decisions.
Kubernetes instead of Docker Swarm for small clusters without any traffic or just plain Docker.
Expensive Github instead of free Bitbucket.
Asana, Jira, Pivotal Tracker instead of free Trello.
Slack.
Self-hosting of non-core tools.
Team retreats nobody wants.
Team events nobody wants.
Team members with weak communication skills.
Teams which are not synced, where some come after lunch and some finish few minutes later.
Using the too expensive AWS and Google Cloud where DO would have been totally fine.
People who can't admit their mistakes or even blame others for theirs.
Bad architectural decisions.
Kubernetes instead of Docker Swarm for small clusters without any traffic or just plain Docker.
Expensive Github instead of free Bitbucket.
Asana, Jira, Pivotal Tracker instead of free Trello.
Slack.
Self-hosting of non-core tools.
Team retreats nobody wants.
Team events nobody wants.
Team members with weak communication skills.
Teams which are not synced, where some come after lunch and some finish few minutes later.
Competition. A lot of people generally like to harm their colleague's odds of success - they pass accountability, responsibility, work, force them to attend more meetings, force them to use more communication software. Basically trying to look good by making someone else look bad.
It would be very effective if everyone worked together towards a common goal. This really isn't an issue in fast growing companies but the slower the growth of a company is, the more destructive competition sinks in.
It would be very effective if everyone worked together towards a common goal. This really isn't an issue in fast growing companies but the slower the growth of a company is, the more destructive competition sinks in.
Mandated technology decisions. Strap in for a dozen man-years of technical debt on a platform your team doesn't understand but now must debug.
Building the main campus in SF Bay then formalizing a location pay differential, then SF Bay staff pestering the distributed talent to move to SF Bay and fly to SF Bay for every team bonding/etc until they quit or move to SF Bay (and then quit as they actually need double the salary just to get back to their previous standard of living.)
At big companies, just communication between departments. Startup idea: a "heroku" for enterprises, where companies own their own "heroku", and each division/group provides a "plugin" that they manage. Would make things much more consistent among departments, and integrating solutions so much easier.
Convox[1] does somewhat of what you're asking for. The simplest way to describe it is a roll your own PaaS.
[1] https://convox.com/
[1] https://convox.com/
Jupming on every framework as soon as it comes out - instead of stabilising technology to meet business needs
Not to start a flamewar, but... Cramming expensive engineers shoulder to shoulder in a giant distracting concrete box of an open office, yards from the sales team yammering on the phone all day.
builds that take longer than 2 seconds
Being unwilling to spend money on good tools, forcing everyone to work around the failings of sub-standard free tools. I'm talking about you, TestLink.
Putting product in charge of engineering, or engineering in charge of product.
Or engineering or product in charge of business.
Or engineering or product in charge of business.
Hacker News
Meeting up in person (mostly-remote company)
[deleted]
Undocumented legacy systems.
Union
The great cycle.. lets not code too much ourselves, lets only be experts at our domain, lets use librarys.
The librarys fail, they do not deliver the expected performance, they cant be customized or are suddenly not maintained and we can not port our whole ecosystem.
Lets write most of our code ourselves, get independent from external actors, but we are slow now, glacial and distributed to reinvent a lot of wheels, while others outpace use in our domain. We should use more librarys.
Lets write most of our code ourselves, get independent from external actors, but we are slow now, glacial and distributed to reinvent a lot of wheels, while others outpace use in our domain. We should use more librarys.
Sales force.