Software effort estimation is mostly fake research(shape-of-code.coding-guidelines.com)
shape-of-code.coding-guidelines.com
Software effort estimation is mostly fake research
http://shape-of-code.coding-guidelines.com/2021/01/17/software-effort-estimation-is-mostly-fake-research/
317 comments
The issue is that you are being asked to estimate something that has never been done before. Even houses always go over time and money and that is fairly straight forward.
These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an exact timeline. I really wish we would treat sales people the same way. "How much money is this contract for? What date will it be signed?"
These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years. Some relatively small number of those units. If you want a quote it will take an extra 1/4 of the estimate worth of time for an exact timeline. I really wish we would treat sales people the same way. "How much money is this contract for? What date will it be signed?"
> Even houses always go over time and money and that is fairly straight forward.
This is an incredibly important point. I was "raised" in commercial software at a time when there was a lot of condescension internally towards ourselves as an industry because we couldn't do estimation and planning, and building was the favorite comparison. A $100 million building couldn't run late or over budget. That would be unimaginable, and it doesn't happen because those people are too serious and professional to let it happen, not like us unserious, immature software doofuses. Turns out that was a complete lie, but at one time it warped the profession, because it made people believe the answers already existed, right around the corner, something you would figure out pretty soon just like last year you didn't know regular expressions and now you did.
The problem with that was that when somebody walked in with a button-down shirt, oozing confidence and saying they knew exactly how to consistently deliver software on time and bug-free, those people weren't laughed out of the room. Elaborate heavyweight processes, and their "lightweight" (hah!) cousins, could sell themselves as proven, established solutions. We could believe that bullshit because we had been fed the lie that other industries had conquered these problems.
This is an incredibly important point. I was "raised" in commercial software at a time when there was a lot of condescension internally towards ourselves as an industry because we couldn't do estimation and planning, and building was the favorite comparison. A $100 million building couldn't run late or over budget. That would be unimaginable, and it doesn't happen because those people are too serious and professional to let it happen, not like us unserious, immature software doofuses. Turns out that was a complete lie, but at one time it warped the profession, because it made people believe the answers already existed, right around the corner, something you would figure out pretty soon just like last year you didn't know regular expressions and now you did.
The problem with that was that when somebody walked in with a button-down shirt, oozing confidence and saying they knew exactly how to consistently deliver software on time and bug-free, those people weren't laughed out of the room. Elaborate heavyweight processes, and their "lightweight" (hah!) cousins, could sell themselves as proven, established solutions. We could believe that bullshit because we had been fed the lie that other industries had conquered these problems.
I've seen a lot of industries compared to programming, like construction and manufacturing. For example, people trying to apply lessons from the Toyota Production System to programming.
This doesn't make sense, because we have systematized the construction of executable programs with extremely high reliability and predictability: we call it "Copy File"
It's the design that is the hard part, and the process of coding is the process of iterating on the design. Even turning the design into an executable is extremely reliable, either running via a VM or scripting engine, or compiling. Those are all very repeatable and reliable.
There are other disciplines that focus on creating and improving designs (like writing, painting, architecture, 3D modeling), but it's not like they're reliable or predictable either, unless you're designing very similar things over and over again. That's true of coding as well, but you don't really end up doing that, because once you've done it 3 times you should make a library and then never do that part again, just the unique bits.
In short, yeah, coding is its own thing, and I doubt it'll ever be predictable unless you're wasting a lot of time and effort doing very similar things over and over.
This doesn't make sense, because we have systematized the construction of executable programs with extremely high reliability and predictability: we call it "Copy File"
It's the design that is the hard part, and the process of coding is the process of iterating on the design. Even turning the design into an executable is extremely reliable, either running via a VM or scripting engine, or compiling. Those are all very repeatable and reliable.
There are other disciplines that focus on creating and improving designs (like writing, painting, architecture, 3D modeling), but it's not like they're reliable or predictable either, unless you're designing very similar things over and over again. That's true of coding as well, but you don't really end up doing that, because once you've done it 3 times you should make a library and then never do that part again, just the unique bits.
In short, yeah, coding is its own thing, and I doubt it'll ever be predictable unless you're wasting a lot of time and effort doing very similar things over and over.
I like to compare programming to the marketing industry.
Highly creative work and when deadlines are involved the work becomes spurty (sprints) with corner cutting (scope reduction) to meet unrealistic and over committed deadlines.
Highly creative work and when deadlines are involved the work becomes spurty (sprints) with corner cutting (scope reduction) to meet unrealistic and over committed deadlines.
> It's the design that is the hard part
Design but also requirements. You don't really know what should be built until you have a prototype. And estimating how long it will take to come up with a prototype that does what you think it should do, in other words coming up with requirements, is just the snake eating its tail.
Design but also requirements. You don't really know what should be built until you have a prototype. And estimating how long it will take to come up with a prototype that does what you think it should do, in other words coming up with requirements, is just the snake eating its tail.
This point was recognized quite early. there is an influential article by Jack W. Reeves from 1992 [1] that looks at the manufacturing analogy and argues that "the code is the design" whereas the "manufacturing" is just the compilation and packaging (that is, super cheap and automated). I believe it‘s this line of thinking that inspired Agile practices, where software developers are seen as engineers rather than code monkeys.
[1]: https://www.developerdotstar.com/mag/articles/reeves_design....
[1]: https://www.developerdotstar.com/mag/articles/reeves_design....
Yes.
Coding is not like assembling the machine. Coding is like inventing the machine.
Coding is not like assembling the machine. Coding is like inventing the machine.
It doesn’t even have to be a big construction project to hit budget and schedule issues. Rebuilding our house had us going right back to the drawing board having to get new plans drafted and approved. It worked out much better in the end but the unforeseen cost and time overrun was quite anxiety inducing at the time.
But yeah the idea anyone has magically solved accurately predicting the future is completely bonkers.
But yeah the idea anyone has magically solved accurately predicting the future is completely bonkers.
I like to remember "plans are useless but planning is indispensable". The plan is useless because of the challenge of predicting the future. But the planning is useful because trying to predict the future gets you asking good questions.
Also because planning gets you to make your desires explicit and to put them in a useful framework. Unexpected problems, disruptive as they may be, don't usually derail the whole plan. Some parts may need to be rethought, but others are still valid, and the whole thing keeps you focused.
Programming is isomorphic to theorem proving (according to Curry-Howard).
Therefore trying to estimate a software project is a bit like trying to estimate how long it will take to prove a new theorem in mathematics, assuming you would know which exact theorem you want to prove.
Therefore trying to estimate a software project is a bit like trying to estimate how long it will take to prove a new theorem in mathematics, assuming you would know which exact theorem you want to prove.
Er, isn't it just getting your program to typecheck that's equivalent to proving the theorem (by construction)?
I tend to find this isn't the main consumer of my software budget :-)
I tend to find this isn't the main consumer of my software budget :-)
In a way I think yes but just getting the type-check right would be like proving ANY mathematical theorem. It is very easy to prove mathematical theorems if you don't care what those theorems are.
The challenge in mathematics, and programming, I believe, is what ("lemmas") to prove so that you can then prove more things that are useful for engineering, science, etc. . What functions to write in your program so that you can get its "main function" to do something useful.
Add to that the fact that much of the "type-checking" with current practical programming languages still has to be done informally, to describe what we expect of the result and then prove (to yourself and the client) that the program does that.
A lot of the thought-process that goes on in programmers' heads is informal reasoning proving things to yourself, "If I make this change here then that thing there must have a correct value, and then ..."
The challenge in mathematics, and programming, I believe, is what ("lemmas") to prove so that you can then prove more things that are useful for engineering, science, etc. . What functions to write in your program so that you can get its "main function" to do something useful.
Add to that the fact that much of the "type-checking" with current practical programming languages still has to be done informally, to describe what we expect of the result and then prove (to yourself and the client) that the program does that.
A lot of the thought-process that goes on in programmers' heads is informal reasoning proving things to yourself, "If I make this change here then that thing there must have a correct value, and then ..."
There was a submission on this (in the annals of HN) where someone made the point that, more often than not and under normal project scenarios, unexpected problems usually do derail the entire software engineering plan.
The gist being: estimates are given with zero as a lower bound, and some upper bound. In reality, there is no upper bound. Software engineering can and does involve unexpected problems that would take infinite time to solve.
Ergo, project timelines are more accurately averaged between {min_time} and {infinite}, and, given that, remaining risk is the only true metric that should be reported. I.e. how far are we along in terms of project completed, such that we can say only x% of the project remains, and therefore that 1-x% did not explode.
The gist being: estimates are given with zero as a lower bound, and some upper bound. In reality, there is no upper bound. Software engineering can and does involve unexpected problems that would take infinite time to solve.
Ergo, project timelines are more accurately averaged between {min_time} and {infinite}, and, given that, remaining risk is the only true metric that should be reported. I.e. how far are we along in terms of project completed, such that we can say only x% of the project remains, and therefore that 1-x% did not explode.
The fundamental problem with estimates is that the recipient believes that the estimate will always turn out to be true, which is obviously not correct.
So, why can't we express the upper bound in combination with uncertainty?
Example: "There is a 50% chance that the feature will be delivered within 1 month."
As time progresses confidence starts increasing and we get closer and closer to how much time it actually takes (be that 5 days, 5 months or 5 years). On the day the feature is completed, confidence is 100% because the real delivery date has become known.
So, why can't we express the upper bound in combination with uncertainty?
Example: "There is a 50% chance that the feature will be delivered within 1 month."
As time progresses confidence starts increasing and we get closer and closer to how much time it actually takes (be that 5 days, 5 months or 5 years). On the day the feature is completed, confidence is 100% because the real delivery date has become known.
More than that we need to be flexible to adjust a plan. You’d be amazed at how many people who quote that aphorism aren’t actually flexible. At least with a building they are also motivated by not leaving a physical monument to their failure.
What is then terrifying to me then is when I find myself working with people who manage to, by way of treating "estimates" as "quotes", make it so demoralizing to plan anything--as it will be "used against you" later (which they do even if they were adamant they wouldn't)--that everyone does everything they can to discourage and sabotage any attempt at planning, because planning is valuable :(.
its more than that. there is a loose correspondence between plan and work. so changes in plans can often be projected onto changes in work. the same goes for expectations and dependencies.
the other thing about the plan is that it attempts to capture what completion might look like. where i come from, the things software developers do right before releasing code look very different than the things they do when they are starting a new project. the plan can allow you to try to do some orchestration, ensure minimum quality bars are met, etc.
having a fuzzy map of the future that you're constantly having to correct sucks. having no map at all leads to the kind of carpe diem development organizations that never seem to finish anything.
the other thing about the plan is that it attempts to capture what completion might look like. where i come from, the things software developers do right before releasing code look very different than the things they do when they are starting a new project. the plan can allow you to try to do some orchestration, ensure minimum quality bars are met, etc.
having a fuzzy map of the future that you're constantly having to correct sucks. having no map at all leads to the kind of carpe diem development organizations that never seem to finish anything.
I'll raise (shrink) you an estimated-at-1-week landscaping project for my house that ended up taking just over 2. Wasn't particularly anxiety inducing, though, as we expected some overrun and both parties (re)negotiated in good faith.
Oh man, this hits home so much.
I used to lead a software department in a small company that sold software to, mainly, the construction industry, so the go to comparison was building houses, which according to my boss, were always on time and on budget, unlike software.
The funniest thing about it was that our software was used by huge projects: Hinkley Points C, HS2, etc ... which obviously have an great track record of being on time and on budget ಠ_ಠ
I used to lead a software department in a small company that sold software to, mainly, the construction industry, so the go to comparison was building houses, which according to my boss, were always on time and on budget, unlike software.
The funniest thing about it was that our software was used by huge projects: Hinkley Points C, HS2, etc ... which obviously have an great track record of being on time and on budget ಠ_ಠ
From what I've heard, in construction most of the delays happen with the foundations.
Everything that's below the ground is hard to predict, because you seldom know exactly what's in the ground.
Once your construction has progressed above ground, it's somewhat more predictable.
(But delays are still possible.)
Everything that's below the ground is hard to predict, because you seldom know exactly what's in the ground.
Once your construction has progressed above ground, it's somewhat more predictable.
(But delays are still possible.)
There's also the problem of of a company working on multiple construction sites in parallel.
In software engineering, this also happens of course, especially if you develop in house. And you also need to factor in the maintenance, as it's usually performed by the same team.
In software engineering, this also happens of course, especially if you develop in house. And you also need to factor in the maintenance, as it's usually performed by the same team.
'I really wish we would treat sales people the same way. "How much money is this contract for? What date will it be signed?"'.
In well-run sales teams, that is exactly what happens. There is a constant move towards more accuracy and tightness in forecasting.
In poorly run sales teams, the reps basically set their owns quotas (padded, of course, to mitigate against massive uncertainty) and then are very imprecise at managing the WIP. Which sounds like a poorly run engineering team too. In sales, though, at least there is a very objective measure ($$$ closed) at the end of the period. Also, the period is fixed typically (months, quarters, etc).
Engineering is obviously different... what's the measure? LOCs? Obviously not...
Edit: just to be clear, I'm saying that well-run sales teams and well-run engineering teams probably have a lot in common, not that they are exactly the same.
In well-run sales teams, that is exactly what happens. There is a constant move towards more accuracy and tightness in forecasting.
In poorly run sales teams, the reps basically set their owns quotas (padded, of course, to mitigate against massive uncertainty) and then are very imprecise at managing the WIP. Which sounds like a poorly run engineering team too. In sales, though, at least there is a very objective measure ($$$ closed) at the end of the period. Also, the period is fixed typically (months, quarters, etc).
Engineering is obviously different... what's the measure? LOCs? Obviously not...
Edit: just to be clear, I'm saying that well-run sales teams and well-run engineering teams probably have a lot in common, not that they are exactly the same.
I think you're talking about goals, which isn't the same as estimates.
A well run sales team, like a well run dev team do benefit from goals. Like increase X by Y. That's because it can guide your decision process of what you'll spend your time on and what you won't. It can also push for focused innovation around a particular area.
But I think OP meant what if you asked the sales team: How much time will it take you to close the deal with X? Would they be able to estimate that? Which is different from asking them: "Try and close X Y% faster than it took us to close Z last time." The former would be asking for an estimate, the latter would be setting a goal.
A well run sales team, like a well run dev team do benefit from goals. Like increase X by Y. That's because it can guide your decision process of what you'll spend your time on and what you won't. It can also push for focused innovation around a particular area.
But I think OP meant what if you asked the sales team: How much time will it take you to close the deal with X? Would they be able to estimate that? Which is different from asking them: "Try and close X Y% faster than it took us to close Z last time." The former would be asking for an estimate, the latter would be setting a goal.
Sales team certainly understand the sales cycle and different velocities involved.
Eg - two jobs we ago sold to finance playera from small hedge funds to huge asset managers to central banks and finance ministries. You could sell more small hedge funds by knocking on more doors but you understood that the big players took months and years to decide. And of course we always tried to hone the sales org and process to arm the sales people with tools and data to move faster. We certainly held them much more accountable than devs.
It's actually not a fair comparison - software development is often like golf - the only limit to your velocity is you while sales can't really move faster than customer's process in many cases.
(Yes I obviously know dev processes can have external dependencies too)
Eg - two jobs we ago sold to finance playera from small hedge funds to huge asset managers to central banks and finance ministries. You could sell more small hedge funds by knocking on more doors but you understood that the big players took months and years to decide. And of course we always tried to hone the sales org and process to arm the sales people with tools and data to move faster. We certainly held them much more accountable than devs.
It's actually not a fair comparison - software development is often like golf - the only limit to your velocity is you while sales can't really move faster than customer's process in many cases.
(Yes I obviously know dev processes can have external dependencies too)
> the only limit to your velocity is you
I have never seen this to be the case outside small, self-contained projects. Even with a fixed scope (its own can of worms), major slow-downs often clearly come from outside the team—unexpected or unreasonable infrastructure limitations, problems with the products of other teams, shortcomings of internal or external tools and libraries, unclear or inconsistent evaluation of the work by its eventual users... etc.
Moreover, in cultures tied to estimates-as-quotes (or estimates-as-deadlines), there is often significant pressure for "product" teams not to try to fix these problems, even for themselves. It would be net faster for us to write our own version of X than to try to integrate the work of team Y? But they've already done the work, why would you want to do something redundant?
In a world where technical teams have clear ownership, this is less of a problem. Team Y giving you problems? Work around it the best way you can. Infrastructure not supporting your needs? Hack together your own on top of lower-level pieces until it's ready. But organizations with cultures that give technical teams the space to do this also have far less issues with estimates—both in having better estimates and being able to handle estimate uncertainty gracefully.
I have never seen this to be the case outside small, self-contained projects. Even with a fixed scope (its own can of worms), major slow-downs often clearly come from outside the team—unexpected or unreasonable infrastructure limitations, problems with the products of other teams, shortcomings of internal or external tools and libraries, unclear or inconsistent evaluation of the work by its eventual users... etc.
Moreover, in cultures tied to estimates-as-quotes (or estimates-as-deadlines), there is often significant pressure for "product" teams not to try to fix these problems, even for themselves. It would be net faster for us to write our own version of X than to try to integrate the work of team Y? But they've already done the work, why would you want to do something redundant?
In a world where technical teams have clear ownership, this is less of a problem. Team Y giving you problems? Work around it the best way you can. Infrastructure not supporting your needs? Hack together your own on top of lower-level pieces until it's ready. But organizations with cultures that give technical teams the space to do this also have far less issues with estimates—both in having better estimates and being able to handle estimate uncertainty gracefully.
Moving faster and increasing impact of deliveries, those are goals you can work towards, that do not involve needing to estimate anything.
The question is: do you for every sale provide an ETA of when you think the sale will close, with projected key milestones along the way and their dates, all before even beginning engaging the would be customer?
The question is: do you for every sale provide an ETA of when you think the sale will close, with projected key milestones along the way and their dates, all before even beginning engaging the would be customer?
// do you for every sale provide an ETA of when you think the sale will close, with projected key milestones along the way and their dates, all before even beginning engaging the would be customer?
No because that's kind of useless.
With software, the estimate is (or should be) part of the go-no-go decision for the project. "Oh, it's gonna take us 10 years to build this? Maybe we should reconsider"
No because that's kind of useless.
With software, the estimate is (or should be) part of the go-no-go decision for the project. "Oh, it's gonna take us 10 years to build this? Maybe we should reconsider"
Yes, it's like that. You have to move deals through many stages. There's expectations of timing into each stage in addition to quota. Most places will fire even senior sales people if they don't perform for a few quarters.
"$$$ closed at the end of the period" causes negative externalities at the expense of value delivered in the long run (overpromising, misleading, mispricing). This isn't unique to sales/engineering, I've heard of it playing out the same way in "factory" law firms where sales-lawyers and delivery-lawyers are separate.
> at least there is a very objective measure ($$$ closed) at the end of the period.
That is far from an objective measure standing alone, unless every salesperson generates all their own leads and the distribution of potential leads is random.
But generally your more experienced members will get the bigger potential customers.
So now you're back at square one trying to judge their success in relation to the anticipated likelihood of their leads.
That is far from an objective measure standing alone, unless every salesperson generates all their own leads and the distribution of potential leads is random.
But generally your more experienced members will get the bigger potential customers.
So now you're back at square one trying to judge their success in relation to the anticipated likelihood of their leads.
i think well run teams in general have a lot in common. Not just sales or engineering but all well run teams.
I used to work for a company that sold custom business systems. When the customers did their parts, i.e., signed of specifications and did their acceptance testing within schedule, we usually delivered within estimate and in time. This was of course applications built with boring technology where 80% was simple CRUD. But the market for this should not be underestimated.
Sales teams do EXACTLY that. Go into salesforce and you’ll see the estimates.
Potential deal with client Foo will be for $X, to be signed on Y day, with Z% chance of closing.
On an individual level, it may be more precise than it is accurate, but it’s a fairly standard and important part of sales operations. And yes, the estimate dates are often just ways to express orders of magnitude (put next Friday as estimate dates not because it’ll definitely close next Friday, but bc it’ll probably take about 2 weeks). The % chance of closing are typically pre determined by what stage of the deal you’re in.
However, when applied across dozens, hundreds, thousands of potential deals, it allows the organization to understand their sales pipeline and sales velocity fairly accurately.
Of course the numbers are gamed a bit. And some sales people are much worse at estimating than others, or less consistent about keeping the numbers up to date. But the benefit of these practices are massive.
Potential deal with client Foo will be for $X, to be signed on Y day, with Z% chance of closing.
On an individual level, it may be more precise than it is accurate, but it’s a fairly standard and important part of sales operations. And yes, the estimate dates are often just ways to express orders of magnitude (put next Friday as estimate dates not because it’ll definitely close next Friday, but bc it’ll probably take about 2 weeks). The % chance of closing are typically pre determined by what stage of the deal you’re in.
However, when applied across dozens, hundreds, thousands of potential deals, it allows the organization to understand their sales pipeline and sales velocity fairly accurately.
Of course the numbers are gamed a bit. And some sales people are much worse at estimating than others, or less consistent about keeping the numbers up to date. But the benefit of these practices are massive.
Houses have been done before. Enough time that we are good at those estimates. Once you have built the same house a dozen times you can estimate to within a couple hundred dollars. Contractors do this all the time for spec houses. Spec meaning that they are building it and the planning to sell it once it is complete. I know several contractors who do this all the time. They have never had someone order a spec house and come in anywhere close to the estimates. Everyone goes over the light fixture budget ($600) by at least $3000. Similar for the kitchen and bath fixtures. It isn't unheard of to double the cost of the house on seemingly minor upgrades.
> These days, I only give estimates in terms of units but without numbers. Hours, days, weeks, months, quarters or years.
Then you must be either an independent contractor or have some magical source of job security. I couldn't get away with this kind of estimate.
Then you must be either an independent contractor or have some magical source of job security. I couldn't get away with this kind of estimate.
Probably more to do with personality mix than circumstance. I've had bosses where I could say things like that and get away with it. I've also had bosses where if I said something like that, I'd be packing up my desk.
Ok, granted, an enlightened boss might accept that kind of estimate, I suppose. And without the higher-ups breathing down his/her neck.
If you give a bonus for getting done on time/early, construction will finish pretty darn close to the estimates. Construction is underbid at least partly because the incentives are setup for underbidding.
A lot of the rest is Planning fallacy[1], particularly with individual contractors, but larger contracting businesses can absolutely come up with realistic estimates based upon past experience.
1: https://en.wikipedia.org/wiki/Planning_fallacy
A lot of the rest is Planning fallacy[1], particularly with individual contractors, but larger contracting businesses can absolutely come up with realistic estimates based upon past experience.
1: https://en.wikipedia.org/wiki/Planning_fallacy
Estimating in terms of units is mostly for internal teams with a fixed payroll. It can work really great in that context. But imagine that you are asking for project funding. The person across the table logically wonders: how much do you need? You yourself logically wonder: how much do I ask for? In that environment, can either party really accept “who knows” or “5.5 units” as the answer unless money is not a concern for anybody?
In my opinion, the estimate gives them a sense of scale. If some small number of months would cost too much, then you know and can start to have an investigation project of a week to better help define what should be in and what shouldn't.
They do treat sales people that way. Sales is ruthlessly managed and expectations are very important, especially at a public company.
At least for government projects the 'wrong' estimates are intentional. Admins always take the cheapest quote, so to earn money everything happening during construction has to be 'out of scope'. Like winter, or different soil layer than specified, or whatever.
Software doesn‘t exist in a vacuum. The marketing team needs to know when the landing page / qr code (whatever) is implemented, because they need to plan their marketing budget. The customer care agent that gets shouted at because feature xyz is broken / not working as the customer expects, would like to give the customer an answer when it will be fixed. The customer care manager is also worried about her team members‘ retention.
So having a rough or concrete time line / roadmap is helpful for these parts of the business.
I also believe that a feature request is incomplete without an estimate. It is like a friend asking you to go buy him a car. What does he need a 3000$ subaru from 2003 or a 2021 Ferrari? Of course the friend can give you an insanely accurate list of specifications, but giving you a budget + some guidelines will probably be more effective for your friend and more fun for you. Especially considering that you know much more about cars (read software) than your friend. Knowing the budget you will be able to make much better choices.
If you take the allegory of a quote (which I think is correct) then you need to consider that the haggling process always also includes scope of the product. „I‘ll give you 5000$ for the car, but only if you include a free car wash“. And you can always use an estimate as defense against feature creep. „We are running low on budget, I suggest descoping xyz into the next iteration. Otherwise we won‘t be able to ship on time“. If on time is not defined then you can never have this conversation.
So having a rough or concrete time line / roadmap is helpful for these parts of the business.
I also believe that a feature request is incomplete without an estimate. It is like a friend asking you to go buy him a car. What does he need a 3000$ subaru from 2003 or a 2021 Ferrari? Of course the friend can give you an insanely accurate list of specifications, but giving you a budget + some guidelines will probably be more effective for your friend and more fun for you. Especially considering that you know much more about cars (read software) than your friend. Knowing the budget you will be able to make much better choices.
If you take the allegory of a quote (which I think is correct) then you need to consider that the haggling process always also includes scope of the product. „I‘ll give you 5000$ for the car, but only if you include a free car wash“. And you can always use an estimate as defense against feature creep. „We are running low on budget, I suggest descoping xyz into the next iteration. Otherwise we won‘t be able to ship on time“. If on time is not defined then you can never have this conversation.
Agreed, marketing (and everyone, really) needs at least rough estimates. Software is notoriously hard to predict. As time goes on, estimates get updated as more is known about the problem and the org's ability to solve it. So, by definition the first estimate anyone makes at the beginning is the least reliable one, by a long shot, but it's something to start with that gets refined as time goes on.
I think where companies can get in trouble with software is when the initial rough estimate somehow gets blessed as the plan of record to cement in everyone else's critical path, and subsequent estimates are treated as engineering screwups rather than more informed predictions. I don't know what the solution is, because stuff like this can happen with everyone involved being super competent and having the best intentions. And sometimes there are engineering screwups, neither side is always blameless.
Power imbalances between organizations are where natural creative conflicts go bad. In a less healthy company, whichever org is on top politically tends to be blameless, whether it's fair or not, which ultimately hurts everyone's ability to do their best work because conflicts between orgs are less likely to be resolved based on what's best for the company or project as a whole. In ideal companies, orgs are balanced such that they can push back on each other for good reasons, but not for zero-sum, shit-rolling-downhill reasons. I'm pretty sure ideal companies only exist briefly or never though.
I think where companies can get in trouble with software is when the initial rough estimate somehow gets blessed as the plan of record to cement in everyone else's critical path, and subsequent estimates are treated as engineering screwups rather than more informed predictions. I don't know what the solution is, because stuff like this can happen with everyone involved being super competent and having the best intentions. And sometimes there are engineering screwups, neither side is always blameless.
Power imbalances between organizations are where natural creative conflicts go bad. In a less healthy company, whichever org is on top politically tends to be blameless, whether it's fair or not, which ultimately hurts everyone's ability to do their best work because conflicts between orgs are less likely to be resolved based on what's best for the company or project as a whole. In ideal companies, orgs are balanced such that they can push back on each other for good reasons, but not for zero-sum, shit-rolling-downhill reasons. I'm pretty sure ideal companies only exist briefly or never though.
And to continue your metaphor, most software estimation comes from someone describing a car they like, and based on your mock ups you and others decide on a price, how long is acceptable to build, etc.
So then you goto build the car and everything's going great till you find out that the latest research shows the specific design of engine you're using doesn't work with the eco friendly fuel that is now being mandated by the government. And suddenly killing cows is uncool so you have to change all your chairs to fabric or source fake leather, but half of your company thinks that endorsing fake leather is hide-appropriation and you're now spending a third of your time in HR going through training.
And eventually you get about a week out from delivering the car and the customer calls your PM and says they want the car to have a top speed 30mph over the original specs while maintaining the fuel efficiency rating. And the doors should swing up, instead of out. Like billionaires'.
So then you goto build the car and everything's going great till you find out that the latest research shows the specific design of engine you're using doesn't work with the eco friendly fuel that is now being mandated by the government. And suddenly killing cows is uncool so you have to change all your chairs to fabric or source fake leather, but half of your company thinks that endorsing fake leather is hide-appropriation and you're now spending a third of your time in HR going through training.
And eventually you get about a week out from delivering the car and the customer calls your PM and says they want the car to have a top speed 30mph over the original specs while maintaining the fuel efficiency rating. And the doors should swing up, instead of out. Like billionaires'.
Kinda. Except that's "need for a definite date" is an expectation that they could be flexible about if forced. If other professions genuinely understood that software estimation is inaccurate by its nature and that accurate predictions of completion dates are, by their nature, impossible, then the uncertainty can be managed.
But because it seems so simple to estimate how long a piece of work will take for everyone else, it's really difficult to get other people to realise how hard this is.
I've sat in meetings saying "I don't know how long this is going to take" repeatedly until the rest of the organisation works out that they're not going to get a definitive date. Then suddenly the marketing team plans for "OK, well if it's done by this date then we'll have enough time to make this trade fair. If not then we can skip that and do the other one". They'd like to have a definite date because it makes their job easier. But if they can't have one, they can work around it.
> I also believe that a feature request is incomplete without an xestimatex delivery date attached
This is so true, as long as the devs have control over the scope. "We don't know and can't predict how long it's going to take to build the thing. If we can't build it all by the deadline, what's more important: delivering something on the deadline, or missing the deadline and delivering the entire feature set?"
To use your car haggling analogy: You've given me $5K to buy a 4-door car less than 5 years old. If I can't find one like that, what's more important: the budget, the doors or the age?
But because it seems so simple to estimate how long a piece of work will take for everyone else, it's really difficult to get other people to realise how hard this is.
I've sat in meetings saying "I don't know how long this is going to take" repeatedly until the rest of the organisation works out that they're not going to get a definitive date. Then suddenly the marketing team plans for "OK, well if it's done by this date then we'll have enough time to make this trade fair. If not then we can skip that and do the other one". They'd like to have a definite date because it makes their job easier. But if they can't have one, they can work around it.
> I also believe that a feature request is incomplete without an xestimatex delivery date attached
This is so true, as long as the devs have control over the scope. "We don't know and can't predict how long it's going to take to build the thing. If we can't build it all by the deadline, what's more important: delivering something on the deadline, or missing the deadline and delivering the entire feature set?"
To use your car haggling analogy: You've given me $5K to buy a 4-door car less than 5 years old. If I can't find one like that, what's more important: the budget, the doors or the age?
That's the thing, if other departments have a timeline they need that should be the estimate.
I think it's a lot more natural in business software to say something like "we need the best version of XYZ you can do in three months" rather than saying "we need XYZ" and hoping the estimate is three months. That way everyone's cards are on the table and can work together on getting what they need.
I think it's a lot more natural in business software to say something like "we need the best version of XYZ you can do in three months" rather than saying "we need XYZ" and hoping the estimate is three months. That way everyone's cards are on the table and can work together on getting what they need.
I need to know what the stock market is going to do next week. Otherwise, how am I going to become a billionaire? My broker’s refusal to tell me literally costs much more than all the returns he’s ever generated. Who does he think he is?
I once had a manager that thought he could negotiate estimates. That is like negotiating with the weatherman about the weather. Sure, maybe in the end you can convince them it will be finished sooner. But the fact is that it really doesn't change reality.
Typically there are many dimensions to a solution, the negotiation can be about trading one dimension for another.
- does it have to be instant or can we run it as a batch
- maybe it’s enough to have this tool in English only
Etc.
- does it have to be instant or can we run it as a batch
- maybe it’s enough to have this tool in English only
Etc.
Negotiating scope is a different mindset than negotiating estimates.
Even better would be making sure the team understands both the real goal they're trying to achieve and how time plays into it, letting the team make scope trade-offs like that as they go along.
Even better would be making sure the team understands both the real goal they're trying to achieve and how time plays into it, letting the team make scope trade-offs like that as they go along.
All discussions about estimates are really just discussions about scope. I try to make that as clear as possible when I'm talking about time estimates. There is no 'negotiating' estimates. If someone is being a little pushy with trying to get a shorter estimate, I will always just rephrase it as a trade-off. 'Doing it in fewer than X months would mean dumping X feature/walking back on Y commitment,' whatever the case may be.
I think in general these conversations always go better when you make the other person feel like they have a choice. Really, in today's corporate world everyone is playing CYA at all times. The person trying to bring down the estimate is probably doing so because they are anticipating what their boss will say. By listing potential trade-offs for them, I'm helping them by giving them fodder for that talk that they'll be facing, and they'll be able to give their boss a choice.
You always get better reactions out of people when they have choices, rather than just presenting a stone wall to them. Having said that, some times the person will really just keep being pushy, and then fortitude is necessary. If you are someone who has trouble definitively saying 'no' when it's required, you're doing your team a disservice by being the person giving estimates. You really need to never slip with this; all it takes is for you to one time cave and say, "okay, maybe we can try to shave a week off" and now that person knows that your estimates are negotiable.
If you consistently phrase them as facts rather than bids (like the sort of estimate a contractor gives you for a house), you'll get better results.
I think in general these conversations always go better when you make the other person feel like they have a choice. Really, in today's corporate world everyone is playing CYA at all times. The person trying to bring down the estimate is probably doing so because they are anticipating what their boss will say. By listing potential trade-offs for them, I'm helping them by giving them fodder for that talk that they'll be facing, and they'll be able to give their boss a choice.
You always get better reactions out of people when they have choices, rather than just presenting a stone wall to them. Having said that, some times the person will really just keep being pushy, and then fortitude is necessary. If you are someone who has trouble definitively saying 'no' when it's required, you're doing your team a disservice by being the person giving estimates. You really need to never slip with this; all it takes is for you to one time cave and say, "okay, maybe we can try to shave a week off" and now that person knows that your estimates are negotiable.
If you consistently phrase them as facts rather than bids (like the sort of estimate a contractor gives you for a house), you'll get better results.
You can tell when they are negotiating time instead of scope. It will be something like "surely we can do this in 2 days not 5?" Instead of "we probably will take 2-5 days to do this but need it in 2 so something of 3 days needs to come out somewhere else". The first one is saying 'work harder and just do it.' The second is 'ok it may take 2-5 days but lets pull something else out to make up for the deadline we want'.
I found orgs that negotiate time did not talk to their team in the first place. Then went on to promise something on behalf of the team. Now they do not want to eat the humble pie of 'I ran the numbers and you only get half' instead they get it going and just overbudget on the thing. Which just makes everyone in the process mad.
I found orgs that negotiate time did not talk to their team in the first place. Then went on to promise something on behalf of the team. Now they do not want to eat the humble pie of 'I ran the numbers and you only get half' instead they get it going and just overbudget on the thing. Which just makes everyone in the process mad.
I think this is a vital point. Estimates help narrow down what is important and what isn't.
Maybe you want features x, y and z. If it takes a day then X is worthwhile, if it's going to take a month don't bother. Z must be done so if that's going to take a long time we need to drop y.
One of the best product owners I've worked for would often ask for estimates and base plans and features from those. Some things would be easy technically and valuable, others harder and just a nice to have (but worth it if it only pushed things back a day).
Maybe you want features x, y and z. If it takes a day then X is worthwhile, if it's going to take a month don't bother. Z must be done so if that's going to take a long time we need to drop y.
One of the best product owners I've worked for would often ask for estimates and base plans and features from those. Some things would be easy technically and valuable, others harder and just a nice to have (but worth it if it only pushed things back a day).
I remember an interview at a well known tech-ish company where the question started out as “Estimate this project” with a good description of the scope and team. After I said something like 4 months, interviewer said, “OK. The CEO says it needs to be done in 2 weeks. What now?”
I thought this was a basic project management question about trading off quality, scope, schedule, staffing, etc. and offered the usual suggestions of cutting scope (nope CEO demands entire scope), cutting quality (nope can’t compromise quality), increasing the team, contracting out, reprioritizing and realeasing P1s in 2 weeks (nope, nope, nope). Obviously I didn’t get the job.
Later I learned from an acquaintance inside the company that the right answer was probably “Work the team 24 hour days, nights and weekends, for 2weeks to deliver what the CEO wants. I think I dodged a bullet there.
I thought this was a basic project management question about trading off quality, scope, schedule, staffing, etc. and offered the usual suggestions of cutting scope (nope CEO demands entire scope), cutting quality (nope can’t compromise quality), increasing the team, contracting out, reprioritizing and realeasing P1s in 2 weeks (nope, nope, nope). Obviously I didn’t get the job.
Later I learned from an acquaintance inside the company that the right answer was probably “Work the team 24 hour days, nights and weekends, for 2weeks to deliver what the CEO wants. I think I dodged a bullet there.
In my experience, the sort of people who believe in negotiating estimates are also the sort of people who only believe there's one dimension that can be traded off: how much unpaid overtime will the developer work?
I hate to sound cynical, but it can change reality if he were to use the estimate to extort overtime out of the team by citing their own estimates. I don't think it works out medium to long term, but could help him short term with the right teams. I assume they were also hiring for pliability/submissiveness.
The big cost tends to be software quality as it is less visible/harder to measure than dates and scope.
The big cost tends to be software quality as it is less visible/harder to measure than dates and scope.
In practice, the lower you can get the estimate, the more it will shoot back in your face.
I can say that this company had the worst planning I have ever seen, with some project not being able to finish at all.
I can say that this company had the worst planning I have ever seen, with some project not being able to finish at all.
I find it much less stressful to deal with since I've made peace with the fact that some managers want to be lied to.
Be careful because those lies become expectations for everyone that manager speaks to.
Yeah I've had a couple of managers try this. I always tell them my estimate is how long I think it will take take. But if he believes that estimate is incorrect he's welcome to modify it as he sees fit. (Over e-mail of course).
Sometimes they'll modify it, sometimes they won't but at least I won't have to argue about it or get in a situation later where suddenly they pretend like their estimate is my estimate.
Sometimes they'll modify it, sometimes they won't but at least I won't have to argue about it or get in a situation later where suddenly they pretend like their estimate is my estimate.
You and I act like weathermen in this situation.
But plenty of engineers do estimate based on what they think they can get away with or other negotiable ways.
But plenty of engineers do estimate based on what they think they can get away with or other negotiable ways.
I used to be a non-technical manager “managing” a team of developers. I later (through circumstances) became a developer. Looking back at the games my team played with me I can see why many managers are cynical about the estimates they are given and attempt to negotiate.
I always overestimated because it was the only way that I could ever devote anytime at all to technical debt reduction.
I'd say you're including necessary maintenance in your estimates, not overestimating!
Not claiming it applies to you, but a lot of managers don't understand the difference between an estimate and a deadline. It comes back in all kinds of weird ways.
[deleted]
Sounds like the type that thinks that if they blow enough smoke up your arse then you get things done quicker, and they look better. I hope they die out.
They did die out. Their planning was crazy, promises to customers never achieved, some projects even completely cancelled because of major overruns etc.
They also started doing their own estimates since they didn't like mine. 3 months of work with 3 people -> 1 week of work 1 developer. Not kidding about this last one! We didn't finish it in 3 months because of scope creep.
Yeah they went bust.
They also started doing their own estimates since they didn't like mine. 3 months of work with 3 people -> 1 week of work 1 developer. Not kidding about this last one! We didn't finish it in 3 months because of scope creep.
Yeah they went bust.
at that point you are negotiating just how much of your personal time/stress you are going to burn to meet their expectations. Also maybe how many other people you are going to have to short change because they have jumped the queue.
That is the degenerate scenario, but it’s not always true.
Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it.
And I’ve also seen wise use of estimation but that is rarer!
Often I see this haggling down of estimates and then microagression if estimates are not met but no need to work for free to make up for it.
And I’ve also seen wise use of estimation but that is rarer!
> And I’ve also seen wise use of estimation but that is rarer
I've got a question for you here, what is a wise use of estimation?
What problems are estimates the ideal solution for? In the context of software development?
I can think of only two: planning headcount and priorities (aka planning the order in which you do things).
When you focus on those problems, you realize there's no need for an accurate estimate. All you need is relative sizing between requests. This seems twice as much work as that but gives us half the benefit. Ok probably we shouldn't do this one first. And even then, often time impact is all you need for priorities. Like who cares what takes you longer, if you just focus on building the next most impactful thing you'll very often succeed.
The question of: "When are we going to be done this?" Is almost always irrelevant except for what I mentioned "asking for a quote". Like, who cares when it's done? What else you'd rather we do? I can tell you if your alternative will be any faster or not. And is this not the most impactful thing? If so, let's finish it and stop slowing us down with these irrelevant questions.
Now for headcount, it's the same thing. Look how much you did last year with what headcount. Ask yourself if what you're hoping to do this year is of the same relative size, twice as big, half as big, or smaller? And then plan headcount accordingly. If twice as big, double head count. Overtime you'll learn if twice as big requires doubling headcount or 1.5x or 1.2x or 2.3x etc.
I've got a question for you here, what is a wise use of estimation?
What problems are estimates the ideal solution for? In the context of software development?
I can think of only two: planning headcount and priorities (aka planning the order in which you do things).
When you focus on those problems, you realize there's no need for an accurate estimate. All you need is relative sizing between requests. This seems twice as much work as that but gives us half the benefit. Ok probably we shouldn't do this one first. And even then, often time impact is all you need for priorities. Like who cares what takes you longer, if you just focus on building the next most impactful thing you'll very often succeed.
The question of: "When are we going to be done this?" Is almost always irrelevant except for what I mentioned "asking for a quote". Like, who cares when it's done? What else you'd rather we do? I can tell you if your alternative will be any faster or not. And is this not the most impactful thing? If so, let's finish it and stop slowing us down with these irrelevant questions.
Now for headcount, it's the same thing. Look how much you did last year with what headcount. Ask yourself if what you're hoping to do this year is of the same relative size, twice as big, half as big, or smaller? And then plan headcount accordingly. If twice as big, double head count. Overtime you'll learn if twice as big requires doubling headcount or 1.5x or 1.2x or 2.3x etc.
Who cares when it's done? Often there are external deadlines like a start of production of a car for example which can not realistically be moved because too much planning would need to be redone and too much disruption would cause so much overhead that it could break a company.
Also when price and time are fixed, there is only one thing left for negotiation and it is the amount of features being delivered. Sometimes people have a hard time to cope with it but if you do it correctly it's a great thing for everyone. The most important features could be delivered early and the closer to the deadline the non critical features can be implemented which are nice to have but won't break the product if they're not in place. That's a great way to bring as much value as possible without breaking anyone.
Also when price and time are fixed, there is only one thing left for negotiation and it is the amount of features being delivered. Sometimes people have a hard time to cope with it but if you do it correctly it's a great thing for everyone. The most important features could be delivered early and the closer to the deadline the non critical features can be implemented which are nice to have but won't break the product if they're not in place. That's a great way to bring as much value as possible without breaking anyone.
Wise use of estimation is projection. You don't need to do much more than count stories. It would be honest and without agenda.
I've never seen it either.
I've never seen it either.
Yes exactly! When you ask someone for their estimate, you're getting a quote from them.
When you actually want to predict the length of time something will take, you rely on historical data and create a projected forecast. It never involves asking somebody for their best guess. That's how every single "other thing" we predict is handled, so why is software done differently.
Actually, I've seen some of these ideas being talked about in KanBan circles, like tracking lead time and cycle time, and then using monte carlo simulations to forecast, but I've never actually seen them in practice.
When you actually want to predict the length of time something will take, you rely on historical data and create a projected forecast. It never involves asking somebody for their best guess. That's how every single "other thing" we predict is handled, so why is software done differently.
Actually, I've seen some of these ideas being talked about in KanBan circles, like tracking lead time and cycle time, and then using monte carlo simulations to forecast, but I've never actually seen them in practice.
Didn't Joel Spolsky implement it all the way back in the stone age... er, 2007[0]?
The idea is old. The problem is, in many software companies, none of the tasks look alike. So your past data is going to be somewhat dirty, particularly at team level. Perhaps as an individual, intimately familiar with the problems you encounter, being able to introspect your own thought process - perhaps you or me, we could learn to estimate better. Except the existing popular tooling absolutely sucks at facilitating that.
--
[0] - https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
The idea is old. The problem is, in many software companies, none of the tasks look alike. So your past data is going to be somewhat dirty, particularly at team level. Perhaps as an individual, intimately familiar with the problems you encounter, being able to introspect your own thought process - perhaps you or me, we could learn to estimate better. Except the existing popular tooling absolutely sucks at facilitating that.
--
[0] - https://www.joelonsoftware.com/2007/10/26/evidence-based-sch...
> The problem is, in many software companies, none of the tasks look alike. So your past data is going to be somewhat dirty, particularly at team level
Tasks don't really give you anything though. I think if it was measured in terms of features, that would work a lot better.
And with a Monte Carlo simulation, you get to know the time it would take you to complete any random feature on any random period. So maybe you can't perfectly predict that next feature, but in the year, on the average, your predictions should tend towards good accuracy.
And you could take it further, machine learn the feature request to the time it took. And then run the ML inference on any new feature request. Maybe let devs add some labels, it's not exactly estimating, but they could mention if say they think something is "complex", "ambiguous", "straightforward", etc.
I'm sure there'd be ways to do it.
Tasks don't really give you anything though. I think if it was measured in terms of features, that would work a lot better.
And with a Monte Carlo simulation, you get to know the time it would take you to complete any random feature on any random period. So maybe you can't perfectly predict that next feature, but in the year, on the average, your predictions should tend towards good accuracy.
And you could take it further, machine learn the feature request to the time it took. And then run the ML inference on any new feature request. Maybe let devs add some labels, it's not exactly estimating, but they could mention if say they think something is "complex", "ambiguous", "straightforward", etc.
I'm sure there'd be ways to do it.
Monte Carlo here is just a way of adding together estimate distributions without doing any of the fancy math. The variance of resulting compound estimate strongly depends on the variance on inputs.
Feature-based estimation seems to coarse to me. Not only none of the features is ever like the others once you dig into it, they're also interconnected.
Feature-based estimation seems to coarse to me. Not only none of the features is ever like the others once you dig into it, they're also interconnected.
Get something ready for Christmas business? Get a car on the road? Finish the cruise ship in time? ...
> In the context of software development?
While your examples I'm not sure are super relevant from a software development context. I think you're pointing out that the estimate would help you decide when you need to start so you are done before the "real life events that needs the thing".
Is that correct?
Like for example I'm not sure it's useful to learn that no, you will not be ready for the Christmas event. Or no the cruise ship will not be ready in time. But it might be useful to know: "If I want to be ready for the Christmas event when should I get started and how many developers do I need?"
That's a good example I guess, though I haven't seen this issue arise very often in software. I could imagine someone needing to know when to start feature freeze and prep work for scaling up leading up to a big sale event if you run an online store for example.
My take on this is that asking devs to guess that time would obviously not work. And if you did this type of forecasting for any other discipline, you'd instead gather historical data about how much time things took in the past and plot that to forecast how much you'd think it takes in the future. Which is a process that can be done without needing to ask a dev for their personal not data based guesstimate.
While your examples I'm not sure are super relevant from a software development context. I think you're pointing out that the estimate would help you decide when you need to start so you are done before the "real life events that needs the thing".
Is that correct?
Like for example I'm not sure it's useful to learn that no, you will not be ready for the Christmas event. Or no the cruise ship will not be ready in time. But it might be useful to know: "If I want to be ready for the Christmas event when should I get started and how many developers do I need?"
That's a good example I guess, though I haven't seen this issue arise very often in software. I could imagine someone needing to know when to start feature freeze and prep work for scaling up leading up to a big sale event if you run an online store for example.
My take on this is that asking devs to guess that time would obviously not work. And if you did this type of forecasting for any other discipline, you'd instead gather historical data about how much time things took in the past and plot that to forecast how much you'd think it takes in the future. Which is a process that can be done without needing to ask a dev for their personal not data based guesstimate.
I have seen two software examples first hand. Usually it's because a feature can have much more value at a very specific date.
The first one was a feature for one of the minor verticals of the company product. Basically, something that would be nice to have and would be done eventually, but not a priority. However, the big annual trade show for that vertical was coming up, and the company could have a slot for a conference/demonstrate a feature. Marketing/sales were pretty convinced the aforementioned feature could be a big hit at the show, and provide lots of exposure. So we were asked whether it was possible to have it ready by then. If we could, then it would have a lot of value and would be priority 1, otherwise back to the backlog because it would be much harder to advertise it outside of the show.
Second example was caused by the reduction in VAT in Germany last year. Having internationalisation/German translation all of the sudden became pretty valuable because that would help push even more products in Germany.
The first one was a feature for one of the minor verticals of the company product. Basically, something that would be nice to have and would be done eventually, but not a priority. However, the big annual trade show for that vertical was coming up, and the company could have a slot for a conference/demonstrate a feature. Marketing/sales were pretty convinced the aforementioned feature could be a big hit at the show, and provide lots of exposure. So we were asked whether it was possible to have it ready by then. If we could, then it would have a lot of value and would be priority 1, otherwise back to the backlog because it would be much harder to advertise it outside of the show.
Second example was caused by the reduction in VAT in Germany last year. Having internationalisation/German translation all of the sudden became pretty valuable because that would help push even more products in Germany.
I haven't seen this issue arise very often in software
No? Maybe not if you're a software-only company. But for pretty much every hardware (a gadget, a car, a rocket, whatever), the software is part of it and it aint working until the software is.
If you don't want your product to be held shipping because the software isn't ready, you better have some way of estimating when it's going to get done so you can either trade features, personell, or even just knowing that there's no point rushing the hardware because it'll be done way before the code anyway.
No? Maybe not if you're a software-only company. But for pretty much every hardware (a gadget, a car, a rocket, whatever), the software is part of it and it aint working until the software is.
If you don't want your product to be held shipping because the software isn't ready, you better have some way of estimating when it's going to get done so you can either trade features, personell, or even just knowing that there's no point rushing the hardware because it'll be done way before the code anyway.
Inter-team dependencies. Marketing pushes. Customer commitments. New sales dependent on certain features. Building features to increase TAM. Partner commitments for interoperability / joint deployments. Even presentations of your roadmap to customers and prospects to get them bought into your vision of where the product is going.
There's tons of things that require broadly knowing when X is going to be done beyond capacity issues.
There's tons of things that require broadly knowing when X is going to be done beyond capacity issues.
If you create low quality estimates willy nilly then don't be surprised that you will break them.
The above are reasons a business needs estimates / rough delivery timelines. I'm not sure how my discussion of the value of estimates is relevant to the quality of them.
To get a rough idea of what work can be done in the next X weeks, so that the business can forecast.
The wisdom comes in knowing things can change and it's just an estimate. The process of estimating is quite good as it requires finding out a lot more about how you will solve something, and planning things in advance.
The wisdom comes in knowing things can change and it's just an estimate. The process of estimating is quite good as it requires finding out a lot more about how you will solve something, and planning things in advance.
> so that the business can forecast
Forecast to what purpose? What is the business actually trying to achieve for which they believe, maybe wrongly, they need accurate software estimates?
Forecast to what purpose? What is the business actually trying to achieve for which they believe, maybe wrongly, they need accurate software estimates?
I never experienced the dynamic you are talking about. Sounds like your workplace is terrible.
I've almost always had business partners that were just that - partners - invested in the product and the team long term. I am the same way now that I am "the business."
Estimation is a useful forcing exercise for thinking through what something takes, identifying risks and hard parts and deciding upfront what to do about them.
It also becomes a good measure of people's integrity - what they do once it's apparent the estimate is off. 100% of the time, when engineering came back saying "we underestimated X, it's much harder" - it was totally fine.
I've almost always had business partners that were just that - partners - invested in the product and the team long term. I am the same way now that I am "the business."
Estimation is a useful forcing exercise for thinking through what something takes, identifying risks and hard parts and deciding upfront what to do about them.
It also becomes a good measure of people's integrity - what they do once it's apparent the estimate is off. 100% of the time, when engineering came back saying "we underestimated X, it's much harder" - it was totally fine.
I have noticed this article during the "Parkinson's law of triviality"[1] meeting (within phase where technical guys are typically silent) and almost started laughing when someone asked about "over the thumb effort estimates" (that are always becoming carved into rock deadlines).
It really made my day :)
[1] https://en.wikipedia.org/wiki/Law_of_triviality
It really made my day :)
[1] https://en.wikipedia.org/wiki/Law_of_triviality
This is only true when estimates are in concrete units, the way we did it in the 80s and 90s. Unfortunately that still impacts contract structure today and it's hard to change on that level.
TFA is talking about estimates inside projects, which are not time based anymore in any environment that cares about the problems you laid out.
Personally I offer my customers to either pay by the week, or pay by extremely fixed scope with a 3 month definition period up front and a 50% buffer above the estimate. I stress that paying by the sprint lets them change objectives at any time, gives them an always-current report of how much "effort budget" they have left to work with, and guarantees they only pay for what they use. That felt like a risky move the first time I did it, but I never lost any business over it.
TFA is talking about estimates inside projects, which are not time based anymore in any environment that cares about the problems you laid out.
Personally I offer my customers to either pay by the week, or pay by extremely fixed scope with a 3 month definition period up front and a 50% buffer above the estimate. I stress that paying by the sprint lets them change objectives at any time, gives them an always-current report of how much "effort budget" they have left to work with, and guarantees they only pay for what they use. That felt like a risky move the first time I did it, but I never lost any business over it.
I think that everyone still converts what-ever-unit used to time. For example, from one of my earlier jobs, we had "T-shirt"-size units, ranging from XS to XL. It took two weeks after introducing them until someone figured out that XS is two hours, so in a sprint you should be able to do $SPRINT_DURATION / 2 XS tasks. And then the person figured that S was equal to, say 4 hours and so on and we were back on square one. But we didn't estimate in concrete time units. /s
There may be a lot of uncerntainty in Data Science and ML projects. However, recently I started feeling like I actually have it better than someone from pure software engineering sides of things:
For either, there is often a function from time spent to quality. 100% perfection is basically impossible and before that the function increases very slowly, seemlingly logarithmicly.
For SWE, expectations are often close perfect solutions. Too greedy effort estimations cause a lot of trouble. For DS/ML, however, perfect is usually off the table and this fact is widely (not universally though) accepted. When it is accepted to give estimates in this way, suddenly there no harm from being quoted on it and I really don't mind to give estimates anymore, where I just make a guess at a good 80/20 point. If I am wrong with that point, chances are nobody on the outside/higher up ever knows.
This may be different in domains where very clear targets have to be met (e.g., "self driving cars that pass lawmaker's requirements for use on the streets") and then I'd guess it is a true nightmare.
Like this, I never felt overly pressured by ML/DS deadlines over the last years. Some things were great successes, sometimes the quality wasn't great enough and projects were stopped or customers left. But there never really was a case where anyone thought that working extra long might have been an option to meet higher expectations.
I don't really have a solution for SWE, I don't really see how one would sell something like "I can do it in X time and it will only crash / stop working / make mistakes / become too slow / have vulnerabilities so often. More time will lead to fewer problems". This just isn't what's expected. But at least for complex systems and security vulnerabilities, I'd argue it is actually quite true. Guarantees for 100% perfection just aren't realistic. Avoiding the most obvious pitfalls is done rather quickly and the more time spend, the more is needed for further improvements.
For either, there is often a function from time spent to quality. 100% perfection is basically impossible and before that the function increases very slowly, seemlingly logarithmicly.
For SWE, expectations are often close perfect solutions. Too greedy effort estimations cause a lot of trouble. For DS/ML, however, perfect is usually off the table and this fact is widely (not universally though) accepted. When it is accepted to give estimates in this way, suddenly there no harm from being quoted on it and I really don't mind to give estimates anymore, where I just make a guess at a good 80/20 point. If I am wrong with that point, chances are nobody on the outside/higher up ever knows.
This may be different in domains where very clear targets have to be met (e.g., "self driving cars that pass lawmaker's requirements for use on the streets") and then I'd guess it is a true nightmare.
Like this, I never felt overly pressured by ML/DS deadlines over the last years. Some things were great successes, sometimes the quality wasn't great enough and projects were stopped or customers left. But there never really was a case where anyone thought that working extra long might have been an option to meet higher expectations.
I don't really have a solution for SWE, I don't really see how one would sell something like "I can do it in X time and it will only crash / stop working / make mistakes / become too slow / have vulnerabilities so often. More time will lead to fewer problems". This just isn't what's expected. But at least for complex systems and security vulnerabilities, I'd argue it is actually quite true. Guarantees for 100% perfection just aren't realistic. Avoiding the most obvious pitfalls is done rather quickly and the more time spend, the more is needed for further improvements.
I don't really understand what type of environment you're talking about.
Some parts seems to hint the devs are an outside consulting shop, and some parts hint that the devs are on an internal team.
Most internal teams can blow through dev deadlines over and over again for years without any real repercussions. And while outside dev shops can't blow through estimates like internal teams the individual devs usually can unless they're in a leadership role.
Some parts seems to hint the devs are an outside consulting shop, and some parts hint that the devs are on an internal team.
Most internal teams can blow through dev deadlines over and over again for years without any real repercussions. And while outside dev shops can't blow through estimates like internal teams the individual devs usually can unless they're in a leadership role.
And of course once the dev team realises that their "estimates" are actually being treated as quotes, and that their ability to find time to vacation, sleep, or even see their families will be contingent on their quote being sufficiently padded...
Now you have a recipe for a highly adversarial relationship between management and engineering throughout the planning and development process.
Now you have a recipe for a highly adversarial relationship between management and engineering throughout the planning and development process.
I call it the "What's the earliest date you can't prove you won't be finished by?" management question.
Agreed that often times estimates are mistaken for quotes. Aside from that, my personal experience has been that those who work adjacent to software understands that there’s an element of uncertainty. If those that we work with don’t, it is on us to educate.
Improving estimation will eventually improve quality of life for all involved.
It's unfortunate that this HN thread has been reduced to the generic discussion about software estimates when the article is specifically talking about research done on the topic of software estimates.
According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.
If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.
According to the article, proper research remains a struggle due to outdated datasets from before modern agile methodologies, and that the modern datasets from industry are hard if not impossible to gather.
If industry is truly interested in improving software development and estimation, their data should be anonymized and made available to researchers for analysis.
I'd say the problem is more from the academic side. If good data isn't available, then academics should not be publishing papers on toy data. It's meaningless. The goal is not to publish papers but to advance science.
While I tend to agree, the research we environment demands this nonsense. Also, they have to start somewhere to set a record to get future funding. Should probably do more survey studies of what data exists and where the state of things are then suggest new studies and seek funding for it.
Short of large and completely government funded development projects, I think it would be a struggle to get data. Few businesses would be willing to offer up development processes and surrounding data due to potential IP lost. Any organization that has good processes have it in their interest to striffle others from discovering how to improve theirs or learn from the success of others.
Part of the issue is that academia largely just doesn't pay for software development which they can leverage as an accessible cheap data source, it's done as a completely privatized exercise. Any research that requires highly protected commercial processes is pretty difficult to get any traction on unless you're inadvertently rediscovering the same processes (from my experience).
With that said, I feel like the amount of empirical data you'd need is going to be incredibly high, much if it not even currently being collected.
Short of large and completely government funded development projects, I think it would be a struggle to get data. Few businesses would be willing to offer up development processes and surrounding data due to potential IP lost. Any organization that has good processes have it in their interest to striffle others from discovering how to improve theirs or learn from the success of others.
Part of the issue is that academia largely just doesn't pay for software development which they can leverage as an accessible cheap data source, it's done as a completely privatized exercise. Any research that requires highly protected commercial processes is pretty difficult to get any traction on unless you're inadvertently rediscovering the same processes (from my experience).
With that said, I feel like the amount of empirical data you'd need is going to be incredibly high, much if it not even currently being collected.
Research at business schools manage to find all types of data that companies believe is way more integral to their success than information about their software estimation process.
These researchers are choosing to waste their own time and governments money, because it's easier to play with new toy machine learning models than go to networking event and befriend software VPs in order to convince them to give you their data.
These researchers are choosing to waste their own time and governments money, because it's easier to play with new toy machine learning models than go to networking event and befriend software VPs in order to convince them to give you their data.
Nah. You just got a bone to pick. Christ.
Rereading my comment I think you're right. Been frustrated with bureaucracy related to covid (specifically the vaccine) and the lives it's costing. And that frustration was redirected to other people following silly rules with far less at stake.
I still think a lot of research in software engineering is useless because researchers spend too much time focusing on different methods and not enough going out into the world to collect better data. And researchers should be mildly ashamed for doing crappy research.
But also my tone should have been more constructive.
I still think a lot of research in software engineering is useless because researchers spend too much time focusing on different methods and not enough going out into the world to collect better data. And researchers should be mildly ashamed for doing crappy research.
But also my tone should have been more constructive.
Well, the metric is #papers and #citations to get funding. So yes that's true, ideally, but...
Relevant blog post - It's not the incentives it's you.
https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...
https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-in...
Could not disagree more. The premise of this piece is that dieting is just willpower, beating addiction is the same, an entirely individual problem.
The reality is that the incentives are what society is asking for, with ethics etc acting as constraints only, rather than actively rewarded. In that formulation it is inevitable (and optimal) that some people will skate to the edge, and if the edge is poorly enforced they will increasingly go over.
Policies must address the overall actual effects, not just a chimerical ideal.
(It is even more ridiculous coming from someone in psychology.)
The reality is that the incentives are what society is asking for, with ethics etc acting as constraints only, rather than actively rewarded. In that formulation it is inevitable (and optimal) that some people will skate to the edge, and if the edge is poorly enforced they will increasingly go over.
Policies must address the overall actual effects, not just a chimerical ideal.
(It is even more ridiculous coming from someone in psychology.)
[deleted]
> How large are these datasets that have attracted so many research papers?
> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).
> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?
> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?
> The NASA dataset contains 93 rows (that is not a typo, there is no power-of-ten missing), COCOMO 63 rows, Desharnais 81 rows, and ISBSG is licensed by the International Software Benchmarking Standards Group (academics can apply for a limited time use for research purposes, i.e., not pay the $3,000 annual subscription). The China dataset contains 499 rows, and is sometimes used (there is no mention of a supercomputer being required for this amount of data ;-).
> Why are researchers involved in software effort estimation feeding tiny datasets from the 1980s-1990s into machine learning algorithms?
> Grant money. Research projects are more likely to be funded if they use a trendy technique, and for the last decade machine learning has been the trendiest technique in software engineering research. What data is available to learn from? Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
Is this telling me that most theories about "sw estimation best practices" are cargo cults o-O ?
I think the term 'cargo cult' carries some religious history and other baggage that I wouldn't want conflated here, but your feeling that anything we've heard about software estimation is questionable is probably warranted.
IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.
Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.
IMO the theories have sprouted more from management which unfortunately is probably the best equipped today to view the trends over time and compare them to actual results (given the lack of empirical data). If we admit that our ideas of estimation have come from management, we should also admit their conflicts of interest in the matter and the various management fads that come and go. I think as humans we all search for patterns and try to generalize rules to solve our problems, even when we have an incomplete view of things to begin with.
Finally, even with a generalized rule we should realize how different people are team-to-team. I haven't worked on two teams where the same exact processes worked for everyone. The best teams and managers I've had will observe a team over time and tweak the various processes according to what works at that point in time. The worst managers were the ones who idolized a specific style or person and copy-pasted their opinions into their workplace without listening to the team.
I imagine that if Atlassian could get permission from its many JIRA customers, then doing a text-based ml-categorization of issue descriptions alongside completion times would be extremely interesting.
I'd wager that data is just as poor overall, but the sheer volume of data available might be able to help get some sort of consistent conclusions from them.
I'd wager that data is just as poor overall, but the sheer volume of data available might be able to help get some sort of consistent conclusions from them.
I could imagine getting data from tools like Jira, but there is so little consistency on how data is entered and updated, I would have a hard time swallowing any conclusions from that data.
A firm like Pivotal Labs (which, I just discovered, is now VMware Tanzu Labs, which is frankly the saddest thing I have read all day) would have a dataset and a good enough understanding of the assumptions in their Pivotal Tracker software to do good research here. I'm almost certain that they have internal papers that are far beyond the state of the art, so to speak, in the broader academic literature.
What data would be useful here? There are so many confounding factors.
Well, isn’t that the kind of thing researchers ought to be researching?
Ought to? Not my call, as I don't employ any.
I'd say Agile methodologies have done well in exposing the key factors for successful delivery, and they seem not very amenable to generalisation: - creative endeavours such as software creation are fraught with estimation difficulties (contrast, say, with migrating VMs from on-prem to cloud) - whoever prioritises / deprioritises features will have a large say in the accuracy - whatever technologies imposed/chosen will have a reasonable say in the accuracy - whatever architecture decided on will have a reasonable say in the accuracy - unforeseen requirements that may necessitate architecture / tech / people changes have a large say in the accuracy - individuals' skills in a team will have a very large say in the accuracy - how long a team has worked together / in an architecture / with a technology will have a very large say in the accuracy
We know all this. What can we learn? Genuine question.
I'd say Agile methodologies have done well in exposing the key factors for successful delivery, and they seem not very amenable to generalisation: - creative endeavours such as software creation are fraught with estimation difficulties (contrast, say, with migrating VMs from on-prem to cloud) - whoever prioritises / deprioritises features will have a large say in the accuracy - whatever technologies imposed/chosen will have a reasonable say in the accuracy - whatever architecture decided on will have a reasonable say in the accuracy - unforeseen requirements that may necessitate architecture / tech / people changes have a large say in the accuracy - individuals' skills in a team will have a very large say in the accuracy - how long a team has worked together / in an architecture / with a technology will have a very large say in the accuracy
We know all this. What can we learn? Genuine question.
If researchers aren’t doing anything to quantify and assess these ideas around estimation, I don’t know what the point of that entire sub-field of research is, beyond cranking out papers.
The issue here is that if you examine most projects today, it requires effort to collect data about what happened. So much that happens is untracked. That untracked stuff is a source of error.
Dont be so hard on HN, I would say given how niche the actual article topic is, the more generic discussion about software estimation is both relevant and relatable.
I'll be hard on HN here. The more generic discussion is barely relevant to the article. It's a prime example of people seeing an article title as a chance to say something they've already been thinking rather than a chance to read something new.
If someone can conclusively teach inexperienced programmers good approach to estimates (methodology) + help embed this into sales process of a software house-type company, I know some folks who'd love to have this :)
My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)
But at that point nobody cares! People go on with further projects, and no data is ever collected.
Nobody learns.
When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.
Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.
It's insane.
My own experience has been this: people make estimates, client has expectations based on some variant of those, and something later happens but so much change is introduced during the actual software development, that there seems to be no sane way to compare what happened with original estimates. (new features, changed features, lots of new information, market shifts/product vision shifts, quality assumptions change, etc. etc.)
But at that point nobody cares! People go on with further projects, and no data is ever collected.
Nobody learns.
When f*ckups happen, i.e. a gross over- or under-estimate, this is often not shared with the broader organization (ashamed/afraid sales people/PMs/devs say nothing/hide it/sugarcoat it). Client walks away.
Sometimes project is severly underestimated but usually not just because of the software part. Again, no decoupling and estimation of contributing factors is done.
It's insane.
Apple executes massive feature releases on a yearly waterfall-like schedule across hundreds of teams. It requires honesty, transparency, strong leadership (ruthless prioritization), strong cross-team goodwill, strong cross-team collaboration as well as a solid body of experienced top-tier engineers. Other than that there's no magic. While there are teams within apple that does scrum/agile, none of the core OS/frameworks teams does it AFAIK, and I do think it's incompatible with shipping on a hardware-aligned schedule.
Have you noticed their declining OS release quality in recent years? I have, and I now wait 6 months to upgrade.
The triangle is real[1]. Something's gotta give when a deadline can't move, and as we know from Brooks, software development speed doesn't increase linearly with more people. So it's primarily quality or scope that must suffer to hit a software deadline.
[1] - https://en.wikipedia.org/wiki/Project_management_triangle
The triangle is real[1]. Something's gotta give when a deadline can't move, and as we know from Brooks, software development speed doesn't increase linearly with more people. So it's primarily quality or scope that must suffer to hit a software deadline.
[1] - https://en.wikipedia.org/wiki/Project_management_triangle
People have been saying that since OS X first came out, and OS 9 was so buggy it had a famous cute bomb icon for when it crashed.
I'm using MacOs/OsX since Lion, and I can say that it was very stable (even on my Hackintosh) and I observed stability issues increasing with all the following versions (on real Apple HW) and I'm on the same ship as GP never upgrading to newer versions until at least 6 months pass.
Yeah, first few OS X releases were famously buggy as they were figuring this UNIX stuff and the OS 9 compatibility, but they stabilized about 10.4 Tiger.
Honestly the OS X itself (uhhh macOS now) is still fine, even with Big Sur and its weird iOS controls, however the other software is sometimes weird; but - let's say Numbers was never good and iWeb was always downright bad. And iTunes was bad in Jobs era and is still bad (as Music).
I however still kind of miss when native Cocoa apps were sort of good and not just ported iOS app via Catalyst or something. But whatever. I'm old I guess
Honestly the OS X itself (uhhh macOS now) is still fine, even with Big Sur and its weird iOS controls, however the other software is sometimes weird; but - let's say Numbers was never good and iWeb was always downright bad. And iTunes was bad in Jobs era and is still bad (as Music).
I however still kind of miss when native Cocoa apps were sort of good and not just ported iOS app via Catalyst or something. But whatever. I'm old I guess
Don't the bomb icon and "Sad Mac" go back to the very beginning?
When OS X came out 20 years ago, it was the star of the show. Nowadays it's just an annoyance that Apple has to keep around. The quality has demonstrably fallen, and the internal struggles it's had within Apple have already been documented.
Honestly, Apple's decline traces back to losing Steve Jobs. The strategic changes were immediate, (and aimed at shorter term gains, things that people besides Jobs thought were being left on the table) the effects just took a long time to really add up to something meaningful. IMO, they never really recovered, they just are massive and have a HELL of a lot of momentum. It takes time to undermine that all even if you do everything wrong. They definitely are still doing some things right, but IMO, the type of thing you are complaining about is a leadership issue.
Adobe executes (executed) massive releases on a well-defined cadence (1.5 yr per "creative suite"). Sure, I won't say they don't have collaboration, or top-tier engineers etc. But in the end, the success of this well defined cadence boiled down to the simple strategy of "we release what is ready". Everybody knew the deadlines well in advance, and everybody strives to 'make it' - but, (not so) occasionally, some teams didn't and you could see entire features get ruthlessly cut out. Simply no other way of synchronizing across the entire company... you cut scope to make it in time, and if you can't make it, that's it, your feature gets cut. With enough teams working, you still get features to showcase at each release.
That's pretty much how Linux kernel releases work. Just having a regularly scheduled release train that always runs on time – that you can catch (or not) is great.
> "we release what is ready"
this really is key, if you can do it
alot of companies though, are "business driven" which means releases and schedules are set beforehand, which makes this hard... (but if you can do it, its great!)
this really is key, if you can do it
alot of companies though, are "business driven" which means releases and schedules are set beforehand, which makes this hard... (but if you can do it, its great!)
What happens when you can release whenever. This question is probably quite relevant to most developers, because CD (continuous delivery) is popular.
> Apple executes massive feature releases on a yearly waterfall-like schedule across hundreds of teams. It requires [...]
And no doubt a great deal of crunch time, their fair share of delays, some major messes and the occasional outright cancellation. And if I'm not wrong their roadmap is not public information.
I personally find it hard to believe that a large company can exhibit all those adjectives you list. Everyone finds this hard.
And no doubt a great deal of crunch time, their fair share of delays, some major messes and the occasional outright cancellation. And if I'm not wrong their roadmap is not public information.
I personally find it hard to believe that a large company can exhibit all those adjectives you list. Everyone finds this hard.
When you have hundreds of teams you can cut a bunch of features and it still looks like you are massively productive each release :) Not everyone has that luxury unfortunately. Some of us are a single team of a handful of developers.
How do you know they are thoroughly waterfall? How do you know they don’t use agile, time-boxing, etc.? Apple is so secretive, it feels like we know of little other than what they are confident about releasing. The things that weren’t done on a schedule compatible with a hardware release, how often are they simply postponed? And for those postponements that they do admit, how many are waved away with a creative explanation? I think you need a good deal more information to back up your point about Apple’s work management.
I worked there 6 years. It is 100% Waterfall. Things that are too buggy by July 1st are delayed by 12 months.
How do you know it's waterfall-like? If you ship working software every 2 weeks and every 26th one you release, hooray, you have yearly software that's super likely to work well.
Apple produces a new build of iOS or MacOS every 24 hours. For the many years that I had to suffer through that, about 40% of those builds were dogshit and unusable. The public gets the 360th release, and hopefully by then the PMs have driven enough burnout in the engineers that it is mostly acceptable to users.
Makes sense!
That doesn't imply anything about estimates.
Do they tell you at the beginning of that cycle what features to expect? Or do they constantly work on features, and then at presentation time do a freeze?
Do they tell you at the beginning of that cycle what features to expect? Or do they constantly work on features, and then at presentation time do a freeze?
[deleted]
The macbook range had years of delays
I think it's because inexperienced programmers don't think about the "soft" costs of programming.
Requirements gathering, general communication, testing, production rollout, coordination, research, final approval - it all adds up. For a 2 week project you might only actually spend a day of ass-in-seat coding. And that ass-in-seat coding is what they tend to estimate.
Requirements gathering, general communication, testing, production rollout, coordination, research, final approval - it all adds up. For a 2 week project you might only actually spend a day of ass-in-seat coding. And that ass-in-seat coding is what they tend to estimate.
I think this is a big factor in estimation inaccuracy for really new programmers - but it is pretty well fixed by project managers following estimation rules of thumb - like double it and add five days... The bigger issue that I am less certain about being teachable is the experience side of things. I've worked in industry for a while and I know how long different sorts of patterns will take to write and what sorts of complications different sorts of add-ons will add to those numbers. Some projects are about where my naive brain fifteen years ago would put them and other ones are wildly different due to complexities that aren't visible in a surface level examination. Learning to sniff out those complexities seems to just take experience (not with every individual scenario - but it's a pattern analysis thing).
> I think it's because inexperienced programmers don't think about the "soft" costs of programming.
That's what the research has previously found[0] decades ago -- engineers will generally correctly estimate the part they are best at, but will either neglect or under-estimate other work. The old-school ways of dealing with this are canned Work Breakdown Structures and checklists.
This is not unique to software.
[0] van Genuchten, "Why is Software Late? An Empirical Study of Reasons For Delay in Software Development" https://web.archive.org/web/20160914040151/https://lab.cs.ru...
That's what the research has previously found[0] decades ago -- engineers will generally correctly estimate the part they are best at, but will either neglect or under-estimate other work. The old-school ways of dealing with this are canned Work Breakdown Structures and checklists.
This is not unique to software.
[0] van Genuchten, "Why is Software Late? An Empirical Study of Reasons For Delay in Software Development" https://web.archive.org/web/20160914040151/https://lab.cs.ru...
I do a fair bit of writing at work of various different types of things.
For stuff that I'm doing on my own? I can estimate pretty well. I know more or less how long the writing will take based on length. I'll allocate more time for research if I need to do more than a base amount. And then I add some slack in case I'm struggling with something, priority interrupts happen, or I'm just not into it on some given day. (Underpromise. Overdeliver.) It only gets harder with bigger projects. It helps that I'm quite fast compared to most people so even with significantly underpromising people are usually very happy with the estimate.
But where it's really hard is work that involves getting buy-in and input from a lot of different people. Those always take forever. (And at this point I pretty much know they'll take forever, a lot of which is just latency rather than actual active work.)
For stuff that I'm doing on my own? I can estimate pretty well. I know more or less how long the writing will take based on length. I'll allocate more time for research if I need to do more than a base amount. And then I add some slack in case I'm struggling with something, priority interrupts happen, or I'm just not into it on some given day. (Underpromise. Overdeliver.) It only gets harder with bigger projects. It helps that I'm quite fast compared to most people so even with significantly underpromising people are usually very happy with the estimate.
But where it's really hard is work that involves getting buy-in and input from a lot of different people. Those always take forever. (And at this point I pretty much know they'll take forever, a lot of which is just latency rather than actual active work.)
Programmers can also try to come up with reasonable estimates, only for management (product, middle or upper - or often all of them) to come back and assert that the estimates are off base and to "reconsider." Not much to do at that point but to give a lower estimate and hope it works out. If you make that your hill to die on as a programmer you will find yourself out of a job with only your integrity to keep you warm at night.
> ...nobody cares!
> Nobody learns.
> It's insane.
Estimation is really hard, especially if you're dealing with new challenges, new people, and new expectations. Throwing moving targets (unplanned changes) into the mix, as you noticed, makes estimation even more difficult.
The thing is, it very rarely is the case that a wrong estimate will sink a project. Things get delivered late all the time. No one dies. Most stakeholders aren't like two year olds throwing a tantrum because an unforeseen problem delays a deliverable by a few days (admittedly some are close).
The actual definition of failure also needs to be considered. Is it considered a failure because a project manager ended a colored gantt bar at a particular date and the actual time was longer? Or is it a failure because the customer cancelled the contract because a deliverable was not available on the agreed-upon date? Or is it something far more nebulous, like a slow death march towards a misguided goal that burns everyone out and compels them to sip artisanal coffee and daydream about leaving instead of grinding onwards and hitting targets?
Ultimately, people "don't care" because accurate estimates are not the most critical problem on their mind. Most of the time an accurate answer to "When will it [the deliverable] be done?" has about as much gravitas as guessing the number of jellybeans in a jar at a kid's birthday party.
Doing a retrospective and analyzing what went wrong when a project delivery slipped expectations is certainly reasonable but it so easily slips into a blame storm and/or people making commitments that aren't realistic. Getting straight answers from folks for this kind of analysis requires a level of psychological safety that just isn't present in most workplaces.
Estimation is really hard, especially if you're dealing with new challenges, new people, and new expectations. Throwing moving targets (unplanned changes) into the mix, as you noticed, makes estimation even more difficult.
The thing is, it very rarely is the case that a wrong estimate will sink a project. Things get delivered late all the time. No one dies. Most stakeholders aren't like two year olds throwing a tantrum because an unforeseen problem delays a deliverable by a few days (admittedly some are close).
The actual definition of failure also needs to be considered. Is it considered a failure because a project manager ended a colored gantt bar at a particular date and the actual time was longer? Or is it a failure because the customer cancelled the contract because a deliverable was not available on the agreed-upon date? Or is it something far more nebulous, like a slow death march towards a misguided goal that burns everyone out and compels them to sip artisanal coffee and daydream about leaving instead of grinding onwards and hitting targets?
Ultimately, people "don't care" because accurate estimates are not the most critical problem on their mind. Most of the time an accurate answer to "When will it [the deliverable] be done?" has about as much gravitas as guessing the number of jellybeans in a jar at a kid's birthday party.
Doing a retrospective and analyzing what went wrong when a project delivery slipped expectations is certainly reasonable but it so easily slips into a blame storm and/or people making commitments that aren't realistic. Getting straight answers from folks for this kind of analysis requires a level of psychological safety that just isn't present in most workplaces.
Failure is a tricky thing.
There is a story that Microsoft Word was a staggering failure in terms of actual time and cost to implement versus its estimates.
It then went on to be among the most lucrative ($ billions) pieces of software ever created.
Was Microsoft Word a failure?
There is a story that Microsoft Word was a staggering failure in terms of actual time and cost to implement versus its estimates.
It then went on to be among the most lucrative ($ billions) pieces of software ever created.
Was Microsoft Word a failure?
Internal estimates are a political tool, not a technical, financial or scientific one: First, decide if you want to do it or not. Then do the estimate, high if you don't want to do it, low if you do want to do it.
Any kind of reality doesn't factor into it.
In the Word example, somebody really thought this project was worthwhile for some reason, so they gave a very low estimate to get things started. As soon as money has been spent on a project, it isn't likely to be cancelled as easily, so you are safe to continue even if you exceed your original estimate. In the worst case, you need a few excuses.
External estimates, i.e. estimates you use for writing up a contract with an external customer, are different. Those actually do matter, depending on the wording of the contract. But a smart contractor will try to get the contract into a shape such that the above for the internal estimate does apply.
Any kind of reality doesn't factor into it.
In the Word example, somebody really thought this project was worthwhile for some reason, so they gave a very low estimate to get things started. As soon as money has been spent on a project, it isn't likely to be cancelled as easily, so you are safe to continue even if you exceed your original estimate. In the worst case, you need a few excuses.
External estimates, i.e. estimates you use for writing up a contract with an external customer, are different. Those actually do matter, depending on the wording of the contract. But a smart contractor will try to get the contract into a shape such that the above for the internal estimate does apply.
The problem is knowing how fast they learn. For a n00b this is basically impossible, and takes cost monitoring to notice their rate of learning. For a person with the basics of the language that they'll be using, and a tiny bit of experience in the libraries they'll be using, things get easier.
For everyone (n00b to senior) I would always recommend designing & breaking things down into no smaller 1/2 day blocks, where nothing went longer than 3 days (for inexperienced), but longer for more experienced. Senior devs should be able to be able to be trusted (to be accurate) for week-long blocks.
Use a Gannt chart (ms project or similar) and monitor the inexperienced, gauging their real-life times with their estimates, adjusting accordingly. Give semi regular feedback at the accuracy so that they (& you, the manager) are working with.
In one of my companies, the most inaccurate we were for any project was that we were late 2 days over 2 years. Build on trust, build on meetings and other time sinks. Include testing, sign offs, etc.
So, absolute n00bs need close mentoring so that the (senior) mentor can begin to gauge the growth rate... Which will be steep... But different for everyone.
For everyone (n00b to senior) I would always recommend designing & breaking things down into no smaller 1/2 day blocks, where nothing went longer than 3 days (for inexperienced), but longer for more experienced. Senior devs should be able to be able to be trusted (to be accurate) for week-long blocks.
Use a Gannt chart (ms project or similar) and monitor the inexperienced, gauging their real-life times with their estimates, adjusting accordingly. Give semi regular feedback at the accuracy so that they (& you, the manager) are working with.
In one of my companies, the most inaccurate we were for any project was that we were late 2 days over 2 years. Build on trust, build on meetings and other time sinks. Include testing, sign offs, etc.
So, absolute n00bs need close mentoring so that the (senior) mentor can begin to gauge the growth rate... Which will be steep... But different for everyone.
It's very useful to break tasks down so instead of a month long "upgrade elasticsearch" story, there's dozens of day-or-less tasks to work on. Often it's not noticed how much effort goes in to creating these new smaller tasks. If you aren't extremely familiar with the targeted version of elasticsearch, you'll have to spend significant amounts of time investigating, planning, and writing.
You can get very high accuracy on estimates by spending lots of time creating smaller stories and encouraging people to over estimate. This may not be in the best interest of the project or the business.
You can get very high accuracy on estimates by spending lots of time creating smaller stories and encouraging people to over estimate. This may not be in the best interest of the project or the business.
That means the business is taking safe bets. Depending on the industry this makes a lot of sense. If you are in a small business this probably won't help you.
More important than estimating is knowing how to sequence the work. I often see teams try to build the entire DB schema first, or built the interface out one wire at a time. The agile approach of building small, vertical parts of each feature tends to work much better in my experience.
For example say you have a CRUD application with a page for editing and searching complex objects. Rather than building the entire search page with all of its filters, and then building the edit page with all of its fields, you build a simplified version of the search page with only a few filters and a simplified version of the edit page with fields that correspond to those filters. You also make sure the functionality integrates with the backend to form a cohesive, vertical feature that can go through QA, and possibly the client. Once that is working you expand the list of fields and filters.
This approach tends to surface design issues much faster, and is more adaptable to change requests from the client.
For example say you have a CRUD application with a page for editing and searching complex objects. Rather than building the entire search page with all of its filters, and then building the edit page with all of its fields, you build a simplified version of the search page with only a few filters and a simplified version of the edit page with fields that correspond to those filters. You also make sure the functionality integrates with the backend to form a cohesive, vertical feature that can go through QA, and possibly the client. Once that is working you expand the list of fields and filters.
This approach tends to surface design issues much faster, and is more adaptable to change requests from the client.
agree with the adaptability of change requests from clients, but i would say only insofar as the design issues you're mentioning are actually addressed as quickly as they are surfaced. Otherwise this type of building eventually grinds to a halt.
another commenter sort of hinted at this, but this type of process has a downside of prioritizing short term design decisions that lead to working code. It takes time and sometimes a lot of thought to think about the system and the problems deeply and ask yourself if you've created the right primitives.
another commenter sort of hinted at this, but this type of process has a downside of prioritizing short term design decisions that lead to working code. It takes time and sometimes a lot of thought to think about the system and the problems deeply and ask yourself if you've created the right primitives.
Although when working like this there can be a "Good enough" inertia
Make a complete list of things to develop in a spreadsheet, "complete" means every single item the customer wants to see in the product, not only every single button but also every single label, that should be definable by reading the project specs or mockups. I think "forgetting things" is the first big mistake leading to under estimations.
Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work.
I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV
Only problem: the customer might not like what he gets, even if it works exactly as planned.
Add an estimate that you multiply by three in front of each, ie. if the dev thinks 1 hour then they should put 3. From my experiments: no multiplying factor turned to nightly and weekend work, multiplying by 2 turned to long (10 hours) work days, multiplying by 3 turned into comfortable work hours and quality work.
I've had success delivering quality code and respecting deadlines since 2012 or so with this system, but YMMV
Only problem: the customer might not like what he gets, even if it works exactly as planned.
> "complete" means every single item the customer wants to see in the product,
Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.
Ah, but the customer does not know everything they want, due to the fractal nature of reality. The closer you get to the end product, the more detail is resolved, and more work is added.
Oh man, thank you for the fractal nature of reality. That is the perfect term to describe what I saw about ever-changing requirements. No matter how closely you define something, things will always keep changing on all levels when closer to actually "delivering" / launching. Adding time, costs, confusion, frustration. Fractal, yes, that's it :)
> No matter how closely you define something, things will always keep changing on all levels when closer to actually "delivering" / launching
Xeno's paradox [0] doesn't apply here. Some projects do actually finish.
[0] https://en.wikipedia.org/wiki/Zeno%27s_paradoxes#Achilles_an...
Xeno's paradox [0] doesn't apply here. Some projects do actually finish.
[0] https://en.wikipedia.org/wiki/Zeno%27s_paradoxes#Achilles_an...
Another way of looking at this is that (unless there is a complete disaster) as you get closer to the end the uncertainty in the estimate decreases. When you get to the end, you know precisely how long it took.
Although this sounds flippant, the design process is where unknowns should be uncovered. It is easier to change design before a line of code is written.
Design is precisely what is needed in order for the dev team to break down the projects into tiny chunks where nothing is unknown.
Your time estimates should always include time for research and design. The customer can sign off on what you're about to write at the end of that (whereas beforehand you give them a wild ballpark figure made by senior devs - not management!)
Your time estimates should always include time for research and design. The customer can sign off on what you're about to write at the end of that (whereas beforehand you give them a wild ballpark figure made by senior devs - not management!)
Yeah but too many customers change their mind even after they've signed off. Then you're left with strictly enforcing a contract or making adaptations.
You're right, design is very very necessary and should be given plenty of time. But it's not a magic bullet.
You're right, design is very very necessary and should be given plenty of time. But it's not a magic bullet.
In terms of contract, I have found this works best (recommended by my lawyer (a former ASF lawyer)): 30% upfront, 50% at first delivery, then they have 2 months to find bugs, final delivery is when the bugs are fixed that's when you take the last 20%. This way, you're certain to always get 80%.
If they want the liberty to "change their mind on the fly" then they should buy man-days which is a completely other thing. Or, they can buy supplementary mandays for whatever they want at the daily rate without changing the contract, or, they can just wait to finish this contract and start another one.
If they want the liberty to "change their mind on the fly" then they should buy man-days which is a completely other thing. Or, they can buy supplementary mandays for whatever they want at the daily rate without changing the contract, or, they can just wait to finish this contract and start another one.
it's pretty a well known saying that the last 10% of any project takes 90% of the time.
ReRe's Law of Repetition and Redundancy [4] somehow does not apply to you:
A programmer can accurately estimate the schedule for only the repeated and the redundant. Yet,
A programmer's job is to automate the repeated and the redundant. Thus,
A programmer delivering to an estimated or predictable schedule is...
Not doing their job (or is redundant).
I think the explanation was made in another comment where they say "don't forget to estimate time for research & design", which explains why the *3 multiplier works for me. And yes, I feel that most of the work is redundant, there's always like network services of some sort or web pages, databases, linux servers, a humain interface, other services depending on the project like a message queue or a caching server or both, background workers and scheduled code execution... things I had been doing for ~10 years back in 2012. Data structures, workflows, they change between gigs, and each gig has its own particularity, but 90% is the redundant from a gig to another: HTML/CSS/JS/Python/SQL/Networking/Storage/Linux stuff
I've done backlogs before; both very high-level and very detailed; with and without mockups (even final graphic designs) and detailed description in the form of multi-page spec or user stories (depending on the fashion of the decade I was doing it in :).
I don't think having the detailed requirements made estimates much more accurate. However it made creating those estimates take an order of magnitude longer - developers needed much much more time to absorb all the details.
At that point I decided I only ever want to work with senior developers who have good imagination and can give reasonable triplets (solution, # of devs to do it, approximate time) for requested pair (idea, quality level).
I don't think having the detailed requirements made estimates much more accurate. However it made creating those estimates take an order of magnitude longer - developers needed much much more time to absorb all the details.
At that point I decided I only ever want to work with senior developers who have good imagination and can give reasonable triplets (solution, # of devs to do it, approximate time) for requested pair (idea, quality level).
Software development is the process of writing a complete specification, including every single button and label. If you have a formalised complete design, you could write a compiler to do the work for you.
I forgot to add this to my other comment. Yes, a multiplier works wonders. (I typically added 30% to my own estimates, which covers all the little incidentals that trip you up, and coffee breaks).
I've not been forced to work longer than an 8hr day since the 90's.
I thought your comment stating that time should be added for both research & design was actually a reasonable explanation for why I need the *3 multiplier!
Back when I was a product manager, there was an engineering manager who would drive me crazy. He had this idea of a 90% schedule, by which he meant a 90% chance of hitting if everything went as planned. Everything never went as planned.
He probably read Goldratt's follow-on to The Goal, Critical Chain. Critical Chain Project Management is the specific term. Goldratt tried to sell software to companies that would help them make schedules like that, but a large component (often ignored) was collecting historic data, and running models. Most companies might do one of those, but rarely both, and of course models with only speculative numbers are absolute garbage. In my experience, places that tried to apply CCPM never actually grokked it, and it was some manager's obsession because they were buzzword crazed (like many big-A Agile offices).
I think this was a little before that (mid-90s) but maybe something related.
If someone can teach _any_ programmer how to estimate accurately they will literally print money. Its inherently a hard problem (I'd argue a siren song). To truly succeed you need to perform a massive stakeholder education on software development, hidden costs, and the highly dynamic nature of the beast.
Then you need to train the IT professionals to build the dynamic nature into the conversation and process. You dont quote a number, you give a range. You communicate confidence levels and uncertainty clearly and continuously. You update assumptions as new information comes available. You _push back_ when the uncertainty is frowned on.
It can be done successfully on a micro scale sure, by working on it within an organization and/or team. But its not scalable and its not a 5-step plan. Its hard.
Then you need to train the IT professionals to build the dynamic nature into the conversation and process. You dont quote a number, you give a range. You communicate confidence levels and uncertainty clearly and continuously. You update assumptions as new information comes available. You _push back_ when the uncertainty is frowned on.
It can be done successfully on a micro scale sure, by working on it within an organization and/or team. But its not scalable and its not a 5-step plan. Its hard.
As long as humans are involved in defining requirements, inaccuracy of estimate will be proportional to the cusomters' incompetence and level of bad politics of the organisation as a whole ;-)
Estimates don't give you the time when something will be done. You can just be almost sure that the thing will not be ready before this time. And that's still something.
The principles behind the Manifesto for Agile Software Development mention only retrospectives, not estimates. You could say they value learning over guessing.
When people ask me how can they learn, I tell them to find old project schedules and other project documents. Looking at past projects makes the learning more objective, and it is not happening under time pressure.
When people ask me how can they learn, I tell them to find old project schedules and other project documents. Looking at past projects makes the learning more objective, and it is not happening under time pressure.
"Estimates" are for things you've done before - like you can estimate building a house, because people have built houses before. The more like an existing house, the better you can estimate it.
Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.
Software is invention and construction. The construction part is pretty easy to estimate. The invention part is ... very very hard. I'd like to say it's impossible. I'd like to see the software industry use a different word than estimate.
You've perfectly nailed the fundamental issue here.
For many software projects, you simply cannot make meaningful granular estimations for parts of them. It doesn't matter how many story-point poker sessions you hold. Some software work cannot be reduced to a positive integer. In our business scopes creep, bugs plague us, and myriad issues make precise estimations pointless. It's engineering Numberwang.
I get that folks want to tame a chaotic world. But sometimes you can't. And the software engineering field in particular feels wildly neurotic about pretending otherwise. Sometimes I imagine applying our project management tools to other technical fields and laugh at how insane we must look.
For many software projects, you simply cannot make meaningful granular estimations for parts of them. It doesn't matter how many story-point poker sessions you hold. Some software work cannot be reduced to a positive integer. In our business scopes creep, bugs plague us, and myriad issues make precise estimations pointless. It's engineering Numberwang.
I get that folks want to tame a chaotic world. But sometimes you can't. And the software engineering field in particular feels wildly neurotic about pretending otherwise. Sometimes I imagine applying our project management tools to other technical fields and laugh at how insane we must look.
> It's engineering Numberwang.
Can I just say that is the greatest description of software estimation I have ever read? I am 100% stealing that.
Can I just say that is the greatest description of software estimation I have ever read? I am 100% stealing that.
LET'S ROTATE THE SPRINT BOARD
We use it internally ourselves. It’s from That Mitchell and Webb Look.
That estimation is hard does not make it impossible. That it may be inaccurate does not make it worthless. Special pleading that our industry is somehow the only one that faces uncertainty and novelty doesn't really hold water.
> "Special pleading that our industry is somehow the only one that faces uncertainty and novelty doesn't really hold water."
I'm sure you're right. I don't think we should abandon all hope. I just think we should be more honest. I also think that software has fewer constraints than other industries, which is a blessing and a curse for estimation
I'm sure you're right. I don't think we should abandon all hope. I just think we should be more honest. I also think that software has fewer constraints than other industries, which is a blessing and a curse for estimation
Software is unique because it can be copy pasted. A builder that makes a house in 4 months the first time might estimate 3.5 months for a near identical house. But to deploy the same desktop app on two computers you merely reuse the installer from the first computer, and to recreate the same page in a different app you can copy the code, so no one ever estimates doing the same work twice. Every software problem has at least a little novelty, because if it isn't at least a little different, the work is already done.
>I get that folks want to tame a chaotic world. But sometimes you can't.
No matter how little you know about something you want to estimate, there is an infinite difference between what you do know and knowing nothing. The right thing to do is to represent your knowledge without fabricating it.
Saying "I can't make a point estimate, therefore I can say nothing" is a false dichotomy.
Even if you say, x is between 0.001 and 10^10, you are narrowing things down.
There is a great paper mentioned periodically on HN, where they examined the famous Drake equation and show that when you get rid of point estimates and use reasonable ranges for each variable, the "paradox" that we don't see aliens vanishes. Despite many of the variables having uncertainty of many orders of magnitude.
No matter how little you know about something you want to estimate, there is an infinite difference between what you do know and knowing nothing. The right thing to do is to represent your knowledge without fabricating it.
Saying "I can't make a point estimate, therefore I can say nothing" is a false dichotomy.
Even if you say, x is between 0.001 and 10^10, you are narrowing things down.
There is a great paper mentioned periodically on HN, where they examined the famous Drake equation and show that when you get rid of point estimates and use reasonable ranges for each variable, the "paradox" that we don't see aliens vanishes. Despite many of the variables having uncertainty of many orders of magnitude.
> Some software work cannot be reduced to a positive integer
I'd take that a step further: if software work can be reduced to a positive integer, you're doing it wrong.
I'd take that a step further: if software work can be reduced to a positive integer, you're doing it wrong.
I'd beg to differ. There is very little invention going on. Most software solutions tackle well-known problems, customized to a particular business need. Akin to building a house, but with specific owner requirements (three-car garage, etc).
It gets a little complicated partly because of the industry's penchant for reinventing its tools on a rolling basis. In the trades, technology remains largely unchanged over decades, and only truly useful new advances are integrated into workflows (for example, laser level vs plumb bob or spirit level).
In software, new technological approaches are adopted on a continuous basis, and so a lot of effort is spent on learning how to solve old problems with new tools.
It gets a little complicated partly because of the industry's penchant for reinventing its tools on a rolling basis. In the trades, technology remains largely unchanged over decades, and only truly useful new advances are integrated into workflows (for example, laser level vs plumb bob or spirit level).
In software, new technological approaches are adopted on a continuous basis, and so a lot of effort is spent on learning how to solve old problems with new tools.
> Most software solutions tackle well-known problems, customized to a particular business need. Akin to building a house, but with specific owner requirements (three-car garage, etc.).
Not really. The "building" part in software really is compiling and deploying your code. With CI/CD being set up, this takes a few minutes and requires almost no effort.
So what's left is not "building" the house but "designing" it. Unfortunately, the comparison you give fits more for buying a software and installing/configuring it. Then yes, this should be rather well estimatible, given that you have done it before.
But most software developers are not paid for that. They are paid to design a space station that has never existed before. Sure, they can use existing parts (libraries, frameworks) but it is still design in the sense that certain expectations are simply not possible. Or someone wants that the space station is connected to the other space station on another planet and we don't even know how the other space station looks like (it's built by another company). Who knows if it's even possible to connect them? And if not, the other space station might have to be adjusted to be connectable and this must be done by the other company. How long will that take? Who knows...
By the way, don't think about a space station like the ISS here. Please think about a space station like the star wars death star. Because that's the complexity of most software where estimates are desired and important.
And we are only talking about design, not building. The building part is easy - once we have designed it, we can actually copy it and have 100 of these exact space stations easily.
Not really. The "building" part in software really is compiling and deploying your code. With CI/CD being set up, this takes a few minutes and requires almost no effort.
So what's left is not "building" the house but "designing" it. Unfortunately, the comparison you give fits more for buying a software and installing/configuring it. Then yes, this should be rather well estimatible, given that you have done it before.
But most software developers are not paid for that. They are paid to design a space station that has never existed before. Sure, they can use existing parts (libraries, frameworks) but it is still design in the sense that certain expectations are simply not possible. Or someone wants that the space station is connected to the other space station on another planet and we don't even know how the other space station looks like (it's built by another company). Who knows if it's even possible to connect them? And if not, the other space station might have to be adjusted to be connectable and this must be done by the other company. How long will that take? Who knows...
By the way, don't think about a space station like the ISS here. Please think about a space station like the star wars death star. Because that's the complexity of most software where estimates are desired and important.
And we are only talking about design, not building. The building part is easy - once we have designed it, we can actually copy it and have 100 of these exact space stations easily.
> "There is very little invention going on."
If you said there is very little fundamental research going on, I would agree with you; but even putting together building blocks in a new way is invention. If you look at patents (arguably a unit of invention), most of them don't have fundamentally new building blocks, just existing blocks put together in new ways.
If you said there is very little fundamental research going on, I would agree with you; but even putting together building blocks in a new way is invention. If you look at patents (arguably a unit of invention), most of them don't have fundamentally new building blocks, just existing blocks put together in new ways.
Very true, I think those estimates are actually two ideas/types crammed into one value.
1. The construction part, as you said, can be estimated.
2. I'd just call the other thing "allocated time" instead of "estimated time".
Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem.
For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.
1. The construction part, as you said, can be estimated.
2. I'd just call the other thing "allocated time" instead of "estimated time".
Any time someone asks me how long it will take me to fix a bug that I haven't really looked at yet, or to plan some new feature or something like that, and they badly need a number, I ask them how much time I should allocate to that. I can't promise to have something like that done by that time, but it gives us both an idea about how to treat that problem.
For example, we could allocate two hours to fix a bug, with the understanding that if that turns out to not be enough then we'll need to talk about workarounds. Or we can allocate two days to plan a new feature, and the best solution we can think of in that time shall be the one we use.
This is brilliantly put. That's pretty much what engineering is – constraints and tradeoffs – giving software project planning/execution a healthy vocabulary to talk about makes it so much better. Much of the crappy feeling that developers go through by putting themselves into a corner can be avoided if everyone had better mental/language models to think/talk about it.
Yes. So you can more accurately estimate replicating components you’ve written before, eg another database client, but estimating new software is a successive sequence of uncertainty reduction. The first pass being highly intuitive, the second pass, with more of the details worked out, less so... until you’ve completed the first working version, and you can finally say in hindsight, it actually took this long.
Yes! I try to always phrase it as "the part I can see from here will take at least X time".
Yes, and if you find yourself getting better at estimation, that's probably because you have failed to build proper abstractions. With proper abstractions, the cost of the same stuff should be minimal, so the part with no good priors predominates.
I mostly agree, but for research, estimating the estimation is often good enough in practice.
For me it often something like this:
I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.
For me it often something like this:
I don't have a clue how the hell to do what you asking for. But maybe implementing .. might help. Can't guarantee but it might. I think I can confirm or disprove that spending .. on the prototype subject to following limitations.. If then we find out it actually works for you, we'll go from there but approximately gonna take .. extra to rework the prototype into production-quality stuff. If it won't, I'll think about something else to try.
[deleted]
I have been involved in Software engineering research a bit, even have a first-author short paper. I was struck by just how much pointless, low-effort papers there are in this domain. People have been researching about bug prediction for over two decades now, and judging by the paper quantity, this isn't a niche area. Yet how many organizations do actually employ those systems in real-world? Can't comment on industry, but I haven't found a single open-source program that does that [1].
Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research.
[1] I saw Mozilla attempt it, but not sure if it's currently in use.
Now I know "most papers are pointless" is a common complaint in science, specially in my area of focus- machine learning. But I can't shake the feeling that the situation is particularly worse in software engineering related academic research.
[1] I saw Mozilla attempt it, but not sure if it's currently in use.
Couldn't agree more. Software engineering is a very practical, fast-moving subject, and that makes most academic work even more irrelevant - software engineering academics often don't have the first clue about what real software engineering is about, because they don't have recent industrial experience.
I am 20 years in development business now.
This simple rule of thumb works for me and the team:
(Your honest and concise estimation) * 3
There are just to many unknowns you cannot foresee. Software development is complex.
There are just to many unknowns you cannot foresee. Software development is complex.
same experience level, similar formula. Take your gut estimate, double it, double that new value, then add another "increment".
e.g. say you think it will take a day, so 2 days, 4 days, add another day, likely estimate is 5 days.
My pet theory is that when we estimate, we typically think of how long it will take to figure out a working solution to the problem, but forget about how long it takes to debug it, add tests, rework for changed requirements and unexpected nuances, and then roll it out and do any training, etc.
e.g. say you think it will take a day, so 2 days, 4 days, add another day, likely estimate is 5 days.
My pet theory is that when we estimate, we typically think of how long it will take to figure out a working solution to the problem, but forget about how long it takes to debug it, add tests, rework for changed requirements and unexpected nuances, and then roll it out and do any training, etc.
Great anecdote posted here a few years ago: https://www.quora.com/Why-are-software-development-task-esti...
Great read!! :D
This, the famous "prediction* Pi" rule seems to be correct also in my experience.
I make the most optimistic prediction and then * PI.
I make the most optimistic prediction and then * PI.
I use a similar methodology.
The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).
The part you didn't mention is that for most businesses it's better to be over than under in the estimation. I also explain this thought process to the various stake holders. We can certainly try to tighten up an estimate, but that runs a higher risk of being under which is usually a worse outcome (promised launch dates are missed, marketing is missed/happening, customers are told, etc...).
I've been hearing this rule for years now, and I think the coefficient is increasing. It started at 1.5x, now it's at 3x... Is this just me?
I'm such an optimist! I multiply by 2.2.
That's a recursive definition (estimate * 3) = ((estimate * 3) * 3) = (((estimate * 3) * 3) * 3)... But if you do that for a few years, "your honest and concise estimation" starts to grow because you've seen how it usually takes longer than expected, and your coefficient can approach 1.
I couldn't agree more. This is what I do.
I multiply by 4. Specifically, giving estimates on when features will be deployed/usable by end users.
yes, going from identification of the bug/idea to the solution in end users hands is a lot different than just the coding estimate. On teams with multiple levels of management both engineering side and customer side the actual code required to fix/implement the solution is such a small part of the overall effort you could, ironically, consider it immaterial.
Software development which is a repeatable and already defined process is totally possible to predict and estimate. Most tasks of repeatable processes follow normal distribution and is predictable. Deviations from expected mean will be due to predictable factors of the environment such as failed disk or sleepy programmer. You can apply arbitrary six sigma methodology to measure such process with accuracy.
The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another.
Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts.
Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance.
Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity.
Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.
The problem in software though is that such a repeatable process would be immediately automated away by writing a function, library, framework or any such tools that programmers use on a daily basis without much thinking. Unlike in building construction, to which programming discipline is often wrongly likened to, where construction companies simply cannot "write a function" to deploy cookie cutter houses or bridges one after another.
Therefore software engineering is never a repeatable process, unless crappy tools are used, which don't allow for sufficient abstraction of repeatable parts.
Tasks in software disciplines therefore don't follow a normal distribution. They follow exponential distribution most often. Most issues go unnoticed. Majority are just so tiny and ofthen considered business as usual. Every time you get stuck and have to look up a solution in docs or stackoverflow technically is an issue, but never gets reported in an issue tracker for its triviality. There are however issues which are orders of magnitude larger than what management expects when they occassionally sampling issue trackers. Some issues lead to critical design flaws which could need a full blown rewrite for example, or ever lasting and expensive hackery in case the executive decision is to work around the critical design flaw. These issues can take years to accomplish or take massive amount of pain endurance.
Trying to estimate a process with such exponential distribution and making sense of averages or other statistics of such distribution is borderline insanity.
Why not just go to physics department and ask when the next Theory of Relativity will be invented and how much budget and story points those guys need.
"a repeatable process would be immediately automated away by writing a function, library, framework or any such tools"
This is the crucial point here.
Source code is a (almost) self-assembling blueprint.
The actual product that will be build is the software and software is a configuration of matter, in this case of a computer.
The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house.
With better robotics, we will probably get there at some point in the future.
And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.
This is the crucial point here.
Source code is a (almost) self-assembling blueprint.
The actual product that will be build is the software and software is a configuration of matter, in this case of a computer.
The source code/blueprint for a house is not self-assembling. Compiling such a blueprint requires you to configure the building materials in a way that they become a house.
With better robotics, we will probably get there at some point in the future.
And with software we will always be in a place where you either do new stuff the first time manually or with crappy tools the 100th time.
The most useful piece of advice I have gotten for estimates is that they are all junk until someone sits down and tries to do the work. For big jobs, 20% of it, small jobs pushing 50% of the work.
In every single case when you've done that much work, I seem to wind up with a reasonable estimate.
If we've done the job before and have data on it, also reasonable.
Double or triple anything else.
In every single case when you've done that much work, I seem to wind up with a reasonable estimate.
If we've done the job before and have data on it, also reasonable.
Double or triple anything else.
Scrum dogma is that estimates are for complexity, not effort or timing. The points you track in JIRA are meant to reflect how much of the current backlog is complete and how much is remaining. That can be extrapolated into timing but can't be done up front.
If it can't be done upfront it's largely useless for business folks ("so you have an estimate but can't translate it into time? so why do it? go away" :).
If it assumes remaining work is equivalent in time to done work (notion of velocity), it's (in my view) very optimistic. People learn the nature of the beast as time progresses. Collective learning happens slowly. A lot of important work is usually only discovered and planned in the second half of the actual (not original) timeline. A lot of important stakeholders are naturally only introduced into projects close to its finalization, which creates a flurry of new activity and discoveries. This makes projects late.
But where that real halftime is - most people I worked with rarely know. I was no better, I also rarely knew.
The only way to manage this I found was the basic rule: build, demo, decide what to do next. Don't get attached to original backlog and grand plans for the future. It won't work that way. Just build and make it work, launch ASAP, get decisions on a ~weekly basis done. Ignore long term planning. It's not helpful.
Sadly, few business folks want that of course :)
If it assumes remaining work is equivalent in time to done work (notion of velocity), it's (in my view) very optimistic. People learn the nature of the beast as time progresses. Collective learning happens slowly. A lot of important work is usually only discovered and planned in the second half of the actual (not original) timeline. A lot of important stakeholders are naturally only introduced into projects close to its finalization, which creates a flurry of new activity and discoveries. This makes projects late.
But where that real halftime is - most people I worked with rarely know. I was no better, I also rarely knew.
The only way to manage this I found was the basic rule: build, demo, decide what to do next. Don't get attached to original backlog and grand plans for the future. It won't work that way. Just build and make it work, launch ASAP, get decisions on a ~weekly basis done. Ignore long term planning. It's not helpful.
Sadly, few business folks want that of course :)
For one, it doesn't assume pace is consistent. Tracking velocity is part of the game too and it can vary with capacity, talent and maturity.
Secondly, telling business that we can't estimate accurately is more useful than saying we can estimate accurately and being wrong (which is guaranteed). It's a tough sell but it's necessary.
Secondly, telling business that we can't estimate accurately is more useful than saying we can estimate accurately and being wrong (which is guaranteed). It's a tough sell but it's necessary.
Agree on the second point. I tried to convince some managers and sales guys to stop giving estimates at all and build trust otherwise, didn't work :)
As for consistent, my point was if you have 100 SP in backlog, delivered 50, then I contend you are not in the mid-point of the project. This is a dangerous assumption. You don't know where you are in my experience.
As for consistent, my point was if you have 100 SP in backlog, delivered 50, then I contend you are not in the mid-point of the project. This is a dangerous assumption. You don't know where you are in my experience.
It is if you're following agile principles. By the time you've completed a few stories you should be delivering every feature at production quality with all the automation pipeline and deployments ready. The question from that point is always did we build enough features for a launch.
I agree that building and demoing something small is key. The larger and more complex a project will be, the harder it is to estimate.
I’ve been trying to distill this into a framework that is accessible to both development and business folks [0]. I've used it a few times and it has resulted in a few projects getting cancelled due to a massive difference in initially expected cost and post-analysis expected cost.
0 - It's basically just Scrum oriented towards providing upfront estimates. https://jonpurdy.com/pm-framework/how-to-quickly-plan-and-es...
I’ve been trying to distill this into a framework that is accessible to both development and business folks [0]. I've used it a few times and it has resulted in a few projects getting cancelled due to a massive difference in initially expected cost and post-analysis expected cost.
0 - It's basically just Scrum oriented towards providing upfront estimates. https://jonpurdy.com/pm-framework/how-to-quickly-plan-and-es...
I'm gonna play the devil's advocate here.
> Sadly, few business folks want that of course :)
Business folks want this usually because it's their money you're playing with.
> Ignore long term planning. It's not helpful.
So, no don't tell people to ignore long term planning otherwise people who give you money are going to stop giving you money to "build, demo and decide". Unless you're on your own dime (e.g. you pay yourself or are self funded), then I wouldn't take this advice.
> Sadly, few business folks want that of course :)
Business folks want this usually because it's their money you're playing with.
> Ignore long term planning. It's not helpful.
So, no don't tell people to ignore long term planning otherwise people who give you money are going to stop giving you money to "build, demo and decide". Unless you're on your own dime (e.g. you pay yourself or are self funded), then I wouldn't take this advice.
> Scrum dogma is that estimates are for complexity, not effort or timing
The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates.
The “dogma” you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.
The theory of story points (which originate outside of Scrum and are not part of Scrum proper) is that task-specific time estimates in creative intellectual work are extraordinarily unreliable and expending more effort on them doesn't improve them, but broad-banded complexity class evaluation mixed with empirical observation of the teams velocity produces time estimates that are (while still extremely fuzzy) both better and much lower effort, once you have the basic tracking in place, than task-specific direct estimates.
The “dogma” you report seems like something that might be a derivative of that that has lost track of rationale and purpose, reducing it to a cargo cult practice.
Right, the estimates are not even in the guide: https://www.scrumguides.org/scrum-guide.html#sprint-planning
Planning poker and such are useful as they encourage the team to discuss differences and help identifying stories which are too large.
Planning poker and such are useful as they encourage the team to discuss differences and help identifying stories which are too large.
My understanding is that for a team and project that's been going for a while, scrum points can be roughly turned into time estimates by looking at the history of stories that were rated for that many points and averaging.
But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head.
And IME, it's a lot less stressful on the programmer to estimate points rather than time.
But that obviously won't work from the start, and it won't be accurate... Just better than nothing and (hopefully) better than what a programmer will estimate in their head.
And IME, it's a lot less stressful on the programmer to estimate points rather than time.
Scrum has destroyed my organisation. It’s like working with zombies.
How to turn your organisation into a cargo cult 101.
How to turn your organisation into a cargo cult 101.
Not only do we not know how to predict how long a software project will take, we don't even know how to predict what the end product will look like.
So who are we kidding?
Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team.
About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.
So who are we kidding?
Another way to look at it: take a small one-person project and assign it to three different developers. You may get wildly different results. How could you have predicted those differences in advance? Let alone apply that type of prediction across a large team.
About a dozen years ago I gave a presentation to the Silicon Valley Software Process Improvement Network (does it still exist?) My presentation: "Unsolved Problems of Software Maintenance." You think predicting greenfield development is difficult? Try predicting maintenance work, where figuring out what to do can be more than half the work.
Excellent point.
Can you share your presentation or at least some of your thinking behind it?
Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).
Can you share your presentation or at least some of your thinking behind it?
Based on what you know, how do you frame this problem? Imagine you had an impressionable audience of 10,000 software professionals (C-level people, managers, developers, UX people, customer support, and so on).
Sometimes a request for an “estimate” is really a request for a promise, quotation, a guarantee that something will be delivered by X time or cost.
It's easy to detect this:
Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%?
If you get emotional pushback to discussing uncertainty, they are looking for a promise.
It's easy to detect this:
Gently begin a discussion of how much uncertainty is tolerable, do they want to know the number we are 50% likely to hit? 80%?
If you get emotional pushback to discussing uncertainty, they are looking for a promise.
Yes, many of us are too likely to interpret the word "estimate" at face value. It is a wonderful idea to view this instead as only a starting point -- an information-gathering conversation -- as how to provide your customer or other stake-holders what they need.
If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)
If you are some combination of lucky, influential, and persuasive, you might have some ability to shape the contours of their expectations. :)
I had access to information about historical projects, so I compared the actual amount of time taken to the estimated time at the beginning, for every software project in the history of my organization.
I found that on average, things take twice as long as expected.
So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it.
A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot.
But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.
I found that on average, things take twice as long as expected.
So, I was like, now I know how to estimate any project. I figure out what seems reasonable based on known factors...and double it.
A way to look at this is the "unknown unknowns" of anything empirically average to a 100% overshoot.
But this doesn't fly with the project managers I work with, because they can only see that as illegitimately padding an estimate.
Hofstatder's law: It always takes longer than you expect, even after accounting for Hofstatder's law.
My problem with this is all estimation is hard. Period. Quantitative discussions of something that has been done? Fairly easy, if still not accurate.
Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done.
Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?
Discussing something that hasn't been rehearsed before? You can really only discuss how long you are willing to work on it. Not how long it will take to get done.
Fun examples. How long would it take you to clean out your fridge? How long would it take you to learn to play a song on piano?
I like this analogy. But I was picturing myself asking my sales manager "How long would it take you to learn to play a song on piano?", and I'm pretty sure he would reply "but you touched a piano before, you are supposed to be a professional piano player ! A professional piano player surely knows how long it would take him to learn how to play a song". So I guess he would miss the point totally :/
Ironically missing that some folks will never be able to learn some songs.
And, learning to play an existing song doesn't necessarily translate to being able to write one.
And, learning to play an existing song doesn't necessarily translate to being able to write one.
Estimation is hard if you are stuck in the mindset that you have to make a point estimate.
It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that.
It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know.
It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.
It's not hard at all if you are willing to be (and are allowed to be) honest about the uncertainty of the inputs and calculate the uncertainty of the final result based on that.
It's true that people may demand precision that you can't give them. But at the same time, you know something and it is simple to compute what you know.
It's like Fermi estimation, that everyone hates so much and claims is so useless to interview for.
Even distributions can fail in execution. I can't remember the name, but the joke rule that work expands to fit the time given to it. Such that, if all work expands to the max of the distribution, your time is easily set at the point of the max.
Of course, this has been tried and doesn't work. The max time on many distributions is effectively infinity. Which, fails. For obvious reasons. And is why I assert you can really only talk of how long you are willing to work on something. Not how long it will take.
Of course, this has been tried and doesn't work. The max time on many distributions is effectively infinity. Which, fails. For obvious reasons. And is why I assert you can really only talk of how long you are willing to work on something. Not how long it will take.
This!
If you estimate, estimate a distribution.
If you estimate, estimate a distribution.
I wish as much attention was paid to perform post mortems regularly then to only do estimation. You know, actually look at "hey, this is what we guessed and this is what actually happened".
I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?"
So clearly, there's value, at least when we've done them.
I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".
I've had to fight to actually hold post mortems, and every time I've done this, the manager ends up asking, "hey, can I share this?"
So clearly, there's value, at least when we've done them.
I'm amazed at how few places even perform a complete feedback loop. It's just, "when can you get this done?" and, "is it done yet?".
To tie this a bit more into the actual article: It might be even more accurate to just ask people in a post mortem for some feedback, instead of trying to build some data set based on estimates and SLOC. Like an exit poll.
I find this amusing.
Know how much a Honda Accord costs?
About 25 Grand.
Know how much a Mercedes S450 costs?
About three times as much.
They are both great cars, that will be rewarding to own.
The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface.
The reason is that all those little things that go into high quality take lots of time.
I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot.
However, if you want that app to be ship quality, and highly usable, you're looking at over a month.
The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.
Know how much a Honda Accord costs?
About 25 Grand.
Know how much a Mercedes S450 costs?
About three times as much.
They are both great cars, that will be rewarding to own.
The Mercedes doesn't have 3 times more parts, but it probably took four times longer to make, and they paid the folks that make it, a lot more than the Honda. It's actually, probably better "bang for the buck," although it won't seem like it, on the surface.
The reason is that all those little things that go into high quality take lots of time.
I can write a pretty complete app that does some cool stuff in a day or two. I do it often, when I write test harnesses for my libraries and whatnot.
However, if you want that app to be ship quality, and highly usable, you're looking at over a month.
The thing is, many folks would consider my test harness lash-ups to be their "shipping" product, and will use that as a basis for estimation.
Your base assumption seems to include that quality is valuable for its own sake. I don't totally disagree, but I'm wary of assigning value based on effort rather than output.
Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.
Depending on why you are buying a car, the Accord is very likely much better bang for the buck than the S-class Mercedes. And depending on the situation, the prototype is often better value than the shippable product.
You are exactly correct.
That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale.
I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger."
Our prototypes were incredibly expensive. A $2,000 (retail) body prototype would be insured at half a million bucks.
I just find, in my experience, managers expect Mercedes-level quality, for Kia (not even Honda) prices, and (if you are lucky) mature Honda assembly line development speed.
That can lead to real Jurassic-scale disasters. If you want that level of quality, you need to plan for it.
I'm working on the app that I'm doing, because I was helping a friend of mine evaluate contractors for his dream.
The promises they made were absurd. After a couple of these, I just said "Screw it. I'll do it for you."
He'll be getting Mercedes quality, but he'll need to wait a bit longer for it. I am pretty good at doing damn good; damn fast.
But he also won't be paying a dime for it, so I think he's good with that.
That is why Honda is a bigger company than Mercedes. They do decent quality (Hondas cost more than Kias), but at scale.
I worked for a "Mercedes-level" photographic equipment company for years. People were often quite surprised, when I told them the size of the company. It was like Roadhouse "I thought you'd be bigger."
Our prototypes were incredibly expensive. A $2,000 (retail) body prototype would be insured at half a million bucks.
I just find, in my experience, managers expect Mercedes-level quality, for Kia (not even Honda) prices, and (if you are lucky) mature Honda assembly line development speed.
That can lead to real Jurassic-scale disasters. If you want that level of quality, you need to plan for it.
I'm working on the app that I'm doing, because I was helping a friend of mine evaluate contractors for his dream.
The promises they made were absurd. After a couple of these, I just said "Screw it. I'll do it for you."
He'll be getting Mercedes quality, but he'll need to wait a bit longer for it. I am pretty good at doing damn good; damn fast.
But he also won't be paying a dime for it, so I think he's good with that.
>That is why Honda is a bigger company than Mercedes
I was curious if this was accurate. Here are some recent numbers for revenue and unit sales:
Honda -> $144B revenue
- 12.4M motorcycles, ATVs, etc.
- 3.3M automobiles
- 5.7M generators, small engines, etc.
Daimler -> $207B revenue
- 2.4M automobiles
- 0.5M trucks
- 0.4M vans
- 0.03M buses
So, Daimler seems to be about 40% larger in revenue, but Honda sells almost 40% more automobiles. But 2.4M automobiles for Daimler seems like it's still comparable in scale.
Honda seems to still be largely a motorcycle and small engine company, in the global picture, while Daimler has a substantial business in heavy duty commercial vehicles including nameplates that aren't necessarily associated with Mercedes by the average person.
I was curious if this was accurate. Here are some recent numbers for revenue and unit sales:
Honda -> $144B revenue
- 12.4M motorcycles, ATVs, etc.
- 3.3M automobiles
- 5.7M generators, small engines, etc.
Daimler -> $207B revenue
- 2.4M automobiles
- 0.5M trucks
- 0.4M vans
- 0.03M buses
So, Daimler seems to be about 40% larger in revenue, but Honda sells almost 40% more automobiles. But 2.4M automobiles for Daimler seems like it's still comparable in scale.
Honda seems to still be largely a motorcycle and small engine company, in the global picture, while Daimler has a substantial business in heavy duty commercial vehicles including nameplates that aren't necessarily associated with Mercedes by the average person.
Good point. I was thinking cars, but Europe seems to have every cab as a Mercedes.
That's a lot of cars.
That's a lot of cars.
High quality != high feature count. You are conflating the two with your anology.
You don't want to daily drive a 30 year old performance oriented Mercedes.
An Accord however, most generations can hit 300k miles easily.
The Mercedes is more expensive because they put more investement into refinement and luxury. Make the doors feel right. Ensure the torque curve is flat and starts at a low RPM, usually using turbo chargers that will fail by 200k miles. Lots of carefully placed sound deadening.
This isn't higher quality, it's extra features.
You don't want to daily drive a 30 year old performance oriented Mercedes.
An Accord however, most generations can hit 300k miles easily.
The Mercedes is more expensive because they put more investement into refinement and luxury. Make the doors feel right. Ensure the torque curve is flat and starts at a low RPM, usually using turbo chargers that will fail by 200k miles. Lots of carefully placed sound deadening.
This isn't higher quality, it's extra features.
> but it probably took four times longer to make,
I doubt it. Luxury goods don't proportionately more cost much more to produce than commodity goods.
Instead they often derive their value from perceptions and exclusivity. That is people perceive the object to be more value for their own reasons, or they are made more expensive as a proof point of exclusivity / conspicuous consumption
https://en.wikipedia.org/wiki/Conspicuous_consumption
I doubt it. Luxury goods don't proportionately more cost much more to produce than commodity goods.
Instead they often derive their value from perceptions and exclusivity. That is people perceive the object to be more value for their own reasons, or they are made more expensive as a proof point of exclusivity / conspicuous consumption
https://en.wikipedia.org/wiki/Conspicuous_consumption
Mercades cars do often have more features, but I also am skeptical of claims that they are of higher quality.
In my experience Mazdas have the best tech/features at various price levels, compared to the competition's offering of the same feature
We should not conflate quality with value. [1]
[1] https://moznainaczej.com.pl/Download/Teal%20Doctrine/A.Blikl... (13.10 Quality and value)
[1] https://moznainaczej.com.pl/Download/Teal%20Doctrine/A.Blikl... (13.10 Quality and value)
Welcome to Hacker News!
From your link:
> The more the product promises the customer, the more luxurious it is; the greater the extent to which these promises are fulfilled, the better the product’s quality.
While this is an interesting way of thinking about it, I find personally it to be both non-standard and not useful.
I'm generally not inclined to force or coin a definition of a concept that is intrinsically messy. Instead, I think it is better to take almost an anthropological view: unpack what different people and cultures mean by a word.
With this in mind, I recommend the following article from MIT's business school, because it embraces the vagueness and complexity of quality:
https://sloanreview.mit.edu/article/what-does-product-qualit...
From your link:
> The more the product promises the customer, the more luxurious it is; the greater the extent to which these promises are fulfilled, the better the product’s quality.
While this is an interesting way of thinking about it, I find personally it to be both non-standard and not useful.
I'm generally not inclined to force or coin a definition of a concept that is intrinsically messy. Instead, I think it is better to take almost an anthropological view: unpack what different people and cultures mean by a word.
With this in mind, I recommend the following article from MIT's business school, because it embraces the vagueness and complexity of quality:
https://sloanreview.mit.edu/article/what-does-product-qualit...
> It's actually, probably better "bang for the buck," although it won't seem like it, on the surface.
A car's primary function is to move its passengers from point A to point B, safely and timely. Both cars do this very well, but the Mercedes will probably be costlier over their respective lifetime (fuel, parts, service).
The only way for it to be a better bang for buck would be to turn its secondary functions (comfort, prestige, signalling) into a primary tool. For instance, if a broker or salesperson finds that owning the car gives them the added confidence to project the image they want during negotiations and thus contributes to their overall success, then it becomes a worthy investment.
A car's primary function is to move its passengers from point A to point B, safely and timely. Both cars do this very well, but the Mercedes will probably be costlier over their respective lifetime (fuel, parts, service).
The only way for it to be a better bang for buck would be to turn its secondary functions (comfort, prestige, signalling) into a primary tool. For instance, if a broker or salesperson finds that owning the car gives them the added confidence to project the image they want during negotiations and thus contributes to their overall success, then it becomes a worthy investment.
My experience is that most customers are unwilling to wait for and pay for quality, especially if someone in the market sells a competing product (which itself is of low quality)
Don't do an estimate, build a prototype.
Don't do an estimate, build a prototype.
Don't do an estimate... seriously... build a prototype.
And then throw it out.
And THEN do an estimate. It will probably be pretty accurate.
Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.
Don't do an estimate, build a prototype.
Don't do an estimate... seriously... build a prototype.
And then throw it out.
And THEN do an estimate. It will probably be pretty accurate.
Not always easy to do this in a tech company but as an eng director I was able to get it done and it changed a seriously broken process based on multi-week scoping and estimation futility.
I had a conversation about estimates during a recent interview. I asked about how the company deals with those, and the interviewer said they don't do estimates because there's never been a time where something productive came out of one, and I think it makes sense.
In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.
In my experience, when an estimate is spot on the world goes on as if nothing happened. When it's incorrect, all hell breaks loose and it's every man for himself. And at the end of the day, all of the blame ends up on the person who guessed wrong. I'm glad I don't have to deal with that anymore.
When a company uses software estimation, it suggests strong distrust towards the software people and is looking for justification of those huge costs.
Most often it means some shitshow happened or is still going on in there.
Reminds me of the excellent 2010 talk, “What We Actually Know About Software Development, and Why We Believe It’s True”:
https://vimeo.com/9270320
Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).
https://vimeo.com/9270320
Bit long, but well-presented and worth a listen for any practicing software developer (or person who manages developers).
I seriously don't understand the obsession with estimates in the paradigm it is pushed in. Want to be steady at a reliable pace, work for a few months and figure out the average and deviations. Need a priority on tasks, use a priority queue using whatever method you assign it by, often a combination of deadline, value added, complexity and more.
Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models?
Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.
Yes, I get on a high level, one needs to be able to say "yes we can make this before [date]", when using big deadlines. How often do people really have to finish something before a critical deadline or decide to drop it right then and there? I'd wager most software devs do not, let alone biweekly deadlines. Isn't agile methodology supposed to help us fight against artificially tight deadlines (customer collaboration)? Isn't the SaaS model combined with "deploy any time" designed to be profitable in accordance to features implemented? Then, why push estimates so strongly in whatever flavor of the month Scrum version BigCorp wishes to use today? What do they even add at that point? If they are really so important, why do we always feel the need to introduce human error when we can extrapolate former experiences with computer models?
Maybe it is just me being cynical. The entire need to hold so fiercely onto an estimate reeks of micromanagement and desire to push responsibility entire onto the lower ranks.
That is absolutely what it is. For example, people are comparing favorably to how sales reps are now forced to record everything they do in something like Salesforce to try to prove they are productive, which by all accounts is pretty awful for the actual people doing the selling and doesn't actually improve sales--it just makes management feel like they're doing something.
One important thing I've managed to get a couple managers to track for quarter/half planning purposes: uncertainty. Prior to that, we'd of course hemmed and hawed and verbalized the fuzziness of our estimates... but only the final decided-on not-super-pessimistic number was written down, and decisions were made based on those. That was a major source of our estimating problems.
Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are.
It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.
Everyone wants estimates. For decent reasons. But until we've done a similar thing before, they're utter fabrication. We can take a semi-educated guess, but we know they're still guesses... so for things we don't have concrete answers for, we give small/medium/large/extreme markers for how uncertain we are.
It's fine to take on a big unknown or two. You might even get them both done in a half. But they better be worth it (or you have to decide when to cut your losses), because completing those two could consume all of your resources... and if that happens and you didn't commit everyone to it up-front, you won't get it done this half. Making that tradeoff more explicit managed to get us signed up for fewer low-impact-but-highly-uncertain projects that would inevitably balloon out of control but never be cut.
I've been in this business long enough to know that point estimates are always wrong. A proper estimate is a range with a confidence interval. When forced to do a fixed bid, you have to raise the price even higher to the upper end of the cone of uncertainty.
This agrees with my experience as well. Stop giving point estimates.
I've been using the Rumsfeld Matrix to classify software estimations into 4 categories:
Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments).
Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code).
Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity).
Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.
Known knowns: familiar tasks that can be reliably estimated from past experience. You can improve these estimates by better knowledge sharing (internal wiki, adding comments).
Unknown knowns: tasks that can be increasingly better estimated the more time you spend on estimating (for example creating Draft PRs with pseudo code).
Known unknowns: tasks that haven’t been done before, so estimations need a decent buffer (30-50%) to account for research and potential blockers. You can improve these estimates by benchmarking your previous efforts combined with the team's skill levels (aka sprint story point velocity).
Unknown unknowns: the unforeseen blockers that come out of nowhere, can't really be estimated, and can really disrupt a project's schedule. Improving these estimates is really hard. The key to improving these is by improving team/company communication and building agile feedback loops so you can identify these issues early and reprioritize as needed.
It's worth watching Software Schedules by Joel Spolsky has some of the best insight regarding evidence based scheduling. It's extremely interesting - https://www.youtube.com/watch?v=EUS4ktQJOSY
At least in a business setting I think the whole concept of a project needs serious reconsideration. We end up more often than not trying to fit developing a digital product into an enormously stupid gantt chart to execute some poorly thought “business requirements”. I prefer to talk products and not projects, I deliver the full thing including “growth” as adoption doesn’t come “if you build it” even within a Company setting. If you are building a product you can also get closer to those with the real problem willing to fund you with real budgets. On top of everything else if you are making users happy they will not chase you on fake estimates but rather work with you to get stuff done.
Funny when you look at the actual definition of a project straight from PMP.
"a temporary endeavor undertaken to create a unique project service or result."
No "projects" in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption.
We in software think of projects really as things which we create and which need maintenance in order to live. There is never some "end" to it.
Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.
"a temporary endeavor undertaken to create a unique project service or result."
No "projects" in software fhat I know of are actually temporary. They only end when management fires the people behind them, it gets cut off or there is no adoption.
We in software think of projects really as things which we create and which need maintenance in order to live. There is never some "end" to it.
Because it doesn't even conform to its own definitions, we could therefore conclude that the whole PMP project management discipline, as applied to software, is a scam.
I feel like I can get pretty good estimates on the following conditions:
- the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final
Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.
- the application is thoroughly specced. You might need WEEKS for this. - all variables are taken care of. Stack is known, and you’ve experience with all parts involved. If you don’t, get familiar with the parts first. Again, might take weeks. - there is no implicit functionality. It is either explicit or not included. - there are clear boundaries and rules to prevent feature creep. - you cannot estimate an estimate - all designs and UX are final
Now the problem is, this estimate is really expensive, because it’s actual work. It takes about 10-25% of the total project time to estimate the project.
> the application is thoroughly specced. You might need WEEKS for this
More weeks than it will end up taking to build the finished product, in fact.
More weeks than it will end up taking to build the finished product, in fact.
The first key mistake of software estimation in industry is that we rarely talk about confidence (e.g “I’m 50% confident it’ll be shipped in 5 hours, but 99% confident it’ll ship before end of week”). The fact that this is not part of planning means it’s often not part of the conversation and that we seldom think about estimation as a tuple of value and confidence.
The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.
The second key mistake is forgetting that estimates (at least in software) are an upper bound. So high estimates are often “better”.
In his book “Code Complete”, Steven McConnel speaks about metaphors. He reasons that metaphors are necessary to be a good developer as it helps visualize the act of coding. The metaphor he prefers is the “construction” metaphor. This metaphor he argues best explain the act (some would say art) of programming and gives developers a language to speak in that brings clarity to the development process.
When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc)
The problem with that metaphor IMO is that it's not actually what software development is.
The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building.
And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does.
I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.
When in construction you prepare the building site, lay a foundation, frame the house, put siding and a roof on it, and plumb and wire it. This is equivalent to programming. In other words through the lens of the construction metaphor, the developer is someone who ultimately build someones house by working together with different disciplines (Architects, designers, contractors etc)
The problem with that metaphor IMO is that it's not actually what software development is.
The proper metaphor for software development is more "engineering and development of construction site equipment and material" i.e. a developer is not building buildings they are building the things necessary for building the building.
And so the developer will often find themselves in a situation where the material doesn't exist and they have to invent it it or the material exist but we dont know how it's going to work with some other material needed or the equipment used for that material isn't made yet or doesn't work with that material even though it normally does.
I.e. a developer is inventing, engineering and building all at the same time and that is what makes it impossible to estimate development regardless of process or metaphor.
The reality is that software development is nothing like engineering or construction, it's totally different. You don't build a quick house, let people live in it and start building the walls whilst they live there.
Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.
Humans like to think via metaphor because it's a least-effort mode of thought but sometimes there just isn't one and it's just tough luck and start thinking from first principles instead.
My point was more that the software developer have to not just build with existing material and equipment but have to invent and build those on the fly.
"building with material" implies that material cannot be reused, but code can be copied for effectively zero cost, so it isn't like building, so the comparison to inventing and build equipment on the fly doesn't work.
I think we humans find it hard to accept when something is completely new, when there's no analogy for it, because that's how our brains like to think. But software is just _different_ and it's better to give up on analogy than be misled by it.
I think we humans find it hard to accept when something is completely new, when there's no analogy for it, because that's how our brains like to think. But software is just _different_ and it's better to give up on analogy than be misled by it.
Integrations for time-tracking software with issue tracking software exists, but I've never actually seen them used in combination before. You'd think that issue trackers would always include a very user-friendly time tracker by default, and advertise integrations with popular time trackers just to get that "actual time spent" data. I've also never looked at a burndown chart with any particular interest after knowing you can close an 8h issue after .5h and a .5h issue after 8h of work and the issue tracker will just keep pretending its charts mean anything.
Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.
Yet any company that does a lot of the same work could probably use that. You'd still need to do some extra work and tag your issues by things that you want to observe (and maybe predict), but then you could say things like "after we switched to that fancy new API client generator, our client development times went way down but debugging times went up a bit". And the system could display some time range while you're writing the issue. You could also look at the issues at the end of a sprint and then merge them with other items from the time tracker to see how meetings and other distractions played into the outcome.
The site is down for me, so :
https://web.archive.org/web/20210118200327/http://shape-of-c...
[deleted]
To me the difficulty of estimating software projects isn't about estimating the time / effort to build the thing. The biggest uncertainty is trying to estimate what it is we're trying to build in the first place. i.e. the details of the spec. Often times where I've seen estimates really off is when the scope changes, or more details are discovered etc.
To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.
To borrow the usual building a house analogy, if during software estimation we're actually given how many rooms, how many floors, where are the doors and windows etc, then I think estimates will be a hell of a lot more accurate. Instead, what we typically get is, build a house that can be used by the family of 4, and we'll discover exactly what the family wants as we build.
It is possible to deliver most software on time, but you need to be very clear about scope and unknowns and keep the units of work small.
The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools).
If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation.
When new ideas arise, they should usually be kept for future work.
When work is added, other work needs to be removed or the estimate revised.
If the work runs late, something should be removed from the scope to keep it on time, then done after delivery.
If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.
The broad strokes of the design work need to be done before the estimate (i.e. gather requirements, identify constraints, decide on direction and tools).
If work is unknown or in large chunks, it needs to be broken up into smaller well-understood chunks before estimation.
When new ideas arise, they should usually be kept for future work.
When work is added, other work needs to be removed or the estimate revised.
If the work runs late, something should be removed from the scope to keep it on time, then done after delivery.
If work involves integration with other systems this is much harder unfortunately, but for some classes of software (much business software), it is not so hard to estimate.
>Those estimation datasets that were flogged to death in the 1990s using non-machine learning techniques, e.g., regression.
>...non-machine learning techniques, e.g., regression.
Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.
>...non-machine learning techniques, e.g., regression.
Is this where we are now? The connotation of "machine learning" doesn't include regression? Wow.
They (almost certainly) mean linear regression, which many people appear to regard as statistics and boring, as opposed to single layer neural networks, which are cool.
To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).
To be fair, this is a pretty common perspective in computer science academia and adjacent fields (and probably occurs in a lot of fields where people don't focus on statistical learning..ahem I meant ML).
How do other disciplines estimate NRE? Do they have the same problems with missed predictions?
Sure. A lawyer may have a pretty good idea of how a particular task or case is likely to play out but there's a lot that isn't under their control. I imagine something like drafting a will is relatively straightforward but I also imagine a criminal case has a huge number of variables.
Well, lawyers take a retainer. Doctors are paid per-diagnostic-visit and per-procedure. Accountants bill hourly. Most professions have this - electricians, plumbers, locksmiths, etc.
This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple "feature factories" (see also: Developer Hegemony[0]).
[0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-...
This business model is normalized for other professions, and it should be for software engineering too. As a profession, we should move more towards partnering with organizations to realize business value through software rather than being simple "feature factories" (see also: Developer Hegemony[0]).
[0] https://daedtech.com/developer-hegemony-the-crazy-idea-that-...
You only quote the fixed bits and charge by the hour for the rest. If you need to give a quote for an unknown you multiply the quote by N to cover a worst case.
My issue with effort estimates is that it is taboo to report estimates with uncertainty (think how silly that sounds). I'll often give a low-end / high-end and the person asking for the estimate will tell me they want a single number. If you did this in physics you would get laughed and for good reason. It's bad science, and it's bad forecasting.
You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?
You should never claim better precision than you know. It's often more accurate to say "more than an hour but less than a week" than to say "6 hours". We learn scientific precision in middle school... why is modern workflow management still so bad at uncertainty propagation?
I need to get a check list for the ever so popular criticisms of academia article. This one surely would hit plenty of boxes: scientists don’t pay attention to industry, researchers only care about grants, etc. It’s a lot of words just to say “they need bigger datasets”, which is also a check box.
The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.
The article ends by suggesting using GitHub. When I was a PhD student I recall open source projects being a standard data source for people who did software engineering research. But it’s not my field, so I’m not sure if effort estimation really has missed this.
I think a better title would have been "most effort estimation literature relies on small datasets". The use of "fake" can be read as insinuating deliberate malpractice.
I bet it is possible to give precise estimates, the problem is the amount of time and effort to produce such estimation compares with the actual execution.
I think the problem is that if you have a timeline you can produce the result within time and budget, but at the expense of quality.
The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).
The time and cost estimates do not measure the amount of technical debt in the outcome. it may good great on paper but that is because of a vaguely defined specification which says nothing about the quality, including amount of te4chnical debt (meaning mostly maintainability).
In my previous company, we needed to put project code into timesheet for every activity. Of course, requirement gathering is also an activity. However, before you get the budget, you don't have the project code. Also, you can't get the budget before you have the estimate. It means I needed to give an estimate before I know what system I am going to build.
How about getting a budget to do an estimation/PoC/prototype for a project?
I noticed that my estimates usually end up 3x as high as the next one and i feel pretty confident about them
Hasn't anyone tried to replace time estimates with probabilities?
(Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)
(Like "There's a 80% chance we'll finish this today." ) You immediately start to understand why it can take so much longer than expected (You've rolled a dice before; I want a six... How long will it take)
I don't know if I am supposed to roll a 20 on a D20 or a 8 on a D8 to begin with.
Time estimation on a macro or micro level is just educated feelings.
The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice.
I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.
Time estimation on a macro or micro level is just educated feelings.
The thing I notice with people that are forced to estimate time and have nagging managers is that they start to report time according to how much the task was estimated to, to make the burndown chart nice.
I.e. if one task is finished early a late running one gets the hours giving the illusion of getting better at estimating. I think Joel's estimation method for Frogbugz had the same fate linked in somewhere above.
That's how I always felt about all the estimating parts in CS courses. They were all made up BS that would only work on very pre-defined things, like creating a new DB or website based on a template. aka "IT" and not "SE" stuff.
My bulletproof formula for doing estimations in corporate environments:
InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order
Example: 1 day (MD) = 2 weeks 2 weeks = 4 months
InternalEstimation = estimate the work as truly as possible (ie. in MD) ExternalEstimation = double the value of InternalEstimation, and increase units by one order
Example: 1 day (MD) = 2 weeks 2 weeks = 4 months
I think there is a big difference between building something totally new versus a modification to an existing system. If you are building something totally new, I do not think there is much value in estimates with granularity lower than a month.
I think, software development works best without deadlines, but then you have to find ways to finance yourself while people wait.
Unfortunately, the business folks insist on making up arbitrary dates and costs. It’s the way of the world.
Estimation seems like a paperwork generation exercise anyway, so why not take it all the way to research?
Would software effort estimation work better if software developers were held to fixed-price bids?
Jonathan Stark saved my soul from this estimates nightmare. You should check him out!
Exactly, you can't really predict what can't be predicted
Redirect loop on TFA?
Update: working now.
Update: working now.
Broken Link....
So research into software estimation also seems to follow Hofstadter's Law: “It always takes longer than you expect, even when you take into account Hofstadter's Law.”
[deleted]
It is provably impossible to predict development effort.
The difference is when you're asked for a quote, you're asked how much you will be charging, with the expectations that you'll be willing to eat into your own margins to give a lower quote. That's why it's a negotiation, where you negotiate how much extra effort, time and headcount you're willing to give, how much tech dept you're willing to take, etc., for the privilege of getting their business.
If you see it for what it really is, you'll see that it works pretty well actually. The business gets more out of you for less to them. It was never about having an accurate timeline or helping with planning or prioritizing, and always about negotiating a better contract with the dev team.
Now keep in mind that the "business" in this case is a person who need to report that through their amazing prowess of administration and management, they personally managed to get X feature out during their last review cycle at Y cost with impact Z. This person will not need to deal with developer satisfaction, retention and performance. They will not need to deal with the impact the lower margins they pushed for had on the next feature delivery, or the continued maintainance of the systems. And if the dev team had to lower the quality too much in order to meet the quote they put out, that will be 100% their fault, the "business" will know not to use them for their next contract, or they'll expect the dev team to take on fixing all the issues at their own expense once more.