Lessons learned from the big rewrite(anton-pirker.at)
anton-pirker.at
Lessons learned from the big rewrite
http://www.anton-pirker.at/the-big-rewrite-war-story/
39 comments
Doing it incrementally is definitely the best. However, in your scenario, the problem of a 'weird' codebase people have a hard time using is not addressed by requiring further development in it. Database layout is usually not the problem if the system was at all functional.
Having done this a few times, when switching from old to new, the new framework would have a DB agnostic ORM. What works nicely is building the models, duplicate templating, getting a mirror working. Shut down the old site, and then it's possible to go on with normal development.
So, it is important to consider rewrite and redesign as separate goals.
Having done this a few times, when switching from old to new, the new framework would have a DB agnostic ORM. What works nicely is building the models, duplicate templating, getting a mirror working. Shut down the old site, and then it's possible to go on with normal development.
So, it is important to consider rewrite and redesign as separate goals.
The thing is though - if no one understands the existing system, how could you ever hope to build a new one that provides the same functionality? You'll have to interrogate that system to find out how it functions in order to replicate it. I find that this is often easier done by incremental refactorings of the legacy codebase (And writing a test harness in the process). So the rewrite really becomes a mission of analysing/documentation.
However messy it may be, there's usually a lot of domain knowledge to be learned from such a legacy codebase.
However messy it may be, there's usually a lot of domain knowledge to be learned from such a legacy codebase.
In this case porting from perl to python makes it much harder and seems pointless. Why not just keep it in perl but incrementally move to a modern perl framework?
Why do we see so many more posts about Django compared to 'modern Perl framework'? Python has killer features like simple syntax and developer familiarity. Django has GeoDjango.
This isn't even a question to me, but to the blog author. My. Point is about order of steps in the rewrite. It is not harder to rewrite first, redesign second, it is easier. I've done it multiple times in c++/c#, perl/python, php/python, r/python.
This isn't even a question to me, but to the blog author. My. Point is about order of steps in the rewrite. It is not harder to rewrite first, redesign second, it is easier. I've done it multiple times in c++/c#, perl/python, php/python, r/python.
One consideration to switch from Perl to Python was that it is easier to find Python/Django developers because more young developers learn Python than Perl. Perl developers (at least in Austria) are mostly seniors and thus most of them already have a good job. (And do not want to change jobs) And a mature Perl developer is more expensive than a young Python developer.
Working incrementally definitely makes sense. Also if you don't see a chance to do it in an incremental way you should probably take a step back and ask yourself if the old application is maybe too huge and monolithic and check if there's a possibility to make it more modular. So you could split one huge application in smaller parts/services which can easier evolve seperately from the rest.
Very much agree. Slicing the existing application up in smaller parts and migrating those piecemeal is another form of incremental rewriting, as opposed to working through one layer at a time.
[deleted]
It's sometimes a judgement call, though. If an incremental approach is too expensive, haphazard, or just not technically sound, doing a soft launch and gradually transitioning users can work well too. This obviously has it's own drawbacks and complexities but it's a good alternative if an in-place upgrade isn't feasible. That said, at the end of the day an all-or-nothing release really should be the last resort.
> The first thing I will do the next time is taking more time to analyze the old system thoroughly and write a technical spec what the new system should do. A simple "What the old system is doing now" as a spec for the rewrite does not cut it. No one knows what the old system is doing exactly so it is necessary to take the time to write a spec.
I had the same experiences last year. There wasn't even a single person who knew the 'old system' on the whole. A receipt for many detours and much extra work.
I had the same experiences last year. There wasn't even a single person who knew the 'old system' on the whole. A receipt for many detours and much extra work.
It is not that simple and comes back to the same old maxim - as long as the only documentation of what the system is doing and why is its source code, you are doomed.
If you have a written detailed guidelines of what you are doing and why, which design decisions were made and why, which data representation is used and why and what corresponding algorithms are and why, then one could re-write parts of the system in different languages/frameworks just for fun or to explore and compare.
Coding or rewriting is second to deep, hierarchical understanding of hows and whys. This is how the most of programming exercises are designed - once you have understood how it should be, which results each procedure should return (wishful thinking and tests before code, to define domain and range of a procedure - ideas as old as SICP) then coding is the most trivial part.
Look at antirez - he is a rare example of a guy of deep understanding before code.
If you have a written detailed guidelines of what you are doing and why, which design decisions were made and why, which data representation is used and why and what corresponding algorithms are and why, then one could re-write parts of the system in different languages/frameworks just for fun or to explore and compare.
Coding or rewriting is second to deep, hierarchical understanding of hows and whys. This is how the most of programming exercises are designed - once you have understood how it should be, which results each procedure should return (wishful thinking and tests before code, to define domain and range of a procedure - ideas as old as SICP) then coding is the most trivial part.
Look at antirez - he is a rare example of a guy of deep understanding before code.
I've done the big rewrite a few times, and what I've learned from it is that a system so "bad" that it is in need of a rewrite does not have a caretaker who understands it well enough to provide perfect documentation of choices made and reasoning behind them. In fact, a system that bad typically has plenty of choices which were made for no good reason at all. So, you never go into the big rewrite with perfect documentation or good tests, because if you had those you wouldn't need a big rewrite.
Uncle Bob made this point very salient in one of his videos:
If there was some catastrophic, quirky failure of backup, and you could only save either (a) all your production code, sans test, or (b) only your tests, which would you rather save?
Assuming you've written thorough tests, they win out over code every time. The tests remove all ambiguity.
I'm reminded of a story that illustrates the point:
A young woman always cuts the ends off of her pot roasts before cooking them, following her mother's example. One day, she asks her mother why they do that. The mother doesn't recall, so she asks the grandmother. The grandmother looks at them both, shrugs, and says, "When I was your age, we lived in an apartment with a very small oven. We had to cut the pot roast down to make it fit!"
If there was some catastrophic, quirky failure of backup, and you could only save either (a) all your production code, sans test, or (b) only your tests, which would you rather save?
Assuming you've written thorough tests, they win out over code every time. The tests remove all ambiguity.
I'm reminded of a story that illustrates the point:
A young woman always cuts the ends off of her pot roasts before cooking them, following her mother's example. One day, she asks her mother why they do that. The mother doesn't recall, so she asks the grandmother. The grandmother looks at them both, shrugs, and says, "When I was your age, we lived in an apartment with a very small oven. We had to cut the pot roast down to make it fit!"
Oh come on! Reality check please :)
Tests alone provide little value to anyone. At least if you had the production code you could keep it running in its current state while you added back those important tests bit by bit.
Most of the tests wouldn't be needed as good code tends to be churn-free. Finally if you wrote the code TDD style, then writing tests for even the hairy bits would be "easy" compared to rewriting the whole system.
Tests alone provide little value to anyone. At least if you had the production code you could keep it running in its current state while you added back those important tests bit by bit.
Most of the tests wouldn't be needed as good code tends to be churn-free. Finally if you wrote the code TDD style, then writing tests for even the hairy bits would be "easy" compared to rewriting the whole system.
I'm going to have to respectfully disagree. :) (Or, with apologies to Adam Savage, "I reject your reality check and substitute my own!")
A huge portion of my day-to-day agony at my current job stems from a lack of tests. The developers on my team (all lifelong employees, where I'm a contract hire) are terrified of making more than the absolutely necessary changes to code. I can't replace literally copy-pasted code with a shared method without push-back. To even speak of refactoring is verboten, at least in front of our internal customers. Testing small changes can take days, as it must go through QA and the customer, and surprising bugs still slip through. As someone who went through Fowler's Refactoring as bedside reading, this process and turnover rates drives me batty.
Here's the thing: We have the production code up and running, and I'm already adding tests where I can, but we constantly run into implicit requirements. Even the senior developer, who's worked on this code base for years, keeps discovering things about it. We continue to discover genuine bugs, too. The production code empirically lacks enough information to reproduce the tests.
And "good code" begs the question. In fact, it begs two questions: (a) How do you know code is "good" if it isn't tested empirically, and (b) how can code ever be churn-free when the requirements themselves keep changing?
A huge portion of my day-to-day agony at my current job stems from a lack of tests. The developers on my team (all lifelong employees, where I'm a contract hire) are terrified of making more than the absolutely necessary changes to code. I can't replace literally copy-pasted code with a shared method without push-back. To even speak of refactoring is verboten, at least in front of our internal customers. Testing small changes can take days, as it must go through QA and the customer, and surprising bugs still slip through. As someone who went through Fowler's Refactoring as bedside reading, this process and turnover rates drives me batty.
Here's the thing: We have the production code up and running, and I'm already adding tests where I can, but we constantly run into implicit requirements. Even the senior developer, who's worked on this code base for years, keeps discovering things about it. We continue to discover genuine bugs, too. The production code empirically lacks enough information to reproduce the tests.
And "good code" begs the question. In fact, it begs two questions: (a) How do you know code is "good" if it isn't tested empirically, and (b) how can code ever be churn-free when the requirements themselves keep changing?
Heh, that agony description was eerily familiar, and I'd be the employee! The hard part is not being afraid to make an obviously right change, even if it means spending ages manually testing things and then continuing to follow up weeks later when the non-obvious things you didn't think of break :)
Even in the face of changing requirements I've found that there are substrates of code that do their job without (showing?) major bugs for years. They never needing fixing. That's your good code. If you had unit tests, you'd never run them anyway because you never touch that code. They are the "select()"s of this world, but specific to whatever you're writing.
Now if the whole system had to be rewritten, the old tests would be useless anyway. You mention requirements that have changed, so those tests you had would no longer apply. You'd have to write a new system and new tests ;-)
FWIW I've found that peer code review and static analysis tools give you a lot of what unit tests would anyway, without the overhead of having to write the tests and update the dead code when you rewrite the affected part of the system.
I mean... what's the point of code anyway? Is it to be "perfect"? Or is it to do a job, earn you money, and be more or less maintainable? Constantly refactoring to stay in the same place is worse than copy-pasting code, from a business owner's POV. Sometimes it is better to have 2 copy pasted methods that are deployed and the customers are happy, than trying to have 1 perfect method sat in development. If those 2 methods never need updating again, you wasted time refactoring. If you would need a third copied method later on, that's the time to refactor.
Maybe I'm being a bit devil's advocatey. Unit tests and refactoring are good tools, but they are not "free" (since you have to test the real system anyway at some point) and Uncle Bob is a bit of a snake oil salesman anyway :-P
Even in the face of changing requirements I've found that there are substrates of code that do their job without (showing?) major bugs for years. They never needing fixing. That's your good code. If you had unit tests, you'd never run them anyway because you never touch that code. They are the "select()"s of this world, but specific to whatever you're writing.
Now if the whole system had to be rewritten, the old tests would be useless anyway. You mention requirements that have changed, so those tests you had would no longer apply. You'd have to write a new system and new tests ;-)
FWIW I've found that peer code review and static analysis tools give you a lot of what unit tests would anyway, without the overhead of having to write the tests and update the dead code when you rewrite the affected part of the system.
I mean... what's the point of code anyway? Is it to be "perfect"? Or is it to do a job, earn you money, and be more or less maintainable? Constantly refactoring to stay in the same place is worse than copy-pasting code, from a business owner's POV. Sometimes it is better to have 2 copy pasted methods that are deployed and the customers are happy, than trying to have 1 perfect method sat in development. If those 2 methods never need updating again, you wasted time refactoring. If you would need a third copied method later on, that's the time to refactor.
Maybe I'm being a bit devil's advocatey. Unit tests and refactoring are good tools, but they are not "free" (since you have to test the real system anyway at some point) and Uncle Bob is a bit of a snake oil salesman anyway :-P
> Oh come on! Reality check please :)
I agree. Typical Uncle Bob BS
With the source code you have the product. You lose nothing, maybe some information on how it works in some corner cases (which is derivable from the source code anyway)
I agree. Typical Uncle Bob BS
With the source code you have the product. You lose nothing, maybe some information on how it works in some corner cases (which is derivable from the source code anyway)
'once you have understood how it should be, which results each procedure should return then coding is the most trivial part.'
I wholeheartedly agree with this! The big challenge is that you have the patience (and the patience of the management) to understand the old system thoroughly before starting to write new code.
I wholeheartedly agree with this! The big challenge is that you have the patience (and the patience of the management) to understand the old system thoroughly before starting to write new code.
Doomed is a bit harsh perhaps. Documentation, even if it exists, is never perfect. I don't think you can completely get away without some study of the old code base.
A good rule of thumb is that reading code takes at least as long time as writing code.
A good rule of thumb is that reading code takes at least as long time as writing code.
There are two observations. One, very general, is that the best writing is re-writing. It is mostly about scientific writings, when your understanding grew and clarified since the first writing.
The second one is "the second system effect" from Mythical Manmonth, which says that when people re-design a system (or a language) they tend to make a kitchen sink from it, throwing in everything that comes to their minds (hello, Rust?)
There is also a common sense notion that a sequel for the sake of a new version is usually much worse than the original.
What is common of them all? It seems that still the best writing is rewriting, but only if your understanding of the details has been grown and now you have less clutter in your mind. But if you do a rewrite without much thinking or just to get shit done, there is a high probability that you will fail.
The second one is "the second system effect" from Mythical Manmonth, which says that when people re-design a system (or a language) they tend to make a kitchen sink from it, throwing in everything that comes to their minds (hello, Rust?)
There is also a common sense notion that a sequel for the sake of a new version is usually much worse than the original.
What is common of them all? It seems that still the best writing is rewriting, but only if your understanding of the details has been grown and now you have less clutter in your mind. But if you do a rewrite without much thinking or just to get shit done, there is a high probability that you will fail.
For those who've not already come across it I would thoroughly recommend folk taking a gander at Michael Feathers book "Working Effectively with Legacy Code" on this topic.
Lots of solid advice and patterns for approaching thing sort of thing.
Lots of solid advice and patterns for approaching thing sort of thing.
To number 2 (Migrating the database is hard) I would also add that it is important to verify the integrity of the migrated database in some way afterwards. Even a simple assertion like "the number of user accounts (or whatever) is the same in both databases" will help greatly.
Yes, right! I forgot about this in the blog post. We counted the rows of some tables. Unfortunately we could not do this with all tables because we had dead data in the old database (ex: child rows without parents). But with the new database we made sure to have all the database features and checks in place to guarantee referential integrity. We also did spot tests to make sure the data is as expected.
After spending 3months rewriting an huge module for our many websites, initially estimated for 3weeks at most, I can say that it's easy to make mistake but also incredibly good for the company.
- Mixing refactoring and new feature.
- Introducing new technology
- Changing behavior of old code that were not directly concerned by the rewrite
...
Many things that should have been done after that. But, all I can say is if the company can handle this amount of work, it will always be a very good way to improve the quality and durability of the website.
Because if you simply rewrite code, there is a big chance that your company will assign to you, new task that are not related at all and you will never be able to introduce these feature.
- Mixing refactoring and new feature.
- Introducing new technology
- Changing behavior of old code that were not directly concerned by the rewrite
...
Many things that should have been done after that. But, all I can say is if the company can handle this amount of work, it will always be a very good way to improve the quality and durability of the website.
Because if you simply rewrite code, there is a big chance that your company will assign to you, new task that are not related at all and you will never be able to introduce these feature.
I have an issue with iframe perf test. It is very reckless and a waste of user bandwidth. One can reroll user requests from web server access logs and it would be a controlled perf test with real user behaviour.
Lesson 1: Don't ever attempt a big rewrite?
Sure you can attempt a rewrite! But you must make sure that you know what are getting into. It is not an easy thing to do, but once the rewrite is done your live gets way better :)
Having had rewrites of large projects go spectacularly well and also spectacularly badly, I suspect one factor to be mindful about is if the rewrite facilitates the "second-system effect" - I've had that happen where the rewrite of a messy but successful suddenly allows the introduction of lots of cool features you've wanted to add for ages and you actually end up with a "simplified" system that is actually a lot more complex than the original.
NB I've only had one project go badly wrong that way and I guess it was a useful learning experience :-)
NB I've only had one project go badly wrong that way and I guess it was a useful learning experience :-)
I agree; instead of doing big rewrites, do a bunch of small rewrites.
Especially try to avoid "big bang" switchovers.
Especially try to avoid "big bang" switchovers.
The OP's big rewrite was a lot of work. But I suspect that incrementally improving the legacy system to the point where it was as good as the new one would have been even more work. Incremental improvement is the right way to make small improvements. But a rewrite can be the right way to achieve root-and-branch reform.
I wasn't advocating a gradual improvement of the old system.
I meant identifying N particular tasks that need replacing, for example user accounts, content management, front-end, etc. Then for each one, isolating that part of the old system, eg. using a proxying layer. A replacement can then be built alongside, with the proxy sending all requests to both. Results can be compared, and differences treated as test failures. When the replacement is working, the proxy can be switched to serve the new system's responses, and once everything's settled down, the old system can be turned off.
Once all of the new system is in place and replicating the behaviour of the old system, the rewrite is finished. New features and improvements can be worked on as normal, as a separate phase.
What's more, we only need to feature-freeze the component that's being migrated, rather than the whole thing (as long as coding standards for new features enforce our newly-imposed compoentn isolation).
I meant identifying N particular tasks that need replacing, for example user accounts, content management, front-end, etc. Then for each one, isolating that part of the old system, eg. using a proxying layer. A replacement can then be built alongside, with the proxy sending all requests to both. Results can be compared, and differences treated as test failures. When the replacement is working, the proxy can be switched to serve the new system's responses, and once everything's settled down, the old system can be turned off.
Once all of the new system is in place and replicating the behaviour of the old system, the rewrite is finished. New features and improvements can be worked on as normal, as a separate phase.
What's more, we only need to feature-freeze the component that's being migrated, rather than the whole thing (as long as coding standards for new features enforce our newly-imposed compoentn isolation).
For testing performance i would recommend https://github.com/buger/gor (i'm author).
Nice, I was just going to point out that using an iframe really seems like a bad idea.
Any drawbacks of #3?
It's a really easy thing to do. But you need enough server resources for this. A mirror of the current production system would be the best thing. If you only have one server where the live system is running and there is not enough resources on this server you can not do this.
I rewrote my product 6 or 7 times in the past 6 years. 4 of those years were written in isolation by myself. 1.5 years I worked as a developer which I HATED and fired from (refusing to work excessive overtimes and as a result impossible to find a job in this competitive labor market) and working on my product again. I open sourced the old code that I "threw" out to start all over.
Technically, never reached market until 5 months ago. I have of course, launched after each time the product was finished but would give up due to being burnt out etc. An entire year spent starting from scratch which would be my 7th attempt.
3 month in after launch I was going to rewrite it again, I decided, this would've been my last attempt in the software business. I decided that enough was enough, this is it. I ALMOST rewrote it again but having done that 7 times already, instead of doing that, I just ripped out a lot of code, made architectural changes and reused what I could, whatever was tested. 8 times a charm....
My lesson learned from past 6 years pursuing insanity, restarting from scratch means you will have to solve the same problems again and again. A blank canvas is NOT a solution. Don't do that, it's insane, I was insane but not anymore.
Technically, never reached market until 5 months ago. I have of course, launched after each time the product was finished but would give up due to being burnt out etc. An entire year spent starting from scratch which would be my 7th attempt.
3 month in after launch I was going to rewrite it again, I decided, this would've been my last attempt in the software business. I decided that enough was enough, this is it. I ALMOST rewrote it again but having done that 7 times already, instead of doing that, I just ripped out a lot of code, made architectural changes and reused what I could, whatever was tested. 8 times a charm....
My lesson learned from past 6 years pursuing insanity, restarting from scratch means you will have to solve the same problems again and again. A blank canvas is NOT a solution. Don't do that, it's insane, I was insane but not anymore.
Stop chasing your tail. I have no idea what you are working on, but if you are not trying to solve a hard CS problem and you are trying to build a business ...
NLP from texts detected from the audio in video files. 100% solved? No. Others have attempted or marketing their own solution? Yes.
I was just really crazy and stupid to pursue my own vision of what the solution would be when in reality I lacked the experience (I started with barely knowing how to code).
I was just really crazy and stupid to pursue my own vision of what the solution would be when in reality I lacked the experience (I started with barely knowing how to code).
Maybe they could have started by refactoring the schema in-place (That is, within mysql and webtek). Then change from mysql to pgsql - still behind webtek. Maybe separating the change of webtek -> django from the interface change is a bit harder, but you could still try to restrain your self as much as possible and then make a second iteration of the interface after the initial change.
I've taken this approach before, when migrating a big product and it worked quite well. It seems as a lot of extra work at first, but I'm pretty sure that is not actually case.