The 25 million lines of code is only the source code of Oracle Database written in C.
The test cases are written in a domain specific language named OraTst which is developed and maintained only within Oracle. OraTst is not available outside Oracle. The OraTst DSL looks like a mixture of special syntax to restart database, compare results of queries, change data configuration, etc. and embedded SQL queries to populate database and retrieve results.
I don't know how many more millions of lines of code the tests add. Assuming every test takes about 25 lines of code on an average (every test used to consume about half of my screen to a full screen), we can estimate that the tests themselves consume close to another additional 25 million lines of code to 50 million lines of code.
I have not found this to be true at all. I have seen both US and Indian developers adding good code as well as ugly code to the Oracle Database product.
The actual damage was done much before I had joined Oracle. It appears that somewhere in the early 2000s, the Oracle codebase went from manageable to sphagetti monster. The changelog showed more changes from US developers than Indian developers at that time. Once the damage was done, all developers whether from the US or India now need to follow this painful process to fix bugs and add features.
I was given 5 minutes to ask questions after each round of interview. That part was ordinary too. That's what most of the other companies do (FAANG or otherwise).
> Tests that run for 30 hours is an indication that nobody bothered writing unittests.
Yes, they were not unit tests. There was no culture of unit tests in the Oracle Database development team. A few people called it "unit tests" but they either said it loosely or they were mistaken.
Unit test would not have been effective because every area of the code was deeply entangled with everything else. They did have the concept of layered code (like a virtual operting system layer at the bottom, a memory management layer on top of that, a querying engine on top of that, and so on) but over the years, people violated layers and wrote code that called an upper layer from lower layer leading to a big spaghetti mess. A change in one module could cause a very unrelated module to fail in mysterious ways.
Every test was almost always an integration test. Every test case restarted the database, connected to the database, created tables in it, inserted test data into it, ran queries, and compared the results to ensure that the observed results match the expected results. They tried to exercise every function and every branch condition in this manner with different test cases. The code coverage was remarkable though. Some areas of the code had more than 95% test coverage while some other areas had 80% or so coverage. But the work was not fun.
That would have been fun but in reality there was no downtime. Developers like me were expected to work on two to three bugs/features at a time and context switch between them.
If I submit my test jobs today to the farm, the results would come one or two days later, so I work on another bug tomorrow, and submit that. Day after tomorrow, I return to the first bug, and so on.
On the contrary, the interview was an ordinary one. The screening round consisted of very basic fizzbuzz type coding ability checks: Reversing a linked list, finding duplicates in a list, etc.
Further rounds of interviews covered data structure problems (trees, hashtables, etc.), design problems, scalability problems, etc. It was just like any other interview for software engineering role.
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
I worked at Oracle for a few years in their database development group (internally known as "RDBMS"). You would be surprised to know that they follow a strictly waterfall process to perform database development. The results are horrible.
Process:
- Let us say a release is planned for January 2018. Projects are decided around June 2017. By the way, In Oracle RDBMS, a project essentially means anything more than 5 lines of code change.
- Each developer is assigned a project.
- The developer is given a deadline to write a design spec by August 2017. Developer spends 3 months in writing a design spec. A lot of bike-shedding occurs for those 3 months on mundane topics like whether a certain timeout should be 0.5 seconds or 1 second.
- Then the developer is given another deadline, say September 2017, to write a functional spec. More bike-shedding.
- Then the developer is given yet another deadline, say October 2017, to write a test spec. More bike-shedding.
- 5 months have passed and not a single line of code has been written. But 3 large documents with about 10000 words in total and another 10000 words of bike-shedding has been written.
- Finally, the developer is given another deadline, say November 2017, to complete writing the code with tests.
Results:
- Developer writes 10 lines of code and checks in their code by November 2017. Project completed!
- Everything looks fine for a few days.
- About a million or so tests run over the next few weeks. 1000s of tests are found to be broken. So much for 5 months of spec reviews! (Note: This happens for every such project, about 20 or so projects in a year.)
- The bugs are distributed to various developers. It takes them another 12 months to figure why each test broke, how tightly every layer and corner of the code is coupled with each other, and how they are solving the biggest puzzles of their lives: How to fix one thing without breaking 10 more things?!
- The original design, functional and test specs go out of the window. This is really bad time. Management is unhappy. Larry is unhappy! We got to fix those tests, no matter what it takes to fix them. Radical decisions are made as developers discover clever tricks to fix one part of the code while somehow finding a way to keep the other part of the code happy.
- 12 months have passed. It is November 2018 now! (Remember, the project was scheduled to be released 11 months ago!) It looks like everything is fixed. The whole 10 line project is somehow held together by tests. Nothing appears to be broken. But nobody is really sure.
I am glad there are other saner companies where people understand that waterfall model development is flawed and embrace agility. Having done waterfall for a few years, agile development is a breath of fresh air where I can spend more time writing code than writing documents, where I can evolve the code based on actual user feedback rather than feedback from bike-shedding by people who have not contributed a single line of code to the software.
TL;DR - An employee found fire safety violations in one of its Bangalore offices. He reported the violations internally. Nobody addressed the violations. He reported the violations to the cops. Oracle sacked and sued the employee.
Oracle is a safe haven for incompetent managers and so called team leads. I know a few ex-colleagues of mine who left Oracle to join other companies and were shocked to see how fast and agile other companies were.
It is not in Oracle's DNA to hold managers accountable or fire them for management blunders or shoddy management. So when some of these managers and team leads from Oracle joined other companies, they found much to their surprise that they were answerable to higher management for the state of the project they were managing. They could not keep up with the pace of development or having concrete responsibility, and returned to Oracle to live a purposeless but relaxed life where one would earn less but otherwise thrive in the company by doing little to no actual work.
- A couple of years ago, the company was providing us Oracle branded corporate diary for free. An employee was allowed to order one diary. The cost center would pay the price of the diary which if I remember correctly was $3.00. There were many managers who did not allow their departments to order this diary in order to save that $3.00 per employee for their cost centers. "But I want the Oracle branded diary!" "No, we need to save money!" "I will be proud to show the Oracle branded diary to my family" "No, we need to save money!" "Then why bother providing the diaries in the first place?" "No, we need to save money!" And all this commotion happened when Oracle was spending millions of dollars in legal fees for its lawsuit against Google.
- In a meeting with a very senior VP of our department, we asked our senior VP why there were no fun activities in our team, like a day out for fun, or even something as small as a team lunch. The response by the senior VP to the best of my recollection was, "Think not what the company can do for you. Think what you can do for the company. You guys are incredibly privileged to work for Oracle. What more can one need? This is not the time to think about fun. This is the time to think how you can reduce the number of bugs in our product." It was no surprise later that within a year, all the smart developers of our team left to other companies who not only pay better than Oracle, but also offer diaries without fuss, and have team lunches and other fun events.
Disclaimer: Worked for Oracle for 3 years. Will never work for Oracle again.
The test cases are written in a domain specific language named OraTst which is developed and maintained only within Oracle. OraTst is not available outside Oracle. The OraTst DSL looks like a mixture of special syntax to restart database, compare results of queries, change data configuration, etc. and embedded SQL queries to populate database and retrieve results.
I don't know how many more millions of lines of code the tests add. Assuming every test takes about 25 lines of code on an average (every test used to consume about half of my screen to a full screen), we can estimate that the tests themselves consume close to another additional 25 million lines of code to 50 million lines of code.