I was also surprised by the low score of NS. I think the ranking puts too much weight on pricing.
NS is quite expensive, but the service is way better than say, DB. In my last trip from the Netherlands to Germany, officially 2:45 hours, I had a 45 minutes delay just on the German side, on both ways, at least they are consistent.
For some reason, when talking about Japanese authors we (almost) always forget about: Ryunosuke Akutagawa [1], author of short stories like Rashomon or In a grove [2] (this actually the story that inspired Kurasawa's Rashomon)
You are right, it is not specific to software, just wanted to add more context and also, because I don't know the answer and I believe there are multiple factors, but my two cents: a mix of history and culture.
By the time software came around, Japan already had a well stablished manufacturing industry that had a reputation for quality (this wasn't always the case).
Software was something totally new and wasn't considered important, why? perhaps management didn't understand it or didn't know how to handle it and did nothing.
The above made a career in software not very attractive compared to other industries. Since then, companies struggle to get good developers, who can move to more attractive sectors or countries than trying to change things within Japan with its rigid hierarchy.
How things move in Japan is generally by someone breaking the status quo and the rest following if it is a success. That was the case with Toyota in manufacturing for example. For software it hasn't happened.
Another problem IMHO is that code is invisible to users. If the app works, people don't care if the code is beautifully made or not, and without consumer pressure, management doesn't care either. Compare that with the complains they'll get if a physical object had the sightless defect.
Finally, there is a higher entry barrier for physical products than software. Any teenager can build an app in her bedroom, it is a different story to develop a bullet train. Inefficiencies and slow decision processes in manufacturing are less obvious than in tech (though manufacturing industries in Japan are struggling as well because of this), and Japan simply isn't competitive in this area, making it less attractive to invest.
And perhaps, the most important: People got used to it and don't have higher expectations.
Blaming it to the keiretsu is too simplistic. Here other factors that in my experience (living, studying & working) in Japan contribute:
- Graduates don't apply to a job, but to a company. The company decides where to place then. The first years are generally a rotation between departments until it is decided where to allocate them. This means that they will be often misplaced in positions for which they don't have the necessary background nor motivation to learn or contribute because in 6 months they'll be somewhere else.
- It is not uncommon for developers not to have a CS or coding background. They learn on the job how to "program" but lack best practices, etc. and figure out things as they go.
- Standing out is frown upon
- People who are good at their jobs are generally rewarded with more work. One can know who the manager's favourite is by who busy the person is. At the end, the good ones end up burned out, over-stressed and brain death.
- Looking busy or hard working is more important than the outcome.
- Combine the above, and there is no point to do a good job: There is no reward other than more pressure to deliver but on the other hand, as long as you look busy, not delivering is not "punished". Clear what option most people will take.
- As already commented, decisions are top down and often very conservative replicating old methods digitally.
- Many customer facing products will try to cover as many cases as possible to avoid complains, perceived discrimination or causing trouble, for example. This results in over bloated software, websites, flyers full of information, etc.