A large data center has 20-30k machines. Amazon disclosed that back in 2014. Most likely you want to partition the machine into some chunks, so each application or each database only use some of them. So you don't need to talk to many machines at once.
In general, a distributed database can be really fast. DynamoDB claim to be 3ms (AWS reinvent 2014). The current technology can probably achieve close to 1ms, which is equivalent to memcache. With such performance, you don't really need local storage.
You can get much faster IO if you use many local SSDs. The downside is utilization. It is very rare a single machine has a workload that fully utilize local disk. You end up over provision greatly fleet-wise to get high performance. A managed database over a network is more likely to utilize disk/SSD throughput.
Write to disk has no practical latency because of write buffer, either local file system or remote database. Flush to disk would be slow unless you use SSD.
On the other hand, a single machine has limited reliability. If one wants to have high availability, they needs to dual write to another machine, which also has network latency.
I saw many comments about stateful workloads. I am not sure it is a necessary issue for cloud environment.
Within a zone or a cluster, the latency is about 1ms, which is faster than most hard disks. The network bandwidth is on par with disk throughput. What we really need is a faster database and a faster object storage that can match the network performance (1ms and 10Gbps), then all workloads can be stateless.
If one uses a VM on GCP, the VM has no local storage besides the small local SSDs. Practically even the VM is stateless besides some cache.
The short answer is layering. For example, to support mutual TLS, you need a system that distributes and rotates TLS certs to different nodes. If we don't want to add to many features to Kubernetes, then we need another layer. That becomes Istio or something like Istio. That is roughly how Istio came to exist in the first place, as it models after several Google internal systems that are on top of Borg instead of being part of Borg.
Such policy existed since the very beginning of Google APIs, and is well documented within the company. Anyone who worked at Google should be aware of it.
People often consider to use this information for reliability and performance, but you can do much more with the data. For example, if a method has low latency, you can use short deadline with fast retry to improve reliability. If you see a sudden jump of certain usage, you can consider to use batching and caching to reduce your cost. If you see an unexpected usage of a service, you know someone introduce a new dependency in your system. Google teams often use the same data to understand how large services work and how they are correlated.
The app can post your data to anywhere it has access to. This is commonly known as data exfiltration. The common way to prevent that is to run the app in a secure sandbox. Most OS don't provide such capability in a usable way.
If a node goes down, another node will replace it quickly. The node (aka tablet server) doesn't own any data. The data is stored on lower level storage layer.
In general, a distributed database can be really fast. DynamoDB claim to be 3ms (AWS reinvent 2014). The current technology can probably achieve close to 1ms, which is equivalent to memcache. With such performance, you don't really need local storage.
You can get much faster IO if you use many local SSDs. The downside is utilization. It is very rare a single machine has a workload that fully utilize local disk. You end up over provision greatly fleet-wise to get high performance. A managed database over a network is more likely to utilize disk/SSD throughput.