Sure, you can. But on advantage of IPv6 is you addresses are globally unique. If you ever need to connect two networks that were created independently (like different companies), unique addresses, even local are very helpful.
I've given a try to IPv6 in a company with few tens on servers in a 2 DCs, an office + additional location, 3 ISPs in total. For me the real challenge is not just different way to write an IP address or doing NAT. The challenge is that IPv6 changes a lot of unexpected things:
- Our ISPs support IPv6 but routing quality is way worse than IPv4 including occasional inability to connect to some networks or greater latency than IPv4. I had to create tickets with such issues understood that most probably they just don't have IPv6 BGP sessions to all their upstream providers they connect.
- How the VPN (an employee / road warrior setup) should be configured since from the routing perspective you don't need a VPN to connect from your home to the office? Assuming both have proper IPv6 connection and all devices in the office and your laptop have a globally addressable IP address. Employee can have IPv4 or dual stack at his home, where is dual stack in the office. Very confusing. Looks like Fortigate also don't have an idea and decided to not support such case.
- You have to be careful with site-to-site VPN since even your internal services like database are now globally addressable. You really need proper firewall rules / routing policies to not leak unencrypted packets over internet.
- SLAAC is cool but doesn't provide DNS configuration. (there is RFC8106 but is it supported by all OSes?). You need DHCPv6 for that. You have to choose: use only DHCPv6 or SLAAC + DHCPv6 or just relay on the vast that DNS will be proviedd by DHCP IPv4 in a dual stack setup.
- The way of providing high availability gateway address in a network is different. You need router advertisement where you can provide priorities. That actually is much better than any other VIP mechanisms (no issue with MAC table updates, etc.) but you need to know that.
- OSPF works a bit differently. For example: there is no authentication in router communication in OSPF itself, you are supposed to use IPSec.
With BGP (especially anycast) you don't have direct influence where the requests lands. You can steer traffic by techniques like AS prepend, some priorities per BGP session (I'm not an expert in BGP) but eventually it's not CDN that decides where the request will be routed. It's decided by routers of the client's ISP and backbone networks, each taking decision themself (shorted AS path from his point of view + BGP policies set by network operators), not by you. You can't for example split traffic between DCs with specific proportion (like 30% here and 70% there). You can't split by something else than network properties like forward request to the region that has better chance to have this content in the cache.
With DNS and dynamic responses you are directing request to specific DC, even server, almost on every request. It may be dedicated for this traffic type (live stream different than static images etc). Your DNS server can take the hostname ("www.google.com") into consideration - BGP doesn't even know the hostname in the URL. If you wanted to do it with BGP you would need to place specific content to a dedicated /24 subnet and that is impossible considering how many IPv4 addresses are available.
BGP doesn't even consider network latency, current network load. CDN knows load on their machines, on their network link, where given content is placed. The bottleneck may be storage, network or CPU processing, different for different sites and content type. They need to direct traffic on request basis considering this and at least the hostname from the URL. That's why DNS is used first.
Varnish (HTTP caching only) also uses one thread per client. I believe worker threads are used to handle reuqests while a dedicated thread handles all the idle connections between requests using epoll(). Also per-threads stack size is lowered so thousands of threads don't occupy massive amount of memory.
Single threaded HTTP servers have their own issues. If the bottleneck is the storage then lack of async open()/stat() and some other calls is problematic. We feel that serving hundreds of millions of files (long tail content) from slow storage using nginx. For that reason you can configure nginx to spawn multiple processes.
Our security team complained that we have some services like monitoring or SSH access to some Jump Hosts accessible without a VPN because VPN should be mandatory to access all internal services. I'm afraid once comply we could be in similar situation where Facebook is now...