The downside is obviously that uncached queries take much longer (adding >100ms) and more queries are uncached since you can't share the cache with a large user-base. Unless you just visit the same websites over and over again, this results in worse overall performance.