Yesterday the replica randomly disconnected from the master and could no longer reconnect, the resyncs were failing because a replication buffer on the master was being hit (https://redislabs.com/blog/top-redis-headaches-for-devops-re...). Once that buffer was increased the replica was able to sync the snapshot from the master but for some reason was taking a very long time to load the snapshot, during this time the sentinels thought the replica was healthy again and started allowing the application to read from them, of course Redis responded with an error "dataset loading". We were running Redis 6.0.8 and upgrading the replica to 6.2.1 allowed the replica to sync and become healthy in seconds.
I'm still not sure why the sentinels thought the replica was healthy as issuing commands to it always returned an error.
Yesterday the replica randomly disconnected from the master and could no longer reconnect, the resyncs were failing because a replication buffer on the master was being hit (https://redislabs.com/blog/top-redis-headaches-for-devops-re...). Once that buffer was increased the replica was able to sync the snapshot from the master but for some reason was taking a very long time to load the snapshot, during this time the sentinels thought the replica was healthy again and started allowing the application to read from them, of course Redis responded with an error "dataset loading". We were running Redis 6.0.8 and upgrading the replica to 6.2.1 allowed the replica to sync and become healthy in seconds.
I'm still not sure why the sentinels thought the replica was healthy as issuing commands to it always returned an error.