There is a well-known paper related to a statistical zero-knowledge proof about Kolmogorov complexity, but this proof introduced is considered a perfect ZKP
Methodology for comparison: train zstd dictionary on enwik9. Then build my dictionary as most common words in enwik9. Mine does 13% better because of the way I discovered how you can generate dictionary replacement symbols
I used the z-score. How can you claim that the digits of pi are random, yet a random forest classifier predicted better than the distribution probability. Your claim implicitly means "there is no structure." The hard thing to understand is that the classifier didn't see the test set, so what structure did it learn? At the very least this is an interesting question
Also, I am testing different ranges of digits other than first 10,000, but the problem with other ranges is that the distribution of digits is highly imbalanced and the model is not showing statistical significance, but models have a harder time when the distribution of classes is not 50/50, so I think its not quite fair to evaluate the model on these ranges.
So why do you think the first 10,000 digits are somewhat predictable?
The issue is not with getting the digits, the issue is with running a large model for larger digit ranges. I tried running with 10,000,000 digits and haven't gotten a prediction yet.