I think this is such an interesting question! Perhaps a new science of creative management. I guess there were a few places that presumably seemed to do this successfully (e.g. bell labs :)). There is probably a 51% problem somewhere in there, where the institution would need to make sure it isn't captured by those who would 'phone it in'. Maybe you could pay people a healthy sum to quit at any time.
Managing a process which has no defined outcome, but whose local quality and direction can be measured seems like an interesting research path in itself. I think the institute would have to be bootstrapped and a lot of learning would have to take place to make a sustainable version of it. Also, as you say hiring would be challenging (though maybe no more challenging than hiring professors). However, I'll say that I can recognize at least one phenotype of researcher reliably who would thrive there.
I'm a researcher in Biotech, and honestly, a working at curiosity driven research institute is all I've ever wanted from a career. I have the credentials and have proven myself academically, and in industry, but this kind of position is just so hard to find! I really just need time and a small amount of resources to work on ideas. I usually struggle to get more than 20% of my time working on my own ideas.
I've spoken with colleagues over the years, and when I bring up this desire, so many of them feel the same way. Many of these people are incredibly accomplished and come from top institutions. What a waste to not give that magnitude of creativity an outlet! The only path I see to this type of life now, is independent wealth. It just strikes me as such an opportunity to create a place for these people. If you build it they will come!
Managing a process which has no defined outcome, but whose local quality and direction can be measured seems like an interesting research path in itself. I think the institute would have to be bootstrapped and a lot of learning would have to take place to make a sustainable version of it. Also, as you say hiring would be challenging (though maybe no more challenging than hiring professors). However, I'll say that I can recognize at least one phenotype of researcher reliably who would thrive there.