I helped build a similar setup a "long" time ago to build a sky HDR database. We had over 40k images across hundreds of days. The longer exposures definitely did not help (even with a ND filter) and by the end the camera had some broken pixels along the sun trail. Nothing noticeable visually, but likely not good for computer vision purpose. If we were to redo this it would be definitely a valid concern to address.