Tbh I can't follow your logic here. You mentioned that in VanJS there are things that need to be careful with. But the same is true for other frameworks, even for plain Vanilla JavaScript. Thus what exactly the point that you're trying to make?
Complexity can't be avoided for extremely complex use cases. But that doesn't mean a simple solution that can work for most of the use cases has no value.
With bundler like esbuild, all JS code, including 3rd party libraries can be bundled into a single file. You don't need an extra round trip for a library.
JSX is redundant, at best. VanJS demonstrated ordinary JavaScript syntax is as good as JSX, if not better. How can something feel sugar-ish if it's actually more verbose than ordinary JS code? Plus, you can't really execute JSX code in browser's developer console.
The fact that you have to manually maintain the binding between states and UI elements and propagate state changes to UI elements is exactly the thing offered by VanJS, or other popular reactive frameworks (despite with a much larger bundle size)
https://vanjs.org is another example in that direction. It illustrates that it's even possible to have a full-fledged solution of composing complex reactive DOM tree with just plain JavaScript functions.
Complexity can't be avoided for extremely complex use cases. But that doesn't mean a simple solution that can work for most of the use cases has no value.