dcode · HackerTrans

> the Web side is largely complete and successful

Perhaps from a C++ perspective, say a monolith with kilobytes of bindings driving a canvas. Yet

> the original purpose of Wasm

includes, as per the charter, "and interoperate gracefully with JavaScript and the Web", which is far from complete and, thanks to WASI and the Component Model, hardly improving. I guess it will remain a mystery how WASI could basically replace something as obviously useful as WebIDL-bindings - that is, until someone figures out.

dcode

4 anni fadiscuss

I guess we are about to find out whether there is substance to the precedents. My bet is on "what can go wrong, will go wrong", even more so on Web scale. Let's hope I'm wrong.

dcode

4 anni fadiscuss

I do understand the desire to switch all languages and systems to one encoding, of course. However, Switching a WTF-16 language to UTF-8 removes previously valid values from strings, then exchanging what's errors/mutation on Component boundaries right now with errors/mutation when using string APIs. Can't be done in a backwards-compatible way, and all these languages have a lot of existing code. If backwards compatibility is a goal (say when using a breadcrumbs mechanism as in Swift), one still ends up with WTF-8 underneath, which maps to WTF-16, but is not UTF-8. Hence why I think it's impossible, because the only way to pull this off is by replacing affected string APIs (and/or accepting that old APIs then throw or mutate). Likewise, I see a possible future where JS adopts breadcrumbs, but then with WTF-8 (and perhaps a well-formedness flag), not guaranteed UTF-8. In your use case, that would yield a fast-path if a string is well-formed, but still with the same old fallback. Plus, of course, that having a systems fast-path implies that there is a corresponding JS-interop slow-path (when using AS).

dcode

4 anni fadiscuss

In more technical jargon, it is about value spaces. All UTF-8 strings map to WTF-16 strings semantically (lists of Unicode Scalar Values are a subset of lists of Unicode Code Points), but some WTF-16 strings do not map to UTF-8 (Unicode Scalar Values exclude some Code Points). That's something UTF-8-based languages have to deal with anyway (on the Web, which is WTF-16, or any other mixed system), but it's odd that the expectation has become that even languages that map to each other incl. to JS/Web APIs now have to share a problem that does not exist in their native VMs for reasons. To emphasize: These languages just do what's perfectly fine in WebIDL, JSON, ECMAScript and their own language specifications.

  let myString = "...";
  let returnedMyString = someFunction(myString); // might or might not cross boundary
  if (myString == returnedMyString) {
    // sometimes silently false
  }

I am having a hard time to understand how so many people prefer this as the outcome of a Web / polyglot / language neutral standard, especially since affected languages cannot change, and the problem is so trivially avoided, say with a boolean flag "this end is WTF-16, it's OK if the other end is as well" (otherwise use well-formed/UTF-8 semantics).

For context, I once gave a presentation about the pitfalls: https://www.youtube.com/watch?v=Ri2NMnSQo4o

dcode

4 anni fadiscuss

To me it's not so much a question of what's the best / recommended (for new languages) / most used encoding. It's rather the observation that there are so many popular languages operating on Unicode Code Points, not Unicode Scalar Values, and Wasm said it wants to support these (as) well, incl. integration with JS/Web APIs which already share their semantics. And for all of these, the restriction is unnecessary and trivially avoidable by allowing them to pass strings to each other, or from/to JavaScript, without errors or mutation, while not changing anything for those preferring UTF-8. Looking at this the other way around might be valuable as well: What if one would design a Component Model for any of the affected languages? There, introducing an unnecessary discontinuity would be a design mistake I'd argue.

dcode

4 anni fadiscuss

That's not the issue / impression. The issue is that well-formed UTF-16 is rarely used in practice. All of JavaScript, Java, C#, Dart, Kotlin etc. effectively use WTF-16 for compatibility and performance reasons, and that's what's semantically distinct from UTF-8. These have asymmetric value spaces, so that strings in "legacy" languages would sometimes throw or implicitly mutate synchronously. Mixed systems typically use WTF-8 as the common denominator for this reason, i.e. not UTF-8, but Wasm decided against.

dcode

4 anni fadiscuss

The value spaces are in fact asymmetric. In Unicode jargon, UTF-8 is a "list of Unicode scalar values" while WTF-16, i.e. UTF-16 as seen in practice, is a "list of Unicode code points (except surrogate pairs)". Unicode code points are a superset of Unicode scalar values, aka "list of Unicode Code Points except surrogate code points", hence conversion from WTF-16 to UTF-8 is lossy. In WTF-16 languages, this is not a problem because the system is designed for WTF-16, but it becomes one if the value space is restricted further, here to UTF-8.

That's why, in such mixed systems, https://simonsapin.github.io/wtf-8/ is typically used. Not UTF-8. And Wasm is such a mixed system.

dcode

4 anni fadiscuss

Except that picking the most restrictive option prevents two components that are using the same less restrictive semantics to communicate with each other securely (without throwing exceptions or silently mutating data), e.g. Java <-> Java, AssemblyScript <-> JavaScript. To illustrate, if one would design a Component Model for Java, restricting to UTF-8 would make the Component Model for Java a hazard for Java. The same effect happens in a multi-language Component Model, where some languages then work and others don't. Hence "pick the best option" falls short. The argument in all these questionably stonewalled discussions is basically to allow these languages and use cases to exist, which could be as trivial as to make UTF-8 the default if the WASI folks so wish, but also have a Boolean flag for "don't eagerly mutate on WTF-16 pass-through". Yet, even though trivial and rather obvious, this has been fought relentlessly since 2017, and surely one has to wonder why this vehemence.

dcode

4 anni fadiscuss

Despite that speed is not the critical concern (asymmetric value spaces are), UTF-8 must guard against invalid byte sequences, while WTF-16, where all possible values are valid as long as byte length is a multiple of 2, does not. In practice, the guard in UTF-8 is part of the copy loop over a boundary, typically from untyped memory to untyped memory, while WTF-16 can indeed just memcpy. Unless SIMD can be utilized, the difference is about that of a loop over a load into a branch vs. a memcpy, in case this helps to quantify. Expect an additional final memcpy if the UTF-8 copy should fail before storing anything to the receiver's memory.

dcode

4 anni fadiscuss

Well, I tried. Anyway, even if I was the abhorrent monster you keep painting in your almost exclusively ad hominem argumentation while accusing me of what you are undoubtedly guilty of yourself, I'd argue that none of this justifies plain ignoring technical concerns in a standardization effort. This exchange is an almost perfect reflection of the practices prevalent in the Wasm CG, that made the Component Model, and likely other things elsewhere, possible almost uncontested. And surely this is deliberate abuse, and I hope people can see that. And coincidentally, that's exactly my critique. I hope nobody is surprised that being at the receiving end of this, despite your best efforts, for years, is an extraordinarily frustrating experience, and that this is exactly the point.

dcode

4 anni fadiscuss

Allow me to focus on the technical arguments, that I think fall short. Sure, one can go and make a second string type, or a bunch of throwing APIs. I'd question that this actually improves a language in a tangible way, or that anyone would want that for a rational reason that is not induced from the outside. When not doing so, it's incompatible in the sense of the word. Whether incompatibility isn't much of a deal depends, I guess. For someone not having a respective use case, perhaps, but for someone else having exactly that use case, say hashing substrings to then discover unexpected collisions, or streaming 1K chunks of strings over component boundaries, then discovering mojibake after concatenating, it might very well be significant, or even expensive. I mean, there are good reasons all those languages try the best they can to prevent that in their native habitats. What amazes me is that all that came to be because someone has formulated a desire, that could easily be fulfilled in addition with a boolean flag for W/UTF, but refuses to include such a trivial compromise, which surprisingly has more weight than any evidence, or precedents like WebIDL, JSON, or the various language standards. I find this highly concerning, since it conflicts with my understanding of responsible engineering. Also conflicts with Wasm's communication, that literally states that Wasm executes in the same semantic universe as JavaScript, and maintains backwards-compatibility with the Web. Perhaps, if there is something fruitful to spin a narrative around, then that these decisions undermine the exact value proposition of AssemblyScript, that was supposed to be used in tandem/closely with JavaScript, which now becomes risky on the fundamental level of the most prominent higher-level data type, strings. Plus, of course, when two AssemblyScript components communicate. That makes these string decisions particularly unfortunate for me personally after having spent all that time and effort, working towards Wasm's goals in good faith, perhaps explaining my persistence on the matter. Quite a dilemma.

dcode

4 anni fadiscuss

Perhaps this is missing some necessary context. Note that the threads happened before a resolution, and have been created a few days before using them as an argument in Wasm. There was and still is no such design principle, because it doesn't make sense. Similarly, the arguments in the thread referenced are nonsense. JSON and CSS modules are files, which have different requirements than API calls. importScript the same, loads source code. Also note that JSON actually preserves idiomatic JS strings in data (APIs) through escape sequences, even though it stores as UTF-8 (files). Preserving in JSON makes sense, since sometimes JSON.stringify/parse are used for things like synchronous IPC, where maintaining integrity is critical. The Component Model should do the same, but refuses. The others are networking APIs, where UTF-8 is common. These are asynchronous and typically untrusted, where the protocols mandate UTF-8, and not maintaining synchronous state is much less problematic there. However, mutating string data as a side-effect of a normal function call is nothing less than a hazard, since some strings then seemingly randomly won't compare equal anymore after a call and stuff like that. WTF-16 (what JS effectively specifies) -> UTF-8 is lossy. The people in the threads know that very well, but keep bringing up the same already debunked arguments over and over again nonetheless to get their way. There, the thread suggests to encourage UTF-8 for new JavaScript APIs as well, which plays into the desires on the Wasm end yet is well-known to be impossible to pull off in JS. Applying this to JS conflicts with the ECMAScript specification, because JS has the same hard backward-compatibility constraints as Java, Dart, Kotlin, C# and other languages that evolved from UCS-2 to WTF-16. None of these languages can change their string semantics, in particular not to something akin to UTF-8, since it's semantically more restrictive and hence would break execution of existing and, arguably, all future code using idiomatic APIs like substring without being aware of potential mutation. It's subtle, I admit, but given that the arguments are factually invalid, the threads can only be one of two things: Incompetence or dishonesty. As always in political contexts, the mere mortal wonders what's better. Given foregoing discussions, where the same people participated, I would rule out incompetence. Now go there and question, and always the same gate keepers show up, simulating responses in good faith. Killing you with kindness, or however that's called. Dare to be unimpressed and follow-up, and someone quick-draws a CoC. That's why I decided to try something new, there pinging various people from the TAG, in the hope to finally get eyes on this behavior by someone knowledgeable, which obviously failed as well. Not sure if that helps, but that's some of the background :)

FWIW, here's the presentation I wanted to give to the Wasm CG that explains the details and pitfalls in an easier to digest way: https://www.youtube.com/watch?v=Ri2NMnSQo4o

dcode

4 anni fadiscuss

It is almost funny, almost sad, that we were in this for nearly five years, every single day, and despite that a cartel created for the occasion completely evaporated us politically for pointing out "technically not wrong" technicalities that nobody ever had the intention to address, this is how we are perceived still by the same people we once thought we were giving a voice to :) Yet, somehow, Wasm has turned out to be almost indistinguishable of what was once presented or what is still marketed by its outdated supplementary material. Openness? Portability? Security? Web interop? Multi-language? Professional conduct? Giggles through rose-colored glasses. https://github.com/WebAssembly/WASI/issues/401 —the AS guy

dcode

5 anni fadiscuss

AssemblyScript maintainer here. Can confirm, other language communities have practically zero influence on the design. In fact, AS was present during "specification" and has been completely annihilated for disagreeing with what the chairing majority has come up with there. It's particularly ironic that the Component Model suddenly targets Web integration again, or that this is now coined "language neutral". In relation, the syntax is more of a minor detail.

dcode

5 anni fadiscuss

Sure, we could resolve this problem by either a) giving these languages a separate fitting string type to use internally or externally (Rust for instance can use 'string' everywhere) or b) integrating their semantics into the single one so they are covered as well as first-class citizens. And coincidentally, that would fit JavaScript perfectly, which is rather surprising being off the table in a Web standard. Yet we are polling on having a "single" "list-of-USV" string type, likely closing the door for them forever with everything it implies.

dcode

5 anni fadiscuss

This is common indeed, and isn't a bug in the affected source languages for reasons. How it displays when printed is irrelevant.

Here's Linus Torvalds explaining it better than I could: https://youtu.be/Pzl1B7nB9Kc?t=263

And sure you can transfer your string that someone else does not consider a string using alternative mechanisms, but then you are only not doing anything wrong because you are not doing it at all for entire categories of languages. There is no integration story for these, and once one mixes with optimizations like compact strings or has multiple encodings under the hood one cannot statically annotate the appropriate type anyhow. And sadly, adapter functions won't help as well when the fundamental 'char' type backing the 'string' type is already unable to represent your language's string.

I also do not understand where the idea that a single language always lives in a single component comes from. Certainly not from npm, NuGet, Maven or DLLs.

Extended this post to provide additional relevant context. It's not a bug, it's a feature.

dcode

5 anni fadiscuss

At that time, it was ~ "you either stop advocating for the security concern, or we make sure you get nothing at all". In fact, I believe we would need to trap to at least make the breakage non-silent, but that would break AssemblyScript even more obviously and only exchange data corruption with denial of service. Not necessarily an improvement if you care about people using what you are building. In hindsight I am not proud of that comment and should have protested, no matter how dire the situation.

dcode

5 anni fadiscuss

The security aspect is separate from whether UTF-16 lowering and lifting is supported. One simply cannot roundtrip every possible DOMString, C#, Java etc. String through the single concept of Unicode Scalar Values without either introducing lots of surface area for a) silent data corruption (what you call "recommended security practice", i.e. strings not comparing equal anymore after what appears to be an innocent function call) or b) for (deliberate) denial of service (when erroring instead). I mean, there is a good reason why all these languages do not do that in between function calls, but actually try very hard to guarantee integrity. And in a real program, an Interface Types function looks like any other function to the developer, just imported somewhere in the codebase, so good luck documenting that. Other than that I do not know how to respond to your subtle insults, please forgive my ignorance.

comments