Your Code Displays Japanese Wrong(heistak.github.io)
heistak.github.io
Your Code Displays Japanese Wrong
https://heistak.github.io/your-code-displays-japanese-wrong/
93 comments
I don't think Han unification is as evil as people make it out to be: the characters that got unified do share the same semantic meaning across languages, whereas Latin vs Cyrillic there is no semantic equivalence since letters are purely orthographic glyphs. The thing that sucks is that there's no real inline way (variation selectors, kind of) to select the language for the CJK glyph.
> the characters that got unified do share the same semantic meaning across languages
It doesn't in practice. For example: 机 in Japanese means "table", while the same codepoint in Simplified Chinese means "machine" (i.e. 機 in Japanese).
It doesn't in practice. For example: 机 in Japanese means "table", while the same codepoint in Simplified Chinese means "machine" (i.e. 機 in Japanese).
For that example, in Chinese 机 is an alternative orthography of 几 (https://en.wiktionary.org/wiki/%E6%9C%BA#Etymology_2) as in e.g. 茶几 – it's not perfect, but language is messy and I don't think Unicode did the worst job of a messy situation.
"UniGreek" didn't happen because character encodings already existed that encoded Latin, Greek, and/or Cyrillic separately. Round-trip conversion is a non-negotiable requirement of Unicode, so merging any of those characters was impossible. There were no encodings that treated Japanese and Chinese simplifications of the same characters differently, hence why UniHan was allowed to happen. For similar reasons, we have twelve meaningless kanji in Unicode that exist purely as typographical errors during the standardization of Shift-JIS[0].
Furthermore, the primary motivation for UniHan wasn't to 'clean up' similar orthographies to avoid encoding homoglyphs. It was to stay inside 16 bits of coding space. Unicode was fighting a civil war against UCS, which proposed a 32-bit codepoint space, which would mean 32-bit characters, which the entire industry NOPE'd out of. Turns out, UCS was right, 16 bits was not enough, and the result is that every system that jumped on the Unicode train early[1] is now permanently cursed with improperly handling less-common kanji and most emoji.
For the record, I consider both UniHan and the hypothetical "UniGreek" a mistake. What characters get encoded in Unicode should match what speakers of a given language would consider distinct characters, not what we can arbitrarily merge to fit under a given coding bitrate. The only viable long-term solution for internationalized text was 32-bit codepoints encoded using a variable length encoding compatible with ASCII. We wound up with 20-bit codepoints, but I suspect at some point we'll need to break UTF-16 some more.
[0] This is known as "yurei moji" in Japanese
[1] Windows, Java, and JavaScript[2] all mishandle codepoints outside the 16-bit basic multilingual plane, which were encoded as pairs of 16-bit surrogate values in a special range of non-codepoints. Modern UTF-16 handling is supposed to treat these as single astral characters and reject broken surrogates, but the systems in question retain Unicode 1.0 era quirks for backwards compatibility.
A few years after the Unicode/UCS wars, Ken Thompson and Rob Pike would propose UTF-8, implementing it in Plan 9. This encoding was and is superior to UTF-16 in every possible way - including support for 31-bit code points, which UTF-16 surrogates can't do. UTF-8 isn't mangled by anything except the above programs... and MySQL, which had to add a second "no seriously it's UTF-8 for real" encoding.
[2] ActionScript inclusive. Yes, Ruffle has its own wide string library because of this.
Furthermore, the primary motivation for UniHan wasn't to 'clean up' similar orthographies to avoid encoding homoglyphs. It was to stay inside 16 bits of coding space. Unicode was fighting a civil war against UCS, which proposed a 32-bit codepoint space, which would mean 32-bit characters, which the entire industry NOPE'd out of. Turns out, UCS was right, 16 bits was not enough, and the result is that every system that jumped on the Unicode train early[1] is now permanently cursed with improperly handling less-common kanji and most emoji.
For the record, I consider both UniHan and the hypothetical "UniGreek" a mistake. What characters get encoded in Unicode should match what speakers of a given language would consider distinct characters, not what we can arbitrarily merge to fit under a given coding bitrate. The only viable long-term solution for internationalized text was 32-bit codepoints encoded using a variable length encoding compatible with ASCII. We wound up with 20-bit codepoints, but I suspect at some point we'll need to break UTF-16 some more.
[0] This is known as "yurei moji" in Japanese
[1] Windows, Java, and JavaScript[2] all mishandle codepoints outside the 16-bit basic multilingual plane, which were encoded as pairs of 16-bit surrogate values in a special range of non-codepoints. Modern UTF-16 handling is supposed to treat these as single astral characters and reject broken surrogates, but the systems in question retain Unicode 1.0 era quirks for backwards compatibility.
A few years after the Unicode/UCS wars, Ken Thompson and Rob Pike would propose UTF-8, implementing it in Plan 9. This encoding was and is superior to UTF-16 in every possible way - including support for 31-bit code points, which UTF-16 surrogates can't do. UTF-8 isn't mangled by anything except the above programs... and MySQL, which had to add a second "no seriously it's UTF-8 for real" encoding.
[2] ActionScript inclusive. Yes, Ruffle has its own wide string library because of this.
[deleted]
> Every European character of the same general design should be merged.
Won't combining Latin and Greek alphabets be horrible? Many formulas for starters, but also things like "β-radiation" (although I suppose b-radiation works too).
I don't really have strong opinions on Han unification as I don't speak those languages, but "Euro Unification" would seem like a right pain, and in general I feel Unicode is often too clever by half.
Won't combining Latin and Greek alphabets be horrible? Many formulas for starters, but also things like "β-radiation" (although I suppose b-radiation works too).
I don't really have strong opinions on Han unification as I don't speak those languages, but "Euro Unification" would seem like a right pain, and in general I feel Unicode is often too clever by half.
I think there are tradeoffs to each approach, and if it had gone the other way, there would be different problems For example characters for languages would be split up between blocks. Among other things this would make collation even more complicated and locale dependent than it already is. And people might complain that it is "western european elitism" that A is in the latin block instead of the greek or cyrrilic block.
Out of curiosity, what languages do you speak and what is your native language?
[deleted]
This is actually less problematic on a Mac and on iOS (if you speak the language) because the rendering engine actually prioritizes CJK set according to OS' language preferences (System Settings → Language & Region → Preferred Languages).
It's a much bigger issue on Linux, Android, and Windows prior to Windows 10, however.
It's a much bigger issue on Linux, Android, and Windows prior to Windows 10, however.
Unless, like me, you have both Japanese and Chinese IMEs installed :(
Ah yes, having both probably makes the prioritization fall apart :(
I have no idea if there's any sane way to make this work. Since I only speak Japanese, my fix on Linux is to prepend IPA PGothic/IPA PMincho (which only contains Japanese glyph) to lang=ja, which worked quite well.
I have no idea if there's any sane way to make this work. Since I only speak Japanese, my fix on Linux is to prepend IPA PGothic/IPA PMincho (which only contains Japanese glyph) to lang=ja, which worked quite well.
It works perfectly for me on Android. It even works with my phone's system language set to English, with Japanese as a fallback.
I haven't used Android since Nokia 7 Plus (back in 2019) but the variant that I had back then (TA-1062) doesn't have Japanese locale settings, despite being Android One and all that. I had to install MoreLocale2 and modify the settings via adb to make it work.
Every phone that was sold in the region I used to live (SEA) also lacks Japanese locale out-of-the-box. I had assumed this was still the case (locale availability tie to region being sold).
Every phone that was sold in the region I used to live (SEA) also lacks Japanese locale out-of-the-box. I had assumed this was still the case (locale availability tie to region being sold).
Interesting, I'm on the SM-G973U1. It always had a Japanese locale iirc. However, I can't recall which version of Android brought fallback locales. I don't remember if it was with this phone or my previous phone that they finally added that feature, but I'm sure it wasn't always a feature.
Just pasted in
刃直海角骨入
in Slack. And it shows the simplified Chinese version, like HN.
刃直海角骨入
in Slack. And it shows the simplified Chinese version, like HN.
Geany 2.0 and gedit 45.0 are rendering Simplified Chinese instead of Japanese. How to fix this?
It still shows a broken character for Japanese in Safari on iOS. Edit: fixed on reload.
I copy/pasted the example into iTerm Build 3.4.8 and got Simplified Chinese.
When copy pasted it here in this comment box (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0), I also got Simplified Chinese. What do you see?
刃直海角骨入
By the way, I do NOT have an Intel Mac, I have an Apple M1.
When copy pasted it here in this comment box (Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:120.0) Gecko/20100101 Firefox/120.0), I also got Simplified Chinese. What do you see?
刃直海角骨入
By the way, I do NOT have an Intel Mac, I have an Apple M1.
I think the point of the page was that unless the site/app/program is explicit about the language being displayed, you'll get Simplified Chinese. As HN is an English-language site, it uses the default. Similarly, your terminal doesn't prioritise Japanese over Simplified or Traditional Chinese.
By way of contrast, if you paste that into the search bar on https://yahoo.jp/ , you'll get the appropriate Japanese characters, as that site is explicitly Japanese.
By way of contrast, if you paste that into the search bar on https://yahoo.jp/ , you'll get the appropriate Japanese characters, as that site is explicitly Japanese.
Instead of explaining to me what I already know from reading the site and testing things out for myself, perhaps you could post some examples and experiences you've found.
Same codepoints. Software can only choose the right font if it knows it's in the Japanese language. I'm sure if you set your browser's language to Japanese it would select the right font, but since it's probably set to English instead, it's selecting a "default" assumption for the language of those glyphs.
App localization (where you specifically select "Japanese" as the language you want an app to display in) is the main target of this article, I think.
App localization (where you specifically select "Japanese" as the language you want an app to display in) is the main target of this article, I think.
Linguistic identification can in principle be a bit smarter than that. Aside from the obvious "if it has kana it is Japanese" metric, there are also differences in code point choice which differentiate Simplified, Traditional, and Japanese. Fully accurate identification is certainly expensive, but quick early-out heuristics could be developed.
This reminds me of guessing codepoint likelihood based on character distribution.
Interestingly I see the Japanese glyph on Apple M1. My locale is English-US and I am a native Chinese speaker.
When I paste it into VS Code it also renders Japanese, but when I click on the unicode code, it goes to https://symbl.cc/en/5203/ and https://symbl.cc/en/76F4/ which renders Chinese.
I am typing out the Chinese equivalent for reference: 刃直 but looks like they also turned into Japanese glyph for display past Chinese input method, in fact I can't type out the Chinese glyph in any apps on macOS.
It is not a big deal to me because I have no issue reading kanji as Chinese characters.
When I paste it into VS Code it also renders Japanese, but when I click on the unicode code, it goes to https://symbl.cc/en/5203/ and https://symbl.cc/en/76F4/ which renders Chinese.
I am typing out the Chinese equivalent for reference: 刃直 but looks like they also turned into Japanese glyph for display past Chinese input method, in fact I can't type out the Chinese glyph in any apps on macOS.
It is not a big deal to me because I have no issue reading kanji as Chinese characters.
[deleted]
I'm on a Apple Silicon Mac and I see Japanese, for what it's worth.
𝔗𝔬 𝔱𝔥𝔢 𝔍𝔞𝔭𝔞𝔫𝔢𝔰𝔢 𝔢𝔶𝔢, 𝔶𝔬𝔲𝔯 𝔱𝔢𝔵𝔱 𝔢𝔰𝔰𝔢𝔫𝔱𝔦𝔞𝔩𝔩𝔶 𝔩𝔬𝔬𝔨𝔰 𝔩𝔦𝔨𝔢 𝔱𝔥𝔦𝔰. ℜ𝔢𝔞𝔡𝔞𝔟𝔩𝔢, 𝔶𝔢𝔰, 𝔟𝔲𝔱 𝔴𝔦𝔱𝔥 𝔰𝔬𝔪𝔢 𝔰𝔦𝔤𝔫𝔦𝔣𝔦𝔠𝔞𝔫𝔱 𝔢𝔶𝔢𝔰𝔱𝔯𝔞𝔦𝔫 𝔞𝔫𝔡 𝔭𝔬𝔰𝔰𝔦𝔟𝔩𝔢 𝔠𝔬𝔫𝔣𝔲𝔰𝔦𝔬𝔫!
The article doesn't focus on it as much, but it's also just as much an issue for Taiwan vs Mainland Chinese script. There's a lot of subtle font differences, and traditional Chinese rendered in a simplified Chinese font looks very much out of place, even if the right hanzi code points are used.
The article doesn't focus on it as much, but it's also just as much an issue for Taiwan vs Mainland Chinese script. There's a lot of subtle font differences, and traditional Chinese rendered in a simplified Chinese font looks very much out of place, even if the right hanzi code points are used.
That kind of problem is much rarer, because Unicode has separate codepoints for traditional/simplified Chinese.
If only the article you're replying to had some example of characters that use the same code points for the same characters in Traditional and Simplified Chinese.
The codepoint mentioned in the article, U+5203, is a perfect example: in most of the world it will display in Simplified Chinese, whereas there's a counterpart U+2F81E that (in practice if not in theory) can be used to reliably display in Traditional Chinese. Only the Japanese rendering doesn't have a codepoint and is forced to use out-of-band methods of indicating the desired appearance.
IMEs do not and should not select U+2F81E, however. That core point exists merely for obscure Unicode compatibility requirements.
That's the Unicode consortium's opinion. Many people believe they're wrong about semantics and that Han unification was a mistake. Regardless of the Unicode consortium's intentions, these compatibility codepoints allow one to write text that displays Traditional Chinese correctly without additional out-of-band information - but there is sadly no equivalent for Japanese.
Except then you get those characters rendered as little squares for Unknown Char on a wide variety of platforms.
There’s good reason to stick to the basic multilingual plane.
There’s good reason to stick to the basic multilingual plane.
> you get those characters rendered as little squares for Unknown Char on a wide variety of platforms.
That can happen in the BMP too. It's a well-known problem with well-known solutions. (E.g. you can ship fonts that support all the codepoints you use).
That can happen in the BMP too. It's a well-known problem with well-known solutions. (E.g. you can ship fonts that support all the codepoints you use).
It isn't. The problem used to be default installed macOS with language set to English put Japanese first (as in CJK) in the language preferences list so they display Simplified Chinese / Traditional Chinese without lang tag wrong.
It looks like now Apple flipped the default to Simplified Chinese first maybe.
It looks like now Apple flipped the default to Simplified Chinese first maybe.
Only for some, because some source character sets had separate codes for them and Unicode had to guarantee a safe round-trip conversion ("source separation principle").
It’s called Unihan for a reason. Using different code points is the exception, not the rule.
Most of them.
Some have "minor" different with the same codepoint.
Some have "minor" different with the same codepoint.
How did you do that (with the font)?
Yeah, quite the hassle. Even the big boys like Apple screw it up[1]
[1] https://blog.frost.kiwi/joyo-kanji-unicode/
[1] https://blog.frost.kiwi/joyo-kanji-unicode/
Why would Unicode bother consolidating code points like this...? Not like it's short on space.
The consolidation effort would not have gone forward unless a significant number of people who have been studying these characters agreed that they were, in fact, the same characters.
Many Hanji/Kanji/Hanja characters have a long history of stylistic variation and simplification for the sake of aesthetics and convenience. For linguists and historians who are used to such variations, the direction of a minor stroke that doesn't alter the meaning would seem to be a purely stylistic choice, just as serifs on Latin characters don't alter the meaning. Some variations just happen to be more popular in some regions/countries/contexts and not others.
The general public, on the other hand, in each country is educated with the single "correct" variation favored by the government. Everything they read now uses the officially approved variation, so other variations look wrong. That stroke should absolutely not protrude to the other side, or you're a filthy barbarian!
The Unicode consortium seems to have listened more to the linguists and historians in this case. The academic stance doesn't always fit with public perception, which is often seasoned with a large pinch of nationalism.
Many Hanji/Kanji/Hanja characters have a long history of stylistic variation and simplification for the sake of aesthetics and convenience. For linguists and historians who are used to such variations, the direction of a minor stroke that doesn't alter the meaning would seem to be a purely stylistic choice, just as serifs on Latin characters don't alter the meaning. Some variations just happen to be more popular in some regions/countries/contexts and not others.
The general public, on the other hand, in each country is educated with the single "correct" variation favored by the government. Everything they read now uses the officially approved variation, so other variations look wrong. That stroke should absolutely not protrude to the other side, or you're a filthy barbarian!
The Unicode consortium seems to have listened more to the linguists and historians in this case. The academic stance doesn't always fit with public perception, which is often seasoned with a large pinch of nationalism.
> The consolidation effort would not have gone forward unless a significant number of people who have been studying these characters agreed that they were, in fact, the "same" characters.
The concrete term is the "normalization rules", which dictate how given arbitrary characters are transformed into domenstic variants. As far as I know most countries with significant Han character usages already had one before Unicode, and the ROK rules were (and still are being) developed alongside with Unicode.
The concrete term is the "normalization rules", which dictate how given arbitrary characters are transformed into domenstic variants. As far as I know most countries with significant Han character usages already had one before Unicode, and the ROK rules were (and still are being) developed alongside with Unicode.
Except, in context of unicode, Han-Unification Rules are very different from Normalization Rules.
Some background can be found in https://www.unicode.org/versions/Unicode1.0.0/V2ch02.pdf
Some background can be found in https://www.unicode.org/versions/Unicode1.0.0/V2ch02.pdf
I think you have mistaken "normalization rules" in this context with Unicode normalization algorithms. I'm specifically talking about documents like [1], which are essentially more detailed and concrete versions of the original Han-unification rules.
(Note that normalization rules themselves are distinct from the eventual unification. It takes further works to actually decide whether the unification is possible or not.)
[1] https://appsrv.cse.cuhk.edu.hk/~irg/irg/irg53/IRGN2420_KRNor...
(Note that normalization rules themselves are distinct from the eventual unification. It takes further works to actually decide whether the unification is possible or not.)
[1] https://appsrv.cse.cuhk.edu.hk/~irg/irg/irg53/IRGN2420_KRNor...
It was, in the old days, 16-bit, IIRC.
Ah yes. Just realized the author linked to its wikipedia page. Whoops.
[deleted]
yeah alphabeticall languages solves this problem by assigning different code points with the same looking character in different languages, which doesn't bring any problem at all..... except maybe domain phising. LMAO
Why is Unicode worried about having separate character sets? Modern computers have plenty of storage space.
If someone enters a form with these characters how can a site know what language to show it as. It makes no sense to work this way.
If someone enters a form with these characters how can a site know what language to show it as. It makes no sense to work this way.
Initially there was a desire to use only 16 bits for all of Unicode. This would allow for only 65K characters - so space was at quite a premium.
The jump from 8 bits to 16 bits effectively doubled memory costs and still wasn't enough to achieve its goal.
The jump from 8 bits to 16 bits effectively doubled memory costs and still wasn't enough to achieve its goal.
That was over 2 decades ago, they have had plenty of time to try and fix this.
Unicode was originally designed with the dream of fitting everything into 16 bits. CJK unification was a controversial measure that enabled this, by combining "the same" characters together in Chinese, Japanese, and Korean. Microsoft (wchar), Sun (Java), Netscape (JS) and many others settled on a 16bit fixed width "Unicode" encoding (nowadays often labelled as UCS-2).
Later it turned out that 16 bits wasn't enough anyway, but we are left with the UCS-2 mess regardless. UCS-2 and CJK unification are historical mistakes in the development of Unicode that we are stuck dealing with.
Later it turned out that 16 bits wasn't enough anyway, but we are left with the UCS-2 mess regardless. UCS-2 and CJK unification are historical mistakes in the development of Unicode that we are stuck dealing with.
Storage limitations from the 80's (as described by sibling comments) is one reason, but there's also another: which characters are considered their own graph (versus variants) is a somewhat fraught geopolitical issue, especially when China and Taiwan are on opposite sides of the argument. The final chapter of Yale professor Jing Tsu's Kingdom of Characters does a great job laying out how this manifests itself at meetings of Unicode working groups.
It wasn't designed for modern computers, inefficiencies also show as processing overhead, not just storage overhead, and in general software bloat can overcome any hardware progress
Ah yes, this. I once downloaded an android flashcard app with a Japanese card deck for studying kanji, and was a quarter of the way through when I noticed that the characters I was memorizing had been displaying incorrectly all along.
I had almost this exact issue in reverse when studying simplified hanzi.
I don’t know how you can expect English speakers to care about this sort of thing, when the vast majority of them don’t even pay attention to how their own language is displayed in software they develop.
Every organisation I’ve worked at meets attempts to correct apostrophes and curly quotes in their English copy with such an intense mixture of indifference, confusion, and annoyance, that anybody who tries is quickly convinced that they are the insane ones to pay attention to such inconsequential nonsense. Use of tabs vs spaces is easier to turn into a legitimate problem with business impact by comparison.
Every organisation I’ve worked at meets attempts to correct apostrophes and curly quotes in their English copy with such an intense mixture of indifference, confusion, and annoyance, that anybody who tries is quickly convinced that they are the insane ones to pay attention to such inconsequential nonsense. Use of tabs vs spaces is easier to turn into a legitimate problem with business impact by comparison.
"Curly quotes" are not somehow the "correct way" to write text. It is your preferred way, and that is something very different.
Does this mean that Han unification was a mistake?
What should software do when it accepts user content? What if a user wants to make a comment containing one quotation in Japanese and one in Chinese?
What should software do when it accepts user content? What if a user wants to make a comment containing one quotation in Japanese and one in Chinese?
The current answer is "use ideographic variation selectors [1] and hope that softwares catch up soon".
[1] https://en.wikipedia.org/wiki/Variant_form_(Unicode)
[1] https://en.wikipedia.org/wiki/Variant_form_(Unicode)
Huh I never knew about the Han Unification project -- seeing as I only know simplified/traditional Chinese, I'm lucky enough to not have noticed any incorrect rendering in my daily usage.
What's the intended solution for encoding text that uses Chinese and Japanese at the same time? For example, in an article discussing the differences between said texts? Is there some kind of hacky workaround, emoji colour selection style, or are you doomed to using pictures?
This also reminds me a lot of the early "auto-detect encoding" tools long ago that tried to guess the encoding based on frequency distributions. There's a good reason to not do that to unified codepoints for CJK, but I wonder if there's a more sophisticated backwards-compatible solution here.
Edit: looking at https://en.wikipedia.org/wiki/Variant_Chinese_characters , some of these differences are really splitting hairs; if you gave me a text with some of the simplified characters swapped out with some of these variants, I would honestly not be able to tell the difference. There's probably more variation in the Chinese fonts out there than I can gleam from the article. I'm not convinced that some of these characters have a non-artistic difference that a fellow Chinese on the street will be able to tell apart.
What's the intended solution for encoding text that uses Chinese and Japanese at the same time? For example, in an article discussing the differences between said texts? Is there some kind of hacky workaround, emoji colour selection style, or are you doomed to using pictures?
This also reminds me a lot of the early "auto-detect encoding" tools long ago that tried to guess the encoding based on frequency distributions. There's a good reason to not do that to unified codepoints for CJK, but I wonder if there's a more sophisticated backwards-compatible solution here.
Edit: looking at https://en.wikipedia.org/wiki/Variant_Chinese_characters , some of these differences are really splitting hairs; if you gave me a text with some of the simplified characters swapped out with some of these variants, I would honestly not be able to tell the difference. There's probably more variation in the Chinese fonts out there than I can gleam from the article. I'm not convinced that some of these characters have a non-artistic difference that a fellow Chinese on the street will be able to tell apart.
> “What's the intended solution for encoding text that uses Chinese and Japanese at the same time?”
For web documents, you can explicitly mark the language and let the browser pick the correct font. Can be applied to any element:
For web documents, you can explicitly mark the language and let the browser pick the correct font. Can be applied to any element:
<span lang=“ja”>…</span>CJK project is just a joke
So let's split CJK? Unicode is a joke anyway. It should be renamed to Multicode.
Did Unicode consider using semantic language markers? Those could even help unify some latin alphabets and save you even more codepoints, e.g., you could have the same codepoints for Greek and English:
start Greek, codepoint 'a' .... end Greek: 'a' wold be 'α' start English, codepoint 'a' ... end English: 'a' would be 'a'
Could even have the same codepoint for Cyrillic alphabets
start Greek, codepoint 'a' .... end Greek: 'a' wold be 'α' start English, codepoint 'a' ... end English: 'a' would be 'a'
Could even have the same codepoint for Cyrillic alphabets
> Did Unicode consider using semantic language markers?
Once upon a time Unicode had inline language tags [1].
[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)#Legacy_us...
> Those could even help unify some latin alphabets and save you even more codepoints, [...]
This is a common misunderstanding; they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified.
Once upon a time Unicode had inline language tags [1].
[1] https://en.wikipedia.org/wiki/Tags_(Unicode_block)#Legacy_us...
> Those could even help unify some latin alphabets and save you even more codepoints, [...]
This is a common misunderstanding; they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified.
Thanks, pity it got poorly designed and cancelled, do you know of a good document/criticism re. why that happened? The Wikipedia linked memos are process garbage docs where copyright header/footer with authors' addresses take more space than the (lack of) substance
> they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified.
Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints? Besides, the codepoint could be "first letter of a Language alphabet" instead of "a letter that looks like A"
> they may look alike but can and often do have different rules (e.g. ordering) underneath, so they have to be disunified.
Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints? Besides, the codepoint could be "first letter of a Language alphabet" instead of "a letter that looks like A"
It's not exactly "cancelled" per se, and you need to look at the correct place (which indeed takes time to learn...).
At the beginning there was an Internet-Draft [1] proposing the use of invalid UTF-8 sequences, with an initial intention to be used in the Application Configuration Access Protocol (RFC 2244). Take a look at that---it is really horrible if you think about that. Nevertheless the proposal was significant because it demonstrated needs for "lightweight language tagging" and a more reasonable proposal was developed in the Unicode Technical Committee [2], which resulted in the current Tags block.
Since they were meant to be used only when there is no other markup method, though, it was "born 'strongly discouraged'" from the beginning [3]. The current "deprecated" status is rather a formal confirmation of this position---[3] even clarifies that they are "indicated as 'strongly discouraged', but reserved for use with special protocols", and Unicode never removes once allocated code points, so Unicode can't do anything more about deprecated characters. Still, the intent is that you should use correct `lang="..."` attributes whenever applicable.
[1] https://datatracker.ietf.org/doc/html/draft-ietf-acap-mlsf-0...
[2] https://www.unicode.org/L2/L1997/97255r.pdf#page=4
[3] https://www.unicode.org/L2/L2001/01301-deprecation.txt
> Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints?
That means you need to first locate all language tags before doing anything, which is wasteful. Almost all Unicode operations are designed to function with only local contexts in comparison.
At the beginning there was an Internet-Draft [1] proposing the use of invalid UTF-8 sequences, with an initial intention to be used in the Application Configuration Access Protocol (RFC 2244). Take a look at that---it is really horrible if you think about that. Nevertheless the proposal was significant because it demonstrated needs for "lightweight language tagging" and a more reasonable proposal was developed in the Unicode Technical Committee [2], which resulted in the current Tags block.
Since they were meant to be used only when there is no other markup method, though, it was "born 'strongly discouraged'" from the beginning [3]. The current "deprecated" status is rather a formal confirmation of this position---[3] even clarifies that they are "indicated as 'strongly discouraged', but reserved for use with special protocols", and Unicode never removes once allocated code points, so Unicode can't do anything more about deprecated characters. Still, the intent is that you should use correct `lang="..."` attributes whenever applicable.
[1] https://datatracker.ietf.org/doc/html/draft-ietf-acap-mlsf-0...
[2] https://www.unicode.org/L2/L1997/97255r.pdf#page=4
[3] https://www.unicode.org/L2/L2001/01301-deprecation.txt
> Don't get it, you'd have different rules apply depending on the language tags, why would you need multiple codepoints?
That means you need to first locate all language tags before doing anything, which is wasteful. Almost all Unicode operations are designed to function with only local contexts in comparison.
I guess some questions may have been answered in the previous HN submission: https://news.ycombinator.com/item?id=29022906
I haven't said much about my personal feeling in that discussion, but honestly speaking as a Korean, I do feel that some Japaneses weigh too much on exact glyphs. These variant glyphs are indeed "wrong", but so does many non-standard shorthands (e.g. ⿸广マ as a shorthand of 魔) or individual preferences (e.g. the first character of Yoshinoya 𠮷野家 is an unusual variant of 吉). Why some are okay while others aren't? I'm yet to hear plausible justifications for this disparity.
I haven't said much about my personal feeling in that discussion, but honestly speaking as a Korean, I do feel that some Japaneses weigh too much on exact glyphs. These variant glyphs are indeed "wrong", but so does many non-standard shorthands (e.g. ⿸广マ as a shorthand of 魔) or individual preferences (e.g. the first character of Yoshinoya 𠮷野家 is an unusual variant of 吉). Why some are okay while others aren't? I'm yet to hear plausible justifications for this disparity.
вє¢αυѕє ιт ƒєєℓѕ ℓιкє яєα∂ιηg єηgℓιѕн тєχт тнαт ℓσσкѕ ℓιкє тнιѕ! It might still be legible, but in some contexts (e.g. official) it's reasonable to want your language to be accurately represented.
Not to mention sometimes you also get kana displaying in a Japanese font and kanji displaying in a stylistically different Chinese font and it looks awful.
Not to mention sometimes you also get kana displaying in a Japanese font and kanji displaying in a stylistically different Chinese font and it looks awful.
Awful at worst, I should say. And as others said, correct uses of `lang` attributes and system-default preferences alleviate most stylistic issues. It looks awful only when applications have a mismatching preference (e.g. Japanese websites that never think about Chinese users).
广+マ can stand for more than one character. That's the fatal problem with CJK unification in general. It takes the correctly entered language character X and replaces it with CJK character Y and then everyone gets confused
⿸广マ is indeed a shorthand for multiple characters due to its phono-semantic construction, but many simplification attempts also had similar ambiguities. ⿸广マ in this sense is just a new character that has multiple meanings from the context. Z-variants merged by the CJK unification are much more subtle than this example.
My point is the same. Your text should be recorded and displayed as you entered it, especially when that can lead to confusion.
> , but many simplification attempts also had similar ambiguities.
No one asked to have our languages changed by Unicode
> , but many simplification attempts also had similar ambiguities.
No one asked to have our languages changed by Unicode
The first Japanese character set was JIS X 0201, which only had half-width katakanas. It is absurd in the modern standard, lacking all hiragana and kanji, and looks really awful (so much that they are used for comical effects today). I'd argue that JIS X 0201 is an extremely radical attempt compared to the Han unification---so why should the Han unification be blamed alone?
My apologies for having higher expectations of computers in 2023 than computers from the 1960s
That doesn't sound fair to me given that the Han unification occurred around 1990...
I think the article doesn’t provide enough arguments why this is even an issue.
For example what is the probability of such character being rendered incorectly in some standart tex? lets say a wikipedia article.
Even more so the argumet that people don’t report this because they are "not speakers of English!” is just an assumption. Not to mention that translation applications are more than good enough for such a task.
For example what is the probability of such character being rendered incorectly in some standart tex? lets say a wikipedia article.
Even more so the argumet that people don’t report this because they are "not speakers of English!” is just an assumption. Not to mention that translation applications are more than good enough for such a task.
>Even more so the argumet that people don’t report this because they are "not speakers of English!” is just an assumption. Not to mention that translation applications are more than good enough for such a task.
Frankly, people have learned helplessness[0] about these oddities and don't think to report them when they see them, so the inference that something isn't serious just because it's not pointed out is weak.
In the first place, the proportion of software users who raise issues on GitHub/other is small, and when devs are a group of people who communicate in characters that are not used in their daily life, the translation apps they have at hand is not very encouraging.
[0]: https://en.wikipedia.org/wiki/Learned_helplessness
(Disclosure: I'm CJK native)
Frankly, people have learned helplessness[0] about these oddities and don't think to report them when they see them, so the inference that something isn't serious just because it's not pointed out is weak.
In the first place, the proportion of software users who raise issues on GitHub/other is small, and when devs are a group of people who communicate in characters that are not used in their daily life, the translation apps they have at hand is not very encouraging.
[0]: https://en.wikipedia.org/wiki/Learned_helplessness
(Disclosure: I'm CJK native)
The issue is that the language is not accurately represented – imagine in English instead of the Latin letter "a" you see the Greek letter "α". It's still legible but it's not unreasonable to ask for an accurate depiction of a language.
The letter “a” has a frequency of ~8%, and it would indead be anoying, but lets say the letter “q” is rendered incorrectly which has a frequency of ~0.1% then thats just some minor issue.
While in a practical sense your argument might hold water (which I doubt, to be honest), this is not just a practical matter. This is also a matter of respect. If you think that mis-representing q is not a problem because it appears so less, then do you also think that disregarding the religious tenets of minorities is fine since they make up such a small part of the population?
I don't know why we shouldn't strive to have computers to reproduce language precisely, since computers are how we communicate most of the time and how the majority of content ends up being preserved. After all, maybe we shouldn't bother with spelling in English either, since everyone can safely approximate the meaning anyway!
Another fun complication with this is that ttf, otf, and woff all have a limit of 2^16 glyphs. Which isn't enough to include separate glyphs for all characters in all three locales, especially if you include ligatures. This is why some fonts, such as Noto have separate font files for Chinese, Japanes, and Korean.
It can also be problematic if you want to include japanese and chinese ideographs in the same document.
It can also be problematic if you want to include japanese and chinese ideographs in the same document.
As a Chinese reading and writing Japanese everyday, they all LGTM
Speaking as someone who reads Chinese and Japanese, this is not as annoying a problem as when a font designed for, say, Chinese is used to display Japanese and does not have the native-Japanese character (say, 働). Europeans using a different Latin script than a subset of English will be intimately familiar with what I am talking about.
I don't think that Han unification itself is too problematic: every variant is attested to some extent in handwriting in all these societies. Furthermore, the web designer conscientious enough will
1. For applications meant to be displayed in a particular language, use a font for that language, in which case the lang-untagged content always appears right
2. For multilingual content ensure that lang tags and fonts are set up right.
If the owner fails to do this, fallback font usage is going to be just as great a problem as variant issues.
I don't think that Han unification itself is too problematic: every variant is attested to some extent in handwriting in all these societies. Furthermore, the web designer conscientious enough will
1. For applications meant to be displayed in a particular language, use a font for that language, in which case the lang-untagged content always appears right
2. For multilingual content ensure that lang tags and fonts are set up right.
If the owner fails to do this, fallback font usage is going to be just as great a problem as variant issues.
The main thing I noticed, wasn't the whole point of Unicode to be a single character set that could work for all languages simulateously? If the same code point is used for multiple characters in different languages, then aren't we pretty much back to code page era?
The lack of this causes so many problems and makes so many things that could be easy so much more difficult.
Why do English, Spanish, German and Italian all share the same homogylhs, even though German has for instance has ß among others, Spanish has ñ, and Italian completely lacks 5e letters JKWXY. Greek or Russian though? Completely different character sets despite containing many of the same glyphs. That to me seems like “Western European Elitism”
There are of course special cases where for instance the upper and lower characters don’t match across languages, but they can be resolved individually rather than as a full set.