Unitools – A suite of tools for working with Unicode in the browser(unicod.es)
unicod.es
Unitools – A suite of tools for working with Unicode in the browser
https://www.unicod.es/
58 comments
> Just text is fine, thank you!
Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text.
It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.
Unicode is text, that's the whole point. "ᐃᓄᐃᑦ ᐅᖃᐅᓯᓕᕆᔨᐅᑉ. ᐃᓄᑦᑎᑑᖅᐳᑦ ᓄᓇᕗᒻᒥ, ᓄᓇᕕᒻᒥ, ᐊᑯᑭᑦᑐᒻᒥ, ᓄᓇᑦᓯᐊᕗᒻᒥ, ᓄᓇᑦᓯᐊᕐᒥ' ᐊᒻᒪᓗ ᐊᓛᓯᑲᒥ" is text, "한글 또는 조선글은 1443년 조선 제4대 임금 세종이 훈민정음(訓民正音)이라는 이름으로 창제하여 1446년에 반포한 문자로, 한국어를 표기하기 위해 만들어졌다.[1][2] 이후 한문을 고수하는 사대부들에게는 경시되기도 하였으나, 조선 왕실과 일부 양반층과 서민층을 중심으로 이어지다가 1894년 갑오개혁에서 한국의 공식적인 나라 글자가 되었고, 1910년대에 이르러 한글학자인 주시경이 '한글'이라는 이름을 사용하였다. 갈래는 표음 문자 가운데 음소 문자에 속한다. 한국에서는 한글전용법이 시행되고 있다" is text, "દેવનાગરી એક લિપિ છે. સંસ્કૃત દેવનાગરી લિપિમાં લખાતી આવી છે. દેવનાગરી લિપિ મૂળ તો સંસ્કૃત માટે જ બની છે, એટલે એમાં દરેક ચિન્હ માટે એક અને માત્ર એક જ ધ્વનિ છે. દેવનાગરીમાં ૧૨ સ્વર અને ૩૪ વ્યંજન છે" is text, and "Алфавиты на основе кириллицы являются или являлись системой письменности для 108 естественных языков, включая следующие славянские языки" is text.
It may be bothersome text I don't understand and you can't be arsed to handle, but it's text nonetheless.
You know, if you give me some ASCII I have a pretty good idea how to interpret it.
For example, I can easily verify that it has no control codes, or is "plain text".
There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
For Unicode, I don't even know where to look up how to do it.
Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.
For example, I can easily verify that it has no control codes, or is "plain text".
There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
For Unicode, I don't even know where to look up how to do it.
Then there are these special encodings in UTF-8 to shelter binary data which failed to decode. Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII (or opaque 8-bit data handled as a black box) because it gives so much more meaning to the things you do with it.
> if you give me some ASCII I have a pretty good idea how to interpret it. For Unicode, I don't even know where to look up how to do it.
That's not because Unicode is not "just text" whereas ASCII is.
It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain".
We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding).
(I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.
That's not because Unicode is not "just text" whereas ASCII is.
It's because "text" is genuinely very complicated and nuanced, and that's the nature of the problem domain. Text is messy. Text is complicated. Text is many things, but text is not "plain".
We should also be careful to distinguish between criticisms of Unicode and criticisms of Unicode encodings. They're related, but not all encodings share the same problems (and some problems are baked into Unicode itself, and can't be solved by a novel encoding).
(I have plenty of complaints about Unicode and opinions about the various Unicode encodings, but the fact remains that there is no one-size-fits-all approach to text which can completely avoid or hide this complexity.
I totally agree that text is complicated. But that just means it's a black box and I can't do anything meaningful to it other than copying it around.
Unless I read the Unicode spec and expect my inputs to also have read it.
Unless I read the Unicode spec and expect my inputs to also have read it.
> But that just means it's a black box and I can't do anything meaningful to it other than copying it around.
It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji).
And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
It means you need some out-of-band context about the domain. The things I can expect do with text if I know I'm parsing C source code is very different from the things I can expect to do with text if I'm parsing Twitter data (especially if the tweets are geotagged to a region that doesn't speak English). Or, if I'm parsing SMS message data (which are likely to contain lots of Unicode emoji).
And quite frankly, pretty much any task involving processing arbitrary input text already has so many complicating factors that have nothing to do with Unicode. There's already pretty much nothing interesting you can do to arbitrary text besides copy it around if all you know is the character encoding. Every real-life problem space has more concrete assumptions about the data you're dealing with, so including assumptions about the character encoding and characters used in that set is not terribly burdensome.
Absolutely. And my go-to context is ASCII whenever it's convenient. This is the only context where I have a clear idea what's inside it.
> You know, if you give me some ASCII I have a pretty good idea how to interpret it.
> For example, I can easily verify that it has no control codes, or is "plain text".
> There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> For Unicode, I don't even know where to look up how to do it.
That's not a problem with unicode.
> Then there are these special encodings in UTF-8 to shelter binary data which failed to decode.
There are no special encodings in UTF-8. What you're talking about is error-handling strategies
> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
The exact same question exists about ASCII, since \xFF is not valid ASCII.
> Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
It's painful that your random manglings don't make sense?
> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
They don't.
> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
Which is already an issue in ASCII.
> So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII
It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did.
And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
> For example, I can easily verify that it has no control codes, or is "plain text".
> There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> For Unicode, I don't even know where to look up how to do it.
That's not a problem with unicode.
> Then there are these special encodings in UTF-8 to shelter binary data which failed to decode.
There are no special encodings in UTF-8. What you're talking about is error-handling strategies
> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
The exact same question exists about ASCII, since \xFF is not valid ASCII.
> Then there are these misplaced things like BOM or strange switching codes which affect the interpretation of the following text. I don't even know how I can interpret a string slice in a meaningful way because of this. It's painful.
It's painful that your random manglings don't make sense?
> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
They don't.
> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
Which is already an issue in ASCII.
> So to wrap it up, yes it is absolutely legitimate to restrict yourself to ASCII
It is, for low values of legitimate. But don't call it text, and more importantly don't call non-ascii content "not text" which is what the person I replied to did.
And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
Sigh, thank you for ranting for me.
I just want to add to what you said:
> > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> > For Unicode, I don't even know where to look up how to do it.
UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4].
In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document.
1. http://www.unicode.org/reports/tr14/
2. http://unicode.org/reports/tr29/
3. http://site.icu-project.org/
4. http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)
I just want to add to what you said:
> > There is not much choice how to split a string into words (split on 0x20? yes. split on 0x0a? mostly. split on 0x09? maybe. allow carriage return? maybe.).
> > For Unicode, I don't even know where to look up how to do it.
UAX #14 and #29 specify perfectly well how to do this[1][2]. There are even C libraries to do this for you[3][4].
In general, if you have a Unicode question, the standard clears things up reasonably well. It's somewhat easy reading for a standardization document.
1. http://www.unicode.org/reports/tr14/
2. http://unicode.org/reports/tr29/
3. http://site.icu-project.org/
4. http://julialang.org/utf8proc/ (Not Julia specific; it's used to implement Unicode support in Julia and is maintained as part of the Julia project. It's a very nice clean C library.)
And what can I do if I don't want to spend the next two months implementing this line breaking algorithm which probably no two implementations did in the same way? (exaggerating a little here).
I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up.
I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y.
I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not.
I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.
I just want to split my text into lines and fields, without second guessing if I did it right. And I want to be confident about interoperability. This means it has to be simple. If it's not simple then it's a sure thing that one of the communication partners screws up.
I need a simple answer how I can compare two strings for certain properties, and not in a way which works in locale X but not in locale Y.
I often don't want a thousand ways to express (almost) the same thing. This is for example quite a headache in practice when you search text. Worse if you search it in a pdf where you typically deal with ligatures and what not.
I never said things are unspecified in Unicode. That's not my point. I think the things you and I want to do with text are just very very different.
OK, wait, what? Of course two implementations will implement line breaking the same way, it's _specified_ how they should do it. That's the whole point of a spec.
ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.
ICU and libutf8proc will both break lines for you. They exist specifically to provide implementations of the various Unicode algorithms. They'll also tell if you two character sequences can be considered identical. ICU even provides regexes where ft will match the ligature.
There is no such one thing as "breaking lines" in Unicode. There are a thousand slightly different ways to do it, in every single locale. That is explicit in the introduction to the first beast you linked yourself...
And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic.
Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.
And assuming that a spec (especially a super-complex one, that moreover constantly evolves), is devoid of ambiguities, and makes every implementation behave in the same way, or even that every implementation implements the largest part of it, is being a tad idealistic.
Sometimes it's all about common sense. For most tasks I will happily continue using strchr or my manual loop and be done with it, thanks.
>> to split a string into words [...] For Unicode, I don't even know where to look up how to do it.
> That's not a problem with unicode.
So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider.
>> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
> The exact same question exists about ASCII, since \xFF is not valid ASCII.
Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible.
> It's painful that your random manglings don't make sense?
If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks).
>> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
> They don't.
They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong?
>> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
> Which is already an issue in ASCII.
Some common sense helps here. Show me what serious fonts don't support the ASCII print range.
> And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
I'm totally with you.
> That's not a problem with unicode.
So how do I know or make a reasonable decision what's "whitespace" in Unicode? There isn't just these few characters <= 0x20. http://www.fileformat.info/info/unicode/category/Zs/list.htm lists just 17 characters, which does not include newline, tab or CR. And maybe the list will be expanded over time? I'm also not sure if there are other categories or special cases which I might have to consider.
>> Did you ever do a 'sed /\xFF/d' and asked yourself how the hell this is interpreted (in a UTF-8 locale)?
> The exact same question exists about ASCII, since \xFF is not valid ASCII.
Read that as C locale. The point is, there are not many ways that can be handled, and consequently not much surprise is possible.
> It's painful that your random manglings don't make sense?
If you consider basic and practical things, like interpreting input as lines and words/fields, random manglings, then yes. But there are actually many applications that can profit from a "free monoid" property (i.e. the concatenation operation being associative, without "chemical reaction" between the concatenated chunks).
>> Then there are these combining code points which repeat the UTF-8 ceremony at a higher level of abstraction.
> They don't.
They are only meaningful as a combination sequence, and otherwise there is semantic illness, or am I wrong?
>> Then there are so many characters where you just can't know how they will be displayed (if at all) at the receiving side.
> Which is already an issue in ASCII.
Some common sense helps here. Show me what serious fonts don't support the ASCII print range.
> And if you think dealing with unicode is troublesome, for your sake stay far away from anything having to do with localisation, unicode is the basement level of localisation issues.
I'm totally with you.
Have you considered that perhaps the fault is not in your stars but in yourself? (That is, a thing is not inherently negative just because you do not care to understand it.)
For some context, read the top level OP. Of course it's not inherently negative.
For internal use, OK, fine. But what part of the following is not "just text"? "Don’t be naïve. £100 is a lot more than €100—and Hélène knows you know that."
While you can't represent that as plain ASCII, I think Windows-1252, an 8-bit single-byte encoding, can handle that quite well.
That's an excellent point. Let's generalize that Windows-centric idea of a code page to something that can handle most/all known languages: a Universal Coded character set, if you will.
I guess we can get some arbitrary text, then check whether all its characters can be encoded as Windows-1252, store it with a note that it is in Windows-1252, and then check that note before decoding it. Clearly so much better than UTF-8.
They tried: https://en.wikipedia.org/wiki/ISO/IEC_2022
It was... horrible.
It was... horrible.
There is an art in discovering which encoding the text vs. represented by application, by making sense of nonsense.
https://jeffquast.com/soviet-encodings.jpg
https://jeffquast.com/soviet-encodings.jpg
Sure I mean who wants to visit Госуда́рственный Эрмита́ж or eat 삼겹살? Moonspeak's not really text right?
I'd argue that there's no meaningful difference between ASCII text and UTF-8 that just happens to not use any characters higher than '~'.
As long as there's no BOM, anyway.
As long as there's no BOM, anyway.
s/meaningful//
Any ASCII text is ipso facto UTF-8 text.
Any ASCII text is ipso facto UTF-8 text.
If you consider representation, yes. If you consider semantics, no.
Can you explain?
UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII.
Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
UTF-8 was deliberately designed to be "backwards compatible" (for want of a better phrase) with ASCII.
Absent a BOM mark, there is no way to tell the difference between a piece of ASCII text on the one hand and, on the other, a piece of UTF-8 text that uses no code point above U+007f. How do they differ semantically?
Loading it from disk, there's no way to tell the difference, because the representation is the same and a selection of bytes are a selection of bytes.
Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc).
The difference isn't the representation; the difference is the interpretation that we attach to the representation.
Say you read the text into a program. You can load the text into an AsciiString variable or a UnicodeString variable, and it's valid in either of those cases. They have the same binary representation...but they're semantically different. They each have properties that the other one doesn't (ASCII has random access, Unicode is linear access, either one could have an ASCII string appended to it, but only the Unicode one could have a Unicode string appended, etc).
The difference isn't the representation; the difference is the interpretation that we attach to the representation.
Yes. I already tried to explain with an analogy about faithfulness (which I removed because it got many downvotes).
When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode).
When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning.
Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
When you deal with ASCII, that's also the semantic level on which you operate. Much simpler than Unicode (I don't think there are many people who really understand Unicode).
When you have a Unicode string that uses only characters that are also present in ASCII, that doesn't make it an ASCII string. It has more meaning.
Btw that has nothing to do with BOM. BOM is itself not part of ASCII, and not encoded as a single byte, and it doesn't affect single-byte sequences.
Are you talking about how, when someone is limited to ASCII, he makes substitutions he wouldn't make if he were using Unicode? Such as using :-) instead of U+263A?
Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII?
Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
Or are you maybe talking at a software level, where you can make some simplifications and assumptions about a string when it is guaranteed to be ASCII?
Because otherwise I can't see how there could be any semantic or meaningful difference between ASCII and Unicode when the text is the same. An ASCII 'a' has exactly the same meaning as a Unicode 'Latin Small Letter A'.
You can look at it from an information theoretic standpoint. The information you get from the string is also all the things that it does not contain.
Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.
Btw. I have no clue why I am constantly downvoted. Whoever did this, I don't think there is an incentive to downvote my two ancestor comments. It's demotivating.
Didn't downvote but I can see why they got downvoted: they make a claim, but when asked for a specific argument, they go roundabout.
In what SPECIFIC way is an only-ascii string not a UTF-8 string for ALL intends and purposes?
You throw some words like "semantics" but no actual arguments.
In what SPECIFIC way is an only-ascii string not a UTF-8 string for ALL intends and purposes?
You throw some words like "semantics" but no actual arguments.
I think I answered that in the clearest way I (personally) could, right in the comment to which you replied.
But wherein does that meaning lie? If the two strings represent the same text, you can't tell the difference between them, because there is none.
(I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
(I guess technically you can't tell the difference between an ASCII string and a string of random 8-bit numbers in the range 0 <= x <= 127, but if when represented as ASCII characters those numbers are meaningful -- comprehensible English, for example -- the odds are good it's the former not the latter.)
I think what he's getting at is that whatever program is reading the text must be prepared to read UTF-8 and not just ASCII. Sure, the underlying data is the same, but the data type is not the same; analogously, a byte with the decimal value of 65 might be the number 65 if cast as an integer, but might be the character 'A' if cast as a char.
[deleted]
> Just text is fine, thank you!
And Unicode has a non-negligible footprint.
I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU).
Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ...
I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.
And Unicode has a non-negligible footprint.
I tried few times to handle that in C or C++ and everything becomes complicated (even with a library like ICU).
Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ? - ...
I read Java strings are switching to ASCII (or 16b) because it's too much inefficient.
I think most software (like programming languages) has actually switched to UTF-8 for internal representation, because it has low footprint (for most applications).
You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
You give up efficient O(1) indexing, but I don't think you typically need that. Indexing is far less meaningful for Unicode compared to ASCII.
> Each time,I had to ask myself: - What's the encoding ? - How to detect it ? - What's the collation ?
You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into.
Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
You always have to ask yourself that if you deal with text, otherwise your application will most likely be broken. Heck, Unicode actually made this easier because you can use it as an internal encoding to convert everything else into.
Yes, Unicode has overhead and complexities, but these are because text and language is complicated.
If you're dealing with text from outside you have to handle encodings whether you like it or not. (I mean you can blindly copy bytes if you like, but you end up with corrupt data that you have no idea what it means, and may ultimately have to spend months or years recovering that information. I'm speaking from experience here).
Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
Internally it very rarely matters what representation you use as long as you know what it is - pick one particular representation to use for all your internal functions and stick to it and you're fine. You shouldn't need to do any detection except at the boundaries, and efficiency concerns are usually overblown.
I've been practicing this approach for awhile with great success: http://utf8everywhere.org/
> ... This document also recommends choosing UTF-8 for internal string representation in Windows applications ...
I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now.
http://mingw-w64.org/doku.php
I used mingw32-w64 to cross-compile for Windows (with iconv) and I can say it works great now.
http://mingw-w64.org/doku.php
An article I always recommend on this topic:
"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky
http://www.joelonsoftware.com/articles/Unicode.html
"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)" by Joel Spolsky
http://www.joelonsoftware.com/articles/Unicode.html
Getting Unicode wrong can kill.
http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two...
> The use of "i" resulted in an SMS with a completely twisted meaning: instead of writing the word "sıkısınca" it looked like he wrote "sikisince." Ramazan wanted to write "You change the topic every time you run out of arguments" (sounds familiar enough) but what Emine read was, "You change the topic every time they are fucking you"...
http://gizmodo.com/382026/a-cellphones-missing-dot-kills-two...
> The use of "i" resulted in an SMS with a completely twisted meaning: instead of writing the word "sıkısınca" it looked like he wrote "sikisince." Ramazan wanted to write "You change the topic every time you run out of arguments" (sounds familiar enough) but what Emine read was, "You change the topic every time they are fucking you"...
From the article, it sounds as if that wasn't the only factor...
Besides the obvious batshit craziness of those involved, that's not unique to unicode.
After all, the turkish encoding wasn't ASCII in the DOS/Unix years but ISO 8859-9 (and various other, e.g. an IBM one etc).
After all, the turkish encoding wasn't ASCII in the DOS/Unix years but ISO 8859-9 (and various other, e.g. an IBM one etc).
Despite the tongue-in-cheek tagline on the site, these are some really neat tools.
I agree! It's always disappointing when an invitation to contest a controversial statement outweighs the information they were actually trying to relay. Would love to seem more comments and discussion on clever applications of some of these tools.
Heck even EBCDIC isn't dead. Nothing dies forever as much as you'd like it to.
Embedded systems would like to talk with you ...
Doesn't translate "é" as "&eactue;" ("é" instead).
It generally changes characters to numerical character references, not HTML character entities (which, incidentally, are a rather limited set of characters and pretty much useless these days – they were useful to include a few non-ASCII latin characters from Latin 1 in the mid-90s into HTML pages with browser charset detection messing things up if included directly; by now you just use Unicode and forget that a named set of entities ever existed).
>...by now you just use Unicode and forget that a named set of entities ever existed...
OK.
When I was in PHP, I often used "htmlspeciachars()" and it does this kind of transform. That's why I tried that.
http://php.net/manual/en/function.htmlspecialchars.php
I had some problems too with " " because it's 160 in ASCII and not a whitespace (32).
Edit: 160
OK.
When I was in PHP, I often used "htmlspeciachars()" and it does this kind of transform. That's why I tried that.
http://php.net/manual/en/function.htmlspecialchars.php
I had some problems too with " " because it's 160 in ASCII and not a whitespace (32).
Edit: 160
The non-breaking space is certainly not 256 in ASCII (ASCII only extends to 127) and not even in Latin 1 (goes up to 255) and also not in Unicode – it's 160. While you can use in HTML you can also use   or  , both of which work in XML as well. And since numeric and hexadecimal character references are much easier to generate (you don't need a lookup table) I'd say they're a far nicer choice these days.
> ... in HTML you can also use   or   ...
Yes! You're right.
>... I'd say they're a far nicer choice these days.
It was a long time ago for a scrapping project (garbage HTML).
Yes! You're right.
>... I'd say they're a far nicer choice these days.
It was a long time ago for a scrapping project (garbage HTML).
[deleted]
ASCII 4 Life!
Unicode is ideal for storing text exactly as the user wanted it. Which may be crazy.
But when I am writing a program for internal use, or creating a communication protocol, or trying to parse one program's output into another, or trying to make a 7-segment display show some characters, I don't want to have to handle these crazy possibilities. Just text is fine, thank you!