Down with Unicode! Why 16 bits per character is a right pain in the ASCII

Cardinal sin of computing

The fact that some programmer, in an attempt to show the "benefit of Unicode", should use a 'double' variable for PI and only give 6 figures tells you they should be executed and their programs not!

But yes, you speak the truth - UTF-8 is better for all practical reasons because it won't break old software/code and yet it allows all characters you (and your customers/users) might want. Subject to matching system fonts - a rant for another day...

For the love of (deity), make your code uses UTF-8. The world has decided. Text files are UTF-8, websites are (should be) UTF-8, emails should be UTF-8 and SMSes are UTF-8. So, use UTF-8.

Good idea

I use UTF-8 for everything (some Welsh characters aren't supported in the usual European sets) - but could someone please give Microsoft a good slapping - just wasted ages trying to get data containing Welsh characters (ŵ and ŷ - see, el Reg can handle them) from an Excel spreadsheet via csv into a mySQL DB - nightmare! Excel output to csv can't do UTF-8. I ended up pasting into OpenOffice, then exporting.

Would be really interesting if computers were invented in China.

Wonderfull.

Why did no-one point out to me earlier that Unicode isn't 16 bits any more?

I had noticed however that if you wanted Web Sites, Browsers & SQL all to talk nice that UTF-8 was best.

Java

The Java devs could have saved themselves a lot of bother by just calling "char" an unsigned short and being done with it. Now they just look silly by calling a character a "codepoint" and a utf-16 unit a "char".

Mind, their punishment was to be swallowed whole by Oracle. Rough justice.

Got to keep it

If every character needs 16 bits rather than 8 then the NSA will only be able to store half as much of our stuff

Machine code £

The comment about printing the £ (I hated the # sign) reminded me of my DOS days. I solved it by writing a small 90 byte machine code routine (most bytes were my credit line) that loaded through config.sys that redirected the print code to see the £ code rather than the hash code. Staff often asked me what the line

"Money added to system"

meant when they switched the machine on but then I always did have a weird sense of humor.

Fixed Length

As UTF-8 only represents characters up to 31 bits in length, the alternative of every character taking 32 bits still remains another valid, if wasteful, option.

UTF-8 is somewhat wasteful as well, often requiring three bytes instead of two, or two bytes instead of one; stateful encodings can do much better.

Should be titled "Down with UTF-16"

Unicode is a good standard and it was written by clever guys. There's nothing wrong with Unicode's approach of mapping each character to a code point, and adding an intermediate step requiring encoding it into bytes. Far better than the ugly mess of codepages that preceded Unicode.

UTF-8 is part of Unicode and it's a damn good encoding.

Control Data had it right

The old CDC mainframes used a 6-bit character set - and no multibyte codes (until some idiot went and wanted lower case put in too).

Bravo!

I spent a lot of time fighting with text encodings when designing the .mobi file format for Mobipocket (and later Kindle). The conclusion was also that UTF-8 wins everywhere. The self-sync feature is superb. As for the "hassle" of handling a variable-length character encoding, you soon realize that:

- in most cases, you need the length of your string in bytes (for memory allocation, string copying, ...)

- cases where you need to decode UTF-8 to code points are rare, mostly when you display those characters and then you usually display the whole string from first to last byte so going through the bytes in sequence to decode the code points is not wasteful.

- the typical case when you need to know the characters is parsing, BUT, ALL keywords and ALL control characters in ALL computer languages are below the 128 code point so you can actually parse UTF8 as if it was ASCII and never care about the multi-byte encoding outside of string litterals.

So yes, UTF-8 everywhere!

The Youth Of Today

Some of us still dream in EBCDIC.

"public static final double π = 3.14159;"

No its not.

Joel Spolsky

"he royally patronises programmers"

He sure does. All the time. And not just about Unicode.

Mr. U will not be missed

(see ee cummings).

I must say that Perl makes it not too painful to deal with Unicode.

I am slight disappointed with Ms. Stob, though, for not riffing on The U and the Non-U...

Global posts

Discovered last week that Windows Notepad won't display all pasted UTF-8 characters - but it does preserve the binary values. So saving from Word in "TXT" UTF-8 format with an HTM suffix does appear correctly on a browser page.

Very useful for a hobby task that indexes public FaceBook and YouTube postings which can be written in just about any language. A quick screen scrape of Google Translate then combines a translation with the original.

Err....Qt has always tended to use UTF-8, not plain old UCS-2, for file i/o...

(Also, one 16-bit value equals one character wasn't true even from the start, even without taking Chinese into account; consider polytonic Greek for example)

Go Forth

I've been using UTF-8 in my Forth code for years; it's nice to be able to use function names (all right, "words" as we right-minded Forth programmers call them) maths and logical symbols in them.

Unicode needs to be taken out back and shot

Not just shot once, but repeatedly.

One of the principals of Unicode is to separate the character from the representation of the character. In other words, ASCII 65 (decimal) is "A". How your system chooses to display "A" is up to the system. The character is transmitted as decimal 65 no matter what the display representation is.

Unicode promptly goes on to rubbish this ideal.

Pre-Unicode Asian fonts had "full-width" representations of ASCII characters so displays that mixed ASCII and Japanese characters kept their formatting as the characters had the same width, while the usual ASCII characters were narrower and hence broke formatting.

Unfortunately this lives on in Unicode, shattering the idea that the display of the character is independent of the code point of the character because there are now two different Unicode code points that both print out a Latin-1 "A" (and also the rest of the alphabet and numbers and punctuation). In reality, the full width "A" should not be U+FF21, it should be decimal 65 with the renderer deciding if it should be full width or not.

This has caused me more than one problem in the past with things that sometimes correctly handle the full-width and ASCII mix and sometimes don't.