// Write the first character:
Console.WriteLine(s[0]);
 
// Reverse the string:
var backwards = s.ToArray();
Array.Reverse(backwards);
Console.WriteLine(new string(backwards));
 
// List the characters, separated by commas:
Console.WriteLine(string.Join(", ", s.ToArray()));

All of these are potential bugs.

Just set the following and you’ll get some nice output:

string s = "𠂔";

�
��
�,�

There are other problems as well:

string A = "𝔄"; // MATHEMATICAL FRAKTUR CAPITAL A, if you can't see it
Console.WriteLine(char.IsUpper(A[0])); // False
Console.WriteLine(char.IsUpper(A[1])); // False
Console.WriteLine(char.IsUpper(A,0)); // True - this is the correct way

Why is this?

Console.WriteLine(char.GetUnicodeCategory(A[0])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A[1])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A,0)); // UppercaseLetter

Ah.

The UCS-2 Hangover

These are all symptoms of ‘migrated’ libraries, originally developed for UCS-2 (in the early days of Unicode), and now touted as UTF-16.

Whereas UCS-2 was a fixed-width, 16-bit format, UTF-16 is a variable-width format which uses pairs of ‘surrogate’ 16-bit units to support characters that UCS-2 does not. These are the characters outside the Basic Multilingual Plane (BMP), sometimes termed the ‘astral characters’.

Libraries with char as a 16-bit quantity are living a lie. They live in a delusional world of fixed-width characters, where they let you believe you can index, substring, and play around at will, whereas:

• You cannot select arbitrary substrings — you might get half a character included.
• You cannot index into the string arbitrarily — you might select half a character.
• You cannot treat a string as a bunch of isolated characters — the surrogate pairs will get broken up.

So?

The general response seems to be “it doesn’t matter, most of the stuff I deal with is in the BMP, so I don’t care”.

A simple example of the problem with this: if any of these ‘orphan surrogates’ gets into an XML document, you have an application-stopping bug on your hands. (Just ask LINQPad, which refuses to run any of the examples in the first block of code. Luckily it has good exception handling.) A quick Google search shows that many developers are already encountering this problem.

All of the latest CJK extensions (B, C, D) also live in the Supplementary Ideographic Plane (SIP, the second non-BMP plane). I haven’t really been able to find out just how many of these characters are actually used in any real sense, but you’re going to encounter them eventually (and recent additions to Unicode might start turning up).

There’s a hole in your abstraction

The problem with UCS-2–flavoured UTF-16 is that it hands all the work of handling surrogate characters correctly over to the developer.

Underlining the third character in a string sounds like a simple task. But, if someone slips in a non-BMP character, then…

foreach (var s in new[] {"0123456789", "〇一二三四五六七八九", "𝟘123456789"})
{
	Console.WriteLine(s.Substring(0,2) + "<span style='border:2px #FF4500 solid'>" + s[2] + "</span>" + s.Substring(3));
}

0123456789
〇一二三四五六七八九
𝟘123456789

…and now the developer has to write their own linear-time indexing functions to actually find the character they need. This is not a good abstraction.

From here

Once you give up the idea that you can safely treat UTF-16 as a fixed-width format, it really doesn’t have anything going for it. (Perhaps this is why the idea is so firmly embedded — elements of cognitive dissonance.)

Here’s a quick (mainly biased) diagram:

Because I had much trouble finding a free copy of the document on the ’Net, I have made a table of the symbols and their functions below. I have marked those not present in Unicode as �.

Some examples: copy is usually [⎈ + C] and close is usually [⎇ + F4].

If we want to restrict ourselves to assigned, non-control, non-private Unicode characters, then by my reckoning that gives us 129,775 available characters.

wget http://unicode.org/Public/UNIDATA/UnicodeData.txt
awk -F ';' UnicodeData.txt -f countUnichars.awk | bc

countUnichars.awk source:

BEGIN { print "ibase=16" } # set bc to hex mode
 
$2 ~ /Private/ { # skip any lines with "private" in the description
    getline;
}
 
n { # if n is set, then print the range for bc to calculate
    printf("(%s-%s+1)+",$1,n);
    n="";
}
 
$2 ~ /First>/ { # set n if the start of a range
    n=$1;
    getline;
}
 
$3 !~ "C." { # otherwise count anything that isn't some kind of a control character
    i++;
}
 
END { # print out the count of everything else
    printf("%X\n",i)
}

This means we can store exactly 2377 bits (297 bytes) per message (this is ), so if we use a 16-colour palette we can store about 594 pixels (), which can almost reproduce the Mona Lisa thumbnail in the contest page.

Not only does Google search fail to recognize that “it’s” is a word, it also ignores the quote marks, searching for the phrase as individual words.

XML 1.1 allows you to insert characters from the C0 range as long as they are escaped as character entity references, and mandates that you do the same for those from the C1 range.

This little fact is the reason why not all XML 1.0-valid documents are valid under XML 1.1. Nasty.