// Write the first character:
Console.WriteLine(s[0]);
 
// Reverse the string:
var backwards = s.ToArray();
Array.Reverse(backwards);
Console.WriteLine(new string(backwards));
 
// List the characters, separated by commas:
Console.WriteLine(string.Join(", ", s.ToArray()));

All of these are potential bugs.

Just set the following and you’ll get some nice output:

string s = "𠂔";

�
��
�,�

There are other problems as well:

string A = "𝔄"; // MATHEMATICAL FRAKTUR CAPITAL A, if you can't see it
Console.WriteLine(char.IsUpper(A[0])); // False
Console.WriteLine(char.IsUpper(A[1])); // False
Console.WriteLine(char.IsUpper(A,0)); // True - this is the correct way

Why is this?

Console.WriteLine(char.GetUnicodeCategory(A[0])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A[1])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A,0)); // UppercaseLetter

Ah.

The UCS-2 Hangover

These are all symptoms of ‘migrated’ libraries, originally developed for UCS-2 (in the early days of Unicode), and now touted as UTF-16.

Whereas UCS-2 was a fixed-width, 16-bit format, UTF-16 is a variable-width format which uses pairs of ‘surrogate’ 16-bit units to support characters that UCS-2 does not. These are the characters outside the Basic Multilingual Plane (BMP), sometimes termed the ‘astral characters’.

Libraries with char as a 16-bit quantity are living a lie. They live in a delusional world of fixed-width characters, where they let you believe you can index, substring, and play around at will, whereas:

• You cannot select arbitrary substrings — you might get half a character included.
• You cannot index into the string arbitrarily — you might select half a character.
• You cannot treat a string as a bunch of isolated characters — the surrogate pairs will get broken up.

So?

The general response seems to be “it doesn’t matter, most of the stuff I deal with is in the BMP, so I don’t care”.

A simple example of the problem with this: if any of these ‘orphan surrogates’ gets into an XML document, you have an application-stopping bug on your hands. (Just ask LINQPad, which refuses to run any of the examples in the first block of code. Luckily it has good exception handling.) A quick Google search shows that many developers are already encountering this problem.

All of the latest CJK extensions (B, C, D) also live in the Supplementary Ideographic Plane (SIP, the second non-BMP plane). I haven’t really been able to find out just how many of these characters are actually used in any real sense, but you’re going to encounter them eventually (and recent additions to Unicode might start turning up).

There’s a hole in your abstraction

The problem with UCS-2–flavoured UTF-16 is that it hands all the work of handling surrogate characters correctly over to the developer.

Underlining the third character in a string sounds like a simple task. But, if someone slips in a non-BMP character, then…

foreach (var s in new[] {"0123456789", "〇一二三四五六七八九", "𝟘123456789"})
{
	Console.WriteLine(s.Substring(0,2) + "<span style='border:2px #FF4500 solid'>" + s[2] + "</span>" + s.Substring(3));
}

0123456789
〇一二三四五六七八九
𝟘123456789

…and now the developer has to write their own linear-time indexing functions to actually find the character they need. This is not a good abstraction.

From here

Once you give up the idea that you can safely treat UTF-16 as a fixed-width format, it really doesn’t have anything going for it. (Perhaps this is why the idea is so firmly embedded — elements of cognitive dissonance.)

Here’s a quick (mainly biased) diagram:

s = cmds.sphere()
r = cmds.sphere(s, query=True, radius=True)
cmds.sphere(s, edit=True, radius=10)

Nasty.

So, here is a wrapper to make things nicer:

import maya.cmds as cmds
 
class Wrapper(object):
	def __init__(self, other):
		self.__dict__['me'] = other()
 
	def __getattr__(self, thing):
		i = 0 if cmds.attributeQuery(thing, node=self.__dict__['me'][0], exists=True) else 1
		return cmds.getAttr(self.__dict__['me'][i] + "." + thing)
 
	def __setattr__(self, thing, value):
		i = 0 if cmds.attributeQuery(thing, node=self.__dict__['me'][0], exists=True) else 1
		cmds.setAttr(self.__dict__['me'][i] + "." + name, value)

The above example is now:

s = Wrapper(cmds.sphere)
r = s.radius
s.radius = 10

• I ran into this bug (here’s the fix)
• ATi’s dual monitor setup is still too hard (here’s how to do it)
• ATi’s drivers can’t do compiz and video at the same time (Install the latest drivers to fix it)

How did we all miss that before?

—Peaker

The answer (of course!) is that while we might have, they didn’t. For example, Edward Kmett’s category-extras package has had Control.Monad.Indexed available for use in Haskell since early last year, while the concept for its implementation in Haskell has been around since at least 2005 (see the paper “Monadic Augment and Generalised Short Cut Fusion”). Needless to say, Oleg has been there too.

For me, this prompted a vaguely-related revelation. Say that we have a Category class:

class Category (⤳) where
	id ∷ a ⤳ a
	(∘) ∷ (b ⤳ c) → (a ⤳ b) → (a ⤳ c)

I had recently been wondering how categories other than (→), such as graphs (and even the trivial category) fit into this model. Now that I have figured it out, the answer is simple; it is a matter of arity. Graphs and other datastructure-related categories don’t have the extra type arguments (unless you’re working in a dependently-typed language), and so can’t fit into this. What we want to do is fake the extra arguments, by having some extra ones that we can just ignore (the name, FF, is short for FakeFake, i.e. two fake arguments).

newtype FF f a b = FF { unFF :: f }

Now we can define the trivial category!

class Category (FF ()) where
	id = FF ()
	f ∘ g = FF ()

Viewed in this light, it is obvious why (as Dan notes in the original post) Monoids give rise to Monads and Categories give rise to ParametrizedMonads, as we have:

class Monoid m ⇒ Category (FF m) where
	id = FF mempty
	(FF f) ∘ (FF g) = FF (mappend f g)

Next, we want to write implementations for all our old Monads in the new PMonad class. We might do it like this (with a new newtype for the different arguments of this datatype, RealFakeFakeReal):

newtype RFFR f x y a = RFFR { unRFFR ∷ (f a) }
 
class PMonad m where
	(>>=) ∷ m s1 s2 a → (a → m s2 s3 b) → m s1 s3 b
	return ∷ a → m s s a
 
instance (Old.Monad m) ⇒ PMonad (RFFR m) where
	(RFFR x) >>= f = RFFR (x Old.>>= unRFFR ∘ f)
	return x = RFFR (Old.return x)

There is a big issue that this highlights. It is hard to work with types of different arities in Haskell. (This is noted in the ‘quantified contexts’ proposal.)

As an example, since we now know that every Monoid is a Category, we might think to drop Monoids and just use Categories everywhere (I should note that this is actually a silly idea !). Unfortunately, we can’t just do this as we’ll have to use FF wrappers/unwrappers everywhere:

x = unFF $ FF [1,2,3] ∘ FF [4,5,6]

I have not yet found a workaround for this. Help would be nice