At face value, setting a field to null means that the referenced object is now unrooted from the class that owns the field and, if that was the last root of that reference, the Garbage Collector (GC) is now free to release the memory used by the object that was referenced by that field. Although advanced, this seems all very academic because the amount of time between unrooting the reference and the return from Dispose (and thus the unrooting of the parent object) would seem like a very short amount of time. Even if the amount of time between these two actions is small, setting a single field to null (i.e. a single assignment) seems like such a minor bit of code to provide no adverse affects. The prevalent opinion seems to be that the GC “handles” this case and does what is best for you without setting the field to null.

In fact, this “short amount of time” can be so short as to be negative! A class that nothing else refers to is unrooted as soon as it no longer refers to itself, so objects in .NET can be collected while their instance methods are still executing. I first learned about this from Chris Brumme’s blog. Here is an example that shows this behaviour:

The output is:

start method
collected
end method

Once the method Go() has no references to this, it is eligible for collection. (What this means is that setting fields to null will actually extend the lifetime of the parent object ever-so-slightly.)

However, the code that a using is transformed into will hold onto a reference to the object while its Dispose() method is called, so you won’t get the same behaviour with the following code:

The output is:

start method
end method
collected

This is because the using statement is transformed to something like this:

I think (but cannot confirm) that the reference being held is that of Slow in the outer scope. We can restore the previous behaviour if the using transform was written as this instead:

Either way, I wouldn’t bother setting fields to null in the Dispose() method. (For one thing, these fields can no longer be declared readonly.) And most of the time, the GC is smarter than you are.

I know this might seem like an absolutely silly question to ask, yet I am too curious not to ask…

Why did “i” and “j” become THE variables to use as counters in most control structures?

The question has generated many answers, from scholarly to spurious — but the thing that has struck me is that no one has attempted to cite their sources or do any research. Why is this, when we live in a time when primary sources are more widely available than ever?

Let’s start with the claims that FORTRAN was the original source for their use in programing languages—while perhaps not the ultimate origin, it may have been the reason that they became widespread in the programming community.

The original manual for Fortran ¹ for the IBM 704 is readily available online. The first thing I notice is the glorious cover:

And sure enough, we can find the definition for the integral variables:

Unfortunately, the path stops here. I can’t find any references by Backus (or anyone else) as to why they chose IJKLMN as the integer variables. However, due to the fact that integer variables in Fortran “are somewhat restricted in their use and serve primarily as subscripts or exponents”, ² I am forced to the conclusion that they were used in imitation of those in mathematics. I don’t think we’ll ever know exactly who or when they were introduced to Fortran itself.

What we can do, however, is have a look at when they arose in mathematics. The usual place that i, j, etc. arise is in ‘sigma notation’, using the summation operator Σ. For example, if we write:

We mean i (= 1) + i (= 2) + i (= 3), until i = 100, and we can calculate the answer as 1 + 2 + 3 + 4 + … = 5050. So where did this notation itself come from?

The standard work on the history of mathematical notations is A History of Mathematical Notations by Florian Cajori. ³ He states that Σ was first used by Euler, in his Institutiones calculi differentialis (1755). We can see the part in question here:

This reads (translation by Ian Bruce, from 17thCentryMaths.com):

26. Just as we have been accustomed to specify the difference by the sign Δ, thus we will indicate the sum by the sign Σ; evidently if the difference of the function y were z, there will be z = Δy; from which, if y may be given, the difference z is found we have shown before. But if moreover the difference z shall be given and the sum of this y must be found, y = Σz is made and evidently from the equation z = Δy on regressing this equation will have the form y = Σz, where some constant quantity can be added on account of the reasons given above; […]

Evidently this is not the Σ we are looking for, as Euler uses it only in opposition to Δ (for finite differencing). In fact, Cajori notes that Euler’s Σ “received little attention”, and it seems that only Lagrange adopted it. Here is an excerpt from his Œuvres (printed MDCCCLXIX):

Again, we can see Σ is only used in opposition to Δ. Cajori next states that Σ to mean “sum” was used by Fourier, in his Théorie Analytique de la chaleur (1822), and here we find what we’re looking for:

The sign Σ affects the number i and indicates that the sum must be taken from i=1 to i=1/0. One can also contain the first term 1 under the sign Σ, and we have: [equation]

It must then have all its integral values from -1/0 up to 1/0; that is what one indicates by writing the limits -1/0 and +1/0 next to the sign Σ, that one of the values of i is 0. This is the most concise expression of the solution. ⁴

Since Fourier explains Σ several times in the book, and not just once, we can assume that the notation is either new or unfamiliar to most readers. ⁵ In any case, it doesn’t really matter who invented it, because while we have found our Σ, Fourier doesn’t explain why he uses i. In fact, since he uses it to index sequences in other places it appears it must be an already-existing usage. ⁶

A quick glance at the text by Euler above shows that he uses indexing very rarely (despite the subject of the text being a prime candidate!), and when he does, he uses m.

And, this is as far as I got. Time to publish this.

	object it = new SomeStruct { Item = 1 };
 
	Floatsy(it);
 
	Console.WriteLine(((SomeStruct)it).Item);

At runtime, it will throw an InvalidCastException!

Here’s how. In .NET, each object has also associated with it a value which determines the type of the object. In memory, this is stored before the object’s data, like so (I got this information from the MSDN article Drill Into .NET Framework Internals to See How the CLR Creates Runtime Objects):

Now, if we can write to that, we can set the type of the object to whatever we want!

There’s one small problem — in .NET we can’t take the address of a managed object (which we need in order to write to the object in memory). There are various reasons for this, one of them being that the garbage collector likes to be able to move objects around. Being able to take arbitrary pointers of objects would mean that the pointers could become invalidated.

What we can do is to pin a struct, which allows us to retrieve the address of it (this facility exists so that users can pass pinned managed structs into unmanaged code as pointers). Here’s how to pin a struct:

    var handle = GCHandle.Alloc(o, GCHandleType.Pinned);
    var addr = handle.AddrOfPinnedObject();

We can then write to the type handle like so (both the sync block and type handle are 32 bits):

    *(((int*)addr)-1) = someValue;

Another problem is that all of this is only true for values on the heap. As far as I can tell, the static type of the variable is what .NET uses to identify values on the stack. So in order for this to work, we must use a boxed copy of a struct.

Finally, here’s some demo code, showing runtime changing of types for both a primitive and custom struct:

using System;
using System.Runtime.InteropServices;
 
unsafe class Program
{
    struct SomeStruct { public int Item; }
 
    public static void Main()
    {
        object notAFloat = 1;
        object me = new SomeStruct {Item = 1};
 
        Console.WriteLine("{0} ({1})", notAFloat, notAFloat.GetType());
        Floatify(notAFloat);
        Console.WriteLine("{0} ({1})", notAFloat, notAFloat.GetType());
        Console.WriteLine();
        Console.WriteLine(me);
        Floatify(me);
        Console.WriteLine(me);
    }
 
    static void Floatify<T>(T o)
    {
        var handle = GCHandle.Alloc(o, GCHandleType.Pinned);
        var addr = handle.AddrOfPinnedObject();
 
        object f = (float)1.0;
        var handle2 = GCHandle.Alloc(f, GCHandleType.Pinned);
        var addr2 = handle2.AddrOfPinnedObject();
 
        // copy type handle of a float to the object
        *(((int*)addr)-1) = *(((int*)addr2)-1);
 
        handle2.Free();
        handle.Free();
    }
}

The output is:

1 (System.Int32)
1.401298E-45 (System.Single)

Program+SomeStruct
1.401298E-45

Here is the code:

var crlf = @"(\r\n)";
var wsp = @"[ \t]";
var vchar = @"[\u0021-\u007e]";
var obsNoWsCtl = @"[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f]";
var commentText = @"([\u0021-\u0027\u002a-\u005b\u005d-\u007e]|" + obsNoWsCtl + ")";
var quotedPair = @"\\(" + vchar + "|" + wsp + @"|[\r\n\0]|" + obsNoWsCtl + ")";
var quotedText = @"([\u0021\u0023-\u005b\u005d-\u007e]|"+obsNoWsCtl+")";
var domainText = "([\u0021-\u005A\u005e-\u007e]|" + obsNoWsCtl + ")";
var atomText = "[a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]";
var fws = "(" + wsp + "+" + "(" + crlf + wsp + "+)?|(" + crlf + wsp + "+)+)";   
var comment = @"(\((((?'paren'\()|(?'-paren'\))|"+commentText+"|"+fws+"|"+quotedPair+@")*(?(paren)(?!)))\))";
var cfws = "(" + comment + "|" + fws + ")*";
var atom = "(" + atomText + "+)";
var quotedContent = "(" + quotedText + "|" + quotedPair + ")";
var quotedString = "(\"(" + fws +"?" + quotedContent + ")*" + fws + "?\")";
var dottedAtoms1 = "(" + cfws + "?(" + atom + "|" + quotedString + ")"  + cfws + "?)";
var dottedAtoms = "(" + dottedAtoms1 + @"(\." + dottedAtoms1 + ")*)";
var localPart = "(?'localPart'" + dottedAtoms + ")";
var domainLiteral = "(" + cfws + @"?\[(" + fws + "?" + domainText + ")*" + fws + @"?\]" + cfws +"?)";
var domain = "(?'domain'" + dottedAtoms + "|" + domainLiteral + ")";
var email = "^" + localPart + "@" + domain + @"\z";
 
// Console.WriteLine(email);
 
var valid = @"I.                        
 am.                  
 a.      
 nice.
 guy@(yeah)you.com";
 
var success = Regex.Match(valid, email, RegexOptions.ExplicitCapture|RegexOptions.Singleline);
Console.WriteLine("Parsed: {0} <@> {1}", success.Groups["localPart"], success.Groups["domain"]);

And here is the regex. Despite supporting comments and a more recent RFC, it’s about ⅔ of the size of Perl’s Mail::RFC822::Address .

^(?'localPart'((((\((((?'paren'\()|(?'-paren'\))|([\u0021-\u
0027\u002a-\u005b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u
000e-\u001f\u007f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|
\\([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c
\u000e-\u001f\u007f]))*(?(paren)(?!)))\))|([ \t]+((\r\n)[ \t
]+)?|((\r\n)[ \t]+)+))*?(([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)|(
"(([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)?(([\u0021\u0023-\u
005b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u
007f])|\\([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000
b\u000c\u000e-\u001f\u007f])))*([ \t]+((\r\n)[ \t]+)?|((\r\n
)[ \t]+)+)?"))((\((((?'paren'\()|(?'-paren'\))|([\u0021-\u00
27\u002a-\u005b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u00
0e-\u001f\u007f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\
([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u
000e-\u001f\u007f]))*(?(paren)(?!)))\))|([ \t]+((\r\n)[ \t]+
)?|((\r\n)[ \t]+)+))*?)(\.(((\((((?'paren'\()|(?'-paren'\))|
([\u0021-\u0027\u002a-\u005b\u005d-\u007e]|[\u0001-\u0008\u0
00b\u000c\u000e-\u001f\u007f])|([ \t]+((\r\n)[ \t]+)?|((\r\n
)[ \t]+)+)|\\([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\
u000b\u000c\u000e-\u001f\u007f]))*(?(paren)(?!)))\))|([ \t]+
((\r\n)[ \t]+)?|((\r\n)[ \t]+)+))*?(([a-zA-Z0-9!#$%&'*+/=?^_
`{|}~-]+)|("(([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)?(([\u00
21\u0023-\u005b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u00
0e-\u001f\u007f])|\\([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-
\u0008\u000b\u000c\u000e-\u001f\u007f])))*([ \t]+((\r\n)[ \t
]+)?|((\r\n)[ \t]+)+)?"))((\((((?'paren'\()|(?'-paren'\))|([
\u0021-\u0027\u002a-\u005b\u005d-\u007e]|[\u0001-\u0008\u000
b\u000c\u000e-\u001f\u007f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[
\t]+)+)|\\([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u0
00b\u000c\u000e-\u001f\u007f]))*(?(paren)(?!)))\))|([ \t]+((
\r\n)[ \t]+)?|((\r\n)[ \t]+)+))*?))*))@(?'domain'((((\((((?'
paren'\()|(?'-paren'\))|([\u0021-\u0027\u002a-\u005b\u005d-\
u007e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f])|([ \t
]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\([\u0021-\u007e]|[ \t]|
[\r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f]))*(?
(paren)(?!)))\))|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+))*?(
([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)|("(([ \t]+((\r\n)[ \t]+)?|
((\r\n)[ \t]+)+)?(([\u0021\u0023-\u005b\u005d-\u007e]|[\u000
1-\u0008\u000b\u000c\u000e-\u001f\u007f])|\\([\u0021-\u007e]
|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007
f])))*([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)?"))((\((((?'pa
ren'\()|(?'-paren'\))|([\u0021-\u0027\u002a-\u005b\u005d-\u0
07e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f])|([ \t]+
((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\([\u0021-\u007e]|[ \t]|[\
r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f]))*(?(p
aren)(?!)))\))|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+))*?)(\
.(((\((((?'paren'\()|(?'-paren'\))|([\u0021-\u0027\u002a-\u0
05b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u0
07f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\([\u0021-\u0
07e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\
u007f]))*(?(paren)(?!)))\))|([ \t]+((\r\n)[ \t]+)?|((\r\n)[
\t]+)+))*?(([a-zA-Z0-9!#$%&'*+/=?^_`{|}~-]+)|("(([ \t]+((\r\
n)[ \t]+)?|((\r\n)[ \t]+)+)?(([\u0021\u0023-\u005b\u005d-\u0
07e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f])|\\([\u0
021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-
\u001f\u007f])))*([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)?"))
((\((((?'paren'\()|(?'-paren'\))|([\u0021-\u0027\u002a-\u005
b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007
f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\([\u0021-\u007
e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u0
07f]))*(?(paren)(?!)))\))|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t
]+)+))*?))*)|(((\((((?'paren'\()|(?'-paren'\))|([\u0021-\u00
27\u002a-\u005b\u005d-\u007e]|[\u0001-\u0008\u000b\u000c\u00
0e-\u001f\u007f])|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\
([\u0021-\u007e]|[ \t]|[\r\n\0]|[\u0001-\u0008\u000b\u000c\u
000e-\u001f\u007f]))*(?(paren)(?!)))\))|([ \t]+((\r\n)[ \t]+
)?|((\r\n)[ \t]+)+))*?\[(([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]
+)+)?([!-Z^-~]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f
]))*([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+)?\]((\((((?'paren
'\()|(?'-paren'\))|([\u0021-\u0027\u002a-\u005b\u005d-\u007e
]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f])|([ \t]+((\
r\n)[ \t]+)?|((\r\n)[ \t]+)+)|\\([\u0021-\u007e]|[ \t]|[\r\n
\0]|[\u0001-\u0008\u000b\u000c\u000e-\u001f\u007f]))*(?(pare
n)(?!)))\))|([ \t]+((\r\n)[ \t]+)?|((\r\n)[ \t]+)+))*?))\z

Basically, any time you use a named capturing group, it actually pushes the capture onto a named stack. You can then pop this stack by using the same capturing group prefixed with a hyphen, like (?<-stackToPop>).

Of course, anyone who finds themselves in this situation is going to ask: can it match XML?

It’s possible that I am missing something completely (it is rather late at night), but … very nearly. I haven’t quite figured out a nested section in the local DTD subset, but no one uses that feature anyway. (Can you spot it?)

Aside from that, most of the well-formedness criteria are handled (the obvious one being element nesting). Things that require non-local information such as entities aren’t handled. I think it is possible to handle duplicate attribute names in this form as well (via a lookahead for duplicate names).

Here is the code, and a test file which shows some stuff that is caught by this. Breaking any of the elements should make it fail:

var surrogate = @"([\ud800-\udbff][\udc00-\udfff])";// .NET can't handle \U10000-\u10FFFF
var c = @"([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|"+surrogate + ")"; 
var s = @"([\u0020\u0009\u000d\u000a]+)";
var nameStartChar = @"([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|" + surrogate + ")";
var nameChar = "(" + nameStartChar + @"|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])";
var name = "(?'name'" + nameStartChar + nameChar + "*)";
var names = "(?'names'" + name + @"(\u0020" + name +")*)";
var nmtoken = "(?'nmtoken'" + nameChar + "+)";
var nmtokens = "(?'nmtokens'" + nmtoken + @"(\u0020" + nmtoken +")*)";
var pereference = "%" + name + ";";
var entityReference= "(?'entityRef'&" + name + ";)";
var charref = @"&\#([0-9]+|x[0-9a-fA-F]+);";
var reference = "(?'reference'"+ entityReference + "|" + charref + ")";
var entityValue = "(?'entityValue'\"([^%&\"]|" + pereference + "|" + reference + ")*\"|'([^%&']|" + pereference + "|" + reference + ")*')";
var eq = "(?'eq'" + s + "?=" + s + "?)";
var versionNum  = @"1\.[0-9]+";
var comment = "(?'comment'<!--((?!--)" + c + ")*-->)";
var PITarget = "(?'pitarget'(?![xX][mM][lL])"+name+")";
var PI= @"(?'PI'<\?" + PITarget + "(" + s + @"((?!\?>)" + c + @")*)?\?>)";
var misc = "(?'misc'" + comment + "|" + PI + "|" + s + ")";
var versionInfo = "(?'versionInfo'"+ s + "version" + eq + "('" + versionNum + "'|\"" + versionNum + "\"))";
var encName = "(?'encName'[A-Za-z][A-Za-z0-9._-]*)";
var encodingDecl = "(?'encodingDecl'" + s + "encoding" + eq + "(\"" + encName + "\"|'"+ encName + "'))";
var sddecl = "(?'sddecl'" + s + "standalone" + eq + "(\"(yes|no)\"|'(yes|no)'))";
var xmlDecl = @"(?'xmlDecl'<\?xml" + versionInfo + encodingDecl + "?" + sddecl + "?" + s + @"?\?>)"; 
var mixed = @"(?'mixed'\(" + s + @"?\#PCDATA" + "(" + s + @"?\|" + s + "?" + name +")*" + s + "?" + @"\)\*|\(" +s + @"?\#PCDATA" + s + @"?\))";
var children = @"(?'children'unsureifpossible)";
var contentSpec = "(?'contentspec'EMPTY|ANY|"+mixed+"|"+children+")";
var elementDecl = "(?'elementdecl'<!ELEMENT" + s + name + s + contentSpec + s + "?>)";
var stringType = "CDATA";
var tokenizedType = "(ID(REF(S)?)?|ENTIT(Y|IES)|NMTOKENS?)";
var notationType = "(?'notation'NOTATION" +s + @"\(" + s + "?" + name + "(" + s + @"?\|" + s + "?" + name + ")*" + s + @"?\))";
var enumeration = @"(?'enumeration'\(" + s + "?" + nmtoken + "(" + s + @"?\|" + s + "?" + nmtoken + ")*" + s + @"?\))";
var enumeratedType = "(?'enumType'" + notationType + "|" + enumeration +")";
var attType = "(?'attType'" + stringType + "|" + tokenizedType + "|" + enumeratedType + ")";
var attValue = "(?'attValue'\"([^<&\"]|" +reference+ ")*\"|'([^<&']|" + reference + ")*')";
var defaultDecl = @"(?'defaultDecl'\#REQUIRED|\#IMPLIED|(\#FIXED"+s+")?" + attValue + ")";
var attDef = "(?'attDef'"+ s + name + s + attType + s + defaultDecl + ")";
var attListDecl = "(?'attlist'<!ATTLIST" + s + name + attDef + "*" + s + "?>)";
var systemLiteral = "(?'systemLiteral'\"[^\"]*\"|'[^']*')";
var pubIdChar = @"[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\u000a-]";
var pubidLiteral = "(?'pubIdLiteral'\"" + pubIdChar + "*\"|'((?!')" + pubIdChar + ")*')";
var externalID = "(?'externalID'SYSTEM" + s + systemLiteral +"|PUBLIC" + s + pubidLiteral + s + systemLiteral+")";
var nDataDecl = "(?'ndatadecl'"+s + "NDATA" + s + name + ")";
var entityDef  = "(?'entityDef'" + entityValue + "|(" +externalID + nDataDecl + "?))";
var peDef = "(?'pedef'" + entityValue + "|"  + externalID + ")";
var GEDecl = "(?'gedecl'<!ENTITY" + s + name + s + entityDef + s + "?>)";
var PEDecl = "(?'gedecl'<!ENTITY" + s + "%" + s + name + s + peDef + s + "?>)";
var entityDecl = "(?'entityDecl'"+ GEDecl + "|" + PEDecl +")";
var publicID = "(?'publicID'PUBLIC" + s + pubidLiteral + ")";
var notationDecl = "(?'notationDecl'<!NOTATION" +  s + name + s + "(" + externalID + "|" + publicID + ")" + s + "?>)";
var markupDecl = "(?'markupdecl'" + elementDecl + "|" + attListDecl + "|" + entityDecl + "|" + notationDecl + "|" + PI + "|" + comment + ")";
var DeclSep = "(?'declSep'" + pereference + "|" + s + ")";
var intSubSet = @"(?'intSubSet'(" + markupDecl + "|" + DeclSep + ")*)"; 
var docTypeDecl = "(?'doctypedecl'<!DOCTYPE" + s + name + "(" + s + externalID+ ")?" + s + @"?(\[" + intSubSet + @"\]" + s + "?)?>)"; 
var prolog = xmlDecl + "?" + misc + "*(" + docTypeDecl + misc + "*)?"; 
var attribute = "(?'attribute'" +name + eq + attValue + ")";
var CDSect = @"(?'CDSect'<!\[CDATA\[((?!\]\]>)"+c+@")*\]\]>)";
var charData = @"(((?!\]\]>)[^<&])*)";
var content = @"(?>" + // minor optimization... don't backtrack over this (makes failing faster)
		@"<(?'openclose'" + name + @")(" + s + attribute + ")*" + s + @"?/>|"+
		@"<(?'open'"+ name +@")(" + s + attribute + ")*" + s + @"?>|"+
		@"</(?=\k'open'" + s + @"?>)(?'close-open'" + name + ")" + s +@"?>|"
		+reference+@"|"
		+PI+@"|"
		+comment+@"|"
		+CDSect+@"|"
		+charData+@")*" + 
	"(?(open)(?!))";
var rootElement = @"(?'root'(<(?'rootName'" + name + ")(" + s + attribute + ")*" + s + @"?>" + content + @"</\k'rootName'" + s + "?>)|(<(?'rootName'" + name + ")(" + s + attribute + ")*" + s + @"?/>))";
 
var document = "^" + prolog + rootElement + misc + "*" + "$";
 
var testDoc = @"<?xml version='1.0' encoding=""utf-8""?><!DOCTYPE nothtml []><items>
	<item available=""yes"" >
		<name> laptop  </name>
		<![CDATA[something14!$]] 1412]]>
		<"+"\U00010000"+@"quantity>  2 &amp; &#121; &#x234f; </"+"\U00010000"+@"quantity>
	</item><?notxml?>" /* or <?xml?> here */ +@"
	<item available=""yes"" x='' y=""&amp;"">
		<name> mouse </name >
		<quantity> 1 " + /* or ]]> invalid here */  @" </quantity>
	</item>
	<item available=""no"" >
		<!----> <!-- --> <!-- - -->" + /* or <!-- -- --> here */ @"
		<name> keyboad </name>
		<quantity> 0</quantity>
	</item>
</items><!-- stuff can go here --> <!-- yup --> <?pi aasd as!@*&$^!*@&$!@ ?>";
 
//Console.WriteLine(document);
Console.WriteLine(Regex.Match(testDoc, document, RegexOptions.IgnorePatternWhitespace|RegexOptions.Singleline|RegexOptions.ExplicitCapture));

And here’s the regex (with apologies to Mail::RFC822::Address):

^(?'xmlDecl'<\?xml(?'versionInfo'([\u0020\u0009\u000d\u000a]+)version(?'eq'([\u0
020\u0009\u000d\u000a]+)?=([\u0020\u0009\u000d\u000a]+)?)('1\.[0-9]+'|"1\.[0-9]+
"))(?'encodingDecl'([\u0020\u0009\u000d\u000a]+)encoding(?'eq'([\u0020\u0009\u00
0d\u000a]+)?=([\u0020\u0009\u000d\u000a]+)?)("(?'encName'[A-Za-z][A-Za-z0-9._-]*
)"|'(?'encName'[A-Za-z][A-Za-z0-9._-]*)'))?(?'sddecl'([\u0020\u0009\u000d\u000a]
+)standalone(?'eq'([\u0020\u0009\u000d\u000a]+)?=([\u0020\u0009\u000d\u000a]+)?)
("(yes|no)"|'(yes|no)'))?([\u0020\u0009\u000d\u000a]+)?\?>)?(?'misc'(?'comment'<
!--((?!--)([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc0
0-\udfff])))*-->)|(?'PI'<\?(?'pitarget'(?![xX][mM][lL])(?'name'([:A-Z_a-z\u00C0-
\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u
218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc0
0-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F
-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\
uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040]
)*))(([\u0020\u0009\u000d\u000a]+)((?!\?>)([\u0009\u000a\u000d\u0020-\ud7ff\ue00
0-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))*)?\?>)|([\u0020\u0009\u000d\u000a]+
))*((?'doctypedecl'<!DOCTYPE([\u0020\u0009\u000d\u000a]+)(?'name'([:A-Z_a-z\u00C
0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-
\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\ud
c00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u03
7F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0
-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u204
0])*)(([\u0020\u0009\u000d\u000a]+)(?'externalID'SYSTEM([\u0020\u0009\u000d\u000
a]+)(?'systemLiteral'"[^"]*"|'[^']*')|PUBLIC([\u0020\u0009\u000d\u000a]+)(?'pubI
dLiteral'"[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\u000a-]*"|'((?!')[a-zA-Z0-9'(
)+,./:=?;!*#@$_%\u0020\u000d\u000a-])*')([\u0020\u0009\u000d\u000a]+)(?'systemLi
teral'"[^"]*"|'[^']*')))?([\u0020\u0009\u000d\u000a]+)?(\[(?'intSubSet'((?'marku
pdecl'(?'elementdecl'<!ELEMENT([\u0020\u0009\u000d\u000a]+)(?'name'([:A-Z_a-z\u0
0C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u207
0-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\
udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u
037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFD
F0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2
040])*)([\u0020\u0009\u000d\u000a]+)(?'contentspec'EMPTY|ANY|(?'mixed'\(([\u0020
\u0009\u000d\u000a]+)?\#PCDATA(([\u0020\u0009\u000d\u000a]+)?\|([\u0020\u0009\u0
00d\u000a]+)?(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u0
37D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDC
F\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-
\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u
2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[
-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))*([\u0020\u0009\u000d\u000a]+)?\)\*|\(
([\u0020\u0009\u000d\u000a]+)?\#PCDATA([\u0020\u0009\u000d\u000a]+)?\))|(?'child
ren'unsureifpossible))([\u0020\u0009\u000d\u000a]+)?>)|(?'attlist'<!ATTLIST([\u0
020\u0009\u000d\u000a]+)(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02F
F\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\
uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u
00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u21
8F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-
\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*)(?'attDef'([\u0020\u0009\u00
0d\u000a]+)(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037
D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\
uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u
00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2F
EF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.
0-9\u00B7\u0300-\u036F\u203F-\u2040])*)([\u0020\u0009\u000d\u000a]+)(?'attType'C
DATA|(ID(REF(S)?)?|ENTIT(Y|IES)|NMTOKENS?)|(?'enumType'(?'notation'NOTATION([\u0
020\u0009\u000d\u000a]+)\(([\u0020\u0009\u000d\u000a]+)?(?'name'([:A-Z_a-z\u00C0
-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\
u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc
00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037
F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-
\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040
])*)(([\u0020\u0009\u000d\u000a]+)?\|([\u0020\u0009\u000d\u000a]+)?(?'name'([:A-
Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u2
00D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\
udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-
\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\u
FDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u
203F-\u2040])*))*([\u0020\u0009\u000d\u000a]+)?\))|(?'enumeration'\(([\u0020\u00
09\u000d\u000a]+)?(?'nmtoken'(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\
u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF
900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u
036F\u203F-\u2040])+)(([\u0020\u0009\u000d\u000a]+)?\|([\u0020\u0009\u000d\u000a
]+)?(?'nmtoken'(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u
037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFD
F0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2
040])+))*([\u0020\u0009\u000d\u000a]+)?\))))([\u0020\u0009\u000d\u000a]+)(?'defa
ultDecl'\#REQUIRED|\#IMPLIED|(\#FIXED([\u0020\u0009\u000d\u000a]+))?(?'attValue'
"([^<&"]|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6
\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u
3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_
a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200
D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\ud
bff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|
x[0-9a-fA-F]+);()))*"|'([^<&']|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00
C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070
-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\u
dc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u0
37F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF
0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u20
40])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*')))*([\u0020\u0009\u000d\u000a]+)?>)|
(?'entityDecl'(?'gedecl'<!ENTITY([\u0020\u0009\u000d\u000a]+)(?'name'([:A-Z_a-z\
u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2
070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff]
[\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D
\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\u
FDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\
u2040])*)([\u0020\u0009\u000d\u000a]+)(?'entityDef'(?'entityValue'"([^%&"]|%(?'n
ame'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\
u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|(
[\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02
FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF
\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300
-\u036F\u203F-\u2040])*);|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u
00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u21
8F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-
\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\
u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uF
FFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*
);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*"|'([^%&']|%(?'name'([:A-Z_a-z\u00C0-\u00D6
\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u
2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udf
ff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FF
F\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]
|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);|(
?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u0
2FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7F
F\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-
\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u
218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc0
0-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-
F]+);()))*')|((?'externalID'SYSTEM([\u0020\u0009\u000d\u000a]+)(?'systemLiteral'
"[^"]*"|'[^']*')|PUBLIC([\u0020\u0009\u000d\u000a]+)(?'pubIdLiteral'"[a-zA-Z0-9'
()+,./:=?;!*#@$_%\u0020\u000d\u000a-]*"|'((?!')[a-zA-Z0-9'()+,./:=?;!*#@$_%\u002
0\u000d\u000a-])*')([\u0020\u0009\u000d\u000a]+)(?'systemLiteral'"[^"]*"|'[^']*'
))(?'ndatadecl'([\u0020\u0009\u000d\u000a]+)NDATA([\u0020\u0009\u000d\u000a]+)(?
'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FF
F\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]
|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u
02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7
FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u03
00-\u036F\u203F-\u2040])*))?))([\u0020\u0009\u000d\u000a]+)?>)|(?'gedecl'<!ENTIT
Y([\u0020\u0009\u000d\u000a]+)%([\u0020\u0009\u000d\u000a]+)(?'name'([:A-Z_a-z\u
00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u20
70-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][
\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\
u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uF
DF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u
2040])*)([\u0020\u0009\u000d\u000a]+)(?'pedef'(?'entityValue'"([^%&"]|%(?'name'(
[:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C
-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud8
00-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0
370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF90
0-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u03
6F\u203F-\u2040])*);|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\
u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2
C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udff
f]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF
\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|
([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())
|&\#([0-9]+|x[0-9a-fA-F]+);()))*"|'([^%&']|%(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D
8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-
\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))
(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u20
0C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\u
d800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);|(?'ref
erence'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u
0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF9
00-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D
6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\
u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\ud
fff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);
()))*')|(?'externalID'SYSTEM([\u0020\u0009\u000d\u000a]+)(?'systemLiteral'"[^"]*
"|'[^']*')|PUBLIC([\u0020\u0009\u000d\u000a]+)(?'pubIdLiteral'"[a-zA-Z0-9'()+,./
:=?;!*#@$_%\u0020\u000d\u000a-]*"|'((?!')[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000
d\u000a-])*')([\u0020\u0009\u000d\u000a]+)(?'systemLiteral'"[^"]*"|'[^']*')))([\
u0020\u0009\u000d\u000a]+)?>))|(?'notationDecl'<!NOTATION([\u0020\u0009\u000d\u0
00a]+)(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u03
7F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0
-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\
u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3
001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u
00B7\u0300-\u036F\u203F-\u2040])*)([\u0020\u0009\u000d\u000a]+)((?'externalID'SY
STEM([\u0020\u0009\u000d\u000a]+)(?'systemLiteral'"[^"]*"|'[^']*')|PUBLIC([\u002
0\u0009\u000d\u000a]+)(?'pubIdLiteral'"[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\
u000a-]*"|'((?!')[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\u000a-])*')([\u0020\u0
009\u000d\u000a]+)(?'systemLiteral'"[^"]*"|'[^']*'))|(?'publicID'PUBLIC([\u0020\
u0009\u000d\u000a]+)(?'pubIdLiteral'"[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\u0
00a-]*"|'((?!')[a-zA-Z0-9'()+,./:=?;!*#@$_%\u0020\u000d\u000a-])*')))([\u0020\u0
009\u000d\u000a]+)?>)|(?'PI'<\?(?'pitarget'(?![xX][mM][lL])(?'name'([:A-Z_a-z\u0
0C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u207
0-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\
udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u
037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFD
F0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2
040])*))(([\u0020\u0009\u000d\u000a]+)((?!\?>)([\u0009\u000a\u000d\u0020-\ud7ff\
ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))*)?\?>)|(?'comment'<!--((?!--)([
\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))
*-->))|(?'declSep'%(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u03
70-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900
-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\
u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2
C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udff
f]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);|([\u0020\u0009\u000d\u000a]+)))
*)\]([\u0020\u0009\u000d\u000a]+)?)?>)(?'misc'(?'comment'<!--((?!--)([\u0009\u00
0a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))*-->)|(?'P
I'<\?(?'pitarget'(?![xX][mM][lL])(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u0
0F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u300
1-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z
\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u
2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff
][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u0009\u0
00d\u000a]+)((?!\?>)([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\ud
bff][\udc00-\udfff])))*)?\?>)|([\u0020\u0009\u000d\u000a]+))*)?(?'root'(<(?'root
Name'(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037
F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-
\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u
00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u30
01-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u0
0B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u0009\u000d\u000a]+)(?'attribute'(?'
name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF
\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|
([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u0
2FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7F
F\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u030
0-\u036F\u203F-\u2040])*)(?'eq'([\u0020\u0009\u000d\u000a]+)?=([\u0020\u0009\u00
0d\u000a]+)?)(?'attValue'"([^<&"]|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\
u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2
070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff]
[\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D
\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\u
FDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\
u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*"|'([^<&']|(?'reference'(?'entityRe
f'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-
\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\u
FFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00
F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001
-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B
7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*')))*([\u0020
\u0009\u000d\u000a]+)?>(?><(?'openclose'(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u
00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2F
EF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:
A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\
u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800
-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u
0009\u000d\u000a]+)(?'attribute'(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00
F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001
-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\
u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2
070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff]
[\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*)(?'eq'([\u0020\u0009
\u000d\u000a]+)?=([\u0020\u0009\u000d\u000a]+)?)(?'attValue'"([^<&"]|(?'referenc
e'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-
\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\u
FDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00
D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00
-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff])
)|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*
"|'([^<&']|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00
F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF
\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-
Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u2
00D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\
udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]
+|x[0-9a-fA-F]+);()))*')))*([\u0020\u0009\u000d\u000a]+)?/>|<(?'open'(?'name'([:
A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\
u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800
-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u037
0-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-
\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F
\u203F-\u2040])*))(([\u0020\u0009\u000d\u000a]+)(?'attribute'(?'name'([:A-Z_a-z\
u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2
070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff]
[\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D
\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\u
FDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\
u2040])*)(?'eq'([\u0020\u0009\u000d\u000a]+)?=([\u0020\u0009\u000d\u000a]+)?)(?'
attValue'"([^<&"]|(?'reference'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00
D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00
-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff])
)(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u2
00C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\
ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\
#([0-9]+|x[0-9a-fA-F]+);()))*"|'([^<&']|(?'reference'(?'entityRef'&(?'name'([:A-
Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u2
00D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\
udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-
\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\u
FDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u
203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*')))*([\u0020\u0009\u000d\u00
0a]+)?>|</(?=\k'open'([\u0020\u0009\u000d\u000a]+)?>)(?'close-open'(?'name'([:A-
Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u2
00D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\
udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-
\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\u
FDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u
203F-\u2040])*))([\u0020\u0009\u000d\u000a]+)?>|(?'reference'(?'entityRef'&(?'na
me'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u
200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([
\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02F
F\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\
uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-
\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);())|(?'PI'<\?(?'pitarget'(
?![xX][mM][lL])(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\
u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uF
DCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D
8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-
\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))
|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u0009\u000d\u000a]+)((?!\?
>)([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff
])))*)?\?>)|(?'comment'<!--((?!--)([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd
]|([\ud800-\udbff][\udc00-\udfff])))*-->)|(?'CDSect'<!\[CDATA\[((?!\]\]>)([\u000
9\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))*\]\]
>)|(((?!\]\]>)[^<&])*))*(?(open)(?!))</\k'rootName'([\u0020\u0009\u000d\u000a]+)
?>)|(<(?'rootName'(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u037
0-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-
\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u
00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C
00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff
]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u0009\u000d\u000a]+)(?'
attribute'(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D
\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\u
FDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u0
0F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FE
F\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0
-9\u00B7\u0300-\u036F\u203F-\u2040])*)(?'eq'([\u0020\u0009\u000d\u000a]+)?=([\u0
020\u0009\u000d\u000a]+)?)(?'attValue'"([^<&"]|(?'reference'(?'entityRef'&(?'nam
e'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u2
00C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\
ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF
\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\u
F900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\
u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*"|'([^<&']|(?'referenc
e'(?'entityRef'&(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-
\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\u
FDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-z\u00C0-\u00D6\u00
D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00
-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff])
)|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*);())|&\#([0-9]+|x[0-9a-fA-F]+);()))*
')))*([\u0020\u0009\u000d\u000a]+)?/>))(?'misc'(?'comment'<!--((?!--)([\u0009\u0
00a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\udbff][\udc00-\udfff])))*-->)|(?'
PI'<\?(?'pitarget'(?![xX][mM][lL])(?'name'([:A-Z_a-z\u00C0-\u00D6\u00D8-\u00F6\u
00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\u2070-\u218F\u2C00-\u2FEF\u30
01-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbff][\udc00-\udfff]))(([:A-Z_a-
z\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02FF\u0370-\u037D\u037F-\u1FFF\u200C-\u200D\
u2070-\u218F\u2C00-\u2FEF\u3001-\uD7FF\uF900-\uFDCF\uFDF0-\uFFFD]|([\ud800-\udbf
f][\udc00-\udfff]))|[-.0-9\u00B7\u0300-\u036F\u203F-\u2040])*))(([\u0020\u0009\u
000d\u000a]+)((?!\?>)([\u0009\u000a\u000d\u0020-\ud7ff\ue000-\ufffd]|([\ud800-\u
dbff][\udc00-\udfff])))*)?\?>)|([\u0020\u0009\u000d\u000a]+))*$

// Write the first character:
Console.WriteLine(s[0]);
 
// Reverse the string:
var backwards = s.ToArray();
Array.Reverse(backwards);
Console.WriteLine(new string(backwards));
 
// List the characters, separated by commas:
Console.WriteLine(string.Join(", ", s.ToArray()));

All of these are potential bugs.

Just set the following and you’ll get some nice output:

string s = "𠂔";

�
��
�,�

There are other problems as well:

string A = "𝔄"; // MATHEMATICAL FRAKTUR CAPITAL A, if you can't see it
Console.WriteLine(char.IsUpper(A[0])); // False
Console.WriteLine(char.IsUpper(A[1])); // False
Console.WriteLine(char.IsUpper(A,0)); // True - this is the correct way

Why is this?

Console.WriteLine(char.GetUnicodeCategory(A[0])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A[1])); // Surrogate
Console.WriteLine(char.GetUnicodeCategory(A,0)); // UppercaseLetter

Ah.

The UCS-2 Hangover

These are all symptoms of ‘migrated’ libraries, originally developed for UCS-2 (in the early days of Unicode), and now touted as UTF-16.

Whereas UCS-2 was a fixed-width, 16-bit format, UTF-16 is a variable-width format which uses pairs of ‘surrogate’ 16-bit units to support characters that UCS-2 does not. These are the characters outside the Basic Multilingual Plane (BMP), sometimes termed the ‘astral characters’.

Libraries with char as a 16-bit quantity are living a lie. They live in a delusional world of fixed-width characters, where they let you believe you can index, substring, and play around at will, whereas:

• You cannot select arbitrary substrings — you might get half a character included.
• You cannot index into the string arbitrarily — you might select half a character.
• You cannot treat a string as a bunch of isolated characters — the surrogate pairs will get broken up.

So?

The general response seems to be “it doesn’t matter, most of the stuff I deal with is in the BMP, so I don’t care”.

A simple example of the problem with this: if any of these ‘orphan surrogates’ gets into an XML document, you have an application-stopping bug on your hands. (Just ask LINQPad, which refuses to run any of the examples in the first block of code. Luckily it has good exception handling.) A quick Google search shows that many developers are already encountering this problem.

All of the latest CJK extensions (B, C, D) also live in the Supplementary Ideographic Plane (SIP, the second non-BMP plane). I haven’t really been able to find out just how many of these characters are actually used in any real sense, but you’re going to encounter them eventually (and recent additions to Unicode might start turning up).

There’s a hole in your abstraction

The problem with UCS-2–flavoured UTF-16 is that it hands all the work of handling surrogate characters correctly over to the developer.

Underlining the third character in a string sounds like a simple task. But, if someone slips in a non-BMP character, then…

foreach (var s in new[] {"0123456789", "〇一二三四五六七八九", "𝟘123456789"})
{
	Console.WriteLine(s.Substring(0,2) + "<span style='border:2px #FF4500 solid'>" + s[2] + "</span>" + s.Substring(3));
}

0123456789
〇一二三四五六七八九
𝟘123456789

…and now the developer has to write their own linear-time indexing functions to actually find the character they need. This is not a good abstraction.

From here

Once you give up the idea that you can safely treat UTF-16 as a fixed-width format, it really doesn’t have anything going for it. (Perhaps this is why the idea is so firmly embedded — elements of cognitive dissonance.)

Here’s a quick (mainly biased) diagram:

using System;
 
namespace ConsoleApplication
{
    public static class MainClass
    {
        // Example:
 
        public static void Main(string[] args)
        {
            var stack = Stack.New<int>(); // real type Stack<Z,int>
 
            var s1 = stack.Push(1).Push(2).Push(3); // real type Stack<S<S<S<Z>>>>
            var s2 = s1.Pop().Pop().Pop(); // real type Stack<Z,int>
 
            // statically disallowed:
            // s2.Pop();
 
            Pop2(s1);
 
            Console.WriteLine(s1.IsEmpty());
            Console.WriteLine(s2.IsEmpty());
        }
 
        // Example of a method requiring "at least 2" items on stack.
        static void Pop2<TRest,T>(Stack<S<S<TRest>>,T> stack)
        {
            T x1;
            T x2;
            stack.Pop(out x1).Pop(out x2);
            Console.WriteLine("{0}, {1}", x1, x2);
        }
    }
 
    // Implementation:
 
    // very weak type level naturals
    // Z = zero, S<x> = successor of x.
    // so, S<Z> = 1, S<S<Z>> = 2, etc.
    public sealed class S<TPrev> {}
    public sealed class Z {}
 
    // internal node type for the linked list
    internal class StackNode<T>
    {
        public StackNode(T item, StackNode<T> next)
        {
            this.item = item;
            this.next = next;
        }
 
        internal T item;
        internal StackNode<T> next;
    }
 
    // the stack wrapper type (sized with types)
    public class Stack<TSize,T>
    {
        internal StackNode<T> node;
 
        // Construct wrapper:
        internal Stack(StackNode<T> node)
        {
            this.node = node;
        }
 
        // Push:
        internal Stack(T item, StackNode<T> node)
        {
            this.node = new StackNode<T>(item, node);
        }
 
        // note that the return type has the size increased by 1
        // S<TSize> ~= TSize++ 
        public Stack<S<TSize>,T> Push(T item)
        {
            return new Stack<S<TSize>,T>(item, node);
        }
    }
 
    // extension methods that perform the operations:
    public static class Stack
    {
        // a new stack has size zero
        public static Stack<Z,T> New<T>()
        {
            return new Stack<Z, T>(null);
        }
 
        // pop with ignored result
        public static Stack<TRest,T> Pop<TRest,T>(this Stack<S<TRest>,T> stack)
        {
            T ignored;
            return stack.Pop(out ignored);
        }
 
        // popping an item removes one S<> from the size (~= TSize--)
        public static Stack<TRest,T> Pop<TRest,T>(this Stack<S<TRest>,T> stack, out T item)
        {
            item = stack.node.item;
            return new Stack<TRest, T>(stack.node.next);
        }
 
        // These are statically known, because of the types:
        public static bool IsEmpty<T>(this Stack<Z,T> stack) { return true; }
        public static bool IsEmpty<TRest, T>(this Stack<S<TRest>, T> stack) { return false; }
 
    }
}

One fun thing about this is that we never have to check whether we’ve hit the bottom of the stack (node.next == null). Since the size is statically part of the type, and there’s no Pop for an empty list, we never have to check for this situation.

This posting on the Java core library mailing list proposes to replace the current quicksort with a new, 2-pivot quicksort.

Ordinary quicksort in Haskell looks something like this:

quicksort [] = []
quicksort (pivot:rest) =
   quicksort [x| x ← rest, x ≤ pivot] ++
   [pivot] ++
   quicksort [x| x ← rest, x > pivot]

Note the similarity of this to the notation for ‘invariants’ used in the email:

[ <= p | >= p ]

Similarly, for the 2-pivot invariants (I have corrected a typo in this):

[ < P1 | P1 <= & <= P2 | > P2 ]

We can derive the code:

quicksort2 [] = []
quicksort2 (only:[]) = [only]
quicksort2 (first:second:rest) =
   quicksort2 [x|x ← rest, x < p1] ++
   [p1] ++
   quicksort2 [x|x ← rest, p1 ≤ x && x ≤ p2] ++
   [p2] ++
   quicksort2 [x|x ← rest, p2 < x]
  where (p1,p2) = if first ≤ second then (first,second) else (second,first)

Which does, in fact, run faster than the ordinary 1-pivot quicksort.

n-pivot quicksort

Of course, this being Haskell we want to take things to their logical conclusions, and allow any number of pivots to be used. My formulation follows:

quicksortN _ [] = []
quicksortN _ (x:[]) = [x]
quicksortN n xs
    | n > length xs = quicksortN ((n `div` 2) `max` 1) xs
    | otherwise = part True pivots
    where
      (takexs,dropxs) = splitAt n xs
      pivots = quicksortN 1 takexs
      part False (p2:[])      = quicksortN n [x|x ← dropxs, p2 < x]
      part False (p1:p2:rest) = quicksortN n [x|x ← dropxs, p1 < x && x ≤ p2] ++ [p2] ++ part False (p2:rest)
      part True  (p1:[])      = quicksortN n [x|x ← dropxs, x ≤ p1] ++ [p1] ++ quicksortN n [x|x ← dropxs, p1 < x] -- remember me? :)
      part True  (p1:p2:rest) = quicksortN n [x|x ← dropxs, x ≤ p1] ++ [p1] ++ part False (p1:p2:rest)

Now there are only 3 things that I can see that are left ‘tweakable’;

• how to reduce ‘n’ when it is greater than length of the list
  • dividing by 2 here
• how to sort the pivots
  • using a recursive call to the 1-pivot version of itself here
• whether or not to reduce the number of pivots as sorting continues
  • ‘no’ chosen here

I have no idea whether these are sensible defaults

Which n should I choose?

Finally, some code for the timings. You’ll need timeit to run it.

main = do
 gen ← getStdGen
 let n = 10^6
 let r = take n $ randoms gen :: [Int]
 print $ sum r -- force list
 timeIt . putStrLn . ("quicksort = "++) . show . sum $ quicksort r
 timeIt . putStrLn . ("quicksort2 = "++) . show . sum $ quicksort2 r
 mapM_ (\x → timeIt . putStrLn . (("quicksortN "++ show x++" = ")++) . show . sum $ quicksortN x r)
    [1..20]

Sample output:

$ ./sort
858124546
quicksort = 858124546
CPU time:   4.60s
quicksort2 = 858124546
CPU time:   3.66s
quicksortN 1 = 858124546
CPU time:   5.19s
quicksortN 2 = 858124546
CPU time:   3.77s
quicksortN 3 = 858124546
CPU time:   3.28s
quicksortN 4 = 858124546
CPU time:   3.27s
quicksortN 5 = 858124546
CPU time:   3.06s
quicksortN 6 = 858124546
CPU time:   3.04s
quicksortN 7 = 858124546
CPU time:   3.04s
quicksortN 8 = 858124546
CPU time:   3.07s
quicksortN 9 = 858124546
CPU time:   3.10s
quicksortN 10 = 858124546
CPU time:   3.14s
quicksortN 11 = 858124546
CPU time:   3.22s
quicksortN 12 = 858124546
CPU time:   3.31s
quicksortN 13 = 858124546
CPU time:   3.30s
quicksortN 14 = 858124546
CPU time:   3.37s
quicksortN 15 = 858124546
CPU time:   3.54s
quicksortN 16 = 858124546
CPU time:   3.41s
quicksortN 17 = 858124546
CPU time:   3.52s
quicksortN 18 = 858124546
CPU time:   3.65s
quicksortN 19 = 858124546
CPU time:   3.63s
quicksortN 20 = 858124546
CPU time:   3.77s

As expected, generalized quicksort performs slower than the equivalent hard-coded quicksorts for specific values of n, but higher values of n than 1 or 2 can perform better.

Preamble; choice of implementation

I decided to go with the latest and greatest version of Scheme: R⁶RS. There are currently two implementations available under Ubuntu Linux: Ikarus and Ypsilon. I installed both so I wouldn’t be swayed by any tempting extensions to the standard. Despite this, I ended up using Ikarus for most testing, as it ran quite a lot faster, although Ypsilon gave much better stack traces.

I also used the streams library for dealing with infinite lists (SRFI 41), which is included with both implementations.

Note: I’ll be providing all the code so that you should be able to copy-paste it into a new file and run it.

(import (rnrs)
    (rnrs r5rs (6)) ; provides 'delay' & 'force'
    (streams))

The Machine

Turing sought to capture the essence of computation. For this purpose he constructed an idealized machine, which can read and write symbols to an infinitely long piece of tape. We are going to model these idealized machines and see what they can do.

First, we want some types to represent the a-machines (the a is for automatic). Each machine has a set of states (m-configurations) it can be in, each of which contains a mapping from a set of symbols to a list of actions and a new m-configuration. I decided that the state would be my basic unit of construction, but twiddled the meaning of m-configuration a tiny bit so that instead of having each configuration contain a mapping from symbols, I would instead have a list of configurations, each one with a list of which symbols activate it. I ended up with the following:

; type for a configuration
(define-record-type m-cfg
            (fields symbols operations next))

This defines a new record type called m-cfg with the fields symbols, operations and next. The define-record-type form defines a constructor (make-m-cfg) and accessors for each field (m-cfg-*). Rather than have a ‘machine’ type, I decided that this would just be left implicit; if we know what the current state is then we can follow the links to next whenever we want to.

The Tape

Now, each machine operates upon an infinitely long ‘tape’. To model this, I use two streams, which are infinite lists. One is the infinite length of the tape to the left of the machine, and the other is the infinite length of the tape to the right of the machine. I decided that the first item in the right list would be the current item that the machine is reading.

; type for a tape (modeled as two stacks)
(define-record-type tape
            (fields left right min max index))

You will note that there are also the fields min, max, and index. These are solely used to track how much of the tape the machine has “visited”. Without this information, we would not know how much of the tape to show when we want to look at it; and since it is infinitely long, this could be a problem!

Representing operations

The operations that a machine can perform consist of:

• Move right
• Move left
• Print symbol
• Erase symbol
• Halt

I implemented these as an enumeration, just in case, but I didn’t actually end up utilizing any of the enumeration features.

; the operations available
(define-enumeration op
  (right left erase print halt)
  op-set)

There is a small difficulty with this representation: the ‘print’ operation needs to be able to take an argument. I decided that operations would always be passed around as lists, and that only ‘print’ would have a second element in the list: its argument. Here is a little shorthand to make this representation easier:

; these need to be lists, because print takes an argument
(define L (list (op left)))
(define R (list (op right)))
(define P (lambda (c) (list (op print) c)))
(define E (list (op erase)))
(define H (list (op halt)))

Thus, we can represent a list of operations like this: (list R R (P #\A) L E H)—that’s right, right, print ‘A’ (Scheme’s syntax for characters is a little weird), left, erase, halt.

Moving around on the tape

Here is a simple tape; it is completely empty. Note that I use the symbol 'empty to represent empty places on the tape. stream-constant makes an infinite stream of the value(s) supplied.

(define empty-tape
  (make-tape (stream-constant 'empty) (stream-constant 'empty) 0 0 0))

Next we need code to actually implement the operations described above. It is fairly straightforward, but we also have to keep track of the index and max/min points on the tape:

; Tape manipulation:
(define (current-symbol tape)
  (stream-car (tape-right tape)))
 
(define (move-item from to)
  (values (stream-cdr from)
      (stream-cons (stream-car from) to)))
 
(define (move-right tape)
  (let-values (
      [(right left) (move-item (tape-right tape) (tape-left tape))]
      [(min max) (update-min-max (tape-min tape) (tape-max tape) (+ 1 (tape-index tape)))])
    (make-tape left right min max (+ 1 (tape-index tape)))))
 
(define (move-left tape)
  (let-values (
      [(left right) (move-item (tape-left tape) (tape-right tape))]
      [(min max) (update-min-max (tape-min tape) (tape-max tape) (- (tape-index tape) 1))])
    (make-tape left right min max (- (tape-index tape) 1))))
 
(define (print tape symbol)
  (make-tape
    (tape-left tape)
    (stream-cons symbol (stream-cdr (tape-right tape)))
    (tape-min tape)
    (tape-max tape)
    (tape-index tape)))
 
(define (erase tape)
  (print tape 'empty))
 
(define (update-min-max min max i)
  (cond 
    [(< i min) (values i max)]
    [(> i max) (values min i)]
    [else      (values min max)]))

Notice that erase is actually redundant because we can just print 'empty. We also want a “dispatcher” of sorts that takes a value representing an operation and performs that operation. This is where passing the argument around with the ‘print’ came in useful:

; performs an operation on a tape, returns new tape
(define (perform-op tape oper)
  (case (car oper)
    [(op left) (move-left tape)]
    [(op right) (move-right tape)]
    [(op print) (print tape (cadr oper))]
    [(op erase) (erase tape)]
    [(op halt) #f])) ; ungraceful halt!

Running the machine

Now that we have the operations implemented, we can almost run a state against a tape. First we need to figure out just which of the configurations of the state to run. This procedure receives a list of configurations (but they are all part of the same ‘state’) and a symbol, and picks the first configuration that has a matching symbol. Note that the configurations can also have the symbol 'any, which matches anything.

; finds the correct rule to follow for this symbol
(define (find-cfg machine symbol)
  (find (lambda (cfg) (or
                (find [lambda (s) (eqv? symbol s)]
                  (m-cfg-symbols cfg))
                (find [lambda (s) (eqv? 'any s)]
                  (m-cfg-symbols cfg))))
        machine))

Now that we have a way to find out which rule to perform, and how to perform it, we can run it against a tape. This procedure advances the machine to the next state, performing all the operations needed. It returns the new state and a new tape.

; runs a machine forward one step
(define (run-machine tape machine)
  (let ([cfg (find-cfg (force machine) (current-symbol tape))])
    (list (fold-left perform-op tape (m-cfg-operations cfg))
          (m-cfg-next cfg))))

Displaying the tape

Being able to run the machine against a tape isn’t much good if we can’t see the result, so here’s a procedure to print out what it looks like. This is where we need the indexes we kept track of on the tape, so we know when to stop printing.

(define (print-tape tape)
  ; move as far left as possible
  (let ([leftTape (go-far-left tape)])
    ; now go right
    (do ([t leftTape])
      ((> (tape-index t) (tape-max tape)) t)
      (when (eqv? (tape-index tape) (tape-index t))
              (display "["))
      (cond
          [(eqv? (current-symbol t) 'empty) (display ".")]
          [else (display (current-symbol t))])
      (when (eqv? (tape-index tape) (tape-index t))
              (display "]"))
      (set! t (move-right t)))))
 
; moves to the far left of the tape (as far as has been travelled)
(define (go-far-left tape)
  (do ([t tape])
    ((= (tape-index t) (tape-min tape)) t)
     (set! t (move-left t))))

I print out a ‘.’ for each blank, and surround the current symbol with square brackets (actually I’ve changed that to be underlining in this blog post).

Turing Machines!

Now we can test it! Here is my definition of Turing’s first published machine:

; Turing's first published machine!
(define m1 (letrec 
  ([b (delay (list
       (make-m-cfg (list 'any) (list (P #\0) R) c)))]
   [c (delay (list
       (make-m-cfg (list 'any) (list R) e)))]
   [e (delay (list
       (make-m-cfg (list 'any) (list (P #\1) R) f)))]
   [f (delay (list
       (make-m-cfg (list 'any) (list R) b)))])
  b))

Now you see why we needed to import ‘delay’… since Scheme is a strictly-evaluated language, we can’t just letrec each state in terms of the others, so I wrap each one up in delay, then force it in one place; just before we use it in the run-machine procedure.

Other than that it is fairly straightforward, each state has a list of m-configurations, each of which has a list of what symbols it accepts, the actions to take, and the next state to move to. After we letrec, we have the initial state defined as the ‘machine’—in this case, b.

We can run this machine like so:

(define (go t m)
 (do ([tm (list t m)])
  ((eqv? tm #f) (car tm))
  (print-tape (car tm))
  (newline)
  (set! tm (run-machine (car tm) (cadr tm)))))
 
(go empty-tape m1)

This gives us the following output:

.
0.
0..
0.1.
0.1..
0.1.0.
0.1.0..
0.1.0.1.
0.1.0.1..
0.1.0.1.0.
...

This looks correct: P0, R, R, P1, R, R, etc.

Machine redux

The next machine Turing gives is the same as the first one, only in a different form:

; the same machine, only smaller
(define m2 (letrec
         ([b (delay (list
           (make-m-cfg (list 'empty) (list (P #\0)) b)
           (make-m-cfg (list #\0) (list R R (P #\1)) b)
           (make-m-cfg (list #\1) (list R R (P #\0)) b)))])
         b))

It has only one state, but changes depending on what the current symbol is. This produces the same output as the first machine, but in fewer steps:

.
0
0.1
0.1.0
0.1.0.1
0.1.0.1.0
0.1.0.1.0.1
0.1.0.1.0.1.0
0.1.0.1.0.1.0.1
0.1.0.1.0.1.0.1.0
...

You may be wondering why Turing leaves blanks between each printed symbol. He used the convention that only the ‘even’ squares (termed F-squares) would be the output. The ‘odd’ squares (termed E-squares) would be used as a scratch-pad.

Machine-à-trois

The third machine is a little more interesting. Whereas the first two printed out 01010101..., this prints 01011011101111011111...:

; Turing's third machine
(define m3 (letrec
         ([b (delay (list
                   (make-m-cfg (list 'any) (list (P #\ǝ) R (P #\ǝ) R (P #\0) R R (P #\0) L L) o)))]
          [o (delay (list
                  (make-m-cfg (list #\1) (list R (P #\x) L L L) o)
                  (make-m-cfg (list #\0) (list) q)))]
          [q (delay (list
                  (make-m-cfg (list #\0 #\1) (list R R) q)
                  (make-m-cfg (list 'empty) (list (P #\1) L) p)))]
          [p (delay (list
                  (make-m-cfg (list #\x) (list E R) q)
                  (make-m-cfg (list #\ǝ) (list R) f)
                  (make-m-cfg (list 'empty) (list L L) p)))]
          [f (delay (list
                  (make-m-cfg (list 'empty) (list (P #\0) L L) o)
                  (make-m-cfg (list 'any) (list R R) f)))])
         b))

With the output:

.
ǝǝ0.0
ǝǝ0.0
ǝǝ0.0
ǝǝ0.0..
ǝǝ0.0.1
ǝǝ0.0.1
ǝǝ0.0.1
ǝǝ0.0.1
ǝǝ0.0.1
ǝǝ0.0.1
ǝǝ0.0.1..
ǝǝ0.0.1.0
ǝǝ0.0.1x0
ǝǝ0.0.1x0
ǝǝ0.0.1x0
ǝǝ0.0.1x0
ǝǝ0.0.1x0..
ǝǝ0.0.1x0.1
ǝǝ0.0.1x0.1
ǝǝ0.0.1.0.1
ǝǝ0.0.1.0.1
ǝǝ0.0.1.0.1..
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1
ǝǝ0.0.1.0.1.1..
ǝǝ0.0.1.0.1.1.0
ǝǝ0.0.1.0.1.1x0
ǝǝ0.0.1.0.1x1x0
ǝǝ0.0.1.0.1x1x0
ǝǝ0.0.1.0.1x1x0
ǝǝ0.0.1.0.1x1x0
ǝǝ0.0.1.0.1x1x0
ǝǝ0.0.1.0.1x1x0..
ǝǝ0.0.1.0.1x1x0.1
ǝǝ0.0.1.0.1x1x0.1
ǝǝ0.0.1.0.1x1.0.1
ǝǝ0.0.1.0.1x1.0.1
ǝǝ0.0.1.0.1x1.0.1..
ǝǝ0.0.1.0.1x1.0.1.1
ǝǝ0.0.1.0.1x1.0.1.1
ǝǝ0.0.1.0.1x1.0.1.1
ǝǝ0.0.1.0.1x1.0.1.1
ǝǝ0.0.1.0.1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1.1..
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1..
ǝǝ0.0.1.0.1.1.0.1.1.1.0
ǝǝ0.0.1.0.1.1.0.1.1.1x0
ǝǝ0.0.1.0.1.1.0.1.1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0
ǝǝ0.0.1.0.1.1.0.1x1x1x0..
ǝǝ0.0.1.0.1.1.0.1x1x1x0.1
ǝǝ0.0.1.0.1.1.0.1x1x1x0.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1..
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1x1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1..
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1x1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1..
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1
ǝǝ0.0.1.0.1.1.0.1.1.1.0.1.1.1.1

Here we can see the scratch-pad being used. We can also see the beginnings of a pattern of execution; notice how the machine returns to the beginning of the tape after completing each set of 1s.

Opus Magnum

Next up was my main challenge. One of the major contributions of Turing’s paper was to display a machine which could emulate any other machine you wanted. In essence, you don’t actually need lots of different machines. You can just build one, and it can do anything any of the other machines can do! This is the principle behind modern general-purpose computers.

This is a very big, and complex machine. Some of the intricacies not involved in the other machines are:

• m-functions; that is, configurations which accept parameters—luckily, this was surprisingly easy to implement using Scheme; I simply wrap the delayed code inside another lambda
• m-configurations which are parametrized over all symbols on the machine—this is solved by just mapping a lambda over the list of symbols
• variadic m-functions—solved by the magic of case-lambda
• poorly-scanned journal document containing Fraktur and Greek letters

But not only was this by far the biggest and most complex machine supplied by Turing, I had read texts mentioning unspecified bugs in the program.

… and sure enough, I ran into a ‘bug’. There were some configurations used in the machine which weren’t defined in the paper! At first I thought this was due to the low resolution of the PDF, but even enhancing the image didn’t help.

After much supplication and burnt offerings to the God of the Internet, I managed to find that:

Someone else did the work already

Yay!

More specifically, I found a paper entitled Understanding Turing’s Universal Machine — Personal Style in Program Description (which is unfortunately not available to the public), a marvelous paper that not only explains the errors made in detail, but also provides a nice, corrected version of Turing’s exposition of his machine.

After painstakingly re-checking all the states again, I arrived at this:

; need this to generate a couple of cfgs
(define u-symbols (list
		   #\A #\C #\D #\0 #\1
		   #\u #\v #\w #\x #\y #\z
		   #\; #\L #\R #\N
		   #\∷ #\:
		   ))
 
; yo dawg
(define u (letrec
        (
         [f (lambda (C B a) (delay (list
                (make-m-cfg (list #\ǝ) (list L) (f1 C B a))
                (make-m-cfg (list 'any) (list L) (f C B a)))))]
         [f1 (lambda (C B a) (delay (list
                (make-m-cfg (list a) (list) C)
                (make-m-cfg (list 'empty) (list R) (f2 C B a))
                (make-m-cfg (list 'any) (list R) (f1 C B a)))))]
         [f2 (lambda (C B a) (delay (list
                (make-m-cfg (list a) (list) C)
                (make-m-cfg (list 'empty) (list R) B)
                (make-m-cfg (list 'any) (list R) (f1 C B a)))))]
         [fdash (lambda (C B a) (delay (list
                (make-m-cfg (list 'any) (list) (f (l C) B a)))))]
         [fdashdash (lambda (C B a) (delay (list
                (make-m-cfg (list 'any) (list) (f (r C) B a)))))]
         [r (lambda (C) (delay (list
                (make-m-cfg (list 'any) (list R) C))))]
         [l (lambda (C) (delay (list
                (make-m-cfg (list 'any) (list L) C))))]
         [q (case-lambda
              [(C) (delay (list
                (make-m-cfg (list 'empty) (list R) (q1 C))
                (make-m-cfg (list 'any) (list R) (q C))))]
         [(C a) (delay (list
                (make-m-cfg (list 'any) (list) (q (q1 C a)))))])]
         [q1 (case-lambda
               [(C) (delay (list
                (make-m-cfg (list 'empty) (list) C)
                (make-m-cfg (list 'any) (list R) (q C))))]
               [(C a) (delay (list
                (make-m-cfg (list a) (list) C)
                (make-m-cfg (list 'any) (list L) (q1 C a))))])]
         [pe (lambda (C b) (delay (list
                (make-m-cfg (list 'any) (list) (f (pe1 C b) c #\ǝ)))))]
         [pe1 (lambda (C b) (delay (list
                (make-m-cfg (list 'empty) (list (P b)) C)
                (make-m-cfg (list 'any) (list R R) (pe1 C b)))))]
         [pe2 (lambda (C a b) (delay (list
                (make-m-cfg (list 'any) (list) (pe (pe C b) a)))))]
         [c (lambda (C B a) (delay (list
                (make-m-cfg (list 'any) (list) (fdash (c1 C) B a)))))]
         [c1 (lambda (C) (delay (map
                    (lambda (b) (make-m-cfg (list b) (list) (pe C b)))
                      u-symbols)))]
         [ce (case-lambda
               [(C B a) (delay (list
                (make-m-cfg (list 'any) (list) (c (e C B a) B a))))]
               [(B a) (delay (list
                (make-m-cfg (list 'any) (list) (ce (ce B a) B a))))])]
         [ce2 (lambda (B a b) (delay (list
                (make-m-cfg (list 'any) (list) (ce (ce B b) a)))))]
         [ce3 (lambda (B a b g) (delay (list
                (make-m-cfg (list 'any) (list) (ce (ce2 B b g) a)))))]
         [ce5 (lambda (B a b g d e) (delay (list
                (make-m-cfg (list 'any) (list) (ce3 (ce2 B d e) a b g)))))] ; added
         [cp (lambda (C U F a b) (delay (list
                (make-m-cfg (list 'any) (list) (fdash (cp1 C U b) (f U F b) a)))))]
         [cp1 (lambda (C U b) (delay (map
                   (lambda (g) (make-m-cfg (list g) (list) (fdash (cp2 C U g) U b)))
                     u-symbols)))]
         [cp2 (lambda (C U g) (delay (list
                (make-m-cfg (list g) (list) C)
                (make-m-cfg (list 'any) (list) U))))]
         [cpe (case-lambda 
                [(C U F a b) (delay (list
                 (make-m-cfg (list 'any) (list) (cp (e (e C C b) C a) U F a b))))]
                [(U F a b) (delay (list
                 (make-m-cfg (list 'any) (list) (cpe (cpe U F a b) U F a b))))])]
         [e (case-lambda
               [(C) (delay (list
                (make-m-cfg (list #\ǝ) (list R) (e1 C))
                (make-m-cfg (list 'any) (list L) (e C))))]
               [(B a) (delay (list 
                (make-m-cfg (list 'any) (list) (e (e B a) B a))))]
               [(C B a) (delay (list 
                (make-m-cfg (list 'any) (list) (f (e1 C B a) B a))))])]
         [e1 (case-lambda
               [(C) (delay (list
                (make-m-cfg (list 'empty) (list) C) 
                (make-m-cfg (list 'any) (list R E R) (e1 C))))]
               [(C B a) (delay (list
                (make-m-cfg (list 'any) (list E) C)))])]
         [con (lambda (C a) (delay (list
                (make-m-cfg (list #\A) (list L (P a) R) (con1 C a))
                (make-m-cfg (list 'any) (list R R) (con C a)))))]
         [con1 (lambda (C a) (delay (list
                (make-m-cfg (list #\A) (list R (P a) R) (con1 C a))
                (make-m-cfg (list #\D) (list R (P a) R) (con2 C a)))))]
         [con2 (lambda (C a) (delay (list
                (make-m-cfg (list #\C) (list R (P a) R) (con2 C a))
                (make-m-cfg (list 'any) (list R R) C))))]
         [b (delay (list
                (make-m-cfg (list 'any) (list) (f b1 b1 #\∷))))]
         [b1 (delay (list
                (make-m-cfg (list 'any) (list R R (P #\:) R R (P #\D) R R (P #\A) R R (P #\D)) anf)))] ; added "R R PD"
         [anf (delay (list
                (make-m-cfg (list 'any) (list) (q anf1 #\:))))] ; corrected from "(g ..."
         [anf1 (delay (list
                (make-m-cfg (list 'any) (list) (con fom #\y))))]
         [fom (lambda () (list
                (make-m-cfg (list #\;) (list R (P #\z) L) (con fmp #\x))
                (make-m-cfg (list #\z) (list L L) fom)
                (make-m-cfg (list #\ǝ) (list H) fom)
                (make-m-cfg (list 'any) (list L) fom)))]
         [fmp (delay (list
                (make-m-cfg (list 'any) (list) (cpe (e (e anf #\x) #\y) sim #\x #\y))))] ; corrected
         [sim (delay (list
                (make-m-cfg (list 'any) (list) (fdash sim1 sim1 #\z))))]
         [sim1 (delay (list
                (make-m-cfg (list 'any) (list) (con sim2 'empty))))]
         [sim2 (delay (list
                (make-m-cfg (list #\A) (list) sim3)
                (make-m-cfg (list 'any) (list L (P #\u) R R R) sim2)))] ; corrected from "R ..."
         [sim3 (delay (list
                (make-m-cfg (list #\A) (list L (P #\y) R R R) sim3)
                (make-m-cfg (list 'any) (list L (P #\y)) (e mf #\z))))]
         [mf (delay (list
                (make-m-cfg (list 'any) (list) (q mf1 #\:))))] ; corrected from "(g mf ..."
         [mf1 (delay (list
                (make-m-cfg (list #\A) (list L L L L) mf2)
                (make-m-cfg (list 'any) (list R R) mf1)))]
         [mf2 (delay (list
                (make-m-cfg (list #\C) (list R (P #\x) L L L) mf2)
                (make-m-cfg (list #\:) (list) mf4)
                (make-m-cfg (list #\D) (list R (P #\x) L L L) mf3)))]
         [mf3 (delay (list
                (make-m-cfg (list #\:) (list) mf4)
                (make-m-cfg (list 'any) (list R (P #\v) L L L) mf3)))]
         [mf4 (delay (list
                (make-m-cfg (list 'any) (list) (con (l (l mf5)) 'empty))))]
         [mf5 (delay (list
                (make-m-cfg (list 'empty) (list (P #\:)) sh)
                (make-m-cfg (list 'any) (list R (P #\w) R) mf5)))]
         [sh (delay (list
                (make-m-cfg (list 'any) (list) (f sh1 inst #\u))))]
         [sh1 (delay (list
                (make-m-cfg (list 'any) (list L L L) sh2)))]
         [sh2 (delay (list
                (make-m-cfg (list #\D) (list R R R R) sh3) ; corrected from "sh2"
                (make-m-cfg (list 'any) (list) inst)))]
         [sh3 (delay (list
                (make-m-cfg (list #\C) (list R R) sh4)
                (make-m-cfg (list 'any) (list) inst)))]
         [sh4 (delay (list
                (make-m-cfg (list #\C) (list R R) sh5)
                (make-m-cfg (list 'any) (list) (pe2 inst #\0 #\:))))]
         [sh5 (delay (list
                (make-m-cfg (list #\C) (list) inst)
                (make-m-cfg (list 'any) (list) (pe2 inst #\1 #\:))))]
         [inst (delay (list                                 ; note that inst1 is forced here!
                                                            ; this is because it is a zero-arity varargs
                (make-m-cfg (list 'any) (list) (q (l (inst1)) #\u))))] ; corrected from "(g ..." 
         [inst1 (case-lambda
                 [() (delay (map
                       (lambda (a) (make-m-cfg (list a) (list R E) (inst1 a)))
                       u-symbols))]
                 [(x) (case x
                      [(#\L) (delay (list (make-m-cfg (list 'any) (list) (ce5 ov #\v #\y #\x #\u #\w))))]
                      [(#\R) (delay (list (make-m-cfg (list 'any) (list) (ce5 ov #\v #\x #\u #\y #\w))))]
                      [(#\N) (delay (list (make-m-cfg (list 'any) (list) (ce5 ov #\v #\x #\y #\u #\w))))])])]
         [ov (delay (list
               (make-m-cfg (list 'any) (list) (q (r (r ov1)) #\A))))] ; changed from original
         [ov1 (delay (list
               (make-m-cfg (list #\D) (list) (e anf))
               (make-m-cfg (list 'empty) (list (P #\D)) (e anf))))]
         ) b)) ;; start in state 'b'

I don’t think I can blame Turing much for the errors in the paper. It seems as though some arose through printing typos, and attempting to debug this baroque machine by hand, on paper, would have been a difficult task. (I don’t think he even had Visual Studio!)

A machine in a machine

Of course, the final test of this is to see whether this machine can actually emulate another like it is supposed to. I defined a short procedure to set up a machine on a tape according to Turing’s ingenious encoding.

(define (setup-tape tape inits) 
  (let ([t (move-right (print (move-right (print tape #\ǝ)) #\ǝ))]
  	[inits (append inits (list #\∷))])
   (go-far-left (fold-left (lambda (tape init)
  	       (move-right (move-right (print tape init)))) 
	     t inits))))

A quick explanation of Turing’s encoding

The idea is to first simplify, then encode the states. Turing noted that many of the m-configurations’ operations could be considered redundant:

• as mentioned above, instead of having “erase” we can simply print the ‘blank’ symbol
• instead of not modifying the symbol, we simply print the symbol already present

There are then only three types of operation each state needs to perform:

1. print something and go left
2. print something and go right
3. print something and stay put

States which have a sequence of operations can be split into a series of states, each of which transfers control to the next one.

Now, if we encode all the symbols and configurations as numbers, we can write them out. Turing chose to encode them via this scheme:

• configurations’ numbers are the symbol ‘D’ followed by n ‘A’s, where n is the number of the state
• symbols’ numbers are similar, with ‘D’ followed by n ‘C’s. (Turing set ‘blank’ to always be symbol 0, represented as simply “D”)

So to encode each configuration, we write down its number, the symbol it accepts, the symbol it outputs, which direction to move, and which state to go to next. We prefix each configuration with ‘;’. (When we input it into the machine we also sandwich the whole thing between ‘ǝǝ’ and ‘∷’.)

Here is the example which Turing gives in his paper. I have formatted it to make it easier to read. Can you tell what it does?

(define example (setup-tape empty-tape (list 
			     #\; #\D #\A             #\D #\D #\C     #\R #\D #\A #\A
			     #\; #\D #\A #\A         #\D #\D         #\R #\D #\A #\A #\A
			     #\; #\D #\A #\A #\A     #\D #\D #\C #\C #\R #\D #\A #\A #\A #\A
			     #\; #\D #\A #\A #\A #\A #\D #\D         #\R #\D #\A)))

If we translate the symbols to get numbers we have the following:

; 1 0 0 R 2
; 2 0 . R 3
; 3 0 1 R 4
; 4 0 . R 1

This machine prints alternately 0.1.0.1.. In fact, when I first typed it up, I left off an ‘A’ on the 3rd state and couldn’t figure out why the machine was printing 0.11111...!

Just to show it works

Here is some output of the universal machine running the ’0101′ machine. I have only included a snippet as the machine takes a while to get to this stage. You’ll also notice the output format is different from the other machines; this one outputs some state information and the output of the emulated machine (in this case, 0 and 1), separated by colons. So far, after a minute or so, the machine has output 010

.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C
.ǝǝ;.D.A.D.DuCuR.DyAyAy;.D.A.A.D.D.R.D.A.A.A.;.D.A.A.A.D.D.C.C.R.D.A.A.A.A.;.D.A.A.A.A.D.D.R.D.A.∷.:.D.A.D.:.0.:.D.C.D.A.A.D.:.D.C.D.D.A.A.A.D.:.1.:.D.C.D.D.C.C.D.A.A.A.A.D.:.D.CvDvDvCvCvDxD.A.D.:.0.:.D.C

And that’s enough for today! Feel free to post corrections, additions, your own Turing machines, and so on

If we want to restrict ourselves to assigned, non-control, non-private Unicode characters, then by my reckoning that gives us 129,775 available characters.

wget http://unicode.org/Public/UNIDATA/UnicodeData.txt
awk -F ';' UnicodeData.txt -f countUnichars.awk | bc

countUnichars.awk source:

BEGIN { print "ibase=16" } # set bc to hex mode
 
$2 ~ /Private/ { # skip any lines with "private" in the description
    getline;
}
 
n { # if n is set, then print the range for bc to calculate
    printf("(%s-%s+1)+",$1,n);
    n="";
}
 
$2 ~ /First>/ { # set n if the start of a range
    n=$1;
    getline;
}
 
$3 !~ "C." { # otherwise count anything that isn't some kind of a control character
    i++;
}
 
END { # print out the count of everything else
    printf("%X\n",i)
}

This means we can store exactly 2377 bits (297 bytes) per message (this is ), so if we use a 16-colour palette we can store about 594 pixels (), which can almost reproduce the Mona Lisa thumbnail in the contest page.

So here is a very quick and simple plugin; it simply puts a button on the toolbar that you can click when you want to stop playback after the current song. I based the toolbar button code on Alexandre Rosenfeld’s lastfm-queue plugin, since I had no idea where to start with that

Download it here, and put it into ~/.gnome2/rhythmbox/plugins/stop_after_song/. Activate it in Rhythmbox’s plugin dialog.

I didn’t like the fact that in the old code there were places where explicit try points needed to be included. It seems that these arose because the ‘obsolete’ syntax was tacked-on to the EBNF for the normal syntax, creating much overlap. Since I merged the syntaxes together, there are no explicit try points needed (there are some implicit ones, I believe, such as in optional). This makes the code both faster and easier to understand.

module Text.Email.Validation (isValid)
where
 
import Text.Parsec
import Text.Parsec.Char
import Data.Char (chr)
 
isValid :: String -> Bool
isValid x = 	either (const False) (const True) (valid x)
 
simply = (>> return ())
-- simply converts a parser returning something to a parser returning nothing
 
valid :: String -> Either ParseError ()
valid = parse addrSpec ""
 
addrSpec = localPart >> char '@' >> domain >> eof
 
localPart = dottedAtoms
domain = dottedAtoms <|> domainLiteral 
 
dottedAtoms = simply $ (optional cfws >> (atom <|> quotedString) >> optional cfws)
	`sepBy1` (char '.')
atom = simply $ many1 atomText
atomText = simply $ alphaNum <|> oneOf "!#$%&'*+-/=?^_`{|}~"
 
domainLiteral =  between (optional cfws >> char '[') (char ']' >> optional cfws) $
	many (optional fws >> domainText) >> optional fws
domainText = ranges [[33..90],[94..126]] <|> obsNoWsCtl
 
quotedString = between (char '"') (char '"') $
	many (optional fws >> quotedContent) >> optional fws
quotedContent = quotedText <|> quotedPair
quotedText = ranges [[33],[35..91],[93..126]] <|> obsNoWsCtl
quotedPair = char '\\' >> (vchar <|> wsp <|> lf <|> cr <|> obsNoWsCtl <|> nullChar)
 
cfws = simply $ many (comment <|> fws)
fws = (many1 wsp >> optional (crlf >> many1 wsp))
	<|> (many1 (crlf >> many1 wsp) >> return ())
 
comment = simply $ between (char '(') (char ')') $
	many (commentContent <|> fws)
commentContent = commentText <|> quotedPair <|> comment
commentText = ranges [[33..39],[42..91],[93..126]] <|> obsNoWsCtl
 
nullChar = simply $ char '\0'
wsp = simply $ oneOf " \t"
cr = simply $ char '\r'
lf = simply $ char '\n'
crlf = simply $ cr >> lf
vchar = ranges [[0x21..0x7e]]
obsNoWsCtl = ranges [[1..8],[11,12],[14..31],[127]]
ranges = simply . oneOf . map chr . concat

This now passes all of Dominic Sayer’s tests that it is meant to—the domain validation used in Dominic Sayer’s tests is more strict than RFC5322 specifies. Expect this to change!

For those who’d like to know, email addresses that now parse that didn’t before include the often-used (‘|’ is merely to indicate the end of whitespace):

I.                        |
 am.                  |
 a.      |
 nice.|
 guy@(yeah)you.com

In recent times there have been several calls for websites to properly validate email addresses. Invariably, the compiled regex from Perl’s RFC822 is pasted up as The Way To Do It. The problem with this is (as the source code from the Perl module notes) is that email addresses cannot be validated by a simple regex (due to requiring parenthesis-matching). The Perl code addresses this by first stripping out all comments and then parsing via regex.

With this in mind, I thought that implementing the Addr-Spec specification from RFC 5322 (only released less than 6 months ago) might be a good test of the Haskell library Parsec. So, without further ado I went ahead and translated the EBNF from RFC 5322 directly into Parsec.

The mapping is something like this:

juxtaposition

>>

/

<|>

*

many

1*

many1

[]

optional

Here is the result:

import Text.Parsec
import Text.Parsec.Char
import Data.Char (chr)
 
isValid :: String -> Bool
isValid x = let result = valid x in
	either (const False) (const True) result
 
valid :: String -> Either ParseError ()
valid x = parse addrSpec "" x
 
ignore x = x >> return ()
 
addrSpec = localPart >> char '@' >> domain >> eof
 
localPart = dotAtom <|> quotedString <|> obsLocalPart <?> "local part"
domain = dotAtom <|> domainLiteral <|> obsDomain <?> "domain"
 
domainLiteral = optional cfws >> char '[' >>
		many ( optional fws >> dtext) >>
		optional fws  >> char ']' >> optional cfws
		<?> "domain literal"
 
ranges = oneOf . map chr . concat
vchar = ranges [[0x21..0x7E]] -- from Backus-Naur RFC
dtext = ranges [[33..90],[94..126]] <|> obsDtext
qtext = ranges [[33],[35..91],[93..126]] <|> obsQtext
atext = alphaNum <|> oneOf "!#$%&'*+-/=?^_`{|}~"
ctext = ranges [[33..39],[42..91],[93..126]] <|> obsCtext
wsp = char ' '
	<|> char '\t'
	<?> "space or tab"
 
cr = char '\r' <?> "carriage return"
lf = char '\n' <?> "line feed"
crlf = cr >> lf <?> "CRLF line ending"
 
-- # modification: added try
cfws = try (many1 (optional fws >> comment) >> optional fws) <|> ignore fws
-- # modification from RFC: adding try because of overlap
fws = try (optional (many wsp >> crlf) >> many1 wsp)
	<|> many1 wsp
	<|> obsFws
 
-- # modification: added try
comment = between (char '(') (char ')') (many (try (optional fws >> ccontent)) >> optional fws)
	<?> "comment"
ccontent = ignore ctext
	<|> ignore quotedPair
	<|> comment
 
atom = optional cfws >> many1 atext >> optional cfws
dotAtomText = many1 atext >> many (char '.' >> many1 atext)
dotAtom = optional cfws >> dotAtomText >> optional cfws
 
-- # other change from RFC -- merge prefix
quotedPair = char '\\' >> ((vchar <|> wsp) <|> obsQp)
qcontent = qtext <|> quotedPair
quotedString = optional cfws >>	char '\"' >> many (optional fws >> qcontent) >>
	optional fws >>	char '\"' >> optional cfws
	<?> "quoted string"
 
-- # Obsolete syntax
obsNoWsCtl = ranges [[1..8],[11..12],[14..31],[127]]
obsCtext = obsNoWsCtl
obsDtext = obsNoWsCtl <|> quotedPair
obsQtext = obsNoWsCtl
-- # change: see above
obsQp = (char (chr 0) <|> obsNoWsCtl <|> lf <|> cr)
obsLocalPart = word >> many (char '.' >> word) >> return ()
obsDomain = atom >> many (char '.' >> atom) >> return ()
obsFws = many1 wsp >> many (crlf >> many1 wsp) >> return []
word = atom <|> quotedString

A note before I continue: since Parsec by default does no backtracking (in order to remain efficient), there are a couple of places (two that I’ve found so far) where the original EBNF needs to be changed slightly. I have noted these in the source above. It is possible there are a couple more places that need fixing, but I haven’t run this against a large test suite yet to find them. (They are most likely to be in the ‘obsolete syntax’ section.)

And of course, some demonstrations (keep in mind that there is an extra level of escaping operating here… where relevant I’ve included the unescaped email address in a comment):

isValid "porges@example.com" == True
isValid "porges@@example.com" == False
isValid "\"porges@\"@example.com" == True -- # "porges@"@porg.es
isValid "\"por(g)es@\"@example.com" == True -- # "por(g)es@"@porg.es
isValid "porges(comment)@example.com" == True
isValid "porges(comme(nests)nt)@example.com" == True
isValid "porges(comme(nests)nt())@example.com" == True
isValid "porges(()comme(nests)nt())@example.com" == True
isValid "()porges(()comme(nests)nt())@example.com" == True
isValid "((lol)porges(()comme(nests)nt())@example.com" == False
isValid "((lol))porges(()comme(nests)nt())@example.com" == True
isValid "(lol))porges(()comme(nests)nt())@example.com" == False
isValid "((lol))porges(()comme(nests)nt())@example.com" == True
isValid "\"s\\\0\"@example.com" == True
-- # "s\NUL"@example.com, where NUL is actually the
-- # null character! Yep, can't strlen() on email addresses...

I managed to find this post on email addresses by Phil Haack which has the following tests:

[("\"Abc\\@def\"@example.com",True),
("\"Fred Bloggs\"@example.com",True),
("\"Joe\\Blow\"@example.com",True),
("\"Abc@def\"@example.com",True),
("customer/department=shipping@example.com",True),
("$A12345@example.com",True),
("!def!xyz%abc@example.com",True),
("_somename@example.com",True),
("NotAnEmail",False),
("@NotAnEmail",False),
("\"test\\\\blah\"@example.com",True),
("\"test\\blah\"@example.com",True),
-- # Phil gets false for this, which I think is wrong
-- # (Dominic Sayers notes the same at the end of the comment thread)
("\"test\\\rblah\"@example.com",True),
("\"test\rblah\"@example.com",False),
("\"test\\\"blah\"@example.com",True),
("\"test\"blah\"@example.com",False),
("customer/department@example.com",True),
("$A12345@example.com",True),
("!def!xyz%abc@example.com",True),
("_Yosemite.Sam@example.com",True),
("~@example.com",True),
(".wooly@example.com",False),
("wo..oly@example.com",False),
("pootietang.@example.com",False),
(".@example.com",False),
("\"Austin@Powers\"@example.com",True),
("Ima.Fool@example.com",True),
("\"Ima.Fool\"@example.com",False),
("\"Ima Fool\"@example.com",False),
("Ima Fool@example.com",False)]

Next job is to test it against this batch…

Update: Yep, fails in several areas with the obsolete syntax. I’ve fixed one above. (Note that I’m not concerned with the failures in the domain part of the address, as the RFC5322 EBNF for this is more liberal than the tests require.)

Might have to refactor the syntax… there is a large overlap with the obsolete syntax. (Or just use ‘try’, but that’s not so efficient.)

s = cmds.sphere()
r = cmds.sphere(s, query=True, radius=True)
cmds.sphere(s, edit=True, radius=10)

Nasty.

So, here is a wrapper to make things nicer:

import maya.cmds as cmds
 
class Wrapper(object):
	def __init__(self, other):
		self.__dict__['me'] = other()
 
	def __getattr__(self, thing):
		i = 0 if cmds.attributeQuery(thing, node=self.__dict__['me'][0], exists=True) else 1
		return cmds.getAttr(self.__dict__['me'][i] + "." + thing)
 
	def __setattr__(self, thing, value):
		i = 0 if cmds.attributeQuery(thing, node=self.__dict__['me'][0], exists=True) else 1
		cmds.setAttr(self.__dict__['me'][i] + "." + name, value)

The above example is now:

s = Wrapper(cmds.sphere)
r = s.radius
s.radius = 10

How did we all miss that before?

—Peaker

The answer (of course!) is that while we might have, they didn’t. For example, Edward Kmett’s category-extras package has had Control.Monad.Indexed available for use in Haskell since early last year, while the concept for its implementation in Haskell has been around since at least 2005 (see the paper “Monadic Augment and Generalised Short Cut Fusion”). Needless to say, Oleg has been there too.

For me, this prompted a vaguely-related revelation. Say that we have a Category class:

class Category (⤳) where
	id ∷ a ⤳ a
	(∘) ∷ (b ⤳ c) → (a ⤳ b) → (a ⤳ c)

I had recently been wondering how categories other than (→), such as graphs (and even the trivial category) fit into this model. Now that I have figured it out, the answer is simple; it is a matter of arity. Graphs and other datastructure-related categories don’t have the extra type arguments (unless you’re working in a dependently-typed language), and so can’t fit into this. What we want to do is fake the extra arguments, by having some extra ones that we can just ignore (the name, FF, is short for FakeFake, i.e. two fake arguments).

newtype FF f a b = FF { unFF :: f }

Now we can define the trivial category!

class Category (FF ()) where
	id = FF ()
	f ∘ g = FF ()

Viewed in this light, it is obvious why (as Dan notes in the original post) Monoids give rise to Monads and Categories give rise to ParametrizedMonads, as we have:

class Monoid m ⇒ Category (FF m) where
	id = FF mempty
	(FF f) ∘ (FF g) = FF (mappend f g)

Next, we want to write implementations for all our old Monads in the new PMonad class. We might do it like this (with a new newtype for the different arguments of this datatype, RealFakeFakeReal):

newtype RFFR f x y a = RFFR { unRFFR ∷ (f a) }
 
class PMonad m where
	(>>=) ∷ m s1 s2 a → (a → m s2 s3 b) → m s1 s3 b
	return ∷ a → m s s a
 
instance (Old.Monad m) ⇒ PMonad (RFFR m) where
	(RFFR x) >>= f = RFFR (x Old.>>= unRFFR ∘ f)
	return x = RFFR (Old.return x)

There is a big issue that this highlights. It is hard to work with types of different arities in Haskell. (This is noted in the ‘quantified contexts’ proposal.)

As an example, since we now know that every Monoid is a Category, we might think to drop Monoids and just use Categories everywhere (I should note that this is actually a silly idea !). Unfortunately, we can’t just do this as we’ll have to use FF wrappers/unwrappers everywhere:

x = unFF $ FF [1,2,3] ∘ FF [4,5,6]

I have not yet found a workaround for this. Help would be nice

David R. MacIver has recently written this blog post about cleaning up a set of tags. This blog post, on the other hand, is about a nice old Unix tool called ‘awk’.

Awk is one of those programs that is often overlooked. It is really a small domain-specific language for processing text. In some ways it resembles sed, but it is more powerful, and it especially excels at processing line- and field-structured input.

Processing cite-u-like’s data

First of all, before we begin I want to say this this isn’t a criticism of David’s work. Using a general-purpose language like Ruby to process data comes with several benefits. This post is to explain the benefits of awk and what it excels at doing.

Now, cite-u-like’s tag data comes in a pipe-separated format, so we have input like this:

42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|ecoli
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|metabolism
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|barabasi
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|networks
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|control
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|engineering
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|robustness
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|networks
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|strogatz
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|survey

From now on, whenever you see x-separated data, I want you to scream ‘USE AWK!’

Awk scripts look something like this:

pattern { expression }

pattern is used to match against a record, and if successful, the action in the braces (expression) will be carried out. But what is a record? Awk allows you to define what a line is using the variable RS (short for ‘record separator’). By default it is set to \n (so that each line is a record), which is what we want here.

Within the expression you can refer to fields of the current record, using the syntax $n. $0 refers to the whole record, while $1,$2… refer to individual fields.

Awk also allows you to define what the field separator will be via the variable FS.

So how do we set these variables? Awk has two special patterns BEGIN and END, which run before and after everything else. In this case, we want the fields to be separated by |, so we use the pattern:

BEGIN { FS = "|" }

This is rather long-winded, so mawk (an implementation of awk) also allows you to set FS via a command-line option -F.

For this application, we want the last field of the record (we could hard-code it as $4, but we’re exploring awk!). Awk provides the number of fields in the record as the variable NF (number of fields), so we want to access this field. We do so using the record syntax $ and the variable NF.

So to put all this together, what we want is the command:

awk -F "|" '{ print $NF }'

We set the field separator to “|”, and write an awk expression to print the last field of each record. Since we left the pattern empty, this expression is evaluated on every record.

Awk vs. Ruby

So what good is learning all this, anyway? There are a couple of reasons:

• awk is standard on *nix operating systems. In order to use David’s code I had to install Ruby; with awk you can generally count on it being there.
• awk is fast (I should say at least in the interpreter ‘mawk’ which is the standard for Ubuntu). On my machine the awk version completes in under a third of the time that it takes for David’s Ruby version to complete. An interesting thing was that the awk version didn’t even max out the CPU, indicating that it is IO-bound and would go faster if I had faster disks (I’m currently on a laptop).
• Awk is ideal for record- and field-based input, as I hope this post will show you

Filtering data with awk

After the above we use sort and uniq the same as David does to get the results in the following form:

 212595 bibtex-import
 157136 no-tag
  27926 elegans
  27887 celegans
  27825 c_elegans
  27795 nematode
  27738 wormbase
  27736 caenorhabditis_elegans
  18933 review
  15280 all-articles

David uses the following to filter out lines with no alphabetical content:

ruby -ne 'puts $_ if !($_ =~ /^[^a-z]+$/)'

We can use awk’s patterns to do the same thing:

$0 ~ /[a-zA-Z]/

Here we use the ~ (match) operator to write a pattern that matches only the records with an alphabetical character in them. (Remember that $0 refers to the entire record.) Notice also that we can leave off the expression after the pattern, because it defaults to { print }, which is exactly what we want.

In this case, awk really shines. On my machine it outperforms the Ruby version by a factor of 8–9.

Programming with awk

The third task that David does is to consolidate all tags which are differentiated only by hyphens or underscores. That is, ‘a-tag’, ‘atag’, and ‘a_tag’ should all be considered the same. We choose which one to put into the output by whichever one is used the most times (and then we normalize the tag by replacing ‘-’ with ‘_’ so there are only underscores in the output).

Here is David’s code to do the job:

tag_counts = {}
STDIN.lines.each{|l| c, t = l.split; tag_counts[t.strip] = c.to_i }
duplicates = Hash.new{|h, k| h[k] = []}
tag_counts.keys.each{|k| duplicates[k.gsub(/-|_/, "")] << k }
duplicates.values.each{|vs| vs.sort!{|x, y| tag_counts[y] <=> tag_counts[x]} }
 
new_tag_counts = {}
duplicates.values.each{|vs| new_tag_counts[vs[0].gsub(/(_|-)+/, "_")] = vs.map{|v| tag_counts[v]}.inject(0, &:+)}
puts new_tag_counts.to_a.sort{|x, y| y[1] <=> x[1]}.map{|t, c| " #{c} #{t}" }

I’m not going to explain it here, because that’s not the point of the post

Here’s my awk script:

{ tag_counts[$2] = $1 }
END {
	for (tag in tag_counts)
	{
		normtag=tag
		gsub(/-|_/,"",normtag)
 
		count=tag_counts[tag]
		sum[normtag]+=count
 
		if (count > max[normtag])
		{
			names[normtag]=tag
			max[normtag]=count
		}
	}
 
	for (tag in names)
	{
		finaltag=names[tag]
		gsub(/(-|_)+/,"_",finaltag)
		print " " sum[tag] " " finaltag
	}
}