Comments on: Counting Characters in UTF-8 Strings Is Fast(er)

By: 几个汇编/C高性能处理UTF-8的帖子

几个汇编/C高性能处理UTF-8的帖子 — Mon, 17 Nov 2008 11:40:19 +0000

[...] COUNTING CHARACTERS IN UTF-8 STRINGS IS FAST(ER) http://porg.es/blog/counting-characters-in-utf-8-strings-is-faster [...]

By: Colin Percival

Colin Percival — Thu, 05 Jun 2008 09:24:36 +0000

I’ve done even better.

Vectorization yields a 2-4x speedup over your code: http://www.daemonology.net/blog/2008-06-05-faster-utf8-strlen.html

By: Porges

Porges — Thu, 05 Jun 2008 00:47:14 +0000

Hi Savvu, I implemented this as:

int tbl[] = {
    1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,1
};
 
int savvu_strlen(char *s)
{ 
    int cnt = 0;
    while(*s) cnt += tbl[(*s++ >> 4) & 0xF];
    return cnt;
}

It is consistently the slowest or second-to slowest.

I tried implementing it with byte-skipping:

int tbl[] = {
1,1,1,1,1,1,1,1, //one-byte
1,1,1,1, //invalid, but don't go into infinite loop
2,2, //two-byte starter
3, //three-byte starter
4 //four-byte starter
};
 
int porges_strlen(char *s)
{
        int cnt = 0;
        int i = 0;
        while(s[i]) { i += tbl[(s[i] >> 4) & 0x0f]; ++cnt; }
        return cnt;
}

This version is only faster on the byte-skipping tests, and is still about half the speed of what I posted.

By: Savvu

Savvu — Wed, 04 Jun 2008 14:00:44 +0000

while(*s) cnt += tbl[*s++ >> 4]; return cnt;

Setting up tbl is left as an excercise to the reader. If your chars are signed you also need an AND mask.

By: Porges

Porges — Wed, 04 Jun 2008 07:55:23 +0000

Whoops

I think the URL must have tripped me up; I’m so used to Bob Smith being /~bsmith/…

By: matthew

matthew — Wed, 04 Jun 2008 07:15:59 +0000

BTW, his name is Kragen, not Ragen.