porges

Introduction

David R. MacIver has recently written this blog post about cleaning up a set of tags. This blog post, on the other hand, is about a nice old Unix tool called ‘awk’.

Awk is one of those programs that is often overlooked. It is really a small domain-specific language for processing text. In some ways it resembles sed, but it is more powerful, and it especially excels at processing line- and field-structured input.

Processing cite-u-like’s data

First of all, before we begin I want to say this this isn’t a criticism of David’s work. Using a general-purpose language like Ruby to process data comes with several benefits. This post is to explain the benefits of awk and what it excels at doing.

Now, cite-u-like’s tag data comes in a pipe-separated format, so we have input like this:

42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|ecoli
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|metabolism
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|barabasi
42|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:05.373798+00|networks
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|control
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|engineering
43|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:25:51.839281+00|robustness
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|networks
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|strogatz
44|61baaeba8de136d9c1aa9c18ec3860e8|2004-11-04 02:26:33.156319+00|survey

From now on, whenever you see x-separated data, I want you to scream ‘USE AWK!’

Awk scripts look something like this:

pattern { expression }

pattern is used to match against a record, and if successful, the action in the braces (expression) will be carried out. But what is a record? Awk allows you to define what a line is using the variable RS (short for ‘record separator’). By default it is set to \n (so that each line is a record), which is what we want here.

Within the expression you can refer to fields of the current record, using the syntax $n. $0 refers to the whole record, while $1,$2… refer to individual fields.

Awk also allows you to define what the field separator will be via the variable FS.

So how do we set these variables? Awk has two special patterns BEGIN and END, which run before and after everything else. In this case, we want the fields to be separated by |, so we use the pattern:

BEGIN { FS = "|" }

This is rather long-winded, so mawk (an implementation of awk) also allows you to set FS via a command-line option -F.

For this application, we want the last field of the record (we could hard-code it as $4, but we’re exploring awk!). Awk provides the number of fields in the record as the variable NF (number of fields), so we want to access this field. We do so using the record syntax $ and the variable NF.

So to put all this together, what we want is the command:

awk -F "|" '{ print $NF }'

We set the field separator to “|”, and write an awk expression to print the last field of each record. Since we left the pattern empty, this expression is evaluated on every record.

Awk vs. Ruby

So what good is learning all this, anyway? There are a couple of reasons:

• awk is standard on *nix operating systems. In order to use David’s code I had to install Ruby; with awk you can generally count on it being there.
• awk is fast (I should say at least in the interpreter ‘mawk’ which is the standard for Ubuntu). On my machine the awk version completes in under a third of the time that it takes for David’s Ruby version to complete. An interesting thing was that the awk version didn’t even max out the CPU, indicating that it is IO-bound and would go faster if I had faster disks (I’m currently on a laptop).
• Awk is ideal for record- and field-based input, as I hope this post will show you

Filtering data with awk

After the above we use sort and uniq the same as David does to get the results in the following form:

 212595 bibtex-import
 157136 no-tag
  27926 elegans
  27887 celegans
  27825 c_elegans
  27795 nematode
  27738 wormbase
  27736 caenorhabditis_elegans
  18933 review
  15280 all-articles

David uses the following to filter out lines with no alphabetical content:

ruby -ne 'puts $_ if !($_ =~ /^[^a-z]+$/)'

We can use awk’s patterns to do the same thing:

$0 ~ /[a-zA-Z]/

Here we use the ~ (match) operator to write a pattern that matches only the records with an alphabetical character in them. (Remember that $0 refers to the entire record.) Notice also that we can leave off the expression after the pattern, because it defaults to { print }, which is exactly what we want.

In this case, awk really shines. On my machine it outperforms the Ruby version by a factor of 8–9.

Programming with awk

The third task that David does is to consolidate all tags which are differentiated only by hyphens or underscores. That is, ‘a-tag’, ‘atag’, and ‘a_tag’ should all be considered the same. We choose which one to put into the output by whichever one is used the most times (and then we normalize the tag by replacing ‘-’ with ‘_’ so there are only underscores in the output).

Here is David’s code to do the job:

tag_counts = {}
STDIN.lines.each{|l| c, t = l.split; tag_counts[t.strip] = c.to_i }
duplicates = Hash.new{|h, k| h[k] = []}
tag_counts.keys.each{|k| duplicates[k.gsub(/-|_/, "")] << k }
duplicates.values.each{|vs| vs.sort!{|x, y| tag_counts[y] <=> tag_counts[x]} }
 
new_tag_counts = {}
duplicates.values.each{|vs| new_tag_counts[vs[0].gsub(/(_|-)+/, "_")] = vs.map{|v| tag_counts[v]}.inject(0, &:+)}
puts new_tag_counts.to_a.sort{|x, y| y[1] <=> x[1]}.map{|t, c| " #{c} #{t}" }

I’m not going to explain it here, because that’s not the point of the post

Here’s my awk script:

{ tag_counts[$2] = $1 }
END {
	for (tag in tag_counts)
	{
		normtag=tag
		gsub(/-|_/,"",normtag)
 
		count=tag_counts[tag]
		sum[normtag]+=count
 
		if (count > max[normtag])
		{
			names[normtag]=tag
			max[normtag]=count
		}
	}
 
	for (tag in names)
	{
		finaltag=names[tag]
		gsub(/(-|_)+/,"_",finaltag)
		print " " sum[tag] " " finaltag
	}
}

That’s right, you can use awk to do some ordinary programming tasks! Arrays are used using the usual syntax, and awk even has a foreach-style loop for looping over them. I’ll walk through the rest of the script slowly.

First we apply an expression to each record, creating an array of tags and their counts:

{ tag_counts[$2] = $1 }

Then, once everything has finished (the END pattern), we process this array. For each tag, we do the following:

1. Normalize the tag (the gsub function overwrites the variable, so we have to make a copy):

   normtag=tag
   gsub(/-|_/,"",normtag)

   (Notice the similarity between this and Ruby’s equivalent normtag.gsub!(/-|_/, "")!)

2. Get the count for that tag and add it to the count for the normalized version:

   count=tag_counts[tag]
   sum[normtag]+=count

   Like in PHP and Perl, if a value is not present in an array it is automatically added with a default value.

3. Next we check to see if the current tag is the commonest version of the normalized tag, and if so we save its name and count in two other arrays:

   if (count > max[normtag])
   {
   	names[normtag]=tag
   	max[normtag]=count
   }

   Notice again the usefulness of a default value for nonexistent keys: count > max[normtag] will be true if max[normtag] doesn’t exist.

Now we have all we need to print out the answer. For each tag we normalize it to the final version (remembering to make a copy):

finaltag=names[tag]
gsub(/(-|_)+/,"_",finaltag)

Then we print out the line (concatenation is done by simply juxtaposing variables or strings):

print " " sum[tag] " " finaltag

If you’ve been watching closely you’ll notice there is a small difference between the awk and the Ruby scripts; Ruby sorts before outputting, while the awk version will come out in a non-defined order. This is fine! We can use the standard *nix tool ‘sort’ to sort the lines;

awk -f consolidate_tags.awk < tags | sort -nr > fixed_tags

(Note that this fits in with the *nix philosophy of ‘do one thing well’ and reusing small components.)

Again, the awk version outperforms the Ruby by a factor of 3–4.

Reading external commands and files

Unfortunately there doesn’t seem to be a command-line stemming program (a quick Perl script would suffice but it isn’t what we’re here for!), so I’ll skip that stage (here’s one of the aforementioned weaknesses of a non-general-purpose language). Instead we’ll go straight to implementing stopwords.

Here’s David’s Ruby code again:

require "set"
 
stopwords = Set[*
  IO.read("smart.txt").lines.reject{|x| x =~ /^#/}.map(&:strip)
]
 
STDIN.lines.each do |l|
  c, t = l.split
  puts l unless stopwords.include? t.strip
end

And here’s my equivalent in awk:

BEGIN {
    while (getline < "smart.txt")
    { stopwords[$0] = 1 }
}
!($2 in stopwords)

Here I use the ‘getline’ function which does as its name suggests. We make an array of all the stopwords, with 1 as a placeholder value. The pattern is then short and simple: Print every record where the tag isn’t in the stopwords (again, we can leave off the expression to print the whole record).

Note: There is a discrepancy here: David claims this eliminated 46 tags, while I get a value of 368 for both my awk code and his Ruby code.

Again, the Ruby takes about 8 times as long to execute.

Conclusion

Here’s a couple of points:

• Record- or field-oriented data? Think awk.

• Don’t discount it just because it’s venerable. It is very well-suited to its task.

• pattern { expression } syntax is extremely flexible.

• Awk’s regular expressions are fast.

Nowadays everybody wanna talk
    like they got something to say
But nothing comes out
    when they move their lips
Just a bunch of gibberish
And motherf—kers act
    like they forgot about Awk