Courier IMAP & POP3 stats per user

So far, I have been using courier-analog from the Courier Mail Server to generate IMAP bandwidth reports on a per user basis. However, recently, when I copied a user's mailbox of about 600 MB from a remote server via imapsync I was surprised that courier-analog showed only a few KB of traffic. First, I thought that courier-analog only counts outbound traffic, but looking at the source code it seems that it counts only the body values (see below). So, I quickly wrote my own script to get per user stats.

A typical IMAP log line on my server running courier-imap version 4.8.0-3 (from Debian) reads

Jul 29 06:36:47 server imapd-ssl: LOGOUT, user=example, ip=[::ffff:127.0.0.1], \
headers=0, body=0, rcvd=48, sent=292, time=344, starttls=1

Courier IMAP distinguishes headers, bodies, received and sent bytes. Clearly, headers and body can be zero while there is still some traffic. But more importantly, body gives only the bytes of messages sent, apparently:

Jul 23 22:12:28 server imapd-ssl: DISCONNECTED, user=example, ip=[::ffff:127.0.0.1], \
headers=0, body=0, rcvd=14084220, sent=8141, time=123, starttls=1

For simplicity I opted to preprocess the logfile with sed and then sum up the received and sent values with awk.

courier-traffic-imap

#!/bin/bash -e
#
# Calculate traffic of Courier IMAPd per user
 
[[ $1 ]] && DATE="$1" || DATE=$(date +%b\ %e -d yesterday)
[[ $1 == today ]] && DATE=$(date +%b\ %e)
 
LOGFILE=${2:-/var/log/mail.log}
THRESHOLD=2900000 # (almost 3 MB)
UNIT=2  # Bytes (B): 0, KB: 1, MB: 2, GB: 3, ...
 
echo -e "== Courier IMAP traffic stats per user of $DATE ==\n"
 
# We preprocess $LOGFILE with sed, extracting only imapd lines,
# the user name, received and sent bytes.
# Numbers are then summed up by awk, and finally pretty printed
 
# We use uniq to work around a bug in courier imap that fills the logs
# with thousands of identical lines
 
sed -nre "
	s/^$DATE (..:..:..) .* imapd.* user=([^,]*),.* rcvd=([0-9]+), sent=([0-9]+).*$/\\2 \\3 \\4 \\1/p
	" $LOGFILE \
| uniq \
| awk -v threshold=$THRESHOLD -v unit=$UNIT '
	# sum up numbers: uc=count, ui=received, uo=sent, u(i/o)g=global totals
	{ uc[$1]++; ui[$1]+=$2; uo[$1]+=$3; ucg++; uig+=$2; uog+=$3; }
	END {
		# get unit string (ustr) and exponent (uexp)
		split("B KB MB GB TB PB",units); ustr=units[unit+1]; uexp=3*unit
		# print results
		for (user in uc) {
			u[user] = ui[user]+uo[user] # per user sum of received and sent
			if (u[user] > threshold)
				printf "%28s %6d %-2s  [#%5d: r %5d s %5d ]\n",
					user,u[user]/10^uexp,ustr,uc[user],ui[user]/10^uexp,uo[user]/10^uexp \
					| "sort -nr -k 2"
		}
		close("sort -nr -k 2")
		printf "\n%28s %6d %-2s  [#%5d: r %5d s %5d ]\n",
			"-- total --",(uig+uog)/10^uexp,ustr,ucg,uig/10^uexp,uog/10^uexp
	} '

$ bin/courier-traffic-imap 'Jul 23' /var/log/mail.log.1
== Courier IMAP traffic stats per user of Jul 23 ==

                     example    624 MB  [#    3: r     0 s   623 ]
                another-user     90 MB  [#   36: r    40 s    50 ]
            and.another-user     89 MB  [#    7: r    85 s     4 ]
                   and.so.on     68 MB  [#   92: r    10 s    58 ]
                and-so-forth     42 MB  [#   20: r    31 s    10 ]
                
                 -- total --   1002 MB  [# 1027: r   207 s   795 ]

The logfile format is exactly the same. Just replace every imapd with pop3d. Of course, it would be easy to put both into 1 script. However, POP3 traffic is much lower than IMAP traffic. I guess I'll ignore it completely.