SysAdmin'ish Blog

Since the Debian Jessie security upgrade of firefox-esr1) Firefox on our servers stopped offering "Print to LPR" and showed only "Print to file" in the printer dialog.

It turned out that the upgrade apparently relies on GTK3 contrary to the old version that still used GTK2. Previously, we got "Print to LPR" working by adding gtk-print-backends = "file,lpr" to the file /etc/gtk-2.0/gtkrc. There is no /etc/gtk-3.0/gtkrc and running Firefox with strace suggested that system-wide GTK3 settings are read from /etc/gtk-3.0/settings.ini instead. To restore "Print to LPR" use

gtk-print-backends = file,lpr

:!: Hint: Do not put double quotes around "file,lpr" as we did for GTK2!

For reference please note that the following warnings issued by Firefox right when opening the print dialog are misleading but harmless:

** (firefox-esr:17106): WARNING **: Failed to execute child process
"dbus-launch" (No such file or directory)

** (firefox-esr:17106): WARNING **: Couldn't connect to D-Bus system
bus, Could not connect: No such file or directory

** (firefox-esr:26127): WARNING **: GDBus.Error:org.freedesktop.DBus.Error.ServiceUnknown:
The name org.freedesktop.Avahi was not provided by any .service files

These are in fact caused by the default printbackends "cups" and "cloudprint" and the lack of D-Bus. They go away if only "file" and "lpr" are used.

Playing up

User writes

my i pad is playing up … says your server does not like my mails

My reply

Too bad. Apparently, you got up on the wrong side of bed. Try another.

OK nothing happened

A good friend of mine, let's call hir "user", asked for advice since hir PC refused to apply updates and complained about low disk space on "C:". I sent the following instructions:

Press Win-R to open the "Run" dialog, type cleanmgr [Enter] or click [OK]. When it offers to compress old files, don't. All else is fine. But you may want to make sure you have "Windows update cleanup" checked (see screenshot). Screenshot: http://fam.tuwien.ac.at/~schamane/.../resources/disk_cleanup.png

Then I received the following reply:

I wanted to use the link you sent but when I click [OK] in Disk Cleanup nothing happens.

We all know these stories. But it's still bamboozling every time it happens for real, and one of your own users is the main actor ;-)

Adding RFC5322.From to Sendmail logs

Sendmail, by default, logs only the envelope "From" address (AKA RFC5321.From). I wanted to add the "From:" headers of outgoing messages (AKA RFC5322.From).2) So, in sendmail.mc I added the following at the bottom:

Klog syslog
HFrom: $>+LogFrom
R$*        $: $(log From: $&{currHeader} $)

(Note that the block of spaces in the last line have to be TABs. For details of the syntax see e.g. The whole scoop of the configuration file.)

Basically, this defines a map "log" and a ruleset "LogFrom", looks for "From:" and "rewrites" (R) any content via "log" to syslog, prepending "From:". The results in mail.log should look similar to

Sep 21 22:03:34 localhost sm-mta[1364]: u8LK3Y6P001364: From: "Full Name" <sender@example.net>
Sep 21 22:03:35 localhost sm-mta[1364]: u8LK3Y6P001364: from=<sender@mail.example.net.net>, size=724389, class=0, nrcpts=1, msgid=<005c01d21443$2bf4e600$83deb200$@example.net>, proto=ESMTP, daemon=MTA, relay=mail.example.net []
Sep 21 22:03:38 localhost sm-mta[1365]: u8LK3Y6P001364: to=<receiver@example.org>, delay=00:00:04, xdelay=00:00:03, mailer=local, pri=754801, dsn=2.0.0, stat=Sent

These sendmail.cf instructions make Sendmail log "From:" headers of both incoming and outgoing mail. Only the latter was my original intention, however, I appreciate the additional log data for incoming mail, too. It helps debugging and answering user queries since they often and understandably enough only mention RFC5322.From addresses.

2 From:s

However, much to my surprise I found that Sendmail sometimes logged 2 "From:" headers even though the messages definitely had only 1. Turned out that Sendmail indeed inspects contents of attachments if their "Content-type" is message/rfc822.

My colleague Johann Klasek, a true Sendmail aficionado, even digged the source code to confirm my empirical findings. He also came up with the following rules to skip "From:" headers of attachments by means of a macro {InMessage} and an if-then type rewriting:

Klog syslog
HFrom: $>+LogFrom
HContent-Type: $>+CheckMessage
R$*           $: <$&{InMessage}> $1                      are we in a message/rfc822 part?
R<> $*        $: $(log From: $&{currHeader} $) $1        if not then log the From: header
Rmessage/$*   $: $(macro {InMessage} $@ YES $) $1        set mark if Content-Type is message/*

In order to log "From:" headers of attached messages with e.g. keyword "MIME-From:" instead of "From:" add the following line before SCheckMessage (i.e. at the end of ruleset SLogFrom):

R<YES> $*     $: $(log MIME-From: $&{currHeader} $) $1   embedded MIME part From: header

(Again, aligned whitespaces are TAB characters.)

Many thanks to Johann Klasek for his great help!

Suppress IP of authenticated senders in Sendmail

Messages sent by our users got sometimes caught in spam filters due to the fact that Sendmail (like any other reasonable MTA) adds a "Received: from" header with the IP address of the client who submitted the message. This quickly becomes a problem if the IP address is dynamically assigned, e.g. to a user's smartphone, but was previously abused, and is still listed on blacklists.

I was already aware of this problem, and I had it "fixed" on my Postfix installations long ago. I thought, it should be easy to do the same with Sendmail, i.e. suppress the IP address, or if need be the complete header. Much to my surprise I could hardly find any instruction for how to do this. And I thought that should be an FAQ!? Especially, since Sendmail is well established and way older than most other major MTAs.

I could find a few discussions, some 10 years old, of people who wondered how to suppress information that was revealing internal/Intranet network structures. I found practically nobody who was looking for how to surpass the spam filter issue. BTW, I did use more than 1 search engine ;-) Still, I am afraid that now that I am writing this people will send me lmgtfy links.

Anyway, what I did find was, for instance, Removing Sender’s IP Address From Email’s Received: From Header. This page popped up frequently when I searched for how to suppress headers, and it seems to be one of very few indeed. It even addresses my main problem. The author's approach is totally valid, and s/he explains things well and gives pointers. However, I did feel comfortable with completely removing the "from" part, and with redefining confRECEIVED_HEADER without honoring the defaults. I was also afraid that this might even break some other things (like our own spam filter rules).

Another solution that I could find about 2 times goes 1 big step furhter: It suggested to remove the HReceived line(s) from submit.cf altogether. This does work, and it does make some sense for submit.cf, however, only if 2 sendmail daemons are used where 1 is running with the submit.cf and listens to port 587. But, my server is a Debian box with only 1 daemon, and I thought there should be no need for a 2nd.


Joel's Compendium of Total Knowledge (search for Received:) is the only page I found that suggests what I thought was reasonable, i.e. introduce an if-then ($?…$|…$.) evaluating the variable {auth_type} to check whether the client has been authenticated:

define(`confRECEIVED_HEADER', `$?{auth_type}...

By the time I found Joel's advice I was already refreshing my sendmail.cf skills (The whole scoop of the configuration file was a valuable reintroduction). So, eventually, I tried to come up with my own version for confRECEIVED_HEADER.

On Debian, confRECEIVED_HEADER is originally defined in /usr/share/sendmail/cf/m4/cfhead.m4:

define(`_REC_AUTH_', `$.$?{auth_type}(authenticated')
define(`_REC_FULL_AUTH_', `$.$?{auth_type}(user=${auth_authen} $?{auth_author}author=${auth_author} $.mech=${auth_type}')
define(`_REC_HDR_', `$?sfrom $s $.$?_($?s$|from $.$_)')
define(`_REC_END_', `for $u; $|;
define(`_REC_TLS_', `(version=${tls_version} cipher=${cipher} bits=${cipher_bits} verify=${verify})$.$?u')
define(`_REC_BY_', `$.by $j ($v/$Z)$?r with $r$. id $i$?{tls_version}')
define(`confRECEIVED_HEADER', `_REC_HDR_
        _REC_AUTH_$?{auth_ssf} bits=${auth_ssf}$.)

So, even if you don't understand the sendmail.cf syntax you can see that confRECEIVED_HEADER is actually built by concatenating other variables. I wanted to stick to this format in order to re-use the defaults, _REC_HDR_ should be kept (to the best of my knowledge this is also an RFC requirement), nevertheless, it should be rewritten for authenticated senders.

So, in /etc/mail/sendmail.mc I added

dnl # suppress IP of authenticated sender
define(`confRECEIVED_HEADER',`$?{auth_type}from auth (localhost []) $|_REC_HDR_$.

Note: The leading spaces are actually 1 TAB!


Authenticated sender

Received: from auth (localhost []) by mail.fam.tuwien.ac.at
   (8.14.4/8.14.4/Debian-4) with ESMTP id sAHCoctq012610
   (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT);
   Mon, 17 Nov 2014 13:50:38 +0100

Intranet handoff

Received: from wiener.fam.tuwien.ac.at (wienernfs [])   
   by mail.fam.tuwien.ac.at (8.14.4/8.14.4/Debian-4) with ESMTP id
   for <schamane@fam.tuwien.ac.at>; Mon, 17 Nov 2014 17:17:45 +0100
Received: by wiener.fam.tuwien.ac.at (Postfix, from userid 501)   
   id AB3F9461; Mon, 17 Nov 2014 17:17:45 +0100 (CET)

How smart is SmartScreen?

Recently, when I had a closer look at our web access logs, I happened to see a GET request coming from an unknown IP address trying to download a file which had a random file name that we haven't had published anywhere, at least not purposely. The file name was created randomly a few hours before from a download/file sharing script which generates a temporary random link for every single download. These links are removed shortly after the download, so the second attempt was futile since the link was already removed.

After a bit of testing, creating more downloads, using different browsers etc., we concluded that the illegitimate requests were probably done on behalf of Microsoft's SmartScreen feature. Every time we accessed a file using Internet Explorer with SmartScreen turned on (and not using Private Browsing) a few hours later we saw (attempted) downloads using GET requests coming from,, and

All 3 of these IP addresses have in common that they lack rDNS, and their WHOIS data shows only major US providers (Level 3 and InterNap). All requests used faked3) User-Agent strings like for instance "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; FunWebProducts; InfoPath.1; .NET CLR 2.0.50727; Dealio Toolbar 3.4; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; Windows Live Messenger 14.0.8117.0416)".

A web research for these IP addresses and SmartScreen didn't provide much information either. However, the IP addresses are known and SmartScreen's approach has been questioned before:

Calculate Sysadmin day dates

System Administrator Appreciation Day, also known as Sysadmin Day, SysAdminDay, SAD or SAAD, is an annual event (…) to show appreciation for the work of sysadmins and other IT workers. It is celebrated on the last Friday in July.4)

I did a bit of research of how to calculate the last Friday of July for any given year. My own solution was to use ncal and sed or awk:

$ YEAR=2015; ncal 7 $YEAR | awk '/^Fr/{print $NF}'

That's the shortest code I could find. ncal delivers a calender of the whole month. awk then prints the last entry of all Fridays. The same can be achieved with sed, of course:

$ YEAR=2015; ncal 7 $YEAR | sed -n 's/^Fr.* \([^ ]\+\) *$/\1/p'
$ YEAR=2015; ncal 7 $YEAR | sed -nr 's/^Fr.* ([^ ]+) *$/\1/p'   # GNU Sed

Note that the output of ncal and colleagues depends on your locale. Consider prepending LC_ALL=C or whatever fits. So, in Bash this could become LC_ALL=C ncal 7 $YEAR.

Another very nice trick was provided by cbuckley using date. However, it's a bit less obvious ;-)

$ YEAR=2015; date -d${YEAR}0801-$(date -d${YEAR}0801+2days +%u)days +%d

To my surprise, PHP's strtotime is capable of parsing "last Friday of this month" (contrary to GNU date):

$ php -r 'echo date("d", strtotime("last fri of July 2015"))."\n";'

Once I was finished with my own humble attempts I found Last Friday of each month - Rosetta Code ^_^

RSS feeds: Blog posts (abstracts), Blog posts (full text), Comments

See also: SysAdmin'ish Blog Archive & Tag cloud

from 45.9.0esr-1~deb8u1 to 52.2.0esr-1~deb8u1
Since the world of eMail is unstoppably moving towards aligned and authenticated eMail (SPF, DKIM, DMARC, …), and since our mail servers volume is low and we don't need and don't do mail archiving, I was looking for ways to evaluate outgoing eMail by analyzing mail.log. After all, most information was already there. However, I needed the RFC5322.From headers to check domain alignment and make sure we send only eMail we are authorized to send.
HTTP request headers didn't match the known behavior of the pretended browsers.
blog/index.txt · Last modified: 2013-04-21 18:51 by andreas