Little Personal Email Setup

I offered to write up my email server’s configuration for a friend. We’d gotten into a discussion about the horrors involved in managing all the moving parts for his email setup, and questions about other products, etc, etc. I didn’t think my latest experience with Postfix & Dovecot was all that bad, hence my offer. So, this is my write-up of how I got Postfix, Dovecot, Rspamd, ClamAV all linked up and backed by a PostgreSQL database.

Some basics (getting it installed!)

I’m a FreeBSD user, have been for years. So, for me, this was “go hit the ports collection, compile with the appropriate options.” So, use portsnap fetch extract (or update, if you already have ports installed) and get to installing:

• postgresql95-server (at this writing, 96 is available, should be fine)
• postgresql95-contrib (for crypto support!)
• postfix
• dovecot2

Optionally:

• dovecot2-antispam-plugin
• rspamd
• rmilter (milter for rspamd)
• clamav-milter

The optional content is here, and I’ll get into how I configured it. I’d switched to using rspamd for spam filtering. It was also setup to automatically filter suspected spam into the Junk folder, and learn from emails being moved into/out of the Junk folder (unless being moved to Trash, obviously).

Setting up the database

I’m assuming you’ve done the “initdb” and started up postgresql by this point. On FreeBSD, you’d do this by (as root):

# sysrc postgresql_enable=YES
# service postgresql initdb
# service postgresql start

I created a separate user for the email programs to interact with. ‘postgres’ is the default root-level user in my postgresql server (your user may vary based on system).

# psql -U postgres -d template1

create user email with password 'my-password';
create database mail with owner email;
\c mail
create extension pgcrypto;
\q

Now we can create some tables and views. I chose to do this in a way I’d been using for a long time, which isn’t always the easiest to manage, but prevents you from doing stupid things. There will be 3 tables - transport, user, and aliases - and two views (which the programs will largely be using for reference) - view_aliases, and view_users.

# psql -U email -d mail

create table transport (
id serial primary key,
domain varchar(70) not null unique, -- adjust size to suit your needs
destination varchar(20) not null default 'virtual:');

create table users (
id serial primary key,
username varchar(128) not null,
password varchar(128) not null,
domain_id int references transport(id),
enabled boolean not null default true);
create unique index users_username_domain_idx on test_users (lower(username), domain_id);

create table aliases (
id serial primary key,
username varchar(50) not null,
domain_id int references transport(id),
destination varchar(128) );
create unique index alias_username_domain_idx on aliases (lower(username), domain_id);

-- and now the views
CREATE VIEW view_users AS
SELECT (users.username || '@' || transport.domain) AS email, users.password
FROM users
LEFT JOIN transport ON users.domain_id = transport.id;

CREATE VIEW view_aliases AS
SELECT (aliases.username || '@' || transport.domain) AS email, aliases.destination
FROM aliases
LEFT JOIN transport ON aliases.domain_id = transport.id;

This gives us our table structure. Obviously, I want to be able to receive my email, so let’s insert a single entry for testing, and so you can see how it works together. I’ll add an alias just to show that interaction too.

insert into transport (domain) VALUES ('binarydragon.com');
-- I'm going to assume that this gave is id=1 for the rest of this:
insert into users (username, domain_id, password) values ('kovus', 1, '{MD5}' || crypt('mypassword', gen_salt('MD5')));
-- add an alias too.
insert into aliases (username, domain_id, destination) values ('postmaster', 1, 'kovus@binarydragon.com');

Ok, I’m sure that last bit might leave someone scratching their heads asking “why doesn’t the alias destination take advantage of the transport table?” It’s simply because you might want to accept email for an address that then gets forwarded off to another domain. Such as accepting a vanity email for “kovus@binarydragon.com” and forwarding that off to “kovus@othermailhost.com.” Obviously I don’t own othermailhost.com, and won’t accept email for it, so it doesn’t go in my transport table (as we’ll see later), but I can pass it on. The reason to use the transport table to address we do receive is to avoid typographical errors.

Configuring postfix (basics)

So, onto the meat of the configuration. We have our data, so now we can start configuring postfix.

I’ve sort of stripped my main.cf config file down to the minimums. I’ve added some comments to try and make it easier to understand why.

# some basic directories
queue_directory = /var/spool/postfix
command_directory = /usr/local/sbin
daemon_directory =/usr/local/libexec/postfix
data_directory = /var/db/postfix

# since postfix isn't going to manage the actual maildirs
# this value is sort of inconsequential.
mail_owner = postfix

# host configuration
myhostname = mail.binarydragon.com
inet_protocols = ipv4, ipv6
# use the machine's real IPs for inet_interfaces.
# I did want to show that you can use ipv6 addr's too!
inet_interfaces = 10.10.10.10, fe80::1
mynetworks_style = host

# ~40MB message size limit.
message_size_limit = 40960000
unknown_local_recipient_reject_code = 550

# I actually have no idea why there's debug statements here.
debug_peer_level = 2
debugger_command =
  PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
  ddd $daemon_directory/$process_name $process_id & sleep 5

# paths to various programs & directories
# These will be defaults on FreeBSD.
sendmail_path = /usr/local/sbin/sendmail
newaliases_path = /usr/local/bin/newaliases
mailq_path = /usr/local/bin/mailq
setgid_group = maildrop
html_directory = /usr/local/share/doc/postfix
manpage_directory = /usr/local/man
sample_directory = /usr/local/etc/postfix
readme_directory = /usr/local/share/doc/postfix

# scan with clamav milter before processing too much
# if you're not using milters, comment these two lines out!
smtpd_milters = unix:/var/run/rmilter/rmilter.sock, unix:/var/run/clamav/clmilter.sock
milter_default_action = reject


# Recipient delimiter is neat.  You should read up on it, especially
# if this is a personal server.  It lets you use
# kovus+anything@binarydragon.com as an email and emails to that address
# will go to kovus@binarydragon.com automatically.
recipient_delimiter = +

# transport maps - enable this to use non-virtual destinations for domains:
# assumes that virtual domains' destination will be "virtual:"
# Lets us forward mail to other servers, eg:
#  destination: smtp:127.127.127.127
transport_maps = pgsql:/usr/local/etc/postfix/sql-transport.cf
# use transport maps in conjunction with relay domains, for relaying data
# to offsite servers (eg, non-'virtual:' destinations)
relay_domains  = pgsql:/usr/local/etc/postfix/sql-relaydomains.cf

# how to handle virtual mailboxes, where to look for info, etc.
virtual_mailbox_domains = pgsql:/usr/local/etc/postfix/sql-virtual.cf
virtual_uid_maps = static:5000
virtual_gid_maps = static:5000
virtual_mailbox_maps = pgsql:/usr/local/etc/postfix/sql-users.cf
virtual_alias_maps = pgsql:/usr/local/etc/postfix/sql-aliases.cf
# this virtual transport points to our dovecot instance.
virtual_transport = lmtp:inet:127.0.0.1:10025

# use sasl for authentication
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
broken_sasl_auth_clients = yes

# enable TLS for security
smtpd_tls_cert_file = /usr/local/etc/ssl/mail.binarydragon.com.bundle.crt
smtpd_tls_key_file  = /usr/local/etc/ssl/mail.binarydragon.com.key
smtpd_tls_security_level = may
smtpd_tls_loglevel = 1
smtpd_tls_received_header = yes

# setup restrictions
# additional detail available on: http://www.postfix.org/SMTPD_ACCESS_README.html#lists
smtpd_helo_restrictions =
  permit_mynetworks,
  permit_sasl_authenticated,
  reject_non_fqdn_hostname,
  reject_invalid_hostname,
  permit
smtpd_sender_restrictions =
  permit_sasl_authenticated,
  permit_mynetworks,
  reject_non_fqdn_sender,
  reject_unknown_sender_domain,
  permit
smtpd_recipient_restrictions =
  permit_mynetworks,
  reject_unauth_pipelining,
  reject_non_fqdn_recipient,
  reject_unknown_recipient_domain,
  permit_sasl_authenticated,
  reject_unauth_destination,
  reject_rbl_client zen.spamhaus.org,
  permit

A few clarifications:

• Setting up SSL certificates is a different topic completely. I will try to do a writeup on how I use StartSSL to create my certificates.
• I do reference a few files, they’re coming.
• I don’t meant to get into the nitty-gritty of the restrictions on the bottom, so the basics:
• Order matters! We should do the least expensive checks first!
• permit_sasl_authenticated - Allow authenticated users to send emails, no matter what stage.
• permit_mynetworks - Allow our emails from our servers (can specify additional servers besides just this one, but that’s beyond this scope).
• reject_* - It’s pretty efficient to block servers that are misconfigured, such as not providing FQDN, having unknown sender domains (invalid), etc. One could also find information on rejecting domains failing an SPF record lookup, but that’s a different scope too.
• reject_rbl_client ... - Realtime Blackhole List. Wonderful concept - use a 3rd party service (like spamhaus) to reject incoming servers in realtime. This helps to reject servers which start sending out spam (due to misconfigurations, or bad users) and the RBLs include ways for admins to get their servers unlisted once the problem is taken care of. It’s a little more expensive to do RBL than the other checks above it, so we do that one last.

Here’s the additional configurations for postfix, for doing the database lookups. Do note that you’ll be providing a user/password to your database in these files. Postfix is a read-only lookup, so you can get into a bit more depth and create a read-only user if you want. And yes, the only variation between these 4 files is the query.

#sql-aliases.cf
user = email
password = mypassword
dbname = mail
hosts = localhost
query = SELECT destination FROM view_aliases WHERE email = '%s'

#sql-relaydomains.cf
user = email
password = mypassword
dbname = mail
hosts = localhost
query = SELECT domain FROM transport WHERE destination != 'virtual:' AND domain = '%s'

#sql-transport.cf
user = email
password = mypassword
dbname = mail
hosts = localhost
query = SELECT destination FROM transport WHERE domain = '%s' AND destination != 'virtual:'

#sql-users.cf
user = email
password = mypassword
dbname = mail
hosts = localhost
query = SELECT true FROM view_users WHERE email = '%s'

#sql-virtual.cf
user = email
password = mypassword
dbname = mail
hosts = localhost
query = SELECT destination FROM transport WHERE destination = 'virtual:' AND domain = '%s'

So, that’s it for postfix. Now we can get into dovecot’s configuration, since postfix at this point will simply be trying to hand email to dovecot!

Dovecot configuration

Dovecot started spreading their configuration across multiple files, so it somewhat feels like it’s a bit all over the place, but it’s pretty straightforward.

There’s not a lot of stuff in my main dovecot.conf file, but here’s what little there is. We do need lmtp, since that’s how postfix will be delivering email to dovecot (for storage into users’ maildirs).

# dovecot.conf
# Protocols we want to be serving.
protocols = imap pop3 lmtp

dict {
}

# load the actual config files, this is were all the meaty
# configuration for dovecot is done.
!include conf.d/*.conf

# I don't actually have a local.conf, but it tries to load one anyway.
!include_try local.conf

I’m going to only hit on what seems to me to be the important config files (and specifically the non-commentary) from conf.d/*.conf, as a bunch of them have sane defaults. A lot of them just exist with everything commented out, waiting for you to choose what to do!

Authentication & storage

# 10-auth.conf
auth_mechanisms = plain login
!include auth-sql.conf.ext

# auth-sql.conf.ext
# we'll define two things here, where to get the sql data (auth)
# and where to store email.  We're avoiding getting the
# email storage location from the database, since it's the same.
passdb {
  driver = sql
  args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
}
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/home/vmail/%d/%n
}

# /usr/local/etc/dovecot/dovecot-sql.conf.ext
driver = pgsql
connect = host=localhost dbname=mail user=email password=mypassword
default_pass_scheme = PLAIN-MD5
password_query = SELECT email AS user, password FROM view_users WHERE email='%u'

Hopefully there’s not too much confusion here. It’s sort of broken down into 3 files for authentication, as well as the lookup for where to store user email. I do know that I use login/plain for transmission of data. This is sort of only problematic if your clients aren’t supporting TLS, since we’ll get to TLS setup later too. (using the same certs as postfix.)

I don’t think I made any changes to 10-director.conf or 10-logging.conf, so on to 10-mail.conf. We need to specify the mail locations here, as well as define one of the namespaces (inbox).

# 10-mail.conf
mail_location = maildir:/home/vmail/%d/%n/Maildir
namespace inbox {
  inbox = yes
}

We definitely need to modify 10-master.conf, as these’s some services to tweak. We need to enable lmtp on a specific ip/port, and need to add a way for postfix to auth users (for sending email). Everything else was in the example config.

# 10-master.conf
service imap-login {
  inet_listener imap {
  }
  inet_listener imaps {
  }
}
service pop3-login {
  inet_listener pop3 {
  }
  inet_listener pop3s {
  }
}
service lmtp {
  unix_listener lmtp {
  }
  # first addition, add LMTP on a specific port
  inet_listener lmtp {
    address = 127.0.0.1 ::1
    port = 10025
  }
}
service imap {
}
service pop3 {
}
service auth {
  unix_listener auth-userdb {
  }
  # second addition, create a unix sock for postfix to authenticate
  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
    user = postfix
    group = postfix
  }
}
service auth-worker {
}
service dict {
  unix_listener dict {
  }
}

The only reason I had to edit 10-ssl.conf was to give it the same cert data I gave to postfix, since it’s all relevant to both services.

# 10-ssl.conf
ssl = yes
ssl_cert = </usr/local/etc/ssl/mail.binarydragon.com.bundle.crt
ssl_key  = </usr/local/etc/ssl/mail.binarydragon.com.key

Sane defaults exist for the 15-* files. Getting into the 20-*.conf files, though, I wanted some additions.

# 20-imap.conf (optional edits)
# this adds the antispam plugin (optional), which is configured in 90-antispam.conf
# also limited userip connections.
protocol imap {
  mail_plugins = $mail_plugins antispam
  mail_max_userip_connections = 30
}

# 20-lmtp.conf (optional edits)
# Wanted to add a default postmaster address and the sieve plugin (90-sieve*.conf).
protocol lmtp {
  postmaster_address = postmaster@binarydragon.com
  mail_plugins = $mail_plugins sieve
}

# 20-managesieve.conf (optional file)
# this is a completely new file, part of using the sieve series of plugins.
# managesieve allows users to create sieve files which can apply filters
# to their emails.
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
  service_count = 1
  vsz_limit = 64M
}
service managesieve {
  process_limit = 1024
}
protocol sieve {
  managesieve_max_line_length = 65536
  mail_max_userip_connections = 10
  managesieve_max_compile_errors = 5
}

No changes to 20-pop3.conf, 90-acl.conf.

# 90-antispam.conf (optional)
# This is the dovecot antispam configuration.  It lets dovecot run the rspamd
# learning commands, so that rspamd can learn when an email is flagged wrong.
plugin {
  antispam_verbose_debug = 1
  antispam_debug_target = syslog
  antispam_backend = mailtrain
  antispam_mail_sendmail = /usr/local/bin/rspamc
  antispam_mail_spam = learn_spam
  antispam_mail_notspam = learn_ham
  antispam_mail_sendmail_args = -h;localhost:11334
  antispam_spam  = Junk
  antispam_trash = Trash
}

No changes to 90-plugin.conf, 90-quote.conf. I have a new file called 90-sieve-extprograms.conf, but it’s blank. 90-sieve.conf has some interesting content, though. It lets each user have a sieve file & directory. It also defines a global dir, and lets us run a sieve filter against all emails coming in.

# 90-sieve.conf
plugin {
  sieve = /home/vmail/%d/%n/.dovecot.sieve
  sieve_dir = /home/vmail/%d/%n/sieve
  sieve_global_dir = /home/vmail/sieve
  sieve_before = /home/vmail/sieve/sieve_before.d/
        # deprecated imapflags extension in addition to all extensions were already
        # The sieve_extprograms plugin is included in this release.
}

(optional) I wanted to include this here, since all the rest of the sieve documentation is here (at least for now). This is a basic sieve filter, which takes email having the spam header flags, and puts them into the Junk folder for the user.

# /home/vmail/sieve/sieve_before.d/spam.sieve (optional)
require ["fileinto"];
# rule:[SPAM]

# spamassassin uses X-Spam-Flag: YES.
# rspamd uses X-Spam: yes
if anyof (header :contains "X-Spam-Flag" "YES",
          header :contains "X-Spam" "yes")
{
        fileinto "Junk";
        stop;
}

That pretty much wraps up dovecot configuration. It was a little more in-depth than I expected to go, but I wanted to point out a few things I’m using when documenting it. After all, this is what I did….

Rspamd

I don’t remember making too many changes to rspamd. I do remember this being one of the most confusing things I’d ever done.

I can say that I’d adjusted the listen settings in rspamd.conf

# rspamd.conf edits.  listen on localhost only.
  bind_socket: "localhost:11333";
  bind_socket: "localhost:11334";

# worker-controller.inc edits - set the password
password = "my-super-secret-password";

Then I know I trained the rspamd filter based on existing spam/ham I already had. I had been running an email server for a nice long time after all. Just not one with rspamd. There’s some decent instructions on this on the rspamd website.

rmilter

Rspamd sort of listens on a socket, but we don’t have postfix configured to talk to that. The reason for that is simple. Postfix wants to get an email, do it’s thing, then send that email off again. So, if we wanted rspamd to filter emails, we would then need to re-inject the email back into postfix, so that the email could either be bounced, dropped, or forwarded on for delivery. Using the milter engine skips all that muck. With milters, we can have postfix hand the email to the milter system, let it analyze the email, and return a result for what to do with the email (bounce, reject, drop, deliver). Since postfix does this as part of the regular email path, it means that there’s no mucking about with the postfix master.cf file, no reinjection or attempts to skip certain checks on reinjected emails, with possibilities to get those things wrong. It’s glorious.

But we do need rspamd to have a milter then.. which is where rspamd-milter comes in.

# rmilter.conf
pidfile = /var/run/rmilter.pid;


spamd {
        spamd_greylist = yes;
        spamd_soft_fail = yes;
        extended_spam_headers = yes;
        servers = r:localhost:11333;
        connect_timeout = 1s;
        results_timeout = 20s;
        error_time = 10;
        dead_time = 300;
        maxerrors = 10;
        reject_message = "Spam message rejected; If this is not spam contact postmaster at binarydragon.com";
        whitelist = 127.0.0.1/32, 192.168.0.0/16;
        rspamd_metric = "default";
};

bind_socket = unix:/var/run/rmilter/rmilter.sock;
tempdir = /tmp;
max_size = 40M;

limits {
        limit_whitelist = 127.0.0.1/32;
        limit_whitelist_rcpt =  postmaster, mailer-daemon;
        limit_bounce_addrs = postmaster, mailer-daemon, symantec_antivirus_for_smtp_gateways, <>, null, fetchmail-daemon;
        limit_bounce_to = 5:0.000277778;
        limit_bounce_to_ip = 5:0.000277778;
        limit_to = 20:0.016666667;
        limit_to_ip = 30:0.025;
        limit_to_ip_from = 100:0.033333333;
};

# not sure if I can remove this completely.  Doesn't seem important.
beanstalk {
        protocol = tcp;
        lifetime = 172800;
        id_regexp = "/^SomeID.*$/";
        send_beanstalk_headers = yes;
        send_beanstalk_copy = yes;
        send_beanstalk_spam = yes;
};

greylisting {
        timeout = 300s;
        expire = 3d;
        whitelist = 127.0.0.1,
                        192.168.1.1,
                        192.168.2.0/24;
        awl_enable = yes;
        awl_pool = 10M;
        awl_hits = 10;
        awl_ttl = 3600s;
};

This is stripped of all comments. The entire file is pretty massive. The important parts are the bind-socket (for the milter), since we tell postfix to use that socket, and spamd { servers: … } since that defines how to communicate with rspamd!

clamav, clamav-milter

ClamAV is a decent free A/V scanner. I use it because it’s better than nothing, though, definitely not top-ranked. Here’s the clamd.conf & clamav-milter.conf files I use. They’re pretty basic, primary interest is the LocalSocket (clamd.conf), and the ClamdSocket/MilterSocket from the milter conf.

# clamd.conf
LogFile /var/log/clamav/clamd.log
LogSyslog yes
LogFacility LOG_MAIL
PidFile /var/run/clamav/clamd.pid
DatabaseDirectory /var/db/clamav
LocalSocket /var/run/clamav/clamd.sock
FixStaleSocket yes
StreamMaxLength 30M
User clamav
AllowSupplementaryGroups yes
ScanMail yes

# clamav-milter.conf
MilterSocket /var/run/clamav/clmilter.sock
FixStaleSocket yes
User clamav
AllowSupplementaryGroups yes
PidFile /var/run/clamav/clamav-milter.pid
ClamdSocket unix:/var/run/clamav/clamd.sock
LogSyslog yes
LogFacility LOG_MAIL
LogInfected Basic
LogClean Basic

That pretty much sums everything up.

Launch! And pray.

I suppose I glossed right over something… We’ll need to make sure there’s a user named vmail (in group vmail) with a home folder of /home/vmail before any local mail deliver will work. I will try and come back to this write up at some point and see about organizing that.

Typical email things will ensue. Typos, failure to include comments in my post, etc. Getting filtered off the Internet is probably a good one.

Final thoughts

I hate email less than I hate printers. But it’s still a mess, with so many problems, and little updates that have been done over the years (like SPF records). I think I’ve hit a decent medium in terms of complexity for my installation versus ease of use for users (Rspamd/rmilter helped a lot there). I’d done the whole concept with reinjection into postfix before, and boy what a mess that was.

I also know that I wrote this up for someone who was using quotas. I can say that while I don’t use quotas, it would probably be trivial to implement quotas as an additional database field and have Dovecot use that additional field in the quotas configuration.

I do wish I’d written this up sooner. I’d then be able to link to the websites that helped me get this far. Nothing here is work that someone else hasn’t already done, though the particular combination might be a little different.