Debian: IPSec VPN Server

From ReceptiveIT
Jump to: navigation, search

Linux 2.6 Kernel

Since the Linux 2.6 kernel was released, it has had a native IPSec stack in-built that is solid and reliable, based on the KAME IPSec code used in the BSD operating systems. For all those are still using a 2.4 kernel, the IPSec stack has been backported by Herbert Xu of Debian into recent Debian kernel 2.4 packages.

Compiling the Kernel

The relevant parts of the kernel .config are:

* Legacy PTY support - Required for L2TPd
Device Drivers --->
 Character devices --->
  Legacy (BSD) PTY support [Y/n/?] y
* Networking options - Required for IPSec
Networking --->
 Networking support (NET) [Y/n/?] y
  PF_KEY sockets (NET_KEY) [Y/n/m/?] y
  IP: AH transformation (INET_AH) [Y/n/m/?] y
  IP: ESP transformation (INET_ESP) [Y/n/m/?] y
  IP: IPsec user configuration interface (XFRM_USER) [Y/n/m/?] y
* Cryptography options - Required for IPSec
Cryptographic options --->
 Cryptographic API (CRYPTO) [Y/n/?] y
  HMAC support (CRYPTO_HMAC) [Y/n/?] y
  Null algorithms (CRYPTO_NULL) [Y/n/m/?] y
  MD5 digest algorithm (CRYPTO_MD5) [Y/n/m/?] y
  SHA1 digest algorithm (CRYPTO_SHA1) [Y/n/m/?] y
  DES and Triple DES EDE cipher algorithms (CRYPTO_DES) [Y/n/m/?] y
  AES cipher algorithms (CRYPTO_AES) [Y/n/m/?] y

Getting Kernel Modules to load

In order to get the esp4 module autoloaded, the following line referencing esp4 is needed in /etc/modprobe.d/aliases. The full set of ipsec modules are shown for completeness.

alias xfrm-type-2-50 esp4
alias xfrm-type-2-51 ah4
alias xfrm-type-2-108 ipcomp
alias xfrm-type-10-50 esp6
alias xfrm-type-10-51 ah6
alias xfrm-type-10-108 ipcomp6


IPSec is the industry-standard protocol for transparent network encryption and authentication.

Manually Keyed VPN

Automatically Keyed using Pre-Shared Keys

Automatically Keyed using x509 Certificates



This section describes how to build a VPN Server for IPSec/L2TP clients. IPSec/L2TP does not provide any extra security than a IPSec VPN, but it does have the advantage that you can authenticate users against a existing user database, such as LDAP or Active Directory. Microsoft Windows 2000, Microsoft Windows XP and Apple MacOS 10.3+ all have in-built IPSec/L2TP clients, which can be used as a more integrated solution that a sit-on-top IPSec client.

I am basing this howto on;

  • Linux 2.6.x Kernel
  • Racoon (KAME IKE Daemon)
  • Setkey (ipsec-tools)
  • L2tpd
  • OpenSSL

Configuring Racoon

Using Pre-Shared Keys

For road-warrior VPN clients using preshared keys (and aggressive mode), the required racoon.conf is:

log debug; path pre_shared_key "/etc/racoon/psk.txt";

listen {

       isakmp a.b.c.d [500];


padding {

       maximum_length 20;      # maximum padding length.
       randomize off;          # enable randomize length.
       strict_check off;       # enable strict check.
       exclusive_tail off;     # extract last one octet.


remote anonymous {

       exchange_mode aggressive,main;
       doi ipsec_doi;
       situation identity_only;
       generate_policy on;
       proposal_check obey;

       proposal {
               encryption_algorithm 3des;
               hash_algorithm sha1;
               authentication_method pre_shared_key;
               dh_group modp1024;


sainfo anonymous {

       lifetime time 28800 sec;
       encryption_algorithm 3des ;
       authentication_algorithm hmac_md5;
       compression_algorithm deflate ;


Replace a.b.c.d with the public ip address of your gateway.

Using x509 Certificates

Racoon supports the usage of X.509 certificates for the authentication process. These certificates may be checked against a certificate authority (CA). The configuration is similar to the PSK configuration and differs only on the authentication part.

While it is possible to set up a Ipsec/L2TP VPN using pre-shared keys, most likely you will want your windows clients to be road-warriors. A road-warrior is a VPN end-point with an unknown, or dynamic IP address. The use of X509 certificates will allow your road-warrior clients to use main mode key exchange.

  • Note: racoon doesn't let you specify the private key passphrase, so you'll need to generate a passphraseless version of your key to use your certificate.


path certificate "/etc/webCA/certs";

remote anonymous { exchange_mode main; doi ipsec_doi; situation identity_only; generate_policy on;
my_identifier asn1dn; peers_identifier asn1dn; verify_identifier on; certificate_type x509 "vpngateway_cert.pem" "vpngateway_key_nopass.pem";
proposal { encryption_algorithm 3des; hash_algorithm sha1; authentication_method rsasig; dh_group modp1024; }


sainfo anonymous {

       lifetime time 28800 sec;
       encryption_algorithm 3des ;
       authentication_algorithm hmac_md5;
       compression_algorithm deflate ;


There doesn't seem to be an easy way to specify both of these options in the same configuration file. In my tests, if rsasig is available as an option, then a client trying to do pre-shared keys will be refused.

IPSec policy statements

While racoon will generate part of the policy required for a road-warrior connection, the rest of the policy needs to be specified using setkey. Debian manages the IPSec policy with the file /etc/ipsec-tools.conf.


#!/usr/sbin/setkey -f

## Flush the SAD and SPD # flush; spdflush;
# Road Warrior spdadd a.b.c.d[1701][0] any -P out ipsec esp/transport//require;


L2tpd currently is only available in the Debian unstable repository.


port = 1701

[lns default] ip range = - local ip = require chap = yes refuse pap = yes require authentication = yes hostname = LinuxVPNserver ppp debug = yes pppoptfile = /etc/ppp/options.l2tpd.lns length bit = yes


idle 1800
mtu 1400
mru 1400
connect-delay 5000
  • Note: I'm not using proxyarp in this case (I don't want the VPN clients to be able to access the lan the gateway is on - it has other IPSec tunnels which the clients use), but if you need ppp to do it, it should be specified here.

The PPP chap-secrets file should contain your secrets for PPP authentication - these are the usernames and passwords that Windows will ask the user for.


  1. client server secret IP addresses
  • xpuser "sekrit"

xpuser * "sekrit"


As Linux 2.6's KAME IPSec doesn't provide ipsec0-style interfaces in the same way as FreeS/WAN, the iptables setup is different, although we're trying to achieve the same effect - disallow access to the L2TP daemon to unencrypted packets.

We do this using iptable's 'fwmark' match -- marks applied to encrypted packets remain set on the unencrypted packets, and we can use this to determine which packets arrived via the IPSec SA. The 'mark' is internal to the kernel, and can't be set on packets outside of the system, so it's safe to use this match.

iptables -t mangle -A PREROUTING -i eth0 -p esp -j MARK --set-mark 1
iptables -A INPUT -i eth0 -m mark --mark 1 -j ACCEPT

iptables -A OUTPUT -s a.b.c.d -p udp -m udp --sport 1701 -j ACCEPT
iptables -A INPUT -p esp -j ACCEPT iptables -A OUTPUT -p esp -j ACCEPT
iptables -A INPUT -d a.b.c.d -p udp -m udp --dport 500 -j ACCEPT iptables -A OUTPUT -s a.b.c.d -p udp -m udp --sport 500 -j ACCEPT

(Replace a.b.c.d with the public IP address of your gateway)

Installing Server-Side Certificates

The certificate and the private key are stored in the certificate path /etc/certs. This path is set using the option path certificate in the configuration file. The certificates and the certificate revocation lists are stored in PEM format as generated with openssl. For the generation of certificates see the chapter on Debian: openssl. If the certificate of the peer is to be checked against a certificate authority (verify_cert on; is the default), then the certificate of the CA has to be also stored in this directory. For OpenSSL to find the certificate it has to be renamed or linked using the hashed name:

apollo:~/certs# ln -s cacert.pem `openssl x509 -noout -hash < cacert.pem`.0

If the certificate additionally is to be checked against a certificate revocation file (CRL) the CRL must be stored in the same directory using a similar linked hashed name:

apollo:~/certs# ln -s crl.pem `openssl x509 -noout -hash < cacert.pem`.r0

When storing the certificates and the private key it is important to note that racoon cannot decrypt a private key. Therefore the private key must be stored in its decrypted cleartext form. If you created a crypted private key, you have to decrypt it:

apollo:~/certs# openssl rsa -in my_private_key.pem -out my_private_key.pem
read RSA key
Enter PEM pass phrase: certpassword
writing RSA key

Windows Clients

Import Certificate

Start -> Run -> mmc
File -> Add/Remove Snap-in
Add -> Certificates -> Add
Computer account -> Finish
Local computer -> Finish
Expand Certificates (Local Computer)
Right-Click on Personal -> All Tasks -> Import
Next -> vpngateway.p12 -> Next
Type import password -> Automatically select the certificate store based on the type of certificate
Close MMC Console


MTU problems

If you have intermittent problems with your VPN connection, it could be an MTU problem. What you might see is that you can ping machines on the internal network and you can surf or transfer very small files, but you can't copy large files because the connection stalls. Then try decreasing the MTU to 1410 (or perhaps even lower?) by adding the following line to /etc/ppp/options.l2tp:

mtu 1400
mru 1400

This problem may be especially apparent if you connect through the VPN to sites with broken 'Path MTU (PMTU) Discovery' (a letter with an executive summary describing the situation can be found here). Perhaps people with ADSL connections that use PPTP or PPPoE have this problem as well. I myself had no problem with a PPTP ADSL connection ('KPN Mxstream') which uses Alcatel equipment. Reducing the PPP packet size does not help if the connection fails before the authentication has succeeded. Try again with a smaller certificate and without a NATed connection, if you can. If you use NETKEY on kernel 2.6, then kernel 2.6.12+ is recommended if you experience PMTU problems. On a 2.6.12+ kernel you could do:

echo "0" > /proc/sys/net/ipv4/ip_no_pmtu_disc