Racoon is a Internet Key Exchange (IKE) daemon for automatically keying IPsec connections.
Since racoon error messages can be cryptic, here are my compilation of the error messages decoded.
unknown notify message, no phase2 handle found
Racoon got an error message from the peer but was too dumb to spell it out. Try running racoon with -d. Usually, the peer was unable to verify the certificate.
no suitable policy found
You did not set a policy for this IP with setkey. Check /etc/ipsec.conf and run it through setkey -f again.
no policy found
Same as above? Maybe the difference is that you didn't set a policy at all?
trns_id mismatched: my:12 peer:3
I got this when I tried a racoon key exchange over a NATted connection. I think it means that the outer IP (the IP of the NAT gateway) and the inner IP (192.168.*) don't match the same policy rule in /etc/ipsec.conf.
invalid msg length
I have no idea what causes this. It didn't appear to have any negative impact though.
failed to get proposal for responder
I got this when one side was configured for blowfish and the other was configured for 3des in racoon.conf.
unable to get local issuer certificate
This is an openssl error. I got this when the certificate was self-signed instead of signed by a CA the peer recognizes. If it is signed by a CA, openssl was unable to verify. Copy the CA.crt file to /etc/cert (or wherever you told racoon to look for certificates) and run
ln -s CA.crt `openssl x509 -noout -hash -in CA.crt`.0
notify message must be encrypted
This happened to me because one side proposed aes and the other 3des.
failed to get subjectAltName
You forgot to set my_identifier asn1dn; in the remote section.
failed to get my CERT
The path or filename is wrong in racoon.conf. Use
strace -eopen racoon -F
to see which files racoon tries to open. Here is an excerpt from my racoon.conf:
path certificate "/etc/cert" ; certificate_type x509 "knuth.crt" "IPsec Server Key.pem";
This will look for /etc/cert/knuth.crt.
And now a little goodie for poor Cisco users (thanks to Andreas Bogk for this):
%CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 184.108.40.206 failed its sanity check or is malformed
This is Cisco's catch-all error message. If you increase the log level, you will get more detail, but it will be in the debug buffer, not in the syslog messages. The most probably reason for this error message is that the certificate is expired.
The magic incancation is:
conf term logging buffered 8192 debugging ^Z debug crypto isakmp error
then try isakmp and look in the log with
invalid PEM boundary
You foolishly assumed that, since the public key is also part of the private key, it would be sufficient to just give IOS the private key. IOS actually wants something like "BEGIN PUBLIC KEY"... "END PUBLIC KEY"... "BEGIN RSA PRIVATE KEY"... "END RSA PRIVATE KEY" as sequence. Also, IOS will only accept keys with a PEM passphrase. It will ask for the passphrase on import and then save the key without passphrase internally.
If you find that you cannot import PEM keys, your IOS may not have PEM support. As ridiculous as this sounds, IOS 12.3(8)T has PEM support, but IOS 12.3(9)T does not. Another IOS command you should know is "term mon", Andreas finally adds. It makes debug output show up on your terminal (if it's not the console anyway).
And now, finally, something for the Windows people. If your racoon says:
INFO: begin Identity Protection mode. INFO: received Vendor ID: MS NT5 ISAKMPOAKLEY ERROR: ignore information because ISAKMP-SA has not been established yet.
then Windows probably could not find a computer certificate. If you foolishly thought double clicking on the .p12 file would put make certificate known to Windows, think again. You need to do this:
Start -> Run -> MMC File -> Add/Remove Snap-In Choose the Certificate Snap-in Add -> Service Account -> Local computer -> IPSEC Services -> Close, OK Right click on one of the certificate stores in the tree view on the left Import -> Next -> choose your key file and let Windows choose
Setkey is a tool to manipulate and dump the kernel Security Policy Database (SPD) and Security Association Database (SAD).