My server cluster uses JMS messaging for reliable configuration. The setup relies on the ActiveMQ network of brokers architecture, which is, for the most part, fantastic. The brokers automagically discover one another and autocluster themselves.
This morning I had a problem when deploying a new version of my application. It kept hanging while trying to connect to the broker. Turns out the problem was with my new firewall rules. I was blocking the multicast traffic used for discovery. This was easy to figure out – I just stopped iptables and everything worked.
I opened up TCP ports 61616 for ActiveMQ (the defaults), but that didn’t help – the discovery is done over UDP on a different port. But I coulnd’t find that fact documented anywhere. I did a TON of Googling, but couldn’t find a thing about this, so I’m posting my research for posterity.
First I did a `tail -f /var/log/messages` to see what was being blocked by the firewall. Among other things, I noticed
Feb 2 09:28:46 fc515220 kernel: Dropping: IN=eth0 OUT= MAC= src=X.X.X.X DST=188.8.131.52 LEN=90 TOS=0x00 PREC=0x00 TTL=1 ID=0 DF PROTO=UDP SPT=6155 DPT=6155 LEN=70
A little more Googling and I found the following useful documents. First is the RFC for administratively scoped IP multicast which explains clearly:
This document defines the "administratively scoped IPv4 multicast space" to be the range 184.108.40.206 to 220.127.116.11
In other words, it’s the defined range for multicast addresses on a local box, similar to 192.168.x.x or 10.0.x.x.
Next up was this gem from the ActiveMQ source tree:
public static final String DEFAULT_DISCOVERY_URI_STRING="multicast://18.104.22.168:6155";
Huh. Well that’s good to know!
At that point I simply had to add the following rule to my iptables:
/sbin/iptables -A INPUT -p udp -s $IP1/$IP2 -d 22.214.171.124 --dport 6155 -j ACCEPT
where $IP are the range of source IP addresses in my cluster (or just localhost/eth0 if you’re doing this on a single box).
Everything seems to work today. Not sure if that magic number will change in the future. In theory I could probably safely open up everything that comes from my local box to the local multicast range, but no need to do that today.