Splunk Forwarding, Load Balancers, and Proxies

We get a lot of folks on the Splunk Community Slack asking about putting load balancers or proxies between Splunk forwarders (universal or heavy, it doesn’t matter) and indexers. I understand the temptation, and sometimes the need (security policies, network topology weirdness, etc).

Don’t do it. Splunk-to-Splunk (S2S) communications are not HTTP connections. As such, load balancers and proxies will not behave in a way that Splunk expects, and will cause connections to fail unpredictably, drop events, cause duplicate events, or refuse to connect at all.

Load balancers and proxies between forwarders and indexers is also not supported, so if you have a problem with that connection, support will tell you to go pound sand, and remove the load balancer. This may mean some rearchitecting of your deployment, or some network topology changes, so best to get this right from the start.

The best way to load balance connections from forwarders to indexers is to use Splunk’s built-in load balancing.

It would be cool if Splunk forwarders could output HEC, which can be load balanced, but right now that is not possible.

Thanks to the other SplunkTrust members for contributing to this post!