- Establish TCP connection to a server in a different subnet and interface using a PFSense Firewall.
- Allow for SSH, HTTP and HTTPS connections from a LAN client to a FreeNAS box.
- Keep mind intact and don't lose any sleep.
Summary:
- If you are experiencing slow connections that never seem to complete the TCP handshake, make sure your firewall and the rules/chains are correctly configured for the type of connection you are attempting.
- After a couple of hours (to be honest more like 4 hours or troubleshooting 1 hour of blogging), I couldn't figure out why I kept receiving TCP TIME_WAIT and CLOSE_WAIT for a simple HTTP/HTTPS connection to my FreeNAS box.
- Simple topology
- 2 interfaces and 2 networks.
- Create a rule to allow for network A (LAN) to talk to network B (FreeNAS).
- TCP connections from a LAN client to FreeNAS box.
- Complicated results
- Created a simple rule for LAN to FreeNAS with Any Protocol, Any Source, Any Destination, and Any Port.
- Unable to connect to FreeNAS via HTTP/HTTPS
- Successfully able to connect via SSH.
- Successfully able to ping between LAN client and FreeNAS box.
- Unable to connect to FreeNAS HTTP/HTTPs port over netcat:
- netcat <FreeNAS IP> 443
- OK check the logs, seeing a lot log entries for ESTABLISHED, FIN_WAIT_1, FIN_WAIT_2, and TIME_WAIT from LAN client to FreeNAS box.
- Check chrome and inspect the traffic, same info as the logs. Good.
- Identified the issue, but don't understand it enough
- At this point I can see that there is a TCP connection issue, the connection goes as far as ESTABLISHED but there is an issue once the LAN client goes into the FIN_WAIT_1 state.
- Now the problems start, no ACK from FreeNAS box and LAN client goes into FIN_WAIT_2, again no ACK from FreeNAS box now the LAN client goes into TIME_WAIT.
- At this point while client is in TIME_WAIT, any other attempts to connect to the FreeNAS box are going to be blocked because now the wait time needs to expire.
- The wait time is calculated as 2 times the Maximum Segment Lifetime
- MSL on PFSense 30 seconds so that means 2*30 = 60 seconds
- Use the following command to determine MSL on PFSense
- # sysctl net.inet.tcp.msl
- output > net.inet.tcp.msl: 30000
- After TIME_WAIT expires, the connection goes into CLOSE_WAIT and browser reports ERR_CONNECTION_TIMED_OUT
- Solution
- I didn't have one at first and could not determine how to solve this problem.
- Early in the day I recalled an Advanced Feature inside the PFSense Rules from PFSense documentation about Asymmetric Routing and Firewall Rules.
- There is an option to select the handling of State Type to be used for the firewall rule.
- State types - the pfSense software offers multiple options for state handling.
- Keep state - Works with all protocols. Default for all rules.
- Sloppy state - Works with all protocols. Less strict state tracking, useful in cases of asymmetric routing.
- Synproxy state - Proxies incoming TCP connections to help protect servers from spoofed TCP SYN floods. This option includes the functionality of keep state and modulate state combined.
- None - Do not keep any state entries for this traffic. This is very rarely desirable, but is available because it can be useful under some limited circumstances.
- The Keep state is the default set to the rule used in this project, but as far as documentation and what this state does, I found nothing useful.
- Fortunately Sloppy and Synproxy states are documented extensively and I was able to determine a solution now.
- The Synproxy state handling is the solution used to sucessufully establish HTTP/HTTPS connections to the FreeNAS box from the LAN client!
- Here is quick summary of what Synproxy does:
- The LAN Client connects to the Synproxy server (PFSense box).
- The Synproxy server estbalishes the TCP connection to the FreeNAS box on behalf of the LAN client via spoofing LAN client's SYN responses.
- Synproxy server and FreeNAS server continue the handshake and once a connection is opened to the FreeNAS box, the Synproxy server hands it off to the LAN client to communicate with the FreeNAS box.
- The downside
- The PFSense box becomes a proxy for these TCP connections and this could have potential performance impact in much larger environments.
- Overall, a terrific learning experience and really good knowledge for me to share to all of you. Thanks.
Really Really Good References
https://www.pfsense.org/about-pfsense/features.html
https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules
https://en.wikipedia.org/wiki/Maximum_segment_lifetime
https://benohead.com/tcp-about-fin_wait_2-time_wait-and-close_wait/
http://www.serverframework.com/asynchronousevents/2011/01/time-wait-and-its-design-implications-for-protocols-and-scalable-servers.html
http://rhelblog.redhat.com/tag/synproxy/
https://github.com/firehol/firehol/wiki/Working-with-SYNPROXY
https://www.cyberciti.biz/tips/freebsd-lists-open-internet-unix-domain-sockets.html
