Building a Protocol-Aware Reverse Proxy: Lessons from Handling MongoDB’s Unique Framing

The Formal Connector is a protocol-aware reverse proxy that interprets various wire-protocols (HTTP, Postgres, MySQL, Kubernetes, and much more) to provide security teams better visibility and control of their data flows.

Customers often start using Formal for one type of data store and then over time Formal covers more data stores in their stack. To keep the Formal deployment simple we built a feature that enables the Formal Connector to listen on a single port for multiple technologies. This feature works by detecting the incoming wire-protocol and then automatically loading the right wire-protocol interpreter.

Initial approach

The initial approach involved reading the first few bytes of any TCP connection to identify the protocol before forwarding the packets to the appropriate protocol interpreter:

code-snippet-1

The []byte data read from the connection was then passed and explicitly written before using io.Copy to forward the traffic. This approach worked well for most protocols but encountered issues with MongoDB due to its unique protocol framing.

MongoDB expects specific framing for its messages. Sending the initial bytes separately disrupted the framing structure, resulting in communication errors. The misaligned initial bytes caused failed connections and protocol misinterpretations.

The Solution

The issue was resolved by wrapping the initial bytes in a MultiReader alongside the connection. This adjustment ensured that the proxy transmitted both the initial bytes and the rest of the connection stream in a single operation, preserving MongoDB’s expected message format.

Here’s the updated implementation:

code-snippet-2

The new implementation wraps the connection in peekConn, enabling seamless use of io.Copy without requiring an explicit Write. This adjustment preserved MongoDB’s protocol framing while maintaining functionality for other protocols.

This fix not only resolved the issue but also emphasized the importance of understanding protocol-specific behaviors when performing deep packet inspection and protocol detection. It demonstrated the significance of adaptability and the complexities of network engineering.

If that sounds like an interesting challenge to you, join us—Formal is hiring.

Initial approach

The Solution

Speak to an Engineer