RangerIDS - Intrusion Detection System (IDS)



Protocol Decode


Protocol decode IDS take a radically different approach to simple pattern matching IDS products – though sometimes not quite as radically different as the marketing folks would have you believe. With this technique, the IDS detection engine performs a full protocol analysis, decoding and processing the packet contents in the same way that the target client or server application would. It also tends to be stateful. 


Although this may seem like using a sledgehammer to crack a nut, it does have the advantage of highlighting anomalies in packet contents much more quickly than doing an exhaustive search of a signature database. It also has the advantage of greater flexibility in capturing attacks that would be very difficult – if not impossible – to catch using pure pattern-matching techniques, as well as new variations of old attacks. These are attacks which – although changing only slightly from variant to variant – would normally require a new signature in the database for the “traditional” IDS architecture, but which would be detected automatically by a complete protocol analysis. 


One of the first things the protocol decode engine does is to apply rules defined by the appropriate RFCs to look for violations. This can help to detect certain anomalies such as binary data in an HTTP request, or a suspiciously long piece of data where it should not be – a sign of a possible buffer overflow attempt. 


One simple example of how this might work concerns searching Telnet login strings for one of the many well-known login names that rootkits tend to leave behind on the system. A pattern matching system might scan all Telnet traffic for all these patterns, in which case the more patterns you add, the slower it becomes (not always the case, but a reasonable assumption for the purposes of this example). 


In contrast, a protocol analysis system will decode the Telnet protocol and extract the login name. It can then perform an efficient search in a binary-search tree or a hash table for that login name, which should scale much better as new signatures are added.  


In theory, therefore, protocol decoding should offer more efficient processing of traffic and improved scalability as more signatures are added, compared to a pure pattern matching solution. In reality, pattern matching solutions rarely opt for a “brute force” approach, and so the differences are not always as marked as the marketing people would like us to believe.  


Note also, that pattern matching and protocol decoding are not mutually exclusive, as some would lead you to believe. A protocol analysis IDS can only go so far with its protocol decodes before it too will be forced to perform some kind of pattern matching, albeit against a theoretically smaller subset of “signatures”. One major downside, of course, is that if a completely new type of exploit does surface, it is likely that the IDS developer will have to create new code to handle it, whereas the pattern matching approach can allow the administrator to develop a custom signature much more quickly on site. 


Protocol decoding does offer a number of advantages, however. It minimises the chance for false positives if the protocol is well defined and enforced (although false positives can be higher if the RFC is ambiguous), and can also be more broad and general to allow the IDS to detect minor variations of an exploit without having to implement separate signatures. 


You may see this technique referred to in several different ways: 

Each of these terms, if strictly applied, could use a slightly different approach to the problem. For example, we would expect a protocol decode engine to perform the sort of additional pattern matching and length checking mentioned above on the field contents in order to detect specific exploits or buffer overflows. 


Pure protocol validation or Protocol Anomaly Detection engines, however, might go no further than decoding just enough to be able to determine if the packet follows the RFC to the letter. If not, they will raise an alert - but in allowing a packet to pass, they cannot be sure that the contents will not contain a means of exploit that just happens to conform with the RFC.  


Beware the marketing hype in this particular area – no matter what architecture is used, the performance figures and detection rates in a live deployment will speak for themselves.