Covert Channel and Tunneling over the HTTP protocol Detection :
GW implementation theoretical design

v1.1 - November 2003

Simon Castro <scastro [at] entreelibre.com>

Gray-World Team
http://www.gray-world.net


Copyright (c) 2003, Simon Castro <scastro [at] entreelibre.com>.
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with the Invariant Sections being LIST THEIR TITLES, with the Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST. You should have received a copy of the license with this document and it should be present in the fdl.txt file. If you did not receive this file or if you don't think this fdl.txt license is correct, have a look at the official http://www.fsf.org/licenses/fdl.txt licence file.

SUMMARY

CHANGELOG AND NOTES


INTRODUCTION


1. DETECTION THEORY

1.1. SIGNATURE-BASED DETECTION

1.1.1. Basic tunneling example : Reverse-Shell using NetCat

1.1.2. Running a signature-based detection engine

1.2. PROTOCOL-BASED DETECTION

1.2.1. Basic tunneling example : Tunneling SSH with the CONNECT method

1.2.2. Running a protocol-based detection engine

1.3. BEHAVIORAL-BASED DETECTION

1.3.1. Covert channel example : Wrapping arbitrary data into HTTP

1.3.2. Running a behavioral-based detection engine

1.3.3. Statistical approaches

1.3.4. Behavioral-based detection engine and covert channels

2. GW IMPLEMENTATION THEORETICAL DESIGN

2.1. WHAT KIND OF INFORMATIONS ARE WE LOOKING FOR ?

2.1.1. Covert Channels detection : unauthorized tunnels

2.1.2. Covert Channels detection : backdoor communications

2.2. THE LOCATION OF THE DETECTION ENGINE ?

2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?

2.4. WHAT ABOUT THE STATISTICAL METHODS PRESENTATION ?

2.5. WHAT ABOUT THE PROTOCOL-BASED DETECTION ?

2.6. LEARNING PERIOD AND/OR STATIC PRE-CONFIGURATION ?

2.7. DETECTION vs PREVENTION ASPECTS


CONCLUSION

ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED

THANKS

WEBOGRAPHY

CHANGELOG AND NOTES

o v1.1 - November 2003 : Updated the INTRODUCTION and the THANKS parts and added a CHANGELOG AND NOTES part.

o v1.0 - July 2003 : Initial publication.

This paper was originally released at Hitchhiker's World Issue #7 (have a look at http://www.infosecwriters.com/hhworld/).

You will find on http://team.gray-world.net/public the current working version of this paper.

INTRODUCTION

Whereas detecting and/or reacting (proactively or not) to Covert Channels is a topic presented by security researchers in a wide number of papers, the detection of Covert Channels and Tunnels embedded inside the HTTP protocol is a relatively new (theoretical) research area.

Three main detection approaches exist. The signature-based approach involves building and updating a signatures database and notifying the operator when a known signature is found in the network data streams. The protocol-based approach focuses on protocol anomalies or violations and incase the monitored communications turn out to be "abnormal", the operator is notified. The behavior -based approach involves creating behavioral user profiles (users, workstations, servers, network streams, etc.) and using statistical methods to determine if the observed data stream is suspicious.

Although we know simple covert techniques can easily be detected on a production Network Access Control System (NACS), we consider that the more advanced ones described in [1] are more difficult to catch.

Let's first take a look at some concepts related to the main detection approaches to and subsequently focus on a Gray-World theoretical design / implementation.

1. DETECTION THEORY

1.1. SIGNATURE-BASED DETECTION

Using a signature-based detection designed tool involves searching for specific pre-defined patterns (signatures) into the network stream monitored. The related signature appears as something the detector should/must match to trigger an alarm process.

1.1.1. Basic tunneling example : Reverse-Shell using NetCat

NetCat [2] is a "network swiss army knife". We use it in this example to create a reverse-shell communication channel between the internal network and the "very bad and dangerous" [12] W3 public network. Note that this is a widely known example at the present time...

We arbitrary use a Win32 internally located workstation (due to popularity) and suppose the only way to reach the external public resources is to use an HTTP proxy configured in the NACS.

This box uses nc11nt [2], a "configuration" file and a run file :

go_on.bat :


  @echo off
  type go_on.txt
  wait /delay=3 /quiet
  c:\winnt\system32\cmd.exe

go_on.txt :


  CONNECT @DST_SRV_IP:443 HTTP/1.0\r\n
  User-Agent: Thank_you_to_allow_the_connect_method\r\n
  Host: @DST_SRV_IP:443\r\n
  Proxy-Connection: Keep-Alive\r\n
  Pragma: no-cache\r\n
  \r\n
  \r\n

Note : Don't forget to enter twice at the end of the previous text file.

The "evil" [12] internally located authorized user runs nc.exe -L -p 443 on the external located @DST_SRV_IP server and runs nc.exe @NACS_PROXY_IP:NACS_PROXY_PORT -e go_on.bat on the internal located box.

And what does this user do now ?

1.1.2. Running a signature-based detection engine

Running a signature-based detection engine is not as easy as it seems with this example. It would be possible to match cmd.exe related patterns in the data stream, but legitimate websites may set it off if valid files contain these patterns too.

To override such a distinction problem, the detection engine would, for example, have to ask : "Is there an HTTP header in the requests/reponses messages ?" and thus become a kind of protocol-based detection engine.

However, it may be feasible to customize some specific rule sets for existing signature detection engines such as the Snort [7] Network Intrusion Detection System (NIDS). In this case, the engine could consider the rule set as a kind of stop-list to flag the operator.

Also, using an existing signature-based detection engine also gives the opportunity to use its HTTP decoding functionalities.

1.2. PROTOCOL-BASED DETECTION

Using a protocol-based detection designed tool involves searching for protocol anomalies or violations while monitoring a network stream. A protocol -based detection engine should/must be carefully designed concerning what it understands from the protocol specifications and what kind of choice it has to take when different operating systems handle different protocol states.

1.2.1. Basic tunneling example : Tunneling SSH with the CONNECT method

We again suppose that the only way to reach external public resources from the internal network is through a NACS HTTP Proxy.

Using the HTTP CONNECT method to transit arbitrary data as SSH [3] data streams through a NACS HTTP Proxy is something a variety of tools [4] offers at the present time.

For example, a SSH client opens a TCP connection to a proxy client tool which opens a TCP connection to the proxy server and sends a CONNECT string which asks the proxy server to open a TCP connection to a destination SSHd server. When the TCP tunnel (SSH client - Proxy Client - Proxy Server - Destination server) is up, the SSH communication begins.

We let you think of other concrete examples ?

1.2.2. Running a protocol-based detection engine

Using a protocol-based detection approach is something interesting looking at our previous examples.

In the both previous examples (1.1.1. and 1.2.1.), the tools use the HTTP CONNECT method to reach an external public resource. As this HTTP method should only be used to transit a SSL data stream (to the NACS administrator point of view), it would be possible to run a protocol-based detection engine which monitors that a SSL handshake occurs each time a TCP connection transits through the NACS.

Note that we don't suppose here that the SSL connection can be used to transit arbitrary data. If we want to keep the SSL functions such as confidentiality but at the same time, monitor for eventual covert streams, the detection engine won't have to be a protocol-based but merely a behavioral-based one.

The both previous examples are also based on non-HTTP servers (a nc.exe listener and a SSHd server). The data stream generated doesn't have usual HTTP content such as HTTP headers.

We now have something to monitor in the data stream. But, does not this look too easy ?

Indeed, some covert channel and tunneling tools [4] allow the user to only use HTTP requests and responses messages. These tools don't need to use the CONNECT method, but instead, specially crafted HTTP POST requests messages.

Furthermore, other tools even allow setting SSL tunnels to transit arbitrary data avoiding thus any eavesdropper to look at the data stream.

It is, again, possible to design a robust protocol-based detection engine following most of the RFCs specifications to ensure that the data stream is RFC compliant. For example : one HTTP response per HTTP request, a Hostname header field must exist when the HTTP/1.1 protocol is used, etc.

But for carefully designed covert channel and tunneling tools, our protocol-based detection model is not useful anymore...

But be quiet (?), let's now try (!) to detect such tools...

1.3. BEHAVIORAL-BASED DETECTION

Using a behavioral-based detection tool involves creating user profiles as reference points (which can be statically or dynamically defined) to study anomalies in the network stream.

1.3.1. Covert channel example : Wrapping arbitrary data into HTTP

Some tools allow building covert channels through legitimate HTTP data streams using HTTP header parts or HTTP body parts, using different HTTP methods, using covering and steganographic techniques, etc (Refer to [1] - 5. COVERING AND STEGANOGRAPHIC METHODS and to [4] in general).

1.3.2. Running a behavioral-based detection engine

Running a behavioral-based detection engine first involves creating reference user profiles (users, workstations, servers, network streams, etc) in a clean environment. These behavioral profiles are then applied into the production environment and used to match differences between real time user profiles and reference profiles.

Matching a difference is done using statistical methods. A threshold based on the reference profile is calculated and if a production value exceeds the reference threshold, the operator is notified.

1.3.3. Statistical approaches

We present here to the reader some statistical approaches. All of these methods could be used to gather information from the recorded network streams in order to calculate values which would be checked against the reference(s) profile(s).

Probabilistic detection

It involves creating a context which sets that the probability for an event D to appear after an events sequence A, B and C is X% and to group probability contexts into a reference profile.

The reference profile context probability values could be tunable and updated all along the detection engine running period.

Example :

What is the probability for a connection to occur for more than five times during a five minutes delay between the same source address and the same destination address and to exceed one megabyte if we are between 2AM and 3AM ?


  Event_A : connection from source address 's' to destination address 'd'.
  Event_B : connection 's->d' number 'n'.
  Event_C : delay since the beginning of the 's->d' connections 't'.
  Event_D : amount 'a' of data from 's' to 'd' since the beginning of the 's->d' connections in kbytes.
  Event_E : 'Event A' occurs between 'h' (ex: 2AM) and 'H' (ex: 3AM).

  Learning Period on a per day basis ?

  if ('Event_A')
    then
      /* 
      ** Store the number of 's->d' connections occuring at 'h' 
      */
      Learning_Connections['Event_A']['h'] += 1;
      /* 
      ** Store this specific information 
      */
      if (('n' > 5) and ('t' < 5) and ('a' > 1000))
        then
          Learning_Counter['h'] += 1;
          Reset('n','t','a');
      fi
  fi

  Running Period on a per day basis ?

  if ('Event_A')
    then
      /* 
      ** Store the number of 's->d' connections occuring at 'h' 
      */
      Running_Connections['Event_A']['h'] += 1;
      if (('n' > 5) and ('t' < 5) and ('a' > 1000))
        then
          Running_Counter['h'] += 1;
          Reset('n','t','a');
          if ((Running_Counter['h']/Running_Connections['Event_A']['h']) >
              (Learning_Counter['h']/Learning_Connections['Event_A']['h']))
            then
              /*
              ** Apply here some threshold functions ?
              */
              Raise_Alarm("'s->d' probability between 'h' and 'h+1' was exceeded.");
          fi
      fi
  fi

Pure statistical detection

This model involves checking quantitatively the parameter values of specific resources during a certain time (often called "learning period") to generate a reference profile.

When the detection engine then runs in a production period, it dynamically checks the reference profile against real time specific resources values using statistical operations and (basically) if a difference upper than a X% threshold is calculated, an alarm or whatever is raised.

Example :

Run the detection engine in learning mode for 't' hours, hour after hour 'h' and record the amount of data 'a' from the source address 's' to the destination address 'd' and the number of connections 'n' occuring on each 's->d' communications.


  Learning Period ?

  :Initialization
  t_test = 0;
  t_current = 'h';

  /*
  ** Comes here for each connection : from SYN to FIN
  */
  :Begin
  if ('h' > t_begin)
    then
      t_test += ('h' - t_current);
      t_current = 'h';
  fi      

  if (t_test >= 't')
    then
      Learning_Period_Is_Over();
    else
      Learning_Counter['s->d']['h'] += 1;
      Learning_Amount['s->d']['h'] += 'a';
  fi

  Running Period ?

  /*
  ** Comes here for each connection : from SYN to FIN
  */
  :Begin

  Running_Counter['s->d']['h'] += 1;
  Running_Amount['s->d']['h'] += a;

  if (Running_Counter['s->d']['h'] > Learning_Counter['s->d']['h'])
    then
      /*
      ** Apply here some threshold functions ? 
      */
      Raise_Alarm("At 'h', 's->d' exceeded learnt Counter value);
  fi
  if (Running_Amount['s->d']['h'] > Learning_Amount['s->d']['h'])
    then
      /* 
      ** Apply here some threshold functions ?
      */
      Raise_Alarm("At 'h', 's->d' exceeded learnt Amount value");
  fi

Note that this kind of statistical detection is implemented into the "Statistical Packet Anomaly Detection Engine" SPADE [5] module for the Snort [7] NIDS with the "Stealthy Portscan and Intrusion Correlation Engine" SPICE [6] to detect port scans.

Neuronal network detection

That kind of detection is based on the concept that it is possible to set an activities profile for each user in a clean environment.

This profile is then represented as a neuronal network which watches X user actions and tries to guess the X+1 one or learns the user actions during a T period and then tries to guess the next one.

When a guess is false (guess which can be threshold balanced), an alarm or whatever is raised.

Example :

If we take the example of the 'Pure statistical detection' part, we can suppose that we learn each hour, hour after hour, the amount of data sent 'as' or received 'ar' and the number of connections 'n' occuring on each 's->d' communications.

We now present a way to watch users actions during a T period and then try to guess the next one.

  :Begin
  T = Current_Hour();
  N = Number_of_hour_in_the_learning_period;
  Running_Counter['s->d'][T] = Update_The_Current_Hour_Values();

  for ((H = each_hour_of_the_learning_period))
    do
      Average_Counter += Learning_Counter['s->d'][H];
      if (Running_Counter['s->d'][T] > Learning_Counter['s->d'][H])
        then 
          /*
          ** Number of time the number of connections exceeded a previous number 
          ** of connections
          */
          Number_of_match += 1;
        fi
    done

  /*
  ** Number of connections (average) per hour
  */
  Average_Counter /= N;
	
  if (Running_Counter['s->d'][T] > Average_Counter)
    then
      /*
      ** The number of connection for this 'have to guess' hour exceeds the
      ** average value
      */
      if (Number_of_match > 0)
        then
          /* 
          ** The connection number for this 'have to  guess' hour exceeded
          ** Number_of_match times a previously recorded information
          */

          /*
          ** Apply here some other threshold balancing functions ?
          */

          /*
          ** I now guess that this  current number  of connections was not
          ** higher than 75%  of  the  previously  number  of  connections
          ** recorded during the learning period... If I'm wrong  then  we
          ** raise an alarm
          */
          if (((Number_of_match / N) * 100) > 75)
            then
              Raise_Alarm("At H, Running Counter Guess was false");
          fi
      fi
  fi

Scenario approach using the Bayes inference

That kind of detection is based on a signature detection and on an attack scenario probability at the same time.

We could set up a system in which each suspicious matched signature (hypothetical attack) found in the monitored data stream is part of a global set (symptoms) and use each global set to calculate, with a Bayes inference, the probability for a known attack to be on hold knowing the P(Hypothetical attack / Symptoms) probability.

If the detection engine finds a suspicious scenario which probability value is greater than a set threshold, an alarm or whatever is raised.

Example :

Well, I cannot find any good example for this approach... If someone has an idea ?

1.3.4. Behavioral-based detection engine and covert channels

The statistical detection models presented in 1.3.3. could be usable to detect covert channel and tunneling data streams with a less or more high false positive rate.

We can take in consideration the following aspects when thinking about a future implementation : Do we need a learning period or not ? Do we need a way to dynamically tune some of the detection engine parameters such as specific thresholds ? Is there a way to drastically reduce the false positive rate if we only watch some kind of data streams or data patterns ?

2. GW IMPLEMENTATION THEORETICAL DESIGN

Whereas we know it is, at the present time, unbelievable to detect covert and tunneling tools using some of the advanced methods presented in [1], we feel it would be easy to detect and track the most common covering tools.

We will try thereafter to present the design we will use for our development stage so that readers could have an idea of our implementation goals and decide to help us if they want to :)

2.1. WHAT KIND OF INFORMATIONS ARE WE LOOKING FOR ?

We will now join our IDS theories knowledge to the NACS bypassing techniques presented in [1] to design an unauthorized HTTP usage detection engine.

So, what should exactly raise an alarm from a theoretical point of view ? The most common type of Covert Channels utilization are unauthorized tunnels and backdoor communications (refer to [1] for such examples).

We'll suppose for the next parts that the security policy used is an exemplary one and is based on the obvious assumption : "We block everything, then we allow specific and precise access".

An unique outgoing data stream is authorized through a DMZ located mandatory HTTP proxy allowing internal located users to surf the web using the HTTP and HTTPS protocols :


  Workstation_1 -|
  Workstation_2 -|----> Firewall_1 ----> HTTP_PROXY ----> Firewall_2 ----> WWW
  Workstation_X -|(Statefull_80_443_OUT)            (Statefull_80_443_OUT)

* Firewall_1 and Firewall_2 label refer to the firewall rules and not the physical server (i.e. : An unique server could be used).
* The HTTP CONNECT method is only authorized if the destination port is the TCP 443 one.
* Last words regarding this NACS architecture : Redundant network equipments could be used for Firewalls and mandatory HTTP proxies, self-made rotating rules could be done on the OSI layer 2 to break SSL connections occuring for more than a certain delay, etc...

2.1.1. Covert Channels detection : unauthorized tunnels

Covert channel methods establishing unauthorized tunnels are often used to transit real world application data streams. A Covert Channel (CC) client opens a legitimate connection through a NACS and uses this connection to send and receive arbitrary data from a CC server.

Questions are now : "What characterizes such kind of covert channel tunnels ?" and "What kind of tunnel traces can we get from the network streams ?".

We will now present some kind of tunnel traces. Some of them may be redundant to the reader point of view but we are trying here to have a (non- exhaustive) list of tunnel traces.

Unauthorized tunnel traces :

(1) : High amount of inbound/outbound data transiting through the NACS - Inside POST or PUT requests for example - These high data amounts may occur one time only or be generated during a specific period delay.
(2) : Small but frequent amount of inbound/outbound data transiting through the NACS - A CC telnet-like communication channel will, for example, generate a very small but frequent amount of data.
(3) : Requests/responses are sent too frequently or are sent on a curious static interval.
(4) : One "user" requests a small quantity of external resources too frequently.
(5) : The data stream transiting through the NACS contains some specific patterns such as SSH banners for examples.
(6) : The data stream transiting through the NACS generates HTTP protocol anomalies or violations.

The first four tunnel traces could be detected by a behavioral-based detection engine. The fifth one could be detected by a signature-based one while the last tunnel trace could be detected by a protocol-based one.

2.1.2. Covert Channels detection : backdoor communications

A backdoor client gets installed onto the user workstation via email, www script, floppy disk or anything else. Since it cannot build a direct connection to its master server, it fetches the local box mandatory proxy settings and sends an HTTP message through the NACS to its master server :


  POST http://evilhost.com/cgi-bin/server.cgi HTTP/1.0
  Host: evilhost.com
  Content-Length: 42
  Content-Type: application/octet-stream

  Hey, I finally waked me up. Are you ready?

And, of course, the master server is alive and answers with the corresponding HTTP response message telling smartly to the backdoor client : "Please, my dear backdoor client, may I ask you to send me what we defined as precious yesterday night and then I will let you sleep for 1 week ...".

Since the previous generated HTTP traffic is extremely low, it is almost impossible to detect such communication channels. But we will suppose in the next parts that the intruder is not a guru and didn't paid attention to the "Proprietary user defined protocol" mode presented in [1] nor did carefully designed his client/server tool to confuse our detection engine with some of the "Covering and steganographic methods" presented in [1].

So, let's try to imagine some common characteristics and unauthorized traces involved in a backdoor communication.

Unauthorized tunnel traces :

(1) : High amount of inbound/outbound data transiting through the NACS - Inside POST or PUT requests for example - These high data amounts may occur one time only or be generated during a specific period delay.
(2) : Small amount of inbound/outbound data transiting through the NACS.
(3) : The data stream transiting through the NACS generates HTTP protocol anomalies or violations.

2.2. THE LOCATION OF THE DETECTION ENGINE ?

Another question is : "Where is the detection engine located?", it is not a trivial one.

When an HTTP client wants to initiate an HTTP communication with an HTTP server, it opens a TCP connection to the HTTP server. But, if an HTTP intermediary (Not a transparent proxy) is the only way out of the internal network for the HTTP client, the client opens a TCP connection to the HTTP intermediary.

We now face the first design problem : It won't be possible to obtain source AND destination IP addresses from the 3rd layer of the OSI model. On the front of the HTTP intermediary (i.e. : on the internal side), we will get the source IP addresses and behind the HTTP intermediary, we will get the destination ones.

So, the easier way to get source AND destination IP addresses from the data streams between the clients and the proxy server would be to implement a detection engine which analyzes each packet at the 3rd, 4th and upper layers of the OSI model and which would be located on the front of the HTTP intermediaries (i.e. : between the HTTP client(s) and the first HTTP intermediary).

Note that when the NACS HTTP protection scheme is based on transparent proxies, this problem shall not occur.

We should also keep in mind that the corporate web servers of our DMZ can be used to setup scripts such as WSH [8] or FIREPASS [9]. Detecting such a bad [12] traffic may require the detection engine to be located on another network than the user network presented in 2.1. WHAT KIND OF INFORMATIONS ARE WE LOOKING FOR ?

Another detection engine location could be on the middle of each HTTP communication channel between HTTP clients and intermediaries. In other words, it would be feasible to design a 'gateway' HTTP intermediary as it is defined in [10]. This gateway would be located on the front of the standard HTTP intermediaries and would send data to monitor to the detection engine before proxying the requests streams to the standard HTTP intermediaries.

The previous "in-the-middle" location could also be used to design specific engines running as modules for the popular and available HTTP proxies.

2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ?

Analyzing the following information should be sufficient to detect the most common covering tools in the first development stage of the detection engine.

Global values for each client :
  * Number of destination servers (a).
  * Number of TCP connections (same per server).
  * Amount of outbound data (same per server).
  * Amount of inbound data (same per server).
  * Amount of communication channels uptime (same per server).
  * Amount of TCP connection uptime (same per server).

Average values per client :
  * Number of destination servers.
  * Number of TCP connections / destination servers number (a).
  * Amount of outbound data /destination servers number (a).
  * Amount of inbound data / destination servers number (a).
  * Amount of communication channels uptime / destination servers number (a).
  * Amount of TCP connection uptime / destination server number (a).

Destination servers (a) Top20 and Bottom20 (b) values for each client :
  * per number of TCP connections.
  * per amount of outbound data.
  * per amount of inbound data.
  * per amount of time a communication channel stands up.
  * per amount of time a TCP connection stands up.

Notes :
  (a) : This data would be only be convenient if the destination server IP address is the real destination IP address and not the proxy one.
  (b) : The Top20 contains the most accessed servers while the Bottom20 contains the less accessed servers.

The previous Global, Average, Top20 and Bottom20 values could also be calculated on a specific time period - Shall an hour by hour time period be granular enough ? - as it would be possible to monitor the evolution of one configured value all along the detection run time.

It also would be convenient to calculate global average for all of the monitored clients :
  * on a detection run time period.
  * on a specific time period.
  * on all of the Top20/Bottom20 values.

The previous monitoring could be ran during a learning period. This would generate a reference profile of the production network data stream exchanges.

Have a look on the ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED part of this document to learn about the data structure we are actually planning to use.

2.4. WHAT ABOUT THE STATISTICAL METHODS PRESENTATION ?

And you now ask the obvious but quite good question : "Why did this guy tell us about statistical methods ?".

The usual Web user behavior is something which could not be easily defined. Any Web user clicks two times or more on an hyperlink, uses the back and next buttons one time, two times, three times, without any good (?) reason. Any web user clicks and clicks and clicks again and again on the reload button because he thinks the download rate will be faster or because he thinks the page content should/will change ?

This user behavior would decrease our detection engine skills if it watches for the unauthorized tunnel traces presented in 2.1.1. Covert Channels detection : unauthorized tunnels (traces 2, 3 and 4) or presented in 2.1.2. Covert Channels detection : backdoor communications (trace 2).

Plenty of browser add-ons or server-side applications exist which "help" the web crawler to clean the kitchen, to give food to the dog or to choice a gift for ... Basically, all of these proprietary web appliances would increase the amount of TCP connections, the amount of outbound and inbound data, etc.

Again, our detection engine may be raising false positive alarms if it uses the kind of traces described in 2.1.1. Covert Channels detection : unauthorized tunnels (trace 1) and in 2.1.2. Covert Channels detection : backdoor communications (trace 1).

We also have to mention that some browser or web servers may not follow the RFC specifications. Such a browser/server may raise an alarm in a protocol -based detection engine while no covert communication occurred.

And again, our detection engine could be confused, is if it implements a trace presented in 2.1.1. Covert Channels detection : unauthorized tunnels (trace 6) or presented in 2.1.2. Covert Channels detection : backdoor communications (trace 3).

Balancing monitored values using statistical methods should then help to reduce the false positive rate.

Using statistical methods could also be useful to assign different threshold calculation methods regarding the period of the day (night, day, lunch, etc.), the destination web servers, etc...

2.5. WHAT ABOUT THE PROTOCOL-BASED DETECTION ?

It also would be quiet interesting if the application data stream related to the monitored set could be monitored in real time by a third party (or not ?) module which would apply some kind of protocol-based detection : It would then be feasible to check (for example) that connections using the HTTP CONNECT method really begin with a SSL handshake, that exotic HTTP methods are not used (refer to [1] - 3. USING HTTP METHODS), etc...

Another interesting possibility would be to record a whole (or part of a) data stream if an anomaly score is reached. This could lead to further investigation by a third party module ?

2.6. LEARNING PERIOD AND/OR STATIC PRE-CONFIGURATION ?

This question may be easily answered. If the engine design includes a learning period functionality, it would be quiet a good idea to implement some basic configuration through the use of a static file.

This would allow the user to set a detection engine with his specific network parameters. For example, this user may choice to raise an alarm if a TCP connection lasts for more than five minutes or if a same source IP address / destination IP address uploads/downloads more than x kbytes.

2.7. DETECTION vs PREVENTION ASPECTS

I focused this paper on the detection of unauthorized covert channels and tunnels running over the HTTP protocol, but as we had a long discussion with Alex on this topic, here is a few comments about these aspects.

The main goal of an IDS obviously is the detection of something. Sometimes (but more and more since a few years), the IDS implements an active response mode which allow the user to choice the kind of action the IDS must perform when it matches that "something".

For a NIDS (Network IDS), the usual active response mechanisms consist to stop an intrusion (before, while or after it occurs) : Telling a firewall to dynamically load specific rules, sending crafted TCP reset, telling someone to isolate the attacked server, telling someone to jail the bad [12] attacker, etc.

Note that there is few underlying problems related to the NIDS active response modes. One of these problems is that the NIDS shall have an OSI layer 3 address (on a TCP/IP environment) to communicate with the other network devices - but this is not a real problem if this communication occurs on an isolated management network. It only becomes a problem if the NIDS is configured to reset the "bad" [12] connections as it will need a valid IP address on the monitored network and thus could be targeted by the attacker. Another problem resides on the fact that allowing a NIDS to response to an attack may be used by an attacker who will try to trigger some of the NIDS rules in order to DOS part or all of a network resource (including the NIDS itself).

Another concept on the prevention topic exist with the Intrusion Prevention Systems (IPS) which are based on the concept that it is possible not only to detect occuring attacks but also to prevent them to be successful. This is a new security research field at the current time and it is widely commented on the security discussion lists (have a look on [11] for an introduction).

Basically, an IPS acts on the lower layers of the OSI model and is often told "inline" as data streams must transit through it to reach their destination. Does this seems to be quite related to one of the engine location we described in 2.2. THE LOCATION OF THE DETECTION ENGINE ? ?. For the gurus, I'll now suppose that we're dealing with network-IPS.

So you now may ask: what is the main difference between an NIDS and a NIPS ? and the answer is as easy as the difference between their name. A NIDS detects an attack and is then able to inform anyone (including itself) that an attack occurred - but this attack passed through the NACS - whereas the NIPS gets the data stream and tries to detect an attack before passing this data stream to the next intermediaries it protects.

The IPS concept however has to deal with some problems. A NIDS "fails open" - that is : if the NIDS crashes, no one knows it (well, except the admin I suppose ...) - whereas an IPS may "fail closed" - that is : all of the world knows it and you can wait for the phone to ring in a few seconds... . Another problem when adding inline network devices and asking them to detect something is that it may require to increase the resources (memory, cpu, disk storage, ...). A theoretic but last problem we focuse on in this part is that when you have something "inline" watching for some bad [12] guy to come in and that this bad guy knows how to trigger your fantastic new toy, you'll have more to do than only shutdown that equipment as you did it when you were using that old NIDS - And I'm no more speaking of technical jobs to do but merely of how many people you'll have to do that job in your security response team.

CONCLUSION

This paper is a prelude to an implementation as we are actually thinking about the best way to design a detection engine.

We are actually thinking on a Snort [7] preprocessor plugin detection model. Using the Snort capture API, we could process data streams packet per packet and store the information presented in 2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ? in an appropriate data structure.

We feel that the covert channel detection engine should be designed on a two part tool : a data stream monitor which listens on the wire and records data in a proprietary binary form and a correlation engine (modules-based) which would be in charge to investigate the data streams recorded for eventual anomalies. The communication between these two parts could be socket-based for a real time detection model or file-based for a batch detection model. But this is our first step design model ;).

We also feel that it is not possible on a quiet secure NACS scheme to only investigate for anomalies on the 3rd and 4th layers of the OSI model. It would be better to implement a detection engine which basically understands some of the HTTP protocol specifications (header/body, URLs, URIs, etc...) so that we could use the destination servers IP addresses and implement basic protocol-based detection rules.

It is obvious that the network informations we are planning to monitor (as described in the 2.3. WHAT KIND OF INFORMATIONS ARE WE PLANNING TO MONITOR ? part of this paper) could of course be used to watch for covert channel or tunnel occuring over any protocol other than the HTTP one.

To conclude, this is a really interesting research topic and we would appreciate any contribution (remark, suggestion, link to an already done detection tool) helping us to update this paper or giving us hints for our development stage.

ANNEXE 1 - DATA STRUCTURE ACTUALLY DESIGNED


  |----------------|
  |  t_main_infos  |
  |----------------|
          ||
          \/
  |----------------|
  |   t_src_infos  | ===> t_src_infos (x)
  |     (first)    |
  |----------------|
          ||
          \/
  |----------------|
  |   t_dst_infos  | ===> t_dst_infos (x)
  |     (first)    |
  |----------------|
          ||
          \/
  |----------------|
  | t_period_infos | ===> t_period_infos (x)
  |    (first)     |
  |----------------|

Note : Period infos is a kind of modulo. Ex: If the period is 4 hours, then there is 24/4 = 6 period elems of 4 hours... and it is modulo because after a full day period, the algo loops on the first period_info elem.


  /* Double-linked list Elem */
  typedef struct s_infos_list_elem {
    void *datas;
    t_dst_infos_list_elem *next;
    t_dst_infos_list_elem *previous;
  } t_infos_list_elem;

  /* Destination server period informations for a client */
  typedef struct s_period_infos {
    struct timeval starttime;
    u_int32_t number_of_tcp_connections;
    u_int32_t outbound_data_amount;
    u_int32_t inbound_data_amount;
    u_int32_t com_channel_uptime;
    u_int32_t tcp_com_uptime;
  } t_period_infos;

  /* Destination server informations for a client */
  typedef struct s_dst_infos {
    struct in_addr ip_dst;
    u_int16_t dst_port;
    /* first_of_period->datas <=> (t_period_infos *) */
    t_infos_list_elem *first_of_period;
  } t_dst_infos;

  /* Information struct element for each clients */
  typedef struct s_src_infos {
    struct in_addr ip_src;
    /* first_of_servers->datas <=> (t_dst_infos *) */
    t_infos_list_elem *first_of_servers;
    u_int32_t number_of_servers;
  } t_src_infos;

  /* main and global data struct of the detection plugin */
  typedef struct s_main_infos {
    struct timeval starttime;
    /* first_of_clients->datas <=> (t_src_infos *) */
    t_infos_list_elem *first_of_clients;
    u_int32_t number_of_clients;
  } t_main_infos;

THANKS

Big thanks to the GW Team :

o Alex <alex at gray-world.net> for his review, comments, ideas and a "DOS the Internet" party during our mails exchanges :)

o Brian <b.otto at runbox.com> who searched and found :\ my english spelling mistakes.

Thanks to Arun Darlie Koshy <root [at] arunkoshy.cjb.net> who accepted to check the pre-release version of this paper, giving you thus something easier to read :)

Thanks to <simonis [at] att.net> who suggested me that the first phrase of the Introduction part - "Detecting Covert Channels and Tunneling data streams is a relatively new (theoretical) research area. [...]" - was as incomplete as irrelevant.

WEBOGRAPHY

[1]: Exploitation of data streams authorized by a network access control system for arbitrary data transfers : tunneling and covert channels over the HTTP protocol - v1.0 - Alex Dyatlov & Simon Castro , June 2003
http://gray-world.net/projects/papers/covert_paper.txt

[2]: NetCat, Hobbit, 20th of March 1996
http://www.atstake.com/research/tools/network_utilities/

[3]: OpenSSH
http://www.openssh.org

[4]: Network Access Control System (NACS) bypassing
http://www.gray-world.net

[5]: SPADE - Statistical Packet Anomaly Detection Engine
http://www.silicondefense.com/products/freesoftware/spade/

[6]: SPICE - Stealthy Portscan and Intrusion Correlation Engine
http://www.silicondefense.com/products/freesoftware/spice/

[7]: Snort
http://www.snort.org/

[8]: WSH
http://www.gray-world.net/

[9]: FIREPASS
http://www.gray-world.net/

[10]: Hypertext Transfer Protocol -- HTTP/1.1, RFC 2616 , 1999
http://www.w3.org/Protocols/rfc2616/rfc2616.html

[11]: Intrusion Prevention Systems: the Next Step in the Evolution of IDS Neil Desai - 27th of February 2003
http://www.securityfocus.com/infocus/1670

[12]: The Bible
http://www.thebible.net/read/

HTML paper presentation v1.1 / 22/11/2003
Original version : http://gray-world.net/projects/papers/cctde.txt