FireEye Stories

FireEye and Citrix Tool Scans for Indicators of Compromise Related to CVE-2019-19781

On December 17, 2019, Citrix released a security bulletin (CTX267027) that described a vulnerability in Citrix Application Delivery Controller (ADC), Citrix Gateway, and two older versions of Citrix SD-WAN WANOP. At the same time, Citrix released mitigations for these vulnerabilities. The vulnerability, assigned CVE-2019-19781 and deemed Critical in severity, could allow an unauthenticated attacker to perform arbitrary remote code execution via directory traversal.

On January 14, 2020, FireEye published a blog post revealing some of the threat activity we had seen surrounding the vulnerability, including repeated exploitation attempts in the travel, legal, financial, and education sectors. The post also contains detections and other guidance for defending against this threat. A few days later on January 17, FireEye released another blog post detailing the activities of an actor that we observed gaining access to vulnerable devices, cleaning up known malware, and deploying a previously unseen payload to block follow-up exploitation attempts. However, this payload—we now refer to as NOTROBIN—also serves as a backdoor.

A Tool to Help Identify Compromised Systems

To help organizations identify compromised systems associated with CVE-2019-19781, FireEye and Citrix worked together to release a new tool that searches for indicators of compromise (IoC) associated with attacker activity observed by FireEye Mandiant. This tool is freely accessible in both the Citrix and FireEye GitHub repositories.

The free tool is designed to allow Citrix customers to run it locally on their Citrix instances and receive a rapid assessment of potential indications of compromise in the system based on known attacks and exploits. In addition to applying the previously released mitigation steps and installing the permanent updates that have been made available, Citrix and FireEye strongly recommend that all Citrix customers run this tool as soon as possible to increase their overall level of awareness of potential compromise and take appropriate steps to protect themselves.

Compatibility

The tool is designed to run on the following versions of Citrix Application Delivery Controller (ADC), Citrix Gateway, and Citrix SD-WAN WANOP appliances:

  • Citrix ADC and Citrix Gateway version 13.0
  • Citrix ADC and Citrix Gateway version 12.1
  • Citrix ADC and Citrix Gateway version 12.0
  • Citrix ADC and Citrix Gateway version 11.1
  • Citrix ADC and Citrix Gateway version 10.5
  • Citrix SD-WAN WANOP version 10.2.6
  • Citrix SD-WAN WANOP version 11.0.3

Instructions

Download the tool from the release tab on GitHub. You should use the packaged, standalone build because it’s easiest to copy and run a single file.

The IoC Scanner can be run directly on a Citrix ADC, Gateway, or SD-WAN WANOP system. On a live system, the tool will scan files, processes, and ports for known indicators. The tool writes diagnostic messages to the STDERR stream and results to the STDOUT stream. In typical usage, you should redirect STDOUT to a file for review. The tool must be run as root in live mode on a system.

For example:

$ sudo bash ./ioc-scanner-CVE-2019-19781-v1.0.sh > "/tmp/results-$(date).txt"

The IoC Scanner can also inspect a mounted forensic image. In this scenario, pass a command line argument specifying the path to the image root directory. You don't have to be root to run in offline mode.

For example:

$ bash ./ioc-scanner-CVE-2019-19781.sh /mnt/path/to/evidence/root/

Interpreting the Results

This tool was developed by FireEye Mandiant based on knowledge gleaned from incident response engagements related to exploitation of CVE-2019-19781. The goal of the scanner is to analyze available log sources and system forensic artifacts to identify evidence of successful exploitation of CVE-2019-19781. There are limitations in what the tool will be able to accomplish and therefore executing the tool should not be considered a guarantee that a system is free of compromise. For example, log files on the system with evidence of compromise may have truncated or rolled, the system may have been rebooted, or an attacker may have tampered with the system to remove evidence of compromise and/or installed a rootkit that masks evidence of compromise.

The output of this tool will fall into one of three categories:

  1. Evidence of compromise. By default, the tool will only show higher fidelity hits that may indicate a system compromise. This could be anything from executing commands that disclose information (e.g. view the ns.conf or smb.conf configuration files), to installing a backdoor (e.g. NOTROBIN, a coin miner, etc.), or dropping a Perl-based web shell.
  2. Evidence of successful vulnerability scanning. This could be authorized activity from a system administrator or unauthorized activity by an attacker. Any evidence that falls into this category indicates the system was in a vulnerable state (e.g. the mitigation or patch had not been applied) and the first step (out of a two-step process) to exploit CVE-2019-19781 was successful.
  3. Evidence of unsuccessful vulnerability scanning. Any evidence that falls into this category indicates that attempts to scan or exploit the system did not succeed.

We ran this tool on a Citrix ADC appliance that was exposed to the internet and vulnerable to CVE-2019-19781. The scanner identified a lot of evidence of compromise, scanning, and failed exploitation. Let’s walk through these results.

First, the scanner identified many different types of compromise. Figure 1 shows an example of the output. The scanner found:

  • an unexpected listening UDP port, consistent with NOTROBIN,
  • an unexpected process owned by the user nobody,
  • blacklisted terms in files possibly created by exploits,
  • files with metadata consistent with exploitation, and
  • web access logs showing exploit HTTP requests.

Alone, each of these sources of evidence is a strong indicator of compromise. Taken together, we can be confident that this system was compromised. We should initiate a forensic investigation to determine the scope of the compromise.

********************************************************************** MATCH: UDP port 18634, known artifact of NOTROBIN.
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
 **********************************************************************
 identified ports:
 nobody   httpd      33153 3  udp4 6 *:18634               *:*


**********************************************************************
MATCH: unexpected process owned by user 'nobody'
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
processes owned by nobody:
nobody    33153  0.0  0.2  9724  6688  ??  Is   13Jan20 416:45.44 /var/nstmp/.nscache/httpd.


**********************************************************************
MATCH: blacklisted content 'bash'
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
matches for 'bash':
///netscaler/portal/templates/4BSqAXIaOM.xml

**********************************************************************
MATCH: incorrect file permissions
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
files with permissions 644:
///netscaler/portal/templates/VaO8p.xml

**********************************************************************
MATCH: incorrect file permissions
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
files with permissions 644:
///var/vpn/bookmark/rSvRrQhb.xml

**********************************************************************
MATCH: web access logs show      1 instances of successful HTTP exploitation
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
web access log entries:
//var/log/httpaccess.log.3.gz:127.0.0.2 - - [12/Jan/2020:02:05:21 +0000] "GET /vpn/../vpns/portal/yVStWwCFy9BDXBxjIGvCk3h67Gx4Zm8E.xml HTTP/1.1" 200 224868 "https://1.2.3.4/vpn/../vpns/portal/yVStWwCFy9BDXBxjIGvCk3h67Gx4Zm8E.xml" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/58.0" "Time: 135214 microsecs"


**********************************************************************
MATCH: incorrect file permissions
Found evidence of potential compromise.
You should consider performing a forensic investigation of the system.
**********************************************************************
files with permissions 644:
///var/vpn/bookmark/BSmith.xml
///var/vpn/bookmark/JDoe.xml
///var/vpn/bookmark/mxrnpdifet.xml
///var/vpn/bookmark/irdpkcgaco.xml
///var/vpn/bookmark/tohxqlqlwo.xml
Please review the above paths for any unexpected files.
Exploits commonly write to files with these permissions; however, customization of a Citrix NetScaler environment may cause false positives in the above list.
For example, '/var/vpn/bookmark/[legitimate-username].xml' may be valid in your environment.

Figure 1: Example output showing evidence of compromise

Second, the tool used web server access logs to identify scanning activity that targeted this appliance. Figure 2 shows an example of the output. Scanning activity does not confirm a compromise; however, we’ve found that systems left in a vulnerable state and targeted by scanning were often compromised. This evidence also encourages us to initiate a forensic investigation to confirm compromise.


**********************************************************************
SCANNING: web access logs show 4 instances of successful HTTP scanning Found evidence of successful scanning. The device was probably vulnerable during the period of scanning. There is a strong likelihood of compromise. **********************************************************************
web access log entries:
/var/log/httpaccess.log.5:127.0.0.2 - - [12/Jan/2020:12:53:59 +0000] "POST /vpn/../vpns/portal/scripts/rmpm.pl HTTP/1.1" 200 225 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"

/var/log/httpaccess.log.4:127.0.0.2 - - [14/Jan/2020:22:14:37 +0000] "GET /vpn/../vpns/cfg/smb.conf HTTP/1.0" 200 - "-" "-"

/var/log/httpaccess.log.4:127.0.0.2 [15/Jan/2020:00:10:49 +0000] "POST /vpn/../vpns/portal/scripts/newbm.pl HTTP/1.1" 200 135 "-" "curl/7.67.0"

/var/log/httpaccess.log:127.0.0.2 - - [16/Jan/2020:13:49:16 +0000] "HEAD /vpn/../t/../vpns/./cfg/smb.conf HTTP/1.1" 200 - "-" "curl/7.47.0"

Figure 2: Example output showing scanning activity

The scanner looked for HTTP access logs focused on the /vpns/ directory, with a .pl or .conf file extension, and a HTTP status code of 200.  In our experience, this is consistent with successful vulnerability scanning. This might be authorized activity from a system administrator or unauthorized activity by an attacker. Any evidence that falls into this category indicates the system was in a vulnerable state (e.g. the mitigation or patch had not been applied) and the first step (out of a two-step process) to exploit CVE-2019-19781 was successful.

Finally, the tool used web server access logs to identify instances of failed exploitation. Figure 3 shows an example of this output. These entries indicate that an attacker tried to gain access to the appliance but it failed. There is not enough evidence to know why exploitation failed, though reasonable explanations include: the device was since mitigated or patched, there was  a bug in the exploit, or a competing actor is blocking further exploitation. We should consider whether we’ve successfully patched the appliance before deciding to initiate a forensic investigation.


**********************************************************************
FAILED EXPLOITATION: web access logs show 2 instances of failed HTTP exploitation Found evidence of failed exploitation. An actor attempted to exploit the device; however, it failed. This may be due to any of many things, including: - the device was patched correctly - the exploit had a bug - another actor (such as NOTROBIN) blocked it **********************************************************************

/var/log/httpaccess.log.5:127.0.0.2 - - [12/Jan/2020:12:53:59 +0000] "POST /vpn/../vpns/portal/scripts/rmpm.pl HTTP/1.1" 404 225 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0"

/var/log/httpaccess.log.4:127.0.0.2 - - [13/Jan/2020:08:28:31 +0000] "GET /vpn/../vpns/portal/mWK6N4fOqFFgZ7puKwxaAVDV4Kk5zmg8.xml HTTP/1.1" 404 48 "-" "curl/7.58.0"

Figure 3: Example output showing failed exploitation

The scanner looked for HTTP access log entries consistent with exploitation and with a HTTP status code of 404 (“File Not Found”). Any evidence that falls into this category indicates that attempts to scan or exploit the system likely did not succeed.

Remember, the tool will not make an assertion that a system has not been compromised. The tool will only state when IoCs are identified. It will also not provide formal malware family names of all malicious tools and scripts identified on compromised systems, nor will it identify the existence of all malware or evidence of compromise on the system. The tool is limited to the tool-related indicators that FireEye is aware of at the time of release of the tool or tool-related indicators.

If indications of compromise are identified on systems, organizations should perform a forensic examination of the compromised system to determine the scope and extent of the incident.

Additional Information

For more information, Citrix has provided additional context regarding the tool in a blog post.

Download

Download the tool from our Github.

Disclaimer

This tool is not guaranteed to find all evidence of compromise, or all evidence of compromise related to CVE-2019-19781. If indications of compromise are identified on systems, organizations should perform a forensic examination of the compromised system to determine the scope and extent of the incident. This tool is offered AS IS and without warranty.