The Dutch Honeynet chapter

Compiling the Honeyspider Network 2 Source Code

Recently HSN version 2.1 was released, see https://groups.google.com/forum/#!category-topic/honeyspider-network-2/DN2VbD9c618. Honeyspider Network is a highly-scalable system integrating multiple client honeypots to detect malicious websites. It was developed as a joint venture between CERT Polska and NCSC-NL. For more information, see http://www.honeyspider.net.

While a lot of improvements are being made in the development branch and now released in version 2.1, the binaries lack behind. Because of this I tried to compile from Git myself. This proved to be a bit difficult, mainly because of some dependencies not available via Maven anymore. In this blog I will show how to compile and configure HSN2 yourself.

The HSN2 framework is still considered experimental software. Installation and configuration still has its rough edges. The main benefit for using a framework is to have easy access to multiple plugins. If you want to write your own plugin see Niels van Eijk his Java One presentation for an introduction on writing your own plugin:

Compiling from GIT

For these instructions I assume a Debian 8 64-bit base-system. I tested this with the master branch (HSN 2.1) which was just released.

  • start with installing the following packages via apt-get:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
apt-get install \
build-essential \
devscripts \
debhelper \
openjdk-7-jdk \
maven \
python-all \
python-setuptools \
python-pika \
protobuf-compiler \
uuid-dev \
libssl-dev \
libconfig-dev \
libcurl4-openssl-dev \
libjson0-dev \
libarchive-dev \
libemu2 \
git \
libfuzzy2
  • There are two dependencies that cannot be obtained automatically anymore via Maven, so install those first in the following steps.

  • Install commons-ognl-4.0-SNAPSHOT manually (replace the tags):

1
2
3
4
5
6
apt-get install subversion

svn checkout http://svn.apache.org/repos/asf/commons/proper/ognl/trunk/ commons-ognl
cd commons-ognl
mvn package
mvn install:install-file -DgroupId=org.apache.commons -DartifactId=commons-ognl -Dversion=4.0-SNAPSHOT -Dpackaging=jar -Dfile=<REPLACE_WITH_YOUR_DIR>/commons-ognl/target/commons-ognl-4.0-SNAPSHOT.jar
1
mvn install:install-file -DgroupId=swfutils -DartifactId=swfutils -Dversion=0.3 -Dpackaging=jar -Dfile=<REPLACE_WITH_YOUR_DIR>/flexcover-0.90/sdk-modifications-3_2/lib/swfutils.jar
  • Obtain the HSN2 source via github (you can also choose to obtain the development branch):
1
git clone --recursive https://github.com/CERT-Polska/hsn2-bundle
  • Compile with Maven while in the main cloned Git repo:
1
2
cd hsn2-bundle
mvn package
  • When finished, the summary should look like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
[INFO] Reactor Summary:                                                                                                                              [0/1910]
[INFO]
[INFO] HSN2 Commons :: Root .............................. SUCCESS [1.144s]
[INFO] HSN2 Commons :: Protocol Buffers Impl ............. SUCCESS [15.957s]
[INFO] HSN2 Commons :: Bus ............................... SUCCESS [31.892s]
[INFO] HSN2 Commons :: Services Commons .................. SUCCESS [10.683s]
[INFO] HSN2 Commons :: Utils ............................. SUCCESS [1.545s]
[INFO] HSN2 Framework :: Root ............................ SUCCESS [0.718s]
[INFO] HSN2 Framework :: Configuration Manager ........... SUCCESS [1.890s]
[INFO] HSN2 Framework :: Core API ........................ SUCCESS [2.189s]
[INFO] HSN2 Framework :: Workflow :: Engine .............. SUCCESS [41.827s]
[INFO] HSN2 Framework :: Workflow :: HWL Parser .......... SUCCESS [4.230s]
[INFO] HSN2 Framework :: Workflow :: Git Repository ...... SUCCESS [2.938s]
[INFO] HSN2 Framework :: Main ............................ SUCCESS [3.409s]
[INFO] HSN2 Data Store ................................... SUCCESS [9.359s]
[INFO] HSN2 Object Store (MongoDB) ....................... SUCCESS [1.405s]
[INFO] HSN2 File Feeder .................................. SUCCESS [1.899s]
[INFO] HSN2 Web Client ................................... SUCCESS [3:25.166s]
[INFO] HSN2 Shellcode Analyser ........................... SUCCESS [2.737s]
[INFO] HSN2 URL Normalization Service .................... SUCCESS [2.280s]
[INFO] HSN2 JavaScript Analyzer .......................... SUCCESS [5.666s]
[INFO] HSN2 Capture HPC connector service ................ SUCCESS [16.023s]
[INFO] SWF ............................................... SUCCESS [0.001s]
[INFO] SWF CVE Plugin Commons ............................ SUCCESS [0.935s]
[INFO] SWF Tool .......................................... SUCCESS [14.308s]
[INFO] SWF server ........................................ SUCCESS [2.863s]
[INFO] cve_2007_0071 plugin .............................. SUCCESS [8.358s]
[INFO] cve_2009_1869 plugin .............................. SUCCESS [6.312s]
[INFO] SWF Tool Bundle ................................... SUCCESS [0.001s]
[INFO] SWF Service ....................................... SUCCESS [14.592s]
[INFO] HSN2 Service Reporter ............................. SUCCESS [5.716s]
[INFO] HSN2 command console .............................. SUCCESS [5.291s]
[INFO] HSN2 Cuckoo Java .................................. SUCCESS [1:33.914s]
[INFO] HSN2 MD5 To SSDeep ................................ SUCCESS [1.677s]
[INFO] HSN2 DNS Info ..................................... SUCCESS [17.111s]
[INFO] HSN2 Bundle ....................................... SUCCESS [0.004s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:55.242s
[INFO] Finished at: Tue May 17 16:24:39 CEST 2016
[INFO] Final Memory: 41M/113M
[INFO] ------------------------------------------------------------------------

Building Debian packages (and repo)

  • If compiling succeeds, you can also do a build and automatically generate Debian packages. You can of course also run and configure the jar files manually (which can take quite some time to figure out). Run the following script:
1
debian-build.sh
  • If you want to set up a local Debian repro, continue with the following steps:
1
2
apt-get install reprepro
debian-repo.sh yourreponame
  • Install apache2 and put the repo in /var/www/html
  • add in /etc/apt/source.list:
1
deb http://localhost/hsn2debrepo/ experimental main
  • Finally, to test the new repo, do:
1
apt-get update

Install and configure HSN2

Also see http://www.honeyspider.net/Installation.html.

  • First make sure the following packages are installed:
1
apt-get install rabbitmq-server openjdk-7-jre libmozjs185-1.0 libmozjs185-dev
1
apt-get install erlang
  • Make sure that rabbitmq and mongodb are running at startup (and start them)
1
2
3
4
5
update-rc.d rabbitmq-server defaults
update-rc.d mongodb defaults

systemctl rabbitmq-server start
systemctl mongodb start
  • Install the following subset of packages (installation will fail if rabbitmq/mongodb are not started), we skip python-hsn2-thug, hsn2-thug-docker and hsn2-capture-hpc for now.
1
apt-get install hsn2-commons-debian hsn2-cuckoo-java hsn2-data-store hsn2-dnsinfo hsn2-file-feeder hsn2-framework hsn2-js-sta hsn2-md5-to-ssdeep hsn2-norm-url hsn2-object-store-mongodb hsn2-rb-archiveinflate hsn2-rb-clamavnugget hsn2-rb-officecat hsn2-rb-pdffox hsn2-rb-swfscanner hsn2-rb-virustotal hsn2-reporter hsn2-shell-scdbg hsn2-unicorn hsn2-webclient python-hsn2-commons python-hsn2-console python-hsn2-malicious-domains python-hsn2-pcap-analyze python-hsn2-pcap-extract python-hsn2-proto python-hsn2-rb-nugget-commons python-hsn2-url-feeder python-hsn2-yara
  • Disable the following services at startup (they cause errors, start them manually in the right order):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
systemctl disable hsn2-cuckoo-java.service
systemctl disable hsn2-data-store.service
systemctl disable hsn2-dnsinfo.service
systemctl disable hsn2-file-feeder.service
systemctl disable hsn2-framework.service
systemctl disable hsn2-js-sta.service
systemctl disable hsn2-malicious-domains.service
systemctl disable hsn2-md5-to-ssdeep.service
systemctl disable hsn2-norm-url.service
systemctl disable hsn2-object-store-mongodb.service
systemctl disable hsn2-pcap-analyze.service
systemctl disable hsn2-pcap-extract.service
systemctl disable hsn2-rb-archiveinflate.service
systemctl disable hsn2-rb-clamavnugget.service
systemctl disable hsn2-rb-officecat.service
systemctl disable hsn2-rb-pdffox.service
systemctl disable hsn2-rb-swfscanner.service
systemctl disable hsn2-rb-virustotal.service
systemctl disable hsn2-reporter.service
systemctl disable hsn2-shell-scdbg.service
systemctl disable hsn2-url-feeder.service
systemctl disable hsn2-webclient.service
  • Make and run a script to start services manually in the right order. The following uncommented services should work by default:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#!/bin/bash
systemctl start hsn2-framework
systemctl start hsn2-object-store-mongodb
systemctl start hsn2-data-store
systemctl start hsn2-file-feeder
systemctl start hsn2-webclient
systemctl start hsn2-js-sta
systemctl start hsn2-norm-url
systemctl start hsn2-reporter
#systemctl start hsn2-rb-archiveinflate
#systemctl start hsn2-rb-clamavnugget
#systemctl start hsn2-rb-officecat
#systemctl start hsn2-rb-pdffox
#systemctl start hsn2-rb-swfscanner
#systemctl start hsn2-rb-virustotal
#systemctl start hsn2-shell-scdbg
#systemctl start hsn2-swf-cve
#systemctl start hsn2-url-feeder
  • Clone a set of initial workflows to work with:
1
git clone git://github.com/CERT-Polska/hsn2-workflows.git /etc/hsn2/workflows
  • Create file with a url on each line (at least one line) and submit a job:
1
hc j s simple feeder.uri=/home/<USERNAME>/uris.txt
  • Check whether the job was processed correctly:
1
hc j d <NUMBER OF JOB> 

Now that you have the HSN2 framework running you can dive into the different plugins (or write one yourself) and make your own workflows (see /etc/hsn2/workflows). Don’t forget to look trough the logging in /var/log/hsn2 for errors/pointers. A Django based webinterface is also available, see https://github.com/CERT-Polska/hsn2-webgui and http://www.honeyspider.net/Web-Interface.html.

Gert

The Spamhattan Project

Let’s develop a nextgen spamtrap and create intel feeds for .NL

A rising amount of criminals are spreading cryptoware in order to ‘make money’. Cryptoware is ransomware that secretly encrypts files, like documents and pictures, of innocent users. The criminals make money by selling the decryption key. Most of the cryptoware is spread via email. Virus-scanners and anti-spam solutions have a hard time in defending against those threats and often there are no Indicators of Compromise (IoC) that help detecting infected devices in an early phase.

How to defend against this new threat? It’s key to catch new versions and incarnations of cryptoware in an early phase. Most often the cryptoware is spread via email so it was decided to focus on attracting lots of emails by creating a next generation spamtrap honeypot. Why create something new? Many people have reported about running a spamtrap! Why is nobody sharing tools?

We decided to create the spamtrap honeypot based on these elements:

  • Should be Open-Source
  • Be able to receive and analyze spam messages in high volume
  • Act as a regular mailserver, NOT an open-relay (like Shiva)
  • Act as a honeypot: never send out any mail (bounces, ndr, etc)

What will The Spamhattan Project deliver? By deploying the spamtrap in strategic economic sectors in The Netherlands insight in spam targeted at Dutch citizens will be gained. Also information about the actors behind the spamruns can be gathered. Key organizations, like ISPs, major banks and NCSC-NL will be warned in an early phase when new cryptoware campains are detected.

Will this spamtrap be of use to other countries as well? In our current spamtrap deployment we’ve only attached .NL domains to our spamtrap. By deploying the spamtrap yourself and feeding the right spam-sources you will be able to create your own Intel feed. Information and tips&tricks in order to get the right spam feed will be provide in future blog postings and presensations.

A first version of the spamtrap honeypot has been created half a year ago and is currently analyzing loads of spam. Based on the results a next version is currently being developed. Later in 2016 this version will be released under an open-source license.

Jop

Revitalizing a Centralised Honeypot Framework

Bringing the dead back to life

In early 2005 the SURFids Framework, later renamed to SURFcert IDS, was developed (http://ids.surfnet.nl/wiki/doku.php). The unique concept was the centralised detection approach, based on honeypots, with decentralised sensors running OpenVPN. From a marketing perspective ‘IDS’ was chosen in the name, in that age a popular term. Many organisations worldwide have used this open-source framework, however with a last update on the code in 2011, the project slowly died.

In early 2015, several members of the HoneyNED project (https://www.honeyned.nl/), being part of HoneyNet (https://www.honeynet.org/), decided to revitalize SURFids under a new name: Anansi

Anansi is an African folktale character. He often takes the shape of a spider and is considered to be the spirit of all knowledge of stories. He is also one of the most important characters of West African and Caribbean folklore.

With the interest of different parties such as SURFnet (the Dutch National Research Network), a commercial organization and a governmental body, our HoneyNED chapter aims at providing the Honeypot community with a new state-of-the-art open source, centralised honeypot framework. The key concept is to create ‘dumb’ sensors that tunnel attacks through VPN to a centralised server, which runs multiple honeypots and/or analysing algorithms. The centralised server also includes a management interface to administer all sensors, honeypots and detection algorithm. The analysis part will be based on Elastic Search, Logstash and Kibana. Deploying a first proof-of-concept is scheduled for December 2015.

In order to have a sustainable base for Anansi a project board has been founded in which participating organisations collaborate with HoneyNED. All participants stress the importance to put the Anansi open-source code under the umbrella of an independent respectable and known foundation in order to facilitate third parties to join this project. Any third party that wishes an additional Anansi feature can contribute in-kind or will be able to make a financial donation to the foundation. The roadmap and development effort will be governed from this foundation.

Anansi is not intended to re-invent the wheel and therefore we’re looking forward to collaborate with existing projects like T-pot, Arakis and others. Please feel free to contact us if you see opportunities for collaboration.

HoneyNED will keep you updated on the Anansi developments.

Tommy & Rogier