Web Attack Surface Monitoring System: A Step-by-Step Tutorial

In today's digital age, monitoring online assets has become more crucial. Cybersecurity threats are on the rise, and companies must take proactive measures to protect their digital footprint.

In this article, you will learn how to build a simple yet effective monitoring system that can alert you regarding any changes in your sub-domains, take screenshots of active web servers, and host the result locally for easy access.

We will be using open-source tools such as:

• Amass by OWASP: For sub-domain enumeration and its built-in tracking functionality.
• gowitness by sensepost: For taking screenshots and hosting a result in a local web server.
• httpx by projectdiscovery: To probe for web services.
• Notify by projectdiscovery (optional): For sending alerts if any changes occur.

The article assumes that you have some familiarity with these tools and basic knowledge of bash scripting, so let's get started!

Set Up and Installation of the Tools

This article will not cover detailed installation instructions, so please refer to their wiki for assistance regarding the installation. However, if Go is pre-installed on your machine, all the tools mentioned can be installed simply with the following commands.

go install -v github.com/owasp-amass/amass/v3/...@master

go install github.com/sensepost/gowitness@latest

 go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

go install -v github.com/projectdiscovery/notify/cmd/notify@latest

Only installing them is not enough, as they might have dependencies that need to be installed separately. For example, gowitness requires Chromium to use its screenshot functionality. So, refer to their wikis to set them up properly before proceeding.

Setting up Amass

Built and actively maintained by OWASP, Amass is a great tool for attack surface mapping. Although it has many awesome features and functionality, its ability to gather information from various sources sets it apart from others.

For this, you must register and get API keys from the sources (both paid and free). By default, Amass will look for the configuration files in $HOME/.config/amass/config.ini. Once you are done registering and have a bunch of APIs, ensure to include them in a config file.

There are lots of ways to configure Amass to suit your needs. This article will only use its most basic setup for sub-domain enumeration and tracking changes.

Setting up Notify

Notify is a neat Go-based, open-source project developed and maintained by ProjectDiscovery. It can be used to publish the output of several tools to your choice of platforms.

For our demo, we will be using it to send alerts to our Slack channel to keep track of changes.

Set up a configuration profile

Get the Webhook API from your Slack application and include it in a configuration file as shown. By default, Notify will look for the configuration profile in $HOME/.config/notify/provider-config.yaml.

Now, test to see if it's working.

echo "hello word" | notify

Building the Script

Sub-domain enumeration with Amass

We will create a simple function for enumerating sub-domains using Amass.

recon_domains() {
  echo "[*] Starting sub-domain enumeration..."
  amass enum -src -ip -df $DOMAINS_FILE -o $SUBDOMAINS_FILE -dir $AMASS_DIR -config $CONFIG_DIR/amass/config.ini
  echo "[*] Sub-Domain Enumeration finished"
}

Command reference:

-src: Prints the sources for the discovered names 
-ip: Prints the ip addresses for the discovered names
-df: Specifying path to file with list of Root domains
-o: Specifying file to save result 
-dir: Specifying directory in which amass will save its logs, and database
-config: Specifying path to configuration

As you can see, I have only used some of the basic options. You can play around with the options to optimize the Amass. I used Amass for sub-domain enumeration, but you could use amass intel to enumerate root domains as well. It depends on your use case, so play around with the tools and see what works for you.

Probe for active web services using httpx

After running Amass, we will have a file with a bunch of sub-domains it discovered. All the sub-domains will not necessarily have web services running on them. To filter them, we will use httpx to probe each sub-domain in the file to determine whether they have any web services running on them.

find_active_domains() {
  echo "Finding active sub-domains..."
  cat $SUBDOMAINS_FILE | awk '{ print $2; }' | httpx -threads 30 -timeout 60 -retries 5 -silent -o $ACTIVE_DOMAINS_FILE 
  echo "Active sub-domain discovery complete!"
}

Command reference:

-threads: Number of threads to use
-timeout: Request timeout duration (in seconds)
-retries: number of retries before moving to next target
-silent: Silent mode
-o: File where the result will be saved 

Similar to Amass, httpx offers a lot of functionalities and features, so explore and customize the settings according to your need. You can also use config files instead of using options and flags.

Send a notification if any changes are discovered

Amass track can be used to compare the results between two runs and find the changes between them, i.e. removal/discovery of the new sub-domain.

send_slack_notification() {
    echo "[*] Checking for new subdomains..."
    track_output=$(amass track -df "${DOMAINS_FILE}" -dir "${AMASS_DIR}")
    found_output=$(echo "${track_output}" | awk '/Found:/ { print }')
    removed_output=$(echo "${track_output}" | awk '/Removed:/ { print }')

    # Send Slack alert if any new sub-domain is discovered or previously discovered sub-domain was removed from the record.
    if [[ -n "${found_output}" ]] || [[ -n "${removed_output}" ]]; then
        
        echo "Sending Slack notification..."
        notify_output=""
        
        if [[ -n "${found_output}" ]]; then
            echo "Sending notification for new subdomains..."
            notify_output="${found_output}"
        fi
        
        if [[ -n "${removed_output}" ]]; then
            echo "Sending notification for removed subdomains..."
            notify_output="${notify_output}\n${removed_output}"
        fi
        echo -e "${notify_output}" | notify -bulk -silent -pc "${CONFIG_DIR}/notify/provider-config.yaml" -id subdomain_monitoring
        echo "Slack notification sent!"
    fi
}

Here, amass track will compare the latest results with the results from the last run. You can use -last flag to specify the number of runs you want to compare it to. In our case, if any changes (discovery/removal) have occurred between the last two runs, a Slack alert is sent to notify you regarding them.

Take a screenshot and serve the result

After httpx, we will have a list of domains with active web services running on them instead of manually checking each one individually. We will use gowitness, which will do the grunt work for us. It will take screenshots of the website and even spin a web app of its own, hosting the collection of screenshots it took.

take_screenshots() {
  echo "Taking screenshots of active domains..."
  gowitness file -f $ACTIVE_DOMAINS_FILE --db-path $GOWITNESS_DIR/gowitness.sqlite3 -P $GOWITNESS_DIR/screenshots/ --disable-logging
  echo "Screenshots complete!"
}

serve_screenshots() {
  echo "Serving screenshots on local server..."
	
  # Stop the server that was running previously
  if PID=$(pgrep -f 'gowitness server -A'); then
    echo "Stopping existing gowitness server with PID $PID..."
    kill "$PID"
    sleep 2
  fi

  gowitness server -A --disable-logging --db-path $GOWITNESS_DIR/gowitness.sqlite3 -P $GOWITNESS_DIR/screenshots/ &> /dev/null
  echo "Screenshots server started!"
}

The Final Script

With everything put together, the final script should look something like this.

#!/bin/bash

## Variables
PROJECT_DIR='/home/kaxuwa/blog/asm'
DOMAINS_FILE="$PROJECT_DIR/files/domains.txt"
SUBDOMAINS_FILE="$PROJECT_DIR/files/sub-domains.txt"
ACTIVE_DOMAINS_FILE="$PROJECT_DIR/files/active-domains.txt"
AMASS_DIR="$PROJECT_DIR/amass-data"
GOWITNESS_DIR="$PROJECT_DIR/gowitness"
CONFIG_DIR="$PROJECT_DIR/.config"

# Functions
recon_domains() {
  echo "Starting domain recon..."
  amass enum -src -ip -df $DOMAINS_FILE -o $SUBDOMAINS_FILE -dir $AMASS_DIR -config $CONFIG_DIR/amass/config.ini
  echo "Domain recon complete!"
}

find_active_domains() {
  echo "Finding active sub-domains..."
  cat $SUBDOMAINS_FILE | awk '{ print $2; }' | httpx -threads 30 -timeout 60 -retries 5 -silent -o $ACTIVE_DOMAINS_FILE 
  echo "Active sub-domain discovery complete!"
}

take_screenshots() {
  echo "Taking screenshots of active domains..."
  gowitness file -f $ACTIVE_DOMAINS_FILE --db-path $GOWITNESS_DIR/gowitness.sqlite3 -P $GOWITNESS_DIR/screenshots/ --disable-logging
  echo "Screenshots complete!"
}

serve_screenshots() {
  echo "Serving screenshots on local server..."

  if PID=$(pgrep -f 'gowitness server -A'); then
    echo "Stopping existing gowitness server with PID $PID..."
    kill "$PID"
    sleep 2
  fi

  gowitness server -A --disable-logging --db-path $GOWITNESS_DIR/gowitness.sqlite3 -P $GOWITNESS_DIR/screenshots/ &
  echo "$!" > /dev/null
  echo "Screenshots server started!"
}

send_slack_notification() {
    echo "[*] Checking for new subdomains..."
    track_output=$(amass track -df "${DOMAINS_FILE}" -dir "${AMASS_DIR}")
    found_output=$(echo "${track_output}" | awk '/Found:/ { print }')
    removed_output=$(echo "${track_output}" | awk '/Removed:/ { print }')

    # Send Slack alert if any new sub-domain is discovered or previously discovered sub-domain was removed from the record.
    if [[ -n "${found_output}" ]] || [[ -n "${removed_output}" ]]; then
        echo "Sending Slack notification..."
        notify_output=""
        if [[ -n "${found_output}" ]]; then
            echo "Sending notification for new subdomains..."
            notify_output="${found_output}"
        fi
        if [[ -n "${removed_output}" ]]; then
            echo "Sending notification for removed subdomains..."
            notify_output="${notify_output}\n${removed_output}"
        fi
        echo -e "${notify_output}" | notify -bulk -silent -pc "${CONFIG_DIR}/notify/provider-config.yaml" -id subdomain_monitoring
        echo "Slack notification sent!"
    fi
}

echo "Starting web service monitoring script..."

# Recon domains and find active domains
recon_domains
find_active_domains
send_slack_notification

# Take screenshots and serve locally
take_screenshots
serve_screenshots

The script is simple and without fail checks, logging, etc., for demo purposes. You must modify it if you plan on using this extensively for monitoring. Save the script with .sh extension i.e <script_name>.sh, and make it executable chmod u+x script.sh.

Set up crontab

While you can run the script manually at any time, running it regularly is recommended for constant monitoring. For demonstration, we will set up a crontab which will run this script every Monday at 10am. To create a crontab, run crontab -e on your terminal and add the following line at the end of the file.

Here, 0 is the minute (0-59), 10 is for the hour (0-23), followed by the day of the month and month, which are both set to any i.e *.

1 indicates the day of the week (0-7, where both 0 and 7 represents Sunday) and finally, the command to execute, which in our case is the script we just wrote.

Demo and Summary

Since the Amass takes a list of root domains from a file, we must manually create a file with root domains.

echo "example.com" > domains.txt

Make sure the domains.txt file is in the path as stated in the script, then execute the script with the following:

bash /path/to/your/asset-monitoring.sh

Running the script will give us the following result if there are no errors.

With this, you now have a basic monitoring system. It will look for new subdomains and send you alerts on Slack and the web server, which you can check for any irregularities. You can further include other tools, such as gobuster for directory bruteforcing, nuclei for vulnerability management etc.

That's it for this blog. Subscribe using either of the buttons below.