Troubleshooting Guide

Troubleshooting Watchdog

This section provides some troubleshooting tips related to Watchdog.

Initial Checks

Check Port Status If you are experiencing issues with the Watchdog server, the first step is to check the status of the port Watchdog is listening on. Follow these steps to troubleshoot:
- Use netstat to Check Port Status
```
netstat -tuln | grep <watchdog_port> 
```
- Check if Watchdog is listening on the port and bind to 0.0.0.0
Logs
- Set the loglevel to DEBUG in conf file.
- Review the Watchdog logs and the Gunicorn Server logs. Watchdog generates logs in the logdir configured in conf file:
  - Watchdog Logs
  - Gunicorn Logs
Process Status
Check the status of the process if it is running or not.
- Check if the process is running
```
ps aux | grep Watchdog 
```
- Get the details of the process if it's running
```
ps -p <pid> -o pid,ppid,cmd,%cpu,%mem,etime
```

Docker Troubleshooting

Check Docker logs for issues:
```
docker logs container-name
```
If you encounter issues with Kafka, exec into the Kafka-Connect container:
```
docker exec -it kafka-connect /bin/bash
```

Check the status of the MongoDB sink connector:

curl -X GET http://kafka-connect:8083/connectors/mongo-sink-connector/status

{
  "name": "mongo-sink-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "connect-worker-1"
    }
  ],
  "type": "sink"
}

Kafka and Elasticsearch troubleshoot

1. Checking Logs

View logs for a specific container:

docker logs <container-name>

Examples: - docker logs kafka-connect - docker logs elasticsearch

2. Verifying Kafka Connect

Inside the Kafka Connect container:

docker exec -it kafka-connect /bin/bash

Then check connector status:

curl -X GET http://kafka-connect:8083/connectors/mongo-sink-connector/status

A valid Mongo Sink Connector shows:

{
  "name": "mongo-sink-connector",
  "connector": {
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
  },
  "tasks": [
    {
      "id": 0,
      "state": "RUNNING",
      "worker_id": "connect-worker-1"
    }
  ],
  "type": "sink"
}

3. Checking Elasticsearch

If Elasticsearch is running with HTTPS and basic auth:

curl -k -u elastic:BrowsermonElasticAdmin https://localhost:9200/_cluster/health

- -k ignores self-signed certificate errors. - Adjust the user/password as you configured them during installation prompts.

4. Internet Access

Important For the functioning of the Elasticsearch-based URL classification, the following domains must be accessible from the network where your watchdog is deployed.

For ETI:

PhishTank : phishtank.com
URLHaus : urlhaus.abuse.ch

For UCS (if ucs_updates=true in watchdog.conf):

EUNOMATIX Cloud : ucs.eunomatix.com:8000

Common Issues and Solutions

Server Not Starting

Review your Watchdog Config. Ensure that you are using compatible version of Watchdog.
Things that can go wrong in the config file.
MODE: Set the mode to either 'local' or 'cloud' based on your deployment.
BMKEY: Provide the Base64-encoded license key.
AUTHCODE: Add the Base64-encoded authorization code.
BLACKLIST: Specify the blacklist configuration.
LOGDIR: Path to your log directory.
LOGLEVEL: Set the desired loglevel. Set it to DEBUG if the program doesn't work the intended way.
LIMIT: Enable the rate limiting.
RATE: Give the rate on which API is accessible.
ALLOWEDHOSTS: Lists of the Hosts that can access Watchdog.
WATCHDOG_PORT: Port on which watchdog will be run.
CERTFILE: Path to the SSL Certificate File
KEYFILE: Path to the SSL Key File
SSLCONFIG: SSL Config to genrate the cert files.

Server Not Behaving As Expected

Review your SSL certificate is valid and not expired.
Ensure you are not exceeding the max controllers limit.

Contacting Support

Download the Watchdog troubleshooter from the website, to collect all logs and relevant information and save it into a file named watchdog_archive.zip, which you can share with the team that will help you fix the issue.

Run the script in the same folder as the Watchdog binary or optionally use absolute paths for LOGDIR, CERTFILE and KEYFILE in the config file.
Make sure that your config contains absolute path of log and cert files.
To run the troubleshooter, run the following command.

./WatchdogTS /path/to/your/watchdog.conf /path/to/your/Watchdog_binary

Privacy Disclaimer

Watchdog troubleshooter collects all relevant information that can help debug the issue. Here is the list of items BrowserMon troubleshooter will collect:

watchdog_*.log watchdog_gunicorn.log
SSL Certificates
System Information (i.e., platform, platform version, architecture, IP address, MAC address, processor, and RAM)
Process Information
Troubleshooter will run lsof to fetch real-time event properties related to Watchdog.

For further details, you can take a look at the troubleshooter script on Eunomatix’s GitHub.