Guides

Watchdog Server User Guide

Overview

This guide explains how the watchdog server works in simple terms. It focuses on how the server checks a controller's license and provides the correct settings based on which department it belongs to. It also includes a new section on the key expiry feature and instructions on accessing the Grafana dashboard for data analytics.
Workflow
- Step 1: Sending Information from the Controller
The controller gathers details about itself, such as:
```
- Unique ID (GUID)
- Hostname (name of the computer)
- Version (software version)
- IP addresses
- MAC address (unique identifier for network interfaces)
- Operating system (OS)
- Label (a tag to identify the machine)
```
This information is sent to the /api/check-license/ endpoint on the watchdog server.
- Step 2: Checking the License
The server receives the information and checks if the license is valid. It ensures all required details are present and that the GUID is formatted correctly. If everything checks out and the GUID is not already in use, it registers the GUID. If the number of active devices exceeds the allowed limit, the request is denied.
- Step 3: Matching Information to Configuration Settings
The server reads the configuration and mapping files. It uses the details from the controller to find the best matching department based on criteria like label, IP address, hostname, MAC address, and operating system.
- Step 4: Sending Configuration to the Controller
Once the right department is identified, the corresponding settings from the browsermon-watchdog.conf file are selected. These settings are customized based on the controller's operating system (e.g., setting the log directory path differently for Windows and Linux).
- Step 5: Config is sent to Controller
The new config from browsermon-watchdog.conf is sent to the controller along with the valid license message.

Key Expiry Feature
- Overview
  
  The key expiry feature allows administrators to set an expiry date for the keys used by controllers. This ensures controllers must periodically check in with the watchdog server to renew their keys, enhancing security and control over access.
- Workflow
  - Step 1: Generating and Assigning Keys
  When a controller initially registers with the watchdog server or when a key expires, the server generates a new key with an associated expiry date. This key is then assigned to the controller.
  - Step 2: Checking Expiry
  Each time the controller communicates with the watchdog server, the server checks the expiry date of the assigned key.
  - Step 3: Logging Remaining Days
  Upon successful communication, the watchdog server logs the remaining days until the key expires. This information is logged for monitoring purposes.
  - Step 4: Key Expiry
  Once the expiry date is reached, the key is considered expired. The controller will no longer be able to receive configurations or updates from the watchdog server until a new key is generated and assigned.

Implementation Details
- Configuration
  
  Administrators can configure the expiry duration for keys in the watchdog.conf file.
- Logging
  
  The remaining days until key expiry are logged in the watchdog server's log files. Administrators can monitor these logs to ensure timely renewal of keys.
- Grafana Dashboard
  
  The watchdog server integrates with Grafana to provide a comprehensive interface for viewing history and data analytics.
- Accessing the Grafana Dashboard
  
  Users can view the Grafana dashboard by navigating to http://localhost:1514 in their web browser. This dashboard presents the history and analytics data collected by the watchdog server in an intuitive and visual format.
Benefits of the Grafana Dashboard
- Data Visualization: The Grafana dashboard offers various charts, graphs, and tables to help users visualize historical data and trends.
- Customizable Views: Users can customize the dashboard to display the most relevant metrics and information for their needs.
- Real-Time Monitoring: The dashboard allows for real-time monitoring of data, providing up-to-date insights into the performance and status of the watchdog server.
EUNOMATIX Threat Intel (ETI)

If during installation ETI mode was set, then watchdog will automatically create a threat collector to collect fresh threat intel from phishtank and urlhaus. This data is stored in centralized elastic database running on port 9200. Browsermon endpoints will be able to classify browsed urls into threat categories. The threat collector will collect data daily at midnight to keep the database freshly updated. Inside watchdog.conf you can set the value for eti_index_ttl which represent Time-to-live (TTL) in days for Elasticsearch threat index before deletion. For the functioning of the ETI, the following domains must be accessible from the network where your watchdog is deployed.
- ETI API : eti.eunomatix.com
EUNOMATIX URL Classification Service (UCS)

Before Installation if you want to get daily classification updates, set ucs_updates to true in watchdog.conf. If during installation UCS mode was set, then watchdog will automatically create a ucs_client to collect daily classification updates from EUNOMATIX UCS API. Otherwise if ucs_updates is set to false inside watchdog.conf, then no API calls will be made and local snapshot will automatically get restored for UCS Classification. To get UCS daily updates, following cloud URL (https://ucs.eunomatix.com) should be a accessible to the centralized watchdog instance.
- UCS API : ucs.eunomatix.com
Watchdog Deployment Guide (with Kafka and Elasticsearch)
Introduction

This guide explains how to install and configure Watchdog using the watchdog-installer Python script. Watchdog can optionally integrate with Kafka (for data ingestion) and Elasticsearch (for data storage and searching).

The installer supports:
1. Interactive prompts for Docker registry authentication (optional).
2. Enabling/disabling Kafka mode and/or Elasticsearch mode.
3. Automatic creation of necessary directories under /opt/watchdog.
4. File-by-file copy of important Watchdog files (prompts only for /opt/watchdog/watchdog/ overwrites).
5. Automatic generation of a .env file in your current directory, containing the environment variables Docker Compose will need.
6. A final Docker Compose deployment that launches the selected services.

Prerequisites
1. Root/Sudo Access
  The installer must be run as root (or with sudo). It manages system directories (e.g., /opt/watchdog) and sets ownership of data directories.
2. Docker and Docker Compose
3. Docker installed and running (docker ps should work).
4. Docker Compose plugin or Docker Compose CLI installed.
5. Optionally, Docker registry credentials if you plan to pull images from a private Docker registry.
6. Local Files/Directories
7. A local deps/ directory that contains:
  - deps/connect-jars/ (Kafka connector JARs).
  - deps/watchdog/ (Watchdog source files).
  - deps/init-kafka-connect.sh (initialization script).
8. Docker Compose YAML files in the same directory from which you run the installer:
  - docker-compose.base.yml (required).
  - docker-compose.kafka.yml (if enabling Kafka).
  - docker-compose.elastic.yml (if enabling Elasticsearch).
  - docker-compose.eti.yml (if enabling ETI).
  - docker-compose.ucs.yml (if enabling UCS).
9. Optional config files (if needed for custom setups):
  - eti.yml (if elastic_mode=true and you want to override default ES config).
  - Any custom .conf files for Watchdog (placed in deps/watchdog before running the script).
Installation Steps
1. Clone or place the watchdog-installer script in the same directory where your docker-compose.*.yml files exist (because it writes a .env file locally and references the compose files in the current directory).
2. Ensure the script is executable:
```
chmod +x watchdog-installer
```
  If you’re using the Python file directly, you can just run python watchdog-installer install without chmod +x.
3. Run the installer (as root):
```
sudo ./watchdog-installer install
```
4. The script will:
  1. Prompt you for Docker registry authentication (optional).
  2. Prompt whether to enable Kafka/ETI/UCS modes.
  3. If Kafka mode is enabled, prompt for a KAFKA_EXTERNAL_IP.
  4. If Elasticsearch mode is enabled, prompt for username, passwords, etc.
  5. Create /opt/watchdog, /opt/watchdog/kafka_data, and /opt/watchdog/elasticsearch_data as needed.
  6. Copy files from deps/ into /opt/watchdog.
    - connect-jars and init-kafka-connect.sh are forced overwrites (no prompt).
    - The watchdog directory is copied file-by-file with a prompt for each existing file.
  7. Generate a .env file in your current directory (where Docker Compose can see it).
  8. Finally, run docker compose up -d using docker-compose.base.yml, plus the Kafka and/or Elastic Compose files if those modes were selected.
5. Verify installation:
6. Check running containers:
```
docker ps
```
7. If Kafka was enabled:
  - kafka, zookeeper, and kafka-connect containers should be running.
8. If Elasticsearch was enabled:
  - An elasticsearch container (and possibly kibana) should be running (depending on your compose files).

Environment Variables and .env File

The script automatically writes environment variables to a .env file in the current working directory. Docker Compose will automatically load them. If Kafka/Elasticsearch is enabled, you’ll see lines like:
```
KAFKA_EXTERNAL_IP=your.machine.ip
ELASTIC_HOST=elasticsearch
ELASTIC_PORT=9200
ELASTIC_PASSWORD=BrowsermonElasticAdmin
ELASTIC_USERNAME=Browsermon
ELASTIC_USER_PASSWORD=BrowsermonElasticUser
ELASTIC_SCHEME=https
```
You can modify these directly if needed (though re-running the script may overwrite them).

Configuration Files

Depending on your Watchdog setup, you might need additional configuration files within deps/watchdog (which eventually lands in /opt/watchdog/watchdog): 1. watchdog.conf
2. ssl-config.ini
3. mapping.conf
4. browsermon-watchdog.conf 5. elasticsearch.yml (if ETI/UCS=true and you want to override defaults)

Make sure to place these files in deps/watchdog before running the installer if you want them copied to /opt/watchdog/watchdog.

Service Configuration
1. Kafka
2. Typically uses port 8092 (or whatever is in your docker-compose.kafka.yml).
3. Uses Kafka Connect to push data to MongoDB (or other sinks).
4. The init-kafka-connect.sh script is placed in /opt/watchdog, but you typically don’t need to run it manually unless your setup requires it.
5. MongoDB
6. Often deployed alongside Kafka (depending on your docker-compose.kafka.yml).
7. The sink connector is configured to push Watchdog data to MongoDB.
8. Elasticsearch
9. Typically listens on 9200 for HTTP/HTTPS calls.
10. The default scheme is https (from the script prompt) but can be changed if you have a custom ES config.
11. If using elasticsearch.yml, it should be placed in deps/watchdog or your custom location and referenced by docker-compose.elastic.yml.

Updating the Installation

If you re-run the installer and /opt/watchdog is detected, the script enters Update Mode. You will be: - Prompted only for overwriting files inside /opt/watchdog/watchdog. - Other files (like init-kafka-connect.sh or connect-jars) are overwritten automatically. - The script will then re-run Docker Compose to update containers.

Example:
```
sudo ./watchdog-installer install
```
If it sees an existing installation, you’ll be asked:

Existing installation detected at /opt/watchdog
Do you want to proceed with the update? (y/n)

Uninstalling / Cleaning Up

To stop and remove the Watchdog containers (Kafka/Elasticsearch included), run:
```
sudo ./watchdog-installer clean
```
This will: 1. Look for docker-compose.base.yml, docker-compose.kafka.yml, and docker-compose.elastic.yml in your current directory. 2. Run docker compose down -v with whichever files are found, removing containers and volumes.
Note: This does not delete /opt/watchdog or the data directories. If you want to remove them entirely, you can do so manually:
```
sudo rm -rf /opt/watchdog
```

Troubleshooting
- Checking Logs
View logs for a specific container:
```
docker logs <container-name>
```
Examples:
- docker logs kafka-connect
- docker logs elasticsearch
- Verifying Kafka Connect
Inside the Kafka Connect container:
```
docker exec -it kafka-connect /bin/bash
```
Then check connector status:
```
curl -X GET http://kafka-connect:8083/connectors/mongo-sink-connector/status
```
A valid Mongo Sink Connector shows:
```
```json
{
"name": "mongo-sink-connector",
"connector": {
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
},
"tasks": [
    {
    "id": 0,
    "state": "RUNNING",
    "worker_id": "connect-worker-1"
    }
],
"type": "sink"
}
```
```
- Checking Elasticsearch
If Elasticsearch is running with HTTPS and basic auth:

for default configurations
```
curl -k -u elastic:BrowsermonElasticAdmin https://localhost:9200/_cluster/health
```
```
- `-k` ignores self-signed certificate errors.
- Adjust the user/password as you configured them during installation prompts.
```
- Internet Access Important For the functioning of the Elasticsearch-based URL classification, the following domains must be accessible from the network where your watchdog is deployed.
For ETI:
- ETI API : eti.eunomatix.com
For UCS (if ucs_updates=true in watchdog.conf):
- UCS API : ucs.eunomatix.com
If watchdog is behind a proxy server, relevant proxy settings must be enabled in watchdog.conf file before installation.

Example:
```
[proxy]
proxy_mode=false
http_proxy=http://10.10.10.10:1234
https_proxy=https://10.10.10.10:1234
```
- If proxy server has aunthentication, user should enter url according to it e.g http://username:[email protected]:8080
- Both http and https proxies should be specified in watchdog.conf even if you only have http proxy.
- If proxy is running on your host machine you should use host.docker.internal or host.containers.internal depends on your docker setup as host of proxy instead of localhost.
Offline Image Deployment (Optional)

If you have Docker images saved locally (e.g., .tar files) for offline deployment:
1. Load them:
```
docker load -i your_offline_watchdog_image.tar
```
2. Skip Docker Hub Login during the script’s prompts.
3. Ensure the Docker Compose files reference the images you loaded (matching tags).

Guides

Watchdog Server User Guide

Watchdog Deployment Guide (with Kafka and Elasticsearch)