2

Expand public installation and API documentation

This commit is contained in:
2026-03-12 14:35:39 +01:00
parent 34d6d3ddcb
commit f15839cf51
7 changed files with 907 additions and 103 deletions

263
docs/api.md Normal file
View File

@@ -0,0 +1,263 @@
# HTTP API
The daemon serves a small JSON API used by the built-in web UI.
## General behavior
- All endpoints are served from the same address as the web UI.
- Responses are JSON.
- Error responses use the shape `{"error":"..."}`.
- Unsupported HTTP methods return `405 Method Not Allowed`.
- There is no built-in authentication or TLS.
## `GET /healthz`
Liveness probe.
Example response:
```json
{
"status": "ok",
"time": "2026-03-12T18:22:11Z"
}
```
## `GET /api/overview`
Returns summary counters plus recent IP and recent event samples.
Query parameters:
- `limit`
- optional
- default: `50`
- maximum: `1000`
Main response fields:
- `total_events`
- `total_ips`
- `blocked_ips`
- `review_ips`
- `allowed_ips`
- `observed_ips`
- `recent_ips`
- `recent_events`
## `GET /api/events`
Returns recent raw events.
Query parameters:
- `limit`
- optional
- default: `100`
- maximum: `1000`
Each event includes:
- source and profile names
- timestamps
- remote and client IPs
- host, method, URI, path, status, and User-Agent
- decision, primary reason, all reasons, and whether it was enforced
- raw Caddy JSON
- current IP state and manual override
## `GET /api/ips`
Returns current IP state rows.
Query parameters:
- `limit`
- optional
- default: `100`
- maximum: `1000`
- `state`
- optional
- filter by one state such as `review`, `blocked`, `allowed`, or `observed`
Each row includes:
- `ip`
- `first_seen_at`
- `last_seen_at`
- `last_source_name`
- `last_user_agent`
- `latest_status`
- `total_events`
- `state`
- `state_reason`
- `manual_override`
- `last_event_id`
- `updated_at`
## `GET /api/recent-ips`
Returns aggregated IP rows over a recent time window.
Query parameters:
- `hours`
- optional
- default: `24`
- values less than or equal to zero fall back to `24`
- `limit`
- optional
- default: `200`
- maximum: `1000`
Each row includes:
- `ip`
- `source_name`
- `state`
- `events`
- `last_seen_at`
- `reason`
- `manual_override`
- optional `bot`
- `actions`
`actions` contains:
- `can_block`
- `can_unblock`
- `can_clear_override`
## `GET /api/ips/{ip}`
Returns the complete details for one IP address.
Response fields:
- `state`
- `recent_events`
- `decisions`
- `backend_actions`
- optional `investigation`
- `opnsense`
- `actions`
The current implementation returns up to:
- 100 recent events
- 100 decision records
- 100 backend action records
## `POST /api/ips/{ip}/investigate`
Forces a fresh investigation for the selected IP and returns the updated `IPDetails` payload.
This bypasses the cached investigation for that request.
No request body is required.
## Action endpoints
These endpoints accept an optional JSON body:
```json
{
"actor": "alice@example.net",
"reason": "short human-readable explanation"
}
```
If omitted:
- `actor` defaults to `web-ui`
- `reason` defaults to an action-specific fallback
All action endpoints return the updated `IPDetails` payload on success.
### `POST /api/ips/{ip}/block`
- Sets a manual force-block override.
- Default reason: `manual block`.
- If OPNsense is enabled, the daemon adds the IP to the configured alias if it is missing.
### `POST /api/ips/{ip}/unblock`
- Sets a manual force-allow override.
- Default reason: `manual allow`.
- If OPNsense is enabled, the daemon removes the IP from the configured alias if it is present.
### `POST /api/ips/{ip}/clear-override`
- Removes the local manual override.
- Default reason: `manual override cleared`.
- Does not directly call OPNsense.
### `POST /api/ips/{ip}/reset`
Backwards-compatible alias for `clear-override`.
## Response model details
### `investigation`
When present, `investigation` may contain:
- `ip`
- `updated_at`
- `error`
- `bot`
- `reverse_dns`
- `registration`
- `reputation`
### `bot`
- `provider_id`
- `name`
- `icon`
- `method`
- `verified`
### `reverse_dns`
- `ptr`
- `forward_confirmed`
### `registration`
- `source`
- `handle`
- `name`
- `prefix`
- `organization`
- `country`
- `abuse_email`
### `reputation`
- `spamhaus_lookup`
- `spamhaus_listed`
- optional `spamhaus_codes`
- optional `error`
### `opnsense`
- `configured`
- `present`
- `checked_at`
- optional `error`
## Example manual block request
```bash
curl -X POST \
-H 'Content-Type: application/json' \
-d '{"actor":"cli","reason":"confirmed credential stuffing"}' \
http://127.0.0.1:9080/api/ips/203.0.113.10/block
```
## Example recent IP query
```bash
curl 'http://127.0.0.1:9080/api/recent-ips?hours=24&limit=250'
```

237
docs/configuration.md Normal file
View File

@@ -0,0 +1,237 @@
# Configuration reference
The daemon is configured from one YAML file passed with `-config`.
Start from [`../config.example.yaml`](../config.example.yaml).
## Top-level structure
```yaml
server:
storage:
investigation:
opnsense:
profiles:
sources:
```
## `server`
Controls the built-in HTTP server.
- `listen_address`
- default: `127.0.0.1:9080`
- TCP listen address for both the UI and the JSON API.
- `read_timeout`
- default: `5s`
- `write_timeout`
- default: `10s`
- `shutdown_timeout`
- default: `15s`
The HTTP server has no built-in authentication or TLS.
## `storage`
- `path`
- default: `./data/caddy-opnsense-blocker.db`
- SQLite database path.
The parent directory is created automatically if it does not already exist.
## `investigation`
Controls bot detection and external IP lookups.
- `enabled`
- default: `true`
- Enables investigations entirely.
- `refresh_after`
- default: `24h`
- Reserved for future automatic revalidation logic. Current releases do not automatically refresh cached investigations; cached entries are reused until a manual `Refresh investigation` action is triggered.
- `timeout`
- default: `8s`
- Timeout applied to one investigation run.
- `user_agent`
- default: `caddy-opnsense-blocker/0.2`
- User-Agent sent to HTTP-based investigation providers.
- `spamhaus_enabled`
- default: `true`
- Enables Spamhaus DNSBL lookups for non-bot investigations.
- `background_workers`
- default: `2`
- Number of background workers that fetch missing investigations.
- `background_poll_interval`
- default: `30s`
- Delay between background queue refill passes.
- `background_lookback`
- default: `0s`
- If `0s`, the scheduler can pick any known IP missing cached intelligence.
- If greater than zero, only IPs seen within that lookback window are queued.
- `background_batch_size`
- default: `256`
- Maximum number of IPs to queue per scheduler pass.
### Built-in investigation sources
Current releases can collect:
- verified bot matches based on published ranges and reverse DNS logic
- probable bot hints based on the observed User-Agent
- reverse DNS and forward-confirmed reverse DNS
- RDAP registration details such as network name, organization, country, prefix, and abuse contact
- Spamhaus listed or not listed status
## `opnsense`
Controls the optional firewall backend.
- `enabled`
- default: `false`
- When `false`, the daemon stays in review-only mode and does not call OPNsense.
- `base_url`
- required when `enabled: true`
- Example: `https://router.example.test`
- `api_key`
- optional if `api_key_file` is set
- `api_secret`
- optional if `api_secret_file` is set
- `api_key_file`
- recommended
- Path to a file containing the OPNsense API key.
- `api_secret_file`
- recommended
- Path to a file containing the OPNsense API secret.
- `timeout`
- default: `8s`
- `insecure_skip_verify`
- default: `false`
- Only use this for development or tightly controlled environments.
- `ensure_alias`
- default: `true`
- If the target alias does not exist, the daemon will try to create it automatically.
### `opnsense.alias`
- `name`
- required when OPNsense is enabled
- `type`
- default: `host`
- `description`
- default: `Managed by caddy-opnsense-blocker`
### `opnsense.api_paths`
Advanced option for environments where the default OPNsense API endpoints differ.
Defaults:
- `alias_get_uuid`: `/api/firewall/alias/get_alias_u_u_i_d/{alias}`
- `alias_add_item`: `/api/firewall/alias/add_item`
- `alias_set_item`: `/api/firewall/alias/set_item/{uuid}`
- `alias_reconfigure`: `/api/firewall/alias/reconfigure`
- `alias_util_list`: `/api/firewall/alias_util/list/{alias}`
- `alias_util_add`: `/api/firewall/alias_util/add/{alias}`
- `alias_util_delete`: `/api/firewall/alias_util/delete/{alias}`
## `profiles`
`profiles` is a mapping. Each source references one profile by name.
Example:
```yaml
profiles:
public-web:
auto_block: true
suspicious_path_prefixes:
- /wp-admin
```
Supported fields per profile:
- `auto_block`
- When `true`, suspicious matches immediately become `blocked` decisions.
- When `false`, suspicious matches become `review` decisions.
- `min_status`
- default: `400`
- `max_status`
- default: `599`
- Only events within this inclusive status range are evaluated.
- `block_unexpected_posts`
- When `true`, `POST` requests are suspicious unless their normalized path is listed in `allowed_post_paths`.
- `block_php_paths`
- When `true`, paths ending in `.php` are suspicious.
- `allowed_post_paths`
- Exact normalized paths that remain allowed for `POST` requests.
- `suspicious_path_prefixes`
- Prefixes matched against the normalized request path.
- `/` is rejected because it would be too broad.
- `excluded_cidrs`
- CIDRs always allowed for this profile.
- `known_agents`
- Explicit allow or deny rules matched against User-Agent prefixes, CIDR ranges, or both.
### `profiles.<name>.known_agents[]`
- `name`
- Human-readable rule name.
- `decision`
- Required.
- Must be `allow` or `deny`.
- `user_agent_prefixes`
- Optional if `cidrs` is present.
- `cidrs`
- Optional if `user_agent_prefixes` is present.
At least one of `user_agent_prefixes` or `cidrs` must be defined.
## `sources`
`sources` is a list of monitored log files.
Each item supports:
- `name`
- required and unique
- `path`
- required and unique
- `profile`
- required
- Must reference an existing profile name.
- `initial_position`
- default: `end`
- Accepted values: `beginning`, `end`
- `end` means “follow only new lines on first start”.
- `poll_interval`
- default: `1s`
- `batch_size`
- default: `256`
- Maximum number of lines read per poll.
## Validation rules
The configuration loader rejects:
- an empty `profiles` map
- an empty `sources` list
- invalid `server.listen_address`
- duplicate source names
- duplicate source paths
- sources pointing to unknown profiles
- invalid `initial_position` values
- invalid status ranges
- overly broad suspicious prefixes such as `/`
- malformed CIDRs
- invalid known-agent decisions
- missing OPNsense credentials when `opnsense.enabled: true`
## Design note: one source, one profile
One monitored log path equals one profile selection. This makes it easy to monitor, for example:
- one public web vhost with aggressive auto-blocking
- one Gitea vhost with a more conservative review-first profile
- one internal service with no auto-blocking at all
All of them can still share the same OPNsense alias backend because OPNsense configuration is global.

229
docs/install.md Normal file
View File

@@ -0,0 +1,229 @@
# Installation
This document covers both generic Linux deployments and NixOS deployments.
## Prerequisites
You need:
- Linux.
- Caddy access logs written to files in the default JSON format.
- One readable log file per logical source you want to monitor.
- A writable directory for the SQLite database.
- Go 1.25 or newer if you build from source without Nix.
- Optional: OPNsense API credentials if you want firewall alias synchronization.
- Optional: outbound DNS and HTTPS access if you want IP investigation and bot verification.
## Deployment checklist
Before starting the daemon, decide:
- which Caddy log files you want to ingest
- which heuristic profile each log file should use
- whether the daemon should auto-block for each profile or only mark addresses for review
- whether OPNsense integration is enabled
- which address the web UI should listen on
## 1. Configure Caddy access logs
The daemon reads files, not stdout and not journald. Each monitored source should write to its own file.
An example Caddy configuration is available in [`../examples/Caddyfile`](../examples/Caddyfile).
Important points:
- Keep the default JSON log format.
- Use one file per source.
- Make sure the service account running `caddy-opnsense-blocker` can read those files.
- If Caddy sits behind another proxy, make sure Caddy records the actual client IP in `request.client_ip`.
## 2. Prepare OPNsense credentials (optional)
If you want the daemon to block and unblock IP addresses on OPNsense:
1. Create or reuse an API key and secret.
2. Decide which alias name should contain blocked IPs.
3. Keep the credentials in files readable only by the service account.
If OPNsense integration is disabled, the daemon still ingests logs, records decisions, and supports manual review.
## 3. Build the binary
Clone the repository and build the daemon:
```bash
git clone https://git.dern.ovh/infrastructure/caddy-opnsense-blocker.git
cd caddy-opnsense-blocker
CGO_ENABLED=0 go test ./...
CGO_ENABLED=0 go build -o caddy-opnsense-blocker ./cmd/caddy-opnsense-blocker
```
You can also use `nix-build` if you prefer the Nix package defined in this repository.
## 4. Create the runtime user and directories
Example for a non-NixOS host:
```bash
useradd --system --home-dir /var/lib/caddy-opnsense-blocker --create-home --shell /usr/sbin/nologin blocker
install -d -o blocker -g blocker -m 0750 /var/lib/caddy-opnsense-blocker
install -d -o root -g blocker -m 0750 /etc/caddy-opnsense-blocker
install -d -o root -g blocker -m 0750 /var/log/caddy
install -m 0755 caddy-opnsense-blocker /usr/local/bin/caddy-opnsense-blocker
```
If your Caddy logs are group-readable by a different group, either adjust permissions or grant the runtime user supplementary group access.
## 5. Create the configuration file
Start from the repository example:
```bash
cp config.example.yaml /etc/caddy-opnsense-blocker/config.yaml
chmod 0640 /etc/caddy-opnsense-blocker/config.yaml
```
Then edit:
- `server.listen_address`
- `storage.path`
- `profiles`
- `sources`
- `opnsense.*` if OPNsense integration is enabled
For a field-by-field reference, see [`configuration.md`](configuration.md).
Store OPNsense credentials in files such as:
```bash
install -m 0400 -o root -g blocker /path/to/api-key /etc/caddy-opnsense-blocker/opnsense-api-key
install -m 0400 -o root -g blocker /path/to/api-secret /etc/caddy-opnsense-blocker/opnsense-api-secret
```
Then reference those files from `config.yaml`.
## 6. Run the daemon manually once
Before installing a service unit, validate that the daemon starts correctly:
```bash
sudo -u blocker /usr/local/bin/caddy-opnsense-blocker -config /etc/caddy-opnsense-blocker/config.yaml
```
In another terminal:
```bash
curl http://127.0.0.1:9080/healthz
```
If the daemon is bound to another address, adjust the URL accordingly.
## 7. Install the systemd service
An example unit file is available in [`../examples/caddy-opnsense-blocker.service`](../examples/caddy-opnsense-blocker.service).
Typical installation steps:
```bash
cp examples/caddy-opnsense-blocker.service /etc/systemd/system/caddy-opnsense-blocker.service
systemctl daemon-reload
systemctl enable --now caddy-opnsense-blocker.service
```
After that:
```bash
systemctl status caddy-opnsense-blocker.service
journalctl -u caddy-opnsense-blocker.service -f
```
## 8. Verify end-to-end behavior
Recommended checks:
- `GET /healthz` returns `{"status":"ok",...}`
- the UI loads on the configured address
- new suspicious requests appear in the dashboard
- the IP detail page shows the full request history for one address
- manual `Block` and `Unblock` actions produce backend actions if OPNsense is enabled
## 9. Log rotation and upgrades
- The daemon stores inode and offset per source, so it can resume after restarts.
- File truncation or inode changes are detected automatically, which makes the common rename-based log rotation pattern safe.
- Upgrades are usually just “replace the binary, restart the service”. The SQLite database is kept separately in the configured state directory.
## Nix build
Build the packaged binary directly from the repository root:
```bash
nix-build
```
The result symlink contains the packaged daemon under `result/bin/caddy-opnsense-blocker`.
## NixOS module
The repository includes a reusable NixOS module in [`../module.nix`](../module.nix).
### Import from a local checkout
```nix
{
imports = [ /path/to/caddy-opnsense-blocker/module.nix ];
services.caddy-opnsense-blocker = {
enable = true;
credentials.opnsenseApiKeyFile = "/run/secrets/opnsense-api-key";
credentials.opnsenseApiSecretFile = "/run/secrets/opnsense-api-secret";
settings = {
server.listen_address = "127.0.0.1:9080";
opnsense = {
enabled = true;
base_url = "https://router.example.test";
ensure_alias = true;
alias.name = "blocked-ips";
};
profiles.public-web = {
auto_block = true;
block_unexpected_posts = true;
block_php_paths = true;
suspicious_path_prefixes = [ "/wp-admin" "/wp-login.php" "/.env" ];
};
sources = [
{
name = "public-web";
path = "/var/log/caddy/public-web-access.json";
profile = "public-web";
}
];
};
};
}
```
### Notes about the NixOS module
- The module defaults the service user and group to `caddy`.
- The generated configuration file is written from `services.caddy-opnsense-blocker.settings`.
- OPNsense credentials can be injected through systemd credentials with `credentials.opnsenseApiKeyFile` and `credentials.opnsenseApiSecretFile`.
- The default database path is `/var/lib/caddy-opnsense-blocker/caddy-opnsense-blocker.db`.
## Operating without OPNsense
Set `opnsense.enabled: false` to run in review-only mode.
In that mode the daemon still:
- ingests logs
- stores events and IP state
- runs investigations
- serves the UI and API
- records manual override decisions locally
It simply stops short of calling an external firewall API.