infrastructure/caddy-opnsense-blocker
2
Fork 0
Code Issues Activity

Expand public installation and API documentation

Browse Source
This commit is contained in:
Codex, agent ChatGPT
2026-03-12 14:35:39 +01:00
parent 34d6d3ddcb
commit f15839cf51
7 changed files with 907 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

220
README.md
View File

@@ -1,76 +1,119 @@
# caddy-opnsense-blocker
`caddy-opnsense-blocker` is a local-first daemon that ingests Caddy access logs in their default JSON format, evaluates suspicious requests, keeps persistent local state in SQLite, provides a lightweight web UI for review, and blocks or unblocks IP addresses through an OPNsense alias.
`caddy-opnsense-blocker` is a local-first daemon that follows one or more Caddy access log files in the default JSON format, applies per-source heuristics, stores events and investigations in SQLite, exposes a lightweight review UI, and optionally synchronizes manual or automatic block decisions to an OPNsense alias.
## Features
## Highlights
- Real-time ingestion of multiple Caddy JSON log files.
- One heuristic profile per log source.
- Persistent local state in SQLite.
- Local-only web UI with summary cards, a sortable “Recent IPs” view for the last 24 hours, bot badges, and a full request history for each selected address.
- On-demand IP investigation with persistent caching for bot verification, reverse DNS, RDAP, and Spamhaus lookups.
- Background IP investigation workers so missing cached intelligence appears without blocking page loads.
- Manual block, unblock, and clear-override actions with OPNsense-aware UI state.
- OPNsense alias backend with automatic alias creation.
- Concurrent polling across multiple log files.
- Real-time ingestion of multiple Caddy JSON access log files.
- One heuristic profile per log source, so different applications can have different rules while sharing the same OPNsense destination alias.
- Persistent SQLite state for events, IP states, investigations, decisions, backend actions, and source offsets.
- Lightweight web UI with overview cards, a sortable “Recent IPs” table, IP detail pages, decision history, and full request history per address.
- Background investigation workers that fill in missing cached intelligence without slowing down page loads.
- Manual `Block`, `Unblock`, `Clear override`, and `Refresh investigation` actions from the UI or the HTTP API.
- Optional OPNsense integration; the daemon also works in review-only mode.
- Pure-Go build and first-class Nix and NixOS packaging.
## Current scope
## Documentation
This first version is intentionally strong on ingestion, persistence, UI, and OPNsense integration.
The decision engine is deliberately simple and deterministic for now:
- Installation guide: [`docs/install.md`](docs/install.md)
- Configuration reference: [`docs/configuration.md`](docs/configuration.md)
- HTTP API reference: [`docs/api.md`](docs/api.md)
- Example Caddy configuration: [`examples/Caddyfile`](examples/Caddyfile)
- Example systemd unit: [`examples/caddy-opnsense-blocker.service`](examples/caddy-opnsense-blocker.service)
- suspicious path prefixes
- unexpected `POST` requests
- `.php` path detection
- explicit known-agent allow/deny rules
- excluded CIDR ranges
- manual overrides
## Requirements
This keeps the application usable immediately while leaving room for a more advanced policy engine later.
- Linux.
- Caddy access logs written to files in the default JSON format.
- Read access to every configured log file.
- A writable state directory for the SQLite database.
- Outbound DNS and HTTPS access if IP investigation is enabled.
- OPNsense only if you want the daemon to push block or unblock actions to a firewall alias.
## Architecture
## Security model
- `internal/caddylog`: parses default Caddy JSON access logs
- `internal/engine`: evaluates requests against a profile
- `internal/investigation`: performs bot verification and IP enrichment
- `internal/store`: persists events, IP state, manual decisions, backend actions, and source offsets
- `internal/opnsense`: manages the target OPNsense alias through its API
- `internal/service`: runs concurrent log followers and applies automatic decisions
- `internal/web`: serves the local review UI and JSON API
- The built-in web UI and HTTP API do not provide authentication or TLS.
- The default listen address is `127.0.0.1:9080`; keep it on loopback unless another trusted layer protects access.
- OPNsense credentials should be supplied through files, not inline secrets committed to source control.
- Raw Caddy JSON log entries are stored in SQLite for inspection and auditing; plan storage and retention accordingly.
## License
## How it works
This project is licensed under the MIT License. See `LICENSE`.
1. A dedicated follower polls each configured log file and keeps a persistent inode plus offset checkpoint.
2. New Caddy JSON lines are parsed into normalized events.
3. Each source is evaluated against exactly one heuristic profile.
4. Events, decisions, and IP state are written to SQLite.
5. Manual overrides can force an IP to `blocked` or `allowed` regardless of later events.
6. If OPNsense is enabled, block and unblock decisions can be applied to one target alias.
7. Missing IP intelligence is fetched in the background and cached for later UI and API reads.
## State model
- `observed`: traffic was recorded, but the current rule set did not produce a suspicious decision.
- `review`: the rule set matched suspicious behavior, but the configured profile did not auto-block.
- `blocked`: the IP is currently blocked, either automatically or through a manual override.
- `allowed`: the IP is explicitly allowed, typically because of a manual override or an allow rule.
`Clear override` removes only the local manual override. It does not directly add or remove the IP on OPNsense.
## Investigation model
- IP investigations are cached in SQLite.
- The background worker only fills in missing investigations; it does not continuously re-check cached intelligence.
- Opening an IP details page reuses the cached investigation.
- `Refresh investigation` is the explicit action that forces a new lookup.
- Verified bot detection currently uses built-in provider logic for Google, Bing, Apple, Meta, and DuckDuckGo.
- When an address is not identified as a verified bot, the daemon can collect reverse DNS, forward-confirmed reverse DNS, RDAP registration details, and Spamhaus DNSBL status.
## Caddy log requirements
The daemon expects Caddy access log entries in the default JSON structure. In practice, these fields must remain present:
- `ts`
- `status`
- `request.remote_ip`
- `request.client_ip` when available
- `request.host`
- `request.method`
- `request.uri`
- `request.headers.User-Agent`
The parser prefers `request.client_ip` and falls back to `request.remote_ip`. If Caddy itself sits behind another proxy or load balancer, configure Caddy so that `request.client_ip` reflects the real client address before feeding those logs to the blocker.
Use one log file per logical source. Different sources can share the same OPNsense alias while using different heuristic profiles.
## Quick start
1. Generate or provision OPNsense API credentials.
2. Copy `config.example.yaml` to `config.yaml` and adapt it.
3. Start the daemon:
For a local test run:
```bash
cp config.example.yaml config.yaml
CGO_ENABLED=0 go run ./cmd/caddy-opnsense-blocker -config ./config.yaml
```
4. Open the local UI on the configured address, for example `http://127.0.0.1:9080`.
Then open the configured address, for example `http://127.0.0.1:9080`.
## Example configuration
For production deployment instructions, see [`docs/install.md`](docs/install.md).
See `config.example.yaml`.
## Nix and NixOS
Important points:
The repository ships with:
- Each source points to one Caddy log file.
- Each source references exactly one profile.
- `initial_position: end` means “start following new lines only” on first boot.
- The `investigation` section controls how long IP enrichment is cached and whether on-demand Spamhaus lookups are enabled.
- The investigation worker fills missing cached intelligence in the background so the dashboard stays fast while bot badges and cached intel keep filling in. Opening an IP details page reuses the cache; the `Refresh investigation` button is the manual override when you explicitly want a new lookup.
- The web UI should stay bound to a local address such as `127.0.0.1:9080`.
- `package.nix`: reusable package definition
- `default.nix`: convenience entry point for `nix-build`
- `module.nix`: reusable NixOS module
## Web UI and API
Build the package directly from the repository root:
The web UI is intentionally small and server-rendered.
It refreshes through lightweight JSON polling and exposes these endpoints:
```bash
nix-build
```
Detailed NixOS installation examples are in [`docs/install.md`](docs/install.md).
## HTTP API
The UI is backed by a small JSON API. The main endpoints are:
- `GET /healthz`
- `GET /api/overview`
@@ -83,7 +126,20 @@ It refreshes through lightweight JSON polling and exposes these endpoints:
- `POST /api/ips/{ip}/unblock`
- `POST /api/ips/{ip}/clear-override`
The legacy `POST /api/ips/{ip}/reset` endpoint is still accepted as a backwards-compatible alias for `clear-override`.
The legacy `POST /api/ips/{ip}/reset` route is still accepted as a backwards-compatible alias for `clear-override`.
The full API reference, including payloads and response models, lives in [`docs/api.md`](docs/api.md).
## Configuration
See [`config.example.yaml`](config.example.yaml) for a ready-to-edit example and [`docs/configuration.md`](docs/configuration.md) for a field-by-field reference.
Key ideas:
- each `source` points to one log file
- each `source` references one `profile`
- multiple sources can share the same global OPNsense backend configuration
- `initial_position: end` means “start following new lines only” on first boot
## Development
@@ -99,64 +155,22 @@ Build the daemon:
CGO_ENABLED=0 go build ./cmd/caddy-opnsense-blocker
```
`CGO_ENABLED=0` is useful on systems without a C toolchain. The application itself only relies on pure-Go dependencies.
`CGO_ENABLED=0` is useful on systems without a C toolchain. The application depends only on pure-Go packages.
## Nix packaging
## Scope and roadmap
The repository ships with first-class Nix files:
This first public version is intentionally strong on ingestion, persistence, investigation, UI, and OPNsense integration.
The current decision engine is deliberately simple and deterministic:
- `package.nix`: reusable package definition
- `default.nix`: convenience entry point for `nix-build`
- `module.nix`: reusable NixOS module
- suspicious path prefixes
- unexpected `POST` requests
- `.php` path detection
- explicit known-agent allow and deny rules
- excluded CIDR ranges
- manual overrides
Build the package directly from the repository root:
Planned improvements include richer decision strategies, more investigation providers, additional blocking backends, and alternative ingestion transports beyond file polling.
```bash
nix-build
```
## License
Use the NixOS module from another configuration:
```nix
{
imports = [ /path/to/caddy-opnsense-blocker/module.nix ];
services.caddy-opnsense-blocker = {
enable = true;
credentials.opnsenseApiKeyFile = "/run/secrets/opnsense-api-key";
credentials.opnsenseApiSecretFile = "/run/secrets/opnsense-api-secret";
settings = {
opnsense = {
enabled = true;
base_url = "https://router.example.test";
ensure_alias = true;
alias.name = "blocked-ips";
};
profiles.public-web = {
auto_block = true;
block_unexpected_posts = true;
block_php_paths = true;
suspicious_path_prefixes = [ "/wp-admin" "/wp-login.php" "/.env" ];
};
sources = [
{
name = "public-web";
path = "/var/log/caddy/public-web.json";
profile = "public-web";
}
];
};
};
}
```
## Roadmap
- richer decision engine
- optional GeoIP and ASN providers beyond RDAP
- richer review filters in the UI
- alternative blocking backends besides OPNsense
- direct streaming ingestion targets in addition to file polling
This project is licensed under the MIT License. See [`LICENSE`](LICENSE).