Notification Options
flowwler fires notifications on every escalation state transition and on every manual mitigation action via the REST API.
Configuration
Targets are configured globally (all groups) and/or per group:
notifications:
timeout: 10s # per-target send timeout (default: 10s)
retry_count: 2 # retries on failure (default: 0)
targets:
- type: teams
name: ops-channel
url: "https://outlook.office.com/webhook/..."
groups:
- name: datacenter-a
notifications:
targets:
- type: telegram
name: noc-bot
token: "123456:ABC..."
chat_id: "-1001234567890"
Per-group targets receive only events for that group. Global targets receive events for all groups. Both lists are evaluated for every event - a target configured in both lists receives the event twice (once per entry); flowwler does not deduplicate by name.
Event filter
Any target can be restricted to a subset of event types:
- type: webhook
name: activate-only
url: "https://example.com/hook"
events: [activated, escalated] # omit field to receive all events
Valid event types: activated, escalated, holddown, cleared, rules_updated.
Event schema
All backends receive the same Event struct. Human-readable backends (Telegram, Pushover, Teams, Slack) format BPS and PPS; machine-readable backends (webhook, PagerDuty, Alertmanager) carry raw integers.
Text backends (Slack, Teams, Jira, Zammad, Telegram and Pushover) render each FlowSpec rule and suggestion as key=value tokens: a bare frag token marks a non-first-fragment rule or suggestion. The webhook posts the full event JSON, so rules and suggestions carry a fragments field; PagerDuty's custom details carry them on flowspec_rules only (no suggestions), and Alertmanager carries no FlowSpec rules.
| Field | Type | Description |
|---|---|---|
type |
string | activated, escalated, holddown, cleared, or rules_updated |
attack_id |
string | UUID, stable across all events for one attack |
timestamp |
RFC3339 | Time the event was fired |
victim_ip |
string | IP address of the attack target |
group_name |
string | Group name ("manual" for REST API-triggered events) |
rule_name |
string | Rule name if a named rule triggered mitigation; omitted for group-level |
level |
int | Current escalation level (0 on cleared) |
prev_level |
int | Previous level (populated on escalated) |
mitigation_types |
array | Active mitigation types for this event: array of blackhole, subnet-blackhole, flowspec — a level may enable several at once and all of them are listed. Empty array for alert-only levels. |
flowspec_action |
string | FlowSpec action: discard, rate-limit, or redirect. Present on all event types when flowspec is among the active mitigation types; omitted otherwise. |
notification_only |
bool | true on activated, escalated and cleared events when the level has no mitigation configured (alert-only). Omitted when false, and never set on holddown or rules_updated events. |
attack_started_at |
RFC3339 | When the attack was first detected (Idle→Active transition); always present, unlike timestamp this does not change as the attack escalates through levels |
bps |
uint64 | Attack traffic in bits per second |
pps |
uint64 | Attack traffic in packets per second |
peak_bps |
uint64 | Peak bits per second observed so far this attack (omitted if zero) |
peak_pps |
uint64 | Peak packets per second observed so far this attack (omitted if zero) |
threshold_bps |
uint64 | Configured BPS threshold that triggered the level (omitted if not set) |
threshold_pps |
uint64 | Configured PPS threshold that triggered the level (omitted if not set) |
discard_bps |
uint64 | Traffic already being dropped by routers (bits per second) |
discard_pps |
uint64 | Traffic already being dropped by routers (packets per second). Populated on activated/escalated only, same as discard_bps. |
flowspec_rules |
array | FlowSpec rules announced for this event (omitted for non-FlowSpec mitigations; see below) |
suggested_flowspec |
object | FlowSpec rule suggestions derived from flow data (omitted when no traffic pattern is dominant; present on activated/escalated events regardless of the active mitigation type, so it can appear together with flowspec_rules; see below) |
countries |
array | Attack source countries by current BPS, top 10 (omitted when GeoIP isn't configured or resolved no sources; see below) |
FlowSpec rule fields (each entry in flowspec_rules)
| Field | Type | Description |
|---|---|---|
dst_ip |
string | Destination IP matched by this rule (always the victim IP) |
src_ip |
string | Source IP matched by this rule (omitted when no source filter) |
protocol |
uint8 | IP protocol number (0/omitted = any; 6=TCP, 17=UDP, 1=ICMP) |
dst_ports |
[]uint16 | Destination ports (empty/omitted = any). A single-element list may carry an explicit 0 (e.g. an hping3 --destport 0 flood) — a real match criterion, not "any". More than one element is an OR'd match list. |
src_ports |
[]uint16 | Source ports (empty/omitted = any) |
icmp_types |
[]uint16 | ICMP types matched by this rule (empty/omitted = any): the types observed for the source, or the triggering rule's icmp_types list when consolidation relaxed the dimension. Only meaningful for protocol 1 (ICMP) or 58 (ICMPv6). |
icmp_codes |
[]uint16 | ICMP codes matched by this rule (empty/omitted = any), same behavior as icmp_types |
tcp_flags_bitmask |
uint8 | TCP flags bitmask (0/omitted = no constraint; FIN=0x01 SYN=0x02 RST=0x04 PSH=0x08 ACK=0x10 URG=0x20 ECE=0x40 CWR=0x80). Always the triggering rule's tcp_flags mask; flags observed in traffic are never used. Only meaningful for protocol 6 (TCP). |
fragments |
bool | true when the rule matches only non-first fragments (FlowSpec type-12 is-fragment bit, not first fragments or unfragmented packets), copied from the triggering rule's fragments. Omitted when false. |
min_pkt_len |
uint16 | Minimum packet length in bytes (0/omitted = no match) |
max_pkt_len |
uint16 | Maximum packet length in bytes (0/omitted = no match) |
pkt_len_approximate |
bool | Always false/omitted for a live FlowSpec rule notification. min_pkt_len/max_pkt_len, when present, is always real exporter data or an explicit pkt_len match config; a purely derived value is never attached to an announced route. |
action |
string | discard, rate-limit, or redirect |
next_hop |
string | Scrubbing-center next-hop IP. Only for redirect; omitted otherwise |
vrf |
string | VRF route target (ASN:LocalAdmin). Only for redirect; omitted otherwise |
communities |
[]string | BGP communities attached to the route. Omitted when the route has none |
rate_limit_bps |
uint64 | Rate limit in bits/sec. Only for rate-limit; omitted otherwise |
These four optional keys appear in the JSON that is sent as-is: the webhook target (the whole event) and PagerDuty custom_details.flowspec_rules. Text backends do not render them, and Alertmanager carries no FlowSpec rules. The same event is stored in the attack history (GET /api/v1/attacks/{attack_id}); events recorded before flowwler#42 lack the four keys.
icmp_types/icmp_codes/tcp_flags_bitmask/fragments mirror the same-named fields GET /api/v1/flowspec/rules already surfaces on the REST rule object (see rest-api.md) — the notification payload now carries the same full match criteria, so there's no longer a need to cross-check the REST API for these.
Multiple attack sources
When FlowSpec is among the active mitigation types, the escalation engine selects the top-N attack sources by volume (up to max_rules, default 10) and announces one BGP FlowSpec NLRI per source in a single activation call. One notification is fired per state transition (not one per source). That single event contains all rules in flowspec_rules, so every backend will display the full set of matching sources and their match criteria in one message.
flowspec_rules can be absent on an event whose mitigation_types includes flowspec: when no matching source flows were visible and the rule could not be expressed as one fallback route (a match list longer than 5 entries), the mitigation is active with no routes. The routes appear in a later rules_updated event once the 5-second refresh finds matching sources. See Configuration.
Countries (countries)
countries is populated on activated and escalated events, and on holddown→active reactivation — the three transitions that carry a fresh flow snapshot at the moment they fire. It is not populated on holddown-entry or cleared events, since sources may have already stopped flowing by then; it is also omitted for carpet-bomb subnet states, which have no single victim IP to attribute sources to.
countries requires geoip.enabled: true and at least one attack source to resolve to a country; otherwise it's omitted entirely and the notification looks exactly as it does with GeoIP disabled. Each entry is one country's current contribution, sorted by BPS descending and capped to the top 10 — a notification-readability cap only; the full, uncapped per-attack breakdown is always available via GET /api/v1/attacks/{id} regardless of this cap (see rest-api.md).
| Field | Type | Description |
|---|---|---|
country |
string | ISO 3166-1 alpha-2 code, e.g. "US"; empty string ("") when GeoIP resolved a source but not its country — human-readable backends render this as "unknown" in the formatted message text, but the raw JSON field itself is never rewritten |
bps |
uint64 | This country's current contribution in bits per second |
pps |
uint64 | This country's current contribution in packets per second |
source_count |
int | Number of distinct source IPs from this country |
FlowSpec suggestions (suggested_flowspec)
flowwler automatically analyses the live flow distribution and attaches a suggested_flowspec object whenever a dominant traffic pattern is found — computed independently of which mitigation types are actually active. This allows NOC engineers to see what FlowSpec rules would be effective even when none of the active mitigation types use FlowSpec. It is not mutually exclusive with flowspec_rules: when FlowSpec is itself the active mitigation, both fields can be present in the same event, since the suggestion is not gated on the active mitigation type.
The suggestion uses the same selection and consolidation logic as a live FlowSpec mitigation. For a rule-level event it is bounded by the triggering rule's own criteria: only sources the rule counts are candidates, and every suggestion carries the rule's protocol (including an implied one), port and ICMP lists, tcp_flags_bitmask, fragments and pkt_len. coverage_pct is the share of the traffic that rule counts. For group-level events (and GET /api/v1/flowspec/suggest/{ip}) there is no rule, so suggestions consider all inbound sources of the victim and carry none of the rule fields. The ranking metric (BPS or PPS) is chosen based on which threshold was proportionally more exceeded - a PPS-only threshold always ranks by PPS; a BPS-only threshold always ranks by BPS; when both are configured, the metric with the greater ratio of observed/threshold wins.
suggested_flowspec is omitted when:
- No source-level flow records exist for the victim IP
- The dominant pattern covers less than 20% of total attack traffic (too distributed to suggest)
suggested_flowspec object fields:
| Field | Type | Description |
|---|---|---|
ranked_by |
string | "bps" or "pps" - the metric used to rank and select rules |
unique_source_count |
int | Total number of distinct source IPs observed before any consolidation |
spoofed_sources |
bool | true when the source distribution looks randomised (≥ 50 unique sources each contributing < 5 Mbps on average). Indicates per-source FlowSpec rules will be ineffective; destination-only or pattern-only rules are preferable. |
rules |
array | Suggested rules, ranked by the metric in ranked_by |
Each entry in rules:
| Field | Type | Description |
|---|---|---|
src_ip |
string | Source IP matched by this rule (omitted once the rule set is consolidated to a source-agnostic pattern) |
protocol |
uint8 | IP protocol number (0 = any; 6=TCP, 17=UDP, 1=ICMP) |
dst_ports |
[]uint16 | Destination ports (empty/omitted = any; a single explicit 0 is a real match, not "any") |
src_ports |
[]uint16 | Source ports (empty/omitted = any) |
icmp_types |
[]uint16 | ICMP types for this rule: the observed types, bounded by the triggering rule's icmp_types (empty/omitted = any). Only meaningful for protocol 1 (ICMP) or 58 (ICMPv6). |
icmp_codes |
[]uint16 | ICMP codes observed for this rule (empty/omitted = any) |
tcp_flags_bitmask |
uint8 | The triggering rule's tcp_flags mask, set only on suggestions with protocol 6 (TCP); 0/omitted otherwise, and always omitted for group-level events (no rule). Flags observed in traffic are never used. Same bit layout as flowspec_rules[].tcp_flags_bitmask above. |
fragments |
bool | true when the suggestion matches only non-first fragments, from the triggering rule's fragments. Omitted when false, and always omitted for group-level events. |
min_pkt_len |
uint16 | Minimum packet length in bytes (0 = no constraint): the triggering rule's pkt_len, otherwise native exporter data. Never a value derived from bytes/packets. |
max_pkt_len |
uint16 | Maximum packet length in bytes (0 = no constraint), same sources as min_pkt_len |
coverage_pct |
uint8 | Percentage of the inbound BPS (or PPS) this rule would cover. For a rule-level event the base is the traffic the triggering rule counts; for group-level events it is all inbound traffic to the victim. |
fragments is a purely operator-configured match.fragments criterion (see MatchConfig); a suggestion for a fragments: true rule carries it, and so does a live route in flowspec_rules[].fragments. Suggestions never carry a packet length derived from bytes/packets, and pkt_len_approximate no longer appears on them.
Backends
alertmanager
Sends firing/resolved alerts to a Prometheus Alertmanager instance.
- type: alertmanager
name: ops-am
url: "http://alertmanager:9093"
critical_level: 2 # optional; default 2 (also used when set to 0 or below)
Posts an alert for every event type it receives (there is no built-in event-type filtering in the backend itself) — by default that's activated, escalated, holddown, cleared, and rules_updated, since a target with no events: filter receives all event types. Only cleared resolves the alert (sets endsAt); every other event type — including rules_updated, which re-posts the still-firing alert with updated details — leaves it firing. Restrict this with the target's events: filter (see TargetConfig) if you only want it to fire on a subset. The attack_id is included as a label. Severity (warning/critical) is controlled by critical_level: levels at or above it are critical. It defaults to 2.
jira
Opens a Jira issue when an attack is detected and resolves it when the attack clears. Escalation and hold-down events are appended as comments on the existing issue.
- type: jira
name: noc-jira
url: "https://mycompany.atlassian.net"
email: "noc@example.com" # Jira user email (for Basic auth with API token)
api_token: "xxxxxxxxxxxxxxxxxxxx" # Atlassian API token (not your account password)
project_key: "NOC" # Jira project key
issue_type: "Incident" # optional; default: "Task"
resolve_transition: "Done" # optional; transition name to close the issue (default: "Done")
issue_priority: "Normal" # optional; default: "Normal". Escalation events bump it to "High".
tags: [network, ddos-response] # optional; replaces the default labels (flowwler, ddos) on every issue
Setup:
- Log in to id.atlassian.com and generate an API token.
- Create or identify the target Jira project and note its key (e.g.
NOC). - Check the available transition names on your issue workflow - the value of
resolve_transitionmust match exactly (case-insensitive). Common names:Done,Resolved,Close Issue.
Behaviour by event type:
| Event | Action |
|---|---|
activated (first time) |
Creates a new issue with a description table (victim, group, level, traffic, FlowSpec rules and countries if applicable). Priority is set to Normal (or the value of issue_priority). Labels default to flowwler and ddos; configuring tags replaces those defaults. |
activated (re-activation) |
If traffic recovered during hold-down (same attack_id), appends a comment instead of opening a duplicate issue: "Traffic recovered during hold-down — mitigation remains active — 1.20 Gbps / 75.00 Kpps), followed by a Mitigation: line. |
escalated |
Adds a comment: level, new traffic rate, mitigation types. Updates the issue priority to High. |
holddown |
Adds a comment noting the attack has subsided. |
rules_updated |
Ignored. The Jira backend takes no action on this event. |
cleared |
Adds a final comment ("Attack cleared — mitigation withdrawn." plus the last observed traffic as "Final: resolve_transition. |
Issue state is tracked in memory (keyed by attack_id) and survives a config reload as long as this target's configuration is unchanged. If the target's own configuration changes in the reload, tracking for attacks already in progress is lost and their later events are skipped for the Jira backend only.
pagerduty
Sends events to the PagerDuty Events v2 API. Automatically resolves on cleared.
- type: pagerduty
name: ops-pd
integration_key: "abc123def456..."
critical_level: 2 # levels >= this value use "critical" severity; below uses "warning"
Setup: Create a service in PagerDuty with an Events API v2 integration and copy the integration key.
The attack_id is used as the dedup_key, so all events for the same attack are grouped into one PagerDuty incident. A cleared event resolves the incident automatically. For FlowSpec mitigations the custom_details object includes a flowspec_rules array with one structured entry per announced NLRI (same fields as the event schema above). When GeoIP resolved at least one source, custom_details also includes a countries array (same fields as the event schema above).
pushover
Sends a push notification via the Pushover API. Message priority is derived from the escalation level.
- type: pushover
name: ops-po
user_key: "uXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
app_token: "aXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
priority: # optional; omit to use defaults below
0: -1 # cleared/holddown: quiet
1: 0 # level 1: normal
2: 1 # level 2: high (bypasses quiet hours)
3: 2 # level 3: emergency (repeats every 60s until acknowledged)
The priority map is an exact lookup by level. A level with no key uses the default mapping below, so level 4 and above get the default (emergency) unless you add a key for them. Key 0 covers cleared and holddown events.
Default priority mapping when priority is not set:
| Event / Level | Priority | Behaviour |
|---|---|---|
cleared / holddown |
-1 |
Quiet |
| Level 1 | 0 |
Normal |
| Level 2 | 1 |
High |
| Level 3+ | 2 |
Emergency |
Priority range: -2 (silent) to 2 (emergency). Emergency priority requires acknowledgement in the Pushover app and retries every 60 seconds for up to 1 hour.
Message body includes Rule: <name> after the group name when a named rule triggered the mitigation. For group-level mitigations the rule line is omitted. A Threshold: BPS ≥ … / PPS ≥ … line appears after the level number when a BPS or PPS threshold is configured for that escalation level. For FlowSpec mitigations a FlowSpec Rules (N): block is appended before the Attack ID, listing each announced source rule on its own line. A Countries: … line appears when GeoIP resolved at least one source (see Countries).
slack
Sends a message to a Slack channel via an Incoming Webhook. Uses Block Kit with a colored left-border attachment: red for activated/escalated, orange for holddown, green for cleared.
- type: slack
name: ops-slack
url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
Setup:
- Go to api.slack.com/apps and create a new app (or use an existing one).
- Under Features → Incoming Webhooks, toggle Activate Incoming Webhooks on.
- Click Add New Webhook to Workspace, pick the target channel, and copy the webhook URL.
- Paste the URL as
urlin the target config.
Message layout (Block Kit attachment with colored sidebar):
DDoS Attack Detected: 203.0.113.10
─────────────────────────────────────────
Victim 203.0.113.10
Group datacenter-a
Rule udp-amplification ← only shown when a named rule triggered the mitigation
Level 1
Threshold BPS ≥ 1.20 Gbps ← omitted if no threshold configured for this level
Mitigation flowspec (discard)
BPS 1.20 Gbps
PPS 75.00 Kpps
FlowSpec Rules (2): ← only shown for FlowSpec mitigations
#1: `dst=203.0.113.10 src=1.2.3.4 proto=UDP dport=53 → discard`
#2: `dst=203.0.113.10 src=5.6.7.8 proto=UDP dport=53 → discard`
Countries: ← only shown when GeoIP resolved at least one source
US 1.20 Gbps (30 src), DE 400.00 Mbps (12 src)
─────────────────────────────────────────
Attack ID: `3f2b8c1e-5d4a-4e7b-9a60-1c2d3e4f5a6b`
teams
Sends a MessageCard to an MS Teams channel via an Incoming Webhook. The card header color reflects the event type: red for activated/escalated, orange for holddown, green for cleared.
- type: teams
name: ops-teams
url: "https://outlook.office.com/webhook/XXXXXXXX/IncomingWebhook/YYYYYYYY/ZZZZZZZZ"
Setup:
- In Teams, open the channel you want to post to.
- Click the three-dot menu next to the channel name → Connectors.
- Find Incoming Webhook, click Configure, give it a name, and copy the URL.
- Paste the URL as
urlin the target config.
Note: Microsoft is deprecating Office 365 Connectors in favour of Power Automate Workflows. If your tenant has disabled legacy connectors, create a Workflow using the "Post to a channel when a webhook request is received" template and use the Workflow's webhook URL instead - the MessageCard payload is compatible.
Card fields:
| Field | Value |
|---|---|
| Rule | rule name — shown first, before Victim (omitted if group-level) |
| Victim | victim IP |
| Group | group name |
| Level | escalation level |
| Threshold | configured threshold, e.g. BPS ≥ 1.20 Gbps (omitted if not configured) |
| Mitigation | active mitigation types, comma-separated; flowspec carries its action in parentheses, e.g. flowspec (discard); alert only (no mitigation) for alert-only levels |
| BPS | human-readable (e.g. 1.20 Gbps) |
| PPS | human-readable (e.g. 75.00 Kpps) |
| Attack ID | attack UUID |
| FS Rule #1, #2, … / Sources | one fact per FlowSpec rule, e.g. dst=203.0.113.10 src=1.2.3.4 proto=UDP dport=53 → discard (omitted for non-FlowSpec mitigations; a Sources fact with the unique-source count appears instead when only a suggested FlowSpec rule set exists) |
| Countries | US 1.20 Gbps (30 src), DE 400.00 Mbps (12 src) — shown last (omitted when GeoIP resolved no sources) |
telegram
Sends an HTML-formatted message via the Telegram Bot API.
- type: telegram
name: noc-bot
token: "123456:ABCdefGHIjklMNOpqrSTUvwxYZ"
chat_id: "-1001234567890"
Setup:
- Create a bot via @BotFather and copy the token.
- Add the bot to your channel or group and obtain the
chat_id(usegetUpdatesor a helper bot).
Message format:
DDoS Attack Detected
Victim: 203.0.113.10
Group: datacenter-a
Rule: udp-amplification ← only shown when a named rule triggered the mitigation
Level: 1
Threshold: BPS ≥ 1.20 Gbps ← omitted if no threshold is configured for this level
Mitigation: flowspec (discard)
BPS: 1.20 Gbps
PPS: 75.00 Kpps
FlowSpec Rules (2): ← only shown for FlowSpec mitigations
#1: dst=203.0.113.10 src=1.2.3.4 proto=UDP dport=53 → discard
#2: dst=203.0.113.10 src=5.6.7.8 proto=UDP dport=53 → discard
Countries: US 1.20 Gbps (30 src), DE 400.00 Mbps (12 src) ← only shown when GeoIP resolved at least one source
Attack ID: 3f2b8c1e-5d4a-4e7b-9a60-1c2d3e4f5a6b
zammad
Opens a Zammad ticket when an attack is detected and closes it when the attack clears. Escalation and hold-down events are appended as internal notes on the existing ticket.
- type: zammad
name: noc-zammad
url: "https://zammad.example.com"
api_token: "xxxxxxxxxxxxxxxxxxxx" # Zammad API token
group: "Users" # Zammad ticket group
customer: "noc@example.com" # customer email for new tickets
close_state: "closed" # optional; default: "closed"
tags: [flowwler, ddos-response] # optional; tags set on the ticket at creation
Setup:
- In Zammad, go to Profile → Token Access and create a token with the
ticket.agentrole. - Identify the target group name (visible under Manage → Groups).
- Supply a customer email - this must be an existing Zammad user or an email that Zammad can auto-create.
- Check the available state names under Manage → Ticket States - the
close_statevalue must match exactly (case-sensitive). Common names:closed,resolved.
Behaviour by event type:
| Event | Action |
|---|---|
activated (first time) |
Creates a new ticket with a plain-text description (victim, group, level, traffic, FlowSpec rules and countries if applicable). Priority is set to 2 normal. Tags default to flowwler and ddos; configuring tags replaces those defaults. |
activated (re-activation) |
If traffic recovered during hold-down (same attack_id), appends an internal note instead of opening a duplicate ticket: "Traffic recovered during hold-down — mitigation remains active — 1.20 Gbps / 75.00 Kpps), followed by a Mitigation: line. |
escalated |
Appends an internal note: new level, traffic rate, mitigation types. Updates the ticket priority to 3 high. |
holddown |
Appends an internal note that the attack has subsided. |
rules_updated |
Ignored. The Zammad backend takes no action on this event. |
cleared |
Appends a final note ("Attack cleared — mitigation withdrawn." plus the last observed traffic as "Final: close_state. |
Authentication uses Authorization: Token token=<api_token> (Zammad's token scheme - not HTTP Basic). Ticket state is tracked in memory (keyed by attack_id) and survives a config reload as long as this target's configuration is unchanged. If the target's own configuration changes in the reload, tracking for attacks already in progress is lost and their later events are skipped for the Zammad backend only.
webhook
HTTP POST with a JSON body matching the event schema above.
- type: webhook
name: my-hook
url: "https://example.com/flowwler-hook"
headers:
Authorization: "Bearer secret"
events: [activated, cleared]
The headers map is merged into every request. BPS and PPS are raw integers in the JSON body. For FlowSpec mitigations the body also includes a flowspec_rules array containing one object per announced NLRI (see schema above); when GeoIP resolved at least one source, the body also includes a countries array (see Countries).
Testing a target
Use the REST API to send a synthetic test notification to any configured target without waiting for a real attack:
curl -s -X POST http://[::]:9731/api/v1/notifications/targets/my-target/test | jq .
The request bypasses event-type filters and sends a canned activated event (level 1, mitigation_types: ["blackhole"]) once, with no retries. At most one test is accepted per target every 10 seconds; a request inside that window returns 429 Too Many Requests. An optional JSON body overrides victim_ip (default 192.0.2.1) and group_name (default test). The target's health state is updated on success or failure and is visible in GET /api/v1/notifications/targets. See REST API - POST /api/v1/notifications/targets/{name}/test for full details.
Monitoring delivery health
Alongside GET /api/v1/notifications/targets, target health is exposed as Prometheus metrics — useful for alerting when the alerting pipeline itself breaks:
# Send failure rate per target
curl -s http://localhost:9731/metrics | grep flowwler_notify_send_total
# Seconds since a target last succeeded
curl -s http://localhost:9731/metrics | grep flowwler_notify_last_success_timestamp_seconds
The internal "always" targets (attack-history store, and PCAP recording when pcap.enabled: true) receive every event regardless of filters and are not part of the regular target list, so their failures aren't visible via GET /api/v1/notifications/targets — track them via flowwler_notify_always_target_send_failures_total{target} (target is "store" or "pcap") instead. A non-zero rate for target="store" means attack-lifecycle events are being silently lost from history.
See Metrics for full metric descriptions and recommended alerts.
Delivery guarantees
Targets are called concurrently with a configurable timeout (default 10s), so a slow target never delays another. For a given target, events of the same attack are delivered strictly in order (for example escalated is never sent before activated has finished), while different attacks are delivered in parallel. Failed sends are retried up to retry_count times (default 0), but only for transient failures (network errors, timeouts and HTTP 5xx responses; a 4xx response is never retried). Each attempt gets its own full timeout, and retries back off exponentially (0.5s, 1s, 2s, ... capped at 5s). Retried Jira/Zammad resolve/close steps do not repeat the closing comment/note if it was already posted. Events that exhaust all retries are dropped and logged at warn level. The ordering queue is in memory only - if the process restarts during an attack, in-progress events are lost.