Skip to content

Troubleshooting Guide

Log interpretation

All logs are structured JSON, one object per line:

journalctl -u flowwler -f

Escalation events

{"level":"info","msg":"escalation: Idle→Active",
 "victim":"203.0.113.10","group":"datacenter-a","level":1,
 "mitigation":"flowspec","bps":5200000000,"pps":620000,"discard_bps":0}

An attack has been detected. Level 1 mitigation (FlowSpec) has been activated. BPS is ~5.2 Gbps; the router is not yet discarding anything (discard_bps:0).

{"level":"info","msg":"escalation: level up",
 "victim":"203.0.113.10","group":"datacenter-a",
 "prev_level":1,"level":2,"mitigation":"subnet-blackhole",
 "bps":12000000000,"pps":1400000,"discard_bps":0}

Attack exceeded the level 2 threshold after escalate_after elapsed. FlowSpec was withdrawn and a subnet-blackhole was announced.

{"level":"info","msg":"escalation: Active→HoldDown",
 "victim":"203.0.113.10","group":"datacenter-a",
 "bps":180000,"pps":210,"discard_bps":11980000000}

Traffic dropped below the level 2 threshold. Mitigation moved to HoldDown - it will remain active for hold_time (default 15 minutes) before being withdrawn. discard_bps near the original bps means the router is absorbing almost all remaining traffic via the BGP route.

{"level":"info","msg":"escalation: HoldDown→Idle (hold-down expired)",
 "victim":"203.0.113.10","group":"datacenter-a","hold_time":"15m0s"}

Hold-down timer expired. The mitigation is withdrawn and the state machine returns to Idle.

BGP events

{"level":"info","msg":"bgp: blackhole announced",
 "prefix":"203.0.113.10/32","next_hop":"192.0.2.1","community":"65535:666",
 "victim":"203.0.113.10","group":"datacenter-a"}

{"level":"info","msg":"bgp: startup cleanup complete","withdrawn":3,"families":4}

Stale routes from a previous crash were removed on startup.

Decode warnings

{"level":"warn","msg":"flow decode error",
 "router":"10.0.0.1","protocol":"netflow",
 "error":"netflow: unsupported version 5 (want 9 or 10)"}

The router is sending NetFlow v5 on the v9 listener port. Configure the router to export v9 or IPFIX, or switch the router to the netflow5 listener if you want v5 support.

{"level":"debug","msg":"flow data set skipped: template not cached yet",
 "router":"10.0.0.1","protocol":"netflow","domain_id":100,"template_id":256}

Normal at startup. Templates arrive every template-timeout seconds (typically 30–60 s). This message disappears once the first template set is received.

IRR / NetBox prefix resolution

{"level":"info","msg":"irr: resolving prefixes","group":"customers","as_set":"AS64500:AS-CUSTOMERS"}
{"level":"info","msg":"irr: resolved prefixes","group":"customers","asns":47,"routes_v4":312,"routes_v6":89,"duration":"4.2s"}

Normal resolution log. If routes_v4 and routes_v6 are both 0, the AS-SET may be empty or the WHOIS server may be unreachable.

{"level":"warn","msg":"irr: IPv6 route query failed","asn":"AS60735","error":"read tcp ... connection reset by peer"}

The WHOIS server rate-limited or dropped the connection for this ASN. The query is skipped and the process continues. If this is frequent, consider reducing concurrency in the irr: block.

Troubleshooting runbook

No flows being detected despite real traffic

  1. Check Prometheus: flowwler_router_bytes_decoded_total{protocol="netflow"} - is it counting?

    • No: packets are not reaching flowwler. Check UDP routing and firewall (sudo ufw status). Confirm the router is sending to the correct IP and port.
    • Yes: packets are arriving, but no flow records. Likely a template timing issue.

  2. Check for template cache misses: flowwler_netflow_template_cache_misses_total

    • High and rising: data flowsets are arriving before templates. Enable debug logging and look for "flow template stored" - if absent, the template refresh interval is very long or templates are being sent to a different collector.

  3. Enable debug logging (logging.level: debug in config, then sudo systemctl reload flowwler) and look for:

    • "flow template stored" - template received
    • "flow record dropped: no destination IP" - flows decoded but missing dst IP (template lacks key fields)
    • "flow data set skipped" - data arriving before template

BPS readings are wrong

Too high (e.g., 10× actual rate):

  • FIRST_SWITCHED / LAST_SWITCHED fields (21/22) are missing from the template - flow duration cannot be computed, so byte counts are not normalised to BPS correctly.
  • Enable debug logging and check the "field_type_ids" log line - are 21 and 22 present?
  • If absent: configure the router flow template to export timing fields, or reduce active-flow-timeout to 1–2 seconds (forces shorter flow records, reducing the error).

Too low:

  • NetFlow: check flowwler_router_sampling_rate for this router. If the metric is absent, no packets have been received yet. If it shows 1 when the router is sampling at 1:1000, the sampling rate has not been learned yet - flowwler learns it from Options Data records, which can take up to one template refresh interval (typically 1–10 minutes). If the router never exports sampling info via Options Templates, set netflow.default_sampling_rate to the router's 1-in-N ratio as a static fallback.
  • sFlow: verify default_sampling_rate in the sFlow config matches the router's actual sampling ratio.
  • Check that flows are being exported for all transit interfaces, not just one.

Mitigation activates but traffic does not drop

  1. Check flowwler_mitigation_effectiveness - is it rising?

    • Near 0 after 2–3 minutes: the BGP route was not accepted by the peer. Check the peer's import policy and community filtering.
    • Near 1: traffic is being dropped at the router. The remaining traffic is background noise or new flows established after the route was applied.

  2. Inspect BGP routes on the router peer (JunOS):

    show route receive-protocol bgp 192.0.2.100 show route protocol bgp community 65535:666

  3. Inspect flowwler's RIB via the REST API:

    bash curl -s http://localhost:9731/api/v1/bgp/routes

  4. Check the BGP session is established:

    bash curl -s http://localhost:9731/api/v1/bgp/peers curl -s http://localhost:9731/metrics | grep flowwler_bgp_peer_up

HoldDown never expires

The hold-down timer is defaults.hold_time (default 15 minutes), or a per-level hold_time override that applies to all mitigation types, not just FlowSpec. If the victim re-triggers before the timer expires, the hold-down is re-entered at the appropriate level. This is expected during an ongoing attack.

To shorten the recovery time for testing, temporarily set hold_time: 1m in config and send a SIGHUP.

Group subnets are not being matched

  • Confirm the group is loaded: curl -s http://localhost:9731/api/v1/groups
  • Confirm flowwler_group_up{group="..."} is 1.
  • If using IRR or NetBox, check that resolution succeeded - look for "irr: resolved prefixes" or "netbox: resolved prefixes" in the logs with a non-zero routes_v4/routes_v6 count.
  • Check flowwler_irr_last_refresh_timestamp_seconds / flowwler_netbox_last_refresh_timestamp_seconds - if these are absent or very old, the last resolution may have failed.

High CPU or memory

  • A large aggregation_window with many unique 5-tuple sources creates many smoothing state entries. These are pruned each tick for keys that leave the window, so memory is proportional to active unique flows × window depth.
  • High sFlow sampling rates (low sample-rate on the router, e.g. 1-in-100) generate many flow records per second. Consider raising the sampling interval on the router.
  • For IRR resolution with large AS-SETs, reduce irr.concurrency to limit simultaneous WHOIS connections if the host's outbound connection rate is constrained.