Juniper MX: IPFIX inline monitoring, RTBH and FlowSpec

Checked against Juniper’s documentation (Inline Monitoring Services, BGP flow specification, no-validate) and against captures from an MX480. Check every statement against the documentation for your Junos release before you commit it. Lines we haven’t seen verbatim in Juniper’s documentation are marked check on your release.

The examples use these addresses. Replace them with your own:

Flow export with inline monitoring

Heimnull works best with Inline Monitoring Services, which export each sampled packet as an IPFIX record: the ingress and egress interface, the flow direction, the frame size and the first bytes of the frame (IE 315, dataLinkFrameSection, up to 126 bytes on Junos OS), plus an options record with the sampling rate. There’s no flow cache, so samples leave the router within about a second. That gives per-second detail on the Live page and attack detection in about 5.5 seconds.

The configuration has four parts, under [edit services inline-monitoring] and [edit firewall] (Juniper, “Inline Monitoring Services Configuration”):

services {
    inline-monitoring {
        template heimnull {
            template-refresh-rate 60;          /* check on your release: statement names and units */
            option-template-refresh-rate 60;   /* check on your release */
        }
        instance heimnull {
            template-name heimnull;
            maximum-clip-length 126;           /* Junos OS maximum */
            collector heimnull {
                source-address 10.0.0.1;
                destination-address 10.0.0.10;
                destination-port 4739;
                sampling-rate 512;
            }
        }
    }
}
firewall {
    family inet {
        filter heimnull-sample {
            term all {
                then {
                    inline-monitoring-instance heimnull;
                    accept;
                }
            }
        }
    }
    /* the same filter under family inet6 for IPv6 */
}
interfaces {
    xe-0/0/0 {
        unit 0 {
            family inet { filter { input heimnull-sample; } }
        }
    }
}

• Where. Apply the filter on the transit (WAN) units only, on input, so each packet is sampled once. Junos allows up to 16 instances and 4 collectors per instance.
• Sampling rate. Heimnull reads the rate from the options record. The MX we tested sends it only about every 80 seconds, so a lower option refresh rate makes the rate known sooner after a Heimnull restart. Until the rate is known, Heimnull holds the records briefly rather than storing them unscaled.
• Exporter identity. Heimnull identifies the MX by source-address. Keep it fixed, on a loopback.
• IPv6. Apply the same filter under family inet6 on the same units.

Sample both sides, for FlowSpec

The input filter samples a packet before the FlowSpec filter (a forwarding-table filter) can drop it, so inbound samples show an attack at its full rate even while FlowSpec discards it. To see what FlowSpec lets through, also sample on the output of the units facing your own networks (customers, core), and turn on Sampled on both sides for the exporter in Heimnull (Configure → Exporters).

interfaces {
    xe-0/1/0 {                 /* towards customers or the core */
        unit 0 {
            family inet  { filter { output heimnull-sample; } }
            family inet6 { filter { output heimnull-sample; } }
        }
    }
}

Those samples are second copies of traffic already counted at the WAN. Heimnull keeps them out of every total and uses them only for the rate delivered past the router. Without them, FlowSpec will always escalate to a blackhole, because the arriving rate never drops. See FlowSpec.

• Give every WAN unit its role in Heimnull (Configure → Interface settings): Transit or Peering. Every sample not taken on one of those is then treated as a copy.
• On output from a LAG, the MX names the member’s unit (xe-0/3/3.48 for ae0.48). That needs no role.
• Inbound traffic produces about twice the samples.
• The exporter form’s Where samples are taken table lists each kind of sample the MX sends (input or output, both interfaces with their roles) and whether it counts as traffic or as a copy. Samples taken on internal units should all say “copy”.

Alternative: inline-jflow

Inline J-Flow (flow records with active and inactive timeouts) works too. Records arrive a timeout late, though, so detection lags by the inactive timeout. If you use it, set flow-inactive-timeout 10 (the Junos minimum), flow-active-timeout 30 and a template refresh of 60 seconds. Inline monitoring is the better fit.

A discard next hop

Heimnull sends each blackhole with a per-session next hop, set on its BGP page (“IPv4 next hop” and “IPv6 next hop”). Point those next hops at discard routes on the MX:

routing-options {
    static {
        route 192.0.2.254/32 discard;
    }
    rib inet6.0 {
        static {
            route 2001:db8::ffff/128 discard;
        }
    }
}

Alternatively, leave Heimnull’s next hops empty and set next-hop discard in the import policy (see the import policy). Either way, the MX drops the traffic before it reaches the host.

The iBGP session

protocols {
    bgp {
        group heimnull {
            type internal;
            local-address 10.0.0.1;
            family inet { unicast; }
            family inet6 { unicast; }
            /* FlowSpec: leave these out until you use it */
            family inet { flow { no-validate heimnull-flow; } }
            family inet6 { flow { no-validate heimnull-flow; } }
            import heimnull-in;
            export heimnull-out;
            neighbor 10.0.0.10 {
                description "Heimnull";
            }
        }
    }
}

• Send nothing. heimnull-out must reject everything. Heimnull accepts no routes from any session, but anything sent still sits in its BGP speaker’s memory (about 0.8 GiB per million routes). Origin ASNs come from the flow records and Heimnull’s datasets, not from BGP.
• Heimnull’s BGP speaker (GoBGP) listens on the Heimnull host’s port 179.
• In Heimnull, add the session on the BGP page with the MX’s address and AS, and turn on “Announce”.

Import policy: host routes with the community only

policy-options {
    community heimnull-blackhole members 65535:666;
    policy-statement heimnull-in {
        term blackhole-v4 {
            from {
                family inet;
                community heimnull-blackhole;
                route-filter 203.0.113.0/24 prefix-length-range /32-/32;
            }
            then {
                next-hop discard;
                accept;
            }
        }
        term blackhole-v6 {
            from {
                family inet6;
                community heimnull-blackhole;
                route-filter 2001:db8:42::/48 prefix-length-range /128-/128;
            }
            then {
                next-hop discard;
                accept;
            }
        }
        term flowspec {
            /* Heimnull's safety checks already limit the rules to your networks */
            from rib [ inetflow.0 inet6flow.0 ];
            then accept;
        }
        term reject-rest {
            then reject;
        }
    }
    policy-statement heimnull-out {
        then reject;
    }
}

• The route-filter terms repeat Heimnull’s own safety checks: only /32 or /128, only inside your networks. The router stays safe even if Heimnull is misconfigured.
• Rolling out. You can keep then reject in the blackhole terms while Heimnull runs for real. The routes arrive and are visible (show route receive-protocol bgp 10.0.0.10) but are never installed.

FlowSpec

FlowSpec lets Heimnull filter only the attack (for example UDP from source port 123 to one host, rate-limited) instead of dropping everything to the host. In Heimnull, turn on “IPv4 FlowSpec” and “IPv6 FlowSpec” for the session on the BGP page, then choose the action “FlowSpec” or “FlowSpec, escalating to a blackhole” per hostgroup. How rules are built is on the FlowSpec page.

routing-options {
    flow {
        term-order standard;   /* RFC 8955 term ordering, not the legacy default */
    }
}
policy-options {
    policy-statement heimnull-flow {
        term from-heimnull {
            from neighbor 10.0.0.10;
            then accept;
        }
        term reject-rest {
            then reject;
        }
    }
}

• Validation. RFC 8955 section 6 accepts a flow route only if its originator also originated the best unicast route for the destination prefix. Heimnull never announces those unicast routes, so every rule would fail validation and stay inactive. no-validate heimnull-flow (in the BGP group above) skips that check for routes the policy accepts, and heimnull-flow accepts only Heimnull’s session. RFC 9117 relaxes the check for iBGP, but support differs by release; no-validate works regardless.
• What the MX installs. Accepted rules land in inetflow.0 and inet6flow.0, and become terms of the forwarding-table filters __flowspec_default_inet__ and __flowspec_default_inet6__, applied to all interfaces.
• Actions. Heimnull sends discard as traffic-rate 0, and rate-limit as a traffic-rate in bytes per second (RFC 8955: the bits per second you enter, divided by 8). A redirect route target (AS:N) needs a VRF on the MX that imports that target, such as a scrubbing path. DSCP marking rewrites the packets’ DSCP.
• Safety. Heimnull refuses rules whose own-network prefix (the destination for inbound attacks, the source for outbound ones) is outside your networks, whitelisted, or shorter than a host (unless the hostgroup allows aggregate FlowSpec), and rules that match only an address, which would be a blackhole. The route-filter terms of the import policy don’t apply to flow routes.
• Rolling out. As with blackholes, then reject in heimnull-flow lets you watch the rules arrive (show route receive-protocol bgp 10.0.0.10 table inetflow.0) without installing them.

Escalation

With “FlowSpec, escalating to a blackhole”, Heimnull also announces the blackhole if the attack is still above the hostgroup’s rate after the escalation delay (60 seconds by default).

• Which rate. For inbound attacks, the rate delivered past the router: the samples on the output of internal units (Sample both sides). The attack page charts it as “Delivered”. Without those samples, Heimnull sees only the arriving rate, which FlowSpec doesn’t change, so FlowSpec will always escalate to a blackhole if traffic isn’t sampled on both sides.
• Outbound attacks (one of your hosts attacking someone) are counted on the WAN output, after the filter, so their escalation already sees what FlowSpec lets through.

Upstream RTBH (optional)

To have your transit providers drop the traffic before it reaches you, re-advertise the blackholed host route to them with their blackhole community. Each provider documents its own: RFC 7999’s 65535:666 is common, and many also want no-export. A sketch for one provider:

policy-options {
    community transit-a-blackhole members [ 65535:666 no-export ];   /* the provider's, per its docs */
    policy-statement transit-a-out {
        term heimnull-rtbh {
            from {
                protocol bgp;
                community heimnull-blackhole;
                route-filter 203.0.113.0/24 prefix-length-range /32-/32;
            }
            then {
                community add transit-a-blackhole;
                next-hop self;          /* check what the provider expects */
                accept;
            }
        }
        /* ... your normal export terms ... */
    }
}

• Many providers accept /32 blackholes only with their community, only for prefixes they already route for you, and only from the session they expect. Test with one address and the provider’s looking glass before you rely on it.
• Keep the route-filter, so only host routes inside your networks can leave.
• Once upstreams drop the traffic, the MX no longer samples it, and the attack looks over. Heimnull handles this at expiry: it withdraws the blackhole, keeps watching, and blackholes the address again for longer if the attack is still there. See RTBH blackholes.

Check it

• show bgp neighbor 10.0.0.10: the session is Established.
• show route receive-protocol bgp 10.0.0.10 detail: the /32 with its communities while a blackhole is active.
• show route 203.0.113.10/32 detail: the next hop is discard.
• FlowSpec: show route table inetflow.0 detail (or inet6flow.0), show firewall filter __flowspec_default_inet__ (packet and byte counters per rule) and show route flow validation detail.
• In Heimnull, Mitigations → “Mitigate an address” → Preview shows what would be sent before anything is, and each mitigation’s page shows what the session actually received.