Routing Problems

Resolution

The tunnel to Route64 is stable again and the BGP session is established. Traffic was not interrupted [adjust: during the incident, traffic was carried by our other upstreams and IXP peerings / some paths were temporarily affected].

Impact

The Route64 upstream was unavailable or flapping for the duration of the incident. [Describe customer impact here, or: "No customer-facing impact was observed."]

Timeline (CEST)
~10:00: We began setting up the new Route64 tunnel, but no traffic passed through it.
~10:15–10:25: The tunnel came up briefly, then dropped. The BGP session to Route64 flapped repeatedly (hold timer expired).
~10:30: The session established and the full table was imported, but connectivity was lost again shortly after.
~11:15: We moved the tunnel to a different source address and confirmed the tunnel carried traffic in both directions.
~11:35: BGP session established and stable. Incident resolved.
Root cause

Two separate issues combined:

UDP checksum handling. Route64's side sends VXLAN over IPv6 with a UDP checksum of zero, which Linux drops by default. The tunnel therefore worked only intermittently, and recreating it reset our workaround. We now explicitly enable zero-checksum receive on the tunnel interface and keep it in the persistent configuration.
Tunnel endpoint inside an announced prefix (recursive routing). Our tunnel source address was part of a prefix that we announce over the same BGP session. As soon as the session came up, return traffic for the tunnel endpoint was routed into the tunnel itself, which broke the tunnel and brought the session down again. We have moved the tunnel endpoint to an address outside of any prefix announced over this session.
Corrective actions
Tunnel source moved to a provider-assigned address outside our announced prefixes.
Zero-checksum receive made persistent in the interface configuration.
Policy routing added so tunnel traffic leaves via the intended uplink.
We will monitor the session closely and re-enable the full prefix announcement towards Route64 in a controlled step. [adjust if already done]
Preventing recurrence
Tunnel endpoints are never to be placed within prefixes announced over the tunnel's own BGP session.
Configuration changes for tunnels are reviewed against the persisted configuration, so manual fixes do not get lost on interface re-creation.
Alerting on BGP session state for upstreams.

We apologize for any inconvenience.

What we know
Our own equipment sends and receives traffic as expected, and we have
verified our network in both directions. The issue appears to be on the
path to or at the affected providers, and we are working with them.

Summary
We are currently experiencing connectivity issues affecting some of our
tunnel-based network services. Affected services may be unreachable or
show degraded connectivity.

Impact
Some tunnel connections are currently not passing traffic in both
directions. Other services are not affected.

What we know
Our own equipment is sending traffic as expected, but we are not
receiving replies. We are investigating whether the cause is on our side
or on the network path.

Next steps
We are checking our network configuration and working with relevant
providers to isolate the issue.

We are experiencing downtime on our VXLAN interface from Route64. This is due to a routing issue in IPv4. We are working on resolving the issue.