[CONTROLLER-1698] node unreachable if odl under load

Duplicate — Tue, 25 Jul 2023 08:24:40 +0000

The bugs should to track the problem of unreachable nodes, which happenes often, especially if odl is under the load (e.g. bgp suite with 300k prefixes, or if a lot of transactions are produced on nodes).

The scenario in attached logs is simple.
1) produce-transactions rpc are started from all 3 nodes around time 11:15:02
2) On node1 around 11:15:12.xxx the node 3 is marked as unreachable.

Focusing on heartbeats the are sent regularly, but under the load all missing responses come all at once. E.g. on node3

2017-05-24 11:15:06,045 | DEBUG | lt-dispatcher-76 | ClusterHeartbeatSender 2017-05-24 11:15:06,047 | DEBUG | lt-dispatcher-76 | ClusterHeartbeatSender 2017-05-24 11:15:07,055 | DEBUG | lt-dispatcher-47 | ClusterHeartbeatSender 2017-05-24 11:15:08,055 | DEBUG | lt-dispatcher-34 | ClusterHeartbeatSender 2017-05-24 11:15:09,056 | DEBUG | lt-dispatcher-77 | ClusterHeartbeatSender 2017-05-24 11:15:11,977 | DEBUG | lt-dispatcher-79 | ClusterHeartbeatSender 2017-05-24 11:15:11,978 | DEBUG | lt-dispatcher-37 | ClusterHeartbeatSender 2017-05-24 11:15:12,045 | DEBUG | lt-dispatcher-54 | ClusterHeartbeatSender 2017-05-24 11:15:13,056 | DEBUG | lt-dispatcher-79 | ClusterHeartbeatSender 2017-05-24 11:15:15,419 | DEBUG | lt-dispatcher-78 | ClusterHeartbeatSender 2017-05-24 11:15:15,428 | DEBUG | lt-dispatcher-83 | ClusterHeartbeatSender 2017-05-24 11:15:16,055 | DEBUG | lt-dispatcher-75 | ClusterHeartbeatSender 2017-05-24 11:15:17,055 | DEBUG | lt-dispatcher-78 | ClusterHeartbeatSender 2017-05-24 11:15:17,992 | DEBUG | lt-dispatcher-20 | ClusterHeartbeatSender 2017-05-24 11:15:17,993 | DEBUG | lt-dispatcher-20 | ClusterHeartbeatSender 2017-05-24 11:15:18,003 | DEBUG | lt-dispatcher-43 | ClusterHeartbeatSender 2017-05-24 11:15:18,014 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender 2017-05-24 11:15:18,014 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender 2017-05-24 11:15:18,015 | DEBUG | ult-dispatcher-3 | ClusterHeartbeatSender 2017-05-24 11:15:18,020 | DEBUG | lt-dispatcher-43 | ClusterHeartbeatSender 2017-05-24 11:15:18,027 | DEBUG | lt-dispatcher-37 | ClusterHeartbeatSender 2017-05-24 11:15:18,028 | DEBUG | lt-dispatcher-78 | ClusterHeartbeatSender 2017-05-24 11:15:18,031 | DEBUG | lt-dispatcher-78 | ClusterHeartbeatSender | 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat to [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]
| 201 - com.typesafe.akka.slf4j - 2.4.17 | Cluster Node [akka.tcp://opendaylight-cluster-data@172.17.0.6:2550] - Heartbeat response from [akka.tcp://opendaylight-cluster-data@172.17.0.4:2550]

OpenDaylight JIRA

[CONTROLLER-1698] node unreachable if odl under load