[NETVIRT-377] openflow connection broken in 3 node CSIT cluster setup Created: 16/Dec/16  Updated: 08/Apr/19  Resolved: 27/Feb/17

Status: Resolved
Project: netvirt
Component/s: None
Affects Version/s: Boron
Fix Version/s: None

Type: Bug
Reporter: balakrishnan k Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Zip Archive Odl1_karaf_log.zip     Zip Archive Odl2_karaf_log.zip     Zip Archive Odl3_karaf_log.zip     File example2.tar.gz    
External issue ID: 7398

 Description   

In below Run cluster related cases are failed and VM's not getting IP address.
In openstack node observed ODL3 controller connection is not established properly.

https://logs.opendaylight.org/releng/jenkins092/netvirt-legacy-csit-3node-openstack-mitaka-upstream-boron/39/archives/log.html.gz#s1-s3-s2

Manager "tcp:10.29.12.207:6640"
is_connected: true
Manager "tcp:10.29.13.140:6640"
is_connected: true
Manager "tcp:10.29.12.15:6640"
is_connected: true
Bridge br-int
Controller "tcp:10.29.13.140:6653"
Controller "tcp:10.29.12.15:6653"
is_connected: true
Controller "tcp:10.29.12.207:6653"
is_connected: true
fail_mode: secure
Port patch-ext
Interface patch-ext
type: patch
options:

{peer=patch-int}

controller "10.29.13.140" port 6640 connection status is true but 6653 connection status is not displayed.

Next Run all the Test cases are passed this issue is inconsistent.

There are many ERROR in ODL3 controller logs.
uploaded all controller logs.

2016-12-13 20:02:11,373 | INFO | entLoopGroup-7-7 | ConnectionAdapterImpl | 296 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.2.SNAPSHOT | Hello received / branch
2016-12-13 20:02:11,378 | WARN | lt-dispatcher-16 | ReliableDeliverySupervisor | 199 - com.typesafe.akka.slf4j - 2.4.7 | Association with remote system [akka.tcp://opendaylight-cluster-data@10.29.12.207:2550] has failed, address is now gated for [5000] ms. Reason: [Association failed with [akka.tcp://opendaylight-cluster-data@10.29.12.207:2550]] Caused by: [Connection refused: /10.29.12.207:2550]
2016-12-13 20:02:11,380 | ERROR | on-dispatcher-80 | OutboundQueueProviderImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | No queue present, failing request
2016-12-13 20:02:11,381 | INFO | entLoopGroup-7-7 | SystemNotificationsListenerImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | ConnectionEvent: Connection closed by device, Device:/10.29.13.11:50964, NodeId:null
2016-12-13 20:02:11,387 | ERROR | on-dispatcher-82 | OutboundQueueProviderImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | No queue present, failing request
2016-12-13 20:02:11,387 | WARN | on-dispatcher-82 | RpcContextImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:119508379114414
2016-12-13 20:02:11,395 | ERROR | on-dispatcher-69 | OutboundQueueProviderImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | No queue present, failing request
2016-12-13 20:02:11,395 | WARN | on-dispatcher-69 | RpcContextImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:270479877144834
2016-12-13 20:02:11,396 | WARN | on-dispatcher-80 | RpcContextImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:119508379114414
2016-12-13 20:02:11,397 | ERROR | on-dispatcher-80 | OutboundQueueProviderImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | No queue present, failing request
2016-12-13 20:02:11,397 | WARN | on-dispatcher-80 | RpcContextImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:119508379114414
2016-12-13 20:02:11,405 | WARN | entLoopGroup-7-8 | StatisticsGatheringUtils | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Can't write to transaction, transaction chain probably closed.
2016-12-13 20:02:11,405 | WARN | entLoopGroup-7-5 | StatisticsGatheringUtils | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Can't write to transaction, transaction chain probably closed.
2016-12-13 20:02:11,406 | WARN | entLoopGroup-7-8 | StatisticsManagerImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Statistics gathering for single node openflow:270479877144834 was not successful: Device connection is closed for Node : Uri [_value=openflow:270479877144834].
2016-12-13 20:02:11,406 | WARN | entLoopGroup-7-5 | StatisticsManagerImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Statistics gathering for single node openflow:167998386060623 was not successful: Device connection is closed for Node : Uri [_value=openflow:167998386060623].
2016-12-13 20:02:11,406 | INFO | entLoopGroup-7-5 | ConnectionAdapterImpl | 296 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.2.SNAPSHOT | Hello received / branch
2016-12-13 20:02:11,406 | INFO | entLoopGroup-7-8 | ConnectionAdapterImpl | 296 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.8.2.SNAPSHOT | Hello received / branch
2016-12-13 20:02:11,410 | ERROR | on-dispatcher-68 | OutboundQueueProviderImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | No queue present, failing request
2016-12-13 20:02:11,410 | WARN | on-dispatcher-68 | RpcContextImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Xid cannot be reserved for new RequestContext, node:openflow:167998386060623
2016-12-13 20:02:11,420 | WARN | entLoopGroup-7-3 | StatisticsGatheringUtils | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Can't write to transaction, transaction chain probably closed.
2016-12-13 20:02:11,420 | WARN | entLoopGroup-7-3 | StatisticsManagerImpl | 301 - org.opendaylight.openflowplugin.impl - 0.3.2.SNAPSHOT | Statistics gathering for single node openflow:119508379114414 was not successful: Device connection is closed for Node : Uri [_value=openflow:119508379114414].



 Comments   
Comment by Priya Ramasubbu [ 23/Dec/16 ]

Attachment Odl1_karaf_log.zip has been added with description: Odl1_logs

Comment by Priya Ramasubbu [ 23/Dec/16 ]

Attachment Odl2_karaf_log.zip has been added with description: Odl2_logs

Comment by Priya Ramasubbu [ 23/Dec/16 ]

Attachment Odl3_karaf_log.zip has been added with description: Odl3_logs

Comment by Bertrand Low [ 25/Dec/16 ]

The logs may not have similar messages as the first example but the symptoms described in the bug are the same.

Comment by Bertrand Low [ 25/Dec/16 ]

Attachment example2.tar.gz has been added with description: Another instance of the same symptom

Comment by Hari Prasidh [ 27/Feb/17 ]

Issue is not reproducible.

Generated at Wed Feb 07 20:21:24 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.