Operating System: All
Platform: All

[How to reproduce the bug]

The following procedure does NOT always reproduce the issue.
To reproduce the issue, you have to do the procedure several times.

Step 1. Run Karaf.

Step 2. Install VNT Manager features.

feature:install odl-vtn-manager-all odl-adsal-compatibility-all odl-openflowplugin-all

Step 3. Run Mininet.

sudo mn --controller=remote,ip=HOST --topo tree,2

Step 4. Create a virtual network.

I'm attaching the script for it.

Or you can get the script from the following URL.
https://gist.github.com/anonymous/b0ae9794fc97da5eb4c0

Step 5. Try to send ping packets.

h1 ping h3

The above ping packets should be forwarded.
But, due to the bug, it fails occasionally.

If you don't face the issue, stop the Mininet and Karaf, and restart the above step 1, 3, and 5.
(You don't have to do step 2 and 4.)

Attachment vtn_test_script.sh.sh has been added with description: Script to create a virtual network. (See comment 1)

As Shigeru said in the following mail, it seems that this issue is caused by asynchronous notification of data change event.

https://lists.opendaylight.org/pipermail/vtn-dev/2014-September/000639.html

It seems that the OpenFlow plugin notifies MD-SAL of events of switches in the correct order.
However, MD-SAL processes these notification in different threads.
Because the kernel thread scheduler does not guarantee execution order of these threads, the order of these notification for MD-SAL application is unstable.

Does it mean that this issue is caused by the OpenFlow plugin or MD-SAL?

Hi Ed,

It seems to me that VTN Neutron bundle has a bad impact in your setup.

Therefore, could you install VTN features as follows instead of odl-vtn-manager-all?

feature:install odl-vtn-manager-java-api odl-vtn-manager-northbound
feature:install odl-adsal-compatibility-all odl-openflowplugin-all

It does not install odl-vtn-manager-neutron feature.

So tonight I tested this bug more in detail and it looks like the root cause = CONTROLLER-904 = Missing node connectors in AD-SAL Switch manager.

BR/Luis

The details of he test:

VTN gets the node connector s2-eth2 when node registers:

2014-09-30 08:16:28,835 | INFO | notification-113 | VTNManagerImpl | 218 - org.opendaylight.vtn.manager.implementation - 0.2.0.SNAPSHOT | default: addPort: New port: port=OF|2@OF|00:00:00:00:00:00:00:02, prop=PortProperty[name=s2-eth2,cost=1000,enabled]

However adsal switch manager does not show this port:
{
"nodeConnectorProperties": [
{
"properties": {
"name":

,
"state":

,
"config":

},
"nodeconnector": {
"node":

,
"id": "0",
"type": "SW"
}
},
{
"properties": {
"name":

,
"state":

,
"config":

,
"bandwidth":

},
"nodeconnector": {
"node":

,
"id": "1",
"type": "OF"
}
},
{
"properties": {
"name":

,
"state":

,
"config":

,
"bandwidth":

},
"nodeconnector": {
"node":

,
"id": "3",
"type": "OF"
}
}
]
}

This makes VInterface if3 to go DOWN:

2014-09-30 08:16:35,241 | INFO | Thread: default | VTNManagerImpl | 218 - org.opendaylight.vtn.manager.implementation - 0.2.0.SNAPSHOT | default:vBridge-IF:Tenant1.vBridge2.if3: Virtual interface added: VInterface[name=if3,enabled,state=UNKNOWN,entityState=UNKNOWN]

2014-09-30 08:16:35,294 | INFO | Thread: default | VTNManagerImpl | 218 - org.opendaylight.vtn.manager.implementation - 0.2.0.SNAPSHOT | default:vBridge-IF:Tenant1.vBridge2.if3: Port mapping added: PortMap[config=PortMapConfig[node=OF|00:00:00:00:00:00:00:02,port=SwitchPort[name=s2-eth2],vlan=0]]

2014-09-30 08:16:35,294 | INFO | Thread: default | VTNManagerImpl | 218 - org.opendaylight.vtn.manager.implementation - 0.2.0.SNAPSHOT | default:vBridge-IF:Tenant1.vBridge2.if3: Virtual interface changed: VInterface[name=if3,enabled,state=DOWN,entityState=UNKNOWN]

And finally ping fails in the test:

Ping h2 to h4 :: Ping h2 to h4, verify no packet loss | FAIL |
'PING 10.0.0.4 (10.0.0.4) 56(84) bytes of data.
From 10.0.0.2 icmp_seq=1 Destination Host Unreachable
From 10.0.0.2 icmp_seq=2 Destination Host Unreachable
From 10.0.0.2 icmp_seq=3 Destination Host Unreachable

— 10.0.0.4 ping statistics —
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2015ms

BR/Luis

Fix here:

https://git.opendaylight.org/gerrit/#/c/11705

So far it has succeeded 8 out of 8 tries for me

(In reply to Ed Warnicke from comment #7)
> Fix here:
>
> https://git.opendaylight.org/gerrit/#/c/11705
>
>
> So far it has succeeded 8 out of 8 tries for me

Hi Ed,

Thank you for submitting the patch.

I've tested the patch, and it has succeeded 10 out 10 tries in my setup.
I think this patch fixes the bug.

Per Hideyuki's comments, this is fixed for him.