[OVSDB-289] NetVirt Cluster Testing (odl-ovsdb-openstack) - Security Group Errors in karaf.log Created: 04/Feb/16  Updated: 03/May/18  Resolved: 09/May/16

Status: Resolved
Project: ovsdb
Component/s: openstack.net-virt
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Venkatrangan Govindarajan Assignee: ranjithkumar_t
Resolution: Won't Do Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


Attachments: Zip Archive karaf logs.zip     File odl1_log.tgz     File odl2_log.tgz     File odl3_log.tgz    
External issue ID: 5223

 Description   

Performed simple operations using Openstack and netvirt as cluster

Network Create and then created instances, the instances in Compute node not ping successfully.

Errors in karaf.log

a9-4c56-8299-64179f8d445d
2016-02-03 18:49:54,273 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:54,596 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:54,910 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:55,007 | INFO | ntDispatcherImpl | OF13Provider | 283 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | addTunnelPort enter: portName: vxlan-12.14.16.3
2016-02-03 18:49:55,008 | INFO | ntDispatcherImpl | OF13Provider | 283 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | Tunnel vxlan-12.14.16.3 is present in br-int of ovsdb://uuid/7379aa64-0416-45e2-9531-351397cc742a/bridge/br-int
2016-02-03 18:49:55,026 | INFO | ntDispatcherImpl | OF13Provider | 283 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | addTunnelPort enter: portName: vxlan-12.14.16.2
2016-02-03 18:49:55,026 | INFO | ntDispatcherImpl | OF13Provider | 283 - org.opendaylight.ovsdb.openstack.net-virt-providers - 1.2.1.SNAPSHOT | Tunnel vxlan-12.14.16.2 is present in br-int of ovsdb://uuid/bf460c25-5849-4874-9758-4fcbafbf9f5e/bridge/br-int
2016-02-03 18:49:56,666 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:57,096 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:57,399 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d
2016-02-03 18:49:57,710 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:62e7be1b-56a9-4c56-8299-64179f8d445d



 Comments   
Comment by Venkatrangan Govindarajan [ 04/Feb/16 ]

Scenario

ODL1 -> handle neutron requests

ODL2/ODL3 -> Act as OVS Managers and OF Controllers to Openstack

Initial test Both ODL2 and ODL3 were up -> the compute node instances were not able to reach dHCP

Then ODL2 down, the compute node instances are working fine.

Then ODL3 was made down and retried the cerrate networks + instances, the same phenomenon was observed again.

Comment by Venkatrangan Govindarajan [ 04/Feb/16 ]

Observations

During the first try RPC Exceptions seen in ODL2. During other trials

2016-02-03 20:01:53,719 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:01908c18-35e8-4e19-9923-2a3f3e86e2b6
2016-02-03 20:01:53,754 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:2e72cbf6-a3e2-44b1-97a4-410fb0bc7c9d
2016-02-03 20:01:53,779 | ERROR | ntDispatcherImpl | SecurityGroupCacheManagerImpl | 282 - org.opendaylight.ovsdb.openstack.net-virt - 1.2.1.SNAPSHOT | In processPortAdded cachedport port not found in neuton cache: cachedportUuid:cfb1e4ce-9ee6-4742-95bd-f6201257317b

exception was observed.

Comment by Venkatrangan Govindarajan [ 04/Feb/16 ]

Attachment odl1_log.tgz has been added with description: ODL1 logss

Comment by Venkatrangan Govindarajan [ 04/Feb/16 ]

Attachment odl2_log.tgz has been added with description: ODL2 logs

Comment by Venkatrangan Govindarajan [ 04/Feb/16 ]

Attachment odl3_log.tgz has been added with description: ODL3 logs

Comment by Aswin Suryanarayanan [ 18/Feb/16 ]

Venkat, Patch [1] solves issues with the restart/port delete scenarios. Could you please check whether the issue is solved with this patch?

https://git.opendaylight.org/gerrit/#/c/34410/

Comment by Venkatrangan Govindarajan [ 18/Feb/16 ]

Hi Aswin,

I think while verifying for OVSDB-294 I observed the same errors again in karaf.log. Can you please check the logs posted in #5290. Once you feel that is not the issue, I will retest the issue.

Comment by Aswin Suryanarayanan [ 23/Feb/16 ]

An NPE [1] is thrown on floating ip removing logic as mentioned in [2] when neutronL3Adapter.handleInterfaceEvent is called from SouthboundHandler which will prevent networkingProviderManager.getProvider(node).handleInterfaceDelete of SouthboundHandler being called. This should be causing an issue with clean up of SecurityGroupCache.

The NPE is not caused by port being null. As there is sucessful reference of neutron port object before in the flow. So the floating ip list should have list of null entires.

So fixing [2] should sovle this issue.

[1]
java.lang.NullPointerException
at org.opendaylight.ovsdb.openstack.netvirt.impl.NeutronL3Adapter.cleanupFloatingIPRules(NeutronL3Adapter.java:1457)[280:org.opendaylight.ovsdb.openstack.net-virt:1.2.1.Beryllium]
at org.opendaylight.ovsdb.openstack.netvirt.impl.NeutronL3Adapter.handleNeutronPortEvent(NeutronL3Adapter.java:439)[280:org.opendaylight.ovsdb.openstack.net-virt:1.2.1.Beryllium]
at org.opendaylight.ovsdb.openstack.netvirt.impl.NeutronL3Adapter.handleInterfaceEvent(NeutronL3Adapter.java:808)[280:org.opendaylight.ovsdb.openstack.net-virt:1.2.1.Beryllium]

[2]https://bugs.opendaylight.org/show_bug.cgi?id=5290

Comment by Aswin Suryanarayanan [ 24/Feb/16 ]

The error log is observed in odl1 only Which was never shutdown. A node when restarted it will sync the security group cache with the current neutron port objects.So odl2 and odl3 does not have the error.

Comment by Aswin Suryanarayanan [ 21/Mar/16 ]

Venkat,
With bug-5290 has been fixed, could please confirm whether this issue is observed now in your cluster setup?

Comment by ranjithkumar_t [ 09/May/16 ]

Tested with stable/beryllium and master branch. There is no "Security Group" error in karaf log while making the ODL nodes down. It's working as expected. Hence closing this bug and attached the karaf logs.

Comment by ranjithkumar_t [ 09/May/16 ]

Attachment karaf logs.zip has been added with description: karaf logs for all 3 ODL controllers

Generated at Wed Feb 07 20:36:00 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.