[NETVIRT-1106] failures in Create Multiple VPNs test Created: 16/Feb/18  Updated: 07/Jul/18  Resolved: 07/Jul/18

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Oxygen, Fluorine
Fix Version/s: Oxygen-SR1, Fluorine

Type: Bug Priority: Medium
Reporter: Valentina Krasnobaeva Assignee: Philippe Guibert
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Create Multiple VPNs test fails, because another L3VPN, created earlier (with the same name and ID) was not deleted properly by executing deleteL3VPN call.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-oxygen/764/odl_1/odl1_karaf.log.gz

 Comments   
Comment by Kit Lou [ 27/Feb/18 ]

Please provide an update on this issue.  Is this a true blocker?  Any ETA on resolution?

Comment by Daniel Farrell [ 08/Mar/18 ]

pguibert6WIND/valentina.krasnobaeva - We need to get this resolved urgently, it's blocking the imminent Oxygen release.

Comment by Philippe Guibert [ 08/Mar/18 ]

Hi,

ticket is not updated yet.

once integrated, ticket should be updated.

 

Comment by Valentina Krasnobaeva [ 19/Mar/18 ]

reproduced while perfoming operations with dualstack networks

createdL3VPN, which was associated, then disassociated from dualstack network can't be removed, because of exception below: 

2018-03-19T17:25:15,646 | INFO | ForkJoinPool-1-worker-1 | UpdateIdEntryJob | 251 - org.opendaylight.genius.idmanager-impl - 0.4.0.SNAPSHOT | Updated id entry with idValues [], idKey nexthop.10006311.1.1.2/32, pool nextHopPointerPool.174325811
 2018-03-19T17:25:15,646 | TRACE | org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.l3vpn.rev130911.vpn._interface.op.data.VpnInterfaceOpDataEntry_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | VpnInterfaceOpListener | 367 - org.opendaylight.netvirt.vpnmanager-impl - 0.6.0.SNAPSHOT | VpnInterfaceOpListener updated: original VpnInterfaceOpDataEntry{getDpnId=119626733642592, getGatewayMacAddress=00:00:00:00:f5:ad, getLportTag=13, getName=403d9f6c-3db0-4db9-bbd4-892c383e6499, getVpnInstanceName=7ab8cd92-48ca-49b5-94e1-b2921a2616ab, getVpnInterfaceState=Active, isScheduledForRemove=true, augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.l3vpn.rev130911.AdjacenciesOp=AdjacenciesOp{getAdjacency=[Adjacency{getAdjacencyType=PrimaryAdjacency, getIpAddress=11.1.1.2/32, getLabel=100065, getMacAddress=fa:16:3e:69:6e:d4, getNextHopIpList=[10.100.0.11], getSubnetId=Uuid [_value=66be7703-7332-4c32-b375-cca8ab873963], getVrfId=3206:2, isPhysNetworkFunc=false, augmentations={}}, Adjacency{getAdjacencyType=PrimaryAdjacency, getIpAddress=2002:db7:0:2:f816:3eff:fe69:6ed4/128, getLabel=100067, getMacAddress=fa:16:3e:69:6e:d4, getNextHopIpList=[], getSubnetId=Uuid [_value=02c5bb95-6b46-4a8b-b3c4-6329d82178a5], getVrfId=3206:2, isPhysNetworkFunc=false, augmentations={}}]}}} updated VpnInterfaceOpDataEntry{getDpnId=119626733642592, getGatewayMacAddress=00:00:00:00:f5:ad, getLportTag=13, getName=403d9f6c-3db0-4db9-bbd4-892c383e6499, getVpnInstanceName=7ab8cd92-48ca-49b5-94e1-b2921a2616ab, getVpnInterfaceState=Active, isScheduledForRemove=true, augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.netvirt.l3vpn.rev130911.AdjacenciesOp=AdjacenciesOp{getAdjacency=[Adjacency{getAdjacencyType=PrimaryAdjacency, getIpAddress=2002:db7:0:2:f816:3eff:fe69:6ed4/128, getLabel=100067, getMacAddress=fa:16:3e:69:6e:d4, getNextHopIpList=[], getSubnetId=Uuid [_value=02c5bb95-6b46-4a8b-b3c4-6329d82178a5], getVrfId=3206:2, isPhysNetworkFunc=false, augmentations={}}]}}}
 2018-03-19T17:25:15,649 | INFO | ForkJoinPool-1-worker-0 | UpdateIdEntryJob | 251 - org.opendaylight.genius.idmanager-impl - 0.4.0.SNAPSHOT | Updated id entry with idValues [], idKey 3206:2.11.1.1.2/32, pool vpnservices.174325811
 2018-03-19T17:25:16,059 | DEBUG | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Vrfs_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-1 | BgpConfigurationManager | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | received remove Vrfs config val 3206:2
 2018-03-19T17:25:16,059 | DEBUG | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpConfigurationManager | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | received remove Networks config val 2002:db7:0:2::/64
 2018-03-19T17:25:16,060 | INFO | ResourceBatchingManager-0 | ResourceBatchingManager | 260 - org.opendaylight.genius.mdsalutil-api - 0.4.0.SNAPSHOT | Total taken ##time = 3ms for resourceList of size 3 for resourceType BGP-RESOURCES
 2018-03-19T17:25:16,061 | DEBUG | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpRouter | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | Deleting BGP route - rd:3206:2 prefix:2002:db7:0:2::/64 
 2018-03-19T17:25:16,061 | ERROR | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpConfigurationManager | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | networks Delete received exception; Config store updated; undo with Add if needed.
 org.apache.thrift.TApplicationException: BGP RD 3206:2 not configured
 at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[145:org.apache.thrift:0.9.3]
 at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) [145:org.apache.thrift:0.9.3]
 at org.opendaylight.netvirt.bgpmanager.thrift.gen.BgpConfigurator$Client.recv_withdrawRoute(BgpConfigurator.java:406) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.thrift.gen.BgpConfigurator$Client.withdrawRoute(BgpConfigurator.java:387) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.thrift.client.BgpRouter.dispatchInternal(BgpRouter.java:306) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.thrift.client.BgpRouter.dispatch(BgpRouter.java:215) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.thrift.client.BgpRouter.delPrefix(BgpRouter.java:470) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager$NetworksReactor.remove(BgpConfigurationManager.java:1134) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.netvirt.bgpmanager.BgpConfigurationManager$NetworksReactor.remove(BgpConfigurationManager.java:1052) [339:org.opendaylight.netvirt.bgpmanager-impl:0.6.0.SNAPSHOT]
 at org.opendaylight.genius.datastoreutils.AsyncDataTreeChangeListenerBase$DataTreeChangeHandler.run(AsyncDataTreeChangeListenerBase.java:154) [260:org.opendaylight.genius.mdsalutil-api:0.4.0.SNAPSHOT]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:?]
 at java.lang.Thread.run(Thread.java:748) [?:?]
 2018-03-19T17:25:16,062 | DEBUG | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpConfigurationManager | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | received remove Networks config val 11.1.1.2/32
 2018-03-19T17:25:16,062 | DEBUG | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpRouter | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | Deleting BGP route - rd:3206:2 prefix:11.1.1.2/32 
 2018-03-19T17:25:16,063 | ERROR | org.opendaylight.yang.gen.v1.urn.ericsson.params.xml.ns.yang.ebgp.rev150901.bgp.Networks_AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-0 | BgpConfigurationManager | 339 - org.opendaylight.netvirt.bgpmanager-impl - 0.6.0.SNAPSHOT | networks Delete received exception; Config store updated; undo with Add if needed.
 org.apache.thrift.TApplicationException: BGP RD 3206:2 not configured

 As we can see from trace, exception happens, when we execute associateNetworks RPC call

Comment by Daniel Farrell [ 19/Mar/18 ]

valentina.krasnobaeva - I see you've re-opened this bug. Note that it's currently marked as a blocker against Oxygen, and that it's far too late to wait for a fix. Unless a fix is going to be merged immediately I'm going to recommend to the TSC that we don't delay the Oxygen release for this. We need to get an autorelease build (getting lucky and avoiding all "sporadic" failures) and have all projects sign-off on the test results by Wednesday. /cc shague

Comment by Sam Hague [ 19/Mar/18 ]

valentina.krasnobaeva is there a CSIT test failure or are these just exceptions coming out from manual testing? As dfarrell07 mentions this bug was tagged as a blocker so reopening it makes visible to the oxygen release again. Can we lower this to critical?

Comment by Daniel Farrell [ 19/Mar/18 ]

valentina.krasnobaeva - I'm going to go ahead and lower this to critical. I don't think it's realistic to get a fix in for Oxygen. It would have to be a massive regression to justify waiting for it at all.

Comment by Sam Hague [ 06/Apr/18 ]

https://git.opendaylight.org/gerrit/70421

Generated at Wed Feb 07 20:23:15 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.