https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/BGPCEP-988 bgpcep <p>Since release bgpcep-0.16.8, BGP is failing to start properly and establish a BGP session with a peer for Link State family.</p> <p> </p> <p>A fist set of error occurs when BGP Topology Deployer started:</p> <p>opendaylight-cluster-data-akka.actor.default-dispatcher-5 | ClusterSingletonServiceGroupImpl | 261 - org.opendaylight.mdsal.singleton-dom-impl - 8.0.7 | Service group example-bgp-rib-service-group service org.opendaylight.protocol.bgp.rib.impl.config.BGPClusterSingletonService@3a8d67e1 failed to start, attempting to continue<br/> com.google.common.base.VerifyException: expected a non-null reference</p> <p> </p> <p>Then a problem occurs with data store when adding a peer:</p> <p>org.opendaylight.mdsal.common.api.OptimisticLockFailedException: Optimistic lock failed for path /(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2018-03-29)bgp-rib/rib/rib<span class="error">&#91;\{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2018-03-29)id=example-bgp-rib}&#93;</span>/loc-rib/tables/tables<span class="error">&#91;\{(urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2018-03-29)afi=(urn:opendaylight:params:xml:ns:yang:bgp-linkstate?revision=2020-01-20)linkstate-address-family, (urn:opendaylight:params:xml:ns:yang:bgp-rib?revision=2018-03-29)safi=(urn:opendaylight:params:xml:ns:yang:bgp-linkstate?revision=2020-01-20)linkstate-subsequent-address-family}&#93;</span>/attributes</p> <p>This conduct to many errors when attempting to write Link State information into the Datastore.</p> <p>Full log is attached.</p> <p>BGP speaker is a Juniper MX 960 with JunOS 21.1R1.11.</p> <p>Karaf is a fresh compilation of BGPCEP master.</p> <p>Note that the problem does not occur with Phosphorus release.</p> BGPCEP-988

Regression with BGP Link State

Bug High Resolved Done Marek Zaťko Olivier Dugeon regression Tue, 9 Nov 2021 10:56:51 +0000 Sun, 21 Nov 2021 15:00:25 +0000 Sun, 21 Nov 2021 14:18:32 +0000 0.16.8 0.16.9 0.16.11 BGP 0 3 <p>Hi Olivier, can you please describe step by step, how are you starting the karaf and which bundles/features are you installing upon startup? Because I was not able to reproduce this issue, I tried felix and equinox. Reason I'm asking this is because in provided logs I see this chunk at startup:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 2021-11-09T11:35:39,861 | INFO | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-32 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGP Deployer global-bgp started. 2021-11-09T11:35:39,885 | INFO | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-32 | OSGiStatementRegistry | 370 - org.opendaylight.bgpcep.bgp-openconfig-rp-spi - 0.16.10.SNAPSHOT | Stopping 0 StatementProviderActivator instances 2021-11-09T11:35:39,915 | INFO | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-32 | OSGiStatementRegistry | 370 - org.opendaylight.bgpcep.bgp-openconfig-rp-spi - 0.16.10.SNAPSHOT | Starting 1 StatementProviderActivator instances 2021-11-09T11:35:39,957 | INFO | Blueprint Extender: 2 | BlueprintContainerImpl | 27 - org.apache.aries.blueprint.core - 1.10.3 | Blueprint bundle org.opendaylight.bgpcep.bgp-peer-acceptor/0.16.10.SNAPSHOT is waiting <span class="code-keyword">for</span> dependencies [(&amp;(|(type=<span class="code-keyword">default</span>)(!(type=*)))(objectClass=org.opendaylight.mdsal.binding.api.DataBroker))] 2021-11-09T11:35:39,974 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group registered 2021-11-09T11:35:39,994 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGP Deployer global-bgp started. 2021-11-09T11:35:40,002 | INFO | opendaylight-cluster-data-notification-dispatcher-41 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group registered 2021-11-09T11:35:40,035 | INFO | Blueprint Extender: 2 | BlueprintContainerImpl | 27 - org.apache.aries.blueprint.core - 1.10.3 | Blueprint bundle org.opendaylight.netconf.restconf-nb-bierman02/2.0.8 is waiting <span class="code-keyword">for</span> dependencies [(objectClass=org.opendaylight.aaa.web.WebContextSecurer)] 2021-11-09T11:35:40,037 | INFO | Blueprint Extender: 2 | BlueprintContainerImpl | 27 - org.apache.aries.blueprint.core - 1.10.3 | Blueprint bundle org.opendaylight.netconf.restconf-nb-rfc8040/2.0.8 is waiting <span class="code-keyword">for</span> dependencies [(objectClass=org.opendaylight.aaa.web.WebContextSecurer)] 2021-11-09T11:35:40,037 | INFO | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-32 | OSGiDistributedDataStore | 220 - org.opendaylight.controller.sal-distributed-datastore - 4.0.6 | Distributed Datastore type CONFIGURATION started 2021-11-09T11:35:40,046 | INFO | opendaylight-cluster-data-notification-dispatcher-41 | LazyBindingMap | 226 - org.opendaylight.mdsal.binding-dom-codec - 8.0.7 | Using lazy population <span class="code-keyword">for</span> maps larger than 1 element(s) 2021-11-09T11:35:40,064 | INFO | Blueprint Extender: 1 | H2TokenStore | 193 - org.opendaylight.aaa.idm-store-h2 - 0.14.6 | Initialized token store with <span class="code-keyword">default</span> cache config 2021-11-09T11:35:40,067 | INFO | Blueprint Extender: 1 | StoreBuilder | 188 - org.opendaylight.aaa.authn-api - 0.14.6 | Checking <span class="code-keyword">if</span> <span class="code-keyword">default</span> entries must be created in IDM store 2021-11-09T11:35:40,067 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | StringValueObjectFactory | 236 - org.opendaylight.mdsal.binding-spec-util - 8.0.7 | Instantiated factory <span class="code-keyword">for</span> <span class="code-keyword">class </span>org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.Ipv4AddressNoZone 2021-11-09T11:35:40,070 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | StringValueObjectFactory | 236 - org.opendaylight.mdsal.binding-spec-util - 8.0.7 | Instantiated factory <span class="code-keyword">for</span> <span class="code-keyword">class </span>org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.Ipv4Prefix 2021-11-09T11:35:40,071 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | StringValueObjectFactory | 236 - org.opendaylight.mdsal.binding-spec-util - 8.0.7 | Instantiated factory <span class="code-keyword">for</span> <span class="code-keyword">class </span>org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.Ipv6AddressNoZone 2021-11-09T11:35:40,072 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | StringValueObjectFactory | 236 - org.opendaylight.mdsal.binding-spec-util - 8.0.7 | Instantiated factory <span class="code-keyword">for</span> <span class="code-keyword">class </span>org.opendaylight.yang.gen.v1.urn.ietf.params.xml.ns.yang.ietf.inet.types.rev130715.Ipv6Prefix 2021-11-09T11:35:40,139 | INFO | Blueprint Extender: 3 | BlueprintContainerImpl | 27 - org.apache.aries.blueprint.core - 1.10.3 | Blueprint bundle org.opendaylight.bgpcep.bgp-peer-acceptor/0.16.10.SNAPSHOT has been started 2021-11-09T11:35:40,139 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | Closing BGP Deployer. 2021-11-09T11:35:40,147 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group close 2021-11-09T11:35:40,148 | INFO | Blueprint Event Dispatcher: 1 | BlueprintBundleTracker | 204 - org.opendaylight.controller.blueprint - 4.0.6 | Blueprint container <span class="code-keyword">for</span> bundle org.opendaylight.bgpcep.bgp-peer-acceptor_0.16.10.SNAPSHOT [378] was successfully created 2021-11-09T11:35:40,166 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGP Deployer global-bgp started. 2021-11-09T11:35:40,170 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group registered 2021-11-09T11:35:40,172 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | Closing BGP Deployer. 2021-11-09T11:35:40,176 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group close 2021-11-09T11:35:40,178 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | OSGiStatementRegistry | 370 - org.opendaylight.bgpcep.bgp-openconfig-rp-spi - 0.16.10.SNAPSHOT | Stopping 1 StatementProviderActivator instances 2021-11-09T11:35:40,186 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | OSGiStatementRegistry | 370 - org.opendaylight.bgpcep.bgp-openconfig-rp-spi - 0.16.10.SNAPSHOT | Starting 2 StatementProviderActivator instances 2021-11-09T11:35:40,192 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGP Deployer global-bgp started. 2021-11-09T11:35:40,197 | INFO | opendaylight-cluster-data-notification-dispatcher-40 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group registered 2021-11-09T11:35:40,209 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | Closing BGP Deployer. 2021-11-09T11:35:40,213 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group close 2021-11-09T11:35:40,215 | INFO | opendaylight-cluster-data-akka.actor.<span class="code-keyword">default</span>-dispatcher-5 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group instantiated 2021-11-09T11:35:40,230 | INFO | Start Level: Equinox Container: a81c060d-3c5c-4444-a60e-1b22e133ec93 | DefaultBgpDeployer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGP Deployer global-bgp started. </pre> </div></div> <p>which says that BGP deployer is started and closed multiple times which seems rather unusual.</p> <p>That flapping is okay given the startup flux, as we are eagerly registering instances &#8211; we might want to add logging for that.</p> <p>Okay, this part is interesting:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2021-11-09T11:35:40,406 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-5 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group instantiated 2021-11-09T11:35:40,407 | INFO | opendaylight-cluster-data-akka.actor.default-dispatcher-5 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.10.SNAPSHOT | BGPClusterSingletonService example-bgp-rib-service-group instantiated </pre> </div></div> <p>This would seem to indicate we have two of these running concurrently, i.e. did we synchronize on deployer shutdown? If not, that would explain the verifyNotNull() tripping over.</p> <p>After briefly browsing through the code, it seems shutdown paths in rib-impl are (and were for a long time) rather broken w.r.t. singleton lifecycle.<br/> Migration from blueprint to SCR, where components are shut down/restarted as needed is flushing these out. Blueprint was hiding these problems by never issuing a restart.</p> <p>Here it is the sequences I used to install and configure ODL</p> <ol> <li>untar bgpcep karaf</li> <li>launch karaf</li> <li>install restconf feature with feature:install features-restconf and bgp feature with feature:install features-bgp</li> <li>stop karaf and replace the etc/opendaylight/bgpcep/protocol-configs.xml with mine setting (mostly change the AS number and IP address of ODL and disable BGP peer)</li> <li>start again karaf</li> <li>remove old BGP peers (the ones that figure in the original protocol-configs.xml)</li> <li>install the BGP peer that corresponds to my Juniper router</li> </ol> <p>Regarding the log I upload, it corresponds to a fresh startup after installation and configuration I described above (i.e. stop karaf after step 7, remove karaf.log and start karaf again).</p> <p>Note, that it was always necessary to restart karaf after installing bgp and/or pcep features to get them properly start.</p> <p>Ah, thanks for the note, that is a gap I was unaware of. So after this we should be something like third of the way towards not needing that Karaf restart <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>Patch set solved the problem, but introduce a new regression: it is necessary to restart karaf after adding a new peer in order to get the BGP session up and running.</p> <p>I proceed as usual (see above comment) to install the new compiled version. The problem take places just right after adding a new peer  through the REST API. Peer seems correctly added, but BGP session is not fire up. It is necessary to restart karaf to get it up and running. I attached 2 new logs files. Thefirst one, fresh-install.log, captured log when default peers coming from the default configuration file are removed and the new one added. The second one, restart.log, captured the log when karaf is restarted. Note that DEBUG for org.opendaylight.bgpcep.bgp has been enabled before removing peers, but nothing pertinent is revealed and there is no crash and no exception. Just nothing happen.</p> <p>I just updated the patch, added few logs to session creation, do you mind trying it again with debug log level before adding your peer and posting results? I tried reproducing your scenario but I couldn't. I tried the following:<br/> 1. feature:install features-restconf features-bgp<br/> 2. shut down karaf<br/> 3. Modify etc/opendaylight/bgpcep/protocol-configs.xml to contain my desired configuration with new neighbor<br/> 4. start karaf<br/> Neighbor is connecting without any issues in my scenario. I also tried connecting new neighbor via RESTCONF upon restarting karaf, session is initialized just fine for me.<br/> In your fresh-install logs the relevant piece is this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">2021-11-18T14:29:56,582 | INFO | opendaylight-cluster-data-notification-dispatcher-53 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.11.SNAPSHOT | Creating Peer instance IpAddress{_ipv4Address=Ipv4Address{_value=10.194.77.242}} with configuration: Neighbor{afiSafis=AfiSafis{afiSafi={AfiSafiKey{_afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.http.openconfig.net.yang.bgp.types.rev151009.IPV4UNICAST}=AfiSafi{afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.http.openconfig.net.yang.bgp.types.rev151009.IPV4UNICAST}, AfiSafiKey{_afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.http.openconfig.net.yang.bgp.types.rev151009.IPV6UNICAST}=AfiSafi{afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.http.openconfig.net.yang.bgp.types.rev151009.IPV6UNICAST}, AfiSafiKey{_afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.openconfig.extensions.rev180329.LINKSTATE}=AfiSafi{afiSafiName=<span class="code-keyword">interface</span> org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.bgp.openconfig.extensions.rev180329.LINKSTATE}}}, config=Config{peerAs=AsNumber{_value=65522}, peerType=INTERNAL, routeFlapDamping=<span class="code-keyword">false</span>, sendCommunity=NONE}, neighborAddress=IpAddress{_ipv4Address=Ipv4Address{_value=10.194.77.242}}, routeReflector=RouteReflector{config=Config{routeReflectorClient=<span class="code-keyword">true</span>}}, timers=Timers{config=Config{connectRetry=10, holdTime=180, keepaliveInterval=30, minimumAdvertisementInterval=30}}, transport=Transport{config=Config{mtuDiscovery=<span class="code-keyword">false</span>, passiveMode=<span class="code-keyword">false</span>, augmentation=[NeighborTransportConfig{remotePort=PortNumber{_value=179}}]}}} 2021-11-18T14:29:56,583 | INFO | opendaylight-cluster-data-notification-dispatcher-53 | BgpPeer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.11.SNAPSHOT | Starting BgPeer instance IpAddress{_ipv4Address=Ipv4Address{_value=10.194.77.242}} 2021-11-18T14:29:56,584 | INFO | opendaylight-cluster-data-notification-dispatcher-53 | AbstractPeer | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.11.SNAPSHOT | Creating DOM peer chain <span class="code-keyword">null</span> 2021-11-18T14:29:56,584 | INFO | opendaylight-cluster-data-notification-dispatcher-53 | BGPClusterSingletonService | 380 - org.opendaylight.bgpcep.bgp-rib-impl - 0.16.11.SNAPSHOT | Peer instance created IpAddress{_ipv4Address=Ipv4Address{_value=10.194.77.242}} </pre> </div></div> <p>Which is all there is for connecting this peer, no session initialize, nothing, suggesting that <a href="https://github.com/opendaylight/bgpcep/blob/master/bgp/rib-impl/src/main/java/org/opendaylight/protocol/bgp/rib/impl/config/BgpPeer.java#L334" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/bgpcep/blob/master/bgp/rib-impl/src/main/java/org/opendaylight/protocol/bgp/rib/impl/config/BgpPeer.java#L334</a> is false, which seems weird because your neighbor transport config has passiveMode=false. These new logs may bring us more clarity on what exactly is the problem. Thanks</p> <p>Well, this time is working well.</p> <p>I proceed as usual and when adding peer through Rest API, BGP was added and session established. I upload corresponding logs (solved.log), but didn't see anything wrong. Link State topology is correctly retrieved and corresponding Graph correctly fulfill. Don't know if it is due to the new mvn compilation or the log you added. By the way, I setup BGP log to DEBUG before removing default old peers and adding the new one. I let you look to the log in case you found something wrong or some explanations.</p> <p>Regards</p> Relates BGPCEP-958 BGPCEP-992 Development Priority Rank 0|i040lb: