https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/NEUTRON-159 neutron <p>as raised on <a href="https://lists.opendaylight.org/pipermail/neutron-dev/2018-March/001633.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/neutron-dev/2018-March/001633.html</a>, and seen again e.g. in <a href="https://jenkins.opendaylight.org/releng/job/neutron-maven-verify-fluorine-mvn33-openjdk8/25/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/neutron-maven-verify-fluorine-mvn33-openjdk8/25/console</a> for (unrelated) minor change <a href="https://git.opendaylight.org/gerrit/#/c/70354/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70354/</a>, we're occassionally but regularly enough hitting these weird but recurring NeutronNetworkJAXBTest &amp; NeutronFirewallJAXBTest failures:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Failed tests: NeutronFirewallJAXBTest.test_NeutronFirewallPolicy_JAXB:61 NeutronFirewallPolicy JAXB Test 2: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronFirewallJAXBTest.test_NeutronFirewall_JAXB:31 NeutronFirewall JAXB Test 2: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronFloatingIpJAXBTest.test_NeutronFloatingIp_JAXB:34 NeutronFloatingIp JAXB Test 2: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronLoadBalancerHealthMonitorJAXBTest.test_NeutronLoadBalancerHealthMonitor_JAXB:58 NeutronLoadBalancerHealthMonitor JAXB Test 10: Testing tenant_id failed expected:&lt;00045a7b-796b-4f26-9cf9-9e82d248fda7&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronLoadBalancerJAXBTest.test_NeutronLoadBalancer_JAXB:48 NeutronLoadBalancer JAXB Test 8: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronLoadBalancerListenerJAXBTest.test_NeutronLoadBalancerListener_JAXB:62 NeutronLoadBalancerListener JAXB Test 9: Testing tenant_id failed expected:&lt;11145a7b-796b-4f26-9cf9-9e82d248fda7&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronLoadBalancerPoolJAXBTest.test_NeutronLoadBalancerPool_JAXB:47 NeutronLoadBalancerPool JAXB Test 7: Testing Tenant_id failed expected:&lt;1a3e005cf9ce40308c900bcb08e5320c&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronLoadBalancerPoolMemberJAXBTest.test_NeutronLoadBalancerPoolMember_JAXB:47 NeutronLoadBalancerPoolMember JAXB Test 7: Testing tenant_id failed expected:&lt;00045a7b-796b-4f26-9cf9-9e82d248fda7&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronMeteringLabelJAXBTest.test_NeutronMeteringLabel_JAXB:34 NeutronMeteringLabel JAXB Test 4: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronNetworkJAXBTest.test_NeutronNetwork_MultipleProvider_JAXB:79 NeutronNetwork JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronNetworkJAXBTest.test_NeutronNetwork_SingleProvider_JAXB:35 NeutronNetwork JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronNetworkQosJAXBTest.test_NeutronNetworkQos_JAXB:33 NeutronNetwork JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronPortJAXBTest.test_NeutronPort_JAXB:39 NeutronPort JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronPortQosJAXBTest.test_PortQosEnabled_JAXB:42 NeutronPort JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronPortSecurityJAXBTest.test_NeutronPortSecurityDefault_JAXB:77-&gt;test_PortSecurityEnabled_JAXB:87 NeutronPort JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronPortSecurityJAXBTest.test_NeutronPortSecurityDisabled_JAXB:68-&gt;test_PortSecurityEnabled_JAXB:87 NeutronPort JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronPortSecurityJAXBTest.test_NeutronPortSecurityEnabled_JAXB:63-&gt;test_PortSecurityEnabled_JAXB:87 NeutronPort JAXB Test 2: Testing tenant_id failed expected:&lt;9bacb3c5d39d41a79512987f338cf177&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronQosJAXBTest.test_NeutronQosPolicy_JAXB:40 NeutronQosPolicy JAXB Test 2: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronRouterJAXBTest.test_NeutronRouter_JAXB:45 NeutronFloatingIp JAXB Test 5: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSFCFlowClassifierJAXBTest.test_NeutronSFCFlowClassifier_JAXB:39 NeutronSFCFlowClassifier JAXB Test 2: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSFCPortChainJAXBTest.test_NeutronSFCPortChain_JAXB:38 NeutronSFCPortChain JAXB Test 2: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSFCPortPairGroupJAXBTest.test_NeutronSFCPortPairGroup_JAXB:32 NeutronSFCPortPairGroup JAXB Test 2: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSFCPortPairJAXBTest.test_NeutronSFCPortPair_JAXB:35 NeutronSFCPortPair JAXB Test 2: Testing tenant_id failed expected:&lt;4969c491a3c74ee4af974e6d800c62de&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSecurityGroupJAXBTest.test_NeutronSecurityGroup_JAXB:39 NeutronSecurityGroup JAXB Test 4: Testing port range min failed expected:&lt;b4f50856753b4dc6afee5fa6b9b6c550&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSecurityRuleJAXBTest.test_NeutronSecurityRule_JAXB:74 NeutronSecurityRule JAXB Test 10: Testing tenant id failed expected:&lt;e4f50856753b4dc6afee5fa6b9b6c550&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronSubnetJAXBTest.test_NeutronSubnet_JAXB:47 NeutronSubnet JAXB Test 2: Testing tenant_id failed expected:&lt;379ffe2b9cda498d9e17b319733ec889&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronTapFlowJAXBTest.test_NeutronTapFlow_JAXB:33 NeutronTapFlow JAXB Test 2: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronTapServiceJAXBTest.test_NeutronTapService_JAXB:31 NeutronTapService JAXB Test 2: Testing tenant_id failed expected:&lt;aa902936679e4ea29bfe1158e3450a13&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronTrunkJAXBTest.test_NeutronTrunk_JAXB:43 NeutronTrunk JAXB Test 5: Testing tenant_id failed expected:&lt;cc3641789c8a4304abaa841c64f638d9&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronVpnIkePolicyJAXBTest.test_NeutronVpnIkePolicy_JAXB:33 NeutronVpnIkePolicy JAXB Test 2: Testing tenant id failed expected:&lt;ccb81365fe36411a9011e90491fe1330&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronVpnIpSecPolicyJAXBTest.test_NeutronVpnIPSecPolicy_JAXB:34 NeutronVpnIpSecPolicy JAXB Test 2: Testing tenant id failed expected:&lt;ccb81365fe36411a9011e90491fe1330&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronVpnIpSecSiteConnectionJAXBTest.test_NeutronVpnIPSecSiteConnection_JAXB:39 NeutronVpnIpSecSiteConnection JAXB Test 2: Testing tenant id failed expected:&lt;ccb81365fe36411a9011e90491fe1330&gt; but was:&lt;<span class="code-keyword">null</span>&gt; NeutronVpnServiceJAXBTest.test_NeutronVPNService_JAXB:45 NeutronVpnService JAXB Test 6: Testing Tenant Id failed expected:&lt;ccb81365fe36411a9011e90491fe1330&gt; but was:&lt;<span class="code-keyword">null</span>&gt; Tests in error: NeutronFirewallJAXBTest.test_NeutronFirewallRule_JAXB:89 NullPointer Tests run: 62, Failures: 33, Errors: 1, Skipped: 0</pre> </div></div> NEUTRON-159

Sporadic NeutronNetworkJAXBTest & NeutronFirewallJAXBTest failures

Bug Medium Resolved Done Michael Vorburger Michael Vorburger Tue, 10 Apr 2018 10:55:29 +0000 Wed, 10 Oct 2018 13:37:41 +0000 Wed, 10 Oct 2018 13:37:41 +0000 Oxygen Fluorine Oxygen-SR4 Fluorine-SR1 Neon neutron-spi 0 1 <p>I've run the NeutronNetworkJAXBTest about 20'000 times locally (using the <a href="https://javadocs.opendaylight.org/org.opendaylight.infrautils/fluorine/org/opendaylight/infrautils/testutils/RunUntilFailureRule.html" class="external-link" target="_blank" rel="nofollow noopener">org.opendaylight.infrautils.testutils.RunUntilFailureRule</a>) but cannot reproduce this locally. How can this simple test fail only on Jenkins, only every now and then?</p> <p>This is typically indicative of a concurrency timing issue, but I don't see how this could apply here, this is REALLY weird... because the failing tests are really quite simple, just some trivial looking JAX JSON unmarshalling and assert; so I've had a closer look, more because I'm intrigugined by the mystery, although I guess in theory this could be a real problem at runtime in production as well:</p> <p>The 33 failures in NeutronNetworkJAXBTest are because a NeutronObject getTenantID() is SOMETIMES null - however the asserts on getID() which failing all tests do just before passes. What's so special about this tenantID? It has an if isEmpty() return null check in its getter... are there some known concurrency issues with JAXB where this could cause problems?? I'm going to re-order the asserts in the tests to put tenantID last, and see if ALL other properties did get unmarshalled correctly, whenever this hits us next...</p> <p>The NeutronFirewallJAXBTest.test_NeutronFirewallRule_JAXB:89 failure is an NPE where the JaxbTestHelper.jaxbUnmarshall returns null; so that's a little bit different (entire object, not just 1 property).</p> <p>The JaxbTestHelper has nothing obciously wrong, that I can see. May be close the reader? Cache the JAXBContext?</p> <p>&gt; re-order the asserts in the tests to put tenantID last, and see if ALL<br/> &gt; other properties did get unmarshalled correctly, whenever this hits us next...</p> <p>E.g. in both NeutronLoadBalancerHealthMonitorJAXBTest, NeutronLoadBalancerPoolMemberJAXBTest, NeutronSFCPortPairGroupJAXBTest and NeutronMeteringLabelJAXBTest and more this coindentially was already done like this, so all other properties are asserted on just fine there, only getTenantID() then returns null; this supports the theory that there is some weird issue related specifically to the tenant_id getter... hm. <a href="https://git.opendaylight.org/gerrit/#/c/70706/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/70706/</a> will eventually confirm this for good, but I would say there is a high likelyhood that that custom getter is somehow occassionally causing havoc.</p> <p>On the off chance (probably unlikely, but you never know) that this is some wacky sporadic bug in the EclipseLink Moxy JAXB implementation we use for the JSON processing, let's try to bump it to the lastest in <a href="https://jira.opendaylight.org/browse/NEUTRON-160" title="Bump version of EclipseLink Moxy JAXB implementation used in Neutron from 2.6.2 to 2.7.1" class="issue-link" data-issue-key="NEUTRON-160"><del>NEUTRON-160</del></a>, and see if we are lucky that it will help for this to never hit us anymore.</p> <p>This happened again today, but only on 1 of 5 of my Neutron changes that were built successfully today.</p> <p>But c/70717 with the Moxy version bump is not yet merged; I need that to go in before looking any further.</p> <p>This recently happened again on stable oxygen, twice on <a href="https://jenkins.opendaylight.org/releng/job/autorelease-release-oxygen/343/console" class="external-link" target="_blank" rel="nofollow noopener">autorelease-release-oxygen/343</a> and <a href="https://jenkins.opendaylight.org/releng/job/autorelease-release-oxygen/339/console" class="external-link" target="_blank" rel="nofollow noopener">autorelease-release-oxygen/339</a>. I had A Closer Look look through all <a href="https://jenkins.opendaylight.org/releng/view/autorelease/job/autorelease-release-fluorine/" class="external-link" target="_blank" rel="nofollow noopener">autorelease-release-fluorine</a> for the last 1 month, and it's a datapoint worthwhile noting that it hasn't happened on master anymore.</p> <p>So while strictly speaking this is not conclusive proof of course, it supports the theory (or least doesn't contradict it) that my earlier <del><a href="https://jira.opendaylight.org/browse/NEUTRON-160" title="Bump version of EclipseLink Moxy JAXB implementation used in Neutron from 2.6.2 to 2.7.1" class="issue-link" data-issue-key="NEUTRON-160"><del>NEUTRON-160</del></a></del> work may actually have fixed this; I'll therefore attempt to back-port it from master to stable/oxygen.</p> <p>Closing this issue now, as it's not been seen on master Fluorine in a while (see above), and hoping that <a href="https://jira.opendaylight.org/browse/NEUTRON-160" title="Bump version of EclipseLink Moxy JAXB implementation used in Neutron from 2.6.2 to 2.7.1" class="issue-link" data-issue-key="NEUTRON-160"><del>NEUTRON-160</del></a> via <a href="https://git.opendaylight.org/gerrit/#/c/73782/" class="external-link" target="_blank" rel="nofollow noopener">c/73782</a> on Oxygen does the same trick there. We'll re-open if that wasn't it.</p> <p>Seen again today on stable/oxygen on <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/408/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-oxygen/408/</a> <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/75508/" class="external-link" target="_blank" rel="nofollow noopener">c/75508</a> does another bump of Bump EclipseLink Moxy JAXB impl 2.7.1 → 2.7.3 - if we are exceptionally lucky, that fixes some issue. But what we (someone) should really do here is try to run these occassionally failing tests under infrautils'  <a href="https://javadocs.opendaylight.org/org.opendaylight.infrautils/fluorine/org/opendaylight/infrautils/testutils/RunUntilFailureRule.html" class="external-link" target="_blank" rel="nofollow noopener">RunUntilFailureRule</a> and see if it can reproduced running over night - then debug.</p> <p>Seen again today on neon on <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-neon/34/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/autorelease-release-neon/34/</a> .. I just spent 10' having another closer look. E.g. NeutronTrunkJAXBTest - that getTenantID() really cannot be null, I'm very puzzled.</p> <p>Looked again more into this PITA, I still don't really understand how it could happen, but I have just pushed three changes which may help with this tenant_id empty/null business (which, historically, is due to <a href="https://bugs.opendaylight.org/show_bug.cgi?id=4775" class="external-link" target="_blank" rel="nofollow noopener">https://bugs.opendaylight.org/show_bug.cgi?id=4775</a> and its old <a href="https://git.opendaylight.org/gerrit/#/c/31324" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/31324</a> and <a href="https://git.opendaylight.org/gerrit/#/c/31361/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/31361/</a>) ... let's try to:</p> <ol> <li><a href="https://git.opendaylight.org/gerrit/#/c/75926" class="external-link" target="_blank" rel="nofollow noopener">c/75926</a> put that @XmlElement on the getter &amp; setter instead of the field</li> <li><a href="https://git.opendaylight.org/gerrit/#/c/75927" class="external-link" target="_blank" rel="nofollow noopener">c/75927</a> use an EmptyStringAsNullAdapter (XmlAdapter) instead of if in the getter</li> <li><a href="https://git.opendaylight.org/gerrit/#/c/75928/" class="external-link" target="_blank" rel="nofollow noopener">c/75928</a> make tenantID a private field - and cross fingers that (one of these) helps here...</li> </ol> <p>If we don't see it on master for say 2 weeks, then cherry-pick to stable/fluorine and stable/oxygen.</p> <p>&gt; If we don't see it on master for say 2 weeks, then cherry-pick to stable/fluorine and stable/oxygen.</p> <p>done today</p> Blocks NEUTRON-160 Development Rank 0|i03d1b: