https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/INFRAUTILS-18 infrautils <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-verify-oxygen-mvn33-openjdk8/2245/vpnservice/aclservice/impl/target/surefire-reports/org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest-output.txt.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-verify-oxygen-mvn33-openjdk8/2245/vpnservice/aclservice/impl/target/surefire-reports/org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest-output.txt.gz</a> for <a href="https://jenkins.opendaylight.org/releng/job/netvirt-verify-oxygen-mvn33-openjdk8/2245/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/netvirt-verify-oxygen-mvn33-openjdk8/2245/console</a> for <a href="https://git.opendaylight.org/gerrit/#/c/64381/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64381/</a> illustrates a case of AclServiceStatefulTest failing on Jenkins (only), but passing locally, due to some what I suspect to be a timing issue / concurrency bug in JobCoordinator...</p> <p>... basically, this kind of component test failure:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>newInterfaceWithMultipleAcl(org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest) Time elapsed: 30.396 sec &lt;&lt;&lt; ERROR! org.awaitility.core.ConditionTimeoutException: Condition with alias 'TestableJobCoordinatorEventsWaiter' didn't complete within 30 seconds because lambda expression in org.opendaylight.genius.datastoreutils.testutils.TestableJobCoordinatorEventsWaiter: expected &lt;0L&gt; but was &lt;3&gt;. </pre> </div></div> <p>is "just" telling us that the JobCoordinator still has 3 pending jobs that still did not execute, even after the test was waiting for it for 30 seconds. It's possible that Jenkins is slower than local builds, but 30s seems an awfully sufficiently long time for some thread to do do work.. more likely, somehow it got stuck?</p> INFRAUTILS-18

JobCoordinator getIncompleteTaskCount() does not reach 0 in component test after 30s

Bug Highest Verified Done Michael Vorburger Michael Vorburger Fri, 27 Oct 2017 12:30:33 +0000 Thu, 31 May 2018 12:38:19 +0000 Thu, 31 May 2018 12:38:19 +0000 Oxygen General 0 4 <p>Doubt <a href="https://git.opendaylight.org/gerrit/#/c/64685/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64685/</a> will fix this? Merged it anyway as ready now, but probably unrelated.</p> <p>Raised <a href="https://git.opendaylight.org/gerrit/#/c/64806/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64806/</a> and <a href="https://git.opendaylight.org/gerrit/#/c/64805/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64805/</a> to see if we can shed some more light on what exact state it's in when it gets stuck like that...</p> <p><a href="https://git.opendaylight.org/gerrit/64807" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/64807</a> temporarily adds an @Ignore to AclServiceStatefulTest until we figure out what's wrong with JobCoordinator... the idea is that once c/64806/ and c/64805/ are in to give us more debug details when it gets stuck, we'll revert c/64807 and hope to see the failure again on Jenkins, with more details, and then see.</p> <p>I've seen these failures locally as well.</p> <p>I got a failure locally:</p> <p>FAILURE! - in org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest<br/> newInterfaceWithMultipleAcl(org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest) Time elapsed: 120.401 sec &lt;&lt;&lt; ERROR!</p> <p>org.opendaylight.infrautils.testutils.LogCaptureRuleException: Expected no error log, but: Details about stuck JobCoordinator: JobCoordinatorImpl{incompleteTasks=2, pendingTasks=2, failedJobs=0, clearedTasks=228, createdTasks=230, executeAttempts=124, retriesCount=0, fjPool=java.util.concurrent.ForkJoinPool@66cd5f16<span class="error">&#91;Running, parallelism = 4, size = 0, active = 0, running = 0, steals = 58, tasks = 0, submissions = 0&#93;</span>, jobQueueMap={ACL=JobQueue</p> {executing=null, queue=[]}, port1=JobQueue{executing=null, queue=[]} <p>, port2=JobQueue{executing=null, queue=[]}}, jobQueueMapLock=java.util.concurrent.locks.ReentrantLock@315dd1ef<span class="error">&#91;Unlocked&#93;</span>, scheduledExecutorService=java.util.concurrent.ScheduledThreadPoolExecutor@2843960a<span class="error">&#91;Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0&#93;</span>, isJobAvailable=false, jobQueueMapCondition=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@55b87372}</p> <p>org.awaitility.core.ConditionTimeoutException: Condition with alias 'TestableJobCoordinatorEventsWaiter' didn't complete within 120 seconds because lambda expression in org.opendaylight.genius.datastoreutils.testutils.TestableJobCoordinatorEventsWaiter: expected &lt;0L&gt; but was &lt;2&gt;.</p> <p>The queues are empty and FJ pool has nothing - I think the incompleteTasks and pendingTasks counters are wrong. I think the problem is with the static nature of the JobCoordinatorCounters. Since they're enums, there's of course only one instance in the JVM but they're reused per JobCoordinator so a subsequent test may see the residual counts from the prior test's JobCoordinator. This wasn't an issue with the DataStoreJobCoordinator b/c there was a static instance but now each test creates a new JobCoordinator.</p> <p>The JobCoordinatorCounters can't be static - I'll change it and (hopefully) verify it fixes the problem. It failed after 14 runs - I say if it can get to 50 clean then I'll declare victory.</p> <p>Having enums for this seems quite strange — enums aren’t ever supposed to store varying state...</p> <p>Agree - JobCoordinatorCounters should be local to the JobCoordinator instance. I'll push a patch for that - should fix this issue. </p> <p>I made the changes and the AclServiceStatefulTest ran 41 times successfully. One test failed for another reason:</p> <p>FAILURE! - in org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest<br/> newInterfaceWithIcmpAcl(org.opendaylight.netvirt.aclservice.tests.AclServiceStatefulTest) Time elapsed: 1.533 sec &lt;&lt;&lt; FAILURE!<br/> org.junit.ComparisonFailure: Expected and actual beans do not match expected:&lt;...ilter_Egress_10.0.0.[2/32_5000"<br/> flowName = "ACL"<br/> hardTimeOut = 0<br/> idleTimeOut = 0<br/> instructionInfoList = #[<br/> new InstructionWriteMetadata(4bi, 16777214bi),<br/> new InstructionGotoTable(213 as short)<br/> ]<br/> matchInfoList = #[<br/> new MatchMetadata(83886080000bi, 1099494850560bi),<br/> new MatchEthernetType(2048L),<br/> new MatchIpv4Destination(new Ipv4Prefix("10.0.0.2/32"))<br/> ]<br/> ....</p> <p>That looks like a failure in the app code and not the JC. The original failure caused by the JC appears to be fixed.</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/64824/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64824/</a></p> <p>Thanks Tom! Merged c/64824/ (above), and raised new <a href="https://git.opendaylight.org/gerrit/#/c/64827/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64827/</a> to un-revert ignored netvirt aclservice test.</p> <p>PS, clean up in <a href="https://git.opendaylight.org/gerrit/#/c/64869/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64869/</a></p> Development Rank 0|i036xb: