https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1056 OpenFlowPlugin <p>CSIt job and the default table is missing so the suite setup fails.</p> <p> </p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/439/robot-plugin/log_01_l2.html#s1-k1-k1-k9-k2-k3-k1-k1-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-0cmb-1ctl-2cmp-openstack-queens-gate-stateful-neon/439/robot-plugin/log_01_l2.html#s1-k1-k1-k9-k2-k3-k1-k1-k5</a></p> OPNFLWPLUG-1056

Default tables missing

Bug Medium Resolved Done Somashekhar Javalagi Sam Hague csit:failures Mon, 10 Dec 2018 14:11:31 +0000 Tue, 9 Jul 2019 06:04:15 +0000 Tue, 9 Jul 2019 06:04:15 +0000 Neon Sodium 0 6 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> : Why is this a GENIUS JIRA? Tables missing default flows are 18, 60, and 45. Those are netvirt programmed tabled. And if they are already programmed by netvirt, most likely an openflowplugin bug.</p> <p>Thanks, moved to ofp.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> I have added a patch with debug logs, can you please run the csit on the same?</p> <p>We need to figure out which job sees this the most frequent and then try to reproduce it there<br/> with this patch. The job given in the description is a gate job. I've checked the non-gate<br/> job of the same type and this problem hasn't happened in at least the last 30 tries.</p> <p>I'll see what I can figure out, but if anyone else knows please comment.</p> <p>Looks like the 3node cluster jobs see this more frequently than others. This <a href="https://jenkins.opendaylight.org/releng/job/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/" class="external-link" target="_blank" rel="nofollow noopener">tempest one </a><br/> in particular failed because of missing default tables 3 times in the past 30 days (runs once a day).</p> <p>I will create a sandbox job that runs without any test cases, in a loop,<br/> using the distribution from the patch (c/78730) and monitor for any<br/> failures due to missing tables.</p> <p>Here's the <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-ofp-1056-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/" class="external-link" target="_blank" rel="nofollow noopener">sandbox job</a> to try and reproduce this. Note that it<br/> will be purged in two days, so if we don't hit the problem before then I'll<br/> have to recreate.</p> <p>Was able to recreate with the distro created in the <a href="https://git.opendaylight.org/gerrit/#/c/78730" class="external-link" target="_blank" rel="nofollow noopener">debug patch</a></p> <p>here is a link to the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/591/jamo-ofp-1056-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/5/robot-plugin/log_full.html.gz#s1-k1-k1-k9-k2-k3-k1-k1-k4-k2-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">robot failure</a> where you can see table=45 was not found on one of the nodes</p> <p>this is a clustered job so three karaf logs to look at:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/591/jamo-ofp-1056-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/5/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">ODL 1</a><br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/591/jamo-ofp-1056-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/5/odl_2/odl2_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">ODL 2</a><br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/591/jamo-ofp-1056-netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-tempest-stateful-snat-conntrack-neon/5/odl_3/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">ODL 3</a></p> <p>The node with the missing table=45 was the first compute node, and it's ovsdb UUID was 46c9d66c-60a6-4da3-8b58-2c4831689600. I <b>think</b><br/> the ODL that was owning and writing to it was ODL 2, just based off of grepping for that UUID in each karaf.log and seeing things like addPatchPort, etc<br/> in ODL 2 and not in the others.</p> <p>DeviceContext is writing node information to the oper inventory.</p> <p>Rolecontext is responsible to device's mastership election and ownership change callback.</p> <p> </p> <p>FRM registers for ownership callback and will get notified once master is elected for a device. In some cases, Rolecontext is taking time for ownership election but deviceContext is going ahead and writing the node information to the oper inventory. So, apps which listen for node DTCL will go ahead and push default table flows which might be dropped by FRM as it has not yet got ownership callback.</p> <p>The new implementation would be for devicecontext to wait for mastership election to go through and then write the switch information to the oper inventory. This will make sure FRM always has the mastership details when it receives flow information.</p> Development Rank 0|i03ldj: