OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[NETVIRT-788] CSIT Sporadic failures - ElanService suite - openstack instances failing to get dhcp address for more than 3m https://jira.opendaylight.org/browse/NETVIRT-788 netvirt <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/336/log.html.gz#s1-s6-s1-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/336/log.html.gz#s1-s6-s1-k1-k2</a></p> <p>Operating System: All<br/> Platform: All</p> NETVIRT-788

CSIT Sporadic failures - ElanService suite - openstack instances failing to get dhcp address for more than 3m

Bug Resolved Done Faseela K Jamo Luhrsen Mon, 17 Jul 2017 18:00:56 +0000 Mon, 30 Oct 2017 19:52:46 +0000 Wed, 9 Aug 2017 01:47:04 +0000 Carbon General 0 4 <p>some discussion here:<br/> <a href="https://lists.opendaylight.org/pipermail/netvirt-dev/2017-July/005001.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/netvirt-dev/2017-July/005001.html</a></p> <p>but the gist of this bug is that it's coming periodically and when we hit<br/> it, the openstack instances that are created don't get their dhcp ip address<br/> in at least 3m <span class="error">[0]</span>, which is too long. In a passing case example <span class="error">[0]</span> it still<br/> is taking over 2m which may be considered too long as well. Potentially we<br/> have some performance issue in programming flows in time?</p> <p><span class="error">[0]</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/336/log.html.gz#s1-s6-s1-k1-k2-k3-k12" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/336/log.html.gz#s1-s6-s1-k1-k2-k3-k12</a><br/> <span class="error">[1]</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-learn-carbon/67/log.html.gz#s1-s6-s1-k1-k2-k3-k12" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-learn-carbon/67/log.html.gz#s1-s6-s1-k1-k2-k3-k12</a></p> <p>Details of analysis captured in <span class="error">[2]</span> and this looks like OpenStack issue.</p> <p>tl;dr version: <br/> The test fails coz 'nameserver' is not configured, which comes from Metadata service. VMs get IP fom DHCP, but not ssh keys, nameserver etc. Metadata proxy is not sending information about the VMs. </p> <p><span class="error">[2]</span> <a href="https://lists.opendaylight.org/pipermail/netvirt-dev/2017-July/005001.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/netvirt-dev/2017-July/005001.html</a></p> <p>for reference, this is an email exchange with some debugging info:<br/> -------------------------------------------------------------------</p> <p>Vishal,</p> <p>I might be confused, but I think we had two issues last week, and the Elan Suite<br/> issue is not the metadata issue where we got the dhcp lease but not the nameserver.<br/> I think that was just a nitro issue in a gate job. again, high chance that I'm<br/> just confused <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>but, check this link <span class="error">[i]</span>. That's the console log of a VM in the elan suite after<br/> 3m of polling for it's dhcp lease. It looks like dhcp just failed.</p> <p>snippet:</p> <p>Starting network... <br/> udhcpc (v1.20.1) started <br/> Sending discover... <br/> Sending discover... <br/> Sending discover... <br/> Usage: /sbin/cirros-dhcpc <up|down> <br/> No lease, failing <br/> WARN: /etc/rc3.d/S40-network failed <br/> cirros-ds 'net' up at 187.58 <br/> checking <a href="http://169.254.169.254/2009-04-04/instance-id" class="external-link" target="_blank" rel="nofollow noopener">http://169.254.169.254/2009-04-04/instance-id</a> <br/> failed 1/20: up 187.79. request failed</p> <p>Also, I spent some time trying to look for clues in the various logs. I didn't<br/> find anything in the q-* logs which includes the metadata log. The karaf log<br/> has some interesting things. I'm comparing with a job that passed this Elan<br/> Suite Setup (#335 whereas <span class="error">[i]</span> is #337). The failing case has a lot of these<br/> retry and 'timed out' logs around ACL rule programming it seems. stuff like<br/> this:</p> <p>LockManager | 318 - org.opendaylight.genius.lockmanager-impl - 0.2.2.SNAPSHOT | Waiting for the lock acl.flow.priorities.pool.154761518883256.243.PERMITTCP_DESTINATION_1_0 is timed out. retrying again</p> <p>LockManager | 318 - org.opendaylight.genius.lockmanager-impl - 0.2.2.SNAPSHOT | Already locked for acl.flow.priorities.pool.154761518883256.243.PERMITTCP_DESTINATION_1_0 after waiting 1000ms, try 12</p> <p>that general pattern happens a few times once the Elan suite starts, and I didn't<br/> see it in the passing case.</p> <p>Also, I saw a bunch of the mdsalutil-api threads getting terminated in a row due to<br/> some NPE, which was not there in the passing case. example:</p> <p>ERROR | eChangeHandler-1 | AsyncDataTreeChangeListenerBase | 292 - org.opendaylight.genius.mdsalutil-api - 0.2.2.SNAPSHOT | Thread terminated due to uncaught exception: AsyncDataTreeChangeListenerBase-DataTreeChangeHandler-1<br/> java.lang.NullPointerException</p> <p>Having said that, I want to point out that the passing case has it's own set<br/> (a lot) of exceptions ERRORS that are occuring during this Elan Suite Setup,<br/> and since it passes it makes me doubt that the errors listed above are very<br/> helpful.</p> <p>I'll pasted this thread in to the BZ in case others are following there.</p> <p>Thanks,<br/> JamO</p> <p><span class="error">[i]</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/337/log.html.gz#s1-s6-s1-k1-k2-k3-k12-k1-k2-k2-k15" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/337/log.html.gz#s1-s6-s1-k1-k2-k3-k12-k1-k2-k2-k15</a></p> <p>On 07/17/2017 10:47 PM, Vishal Thapar wrote:<br/> > Will take a look at it and see if it is perf degradation. I updated <a href="https://jira.opendaylight.org/browse/NETVIRT-788" title="CSIT Sporadic failures - ElanService suite - openstack instances failing to get dhcp address for more than 3m" class="issue-link" data-issue-key="NETVIRT-788"><del>NETVIRT-788</del></a> with analysis.<br/> > <br/> > 8859 is an OpenStack issue as of now, metadata service to be more specific. So would appreciate help from folks more familiar with it as am clueless when it comes to metadata service.<br/> > <br/> > Regards,<br/> > Vishal.<br/> > <br/> > ----<del>Original Message</del>----<br/> > From: Jamo Luhrsen <span class="nobr"><a href="mailto:jluhrsen@gmail.com" class="external-link" target="_blank" rel="nofollow noopener">jluhrsen@gmail.com<sup><img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/></sup></a></span> <br/> > Sent: 18 July 2017 11:04<br/> > To: Vishal Thapar <vishal.thapar@ericsson.com>; netvirt-dev@lists.opendaylight.org<br/> > Subject: Re: <span class="error">[netvirt-dev]</span> netvirt daily 1node CSIT report (7/17)<br/> > <br/> > Vishal,<br/> > <br/> > every job has this <span class="error">[a]</span> which shows runtime for the full job. Also, we can drill down to each test case to see it's duration, like this one <span class="error">[b]</span> that is checking that VMs get IP addresses in our L2 connectivity suite. That's interested, actually, as it seems that around build 814 (5 days ago) that test case had an increase in run time from aprox 1:20 to 1:50 (30s increase). not sure what to make of that...<br/> > <br/> > Thanks,<br/> > JamO<br/> > <br/> > <span class="error">[a]</span><br/> > <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/buildTimeTrend" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/buildTimeTrend</a><br/> > <span class="error">[b]</span><br/> > <a href="https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/839/robot/netvirt-1node-openstack.txt/Connectivity/01%20L2%20Tests/Check%20Vm%20Instances%20Have%20Ip%20Address/durationGraph?maxBuildsToShow=0&hd=true" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/user/jluhrsen/my-views/view/netvirt%20csit/job/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/839/robot/netvirt-1node-openstack.txt/Connectivity/01%20L2%20Tests/Check%20Vm%20Instances%20Have%20Ip%20Address/durationGraph?maxBuildsToShow=0&hd=true</a><br/> > <br/> > On 07/17/2017 10:11 PM, Vishal Thapar wrote:<br/> >> Jamo will do.<br/> >><br/> >> One question, we recently had a few changes to fix CSIT exceptions. I expected a knock-on impact where things move a bit faster but couldn't figure out how. Is it possible to have a dashboard/graph for test runtimes, similar to fail/success one?<br/> >><br/> >> Regards,<br/> >> Vishal.<br/> >><br/> >> ----<del>Original Message</del>----<br/> >> From: netvirt-dev-bounces@lists.opendaylight.org <br/> >> <span class="nobr"><a href="mailto:netvirt-dev-bounces@lists.opendaylight.org" class="external-link" target="_blank" rel="nofollow noopener">netvirt-dev-bounces@lists.opendaylight.org<sup><img class="rendericon" src="https://jira.opendaylight.org/images/icons/mail_small.gif" height="12" width="13" align="absmiddle" alt="" border="0"/></sup></a></span> On Behalf Of Jamo <br/> >> Luhrsen<br/> >> Sent: 17 July 2017 23:41<br/> >> To: netvirt-dev@lists.opendaylight.org<br/> >> Subject: <span class="error">[netvirt-dev]</span> netvirt daily 1node CSIT report (7/17)<br/> >><br/> >> 75% of our jobs passed 100% in their last run, which is as good as I've seen it in a long time.<br/> >><br/> >> for today:<br/> >><br/> >> opened 8859<br/> >> re-opened 7885<br/> >><br/> >><br/> >> - <a href="https://jira.opendaylight.org/browse/NETVIRT-506" title="CSIT Sporadic failures - tempest.scenario.test_port_security_macspoofing_port" class="issue-link" data-issue-key="NETVIRT-506"><del>NETVIRT-506</del></a> - test_port_security_macspoofing_port<br/> >> <span class="error">[0]</span><br/> >><br/> >> - <a href="https://jira.opendaylight.org/browse/NETVIRT-520" title="CSIT Sporadic failures - Flow(s) missing in VPNService suite on compute node" class="issue-link" data-issue-key="NETVIRT-520"><del>NETVIRT-520</del></a> - Flow(s) missing in VPNService suite on compute node<br/> >> <span class="error">[0]</span><br/> >><br/> >> - <a href="https://jira.opendaylight.org/browse/NETVIRT-788" title="CSIT Sporadic failures - ElanService suite - openstack instances failing to get dhcp address for more than 3m" class="issue-link" data-issue-key="NETVIRT-788"><del>NETVIRT-788</del></a> - ElanService suite - openstack instances failing to get dhcp address for more than 3m<br/> >> <span class="error">[1]</span><br/> >> * for this one, it's not coming every time obviously but I am wondering if by fixing<br/> >> it we'll also be fixing other random failures. Even in the passing case it's taking<br/> >> 2+ minutes to get ips, which is irritatingly long (as a user). Maybe something is<br/> >> bogged down internally and if we figure that out other random CSIT issues could go<br/> >> away.<br/> >> @Vishal, I know you did more debugging on this last week. I put a link to one email<br/> >> thread in the bug. Can you update the bug with any other details you know?<br/> >><br/> >> - <a href="https://jira.opendaylight.org/browse/NETVIRT-780" title="CSIT Sporadic failures - v6 instance connectivity failures in vpnservice suite" class="issue-link" data-issue-key="NETVIRT-780"><del>NETVIRT-780</del></a> - v6 instance connectivity failures in vpnservice suite<br/> >> * all <del>learn</del> jobs<br/> >><br/> >> Thanks,<br/> >> JamO<br/> >><br/> >> <span class="error">[0]</span><br/> >> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-ope" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-ope</a><br/> >> nstack-newton-nodl-v2-upstream-learn-carbon/333/log.html.gz<br/> >> <span class="error">[1]</span> <br/> >> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-ope" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-ope</a><br/> >> nstack-newton-upstream-learn-carbon/336/log.html.gz<br/> >> _______________________________________________<br/> >> netvirt-dev mailing list<br/> >> netvirt-dev@lists.opendaylight.org<br/> >> <a href="https://lists.opendaylight.org/mailman/listinfo/netvirt-dev" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/mailman/listinfo/netvirt-dev</a><br/> >></p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-carbon/467/log.html.gz#s1-s6" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-carbon/467/log.html.gz#s1-s6</a></p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/342/log.html.gz#s1-s6-s1-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-carbon/342/log.html.gz#s1-s6-s1-k1-k2</a></p> <p>here is a new example now that we have extra DEBUG logs introduced and<br/> enabled:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/76/log.html.gz#s1-s6" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/76/log.html.gz#s1-s6</a></p> <p>attached is the karaf.log from the start of the ELAN suite, but the above<br/> logs link can be used to get to the full karaf.log if needed</p> <p>Attachment odl1_karaf.log.gz has been added with description: karaf log from the start of the Elan suite</p> <p>(In reply to Jamo Luhrsen from comment #6)<br/> > here is a new example now that we have extra DEBUG logs introduced and<br/> > enabled:<br/> > <br/> > <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-</a><br/> > ocata-upstream-stateful-carbon/76/log.html.gz#s1-s6<br/> > <br/> > attached is the karaf.log from the start of the ELAN suite, but the above<br/> > logs link can be used to get to the full karaf.log if needed</p> <p>Ignore this please. The Elan Suite failed because of a mistake I had<br/> in the code to disable DEBUG logging. I have a patch to fix that.</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/60794/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/60794/</a></p> <p>I think we now have an example to look at after our better debugs have<br/> been enabled in CSIT and the controller.</p> <p>robot log:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/125/log.html.gz#s1-s6-s1-k1-k2" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/125/log.html.gz#s1-s6-s1-k1-k2</a></p> <p>karaf log:</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/125/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/125/odl1_karaf.log.gz</a></p> <p>you can see a lot of those WARNing messages about waiting for locks<br/> and retrying after the Elan Suite starts. If you grep for ROBOT|DEBUG in<br/> the karaf log you will see a bunch of flow removal lines, but only a <br/> single install message in the timeframe of the Elan Suite. I am stil <br/> wondering if we are stuck (locked and retrying) so nothing gets set up.</p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-carbon/478/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-carbon/478/log.html.gz</a></p> <p><a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/130/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/130/log.html.gz</a></p> <p>Fixing wrong table 200 flows with the below patch :</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/61197/6" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/61197/6</a></p> <ul> <li>table 220</li> </ul> <p>There are multiple aspects to this. It can be any of the following, but might not be always due to the same issue, but you can let me know if any of the following symptoms are seen :<br/> Any issues in Table 17 or Table 220 programming. For example last time when Manu debugged, the flows in Table 220 were something like below :<br/> <span class="error">[1]</span>cookie=0x6900000, duration=127.462s, table=220, n_packets=28, n_bytes=2096, priority=6,reg6=0x8200 actions=load:0x70008200->NXM_NX_REG6[],write_metadata:0x138f000000/0xfffffffffe,goto_table:241<br/> <span class="error">[2]</span>cookie=0x8000007, duration=127.523s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90008200 actions=output:42<br/> <span class="error">[3]</span>cookie=0x6900000, duration=126.910s, table=241, n_packets=1, n_bytes=370, priority=63010,udp,reg6=0x8200/0xfffff00,tp_src=67,tp_dst=68 actions=resubmit(,220)<br/> For the same interface-tag -> 8200, the higher priority service is loading service-index=7, but the next priority service is matching for service-index=9, which will never match. Flows in Table 17 and Table 220 works on this service-binding logic, that for the same interface-tag,<br/> There will be “n” entries in the tables where n = no of services bound on the interface. And the highest priority service will set the next priority in service-index before sending the packet to the service pipeline, so that on a resubmit the packet matches the next service-index in Table 17/220.<br/> 2. ConflictingModificationException for urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node where the flow-id has “DPNID:TABLE-ID:INTERFACE-NAME:SERVICE-INDEX” format, where TABLE-ID = 17 or 220<br/> ConflcitingModificationException for urn:opendaylight:genius:bound-services-state</p> <p>There are multiple aspects to this. It can be any of the following, but might not be always due to the same issue, but you can let me know if any of the following symptoms are seen :</p> <p>1. Any issues in Table 17 or Table 220 programming. For example last time when Manu debugged, the flows in Table 220 were something like below :<br/> <span class="error">[1]</span>cookie=0x6900000, duration=127.462s, table=220, n_packets=28, n_bytes=2096, priority=6,reg6=0x8200 actions=load:0x70008200->NXM_NX_REG6[],write_metadata:0x138f000000/0xfffffffffe,goto_table:241<br/> <span class="error">[2]</span>cookie=0x8000007, duration=127.523s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90008200 actions=output:42<br/> <span class="error">[3]</span>cookie=0x6900000, duration=126.910s, table=241, n_packets=1, n_bytes=370, priority=63010,udp,reg6=0x8200/0xfffff00,tp_src=67,tp_dst=68 actions=resubmit(,220)<br/> For the same interface-tag -> 8200, the higher priority service is loading service-index=7, but the next priority service is matching for service-index=9, which will never match. Flows in Table 17 and Table 220 works on this service-binding logic, that for the same interface-tag,<br/> There will be “n” entries in the tables where n = no of services bound on the interface. And the highest priority service will set the next priority in service-index before sending the packet to the service pipeline, so that on a resubmit the packet matches the next service-index in Table 17/220.</p> <p>2. ConflictingModificationException for urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node where the flow-id has “DPNID:TABLE-ID:INTERFACE-NAME:SERVICE-INDEX” format, where TABLE-ID = 17 or 220</p> <p>3. ConflictingModificationException for urn:opendaylight:genius:bound-services-state</p> <p>(In reply to Sam Hague from comment #14)<br/> > There are multiple aspects to this. It can be any of the following, but<br/> > might not be always due to the same issue, but you can let me know if any of<br/> > the following symptoms are seen :<br/> > <br/> > 1. Any issues in Table 17 or Table 220 programming. For example last time<br/> > when Manu debugged, the flows in Table 220 were something like below :<br/> > <span class="error">[1]</span>cookie=0x6900000, duration=127.462s, table=220, n_packets=28,<br/> > n_bytes=2096, priority=6,reg6=0x8200<br/> > actions=load:0x70008200->NXM_NX_REG6[],write_metadata:0x138f000000/<br/> > 0xfffffffffe,goto_table:241<br/> > <span class="error">[2]</span>cookie=0x8000007, duration=127.523s, table=220, n_packets=0, n_bytes=0,<br/> > priority=9,reg6=0x90008200 actions=output:42<br/> > <span class="error">[3]</span>cookie=0x6900000, duration=126.910s, table=241, n_packets=1, n_bytes=370,<br/> > priority=63010,udp,reg6=0x8200/0xfffff00,tp_src=67,tp_dst=68<br/> > actions=resubmit(,220)<br/> > For the same interface-tag -> 8200, the higher priority service is loading<br/> > service-index=7, but the next priority service is matching for<br/> > service-index=9, which will never match. Flows in Table 17 and Table 220<br/> > works on this service-binding logic, that for the same interface-tag,<br/> > There will be “n” entries in the tables where n = no of services bound on<br/> > the interface. And the highest priority service will set the next priority<br/> > in service-index before sending the packet to the service pipeline, so that<br/> > on a resubmit the packet matches the next service-index in Table 17/220.<br/> > <br/> > 2. ConflictingModificationException for<br/> > urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node where the<br/> > flow-id has “DPNID:TABLE-ID:INTERFACE-NAME:SERVICE-INDEX” format, where<br/> > TABLE-ID = 17 or 220<br/> > <br/> > 3. ConflictingModificationException for<br/> > urn:opendaylight:genius:bound-services-state</p> <p>the problem is that currently, we only see the symptom of this bug in automation.<br/> the only check we have to catch this issue or similar issues, is higher up at the<br/> level where VMs don't get IPs. Ideally, we can add checks for all 3 items above in<br/> our CSIT code, but I'm not sure who can take that on. There are still many other<br/> pending priorities for CSIT work.</p> <p>No need to add any more tests, if a test fails, on urther triaging any of the above symptoms are seen, assign it directly to me <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/wink.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> Development External issue ID 8859 External issue URL Rank 0|i01tyn: Status Whiteboard csit:sporadic_failures