OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[OPNFLWPLUG-1060] instance not able to curl metadata https://jira.opendaylight.org/browse/OPNFLWPLUG-1060 OpenFlowPlugin <p>once in aprox 75 jobs, the apex job we are trying to make stable so it can be a voting<br/> gate saw a failure where one openstack instance did not get it's IP address.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html" class="external-link" target="_blank" rel="nofollow noopener">Failed Job</a></p> OPNFLWPLUG-1060

instance not able to curl metadata

Bug High Resolved Cannot Reproduce Somashekhar Javalagi Jamo Luhrsen apex:gate csit:failures Wed, 12 Dec 2018 00:00:25 +0000 Tue, 19 Mar 2019 09:03:49 +0000 Tue, 19 Mar 2019 09:03:49 +0000 Fluorine-SR2 Neon General openflowplugin-impl 0 7 <p>The job failed both the l2 and external suites for the same reason. In the l2 suite, 6 instances are booted and<br/> all but 5 received their ip address. This is not the case where a single compute node is completely bad like<br/> we've seen before with some main flow missing, or ovs not connected.</p> <p>the karaf.log and neutron server log is attached.</p> <p>This looks like very old dhcp issue where VM couldn't contact metadata service. At the time it was failing to acquire dhcp address and we increased timeout to 3 minutes. Here, it seems to get DHCP IP but something else is missing and not able to contact metadata service <span class="error">[1]</span>. </p> <p>Starting network...<br/> udhcpc (v1.20.1) started<br/> Sending discover...<br/> Sending select for 21.0.0.24...<br/> Lease of 21.0.0.24 obtained, lease time 86400<br/> route: SIOCADDRT: File exists<br/> WARN: failed: route add -net "0.0.0.0/0" gw "21.0.0.1"<br/> cirros-ds 'net' up at 8.07<br/> checking <a href="http://169.254.169.254/2009-04-04/instance-id" class="external-link" target="_blank" rel="nofollow noopener">http://169.254.169.254/2009-04-04/instance-id</a><br/> failed 1/20: up 8.27. request failed<br/> failed 2/20: up 20.71. request failed<br/> failed 3/20: up 33.00. request failed<br/> failed 4/20: up 45.14. request failed<br/> failed 5/20: up 57.38. request failed</p> <p>Route is setup correctly <span class="error">[2]</span></p> <p> default via 192.0.2.1 dev br-ex <br/> 10.9.9.0/24 dev internet_tap proto kernel scope link src 10.9.9.9 <br/> 10.10.10.0/24 dev br-datacentre proto kernel scope link src 10.10.10.250 <br/> 169.254.169.254 via 192.0.2.1 dev br-ex <br/> 172.31.0.0/24 dev docker0 proto kernel scope link src 172.31.0.1 <br/> 192.0.2.0/24 dev br-ex proto kernel scope link src 192.0.2.19 <br/> <span class="error">[heat-admin@overcloud-controller-0 ~]</span>$</p> <p><span class="error">[1]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k1-k13-k2-k2-k7-k1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k1-k13-k2-k2-k7-k1</a></p> <p><span class="error">[2]</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k2-k1-k1-k2-k1-k2-k1-k4-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k2-k1-k1-k2-k1-k2-k1-k4-k5</a></p> <p>some of the .gz files in the logs I've/Vishal has given will be corrupt. Here is a better link to <br/> logs that should be ok:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/562/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/562/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/</a></p> <p>Thanks for poking around <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a>.</p> <p>This is a short job (30-40m) so just pay attention to min:sec<br/> timestamps when trying to correlate between the robot html<br/> logs (which get converted to local time) and all the others.<br/> the VMs in the l2 suite started around 47:00</p> <p>anyway, looking at the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/562/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/controller_sos/sosreport-controllerreport-20181208200152/var/log/containers/neutron/metadata-agent.log.gz" class="external-link" target="_blank" rel="nofollow noopener">meta-data log </a><br/> you can see that the ip from the broken instance never hit<br/> the metadata service. that ip is 21.0.0.24. you can see the<br/> other ips hitting it.</p> <p>in the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/562/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/controller_sos/sosreport-controllerreport-20181208200152/var/log/containers/neutron/dhcp-agent.log.gz" class="external-link" target="_blank" rel="nofollow noopener">dhcp log </a> you can see that 21.0.0.24 was allocated there.</p> <p>nothing stands out to me in the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/562/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/controller_sos/sosreport-controllerreport-20181208200152/var/log/containers/neutron/server.log.gz" class="external-link" target="_blank" rel="nofollow noopener">neutron server log </a> around<br/> that same time frame though.</p> <p>Anyway, I didn't find anything more to go on</p> <p>here is another example of this bug:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/564/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/115" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/564/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/115</a></p> <p>Second job also has exact same issue. Can we log console log for scenario where it does work? May increase logging but will help confirm if the route add error is relevant or not.</p> <p>Issue is VM not able to reach metadata agent. Most likely a routing/arp issue. Can we capture arp table also along with console log, not sure if possible.</p> <p>Also found this bug <a href="https://jira.opnfv.org/browse/APEX-593" class="external-link" target="_blank" rel="nofollow noopener">https://jira.opnfv.org/browse/APEX-593</a> though in this case it goes through but just takes longer. In our case instance-id is never acquired, so suspect some route/arp for metadata is not resolved.</p> <p>Need someone more familiar with metadata and how networking for it works to figure out what is going wrong.</p> <p>After looking through the logs, the metadata agent does not receive the curl from the failed instance. It does however, receive the requests from the instance before and after it on the same network. This looks like an openflow problem, but looking at the flows I don't see an issue. I'm running this in an infinite loop to recreate the same network, subnet, security group, and instance on my machine to see if I can reproduce it.</p> <p>yeah, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a>, that's what I noticed and linked to in an earlier comment about the<br/> meta data server log not seeing the request coming in for that failed instance. I also<br/> didn't see anything stand out in the flow dumps.</p> <p>I can also try to run some extra loops/instances in the csit and let jenkins run<br/> it in a loop. The problem is what debugs to collect if we do hit the failure.</p> <p>and another:<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/567/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/123/robot-plugin/log_full.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/567/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/123/robot-plugin/log_full.html.gz</a></p> <p>I was able to reproduce the problem after 120 iterations. After metadata failed, I was unable to even ping or ssh into the cirros instance (but it did DHCP successfully). The ping request was making it to the instance but the reply was not making it back, and there was no entry in conntrack. Thanks to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=aswins" class="user-hover" rel="aswins">aswins</a> for helping me debug the flows, he noticed that the ping request was actually bypassing conntrack. The ping request should end up in table 220, then piped to 239 to create a conntrack session. However this flow was directly sending it to the VM in table 220:</p> <p>cookie=0x8000007, duration=48738.948s, table=220, n_packets=174, n_bytes=14406, priority=9,reg6=0x20300 actions=output:"tap4386be41-ba"</p> <p>instead of hitting this flow (due to priority):</p> <p>cookie=0x6900000, duration=48738.948s, table=220, n_packets=0, n_bytes=0, priority=6,reg6=0x20300 actions=load:0x90020300->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239</p> <p> </p> <p>Therefore when the packets came back from the VM, conntrack was dropping them because there was no session:</p> <p>#compute: recirc_id(0),in_port(4),eth(src=fa:16:3e:8a:4e:0a,dst=fa:16:3e:58:31:2b),eth_type(0x8100),vlan(vid=1121,pcp=0),encap(eth_type(0x0800),ipv4(src=0.0.0.0/128.0.0.0,dst=0.0.0.0/128.0.0.0,frag=no)), packets:10, bytes:1020, used:0.118s, actions:pop_vlan,5 recirc_id(0),in_port(4),eth(src=fa:16:3e:8a:4e:0a,dst=fa:16:3e:58:31:2b),eth_type(0x8100),vlan(vid=1121,pcp=0),encap(eth_type(0x0806),arp(sip=21.0.0.2,tip=21.0.0.19,op=2/0xff)), packets:0, bytes:0, used:never, actions:pop_vlan,userspace(pid=4294963155,controller(reason=1,dont_send=0,continuation=0,recirc_id=864,rule_cookie=0x822002e,controller_id=0,max_len=65535)),5 recirc_id(0),in_port(5),eth(src=fa:16:3e:58:31:2b,dst=fa:16:3e:8a:4e:0a),eth_type(0x0806),arp(sip=21.0.0.19,tip=21.0.0.2,op=1/0xff,sha=fa:16:3e:58:31:2b), packets:0, bytes:0, used:never, actions:userspace(pid=4286503971,controller(reason=4,dont_send=0,continuation=0,recirc_id=863,rule_cookie=0x822002d,controller_id=0,max_len=65535)),push_vlan(vid=1121,pcp=0),4 recirc_id(0),in_port(5),eth(src=fa:16:3e:58:31:2b),eth_type(0x0800),ipv4(src=21.0.0.19,proto=1,frag=no), packets:10, bytes:980, used:0.118s, actions:ct(zone=5128),recirc(0x35e) recirc_id(0x35e),in_port(5),ct_state(-new-est-rel+inv+trk),ct_mark(0/0x1),eth_type(0x0800),ipv4(frag=no), packets:10, bytes:980, used:0.118s, actions:drop <span class="error">[root@overcloud-novacompute-0 ~]</span># ovs-dpctl show system@ovs-system: lookups: hit:34958088 missed:721437 lost:0 flows: 21 masks: hit:296264755 total:18 hit/pkt:8.30 port 0: ovs-system (internal) port 1: vxlan_sys_4789 (vxlan: packet_type=ptap) port 2: br-int (internal) port 3: br-ex (internal) port 4: eth0 port 5: tap4386be41-ba</p> <p> </p> <p>I then deleted that flow, and I could then ssh into the cirros machine. Then I tried to curl metadata and it worked. I then needed to figure out if the same flow was causing issues for metadata. From there I created a cron job in cirros to curl for metadata every minute. I could see it hitting the metadata agent. I then added the problem flow back, and metadata curls were not coming into the server anymore. Therefore I believe this flow is the same root cause as the ping issue and the reason for this bug.</p> <p>In the CI logs I see that the flow that should go from 220 to 239 is missing, and the problem flow exists in that log (two times?):</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k2-k1-k1-k2-k1-k2-k1-k12-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s1-k2-k1-k1-k2-k1-k2-k1-k12-k5</a></p> <p>cookie=0x8000007, duration=272.526s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xd00,metadata=0x1/0x1 actions=drop cookie=0x8000007, duration=272.522s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xc00,metadata=0x1/0x1 actions=drop cookie=0x8000007, duration=721.965s, table=220, n_packets=118, n_bytes=11146, priority=9,reg6=0x400 actions=output:tuna642e00a5c1 cookie=0x8000007, duration=721.965s, table=220, n_packets=10, n_bytes=448, priority=9,reg6=0x300 actions=output:tun3884a6aa6dc cookie=0x8000007, duration=272.527s, table=220, n_packets=170, n_bytes=14940, priority=9,reg6=0xd00 actions=push_vlan:0x8100,set_field:5217->vlan_vid,output:"br-datacentre-"</p> <p>cookie=0x8000007, duration=272.522s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0xc00 actions=output:"br-datacentre-" cookie=0x6900000, duration=272.470s, table=220, n_packets=107, n_bytes=11218, priority=6,reg6=0xb00 actions=load:0x90000b00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 cookie=0x8000007, duration=272.470s, table=220, n_packets=101, n_bytes=10494, priority=9,reg6=0x90000b00 actions=output:"tapd419615e-50"</p> <p> </p> <p> </p> <p>*cookie=0x8000007, duration=267.979s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90000e00 actions=output:"tap8b015b1e-82"</p> <p>cookie=0x8000007, duration=267.979s, table=220, n_packets=170, n_bytes=13918, priority=9,reg6=0xe00 actions=output:"tap8b015b1e-82"*</p> <p> </p> <p>cookie=0x6900000, duration=248.137s, table=220, n_packets=107, n_bytes=10856, priority=6,reg6=0x1200 actions=load:0x90001200->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239</p> <p>cookie=0x8000007, duration=248.137s, table=220, n_packets=101, n_bytes=10436, priority=9,reg6=0x90001200 actions=output:"tap2559ce3e-48"</p> <p>cookie=0x6900000, duration=721.965s, table=239, n_packets=154, n_bytes=19010, priority=100,ip actions=ct_clear,goto_table:240 cookie=0x6900000, duration=721.965s, table=239, n_packets=16, n_bytes=1168, priority=100,ipv6 actions=ct_clear,goto_table:240 cookie=0x6900000, duration=721.965s, table=239, n_packets=44, n_bytes=1896, priority=0 actions=goto_table:240</p> <p> </p> <p>I'm not sure why the there is no flow to match on reg6, 0xe00 or 0x90000e00 and goto table 239. Could be timing with when the flow dumps are collected? Either way in my reproduction environment there is a bug. We can fix that and see if the problem goes away in CI. Will attach karaf log. Flow and other info here:</p> <p><a href="https://gist.githubusercontent.com/trozet/8a55951c09a7f207ebaf4a111d9d6697/raw/c15f20e405d464d5a13ae8393dea6e5951d11b5d/broken_metadata_instance" class="external-link" target="_blank" rel="nofollow noopener">https://gist.githubusercontent.com/trozet/8a55951c09a7f207ebaf4a111d9d6697/raw/c15f20e405d464d5a13ae8393dea6e5951d11b5d/broken_metadata_instance</a></p> <p> </p> <p> </p> <p>*cookie=0x8000007, duration=267.979s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90000e00 actions=output:"tap8b015b1e-82"</p> <p>cookie=0x8000007, duration=267.979s, table=220, n_packets=170, n_bytes=13918, priority=9,reg6=0xe00 actions=output:"tap8b015b1e-82"*</p> <p>Here the second flow should not have been there. Instead of that there should be a flow that sends packet to table=239 should be there matching reg6=0xe00.</p> <p>The second flow where it directly sends the packet to output port from table 220 is valid for ports for which SG is disabled. Usually the neutron port create happens in this order.(this is when I debugged an older release hope this is still the case.)</p> <p>1)Port with security group disabled is created. (the second flow here will be programmed.)<br/> 2)Port update follows immediately which enables the port security.( The flow programmed in 1 will be replaced with a flow to send to table 239)</p> <p>In this case the openflowplugin may be writing flows out of sync.</p> <p>ok, looks like we're narrowing in on the problem. Thanks guys. Here are two table=220 flow dumps from<br/> a failed and passing case of our CSIT jobs. Keep in mind there are 3 instances on a vlan network and<br/> 3 on a vxlan network:</p> <p>PASSING CASE:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">153: cookie=0x8000007, duration=122.815s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xc00,metadata=0x1/0x1 actions=drop 154: cookie=0x8000007, duration=122.801s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xd00,metadata=0x1/0x1 actions=drop 155: cookie=0x8000007, duration=582.334s, table=220, n_packets=16, n_bytes=1384, priority=9,reg6=0x300 actions=output:tun3884a6aa6dc 156: cookie=0x8000007, duration=582.334s, table=220, n_packets=117, n_bytes=11104, priority=9,reg6=0x400 actions=output:tuna642e00a5c1 157: cookie=0x8000007, duration=122.815s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0xc00 actions=output:<span class="code-quote">"br-datacentre-"</span> 158: cookie=0x8000007, duration=122.801s, table=220, n_packets=236, n_bytes=22292, priority=9,reg6=0xd00 actions=push_vlan:0x8100,set_field:5217->vlan_vid,output:<span class="code-quote">"br-datacentre-"</span> 159: cookie=0x6900000, duration=122.695s, table=220, n_packets=107, n_bytes=11218, priority=6,reg6=0xb00 actions=load:0x90000b00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 160: cookie=0x8000007, duration=122.688s, table=220, n_packets=101, n_bytes=10494, priority=9,reg6=0x90000b00 actions=output:<span class="code-quote">"tapa92fefa9-ca"</span> 161: cookie=0x8000007, duration=118.411s, table=220, n_packets=101, n_bytes=10494, priority=9,reg6=0x90000e00 actions=output:<span class="code-quote">"tap3c18abac-bd"</span> 162: cookie=0x6900000, duration=118.397s, table=220, n_packets=107, n_bytes=11218, priority=6,reg6=0xe00 actions=load:0x90000e00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 163: cookie=0x8000007, duration=100.575s, table=220, n_packets=100, n_bytes=10394, priority=9,reg6=0x90001200 actions=output:<span class="code-quote">"tap80f684f4-60"</span> 164: cookie=0x6900000, duration=100.575s, table=220, n_packets=106, n_bytes=10814, priority=6,reg6=0x1200 actions=load:0x90001200->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 </pre> </div></div> <p>FAILING CASE:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">153: cookie=0x8000007, duration=263.348s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xc00,metadata=0x1/0x1 actions=drop 154: cookie=0x8000007, duration=263.348s, table=220, n_packets=0, n_bytes=0, priority=10,reg6=0xd00,metadata=0x1/0x1 actions=drop 155: cookie=0x8000007, duration=691.042s, table=220, n_packets=16, n_bytes=1384, priority=9,reg6=0x300 actions=output:tun3884a6aa6dc 156: cookie=0x8000007, duration=691.042s, table=220, n_packets=117, n_bytes=11104, priority=9,reg6=0x400 actions=output:tuna642e00a5c1 157: cookie=0x8000007, duration=263.348s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0xc00 actions=output:<span class="code-quote">"br-datacentre-"</span> 158: cookie=0x8000007, duration=263.348s, table=220, n_packets=150, n_bytes=13842, priority=9,reg6=0xd00 actions=push_vlan:0x8100,set_field:5217->vlan_vid,output:<span class="code-quote">"br-datacentre-"</span> 159: cookie=0x8000007, duration=263.336s, table=220, n_packets=92, n_bytes=10116, priority=9,reg6=0x90000b00 actions=output:<span class="code-quote">"tapde1ea417-33"</span> 160: cookie=0x6900000, duration=263.332s, table=220, n_packets=98, n_bytes=10840, priority=6,reg6=0xb00 actions=load:0x90000b00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 161: cookie=0x8000007, duration=252.629s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90000e00 actions=output:<span class="code-quote">"tap0dd6c327-54"</span> 162: cookie=0x8000007, duration=252.629s, table=220, n_packets=157, n_bytes=12997, priority=9,reg6=0xe00 actions=output:<span class="code-quote">"tap0dd6c327-54"</span> 163: cookie=0x8000007, duration=240.479s, table=220, n_packets=100, n_bytes=10394, priority=9,reg6=0x90001200 actions=output:<span class="code-quote">"tap04bf4472-dd"</span> 164: cookie=0x6900000, duration=240.479s, table=220, n_packets=106, n_bytes=10814, priority=6,reg6=0x1200 actions=load:0x90001200->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239 </pre> </div></div> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/567/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/123/controller_sos/sosreport-controllerreport-20181213042218/opt/opendaylight/data/log/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf.log </a> from the above failing case.</p> <p>Reproduced with TRACE debugging for openflowplugin.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/571/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/5/robot-plugin/" class="external-link" target="_blank" rel="nofollow noopener">job logs</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/571/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/5/controller_sos/sosreport-controllerreport-20181215023219/opt/opendaylight/data/log/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf log</a></p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=aswins" class="user-hover" rel="aswins">aswins</a> Which code is responsible for replacing the flow? Will give an idea for markers in case any exceptions in netvirt code.</p> <p>First the interface will be created with out port security later Acl service will update the binding </p> <p>From the logs order in which OF plugin revives the flow programming request seems to be the issue. In the failing case the the flow mod with updated regex after acl service is bound is received before the default bind.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> Should this be an issue in genius?</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Failing Case</b></div><div class="panelContent"> <p>2018-12-15T01:57:27,014 | DEBUG | nc-exe-26 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.9}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2415926784</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>2018-12-15T01:57:27,024 | DEBUG | nc-exe-30 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">7680</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> </div></div> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Success Case</b></div><div class="panelContent"> <p>2018-12-15T01:47:51,184 | DEBUG | nc-exe-20 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.d2cf928a-1c67-48f3-ba78-490534828239, getHardTimeout=0, getId=Uri{_value=260122599241271.220.d2cf928a-1c67-48f3-ba78-490534828239.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=8}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2816</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>2018-12-15T01:47:51,207 | DEBUG | nc-exe-20 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.d2cf928a-1c67-48f3-ba78-490534828239, getHardTimeout=0, getId=Uri{_value=260122599241271.220.d2cf928a-1c67-48f3-ba78-490534828239.9}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=8}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2415921920</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> </div></div> <p>Could be. We should also look at genius servicebinding logs to see if genius is getting service bindings in correct order or not.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> The flow with output to port is the default binding created by IFM, right?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> : Yes!</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> Looks like <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=aswins" class="user-hover" rel="aswins">aswins</a> had right idea but used wrong log entries to explain point.</p> <p>Issue is the default action with output to port is being added AFTER the action to go to Table239 in failure case.</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>SUCCESS</b></div><div class="panelContent"> <p>2018-12-15T01:47:51,184 | DEBUG | nc-exe-20 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.d2cf928a-1c67-48f3-ba78-490534828239, getHardTimeout=0, getId=Uri{_value=260122599241271.220.d2cf928a-1c67-48f3-ba78-490534828239.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, *<font color="red">getOutputNodeConnector=Uri{_value=8}</font>*, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2816</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T01:47:51,204 | DEBUG | nc-exe-20 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=110100480}, getFlowName=acl.egressacl.d2cf928a-1c67-48f3-ba78-490534828239, getHardTimeout=0, getId=Uri{_value=260122599241271.220.d2cf928a-1c67-48f3-ba78-490534828239.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=WriteMetadataCase{getWriteMetadata=WriteMetadata{getMetadata=0, getMetadataMask=1099511627774, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}, Instruction{getInstruction=GoToTableCase{getGoToTable=<b><font color="red">GoToTable{getTableId=239</font></b>, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentations={}}, getEnd=31, getStart=0, augmentations={}}, getValue=2415921920, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=2, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2816</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=6, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T01:54:15,347 | DEBUG | nc-exe-24 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=110100480}, getFlowName=acl.egressacl.d2cf928a-1c67-48f3-ba78-490534828239, getHardTimeout=0, getId=Uri{_value=260122599241271.220.d2cf928a-1c67-48f3-ba78-490534828239.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=WriteMetadataCase{getWriteMetadata=WriteMetadata{getMetadata=0, getMetadataMask=1099511627774, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}, Instruction{getInstruction=GoToTableCase{getGoToTable=<b><font color="red">GoToTable{getTableId=239</font></b>, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentations={}}, getEnd=31, getStart=0, augmentations={}}, getValue=2684357376, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=2, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">2816</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=6, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> </div></div> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>FAILURE</b></div><div class="panelContent"> <p>2018-12-15T01:57:27,016 | DEBUG | nc-exe-26 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=110100480}, getFlowName=acl.egressacl.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=WriteMetadataCase{getWriteMetadata=WriteMetadata{getMetadata=0, getMetadataMask=1099511627774, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}, Instruction{getInstruction=GoToTableCase{getGoToTable=<b><font color="red">GoToTable{getTableId=239</font></b>, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentations={}}, getEnd=31, getStart=0, augmentations={}}, getValue=2415926784, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=2, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">7680</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=6, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T01:57:27,024 | DEBUG | nc-exe-30 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, *<font color="red">getOutputNodeConnector=Uri{_value=12}</font>*, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">7680</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T02:02:08,667 | DEBUG | nc-exe-31 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, *<font color="red">getOutputNodeConnector=Uri{_value=12}</font>*, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=*<font color="red">7680</font>*, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> </div></div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> : Paste the flow on switch in working and non-working case?</p> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Not Working</b></div><div class="panelContent"> <p> cookie=0x6900000, duration=248.818s, table=220, n_packets=0, n_bytes=0, priority=6,reg6=0x1e00 actions=load:0x90001e00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239<br/> cookie=0x8000007, duration=248.817s, table=220, n_packets=157, n_bytes=12369, priority=9,reg6=0x1e00 actions=output:"tapa992567a-ae"</p> </div></div> <div class="panel" style="border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;"><b>Success</b></div><div class="panelContent"> <p>cookie=0x6900000, duration=266.575s, table=220, n_packets=99, n_bytes=9638, priority=6,reg6=0x1b00 actions=load:0x90001b00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239<br/> cookie=0x8000007, duration=266.576s, table=220, n_packets=95, n_bytes=9058, priority=9,reg6=0x90001b00 actions=output:"tapba660f72-15"</p> </div></div> <p>More details at <span class="error">[1]</span></p> <p><span class="error">[1]</span>https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/571/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/5/robot-plugin/log_02_l2.html.gz#s1-k2-k1-k1</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> Please supply a run with TRACE enabled for org.opendaylight.genius.interfacemanager</p> <p>Trace logs for failed port in 220:</p> <p>2018-12-15T01:57:27,014 | DEBUG | nc-exe-26 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.9}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=2415926784, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T01:57:27,016 | DEBUG | nc-exe-26 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=110100480}, getFlowName=acl.egressacl.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=WriteMetadataCase{getWriteMetadata=WriteMetadata{getMetadata=0, getMetadataMask=1099511627774, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}, Instruction{getInstruction=GoToTableCase{getGoToTable=GoToTable{getTableId=239, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentations={}}, getEnd=31, getStart=0, augmentations={}}, getValue=2415926784, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=2, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=6, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T01:57:27,024 | DEBUG | nc-exe-30 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group<br/> 2018-12-15T02:02:08,667 | DEBUG | nc-exe-31 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>Flows in OVS <span class="error">[compute1]</span>:</p> <p> cookie=0x8000007, duration=248.821s, table=220, n_packets=0, n_bytes=0, priority=9,reg6=0x90001e00 actions=output:"tapa992567a-ae"<br/> cookie=0x6900000, duration=248.818s, table=220, n_packets=0, n_bytes=0, priority=6,reg6=0x1e00 actions=load:0x90001e00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239<br/> cookie=0x8000007, duration=248.817s, table=220, n_packets=157, n_bytes=12369, priority=9,reg6=0x1e00 actions=output:"tapa992567a-ae"</p> <p>I will work on this today. I'll just enable this module for TRACE and let it run. Keep in mind, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, that<br/> this is sort of a hard to reproduce failure, but we will hit it eventually.</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">this job</a> is where I'm trying to get this</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=d.arunprakash%40ericsson.com" class="user-hover" rel="d.arunprakash@ericsson.com">d.arunprakash@ericsson.com</a> : We need your help in debugging this from openflowplugin side.. </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> is checking this.</p> <p>I checked the flow programmed by openflowplugin.</p> <p>For table=220 and reg6=0x1e00, found below flow-mods received by plugin in ovs-vswitchd logs,</p> <p>2018-12-15T01:57:27.289Z|02466|vconn|DBG|tcp:192.0.2.19:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x242): DEL_STRICT table:220 priority=9,reg6=0x1e00 actions=output:12</p> <p>2018-12-15T01:57:27.290Z|02468|vconn|DBG|tcp:192.0.2.19:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x244): ADD table:220 priority=6,reg6=0x1e00 cookie:0x6900000 actions=load:0x90001e00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239</p> <p>2018-12-15T01:57:27.291Z|02471|vconn|DBG|tcp:192.0.2.19:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x247): ADD table:220 priority=9,reg6=0x1e00 cookie:0x8000007 actions=output:12</p> <p>2018-12-15T02:02:08.942Z|03073|vconn|DBG|tcp:192.0.2.19:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x310): DEL_STRICT table:220 priority=6,reg6=0x1e00 actions=load:0x90001e00->NXM_NX_REG6[],write_metadata:0/0xfffffffffe,goto_table:239</p> <p>2018-12-15T02:02:08.942Z|03074|vconn|DBG|tcp:192.0.2.19:6653: received: OFPT_FLOW_MOD (OF1.3) (xid=0x311): ADD table:220 priority=9,reg6=0x1e00 cookie:0x8000007 actions=output:12</p> <p> </p> <p>For the corresponding flow-mods, there are respective changes pushed by the plugin in karaf logs.</p> <p>2018-12-15T01:57:27,016 | DEBUG | nc-exe-26 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=110100480}, getFlowName=acl.egressacl.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=WriteMetadataCase{getWriteMetadata=WriteMetadata{getMetadata=0, getMetadataMask=1099511627774, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}, Instruction{getInstruction=GoToTableCase{getGoToTable=GoToTable{getTableId=239, augmentations={}}, augmentations={}}, getOrder=1, augmentations={}}, Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentations={}}, getEnd=31, getStart=0, augmentations={}}, getValue=2415926784, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=2, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=6, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>2018-12-15T01:57:27,024 | DEBUG | nc-exe-30 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>2018-12-15T02:02:08,667 | DEBUG | nc-exe-31 | FrmUtil | 369 - org.opendaylight.openflowplugin.applications.forwardingrules-manager - 0.8.0.SNAPSHOT | Check if flow Flow{getCookie=FlowCookie{_value=134217735}, getFlowName=default.a992567a-ae6c-4acf-9b6c-fda693ac6f18, getHardTimeout=0, getId=Uri{_value=75753836899984.220.a992567a-ae6c-4acf-9b6c-fda693ac6f18.0}, getIdleTimeout=0, getInstructions=Instructions{getInstruction=[Instruction{getInstruction=ApplyActionsCase{getApplyActions=ApplyActions{getAction=<span class="error">[Action{getAction=OutputActionCase{getOutputAction=OutputAction{getMaxLength=0, getOutputNodeConnector=Uri{_value=12}, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}]</span>, augmentations={}}, augmentations={}}, getOrder=0, augmentations={}}], augmentations={}}, getMatch=Match{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.general.rev140714.GeneralAugMatchNodesNodeTableFlow=GeneralAugMatchNodesNodeTableFlow{getExtensionList=<span class="error">[ExtensionList{getExtension=Extension{augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxAugMatchNodesNodeTableFlow=NxAugMatchNodesNodeTableFlow{getNxmNxReg=NxmNxReg{getReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, getValue=7680, augmentations={}}}}}, getExtensionKey=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowplugin.extension.nicira.match.rev140714.NxmNxReg6Key, augmentations={}}]</span>}}}, getPriority=9, getTableId=220, isBarrier=false, isInstallHw=true, isStrict=true, augmentations={}} is dependent on group</p> <p>These logs are for flow-add events. We don't have any logs in code for flow delete events.</p> <p> </p> <p>so what does that mean now? delete is not pushed to the switch? Did you check the output of config/inventory? we see only two flows there, but there are 3 flows on the switch. What would you need to further debug the issue?</p> <p>We are not logging any message after deleting the flows. The config inventory is captured at the teardown state after which some flows programmed for table 220. </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> can I get config inventory dump after all the tests?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, is the <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/560/jamo-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/9/robot-plugin/log_full.html#s1-s2-k2-k1-k1-k3-k1-k2-k35-k3" class="external-link" target="_blank" rel="nofollow noopener">config/opendaylight-inventory:nodes model dump</a> what you want?</p> <p>I'm having some trouble.</p> <p>1) even though I have org.opendaylight.genius.interfacemanager set to TRACE in the <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/81/controller_sos/sosreport-controllerreport-20181220192553/opt/opendaylight/etc/org.ops4j.pax.logging.cfg.gz" class="external-link" target="_blank" rel="nofollow noopener">logging config file</a><br/> there are no TRACE level logs in the <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/81/controller_sos/sosreport-controllerreport-20181220192553/opt/opendaylight/data/log/karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">karaf.log</a></p> <p>2) I cannot hit this bug right now. this <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-netvirt-1538-netvirt-csit-1node-0cmb-1ctl-2cmp-apex-queens-gate-snat-conntrack-oxygen/" class="external-link" target="_blank" rel="nofollow noopener">sandbox job</a> has run for 80+<br/> times now without hitting it</p> <p> </p> <p>Yes Jamo. What Som is saying is the current teardown dump is captured before the log entry for flows is showing up in the karaf log. So he is suspecting if the teardown was captured at a letr point of time, inventory dump would have had 3 flows</p> <p>Looks like you should set only genius.interfacemanager, the config file now says</p> <p>log4j2.logger.org_opendaylight_org_opendaylight_genius_interfacemanager.name = org.opendaylight.org.opendaylight.genius.interfacemanager which is wrong.</p> <p>We are receiving add flow event with flow of priority 9 and then we are receiving update flow event for flow with priority 9 to replace it with flow with priority 6. So eventually we have to do,</p> <p>ADD event: ADD flow priority = 9</p> <p>UPDATE event : DELETE flow priority = 9</p> <p> ADD flow priority = 6.</p> <p> </p> <p>But the UPDATE event is processed first before ADD event.</p> <p>Which is first trying to DELETE flow which doesn't exist followed by adding both the flows.</p> <p>Though DataTreeChange notification is received in proper order, there is some problem in processing the flow mod events.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> : Do you know how to fix this? Or you need any further information?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a> has confirmed that this is something wrong in openflowplugin, so move the Jira to ofp?</p> <p>I'm not totally following what we want then. The link I gave in my comment is the<br/> model dump in the suite teardown of the final suite the job runs. We can't get<br/> that debug much later than that, right?</p> <p>will do</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a>, Can you please enable below log level and reproduce the same?</p> <p>log:set TRACE org.opendaylight.openflowplugin.applications.frm.nodeconfigurator.NodeConfiguratorImpl</p> <p>To see the queuing order of flow-mod requests?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Somashekhar" class="user-hover" rel="Somashekhar">Somashekhar</a>, I wont have time to look at this right now. But, if someone can make<br/> a job in the sandbox, you can use the jenkins build parameters to set custom<br/> logging like you want. You'll need to run it in a loop (once per hour maybe) and <br/> monitor it for when it might fail.</p> <p>I am keep on trying, but not yet able to reproduce. I will update my analysis for added logs onc I am able to reproduce.</p> <p>Created a sandbox and scheduled it to run every hour for a week, but could not see any failures and hence was not able to reproduce. Can be reopened if it is seen again.</p> Development Rank 0|i03lfb: