https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-974 OpenFlowPlugin <p> </p> <p>CSIT tests are failing and looks to be from deserialization errors as shown below.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/817/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/817/odl_1/odl1_karaf.log.gz</a></p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>2018-01-22 20:35:19,822 | INFO | nsole user karaf | core | 112 - org.apache.karaf.log.core - 4.0.10 | ROBOT MESSAGE: Starting test Ping External Network PNF from Vm Instance 1 2018-01-22 20:35:21,619 | WARN | entLoopGroup-5-4 | OFDecoder | 360 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.5.2.SNAPSHOT | Message deserialization failed java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 36864 oxm_field: 6 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly 2018-01-22 20:35:22,617 | WARN | entLoopGroup-5-4 | OFDecoder | 360 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.5.2.SNAPSHOT | Message deserialization failed java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 36864 oxm_field: 6 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly 2018-01-22 20:35:23,619 | WARN | entLoopGroup-5-4 | OFDecoder | 360 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.5.2.SNAPSHOT | Message deserialization failed java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 36864 oxm_field: 6 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly </pre> </div></div> <p> </p> OPNFLWPLUG-974

Message deserialization failed...msgType: 36864 oxm_field: 6 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

Bug Highest Resolved Done Gobinath Suganthan Sam Hague csit:exception Tue, 23 Jan 2018 01:13:52 +0000 Tue, 13 Feb 2018 05:37:24 +0000 Tue, 13 Feb 2018 05:37:24 +0000 Nitrogen Carbon Carbon-SR4 Nitrogen-SR2 Oxygen General 0 6 <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> We haven't merge any patch in stable/nitrogen (specially around de/serializer after 8th dec), so were you seeing this issue before 8th dec as well?</p> <p>Not sure... around that time is when the infra changes were happening and most jobs were aborting. We are collecting tcpdump also for this test case and will attach when the job finishes.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> Could you please set <b>debug</b> logs for this file </p> <p><b>org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl</b> too . Also could you share the wireshark capture for that test case (Ping from external PNF to VM) too.</p> <p>Thanks</p> <p>debug logs are at <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/</a>. I goofed up the tcpdump so need to rerun and collect those with the logs.</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/odl_1/odl1_karaf.log.gz</a></p> <p>This job should have the tcpdump and extra debug logs when it finishes in 60m: <a href="https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197</a></p> <p> </p> <p>job with debug and pcaps:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/log_full.html.gz#s1-t17" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/log_full.html.gz#s1-t17</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/odl_1/odl1_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/odl_1/odl1_karaf.log.gz</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/control_1/tcpdump_extnet__01_external_network__10.30.170.163.pcap.xz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/control_1/tcpdump_extnet__01_external_network__10.30.170.163.pcap.xz</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_1/tcpdump_extnet__01_external_network__10.30.170.99.pcap.xz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_1/tcpdump_extnet__01_external_network__10.30.170.99.pcap.xz</a></p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_2/tcpdump_extnet__01_external_network__10.30.170.86.pcap.xz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_2/tcpdump_extnet__01_external_network__10.30.170.86.pcap.xz</a></p> <p>Verified</p> <p>is a job from 12/06/17 where things passed fine. Shortly after that the infra was changed so we don't have a good history, but around 1/2/18 with  <span class="error">&#91;2&#93;</span> we can see the failure reliably.</p> <p><span class="error">&#91;1&#93;</span> <a href="https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/540/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/540/</a></p> <p><span class="error">&#91;2&#93;</span> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/700/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/700/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> I want to run CSIT with some additional logging. The whole CSIT job takes a while to complete (and sometimes the job fails due to timeout). Is there any way to run only a particular TC in the CSIT suite ?</p> <p>Test Case required to be run:</p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>external network.Ping External Network PNF from Vm Instance 1</td> </tr> </tbody></table> </div> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> The deseriialization is failing due to the absence of any matchdeserializer registered for the match of the packetIn message received. This is due to the unknown "msgType" received. The expected value is "0x8000" but is decoded as "0x9000" and hence the matchdeserializer  could not be found for the packetIn. On analysing the attached wireshark logs, I could not see any packet with msgType as "0x9000" and so I guess this must be due some problem while extracting the "msgType" from the packet in OFplugin. </p> <p>We can't run a single test but we can run a single suite that contains that test. On the job config you just set the suite to run in the SUITES parameter. Let me know if you need that as I don't think you can start a job unless you push it to sandbox.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=shague" class="user-hover" rel="shague">shague</a> Thanks to the wireshark logs you had sent earlier , I was able to hard code and reproduce the issue locally.</p> <p><ins>Observations:</ins></p> <ol> <li>The packetIn message which caused the failure is a ICMP packet punted to controller(which is case for ping to external PNF).</li> <li>The deserialization of this message failed due to the reason I already had mentioned above(the msg type of one of the oxm fields was decoded wrongly).</li> <li>The oxm field "Metadata" was getting wrongly deserialied.</li> <li>This was due to the <b>CtMarkCodec</b> present in the packet(before the <b>Metadata</b> oxm field) was decoding the field wrongly. The field doesnt have <b>mask</b> field in it but still the mask it was read, ie, it was reading the next oxm field bytes too(Metadata) thereby skipping the bytes from which the oxm Metadata is supposed to be read and hence wrong data(skipped 4 bytes) is decoded in the next oxm field deserialization.</li> </ol> <p>This issue is observed only in stable/nitrogen because the ct_mark action is not present in the other releases(carbon,oxygen)</p> <p><ins>Nitrogen flows:</ins></p> <p>cookie=0x6900000, duration=327.336s, table=213, n_packets=5, n_bytes=490, priority=1003,ct_state=+trk,icmp,metadata=0xe0000000000/0xfffff0000000000 actions=<b>ct(commit,zone=5002,exec(set_field:0x1-&gt;ct_mark))</b>,resubmit(,17)</p> <p><ins>Oxygen and Carbon flows:</ins></p> <p>cookie=0x6900000, duration=293.840s, table=213, n_packets=15, n_bytes=1110, priority=1002,ct_state=+new+trk,tcp,metadata=0x130000000000/0xfffff0000000000 actions=<b>ct(commit,zone=5002)</b>,resubmit(,17)</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> I have raised a patch for this :</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/67905/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67905/</a></p> <p>Could someone enlighten us which module/project programs this table 213? And why is this change in pipeline present only in Nitrogen? </p> <p>This issue should be the result of <span class="error">&#91;1&#93;</span>, in which a new action, ct_mark, is used by netvirt. Since the change is not yet merged in branches other than Nitrogen, the issue is not observed there.</p> <p><span class="error">&#91;1&#93;</span>https://git.opendaylight.org/gerrit/#/c/66886/</p> <p>What happened is that <span class="error">&#91;1&#93;</span> used my implementation for ct_mark support, which assumed a mask for the match entry (see <span class="error">&#91;2&#93;</span>).</p> <p>As Aswin mentioned, <span class="error">&#91;1&#93;</span> has only been merged in Nitrogen.</p> <p>If Gobinath's patch <span class="error">&#91;3&#93;</span> fixes the deserialization failed errors, then the problematic flows are probably those with match entries like "ct_mark=0x1", not those with the ct_mark set in the conntrack portion of the action (e.g. "actions=ct(commit,zone=5002,exec(set_field:0x1-&gt;ct_mark)").</p> <p><span class="error">&#91;1&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/66886/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/66886/</a></p> <p><span class="error">&#91;2&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/64042/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/64042/</a></p> <p><span class="error">&#91;3&#93;</span> <a href="https://git.opendaylight.org/gerrit/#/c/67905/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/67905/</a></p> <p>Thanks Bertrand for pointing about the flaws.I've added a check to check the mask as Aswin had suggested in the latest patch.</p> Development Rank 0|i039on: