shague We haven't merge any patch in stable/nitrogen (specially around de/serializer after 8th dec), so were you seeing this issue before 8th dec as well?

Not sure... around that time is when the infra changes were happening and most jobs were aborting. We are collecting tcpdump also for this test case and will attach when the job finishes.

shague Could you please set debug logs for this file 

org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl too . Also could you share the wireshark capture for that test case (Ping from external PNF to VM) too.

Thanks

debug logs are at https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/. I goofed up the tcpdump so need to rerun and collect those with the logs.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/196/odl_1/odl1_karaf.log.gz

This job should have the tcpdump and extra debug logs when it finishes in 60m: https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197

job with debug and pcaps:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/log_full.html.gz#s1-t17

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/odl_1/odl1_karaf.log.gz

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/control_1/tcpdump_extnet__01_external_network__10.30.170.163.pcap.xz

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_1/tcpdump_extnet__01_external_network__10.30.170.99.pcap.xz

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-gate-stateful-nitrogen/197/compute_2/tcpdump_extnet__01_external_network__10.30.170.86.pcap.xz

Verified

is a job from 12/06/17 where things passed fine. Shortly after that the infra was changed so we don't have a good history, but around 1/2/18 with  [2] we can see the failure reliably.

[1] https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/540/

[2] https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-ocata-upstream-stateful-nitrogen/700/

shague I want to run CSIT with some additional logging. The whole CSIT job takes a while to complete (and sometimes the job fails due to timeout). Is there any way to run only a particular TC in the CSIT suite ?

Test Case required to be run:

Avishnoi The deseriialization is failing due to the absence of any matchdeserializer registered for the match of the packetIn message received. This is due to the unknown "msgType" received. The expected value is "0x8000" but is decoded as "0x9000" and hence the matchdeserializer  could not be found for the packetIn. On analysing the attached wireshark logs, I could not see any packet with msgType as "0x9000" and so I guess this must be due some problem while extracting the "msgType" from the packet in OFplugin. 

We can't run a single test but we can run a single suite that contains that test. On the job config you just set the suite to run in the SUITES parameter. Let me know if you need that as I don't think you can start a job unless you push it to sandbox.

shague Thanks to the wireshark logs you had sent earlier , I was able to hard code and reproduce the issue locally.

Observations:

1. The packetIn message which caused the failure is a ICMP packet punted to controller(which is case for ping to external PNF).
2. The deserialization of this message failed due to the reason I already had mentioned above(the msg type of one of the oxm fields was decoded wrongly).
3. The oxm field "Metadata" was getting wrongly deserialied.
4. This was due to the CtMarkCodec present in the packet(before the Metadata oxm field) was decoding the field wrongly. The field doesnt have mask field in it but still the mask it was read, ie, it was reading the next oxm field bytes too(Metadata) thereby skipping the bytes from which the oxm Metadata is supposed to be read and hence wrong data(skipped 4 bytes) is decoded in the next oxm field deserialization.

This issue is observed only in stable/nitrogen because the ct_mark action is not present in the other releases(carbon,oxygen)

Nitrogen flows:

cookie=0x6900000, duration=327.336s, table=213, n_packets=5, n_bytes=490, priority=1003,ct_state=+trk,icmp,metadata=0xe0000000000/0xfffff0000000000 actions=ct(commit,zone=5002,exec(set_field:0x1->ct_mark)),resubmit(,17)

Oxygen and Carbon flows:

cookie=0x6900000, duration=293.840s, table=213, n_packets=15, n_bytes=1110, priority=1002,ct_state=+new+trk,tcp,metadata=0x130000000000/0xfffff0000000000 actions=ct(commit,zone=5002),resubmit(,17)

Avishnoi I have raised a patch for this :

https://git.opendaylight.org/gerrit/#/c/67905/

Could someone enlighten us which module/project programs this table 213? And why is this change in pipeline present only in Nitrogen? 

This issue should be the result of [1], in which a new action, ct_mark, is used by netvirt. Since the change is not yet merged in branches other than Nitrogen, the issue is not observed there.

[1]https://git.opendaylight.org/gerrit/#/c/66886/

What happened is that [1] used my implementation for ct_mark support, which assumed a mask for the match entry (see [2]).

As Aswin mentioned, [1] has only been merged in Nitrogen.

If Gobinath's patch [3] fixes the deserialization failed errors, then the problematic flows are probably those with match entries like "ct_mark=0x1", not those with the ct_mark set in the conntrack portion of the action (e.g. "actions=ct(commit,zone=5002,exec(set_field:0x1->ct_mark)").

[1] https://git.opendaylight.org/gerrit/#/c/66886/

[2] https://git.opendaylight.org/gerrit/#/c/64042/

[3] https://git.opendaylight.org/gerrit/#/c/67905/

Thanks Bertrand for pointing about the flaws.I've added a check to check the mask as Aswin had suggested in the latest patch.