[OPNFLWPLUG-917] PacketIn match message deserialization failed for single-layer-serialization Created: 29/Jun/17  Updated: 09/Jul/18  Resolved: 09/Jul/18

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: Nitrogen, Oxygen, Carbon
Fix Version/s: Oxygen, Oxygen-SR2, Fluorine

Type: Bug
Reporter: Tomas Slusny Assignee: Somashekhar Javalagi
Resolution: Done Votes: 0
Labels: csit:exception
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File odl1_karag.log.tar.xz     File odl1_karag.log.tar.xz     JPEG File packet-snapshot.JPG     File tcpdump_port_6653__05_tempest__10.30.170.44.pcap    
Issue Links:
Blocks
blocks OPNFLWPLUG-919 openflowplugin failing on compilation... Resolved
Duplicate
duplicates OPNFLWPLUG-942 Resubmit actions are changed to drop ... Resolved
External issue ID: 8772

 Description   

In logs I observed recurring exception that deserialization of match for PacketIn failed, that is possibly because also PacketIn needs to be converted to single layer.

Full logs: https://logs.opendaylight.org/releng/jenkins092/openflowplugin-csit-1node-gate-flow-services-only-carbon/76/odl1_karaf.log.gz

Exception:
017-06-28 17:47:37,760 | WARN  | entLoopGroup-5-1 | OFDecoder                        | 182 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.1.SNAPSHOT | Message deserialization failed
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly
	at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.util.ListDeserializer.deserializeList(ListDeserializer.java:52)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.util.MatchDeserializer.deserialize(MatchDeserializer.java:53)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.util.MatchDeserializer.deserialize(MatchDeserializer.java:30)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.PacketInMessageFactory.deserialize(PacketInMessageFactory.java:52)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.deserialization.factories.PacketInMessageFactory.deserialize(PacketInMessageFactory.java:31)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:55)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:49)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:26)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)[114:io.netty.codec:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)[112:io.netty.transport:4.1.8.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[114:io.netty.codec:4.1.8.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[114:io.netty.codec:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)[112:io.netty.transport:4.1.8.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)[114:io.netty.codec:4.1.8.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)[114:io.netty.codec:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)[112:io.netty.transport:4.1.8.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)[116:io.netty.handler:4.1.8.Final]
	at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:39)[182:org.opendaylight.openflowjava.openflow-protocol-impl:0.9.1.SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)[112:io.netty.transport:4.1.8.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:1018)[117:io.netty.transport-native-epoll:4.1.8.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:394)[117:io.netty.transport-native-epoll:4.1.8.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:288)[117:io.netty.transport-native-epoll:4.1.8.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)[111:io.netty.common:4.1.8.Final]
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)[111:io.netty.common:4.1.8.Final]
	at java.lang.Thread.run(Thread.java:748)[:1.8.0_131]


 Comments   
Comment by Tomas Slusny [ 29/Jun/17 ]

OpenFlowJava: https://git.opendaylight.org/gerrit/#/c/59674/

Comment by Tomas Slusny [ 29/Jun/17 ]

OpenFlowPlugin: https://git.opendaylight.org/gerrit/#/c/59675/

OpenFlowJava patch needs to be merged first.

Comment by Tomas Slusny [ 30/Jun/17 ]

Actually it looks like the issue was that we are missing implementation for some Nicira match extension with code 33, but having PacketIn deserializer for single-layer is still good.

Comment by Tomas Slusny [ 10/Jul/17 ]

Abandoned carbon patch as this was not bugfix after all, and here is also new gerrit patch that solves another issue in this (alien message received when plugin is collecting statistics):

https://git.opendaylight.org/gerrit/#/c/60128/

Comment by Tomas Slusny [ 13/Jul/17 ]

Patch was merged, closing.

Comment by Sam Hague [ 11/Oct/17 ]

This exception has been popping up the last few weeks.

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/156/odl1_exception.log.gz

2017-10-10 09:13:45,823 | INFO | h for user karaf | command | 364 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test tempest.api.network
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

Comment by Sam Hague [ 11/Oct/17 ]

Attachment odl1_karag.log.tar.xz has been added with description: odl1_karaf.log

Comment by Sam Hague [ 11/Oct/17 ]

Attachment odl1_karag.log.tar.xz has been added with description: odl1_karaf.log

Comment by Sam Hague [ 11/Oct/17 ]

(In reply to Sam Hague from comment #8)
> Created attachment 2080 [details]
> odl1_karaf.log

In this last karaf.log attached, the exception comes out during the tempest.api.network. That is a suite of tests. Down below is a response form Aswin where he thinks the decoder error is for an IPv6 message.

2017-10-10 09:13:45,823 | INFO | h for user karaf | command | 364 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test tempest.api.network

2017-10-10 09:22:42,517 | WARN | entLoopGroup-7-4 | OFDecoder | 269 - org.opendaylight.openflowjava.openflow-protocol-impl - 0.9.2.Carbon | Message deserialization failed
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly

2017-10-10 09:24:31,591 | INFO | h for user karaf | command | 364 - org.apache.karaf.log.command - 3.0.8 | ROBOT MESSAGE: Starting test tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps

Initial thoughts from Aswin:

Sam,

This should be in the openflow plugin and should be an issue de-serializer. Looks like the oxm_code corresponds to IPV6_ND_TLL [1]. So my guess is the issue should be in [2] or a code related to it. Like we are not loading the de-serializer corectly or there is something wrong with the de-serialization key or so.

Thanks
Aswin
[1]https://github.com/opendaylight/openflowplugin/blob/1f719a8b9a0d298c652898ee84492330d2b298c4/openflowjava/openflow-protocol-api/src/main/java/org/opendaylight/openflowjava/protocol/api/util/OxmMatchConstants.java#L92

[2]https://github.com/opendaylight/openflowplugin/blob/1f719a8b9a0d298c652898ee84492330d2b298c4/openflowjava/openflow-protocol-impl/src/main/java/org/opendaylight/openflowjava/protocol/impl/deserialization/match/OxmIpv6NdTllDeserializer.java

Comment by Abhijit Kumbhare [ 16/Oct/17 ]

Sunil,

Any updates on this? Arun will also check with Sunil and update the bug.

Abhijit

Comment by Sam Hague [ 30/Oct/17 ]

Still present: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/171/odl1_karaf.log.gz

Comment by Sunil Kumar M S [ 31/Oct/17 ]

shague Is it seen in carbon only or in nitrogen/master also?

Comment by Sam Hague [ 14/Nov/17 ]

https://git.opendaylight.org/gerrit/64943

Comment by Sam Hague [ 14/Nov/17 ]

carbon: https://git.opendaylight.org/gerrit/64943 or https://git.opendaylight.org/gerrit/65258
nitrogen: https://git.opendaylight.org/gerrit/65260
master: needs patch

Comment by Abhijit Kumbhare [ 27/Nov/17 ]

Any update Sunil?

Comment by Sam Hague [ 05/Dec/17 ]

Still seen on all branches: https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-carbon/214/odl_1/odl1_exception.log.gz and https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-oxygen/482/odl_1/odl1_exception.log.gz

Comment by Anil Vishnoi [ 09/Jan/18 ]

sunilkumarms Any update on this ? Looks like the mentioned patches are abandoned. Are you working on new patch for this issue?

Comment by Sam Hague [ 06/Mar/18 ]

Another version in the oxygen builds.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/191/odl_1/odl1_karaf.log.gz

2018-03-06T05:28:20,871 | INFO  | pool-163-thread-5 | Ipv6PktHandler                   | 353 - org.opendaylight.netvirt.ipv6service-impl - 0.6.0.SNAPSHOT | Received Neighbor Solicitation request
2018-03-06T05:28:45,175 | WARN  | epollEventLoopGroup-9-1 | OFDecoder                        | 395 - org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl - 0.6.0.SNAPSHOT | Message deserialization failed
java.lang.IllegalStateException: Deserializer for key: msgVersion: 4 objectClass: org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.match.entries.grouping.MatchEntry msgType: 1 oxm_field: 33 experimenterID: null was not found - please verify that all needed deserializers ale loaded correctly
	at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializerRegistryImpl.getDeserializer(DeserializerRegistryImpl.java:69) [395:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserializeEntry(MatchDeserializer.java:105) [386:org.opendaylight.openflowplugin.impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:61) [386:org.opendaylight.openflowplugin.impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.protocol.deserialization.match.MatchDeserializer.deserialize(MatchDeserializer.java:33) [386:org.opendaylight.openflowplugin.impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:66) [386:org.opendaylight.openflowplugin.impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowplugin.impl.protocol.deserialization.messages.PacketInMessageDeserializer.deserialize(PacketInMessageDeserializer.java:29) [386:org.opendaylight.openflowplugin.impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.deserialization.DeserializationFactory.deserialize(DeserializationFactory.java:54) [395:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:51) [395:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.SNAPSHOT]
	at org.opendaylight.openflowjava.protocol.impl.core.OFDecoder.decode(OFDecoder.java:27) [395:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.SNAPSHOT]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88) [58:io.netty.codec:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [58:io.netty.codec:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [58:io.netty.codec:4.1.16.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [58:io.netty.codec:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [61:io.netty.handler:4.1.16.Final]
	at org.opendaylight.openflowjava.protocol.impl.core.IdleHandler.channelRead(IdleHandler.java:37) [395:org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl:0.6.0.SNAPSHOT]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [63:io.netty.transport:4.1.16.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:797) [64:io.netty.transport-native-epoll:4.1.16.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) [64:io.netty.transport-native-epoll:4.1.16.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:293) [64:io.netty.transport-native-epoll:4.1.16.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [60:io.netty.common:4.1.16.Final]
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) [60:io.netty.common:4.1.16.Final]
	at java.lang.Thread.run(Thread.java:748) [?:?]
Comment by Arunprakash D [ 07/Mar/18 ]

shague, could you please enable packet capture during this TC run and attach it.

We will have a look into it.

 

Comment by Arunprakash D [ 07/Mar/18 ]

shague, we were able to see the packet capture in the logs, could you please point out the TCs which throwing this error.

Comment by Sam Hague [ 07/Mar/18 ]

The way to find the test is look for the last "ROBOT MESSAGE: Starting" proceeding the error. Every test start is logged, so any exceptions before the next start log are withing that test.

 

*** error came out before the suites started so this is during infra bringup
2018-03-06T05:28:45,175 | WARN  | epollEventLoopGroup-9-1 | OFDecoder 
2018-03-06T05:29:52,950 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/test/csit/suites/openstack/connectivity/l2.robot" | core                             | 124 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/test/csit/suites/openstack/connectivity/l2.robot
2018-03-06T05:30:08,855 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/test/csit/suites/openstack/connectivity/l2.robot" | core                             | 124 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting test Create VLAN Network net_1
2018-03-06T06:11:00,853 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting suite /w/workspace/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/test/csit/suites/openstack/tempest/tempest.robot" | core                             | 124 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting test tempest.api.network
2018-03-06T06:13:51,448 | WARN  | epollEventLoopGroup-9-1 | OFDecoder
2018-03-06T06:14:01,715 | WARN  | epollEventLoopGroup-9-1 | OFDecoder
2018-03-06T06:15:53,098 | INFO  | pipe-log:log "ROBOT MESSAGE: Starting test tempest.scenario.test_aggregates_basic_ops.TestAggregatesBasicOps.test_aggregate_basic_ops" | core                             | 124 - org.apache.karaf.log.core - 4.1.3 | ROBOT MESSAGE: Starting test tempest.scenario.test_aggregates_basic_ops.TestAggregatesBasicOps.test_aggregate_basic_ops
 

 

Comment by Daniel Farrell [ 08/Mar/18 ]

Arunprakash/sunilkumarms, thanks for working on this, friendly reminder that we need to resolve this ASAP to unblock the imminent Oxygen release.

Comment by Daniel Farrell [ 08/Mar/18 ]

sunilkumarms Can you get a patch cherry-picked on to stable/oxygen for this and make it clear which it is?

Comment by Arunprakash D [ 09/Mar/18 ]

shague, i could see the error message is coming after the 05 tempest suite and the TC 05 tempest.tempest.api.network.

I don't see any TCs failing because of this error and paralleling analysing the logs to fix it.

My question here, do we need to keep this as a blocker for oxygen?

Comment by Daniel Farrell [ 12/Mar/18 ]

Avishnoi - Can you help us understand the patches for this? It's a bit confusing, all over various branches.

shague - To check tests, if this is a regression/blocker/still around.

Comment by Sam Hague [ 12/Mar/18 ]

We can lower the severity from blocker, but should still keep it high. The error is still present but it doesn't look to contribute to the current failing tests.

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-oxygen/228/odl_1/odl1_karaf.log.gz

 

 

Comment by Jamo Luhrsen [ 23/Mar/18 ]

Looking for an update on this one. We have it in our netvirt CSIT and we want to start adding checks in our CSIT to fail when exceptions come out. This will allow our gating jobs to really help prevent new troubles from creeping in.

Comment by Sunil Kumar M S [ 23/Mar/18 ]

Hello jluhrsen, I have progressed well enough on this bug, probably today i will push a patch to fix it. 

Comment by Anil Vishnoi [ 02/Apr/18 ]

sunilkumarms can you update the bug with the patch number ?

Comment by Somashekhar Javalagi [ 22/May/18 ]

Hi shague, Can you please capture the wireshark traces from the controller interface side for the test case 5 (05 tempest Test) in csit?

 

Comment by Sam Hague [ 22/May/18 ]

Do the tcpdump's already collected cover what you need: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-upstream-stateful-fluorine/495/control_1/

compute_1 and compute_2 also have the dumps.

Comment by Somashekhar Javalagi [ 23/May/18 ]

From https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-1node-openstack-queens-gate-stateful-oxygen/289/control_1/tcpdump_port_6653__05_tempest__10.30.170.44.pcap.xz capture(tcpdump_port_6653_05_tempest_10.30.170.44.pcap),

I am able to see that there is a table 50 packet in coming with nxm field match of 33. We are currently not having any deserializer for this perticular type of NXM field. We have to either find a deserializer for this or have to see whether the field value is being sent wrong.

 

Comment by Somashekhar Javalagi [ 07/Jun/18 ]

Hi thapar, shague,

From  packet-snapshot.JPG, I am able to see a packet in message with an nxm match field  with value 33.

For which we don't have any deserializer support. Can you please help me in knowing what functionality this match field is intended to serve and which is the application needs to be notified.

 

Comment by Anil Vishnoi [ 25/Jun/18 ]

thapar shague Just to follow-up with the above question, do you have the details som asked in above comment?

Comment by Sam Hague [ 25/Jun/18 ]

I have no idea what the packet it causing the issue. Do the pcaps captured show which packet is the problem?

Comment by Arunprakash D [ 26/Jun/18 ]

shague, som has attached the packet screenshot of the problematic packet-in matches with this JIRA. Please check it.

 

In brief, NXM class 1 with field 33 is coming with the packet-in matches but there is no support in openflowplugin till now, so it causing the deserialization error. Could you please check why this new matches being added for very few packet-in's. Is it expected or added by mistake?

 

 

Comment by Arunprakash D [ 04/Jul/18 ]

shague raised review for NXM_NX_PKT_MARK (33) support in openflowplugin.

https://git.opendaylight.org/gerrit/#/c/73708/

 

Could you please verify if its working?

Generated at Wed Feb 07 20:33:43 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.