Operating System: All
Platform: All

Attachment odl1_karaf.tar.xz has been added with description: karaf.log

Reopening. Seen in https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-nitrogen/131/odl1_karaf.log.gz

Hi Vivek,
service-binding DS is something used by applications to write/delete service-binding information. If this exception is coming, it has to be debugged from application perspective, probably apps are trying to do a create and delete of service binding from two threads at the same time.
Thanks,
Faseela

Clearer description in the log says :

2017-07-17 07:46:06,355 | WARN | CommitFutures-12 | DataStoreJobCoordinator | 237 - org.opendaylight.genius.mdsalutil-api - 0.3.0.SNAPSHOT | Job: JobEntry

failed
TransactionCommitFailedException{message=Data did not pass validation., errorList=[RpcError [message=Data did not pass validation., severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:genius:interfacemanager:servicebinding?revision=2016-04-06)service-bindings/services-info/services-info[

] does not exist. Cannot apply modification to its children.]]}

Needs to be debugged from ACL side.

snat-conntrack has the issue:

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-snat-conntrack-nitrogen/99/odl1_karaf.log.gz

Not seen in the later runs.

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-ocata-upstream-stateful-snat-conntrack-nitrogen/115/odl1_karaf.log.gz

Please reopen if found again.

Re-opening this as we now see this in carbon sr2. Please see attachments.

ODL VERSION
opendaylight-6.2.0-0.1.20170825rel1943.el7.noarch

Attachment karaf.log.txt.gz has been added with description: karaf log

Attachment odl_info.txt.gz has been added with description: OVS outputs

Attachment karaf_log.txt has been added with description: karaf log

Attachment odl_info.txt has been added with description: OVS outputs

Tim also mentioned that the setup is an all-in-on setup with the ODL on the same node as the compute. The br-ex ip is what is used for the local_ip also.

This exception from Tim's due to ACL being applied on externalInterface incorrectly, is being addressed here: https://git.opendaylight.org/gerrit/62458

https://git.opendaylight.org/gerrit/62458

Adding links to jobs with the issue:

http://logs.openstack.org/05/486905/23/experimental/gate-tripleo-ci-centos-7-scenario008-multinode-oooq-nv/4777aed/logs/undercloud/home/jenkins/overcloud_validate.log.txt.gz

http://logs.openstack.org/05/486905/23/experimental/gate-tripleo-ci-centos-7-scenario008-multinode-oooq-nv/4777aed/logs/subnode-2/var/log/extra/odl/odl_info.txt.gz

http://logs.openstack.org/05/486905/23/experimental/gate-tripleo-ci-centos-7-scenario008-multinode-oooq-nv/4777aed/logs/subnode-2/var/log/extra/odl/karaf.log.txt.gz

Hi Sam,

“Why is this a bad port? This is the patch port on br-int that is going to br-ex. This has to be a valid port right? So that we can push traffic out to the external network? “

ACL service must apply only to Neutron Ports. The port here is a patch-port and not a Neutron port.
And ACL service must not run on such ports.

However the above port is a valid ODL Interface and so interfaceState is available for this port.
Just went through and noticed that ACL Service works on interfaceState as well, thereby creating an ACLInterface entity for something that is not even a Neutron port.

This is a bug in ACL, where we should fix such that we consider only NeutronPorts on InterfaceState (or) InterfaceConfig events and only for them we must construct an ACLInterface, which will be later used for Bind/Unbind logic.

I will try to raise a patch tomorrow.

Here is the exception from Tims latest attachment:

2017-08-28 23:36:32,405 | INFO | nPool-1-worker-3 | VpnInterfaceManager | 360 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.2.Carbon | VPN Interface add event - intfName 110942673897472:br-ex-patch:trunk onto vpnName 108df187-a590-4c50-a599-a476792966cd running config-driven
2017-08-28 23:36:32,406 | INFO | nPool-1-worker-3 | VpnInterfaceManager | 360 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.2.Carbon | Binding vpn service to interface 110942673897472:br-ex-patch:trunk
2017-08-28 23:36:32,414 | INFO | eChangeHandler-0 | VpnOpStatusListener | 360 - org.opendaylight.netvirt.vpnmanager-impl - 0.4.2.Carbon | update: Processing update for vpn 108df187-a590-4c50-a599-a476792966cd with rd 108df187-a590-4c50-a599-a476792966cd
2017-08-28 23:36:32,424 | WARN | nPool-1-worker-3 | VrfEntryListener | 362 - org.opendaylight.netvirt.fibmanager-impl - 0.4.2.Carbon | VRF Table not yet available for RD 108df187-a590-4c50-a599-a476792966cd
2017-08-28 23:36:32,437 | ERROR | pool-16-thread-1 | AbstractAclServiceImpl | 370 - org.opendaylight.netvirt.aclservice-impl - 0.4.2.Carbon | port and port security groups cannot be null
2017-08-28 23:36:32,438 | WARN | pool-16-thread-1 | AclServiceManagerImpl | 370 - org.opendaylight.netvirt.aclservice-impl - 0.4.2.Carbon | Acl action BIND invoking listener org.opendaylight.netvirt.aclservice.StatefulIngressAclServiceImpl failed
2017-08-28 23:36:32,438 | ERROR | pool-16-thread-1 | AbstractAclServiceImpl | 370 - org.opendaylight.netvirt.aclservice-impl - 0.4.2.Carbon | port and port security groups cannot be null
2017-08-28 23:36:32,438 | WARN | pool-16-thread-1 | AclServiceManagerImpl | 370 - org.opendaylight.netvirt.aclservice-impl - 0.4.2.Carbon | Acl action BIND invoking listener org.opendaylight.netvirt.aclservice.StatefulEgressAclServiceImpl failed
2017-08-28 23:36:32,542 | WARN | rd-dispatcher-56 | ShardDataTree | 211 - org.opendaylight.controller.sal-distributed-datastore - 1.5.2.Carbon | member-1-shard-default-config: Store Tx member-1-datastore-config-fe-0-txn-780-0: Data validation failed for path /(urn:opendaylight:genius:interfacemanager:servicebinding?revision=2016-04-06)service-bindings/services-info/services-info[

].
org.opendaylight.yangtools.yang.data.api.schema.tree.ModifiedNodeDoesNotExistException: Node /(urn:opendaylight:genius:interfacemanager:servicebinding?revision=2016-04-06)service-bindings/services-info/services-info[

] does not exist. Cannot apply modification to its children.
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkTouchApplicable(AbstractNodeContainerModificationStrategy.java:281)[108:org.opendaylight.yangtools.yang-data-impl:1.1.2.Carbon]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.checkApplicable(SchemaAwareApplyOperation.java:125)[108:org.opendaylight.yangtools.yang-data-impl:1.1.2.Carbon]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.checkChildPreconditions(AbstractNodeContainerModificationStrategy.java:305)[108:org.opendaylight.yangtools.yang-data-impl:1.1.2.Carbon]

AclService do check if port security is enabled for a port. So the expectation is that port security should be disabled for the ports which is not of Aclinterest. So this should be either a case where enable port security is set to truer a missing a check for the parameter in the acl service code flow.

This exception below from ACL does not cause dataplane problems at FIP.
From the logs, I could see that the external-network VPN is correctly bound to this
same interface, just after the exceptions J from ACL.

The patch was raised by me to have ACL not service these external-facing interfaces.

I believe in our upstream CI, there is always a VXLAN TEP available on all our compute-nodes.
Actually that VXLAN TEP is not at all required for the FIP use-case over FLAT/VLAN networks, but
it looks like the FIP code just ‘peeps for it’ J !

So can you please check (or configure) some VXLAN TEP on that hypervisor where the FIP VM resides,
and then retry the FIP use-case ?

The above observation of mine is based on the following errors seen in Tim’s logs:

2017-08-28 23:37:08,945 | ERROR | nPool-1-worker-3 | VpnFloatingIpHandler | 369 - org.opendaylight.netvirt.natservice-impl - 0.4.2.Carbon | onAddFloatingIp: Unable to retrieve nextHopIp for DPN 110942673897472 to handle floatingIp 192.168.24.104

2017-08-28 23:37:09,087 | ERROR | eChangeHandler-0 | NatUtil | 369 - org.opendaylight.netvirt.natservice-impl - 0.4.2.Carbon | getVpnForRouter : VPN not found for routerID:372363c1-efe6-4348-9e42-093357091509

Hi All,

Problematic Area: (RCA for FIP ping failure)
====================================

VpnFloatingIpHandler à This class will be taking care of installing following DNAT flows on corresponding DPN.

INTERNAL_TUNNEL_TABLE (36) -> PDNAT_TABLE (25) {Single Node DPN, this flow is not required)
L3_LFIB_TABLE (20) -> PDNAT_TABLE (25) {For FLAT/VLAN external provider type this flow is not required. Since no MPLS label was used)
L3_FIB_TABLE (21) -> PDNAT_TABLE (25)

The below code snippet was executed before installing local FIP flow entry

installation issue.

Please let me know if any further comments on doing this proposed code change.

Thanks & Regards,
Karthikeyan.

Karthikeyan, Vivek,

can we tell what is different in the OOO CI that causes this issue? The upstream CI should be about the same here since it uses the autotunnels and tunnels are not created before hand - they are created as the vms are created.

Is it because the OOO is using an all-in-one node where ODL, Os controller and compute are all together so that tunnels would not be created?

This is not about tunnels. It is about presence of a VTEP configuration in ITM for the DPN identified by 110942673897472.

If it is a single node setup, then it is possible that the VTEP creation is never triggered on that setup as part of booting VMs. VTEPs available in ODL and can be viewed by GET of ITM URL /transport-zones.

https://git.opendaylight.org/gerrit/62492

https://git.opendaylight.org/gerrit/#/c/62492