[OVSDB-386] OVSDB unable to configure OVS after restart Created: 14/Dec/16  Updated: 30/Oct/17  Resolved: 08/Aug/17

Status: Resolved
Project: ovsdb
Component/s: Southbound.Open_vSwitch
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Guy Sela Assignee: Unassigned
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: Linux
Platform: PC

Attachments: Zip Archive karaf-first.zip     Text File karaf-second.log     Text File karaf-third.log     Text File karaf.log    
External issue ID: 7365

 Description   

Code is synced from stable/boron on 7/12/2016 (December 7th, 2016).

History:
Started with a clustered setup of 3 ODLS.
Stopped all ODLs, cleaned snapshots+journal+data.
Deleted br-int from OVS.
Started ODLs: OVSDB connection created, but br-int is not created and thus there is no openflow connection.

In that point, I "took aside" one of the ODLs to act as single node.
Ran configure_cluster.sh only with this single ODL.
Deleted the 3 managers of the OVS, and configured it to work only with the single ODL.
Stopped the ODL, cleaned all caches, started the ODL, and finished with the same result: OVS isn't configured by the ODL.

The relevant logs lines are:
2016-12-14 13:47:56,383 | INFO | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | OVSDB Connection from /10.0.97.178:41660
2016-12-14 13:47:56,474 | INFO | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | InstanceIdentifier KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd]]]]}

generated for device connection ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,474 | DEBUG | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Entity Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]} created for device connection ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,474 | INFO | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | OVSDB entity Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]} is registered for ownership.
2016-12-14 13:47:56,485 | DEBUG | lt-dispatcher-17 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | handleOwnershipChanged: EntityOwnershipChanged [entity=Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]}, wasOwner=false, isOwner=true, hasOwner=true, inJeopardy=false] event received for device ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,485 | INFO | lt-dispatcher-17 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | handleOwnershipChanged: this southbound plugin instance is an OWNER of the device ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,671 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | createManagerEntries OpenVSwitch: Open_vSwitch : Row [columns={ovs_version=[ovs_version=[2.5.0]], other_config=[other_config=

{local_ip=10.0.97.178, provider_mappings=physnet1:br-physnet1}

], cur_cfg=[cur_cfg=25], bridges=[bridges=[f183027c-f404-4408-b1c0-9d9f389b5b83]], _uuid=[_uuid=2886d2f0-a150-4568-b36f-0bc0a15456bd], external_ids=[external_ids=

{system-id=dba41967-a7b7-4d07-bf62-307439c54d7f}], ssl=[ssl=[]], datapath_types=[datapath_types=[netdev, system]], db_version=[db_version=[7.12.1]], manager_options=[manager_options=[83a2d227-3e3f-4042-9e22-56295f1e95d0]], next_cfg=[next_cfg=25], system_type=[system_type=[CentOS]], system_version=[system_version=[7.2.1511-Core]], iface_types=[iface_types=[patch, vxlan, geneve, internal, stt, tap, system, lisp, gre, ipsec_gre]]}]
, updatedManagerRows: {83a2d227-3e3f-4042-9e22-56295f1e95d0=Manager : Row [columns={is_connected=[is_connected=false], other_config=[other_config={}], connection_mode=[connection_mode=[]], max_backoff=[max_backoff=[]], _uuid=[_uuid=83a2d227-3e3f-4042-9e22-56295f1e95d0], inactivity_probe=[inactivity_probe=[]], external_ids=[external_ids={}], status=[status={}], target=[target=tcp:10.0.77.178]}]}
2016-12-14 13:47:56,675 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | managerEntries: [ManagerEntry [_key=ManagerEntryKey [_target=Uri [_value=tcp:10.0.77.178]], _numberOfConnections=0, _target=Uri [_value=tcp:10.0.77.178], _connected=false, augmentation=[]]]
2016-12-14 13:47:56,675 | DEBUG | n-invoker-impl-0 | OvsdbManagersUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Update Ovsdb Node : Open_vSwitch : Row [columns={ovs_version=[ovs_version=[2.5.0]], other_config=[other_config={local_ip=10.0.97.178, provider_mappings=physnet1:br-physnet1}], cur_cfg=[cur_cfg=25], bridges=[bridges=[f183027c-f404-4408-b1c0-9d9f389b5b83]], _uuid=[_uuid=2886d2f0-a150-4568-b36f-0bc0a15456bd], external_ids=[external_ids={system-id=dba41967-a7b7-4d07-bf62-307439c54d7f}

], ssl=[ssl=[]], datapath_types=[datapath_types=[netdev, system]], db_version=[db_version=[7.12.1]], manager_options=[manager_options=[83a2d227-3e3f-4042-9e22-56295f1e95d0]], next_cfg=[next_cfg=25], system_type=[system_type=[CentOS]], system_version=[system_version=[7.2.1511-Core]], iface_types=[iface_types=[patch, vxlan, geneve, internal, stt, tap, system, lisp, gre, ipsec_gre]]}] with manager entries : [ManagerEntry [_key=ManagerEntryKey [_target=Uri [_value=tcp:10.0.77.178]], _numberOfConnections=0, _target=Uri [_value=tcp:10.0.77.178], _connected=false, augmentation=[]]]
2016-12-14 13:47:56,685 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Update node with bridge node ref ManagedNodeEntry [_bridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-physnet1]]]]}

], _key=ManagedNodeEntryKey [_bridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-physnet1]]]]}

]], augmentation=[]]
2016-12-14 13:47:56,690 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | setOpenFlowNodeRef: updatedControllerRows: {}
2016-12-14 13:47:56,691 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | createControllerEntries Bridge: Bridge : Row [columns={controller=[controller=[]], other_config=[other_config=

{hwaddr=f6:00:00:ff:01:01, disable-in-band=true}

], mirrors=[mirrors=[]], rstp_enable=[rstp_enable=false], sflow=[sflow=[]], flood_vlans=[flood_vlans=[]], _uuid=[_uuid=f183027c-f404-4408-b1c0-9d9f389b5b83], ports=[ports=[1864a2e0-019a-41c5-8d50-619c94978664, 9afd9d41-ab9c-4c81-ae6e-8df711f8f302, f2af1252-cf12-469e-a9e6-354ea57643ee, 23e3eba5-ed69-4b11-8e24-e5fe492ac4d5]], flow_tables=[flow_tables={}], rstp_status=[rstp_status={}], external_ids=[external_ids={}], mcast_snooping_enable=[mcast_snooping_enable=false], netflow=[netflow=[]], ipfix=[ipfix=[]], datapath_version=[datapath_version=<unknown>], name=[name=br-physnet1], auto_attach=[auto_attach=[]], fail_mode=[fail_mode=[]], datapath_id=[datapath_id=[0000f60000ff0101]], stp_enable=[stp_enable=false], datapath_type=[datapath_type=], protocols=[protocols=[]], status=[status={}]}]
, updatedControllerRows: {}
2016-12-14 13:47:56,693 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | controllerEntries: []
2016-12-14 13:47:56,694 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Built with the intent to store bridge data OvsdbBridgeAugmentation [_bridgeName=OvsdbBridgeName [_value=br-physnet1], _bridgeOtherConfigs=[BridgeOtherConfigs [_bridgeOtherConfigKey=hwaddr, _bridgeOtherConfigValue=f6:00:00:ff:01:01, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=hwaddr], augmentation=[]], BridgeOtherConfigs [_bridgeOtherConfigKey=disable-in-band, _bridgeOtherConfigValue=true, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=disable-in-band], augmentation=[]]], _bridgeUuid=Uuid [_value=f183027c-f404-4408-b1c0-9d9f389b5b83], _datapathId=DatapathId [_value=00:00:f6:00:00:ff:01:01], _datapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, _managedBy=OvsdbNodeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd]]]]}

]]
2016-12-14 13:47:56,817 | ERROR | lt-dispatcher-15 | SimpleShardDataTreeCohort | 176 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.SNAPSHOT | Transaction member-1-datastore-operational-fe-0-chn-1-txn-0 failed to prepare
java.lang.IllegalArgumentException: Metadata not available for modification NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-int}

], modificationType=TOUCH, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point, modificationType=MERGE, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

]=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

], modificationType=WRITE, childModification={}]}]}]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)[38:com.google.guava:18.0.0]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:198)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyWrite(AbstractNodeContainerModificationStrategy.java:113)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:210)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.applyTouch(ListEntryModificationStrategy.java:55)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
....

2016-12-14 13:47:58,569 | WARN | n-invoker-impl-0 | SouthboundUtil | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Read Operational/DS for Node failed! KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd]]]]}

ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalStateException: Transaction chain has failed]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:67)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$1.invoke(TransactionProxy.java:93)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionContextWrapper.maybeExecuteTransactionOperation(TransactionContextWrapper.java:92)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.executeRead(TransactionProxy.java:90)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.singleShardRead(TransactionProxy.java:115)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
....

karaf.log with DEBUG on ovsdb attached.

 Comments   
Comment by Guy Sela [ 14/Dec/16 ]

Attachment karaf.log has been added with description: karaf log

Comment by Guy Sela [ 14/Dec/16 ]

Attachment karaf-first.zip has been added with description: When running in cluster - karaf of first ODL

Comment by Guy Sela [ 14/Dec/16 ]

Attachment karaf-second.log has been added with description: When running in cluster - karaf of second ODL

Comment by Guy Sela [ 14/Dec/16 ]

Attachment karaf-third.log has been added with description: When running in cluster - karaf of third ODL

Comment by Guy Sela [ 14/Dec/16 ]

If after del br-int I recreate it manually, everything works fine.

Comment by Anil Vishnoi [ 02/Feb/17 ]

Guy, Is this something you are testing with new netvirt code?

Comment by Guy Sela [ 05/Feb/17 ]

This bug was tested with code from December 7th, 2016.

Comment by Anil Vishnoi [ 11/Mar/17 ]

Hi Guy,

Looking at the logs, it looks like netvirt/genius project is not setting the external-id properly while creating the bridge or termination point. If you look at the log (karaf.log), it shows that there is only one bridge (br-physnet1) present on the node 2886d2f0-a150-4568-b36f-0bc0a15456bd

2016-12-14 13:48:08,560 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | createControllerEntries Bridge: Bridge : Row [columns={controller=[controller=[]], other_config=[other_config=

{hwaddr=f6:00:00:ff:01:01, disable-in-band=true}

], mirrors=[mirrors=[]], rstp_enable=[rstp_enable=false], sflow=[sflow=[]], flood_vlans=[flood_vlans=[]], _uuid=[_uuid=f183027c-f404-4408-b1c0-9d9f389b5b83], ports=[ports=[1864a2e0-019a-41c5-8d50-619c94978664, 9afd9d41-ab9c-4c81-ae6e-8df711f8f302, f2af1252-cf12-469e-a9e6-354ea57643ee, 23e3eba5-ed69-4b11-8e24-e5fe492ac4d5]], flow_tables=[flow_tables={}], rstp_status=[rstp_status={}], external_ids=[external_ids={}], mcast_snooping_enable=[mcast_snooping_enable=false], netflow=[netflow=[]], ipfix=[ipfix=[]], datapath_version=[datapath_version=<unknown>], name=[name=br-physnet1], auto_attach=[auto_attach=[]], fail_mode=[fail_mode=[]], datapath_id=[datapath_id=[0000f60000ff0101]], stp_enable=[stp_enable=false], datapath_type=[datapath_type=], protocols=[protocols=[]], status=[status={}]}]
, updatedControllerRows: {}

2016-12-14 13:48:08,560 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Built with the intent to store bridge data OvsdbBridgeAugmentation [_bridgeName=OvsdbBridgeName [_value=br-physnet1], _bridgeOtherConfigs=[BridgeOtherConfigs [_bridgeOtherConfigKey=hwaddr, _bridgeOtherConfigValue=f6:00:00:ff:01:01, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=hwaddr], augmentation=[]], BridgeOtherConfigs [_bridgeOtherConfigKey=disable-in-band, _bridgeOtherConfigValue=true, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=disable-in-band], augmentation=[]]], _bridgeUuid=Uuid [_value=f183027c-f404-4408-b1c0-9d9f389b5b83], _datapathId=DatapathId [_value=00:00:f6:00:00:ff:01:01], _datapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, _managedBy=OvsdbNodeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd]]]]}

]]

but when it receives the notification from the switch, it's trying to write the (br-physnet1) port on the bridge br-int (ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-int). br-physnet1 is the local port of br-physnet1, so it should be written on the br-physnet1 node.

2016-12-14 13:48:08,566 | ERROR | lt-dispatcher-18 | SimpleShardDataTreeCohort | 176 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.SNAPSHOT | Transaction member-1-datastore-operational-fe-0-chn-7-txn-0 failed to prepare
java.lang.IllegalArgumentException: Metadata not available for modification NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-int}

], modificationType=TOUCH, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point, modificationType=MERGE, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

]=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

], modificationType=WRITE, childModification={}]}]}]

I looked at the clustering logs (karaf-first.log) where all the devices are owned by the first controller. I can see that SouthboundUtil class is creating bridge with wrong external-id

2016-12-14 15:11:27,754 | INFO | on-dispatcher-49 | SouthboundUtils | 330 - org.opendaylight.ovsdb.utils.southbound-utils - 1.3.2.SNAPSHOT | addBridge: node: Node{getNodeId=Uri [_value=ovsdb://uuid/353105c4-fab2-4329-9c6d-5a15a985ed06], augmentations={interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbNodeAugmentation=OvsdbNodeAugmentation{getConnectionInfo=ConnectionInfo{getLocalIp=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.77.212]], getLocalPort=PortNumber [_value=6640], getRemoteIp=IpAddress [_ipv4Address=Ipv4Address [_value=10.0.77.222]], getRemotePort=PortNumber [_value=47594], augmentations={}}, getDatapathTypeEntry=[DatapathTypeEntry{getDatapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeNetdev, augmentations={}}, DatapathTypeEntry{getDatapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, augmentations={}}], getDbVersion=7.12.1, getInterfaceTypeEntry=[InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeIpsecGre, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeSystem, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeInternal, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeGeneve, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeGre, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeLisp, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypePatch, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeStt, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeTap, augmentations={}}, InterfaceTypeEntry{getInterfaceType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.InterfaceTypeVxlan, augmentations={}}], getManagedNodeEntry=[ManagedNodeEntry{getBridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/353105c4-fab2-4329-9c6d-5a15a985ed06/bridge/br-int]]]]}

], augmentations={}}, ManagedNodeEntry{getBridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/353105c4-fab2-4329-9c6d-5a15a985ed06/bridge/br-physnet1]]]]}

], augmentations={}}], getManagerEntry=[ManagerEntry{getNumberOfConnections=0, getTarget=Uri [_value=tcp:10.0.97.220:6640], isConnected=false, augmentations={}}, ManagerEntry{getNumberOfConnections=0, getTarget=Uri [_value=tcp:10.0.77.212:6640], isConnected=false, augmentations={}}, ManagerEntry{getNumberOfConnections=0, getTarget=Uri [_value=tcp:10.0.77.221:6640], isConnected=false, augmentations={}}], getOpenvswitchExternalIds=[OpenvswitchExternalIds{getExternalIdKey=system-id, getExternalIdValue=720d9f1b-f403-4f9a-99f3-477f7b2c1ea9, augmentations={}}], getOpenvswitchOtherConfigs=[OpenvswitchOtherConfigs{getOtherConfigKey=provider_mappings, getOtherConfigValue=physnet1:br-physnet1, augmentations={}}, OpenvswitchOtherConfigs{getOtherConfigKey=local_ip, getOtherConfigValue=10.0.77.222, augmentations={}}], getOvsVersion=2.5.0}}}, bridgeName: br-int, controller(s): [tcp:10.0.97.220:6653, tcp:10.0.77.212:6653, tcp:10.0.77.221:6653]

External-id should use the same uuid as the ovsdb node uuid, but here the ovsdb uuid is (Node{getNodeId=Uri [_value=ovsdb://uuid/353105c4-fab2-4329-9c6d-5a15a985ed06]) is different then external-id getExternalIdValue=720d9f1b-f403-4f9a-99f3-477f7b2c1ea9.

External-id's are used to figure out the relative node/bridge to write the bridge/termination point statistics data to the operational data store. I think this is something that is causing the issue in your setup.

Can you run the setup again and dump the following logs before you restart the controllers

(1) config/operational data (ovsdb:1) before you restart the controller.
(2) ovsdb-client dump <-- it's ovsdb database dump

Also it's good to put debug in the code section where you are using SouthbounUtils. to create bridge (addBridge) and termination point (addTerminationPoint). I believe if you set the correct external-ids for your bridge and termination point, it should resolve the issue.

Comment by Bertrand Low [ 05/Aug/17 ]

Hi Guy,

With the Carbon distribution distribution-karaf-0.6.2-20170803.213544-260, I am unable to reproduce this bug according to the steps you've listed:

1. Started with a clustered setup of 3 ODLS.
2. Stopped all ODLs, cleaned snapshots+journal+data.
3. Deleted br-int from OVS.
4. Started ODLs: OVSDB connection created, but br-int is not created and thus there is no openflow connection.

In my testing, br-int is always recreated after step 4.

I have installed the features: odl-netvirt-openstack, odl-mdsal-clustering.

I have noticed that, in certain situations, the pipeline flows appear to be incorrect (I'll open a bug for this), but as far as br-int goes, it does appear to be recreated reliably.

Are you still able to reproduce this bug or has it been indirectly resolved?

thanks,
Bertrand

Generated at Wed Feb 07 20:36:15 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.