[OVSDB-386] OVSDB unable to configure OVS after restart

Mon, 30 Oct 2017 15:38:58 +0000

Code is synced from stable/boron on 7/12/2016 (December 7th, 2016).

History:
Started with a clustered setup of 3 ODLS.
Stopped all ODLs, cleaned snapshots+journal+data.
Deleted br-int from OVS.
Started ODLs: OVSDB connection created, but br-int is not created and thus there is no openflow connection.

In that point, I "took aside" one of the ODLs to act as single node.
Ran configure_cluster.sh only with this single ODL.
Deleted the 3 managers of the OVS, and configured it to work only with the single ODL.
Stopped the ODL, cleaned all caches, started the ODL, and finished with the same result: OVS isn't configured by the ODL.

{targetType=interface org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd]]]]}

generated for device connection ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,474 | DEBUG | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Entity Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]} created for device connection ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,474 | INFO | DBConnNotifSer-2 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | OVSDB entity Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]} is registered for ownership.
2016-12-14 13:47:56,485 | DEBUG | lt-dispatcher-17 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | handleOwnershipChanged: EntityOwnershipChanged [entity=Entity{type='ovsdb', id=/(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

]}, wasOwner=false, isOwner=true, hasOwner=true, inJeopardy=false] event received for device ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,485 | INFO | lt-dispatcher-17 | OvsdbConnectionManager | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | handleOwnershipChanged: this southbound plugin instance is an OWNER of the device ConnectionInfo [Remote-address=10.0.97.178, Remote-port=41660, Local-address10.0.77.178, Local-port=6640, type=PASSIVE]
2016-12-14 13:47:56,671 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | createManagerEntries OpenVSwitch: Open_vSwitch : Row [columns={ovs_version=[ovs_version=[2.5.0]], other_config=[other_config=

{local_ip=10.0.97.178, provider_mappings=physnet1:br-physnet1}

], cur_cfg=[cur_cfg=25], bridges=[bridges=[f183027c-f404-4408-b1c0-9d9f389b5b83]], _uuid=[_uuid=2886d2f0-a150-4568-b36f-0bc0a15456bd], external_ids=[external_ids=

{system-id=dba41967-a7b7-4d07-bf62-307439c54d7f}], ssl=[ssl=[]], datapath_types=[datapath_types=[netdev, system]], db_version=[db_version=[7.12.1]], manager_options=[manager_options=[83a2d227-3e3f-4042-9e22-56295f1e95d0]], next_cfg=[next_cfg=25], system_type=[system_type=[CentOS]], system_version=[system_version=[7.2.1511-Core]], iface_types=[iface_types=[patch, vxlan, geneve, internal, stt, tap, system, lisp, gre, ipsec_gre]]}]
, updatedManagerRows: {83a2d227-3e3f-4042-9e22-56295f1e95d0=Manager : Row [columns={is_connected=[is_connected=false], other_config=[other_config={}], connection_mode=[connection_mode=[]], max_backoff=[max_backoff=[]], _uuid=[_uuid=83a2d227-3e3f-4042-9e22-56295f1e95d0], inactivity_probe=[inactivity_probe=[]], external_ids=[external_ids={}], status=[status={}], target=[target=tcp:10.0.77.178]}]}
2016-12-14 13:47:56,675 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | managerEntries: [ManagerEntry [_key=ManagerEntryKey [_target=Uri [_value=tcp:10.0.77.178]], _numberOfConnections=0, _target=Uri [_value=tcp:10.0.77.178], _connected=false, augmentation=[]]]
2016-12-14 13:47:56,675 | DEBUG | n-invoker-impl-0 | OvsdbManagersUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Update Ovsdb Node : Open_vSwitch : Row [columns={ovs_version=[ovs_version=[2.5.0]], other_config=[other_config={local_ip=10.0.97.178, provider_mappings=physnet1:br-physnet1}], cur_cfg=[cur_cfg=25], bridges=[bridges=[f183027c-f404-4408-b1c0-9d9f389b5b83]], _uuid=[_uuid=2886d2f0-a150-4568-b36f-0bc0a15456bd], external_ids=[external_ids={system-id=dba41967-a7b7-4d07-bf62-307439c54d7f}

], ssl=[ssl=[]], datapath_types=[datapath_types=[netdev, system]], db_version=[db_version=[7.12.1]], manager_options=[manager_options=[83a2d227-3e3f-4042-9e22-56295f1e95d0]], next_cfg=[next_cfg=25], system_type=[system_type=[CentOS]], system_version=[system_version=[7.2.1511-Core]], iface_types=[iface_types=[patch, vxlan, geneve, internal, stt, tap, system, lisp, gre, ipsec_gre]]}] with manager entries : [ManagerEntry [_key=ManagerEntryKey [_target=Uri [_value=tcp:10.0.77.178]], _numberOfConnections=0, _target=Uri [_value=tcp:10.0.77.178], _connected=false, augmentation=[]]]
2016-12-14 13:47:56,685 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Update node with bridge node ref ManagedNodeEntry [_bridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

], _key=ManagedNodeEntryKey [_bridgeRef=OvsdbBridgeRef [_value=KeyedInstanceIdentifier

]], augmentation=[]]
2016-12-14 13:47:56,690 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | setOpenFlowNodeRef: updatedControllerRows: {}
2016-12-14 13:47:56,691 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | createControllerEntries Bridge: Bridge : Row [columns={controller=[controller=[]], other_config=[other_config=

{hwaddr=f6:00:00:ff:01:01, disable-in-band=true}

], mirrors=[mirrors=[]], rstp_enable=[rstp_enable=false], sflow=[sflow=[]], flood_vlans=[flood_vlans=[]], _uuid=[_uuid=f183027c-f404-4408-b1c0-9d9f389b5b83], ports=[ports=[1864a2e0-019a-41c5-8d50-619c94978664, 9afd9d41-ab9c-4c81-ae6e-8df711f8f302, f2af1252-cf12-469e-a9e6-354ea57643ee, 23e3eba5-ed69-4b11-8e24-e5fe492ac4d5]], flow_tables=[flow_tables={}], rstp_status=[rstp_status={}], external_ids=[external_ids={}], mcast_snooping_enable=[mcast_snooping_enable=false], netflow=[netflow=[]], ipfix=[ipfix=[]], datapath_version=[datapath_version=<unknown>], name=[name=br-physnet1], auto_attach=[auto_attach=[]], fail_mode=[fail_mode=[]], datapath_id=[datapath_id=[0000f60000ff0101]], stp_enable=[stp_enable=false], datapath_type=[datapath_type=], protocols=[protocols=[]], status=[status={}]}]
, updatedControllerRows: {}
2016-12-14 13:47:56,693 | DEBUG | n-invoker-impl-0 | SouthboundMapper | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | controllerEntries: []
2016-12-14 13:47:56,694 | DEBUG | n-invoker-impl-0 | OvsdbBridgeUpdateCommand | 251 - org.opendaylight.ovsdb.southbound-impl - 1.3.2.SNAPSHOT | Built with the intent to store bridge data OvsdbBridgeAugmentation [_bridgeName=OvsdbBridgeName [_value=br-physnet1], _bridgeOtherConfigs=[BridgeOtherConfigs [_bridgeOtherConfigKey=hwaddr, _bridgeOtherConfigValue=f6:00:00:ff:01:01, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=hwaddr], augmentation=[]], BridgeOtherConfigs [_bridgeOtherConfigKey=disable-in-band, _bridgeOtherConfigValue=true, _key=BridgeOtherConfigsKey [_bridgeOtherConfigKey=disable-in-band], augmentation=[]]], _bridgeUuid=Uuid [_value=f183027c-f404-4408-b1c0-9d9f389b5b83], _datapathId=DatapathId [_value=00:00:f6:00:00:ff:01:01], _datapathType=class org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.DatapathTypeSystem, _managedBy=OvsdbNodeRef [_value=KeyedInstanceIdentifier

]]
2016-12-14 13:47:56,817 | ERROR | lt-dispatcher-15 | SimpleShardDataTreeCohort | 176 - org.opendaylight.controller.sal-distributed-datastore - 1.4.2.SNAPSHOT | Transaction member-1-datastore-operational-fe-0-chn-1-txn-0 failed to prepare
java.lang.IllegalArgumentException: Metadata not available for modification NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd/bridge/br-int}

], modificationType=TOUCH, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point, modificationType=MERGE, childModification={(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

]=NodeModification [identifier=(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)termination-point[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)tp-id=br-physnet1}

], modificationType=WRITE, childModification={}]}]}]
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)[38:com.google.guava:18.0.0]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:198)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyWrite(AbstractNodeContainerModificationStrategy.java:113)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:210)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.ListEntryModificationStrategy.applyTouch(ListEntryModificationStrategy.java:55)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.applyTouch(AbstractNodeContainerModificationStrategy.java:252)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.SchemaAwareApplyOperation.apply(SchemaAwareApplyOperation.java:200)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
at org.opendaylight.yangtools.yang.data.impl.schema.tree.AbstractNodeContainerModificationStrategy.mutateChildren(AbstractNodeContainerModificationStrategy.java:139)[80:org.opendaylight.yangtools.yang-data-impl:1.0.2.SNAPSHOT]
....

ReadFailedException{message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

], errorList=[RpcError [message=Error executeRead ReadData for path /(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)network-topology/topology/topology[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)topology-id=ovsdb:1}

]/node/node[

{(urn:TBD:params:xml:ns:yang:network-topology?revision=2013-10-21)node-id=ovsdb://uuid/2886d2f0-a150-4568-b36f-0bc0a15456bd}

], severity=ERROR, errorType=APPLICATION, tag=operation-failed, applicationTag=null, info=null, cause=java.lang.IllegalStateException: Transaction chain has failed]]}
at org.opendaylight.controller.cluster.datastore.NoOpTransactionContext.executeRead(NoOpTransactionContext.java:67)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy$1.invoke(TransactionProxy.java:93)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionContextWrapper.maybeExecuteTransactionOperation(TransactionContextWrapper.java:92)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.executeRead(TransactionProxy.java:90)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
at org.opendaylight.controller.cluster.datastore.TransactionProxy.singleShardRead(TransactionProxy.java:115)[176:org.opendaylight.controller.sal-distributed-datastore:1.4.2.SNAPSHOT]
....

karaf.log with DEBUG on ovsdb attached.

OpenDaylight JIRA

[OVSDB-386] OVSDB unable to configure OVS after restart