[GENIUS-119] Auto tunnels: race between creation of tun port and bridge ref, port not created

Blocks — Thu, 13 Jun 2019 05:28:05 +0000

What went wrong functionally:

I'm currently working on CSIT to load test auto-tunnels. The test starts 30 docker containers each containing an OVS. Once the containers are running the test quickly set-manager and set-controllers the OVS instances. Test results show that some of the OVS nodes do not get all tunnel ports.

Why?

The reason for this is a race condition where genius attempts to create a tunnel port on the br-int of a node before the BridgeRedEntry is created for that br-int. As such the following code in OvsInterfaceConfigAddHelper#addTunnelConfiguration never enters the if block and the port is never created:

        LOG.debug("creating bridge interfaceEntry in ConfigDS {}", dpId);
        interfaceMetaUtils.createBridgeInterfaceEntryInConfigDS(dpId, interfaceNew.getName());

        // create bridge on switch, if switch is connectedBridgeRefEntry bridgeRefEntry = interfaceMetaUtils.getBridgeRefEntryFromOperDS(dpId);
        if (bridgeRefEntry != null && bridgeRefEntry.getBridgeReference() != null) {
            LOG.debug("creating bridge interface on dpn {}", dpId);
            InstanceIdentifier<OvsdbBridgeAugmentation> bridgeIid =
                    (InstanceIdentifier<OvsdbBridgeAugmentation>) bridgeRefEntry
                    .getBridgeReference().getValue();
            if (createTunnelPort) {
                southboundUtils.addPortToBridge(bridgeIid, interfaceNew, tunnelName);
            }
...

This is evident from the times and line numbers on the following log statements. First we have:

36516 2018-04-08T12:38:16,674 | INFO  | ForkJoinPool-1-worker-3 | OvsInterfaceConfigAddHelper      | 328 - org.opendaylight.genius.interfacemanager-impl - 0.4.1.SNAPSHOT | a      dding tunnel configuration for interface tuneebd1154715
36517 2018-04-08T12:38:16,674 | DEBUG | ForkJoinPool-1-worker-3 | OvsInterfaceConfigAddHelper      | 328 - org.opendaylight.genius.interfacemanager-impl - 0.4.1.SNAPSHOT | c      reating bridge interfaceEntry in ConfigDS 77216042541384

The next line should have been "creating bridge interface on dpn..." (from the code above) but that line never comes because the BridgeRefEntry is only created here:

37304 2018-04-08T12:38:17,159 | DEBUG | ForkJoinPool-1-worker-3 | InterfaceMetaUtils               | 328 - org.opendaylight.genius.interfacemanager-impl - 0.4.1.SNAPSHOT | C      reating bridge ref entry for dpn: 77216042541384 bridge: InstanceIdentifier{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb      .rev150105.OvsdbBridgeAugmentation, path=[org.opendaylight.yang.gen.v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.NetworkTopology, org.opendaylight.yang.gen      .v1.urn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.Topology[key=TopologyKey [_topologyId=Uri [_value=ovsdb:1]]], org.opendaylight.yang.gen.v1.u      rn.tbd.params.xml.ns.yang.network.topology.rev131021.network.topology.topology.Node[key=NodeKey [_nodeId=Uri [_value=ovsdb://uuid/9630ada5-c950-49ec-8599-a53e776ab729/      bridge/br-int]]], org.opendaylight.yang.gen.v1.urn.opendaylight.params.xml.ns.yang.ovsdb.rev150105.OvsdbBridgeAugmentation]}

OpenDaylight JIRA

[GENIUS-119] Auto tunnels: race between creation of tun port and bridge ref, port not created