[NETVIRT-1504] VpnInterfaceManager filling up logs at alarming rate

Thu, 15 Nov 2018 13:52:32 +0000

Netvirt 3 node csit failing after 6 hours. Logs show two things:

1. Default config shard in error state on ODL1.
2. VPNInterfaceManager repeatedly filling log with error messages at a near instantaneous rate without any delays.

Cluster issues need to be looked into, but VPNInterfaceManager shouldn't be filling up logs with ERROR messages at such a high rate.
2018-11-07T00:34:55,909 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 052f2318-dd2a-4530-9703-fdd47b13c3cb not ready, holding on
2018-11-07T00:34:55,909 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 93517e2a-2534-42c4-a334-2454e04fb17f not ready, holding on
2018-11-07T00:34:55,910 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 4efba312-0be0-4a6a-a059-3636b344c2aa not ready, holding on
2018-11-07T00:34:55,911 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 1cb60d50-95dd-4ec6-a78e-375c90b36490 not ready, holding on

Guilty code is in VpnInterfaceManager.java:

@Override
public void add(final InstanceIdentifier<VpnInterface> identifier, final VpnInterface vpnInterface) {
LOG.trace("Received VpnInterface add event: vpnInterface={}", vpnInterface);
LOG.info("add: intfName {} onto vpnName {}", vpnInterface.getName(),
VpnHelper.getVpnInterfaceVpnInstanceNamesString(vpnInterface.getVpnInstanceNames()));
addVpnInterface(identifier, vpnInterface, null, null);
}

addVpnInterface eventuall calls

private void addVpnInterfaceCall(final InstanceIdentifier<VpnInterface> identifier, final VpnInterface vpnInterface,
final List<Adjacency> oldAdjs, final List<Adjacency> newAdjs, String vpnName) {
final VpnInterfaceKey key = identifier.firstKeyOf(VpnInterface.class);
final String interfaceName = key.getName();

if (!canHandleNewVpnInterface(identifier, vpnInterface, vpnName)) {
LOG.error("add: VpnInstance {} for vpnInterface {} not ready, holding on ",
vpnName, vpnInterface.getName());
return;
}

The log messages in above to methods are what are filling up the log.

This code should not be:
1. Retrying forever
2. Retrying without any delay.

OpenDaylight JIRA

[NETVIRT-1504] VpnInterfaceManager filling up logs at alarming rate