Uploaded image for project: 'netvirt'
  1. netvirt
  2. NETVIRT-1504

VpnInterfaceManager filling up logs at alarming rate

XMLWordPrintable

      Netvirt 3 node csit failing after 6 hours. Logs show two things:

      1. Default config shard in error state on ODL1.
      2. VPNInterfaceManager repeatedly filling log with error messages at a near instantaneous rate without any delays.

      Cluster issues need to be looked into, but VPNInterfaceManager shouldn't be filling up logs with ERROR messages at such a high rate.
      2018-11-07T00:34:55,909 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 052f2318-dd2a-4530-9703-fdd47b13c3cb not ready, holding on
      2018-11-07T00:34:55,909 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 93517e2a-2534-42c4-a334-2454e04fb17f not ready, holding on
      2018-11-07T00:34:55,910 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 4efba312-0be0-4a6a-a059-3636b344c2aa not ready, holding on
      2018-11-07T00:34:55,911 | ERROR | jobcoordinator-main-task-0 | VpnInterfaceManager | 359 - org.opendaylight.netvirt.vpnmanager-impl - 0.7.1 | add: VpnInstance b15bbaf1-6b95-46c9-b19d-83d7719804ba for vpnInterface 1cb60d50-95dd-4ec6-a78e-375c90b36490 not ready, holding on

      Guilty code is in VpnInterfaceManager.java:

      @Override
      public void add(final InstanceIdentifier<VpnInterface> identifier, final VpnInterface vpnInterface) {
      LOG.trace("Received VpnInterface add event: vpnInterface={}", vpnInterface);
      LOG.info("add: intfName {} onto vpnName {}", vpnInterface.getName(),
      VpnHelper.getVpnInterfaceVpnInstanceNamesString(vpnInterface.getVpnInstanceNames()));
      addVpnInterface(identifier, vpnInterface, null, null);
      }

      addVpnInterface eventuall calls

      private void addVpnInterfaceCall(final InstanceIdentifier<VpnInterface> identifier, final VpnInterface vpnInterface,
      final List<Adjacency> oldAdjs, final List<Adjacency> newAdjs, String vpnName) {
      final VpnInterfaceKey key = identifier.firstKeyOf(VpnInterface.class);
      final String interfaceName = key.getName();

      if (!canHandleNewVpnInterface(identifier, vpnInterface, vpnName)) {
      LOG.error("add: VpnInstance {} for vpnInterface {} not ready, holding on ",
      vpnName, vpnInterface.getName());
      return;
      }

      The log messages in above to methods are what are filling up the log.

      This code should not be:
      1. Retrying forever
      2. Retrying without any delay.

            thapar Vishal Thapar
            thapar Vishal Thapar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 4 hours
                4h