https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/ODLPARENT-176 odlparent <p>I'm looking at a Java Flight Recording obtained from (internal) scale lab testing, based on Oxygen SR3 code, and thought it was curious / interesting that the following code paths appears very frequently as a big "TLAB allocation".</p> <p>What's interesting is that in a real world scale test this is the only log that shows up as a big "TLAB allocation" - so it seems to me that a closer look at this code is worth it. Perhaps it's even a simple case of reducing an over INFO to a DEBUG log.</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">StackTraceElement java.lang.Throwable.getStackTraceElement(<span class="code-object">int</span>) 3875 StackTraceElement[] java.lang.Throwable.getOurStackTrace() 3875 StackTraceElement[] java.lang.Throwable.getStackTrace() 3875 StackTraceElement org.apache.logging.log4j.core.impl.Log4jLogEvent.calcLocation(<span class="code-object">String</span>) 3771 StackTraceElement org.apache.logging.log4j.core.impl.MutableLogEvent.getSource() 3771 PaxLocationInfo org.ops4j.pax.logging.log4j2.internal.PaxLoggingEventImpl.getLocationInformation() 3771 void org.ops4j.pax.logging.log4j2.internal.PaxAppenderProxy.doAppend(PaxLoggingEvent) 3771 void org.ops4j.pax.logging.log4j2.appender.PaxOsgiAppender.append(LogEvent) 3771 void org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(LogEvent) 3771 void org.apache.logging.log4j.core.config.AppenderControl.callAppender0(LogEvent) 3771 void org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(LogEvent) 3771 void org.apache.logging.log4j.core.config.AppenderControl.callAppender(LogEvent) 3771 void org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LogEvent) 3771 void org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LogEvent) 3771 void org.apache.logging.log4j.core.config.LoggerConfig.log(LogEvent) 3771 void org.apache.logging.log4j.core.config.LoggerConfig.log(<span class="code-object">String</span>, <span class="code-object">String</span>, Marker, Level, Message, Throwable) 3771 void org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(Supplier, <span class="code-object">String</span>, <span class="code-object">String</span>, Marker, Level, Message, Throwable) 3771 void org.apache.logging.log4j.core.Logger.logMessage(<span class="code-object">String</span>, Level, Marker, Message, Throwable) 3771 void org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog0(Level, <span class="code-object">int</span>, <span class="code-object">String</span>, <span class="code-object">String</span>, Throwable) 3771 void org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.doLog(Level, <span class="code-object">int</span>, <span class="code-object">String</span>, <span class="code-object">String</span>, Throwable) 3771 void org.ops4j.pax.logging.log4j2.internal.PaxLoggerImpl.inform(<span class="code-object">String</span>, Throwable) 3646 void org.ops4j.pax.logging.internal.TrackingLogger.inform(<span class="code-object">String</span>, Throwable) 3646 void org.ops4j.pax.logging.slf4j.Slf4jLogger.info(<span class="code-object">String</span>, <span class="code-object">Object</span>) 3542 List org.opendaylight.netvirt.vpnmanager.VpnUtil.getRts(VpnInstanceOpDataEntry, VpnTarget$VrfRTType) 3466 <span class="code-object">boolean</span> org.opendaylight.netvirt.vpnmanager.VpnUtil.lambda$getVpnsImportingMyRoute$10(VpnInstanceOpDataEntry, VpnInstanceOpDataEntry) 3466 <span class="code-object">boolean</span> org.opendaylight.netvirt.vpnmanager.VpnUtil$$Lambda$1872.1909057161.test(<span class="code-object">Object</span>) 3466 void java.util.stream.ReferencePipeline$2$1.accept(<span class="code-object">Object</span>) 3466 void java.util.stream.ReferencePipeline$2$1.accept(<span class="code-object">Object</span>) 3466 void java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Consumer) 3466 void java.util.stream.AbstractPipeline.copyInto(Sink, Spliterator) 3466 Sink java.util.stream.AbstractPipeline.wrapAndCopyInto(Sink, Spliterator) 3466 <span class="code-object">Object</span> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(PipelineHelper, Spliterator) 3466 <span class="code-object">Object</span> java.util.stream.AbstractPipeline.evaluate(TerminalOp) 3466 <span class="code-object">Object</span> java.util.stream.ReferencePipeline.collect(Collector) 3466 List org.opendaylight.netvirt.vpnmanager.VpnUtil.getVpnsImportingMyRoute(DataBroker, <span class="code-object">String</span>) 3466 void org.opendaylight.netvirt.vpnmanager.populator.impl.L3vpnOverMplsGrePopulator.populateFib(L3vpnInput, WriteTransaction) 3466 void org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.processVpnInterfaceAdjacencies(BigInteger, <span class="code-object">int</span>, <span class="code-object">String</span>, <span class="code-object">String</span>, <span class="code-object">String</span>, <span class="code-object">long</span>, WriteTransaction, WriteTransaction, WriteTransaction, Interface) 3466 void org.opendaylight.netvirt.vpnmanager.VpnInterfaceManager.processVpnInterfaceUp(BigInteger, VpnInterface, <span class="code-object">String</span>, <span class="code-object">int</span>, <span class="code-object">boolean</span>, WriteTransaction, WriteTransaction, WriteTransaction, Interface, <span class="code-object">String</span>) 3466 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener.lambda$<span class="code-keyword">null</span>$0(Interface, WriteTransaction, WriteTransaction, WriteTransaction) 3461 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener$$Lambda$1690.1216649502.accept(<span class="code-object">Object</span>) 3461 ListenableFuture org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(CheckedConsumer) 3461 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener.lambda$<span class="code-keyword">null</span>$1(Interface, WriteTransaction, List, WriteTransaction) 3461 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener$$Lambda$1673.843807247.accept(<span class="code-object">Object</span>) 3461 ListenableFuture org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(CheckedConsumer) 3461 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener.lambda$<span class="code-keyword">null</span>$2(Interface, List, WriteTransaction) 3461 void org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener$$Lambda$1660.1073420380.accept(<span class="code-object">Object</span>) 3461 ListenableFuture org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(CheckedConsumer) 3461 List org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener.lambda$add$3(Interface) 3461 <span class="code-object">Object</span> org.opendaylight.netvirt.vpnmanager.InterfaceStateChangeListener$$Lambda$1654.1406813511.call() 3461 void org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.runWithUncheckedExceptionLogging() 3461 void org.opendaylight.infrautils.utils.concurrent.LoggingUncaughtThreadDeathContextRunnable.run() 3461 <span class="code-object">boolean</span> java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec() 3461 <span class="code-object">int</span> java.util.concurrent.ForkJoinTask.doExec() 3461 void java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) 3461 void java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) 3461 void java.util.concurrent.ForkJoinWorkerThread.run() 3461</pre> </div></div> ODLPARENT-176

Reduce GC pressure and lock contention by ditching the PaxOsgi log appender

Improvement Medium Resolved Won't Do Unassigned Michael Vorburger Tue, 13 Nov 2018 13:29:54 +0000 Fri, 15 Mar 2019 17:44:36 +0000 Fri, 15 Mar 2019 17:44:36 +0000 3.1.4 0 4 <p>Hm, I've just had a quick <a href="https://github.com/opendaylight/netvirt/blob/stable/oxygen/vpnmanager/impl/src/main/java/org/opendaylight/netvirt/vpnmanager/VpnUtil.java" class="external-link" target="_blank" rel="nofollow noopener">closer look at VpnUtil</a> (not exactly the code used in this test; but doubtful something signifant related to this changed) and it looks perfectly harmless, nothing obvious jumps out:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> getRts(...) { <span class="code-object">String</span> name = vpnInstance.getVpnInstanceName(); ... LOG.info(<span class="code-quote">"getRts: vpn targets not available <span class="code-keyword">for</span> {}"</span>, name); </pre> </div></div> <p>So this is ... "normal", as in not an application specific issue? It's just a bit curious it only shows up here. </p> <p>I guess the real question here actually is why Pax Logging is doing a <tt>Throwable.getStackTraceElement()</tt> for an INFO when we (hopefully!) don't have have that (expensive) option, the thing which provides stack trace locations logging switched on - that'a a bit surprising, do you agree <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> ?</p> <p>We wouldn't, shouldn't (that's another story) run with INFO logging enabled in production scale tests, so I'm tempted to just close this issue - it's probably not the biggest issue in object allocation and GC -we've got bigger fish to fry.</p> <p>pax-logging is using it to determine which bundle the call comes from. Change logging formatter to not include it and it's gonna disappear.</p> <p>It's getting location info - for line # maybe? I assume it does that for any log operation independent of level. It doesn't seem like that log message should be info - seems more like a warn - but if it occurs a lot and/or is expected to occur then it really should be debug.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> and <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a> Right, now I see... <tt>PaxOsgiAppender</tt> it is. Let me move this issue from netvirt to odlparent, to make it more general. &#8211; So, if there is even the slightest chance that ditching that OSGi specific appender helps with any runtime performance and garbagge collection overhead, then I'd say why don't we do that. The loss of the Bundle in each log message won't be missed by anyone, I think; in particular because in ODL we have clean package names - so the Bundle &amp; package information really is redundant with the package information. Therefore just so proposed in <a href="https://git.opendaylight.org/gerrit/#/c/77790/" class="external-link" target="_blank" rel="nofollow noopener">c/77790</a> ...</p> <blockquote><p>big "TLAB allocation"</p></blockquote> <p>If it’s a TLAB allocation, it isn’t big.</p> <p>I tend to agree: if it's worth emitting into logs (and thereby doing IO) it's probably not a huge deal. If this is showing up, it's probably because VpnUtil is being noisy – dial its logging down (in code or in config) instead...</p> <p>I beg to differ.. it did show up, and it's simple enough to change, and we don't loose any (useful) functionality, so even there is even a chance that this helps reduce objection allocation and GC pressure, I really don't see a reason why we would not do this.</p> <p>Due to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=rovarga" class="user-hover" rel="rovarga">rovarga</a> -1 on <a href="https://git.opendaylight.org/gerrit/#/c/77790/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/77790/</a> (quote: "I am not against this, just against it being done without a major version bump. Downstream deployments can (and really are) expected to tweak logging anyway.") I'm abandoning the proposed change for the short term and moving this issue to Fix Version odlparent 5.</p> <p>Interesting update: Upon looking at a JFR again, I've now noticed (missed this before) that our friend <tt>PaxOsgiAppender</tt> also shows up as a problem in a completely different place, totally unrelated to Memory &amp; GC: In JDK Mission Control's "Lock Instances" I have a big red exclamation mark and this text:</p> <blockquote><p>The program context switches a lot and many threads wait on the same monitor. Consider using fewer threads, or try to decrease lock contention by other means.</p></blockquote> <p>The top #1 entry is something unrelated (Jetty; that may be normal), but the #2 &amp; #4 spots are both ... <tt>PaxOsgiAppender</tt>. Another good reason to get rid of this IMHO. (The #3 spot is <tt>log4j.core.appender.rolling.RollingFileManager.checkRollover</tt>.)</p> <p>Maybe but, as Robert alluded to, if the PaxOsgiAppender is causing issues wrt load then we're probably logging too much unnecessarily which can lead to other issues. Let's first dial down VpnUtil logging. I'll push a patch for that. </p> <p><a href="https://git.opendaylight.org/gerrit/#/c/77906/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/77906/</a></p> <p>As noted, downstreams are in control of how much they log, which should be reasonable amounts. Packagers are free to reconfigure logging (and will probably use something different than plain files just to allow automatic analysis).</p> <p>Let's leave things as they are.</p> Development Rank 0|i03kfr: