https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1845 controller <p>I've been running the ODL Clustering CSIT job from stable/oxygen and analyzing failures, as part of the clustering stabilization effort.</p> <p>During one of these runs, the cluster fails to go to sync status after ODL is restarted (Tell based False). I looked at the node that failed to sync, and see that karaf took just a little over 7 minutes to start up for this test case.</p> <p> </p> <p>Here are the relevant karaf logs:</p> <p> </p> <p>Jun 29, 2018 3:43:47 PM org.apache.karaf.main.Main launch<br/> INFO: Installing and starting initial bundles<br/> Jun 29, 2018 3:43:47 PM org.apache.karaf.main.Main launch<br/> INFO: All initial bundles installed and set to start<br/> Jun 29, 2018 3:43:47 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock<br/> Jun 29, 2018 3:43:47 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Lock acquired<br/> <font color="#d04437">Jun 29, 2018 3:43:47 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired</font><br/> <font color="#d04437">INFO: Lock acquired. Setting startlevel to 100</font><br/> <font color="#d04437">Jun 29, 2018 3:50:48 PM org.apache.karaf.main.Main launch</font><br/> <font color="#d04437">INFO: Installing and starting initial bundles</font><br/> Jun 29, 2018 3:50:49 PM org.apache.karaf.main.Main launch<br/> INFO: All initial bundles installed and set to start<br/> Jun 29, 2018 3:50:49 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock<br/> Jun 29, 2018 3:50:49 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Lock acquired<br/> Jun 29, 2018 3:50:49 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired<br/> INFO: Lock acquired. Setting startlevel to 100</p> <p> </p> <p>I'll also attach the (annotated) karaf log.</p> <p> </p> <p> </p> <p> </p> <p> </p> CONTROLLER-1845

Karaf takes 7 minutes to start

Bug Lowest Resolved Done Arun Sharma Victor Pickard Mon, 2 Jul 2018 20:04:55 +0000 Fri, 5 Oct 2018 15:17:00 +0000 Fri, 5 Oct 2018 15:17:00 +0000 karaf 1 11 <p>Here are the steps in from the controller csit job for restarting ODL (Restart Odl With Tell Based False). <br/>  <br/> 1. kill -9 on karaf pid ( 'ps axf | grep org.apache.karaf | grep -v grep | awk '{print "kill -9 " $1}' | sh' )<br/> 2. Verify karaf is not running<br/> 3. Set Tell Based to False in config file<br/> 4. Copy karaf logs to /tmp (mkdir -p '/tmp' &amp;&amp; rm -vrf '/tmp/log' &amp;&amp; mv -vf '/tmp/karaf-0.8.3-SNAPSHOT/data/log' '/tmp/')<br/> 5. Clean the following directories</p> <ol> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/tmp/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/data/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/cache/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/snapshots/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/journal/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/etc/opendaylight/current/</li> <li>rm -rf /tmp/karaf-0.8.3-SNAPSHOT/etc/host.key</li> </ol> <p>6. Copy logs back to new snapshot dir, as below: # mkdir -p '/tmp/karaf-0.8.3-SNAPSHOT/data' &amp;&amp; rm -vrf '/tmp/karaf-0.8.3-SNAPSHOT/log' &amp;&amp; mv -vf '/tmp/log' '/tmp/karaf-0.8.3-SNAPSHOT/data/</p> <p>Tom, </p> <p>I'll have to get this to fail again in CSIT, and will share the link to the run, which will have the karaf thread dumps you asked for to help debug this issue.</p> <p> </p> <p>If it is due to karaf taking 7 minutes to re-install features, it does hit the disk a lot so I wonder if it's an intermittent issue with disk performance in the jenkins environment.</p> <p>Would need a thread dump when this issue occurs.</p> <p> </p> <p>Need to find a reliable way to reproduce this issue.</p> <p> </p> <p> </p> <p>kill -3 for jvm pid will go to stdout, karaf.out</p> <p> </p> <p>Did CSIT start karaf twice?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a>,</p> <p> </p> <p>which part of the karaf log in the description is "weird" and maybe a problem in csit. I will investigate<br/> and update this jira.</p> <p>I don't have the full context of what is going on here or what this CSIT does, but the first thing that pops out at me is that what is shown in the description of this issue above looks like two Karaf instances started, one at 3:43:47 and then another at 3:50:48 - not one karaf took just a little over 7 minutes to start up. (Look closely, and compare it with the output of when start ODL locally - you'll find "half" of what is shown above.)</p> <p>Does this observation help you?</p> <p>yes there should only be one set. It's as if karaf was started then killed and then started again. </p> <p>We don't have the accompanying csit (e.g. robot) logs for the karaf log in the description, but I was hoping<br/> it was what we'd see when I tracked down a recent example of <a href="https://jira.opendaylight.org/browse/CONTROLLER-1768" title="SyncStatus stays false for more than 5minutes after bringing 2 of 3 nodes down and back up." class="issue-link" data-issue-key="CONTROLLER-1768"><del>CONTROLLER-1768</del></a>. It's not.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vpickard" class="user-hover" rel="vpickard">vpickard</a>, hoping we can get another example of this karaf.log to go with the robot logs. I'll keep looking<br/> myself.</p> <p>From the logs, it is observed that there are messages of JoinSeedNodeProcess which fails with retries before all instances of Karaf Restart - org.apache.karaf.main.Main launch</p> <p>The order of seed-nodes must be uniform across all ODL nodes and availability of first of seed-nodes  is pre-requisite. We can check if at least first condition is satisfied in the target deployment. </p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a></p> <p>One clarification</p> <p>We do not have Quarantine-based suicide and restart (QuarantinedMonitorActor) of ODL process in master - is my understanding correct ?</p> <p>Not seeing any such symptoms on the logs unless QuarantinedMonitorActor had been refactored into some other name retaining suicidal + restart behavior </p> <p>. </p> <p>The quarantined/restart mechanism is still there. But how is what you stated above relevant for this JIRA? This is about what looks to be a 7 min gap during karaf startup and not related to our clustering.</p> <p>I saw something similiar in another run, but in this case, ODL/karaf appears to just hang, i.e., no logs after the first initial startup sequence.</p> <p> </p> <p>Sandbox link:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/181/controller-csit-3node-clustering-vpickard-all-oxygen/23/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/181/controller-csit-3node-clustering-vpickard-all-oxygen/23/</a></p> <p> </p> <p>Final logs in odl3 karaf log:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/181/controller-csit-3node-clustering-vpickard-all-oxygen/23/odl3_karaf.log.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/181/controller-csit-3node-clustering-vpickard-all-oxygen/23/odl3_karaf.log.gz</a></p> <p> </p> <p>Jul 04, 2018 12:43:35 PM org.apache.karaf.main.Main launch<br/> INFO: Installing and starting initial bundles<br/> Jul 04, 2018 12:43:35 PM org.apache.karaf.main.Main launch<br/> INFO: All initial bundles installed and set to start<br/> Jul 04, 2018 12:43:35 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock<br/> Jul 04, 2018 12:43:35 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Lock acquired<br/> Jul 04, 2018 12:43:35 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired<br/> INFO: Lock acquired. Setting startlevel to 100</p> <p> </p> <p> </p> <p> </p> <p> </p> <p>Normal startups look something like this:</p> <p>ul 04, 2018 12:24:53 PM org.apache.karaf.main.Main launch<br/> INFO: Installing and starting initial bundles<br/> Jul 04, 2018 12:24:53 PM org.apache.karaf.main.Main launch<br/> INFO: All initial bundles installed and set to start<br/> Jul 04, 2018 12:24:54 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Trying to lock /tmp/karaf-0.8.3-SNAPSHOT/lock<br/> Jul 04, 2018 12:24:54 PM org.apache.karaf.main.lock.SimpleFileLock lock<br/> INFO: Lock acquired<br/> Jul 04, 2018 12:24:54 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired<br/> INFO: Lock acquired. Setting startlevel to 100<br/> 2018-07-04T12:25:00,969 | INFO | pool-1-thread-2 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.1.5 | Adding features: odl-integration-compatible-with-all/<span class="error">&#91;0.8.3.SNAPSHOT,0.8.3.SNAPSHOT&#93;</span>, odl-jolokia/<span class="error">&#91;1.10.3.SNAPSHOT,1.10.3.SNAPSHOT&#93;</span>, odl-restconf/<span class="error">&#91;1.7.3.SNAPSHOT,1.7.3.SNAPSHOT&#93;</span>, odl-clustering-test-app/<span class="error">&#91;1.7.3.SNAPSHOT,1.7.3.SNAPSHOT&#93;</span>, 42ecb7fe-aed1-4fa7-930f-d13bd83ea61a/<span class="error">&#91;0,0.0.0&#93;</span><br/> 2018-07-04T12:25:01,236 | INFO | features-1-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.1.5 | Changes to perform:<br/> 2018-07-04T12:25:01,237 | INFO | features-1-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.1.5 | Region: root<br/> 2018-07-04T12:25:01,237 | INFO | features-1-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.1.5 | Bundles to install:<br/> 2018-07-04T12:25:01,237 | INFO | features-1-thread-1 | FeaturesServiceImpl | 8 - org.apache.karaf.features.core - 4.1.5 | mvn:com.sun.jersey/je</p> <p> </p> <p>Strange.  Even in the normal run it took 6s before "Adding features..." was logged. What is it doing during that period....  maybe adding feature repos, installing boot features...  we'd need a thread dump (or several) during that period to get an idea.  Does CSIT configure the org.ops4j.pax.url.mvn.cfg to only use the local system dir or does it allow external resolution of artifacts? I'm wondering if it's trying to download artifact(s) over the network.</p> {no format}<br/> <br/> + cat /tmp/karaf-0.8.3-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg<br/> ################################################################################<br/> #<br/> # Licensed to the Apache Software Foundation (ASF) under one or more<br/> # contributor license agreements. See the NOTICE file distributed with<br/> # this work for additional information regarding copyright ownership.<br/> # The ASF licenses this file to You under the Apache License, Version 2.0<br/> # (the "License"); you may not use this file except in compliance with<br/> # the License. You may obtain a copy of the License at<br/> #<br/> # <a href="http://www.apache.org/licenses/LICENSE-2.0" class="external-link" target="_blank" rel="nofollow noopener">http://www.apache.org/licenses/LICENSE-2.0</a><br/> #<br/> # Unless required by applicable law or agreed to in writing, software<br/> # distributed under the License is distributed on an "AS IS" BASIS,<br/> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.<br/> # See the License for the specific language governing permissions and<br/> # limitations under the License.<br/> #<br/> ################################################################################<br/> <br/> #<br/> # If set to true, the following property will not allow any certificate to be used<br/> # when accessing Maven repositories through SSL<br/> #<br/> #org.ops4j.pax.url.mvn.certificateCheck=<br/> <br/> #<br/> # Path to the local Maven settings file.<br/> # The repositories defined in this file will be automatically added to the list<br/> # of default repositories if the 'org.ops4j.pax.url.mvn.repositories' property<br/> # below is not set.<br/> # The following locations are checked for the existence of the settings.xml file<br/> # * 1. looks for the specified url<br/> # * 2. if not found looks for ${user.home}/.m2/settings.xml<br/> # * 3. if not found looks for ${maven.home}/conf/settings.xml<br/> # * 4. if not found looks for ${M2_HOME}/conf/settings.xml<br/> #<br/> #org.ops4j.pax.url.mvn.settings=<br/> <br/> #<br/> # Path to the local Maven repository which is used to avoid downloading<br/> # artifacts when they already exist locally.<br/> # The value of this property will be extracted from the settings.xml file<br/> # above, or defaulted to:<br/> # System.getProperty( "user.home" ) + "/.m2/repository"<br/> #<br/> org.ops4j.pax.url.mvn.localRepository=${karaf.home}/${karaf.default.repository}<br/> <br/> #<br/> # Default this to false. It's just weird to use undocumented repos<br/> #<br/> org.ops4j.pax.url.mvn.useFallbackRepositories=false<br/> <br/> #<br/> # Uncomment if you don't wanna use the proxy settings<br/> # from the Maven conf/settings.xml file<br/> #<br/> # org.ops4j.pax.url.mvn.proxySupport=false<br/> <br/> #<br/> # Comma separated list of repositories scanned when resolving an artifact.<br/> # Those repositories will be checked before iterating through the<br/> # below list of repositories and even before the local repository<br/> # A repository url can be appended with zero or more of the following flags:<br/> # @snapshots : the repository contains snaphots<br/> # @noreleases : the repository does not contain any released artifacts<br/> #<br/> # The following property value will add the system folder as a repo.<br/> #<br/> org.ops4j.pax.url.mvn.defaultRepositories=\<br/> <a href="file:$" class="external-link" target="_blank" rel="nofollow noopener">file:$</a>{karaf.home}/${karaf.default.repository}@id=system.repository@snapshots,\<br/> <a href="file:$" class="external-link" target="_blank" rel="nofollow noopener">file:$</a>{karaf.data}/kar@id=kar.repository@multi@snapshots,\<br/> <a href="file:$" class="external-link" target="_blank" rel="nofollow noopener">file:$</a>{karaf.base}/${karaf.default.repository}@id=child.system.repository@snapshots<br/> <br/> # Use the default local repo (e.g.~/.m2/repository) as a "remote" repo<br/> #org.ops4j.pax.url.mvn.defaultLocalRepoAsRemote=false<br/> <br/> #<br/> # Comma separated list of repositories scanned when resolving an artifact.<br/> # The default list includes the following repositories:<br/> # <a href="http://repo1.maven.org/maven2@id=central" class="external-link" target="_blank" rel="nofollow noopener">http://repo1.maven.org/maven2@id=central</a><br/> # <a href="http://repository.springsource.com/maven/bundles/release@id=spring.ebr" class="external-link" target="_blank" rel="nofollow noopener">http://repository.springsource.com/maven/bundles/release@id=spring.ebr</a><br/> # <a href="http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external" class="external-link" target="_blank" rel="nofollow noopener">http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external</a><br/> # <a href="http://zodiac.springsource.com/maven/bundles/release@id=gemini" class="external-link" target="_blank" rel="nofollow noopener">http://zodiac.springsource.com/maven/bundles/release@id=gemini</a><br/> # <a href="http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases</a><br/> # <a href="https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases</a><br/> # <a href="https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases</a><br/> # To add repositories to the default ones, prepend '+' to the list of repositories<br/> # to add.<br/> # A repository url can be appended with zero or more of the following flags:<br/> # @snapshots : the repository contains snapshots<br/> # @noreleases : the repository does not contain any released artifacts<br/> # @id=repository.id : the id for the repository, just like in the settings.xml this is optional but recommended<br/> #<br/> org.ops4j.pax.url.mvn.repositories=<a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots</a>, <a href="https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror</a>, <a href="http://repo1.maven.org/maven2@id=central" class="external-link" target="_blank" rel="nofollow noopener">http://repo1.maven.org/maven2@id=central</a>, <a href="http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release" class="external-link" target="_blank" rel="nofollow noopener">http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release</a>, <a href="http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external" class="external-link" target="_blank" rel="nofollow noopener">http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external</a>, <a href="http://zodiac.springsource.com/maven/bundles/release@id=gemini" class="external-link" target="_blank" rel="nofollow noopener">http://zodiac.springsource.com/maven/bundles/release@id=gemini</a>, <a href="http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases</a>, <a href="https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases</a>, <a href="https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases" class="external-link" target="_blank" rel="nofollow noopener">https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases</a><br/> {no format} <p>I'm lowering the priority on this issue. This is very sporadic and difficult to reproduce. </p> <p>I do have a patch that should detect when this issue occurs, and do a thread dump:</p> <p><a href="https://git.opendaylight.org/gerrit/#/c/73667/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/73667/</a></p> <p>my local docker container running ODL seems to be stuck in this state. I'm not sure what to do, but I don't want to get out of this state without at least trying to figure out some clues. anyone with ideas, please pass them my way.</p> <p>btw, I'm just now realizing that we are not the only place this kind of thing is seen:</p> <p><a href="https://stackoverflow.com/questions/50734332/karaf-does-not-start" class="external-link" target="_blank" rel="nofollow noopener">https://stackoverflow.com/questions/50734332/karaf-does-not-start</a><br/> <a href="http://karaf.922171.n3.nabble.com/Karaf-4-0-0-6-lines-in-the-logfile-and-nothing-more-td4041410.html" class="external-link" target="_blank" rel="nofollow noopener">http://karaf.922171.n3.nabble.com/Karaf-4-0-0-6-lines-in-the-logfile-and-nothing-more-td4041410.html</a></p> <p>In this state can you use fuser to know the user of /tmp/karaf-0.8.3-SNAPSHOT/lock?</p> <p>fuser -v /tmp/karaf-0.8.3-SNAPSHOT/lock</p> <p>Just to see if this is a file lock problem.</p> <p>yeah, user seems fine:</p> <p>with karaf stopped:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">$ pwd /home/jluhrsen/jamo/repos/OpenDaylight/netvirt-ha-docker/mounts/1/assembly $ sudo fuser lock </pre> </div></div> <p>starting karaf:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">sudo fuser lock /home/jluhrsen/jamo/repos/OpenDaylight/netvirt-ha-docker/mounts/1/assembly/lock: 15756 ps -elf | rg 15756 0 S root 15756 15585 99 80 0 - 2038025 - 15:52 pts/5 00:04:17 /usr/bin/java -Djava.security.properties=/odlha/karaf/target/assembly/etc/odl.java.security -Xms128M -Xmx2048m -XX:+UnlockDiagnosticVMOptions -XX:+HeapDumpOnOutOfMemoryError -Dcom.sun.management.jmxremote -Djava.security.egd=file:/dev/./urandom -Djava.endorsed.dirs=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.172-9.b11.fc28.x86_64/jre/jre/lib/endorsed:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.172-9.b11.fc28.x86_64/jre/lib/endorsed:/odlha/karaf/target/assembly/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.172-9.b11.fc28.x86_64/jre/jre/lib/ext:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.172-9.b11.fc28.x86_64/jre/lib/ext:/odlha/karaf/target/assembly/lib/ext -Dkaraf.instances=/odlha/karaf/target/assembly/instances -Dkaraf.home=/odlha/karaf/target/assembly -Dkaraf.base=/odlha/karaf/target/assembly -Dkaraf.data=/odlha/karaf/target/assembly/data -Dkaraf.etc=/odlha/karaf/target/assembly/etc -Dkaraf.restart.jvm.supported=<span class="code-keyword">true</span> -Djava.io.tmpdir=/odlha/karaf/target/assembly/data/tmp -Djava.util.logging.config.file=/odlha/karaf/target/assembly/etc/java.util.logging.properties -Dkaraf.startLocalConsole=<span class="code-keyword">true</span> -Dkaraf.startRemoteShell=<span class="code-keyword">true</span> -classpath /odlha/karaf/target/assembly/lib/boot/org.apache.karaf.diagnostic.boot-4.1.5.jar:/odlha/karaf/target/assembly/lib/boot/org.apache.karaf.jaas.boot-4.1.5.jar:/odlha/karaf/target/assembly/lib/boot/org.apache.karaf.main-4.1.5.jar:/odlha/karaf/target/assembly/lib/boot/org.osgi.core-6.0.0.jar org.apache.karaf.main.Main cat karaf.log Jul 24, 2018 10:21:47 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Trying to lock /odlha/karaf/target/assembly/lock Jul 24, 2018 10:21:47 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Lock acquired Jul 24, 2018 10:21:47 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired INFO: Lock acquired. Setting startlevel to 100 Jul 24, 2018 10:34:45 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Trying to lock /odlha/karaf/target/assembly/lock Jul 24, 2018 10:34:45 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Lock acquired Jul 24, 2018 10:34:45 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired INFO: Lock acquired. Setting startlevel to 100 Jul 24, 2018 10:37:19 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Trying to lock /odlha/karaf/target/assembly/lock Jul 24, 2018 10:37:19 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Lock acquired Jul 24, 2018 10:37:19 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired INFO: Lock acquired. Setting startlevel to 100 Jul 24, 2018 10:52:11 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Trying to lock /odlha/karaf/target/assembly/lock Jul 24, 2018 10:52:11 PM org.apache.karaf.main.lock.SimpleFileLock lock INFO: Lock acquired Jul 24, 2018 10:52:11 PM org.apache.karaf.main.Main$KarafLockCallback lockAquired INFO: Lock acquired. Setting startlevel to 100 </pre> </div></div> <p>since my reproduction is in a docker container, I'm able to start the container and then bounce karaf as needed.</p> <p>Every time I bounce it I get the same little lockAquired messages in karaf.log, but that's it.</p> <p>However, it seems it's still loading the karaf features and maybe working fine. I see that port 6653 (openflow)<br/> gets lit up when I start karaf.</p> <p>It's still a mystery why the karaf.log is not having any output. The karaf shell "log:display" is also empty.</p> <p>I did see one stdout error when starting the karaf shell, but no idea if it's related at all. here it is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> opendaylight-user@root&gt;23:42:39.944 [Blueprint Extender: 2] ERROR org.opendaylight.netvirt.vpnmanager.LearntVpnVipToPortEventProcessor - Failed to register the entity arpmonitoring </pre> </div></div> <p>One thing to try that may give us some clues, is strace.</p> <p> </p> <p>strace -p &lt;karaf.pid&gt;</p> <p>can't do this in my current setup. docker security stops me. I can maybe start a new container with flags to<br/> allow it, but then I'm worried that this problem will go away.</p> <p>It turns out this is caused by the Pax Logging configuration file having the wrong contents, so the logs aren’t stored in the Karaf log file. Jamo reproduced this, and had a configuration file which only contained the lines from qosservice.cfg. So it seems that QosAlertGenerator in NetVirt is involved — it rewrites the configuration file...</p> <p>When this happens, the container itself is functional. That explains the seven minute thing: the container started, with no Pax logging (so only the initial lines made it), but the container worked so CSIT continued, ended up killing Karaf and starting it again. There is no double-start causing locking issues, it’s two Karaf instances being started in succession but with no logging.</p> <p>nice find. I assume it's getting overwritten with log4j settings instead of log4j2.</p> <p>yeah, this is starting to explain a lot of seeming weirdness. It explains why I couldn't figure out my logs and timestamps when<br/> working on <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a>. At some point my node hit this bug and my automation looped 31 times looking for 1849<br/> but no logs came. So, when I finally hit the bug, there was nothing to see and pass on. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/sad.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>We (<a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=skitt" class="user-hover" rel="skitt">skitt</a>, <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=thapar" class="user-hover" rel="thapar">thapar</a> and I) saw a corrupted logging config file when I started karaf and then quickly exited with a CTRL-C.</p> <p>So we have something troublesome that can pop up if we stop (kill -9, CTRL-C, maybe other ways) a node at the wrong time.</p> <p>I'll see if I can find a more reliable way to reproduce this, if someone can think of a safer way to have netvirt rewrite this logging<br/> configuration file.</p> <p>ok, I can reproduce this locally. The bottom line, I think, is if we kill the karaf process at some perfect time, the<br/> logging config file gets hosed. I don't know how to recover it either, other than having a good copy nearby, which<br/> I can get from the odlparent repo. That copy will have 91 lines in it. After netvirt is started and makes it's qos<br/> additions, it will have 107 lines in it.</p> <p>My most recent reproduction saw that the log file was completely empty, but I've seen it in various truncated<br/> forms as well.</p> <p>this bash script can be used for reproduction. Just replace the ${LOCATION} env var with the root of where you want <br/> it to start karaf and find the logging config file:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> #!/bin/bash <span class="code-keyword">while</span> <span class="code-keyword">true</span> <span class="code-keyword">do</span> # start karaf ${LOCATION}/karaf/target/assembly/bin/start # pick a random value between 1-15 seconds <span class="code-keyword">for</span> when we kill karaf sleepy_time=$((RANDOM % (40 - 20 + 1) + 20)) echo <span class="code-quote">"Waiting $sleepy_time seconds before killing karaf"</span> sleep $sleepy_time # kill it ps aux | grep karaf.main | grep -v grep | awk <span class="code-quote">'{print <span class="code-quote">"kill -9"</span>,$2}'</span> | sh # check line count is right in logging config file linecount=`cat ${LOCATION}/karaf/target/assembly/etc/org.ops4j.pax.logging.cfg | wc -l` <span class="code-keyword">if</span> [ <span class="code-quote">"$linecount"</span> -ne <span class="code-quote">"107"</span> ]; then echo <span class="code-quote">"might have caught the mysterious config file corruption monster"</span>; exit 1 fi done </pre> </div></div> <p>It seems odd for QosAlertGenerator to add fixed logging configuration in code at runtime, usually such configuration is set up on deployment if desired. I don't know the history of this but perhaps we should consider removing it. </p> <p>yeah. or can there be a more atomic way to modify the file such that a badly timed kill won't cause this?</p> <p>That's the risk/danger of killing any process - could leave something in a corrupted state. The levelDB files could get corrupted if it was in the middle of updating a table...</p> <p>of course. and a power outage, OOM crash, kernel panic...</p> <p>You could try deleting the etc/org.opendaylight.netvirt.qosservice.fcg file for your testing if this issue is blocking it. However looking at QosAlertGenerator it looks it would blow up with an NPE if the file doesn't exist but I can push a quick patch for that.</p> <p>ok, what's the theory here? without that file, there will be no actual attempt to modify that logging config file, so we won't have the chance of getting a corrupted file at all?</p> <p>Not sure if an NPE will hurt me will it? not in the case where I'm trying to reproduce the <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a> restart<br/> issue.</p> <p>The qos service bundle would bomb out and fail to start - you don't care about that for your testing but it could affect the diag status checks. For your testing I would suggest to not install integration-all - all you need is the odl-clustering-test-app (ie cars/people)</p> <p>yes. good point. I got ahead of myself here, but at least we found some answers to some mysteries. <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>back to <a href="https://jira.opendaylight.org/browse/CONTROLLER-1849" title="controller not coming up healthy after being killed and restarted (401 after 5m)" class="issue-link" data-issue-key="CONTROLLER-1849"><del>CONTROLLER-1849</del></a> without netvirt features. Really hoping someone will be able to address<br/> this one though.</p> <p>Given that this isn't a clustering problems but Netvirt bug moved to Sam.</p> <p><a href="https://lists.opendaylight.org/pipermail/netvirt-dev/2018-July/007293.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/netvirt-dev/2018-July/007293.html</a></p> <p><a href="https://git.opendaylight.org/gerrit/#/c/74496/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/74496/</a></p> <p><a href="https://jira.opendaylight.org/browse/NETVIRT-1386" title="Better permanent solution for QosAlertGenerator which shall not modify the logging configuration at runtime" class="issue-link" data-issue-key="NETVIRT-1386"><del>NETVIRT-1386</del></a> opened to track discussing if and how a better solution can be re-implemented in a future follow up change to the c/74496 short term removal of the code that modified the logging configuration.</p> Relates NETVIRT-1315 NETVIRT-1386 CONTROLLER-1846 Development Rank 0|i03gan: