Operating System: Mac OS
Platform: Macintosh

Attachment out-of-memory-error.txt has been added with description: Full output of northbound integration test with error

To add some data to Colin's report. I've run into Northbound integration test build failures a lot also (Mac OSX). Normally I keep building it and eventually it will build. I tested from a fresh Mac this week and was never able to get past northbound/integration. I pulled the latest JDK and it did the first try, but it did subsequently fail in an Eclipse assemble-it.

Attaching a mvn debug and an mvn stacktrace. Hit me on IRC if anyone has any ideas or needs a box that can replicate the issue every time.

Mem alloc ENV:
export JVM_ARGS="-Xmx2048m -XX:MaxPermSize=2048m"

Stack Trace Failure:

[INFO] Final Memory: 92M/219M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.15:integration-test (default) on project northbound.integrationtest: Execution default of goal org.apache.maven.plugins:maven-failsafe-plugin:2.15:integration-test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest && /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefirebooter1904471709675583702.jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire3782088130151249964tmp /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire_64358786517243491700tmp
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.15:integration-test (default) on project northbound.integrationtest: Execution default of goal org.apache.maven.plugins:maven-failsafe-plugin:2.15:integration-test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest && /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefirebooter1904471709675583702.jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire3782088130151249964tmp /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire_64358786517243491700tmp
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:225)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution default of goal org.apache.maven.plugins:maven-failsafe-plugin:2.15:integration-test failed: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest && /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefirebooter1904471709675583702.jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire3782088130151249964tmp /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire_64358786517243491700tmp
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:110)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
... 19 more
Caused by: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest && /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefirebooter1904471709675583702.jar /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire3782088130151249964tmp /Users/leslieannedamon/Downloads/brent/controller/opendaylight/northbound/integrationtest/target/surefire/surefire_64358786517243491700tmp
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:485)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:352)
at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:158)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:883)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:747)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:645)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
... 20 more
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,

Thanks,
–
Brent Salisbury

The surefire plugin by default uses a 'forked booter' which launches a separate jvm. This jvm does not inherit the default settings of maven. The default permgen allocation of ~64-72mb is not adequate. I see the following message on enabling memory debugging.

PSPermGen total 79808K, used 79490K [0x000000010ec70000, 0x0000000113a60000, 0x0000000113e70000)
object space 79808K, 99% used [0x000000010ec70000,0x0000000113a109a0,0x0000000113a60000)

I was able to get the tests to run successfully by increasing the permgen size to 128mb.

-DargLine="-XX:MaxPermSize=128m"

Thanks Prasanth.

Can we find a way to put this--and ideally also the MAVEN_OPTS config that works--into the build files so that when people check things out it works?

Greetings,
jvm arguments for tests ran by maven can be put into pom, under sure-fire plugin configuration, e.g.:

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<argLine>-Xmx2048m -XX:MaxPermSize=2048m</argLine>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
</configuration>
</plugin>
</plugins>
</build>

For whatever reason, when I add -DargLine="-XX:MaxPermSize=128m", that works, but when I add the <argLine> command to the configuration of surefire that does not work.

1) The permgen usage appears to be excessive. I monitored the memory usage and the test comes very close to the 128mb. We can increase the permgen size to 256mb (to be safe) to allow the builds to succeed - however, the NorthBoundIT test needs to be investigated since it seems to be loading a lot of bundles.

2) Regarding the MAVEN_OPTS setting, the env variable is picked up by the local installation of maven and the build files have no control. There are two solutions to this -
a) Include a default setEnv.sh|cmd script which sets the environment variables
b) update our docs & README with instructions to setup proper env variable.

I will try to submit a change for the surefire configuration.

Colin: The surefire configuration needs to go into the opendaylight/commons/opendaylight/pom.xml

The following works for me -

diff --git a/opendaylight/commons/opendaylight/pom.xml b/opendaylight/commons/opendaylight/pom.xml
index 98e1d1a..1bb49a1 100644
— a/opendaylight/commons/opendaylight/pom.xml
+++ b/opendaylight/commons/opendaylight/pom.xml
@@ -325,6 +325,9 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
+ <configuration>
+ <argLine>-XX:MaxPermSize=256m</argLine>
+ </configuration>
</plugin>
</plugins>
<pluginManagement>

Prasanth,

I don't know why, but on my Mac I'm seeing that if I add the exact change you described down to the same git hash then I still see the OutOfMemoryError, but running it with command line argument -DargLine="-XX:MaxPermSize=128m", it succeeds.

I'm not sure why, but the pom.xml change is not getting propagated.

Attachment permgen-oom-still-there.txt has been added with description: out of memory errorr persists despite pom.xml change

Colin,
The failsafe plugin configuration needs to be updated with the same memory settings. A few others have seem the same issue and I had to revisit my change -

I have submitted a patch at https://git.opendaylight.org/gerrit/#/c/605/

The fix has been reviewed and merged. Please see https://git.opendaylight.org/gerrit/#/c/605/ for details.

This bug is part of the project to Move all ADSAL associated component bugs to ADSAL.