During perf/scale testing, we see OOM on ODL when a large number of neutron resources are created and deleted we see that ODL is killed due to OOM. Looking at the stdout when the JVM crashed due to OOM, we see:
Heap dump file created [3089813876 bytes in 19.232 secs]
Uncaught error from thread [opendaylight-cluster-data-shard-dispatcher-144] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[opendaylight-cluster-data]
java.lang.OutOfMemoryError: Java heap space
at com.google.common.collect.RegularImmutableMap.createEntryArray(RegularImmutableMap.java:148)
at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:81)
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:294)
at org.opendaylight.controller.cluster.datastore.persisted.FrontendHistoryMetadata.<init>(FrontendHistoryMetadata.java:40)
at org.opendaylight.controller.cluster.datastore.FrontendHistoryMetadataBuilder.build(FrontendHistoryMetadataBuilder.java:54)
at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder$$Lambda$431/741495460.apply(Unknown Source)
at com.google.common.collect.Iterators$8.transform(Iterators.java:799)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)
at org.opendaylight.controller.cluster.datastore.persisted.FrontendClientMetadata.<init>(FrontendClientMetadata.java:38)
at org.opendaylight.controller.cluster.datastore.FrontendClientMetadataBuilder.build(FrontendClientMetadataBuilder.java:77)
at org.opendaylight.controller.cluster.datastore.FrontendMetadata$$Lambda$430/2026307982.apply(Unknown Source)
at com.google.common.collect.Iterators$8.transform(Iterators.java:799)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:258)
at org.opendaylight.controller.cluster.datastore.persisted.FrontendShardDataTreeSnapshotMetadata.<init>(FrontendShardDataTreeSnapshotMetadata.java:71)
at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toSnapshot(FrontendMetadata.java:72)
at org.opendaylight.controller.cluster.datastore.FrontendMetadata.toSnapshot(FrontendMetadata.java:33)
at org.opendaylight.controller.cluster.datastore.ShardDataTree.takeStateSnapshot(ShardDataTree.java:216)
at org.opendaylight.controller.cluster.datastore.ShardSnapshotCohort.createSnapshot(ShardSnapshotCohort.java:68)
at org.opendaylight.controller.cluster.raft.RaftActorSnapshotMessageSupport.lambda$new$0(RaftActorSnapshotMessageSupport.java:52)
at org.opendaylight.controller.cluster.raft.RaftActorSnapshotMessageSupport$$Lambda$123/1533883683.accept(Unknown Source)
at org.opendaylight.controller.cluster.raft.SnapshotManager$Idle.capture(SnapshotManager.java:295)
at org.opendaylight.controller.cluster.raft.SnapshotManager$Idle.capture(SnapshotManager.java:307)
at org.opendaylight.controller.cluster.raft.SnapshotManager.capture(SnapshotManager.java:91)
at org.opendaylight.controller.cluster.raft.behaviors.Follower.lambda$handleAppendEntries$0(Follower.java:254)
at org.opendaylight.controller.cluster.raft.behaviors.Follower$$Lambda$127/742332312.apply(Unknown Source)
at org.opendaylight.controller.cluster.raft.ReplicatedLogImpl.lambda$appendAndPersist$0(ReplicatedLogImpl.java:111)
at org.opendaylight.controller.cluster.raft.ReplicatedLogImpl$$Lambda$128/559701765.apply(Unknown Source)
at akka.persistence.UntypedPersistentActor$$anonfun$persist$1.apply(PersistentActor.scala:206)
- blocks
-
NETVIRT-878 CSIT should help to detect possible memory leaks leading to OOM related to non-closed transactions (and tx chains) early
- In Progress
-
GENIUS-102 New OOM due to more TX leaks seen in the nova-boot scenario (interfacemanager)
- Resolved
-
OPNFLWPLUG-961 New OOM due to more TX leaks seen in the nova-boot scenario
- Resolved
-
OVSDB-435 New OOM due to more TX leaks seen in the nova-boot scenario
- Resolved
-
NETVIRT-883 Umbrella parent issue for grouping all suspected transaction leaks
- Resolved
-
NETVIRT-1010 OOM and other memory issues
- Resolved
- is blocked by
-
NETVIRT-886 Suspected ReadOnlyTransaction leak in StatisticsImpl
- Resolved
-
OVSDB-425 Suspected ReadOnlyTransaction leak in BridgeConfigReconciliationTask
- Resolved
-
OVSDB-426 Suspected ReadOnlyTransaction leak in BridgeOperationalState
- Resolved
-
NETVIRT-1000 Transaction Leaks in neutronvpn, fibmanager & vpnmanager
- Resolved
- is duplicated by
-
NETVIRT-986 java.lang.OutOfMemoryError: GC overhead limit exceeded
- Resolved
- relates to
-
CONTROLLER-1756 OOM due to huge Map in ShardDataTree
- Resolved
-
CONTROLLER-1787 java.lang.Exception: Error creating READ_WRITE transaction on shard default
- Resolved
-
CONTROLLER-1788 Failed to execute RPC (urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet due to org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException
- Resolved
-
OPNFLWPLUG-955 Service group openflow:181281860492823 failed to take ownership
- Resolved
-
OPNFLWPLUG-956 No queue present, failing request
- Resolved
-
OPNFLWPLUG-957 TransactionChainManager... Exception during INITIAL transaction submitting.
- Resolved