[OPNFLWPLUG-390] [LithiumBuild3335] ODL Controller crashed after connecting 500 switches without any links Created: 31/Mar/15  Updated: 27/Sep/21  Resolved: 18/Oct/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: SANDEEP GANGADHARAN Assignee: SANDEEP GANGADHARAN
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: File 500Switch.py     Zip Archive karaf.zip     Zip Archive log.zip    
External issue ID: 2929

 Description   

1) Run karaf
2) Enable the below features
feature:install odl-openflowplugin-flow-services
feature:install odl-dlux-all
feature:install odl-l2switch-all
feature:install odl-l2switch-switch-ui
feature:install odl-l2switch-switch-rest

3) Using mininet connect 500 switches without any links
4) Controller crashed with following Exception

opendaylight-user@root>Exception in thread "Thread-49388" java.util.concurrent.RejectedExecutionException: Task org.opendaylight.openflowplugin.openflow.md.core.HandshakeStepWrapper@5719fc8d rejected from org.opendaylight.openflowplugin.openflow.md.core.ThreadPoolLoggingExecutor@5c01b9dc[Shutting down, pool size = 0, active threads = 0, queued tasks = 1, completed tasks = 0]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.opendaylight.openflowplugin.openflow.md.core.ConnectionConductorImpl.onConnectionReady(ConnectionConductorImpl.java:450)
at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl$3.run(ConnectionAdapterImpl.java:449)
at java.lang.Thread.run(Thread.java:744)
Exception in thread "odl-stat-rpc-oper-thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2694)
at java.lang.String.<init>(String.java:203)
at java.lang.StringBuilder.toString(StringBuilder.java:405)
at java.lang.Throwable.toString(Throwable.java:481)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:97)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)
at org.apache.log4j.OsgiThrowableRenderer.doRender(OsgiThrowableRenderer.java:118)

5) Operational datastore continue to operate and shows 100 switches even if the switches are disconnected.

 Comments   
Comment by SANDEEP GANGADHARAN [ 31/Mar/15 ]

Attachment log.zip has been added with description: karaf logs attached

Comment by Abhijit Kumbhare [ 01/Jun/15 ]

Sandeep,

Can you retest this with both Helium and Lithium redesigns on stable/lithium?

Abhijit

Comment by SANDEEP GANGADHARAN [ 02/Jun/15 ]

(In reply to Abhijit Kumbhare from comment #1)
> Sandeep,
>
> Can you retest this with both Helium and Lithium redesigns on stable/lithium?
>
> Abhijit

Tried 2000 switches without any links and default xmx 2048
After connecting 44 i saw this exception on karaf console
>Exception in thread "pool-30-thread-1" java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid UUID string: invalid-uuid
at com.lmax.disruptor.FatalExceptionHandler.handleEventException(FatalExceptionHandler.java:45)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:147)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.IllegalArgumentException: Invalid UUID string: invalid-uuid
at java.util.UUID.fromString(UUID.java:194)
at org.opendaylight.openflowplugin.applications.statistics.manager.impl.StatisticsManagerImpl.getGeneratedUUIDForNode(StatisticsManagerImpl.java:399)
at org.opendaylight.openflowplugin.applications.statistics.manager.impl.StatListenCommitGroup$2.generatedUUIDForNode(StatListenCommitGroup.java:206)
at org.opendaylight.openflowplugin.applications.statistics.manager.StatisticsManager$StatDataStoreOperation.<init>(StatisticsManager.java:85)
at org.opendaylight.openflowplugin.applications.statistics.manager.impl.StatListenCommitGroup$2.<init>(StatListenCommitGroup.java:167)
at org.opendaylight.openflowplugin.applications.statistics.manager.impl.StatListenCommitGroup.onGroupFeaturesUpdated(StatListenCommitGroup.java:167)
at org.opendaylight.yangtools.yang.binding.util.NotificationListenerInvoker.invokeNotification(NotificationListenerInvoker.java:91)
at org.opendaylight.controller.md.sal.binding.impl.BindingDOMNotificationListenerAdapter.onNotification(BindingDOMNotificationListenerAdapter.java:44)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouterEvent.deliverNotification(DOMNotificationRouterEvent.java:50)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:68)
at org.opendaylight.controller.md.sal.dom.broker.impl.DOMNotificationRouter$1.onEvent(DOMNotificationRouter.java:65)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
... 3 more

I am attaching the logs.

Comment by SANDEEP GANGADHARAN [ 02/Jun/15 ]

Attachment karaf.zip has been added with description: Karaf log for 2175 build

Comment by Sania Zara [ 07/Dec/15 ]

I used the attached script to recreate this bug using Lithium SR2. The controller dint crash so I attached 1000 switches in the said topology but the controller dint go out of memory even this time.
Disconnection of switches via mininet removes all the switches from inventory as well.

Comment by Sania Zara [ 07/Dec/15 ]

Attachment 500Switch.py has been added with description: Script to create 500 switches without any links

Comment by Jozef Bacigal [ 16/Mar/16 ]

Long time no response, if persist open a new bug please.

Comment by yunyunhan [ 18/Oct/16 ]

(In reply to Jozef Bacigal from comment #5)
> Long time no response, if persist open a new bug please.

I still have the similar problems using redist/beryllium-sr3 version.
I have committed new bugs(OPNFLWPLUG-781/6957).

Generated at Wed Feb 07 20:32:20 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.