Setup:
opendaylight stable boron.
JVM heap size: 8 GB
RAM : 32 GB
CPU : 24

Problem Statement:
Huge memory consumption during reconciliation. Old Gen getting filled completely which triggers multiple full GCs but still it is not getting freed.
Frequent and Long full GC pauses cripples the controller.

Scenario:
Multiple reconciliation cycles were triggered.
1) Start the controller
2) Push 10K flows for each switches[40]
3) Start monitoring the heap by using the jstat.
4) Connect the 40 switches.
5) Wait for the reconciliation to finish.
6) disconnect the switch
7) repeat from step 3.

You will observe that the full GC is not able to clear the old gen space. With every reconciliation iteration we see the old gen utilization is increasing. Eventually the utilization hits the max capacity, leading to frequent and long full GCs.