Statistics Manager performance poor for large number of flows

The controller was loaded with 10,000 flows, and things went down fairly spectacularly. The controller was running a mininet with 15 nodes, and a script programmed 10k flows randomly (evenly) distributed over the 15 nodes. All 10k flows are written into the config space properly (we have no problem there).

Things start going really bad when we try to program the flows into mininet. CPU Utilization goes to almost 100% and we start loosing connections to mininet switches. We only manage to program only about 3700 flows (as reported back eventually but the table stats collected from mininet switches). I assume this is because we keep loosing the connection when both the config and oper space start being utilized heavily (stats collection -> oper space plus flows into config space). Since the msgContDump stats (plugin) show 10k enqueued AddFlow messages, I assume the messages got lost either in openflowjava or in the mininet switches themselves (I would suspect openflow java first, because I do not see any real utilization in the mininet switches.)

As the number of flows increases, stats collection starts taking a disproportionate amount of CPU cycles. Interestingly, the bottleneck does not seem to be the MD-SAL data store, but the codecs. I am attaching the profiler run.

Things that needs to be fixed:

• figure out how to do flow indexing (OPNFLWPLUG-227)
• Codecs (which is happening, I guess)
• See if we can write into the MD-SAL data store in batches, rather than
  individual stats.
• Another thought would be to use the DOM API for stats rather than the
  generated Java APIs.