He plugin: Race condition in statistics manager.

I observed that statistics information for some flow entries were missing from operational DS occasionally when I installed so many flow entries (more than 1000 flows per OF switch).

I looked into statistics manager sources, then I found 2 bugs that might drop statistics information.

1. Stats RPC manager (StatRpcMsgManagerImpl) associates transaction container
with the XID returned by stats RPC in txCache. When statistics information
is notified, stats notification listener searches for the container associated
with the XID, and stores received notification into that container.
But statistics information may be notified before the transaction container
is put into txCache because the stats RPC manager does not wait for the
completion of stats RPC on the stats RPC manager thread.

• Although StatRpcMsgManagerImpl.registrationRpcFutureCallBack() is always
  called on the stats RPC manager thread, it uses FutureCallback to detect
  completion of stats RPC. So registrationRpcFutureCallBack() may return
  before stats RPC completes. The XID returned by stats RPC will be put
  into txCache on a future polling thread
  (JdkFutureAdapters.listenInPoolThread()).

• Stats notification listener calls isExpectedStatistics() to check the
  XID, and that check will be done on the stats RPC manager thread
  (statRpcMsgManagerExecutor).

For example, once get-all-flows-statistics-from-all-flow-tables RPC is
invoked, StatListenCommitFlow.onFlowsStatisticsUpdate() may be called
irrespective of whether StatRpcMsgManagerImpl.registrationRpcFutureCallBack()
puts the XID into txCache. onFlowsStatisticsUpdate() will drop flow
statistics in flows-statistics-update notification if the XID is not present
in txCache.

All stats notification listener methods need to wait for stats RPC manager
to put the XID and transaction container into txCache.

2. Stats RPC notification listener method appends the received notification
into the transaction container associated with the XID using
StatRpcMsgManagerImpl.addNotification(). addNotification() creates a task
that appends the received notification into the container, and enqueue it
into the RPC job queue. That task will be executed on the stats RPC manager
thread.

If listener methods receives the last notification (moreReplies flag is not
set), it enqueues a DS operation that stores received statistics information
into operational DS. That operation will be executed on the DS operation
thread (statDataStoreOperationServ).

Therefore, above 2 operations will be executed on different threads in
parallel. But DS operation does not wait for the completion of the task
created by addNotification(). So DS operation may store statistics information
to operational DS before the last notification is added to the transaction
container.

For example, onFlowStatisticsUpdate() enqueues a DS operation that stores
flow statistics into operational DS when moreReplies flag is not set in
the received notification. If that DS operation is scheduled on the DS
operation thread before the task created by addNotification() is scheduled
on the stats RPC manager thread, flow statistics in the last notification
will be dropped.

I believe that the simplest solution is not to use the stats RPC manager
thread. If addNotification() appends the received notification to the
container on the calling thread, DS operation will be enqueud after all
notifications are added to the transaction container. So DS operation does
not need to wait for completion of addNotification() tasks.