[CONTROLLER-528] NotificationBrokerImpl is not scalable Created: 01/Jun/14  Updated: 10/Jun/14  Resolved: 10/Jun/14

Status: Resolved
Project: controller
Component/s: mdsal
Affects Version/s: Helium
Fix Version/s: None

Type: Bug
Reporter: Robert Varga Assignee: Robert Varga
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All


External issue ID: 1120

 Description   

Profiling has shown that NotificationBrokerImpl does not allow for proper scaling. The problem is that each notification incurs a full lock on the listeners map, as it may have been modified by listener registration/unregistration. This leads to severe contention in the notification threads, as they keep bouncing the monitor for no good reason.

We need to bias this structure heavily towards lookups and to do that we need to do the following:

  • make the map a constant in lookup path, accessible via an AtomicReference
  • rebuild it completely when a registration/unregistration occurs, and update the AtomicReference

There is also a race condition present: notification dispatch is scheduled on an executor, so we can end up delivering notifications after the registration has been closed. Given the change in lookup/update balance, this race window will be enlarged, so fix it by having the task dispatch the event through the registration object, so that unregistration occurs atomically.



 Comments   
Comment by Robert Varga [ 02/Jun/14 ]

https://git.opendaylight.org/gerrit/#/c/7593/ achieves lock-free operation, with a few object allocations

https://git.opendaylight.org/gerrit/#/c/7594/ adds a cache for results, which means hot objects will be dispatched even without allocations. cold objects are forced to synchronize (not blocking others, though).

Generated at Wed Feb 07 19:53:15 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.