https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/CONTROLLER-1487 controller <p>when a node connects to ovsdb southbound, there is an entity-owner node<br/> created. for single node, it looks as below. However, when the device<br/> disconnects, it still remains. It does correctly reflect that there is no<br/> ownershipt, but it should be removed entirely.</p> <p>when node connected:</p> <p> "entity-owners": {<br/> "entity-type": [<br/> {<br/> "entity": [<br/> {<br/> "candidate": [</p> { "name": "member-1" } <p> ],<br/> "id": "/network-topology:network-topology/network-topology:topology<span class="error">&#91;network-topology:topology-id=&#39;ovsdb:1&#39;&#93;</span>/network-topology:node<span class="error">&#91;network-topology:node-id=&#39;ovsdb://uuid/bc330598-4581-4d9f-b932-e362b452137b&#39;&#93;</span>",<br/> "owner": "member-1"<br/> }<br/> ],<br/> "type": "ovsdb"<br/> },<br/> {<br/> "entity": [<br/> {<br/> "candidate": [</p> { "name": "member-1" } <p> ],<br/> "id": "/general-entity:entity<span class="error">&#91;general-entity:name=&#39;ovsdb-southbound-provider&#39;&#93;</span>",<br/> "owner": "member-1"<br/> }<br/> ],<br/> "type": "ovsdb-southbound-provider"<br/> }<br/> ]<br/> }<br/> }</p> <p>when node disconnected:</p> <p> "entity-owners": {<br/> "entity-type": [<br/> {<br/> "entity": [</p> { "id": "/network-topology:network-topology/network-topology:topology[network-topology:topology-id='ovsdb:1']/network-topology:node[network-topology:node-id='ovsdb://uuid/bc330598-4581-4d9f-b932-e362b452137b']", "owner": "" } <p> ],<br/> "type": "ovsdb"<br/> },<br/> {<br/> "entity": [<br/> {<br/> "candidate": [</p> { "name": "member-1" } <p> ],<br/> "id": "/general-entity:entity<span class="error">&#91;general-entity:name=&#39;ovsdb-southbound-provider&#39;&#93;</span>",<br/> "owner": "member-1"<br/> }<br/> ],<br/> "type": "ovsdb-southbound-provider"<br/> }<br/> ]<br/> }<br/> }</p> <p>Operating System: All<br/> Platform: All</p> CONTROLLER-1487

entity structures are kept even when the entity is removed. can be used as DOS attack

Bug Resolved Won't Do Unassigned Jamo Luhrsen Sat, 20 Feb 2016 18:09:41 +0000 Tue, 25 Jul 2023 08:24:07 +0000 Tue, 22 May 2018 10:46:56 +0000 clustering 0 9 <p>Hi Jamo,</p> <p>As per the discussion on following thread, this is expected behavior. </p> <p><a href="https://lists.opendaylight.org/pipermail/integration-dev/2016-February/005957.html" class="external-link" target="_blank" rel="nofollow noopener">https://lists.opendaylight.org/pipermail/integration-dev/2016-February/005957.html</a></p> <p>the root issue here is that entity owner service (EOS) does not clean up the entry when it goes "candidateless". So in the case when a lot of new/unique<br/> entities are learned and removed we will continue to use resources and<br/> if repeated there will be an OutOfMemory exception and crash.</p> <p>one quick way to see this is to start an ovs bridge:</p> <p>sudo ovs-vsctl add-br memLeakerBridge</p> <p>connect it to openflowplugin:</p> <p>sudo ovs-vsctl set-controller memLeakerBridge tcp:${ODL_IP}:6633</p> <p>(NOTE: feature installed is openflowplugin-flow-services-rest)</p> <p>cycle through mac addresses:</p> <p>sudo ovs-vsctl set bridge memLeakerBridge01 other-config:hwaddr=00:00:00:00:00:01<br/> sudo ovs-vsctl set bridge memLeakerBridge01 other-config:hwaddr=00:00:00:00:00:02<br/> sudo ovs-vsctl set bridge memLeakerBridge01 other-config:hwaddr=00:00:00:00:00:03<br/> ... and so on</p> <p>here's a hack of a python script to do the cycling:</p> <p>for i in xrange(0x00, 0xFF):<br/> for j in xrange(0x00,0xFF):<br/> for k in xrange(0x00,0xFF):<br/> print(cmd)<br/> cmd = "sudo ovs-vsctl set bridge memLeakerBridge0java.lang.OutOfMemoryError: Java heap space, 'x') + ":" + format(j, 'x') + ":" + format(k, 'x')<br/> time.sleep(2) # without this pause, it doesn't work. I did not investigate</p> <p>As indicated, entities do not have a complete lifecycle (as usual, removal is missing) and this is a bug.</p> <p>I've prototyped it but I'm afraid that removing an entity when "candidateless" will introduce latent timing bugs. I'm seeing timing-related failures even in unit tests. Even if we delay removal via timer (say 15 sec) and recheck that there are no current candidates, there may be a candidate add transaction inflight, in which case the delete would remove it afterwards. I don't see any way to alleviate this potential issue with the way the in-memory data tree works.</p> <p>I'm inclined to leave it as is. The memory footprint for an empty entity node is pretty small so it would likely take millions to run OOM (depending on how much memory is allocated) and, at least with current use cases, complete removal of an entity should be infrequent.</p> <p>Wrt to DOS attack, any config yang list is a potential DOS attack. Eg, one could easily keep putting nodes to the inventory node list via restconf and run it OOM.</p> <p>(In reply to Tom Pantelis from comment #4)<br/> &gt; I've prototyped it but I'm afraid that removing an entity when<br/> &gt; "candidateless" will introduce latent timing bugs. I'm seeing timing-related<br/> &gt; failures even in unit tests. Even if we delay removal via timer (say 15 sec)<br/> &gt; and recheck that there are no current candidates, there may be a candidate<br/> &gt; add transaction inflight, in which case the delete would remove it<br/> &gt; afterwards. I don't see any way to alleviate this potential issue with the<br/> &gt; way the in-memory data tree works.<br/> &gt; <br/> &gt; I'm inclined to leave it as is. The memory footprint for an empty entity<br/> &gt; node is pretty small so it would likely take millions to run OOM (depending<br/> &gt; on how much memory is allocated) and, at least with current use cases,<br/> &gt; complete removal of an entity should be infrequent.</p> <p>I did not specifically keep track of the count, but I think I ran it OOM<br/> with ovsdb entries numbering in the thousands and maybe slightly less<br/> when doing it with openflow entries. If it's important to get a specific<br/> number, I can. Just wanted to give my observation since it seemed like<br/> a lot less than millions.</p> <p>&gt; Wrt to DOS attack, any config yang list is a potential DOS attack. Eg, one<br/> &gt; could easily keep putting nodes to the inventory node list via restconf and<br/> &gt; run it OOM.</p> <p>fair point, but at least in this specific case we can also issue a <br/> delete via restconf. I'm not sure the right way to get rid of the <br/> stale entities, except by restarting.</p> <p>The number will depend on how much memory you allocate to the JVM - I think the default is 2G so thousands would probably do it. But the same issue would occur if you actually had thousands of valid entities.</p> <p>Also doesn't OVSDB/OF leave the inventory node for a bit of time before purging? If so that's accounting for memory as well and will be affected by your blaster script.</p> <p>If you run OOM, the process would likely be hosed and you likely wouldn't be able to issue a delete <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/> And on restart you would likely run OOM again when it restored from persistence, unless you bumped the memory. At least with entities they don't come back on restart since they're operational <img class="emoticon" src="https://jira.opendaylight.org/images/icons/emoticons/smile.png" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>I'm not saying we shouldn't remove the entries - I just don't see a safe/foolproof way to do it automatically w/o introducing potential race conditions that would result in difficult sporadic bugs to track down. Definitely with removing them immediately. Using a timer would be safer and reduce the chance for a race condition. How long is safe? Probably some minutes to hours we could assume the entity won't come back, although it would require more memory for the bookkeeping to track which ones are eligible to purge. But that still wouldn't stop a DOS attack like your script which would hammer it in seconds.</p> <p>(In reply to Jamo Luhrsen from comment #5)<br/> &gt; (In reply to Tom Pantelis from comment #4)<br/> &gt; &gt; I've prototyped it but I'm afraid that removing an entity when<br/> &gt; &gt; "candidateless" will introduce latent timing bugs. I'm seeing timing-related<br/> &gt; &gt; failures even in unit tests. Even if we delay removal via timer (say 15 sec)<br/> &gt; &gt; and recheck that there are no current candidates, there may be a candidate<br/> &gt; &gt; add transaction inflight, in which case the delete would remove it<br/> &gt; &gt; afterwards. I don't see any way to alleviate this potential issue with the<br/> &gt; &gt; way the in-memory data tree works.<br/> &gt; &gt; <br/> &gt; &gt; I'm inclined to leave it as is. The memory footprint for an empty entity<br/> &gt; &gt; node is pretty small so it would likely take millions to run OOM (depending<br/> &gt; &gt; on how much memory is allocated) and, at least with current use cases,<br/> &gt; &gt; complete removal of an entity should be infrequent.<br/> &gt; <br/> &gt; <br/> &gt; I did not specifically keep track of the count, but I think I ran it OOM<br/> &gt; with ovsdb entries numbering in the thousands and maybe slightly less<br/> &gt; when doing it with openflow entries. If it's important to get a specific<br/> &gt; number, I can. Just wanted to give my observation since it seemed like<br/> &gt; a lot less than millions.<br/> &gt; <br/> &gt; <br/> &gt; &gt; Wrt to DOS attack, any config yang list is a potential DOS attack. Eg, one<br/> &gt; &gt; could easily keep putting nodes to the inventory node list via restconf and<br/> &gt; &gt; run it OOM.<br/> &gt; <br/> &gt; fair point, but at least in this specific case we can also issue a <br/> &gt; delete via restconf. I'm not sure the right way to get rid of the <br/> &gt; stale entities, except by restarting.</p> <p>I do not pretend to understand the EOS implementation, but EntityOwnershipShard is based on Shard, hence it inherently contains and internal DataTree and can control what data is inside that.</p> <p>Since entities are stored in a list, the DataTree does not remove them automatically (as it does for-non-presence containers), but it is certainly in the realm of possibility for EntityOwnershipShard to make this list appear and disappear as candidates are added or deleted.</p> <p>This would require non-trivial amount of surgery, I suspect, probably increasing coupling between Shard and EntityOwnershipShard. Since we have splitting out EOS on our plate for Boron, I suggest we tackle this problem once the split is done, make that squeeky-clean and then consider a backport to Beryllium.</p> <p>The EntityOwnershipShard has full control over the data.</p> <p>I'll provide an example race condition:</p> <ul class="alternate" type="square"> <li>node1 registers a candidate for entity1</li> <li>sometime later node1 unregisters its candidate and submits a tx to remove it</li> <li>the EOS leader commits the tx</li> <li>the candidate list DTCL gets triggered and starts processing the change</li> <li>in the meantime, node2 registers a candidate for entity1 and submits tx1 to add it</li> <li>the candidate list DTCL finishes its processing and sends a CandidateRemoved message</li> <li>the leader pre-commits tx1 and replicates</li> <li>the leader receives the CandidateRemoved and sees that there are no more candidates<br/> and submits tx2 to remove entity1. However tx1 is in progress so tx2 is queued</li> <li>consensus is received for tx1 and it is committed - node2 is now in the candidate<br/> list</li> <li>however the leader then commits tx2 which deletes entity1</li> </ul> <p>So we end up with a strange condition where a node thinks it has a candidate but the entity is gone. It would also try to write node2 as the owner but that would fail.</p> <p>This is why we didn't previously deal with entity removal. Maybe there's some trickiness we can do to alleviate this scenario with more EOS/Shard coupling as Robert mentioned. It's possible the EOS could hook into the can-commit/pre-commit (via Tony's new commit cohort stuff) and inspect the modification and abort an entity delete if it has a candidate.</p> <p>(In reply to Anil Vishnoi from comment #9)<br/> &gt; *** <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-618" title="reading table feature property Warning when using openstack with ODL Cluster" class="issue-link" data-issue-key="OPNFLWPLUG-618"><del>OPNFLWPLUG-618</del></a> has been marked as a duplicate of this bug. ***</p> <p>hi,</p> <p>can you please confirm that <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-618" title="reading table feature property Warning when using openstack with ODL Cluster" class="issue-link" data-issue-key="OPNFLWPLUG-618"><del>OPNFLWPLUG-618</del></a> hase been resolved or not.</p> Duplicate OVSDB-297 Development External issue ID 5397 External issue URL Issue Type Rank 0|i02qvj: