https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/OPNFLWPLUG-1026 OpenFlowPlugin <p>If an OVS node is somehow missing a flow that should exist on the switch, currently OFP does not try to proactively fix the issue by reprogramming the flow on the switch. OFP should periodically read all of the flows and groups on the switch and reprogram them if something is missing.</p> OPNFLWPLUG-1026

OFP should proactively check and fix flow table on nodes

Improvement High Open Unresolved Unassigned Tim Rozet neon-release Wed, 11 Jul 2018 20:06:27 +0000 Mon, 27 Sep 2021 09:46:55 +0000 Oxygen forwardingrules-manager 0 4 <p>Installing and removing the flow from device is application's scope. Openflowplugin can't assume that the flow is intentionally removed (assuming some external infrastructure in place is also sharing the switch for some reason) or it was accidently removed from the switch. Whenever a flow is getting removed, it will be removed from the operational data store, so application will get notification about the flow removal. This is an event for application to take an action on, and they can decide what action they want to take.</p> <p>So having this functionality in built to the openflowplugin southbound driver itself is not a right solution, but this functionality can be implemented using an application on top of openflowplugin (like other application -frm, lldp etc), so that user can enable it if they need this functionality. We are happy to receive any contribution regarding this kind of application. Also just to Note : it will require statistics to be enabled in the openflowplugin.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> thanks for your response. I'm somewhat confused as to who is responsible here for ensuring the flows are correct in the switch. My thought process was (take the netvirt scenario) that netvirt would configure some flows in the oper DS to be programmed on the switches. OFP would then read these and program them into the switch. At this point if somehow a flow is deleted on the switch, what netvirt had asked to be programmed hasn't changed, so shouldn't it be OFP job to fix the problem? It sounds like you are saying that the notification that the switch has lost a flow should be punted to the netvirt and he decides what to do. To me it would seem like netvirt should just describe what should be in the switches, and then the southbound handler should ensure it exists and they stay configured as described.</p> <p>To your first point I think you can assume the flow needs to be reprogrammed, regardless of if it was removed accidentally or intentionally. The purpose of assigning ODL as the controller of the switch gives ODL complete control over the flow programming. This is how other OF controllers work like BigSwitch or NEC. If you remove a flow on the switch, it reprograms it immediately.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> This is issue with Netvirt that uses FRM. Expectation is that applications write flows to ConfigDS and then OFP takes over. Here we're seeing flow is added to and present in FRM, but missing from switches. So not a case of flow missing, but flow never showing up. Netvir today doesn't even track operational flows. If it were using RPCs, it would make sense.</p> <p>In other words, this is a bug in -frm. Doesn't that count as part of OFP?</p> <p>I am not sure what is the real ask here. There are two scenarios where flow won't be present in the switch</p> <p>(1) When user configured the flow/group, FRM failed to install the flow/group properly on the switch ( normal installation, or while reconciliation).</p> <p>(2) Flow was installed properly by openflowplugin, but removed by external entity (user, other application sharing switch etc).</p> <p>FRM is responsible for (1) and it should take care of proper programming of the flow/groups. There has been many improvement is been done in this area. So if you are hitting some issue where flow/group is correct, but plugin is not installing the flow, please report that specific issue with error message.</p> <p><a href="https://git.opendaylight.org/gerrit/73531" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/73531</a></p> <p><a href="https://git.opendaylight.org/gerrit/71090" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/71090</a></p> <p><a href="https://git.opendaylight.org/gerrit/70184" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/70184</a></p> <p> </p> <p>Retry mechanism is something that make sense for (2) only and (2) is something that can be usecase specific as well. So just enforcing a common rule of re-installing  flow/group whenever it's missing from operational might not go well with all the usecases. So that's where i was suggesting to implement this as a new feature in the FRM, where user can enable it if it really needs that feature.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=trozet" class="user-hover" rel="trozet">trozet</a> As you mentioned above, ODL as a controller has a complete control over the flow programming, than the question is how the flow is deleted from the switch? It means someone externally deleted the flow from switch (probably intentionally (attack) or accidentally (admin's mistake) . In the production environment both of these scenarios are <b>critical</b> to be notified to the application and the default action of re-installing the flow/group is not a recommended approach because you are probably ignoring the possible network attack, so we can't assume reprogramming as a default action in this case. That's the reason this feature needs to be implemented in a way that user can enable it if they want it, so the application who don't care about any external system removing flows/groups from their controlled switches (like netvirt) can enable this feature by default, but application who wants to take action in these scenarios (like isolate  switch and avoid any programming on the switch as it might be compromised), can keep it disabled.</p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=Avishnoi" class="user-hover" rel="Avishnoi">Avishnoi</a> Agree, we're mainly talking about (1) here. The issues we've seen mostly end up being about flow in FRM not pushed due to leader election being in progress. this mostly happens during bring up, restart or when due to other reasons there is ask timeout and leader re-election kicks in.</p> <p>For 2, though we don't have use cases for external entities currently, do agree a config parameter to enable/disable this functionality would be good way.</p> <p>I believe <a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1028" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/OPNFLWPLUG-1028</a> addresses a part of it. Arun mentions a proper fix for OPENFLWPLUG-1028 in Neon. I think a proper fix for 1028 should address bulk of issues that require this. If we can prioritize a proper redesign/fix for it, it would help.</p> <p>I think retry mechanism would help as a failsafe mechanism. Just like 1028, there might be other such corner cases where a flow can miss being programmed, not to mention bugs in fixes that are coming in or already in. We can't assume that we've fixed all possible scenarios where (1) can occur and having option for retry would help avoid data plane failures for such scenarios.</p> <p><a href="https://jira.opendaylight.org/browse/OPNFLWPLUG-1028" title="Table Miss Entry failed to program in 3 node netvirt CSIT" class="issue-link" data-issue-key="OPNFLWPLUG-1028"><del>OPNFLWPLUG-1028</del></a> is something we are targeting for Neon.</p> <p>Yes, agree that having a retry mechanism is useful. Currently we have the functionality that user/application can use to trigger the device reconciliation, but there is no service that does this automatically for user.</p> <p><a href="https://docs.opendaylight.org/projects/openflowplugin/en/stable-oxygen/specs/reconciliation-cli.html" class="external-link" target="_blank" rel="nofollow noopener">https://docs.opendaylight.org/projects/openflowplugin/en/stable-oxygen/specs/reconciliation-cli.html</a></p> <p>My initial comment was not against the functionality, but moreover where it should be implemented (application level - like FRM or new application) and it should be configurable. As of now i can think of following configuration parameter that we can provide </p> <p>(1) Enable / disable service</p> <p>(2) Retry interval (periodic interval this service should trigger the synchronization for each device, if it's set to 0, it will be triggered on each statistics cycle). Syncing on each statistics cycle can have performance issue, so we need to provide interval so that user can configure it as per their environment and scale.</p> <p>(3) How many attempt it should make to install the missing flows (because in case user install wrong flow, you don't want sync service to keep firing the flow-mod in every iteration).</p> <p>(4) It should remove the flow if it's present in operational datastore but not in config datastore? (Unknown flow installed by someone externally), also how many statistics cycle do we want to wait for this flow to be removed (ideally 2)</p> <p>Probably some more might come up during development phase.</p> Relates OPNFLWPLUG-1028 Development Rank 0|i03gpj: