Operating System: All
Platform: All

Forgot to add the link:

https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-1node-periodic-scale-stats-collection-daily-only-carbon/

So it looks like on nitrogen this is already working, and on carbon it will be working after this cherry-pick will be merged: https://git.opendaylight.org/gerrit/#/c/60196/1

Oh sorry, wrong 100k flows bug, this was meant for 6755 one.

This issue only happens in Carbon now and it is FRM related because flows are not removed from switch when we delete them in the inventory.

This is a regression for Carbon (throughout Carbon) - works in Boron and in Nitrogen.

This is sporadically failing on Nitrogen too. Anyway, here is patch that will resolve this issue: https://git.opendaylight.org/gerrit/#/c/63101/

Yeah, also if you look at the karaf logs when it fails, there are few switch disconnects after DS is cleared in case of nitrogen and a bunch of disconnects in case of carbon. This means the controller gets so busy with the DS clear operation that it misses the ECHO requests from switches and these start to disconnect and reconnect aggravating the problem. If we do not fix this for nitrogen, i will open a blocker for carbon as the issue is much more apparent in carbon.

BTW, to support my observation I started couple of tests with switch ECHO message disabled, and both passed in carbon and nitrogen:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-carbon/154/

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-nitrogen/281/

BR/Luis

This was caused because of heavy load on Netty thread, what was caused probably by single-layer serialization. Explanation: with multi-layer, we are doing conversion on different thread and then serializing simple OFJ data structure. In single-layer, we are doing everything on Netty thread. Overall, it is faster by around 1/3 based on YourKit observations what I did during 100k flow test locally, but all load is on Netty thread, so it disconnects. My patch that I mentioned before solves this by pre-serializing data coming via single-layer and then sending raw bytes to netty thread, what causes almost no load on Netty thread.

Right, you patch works in oxygen:

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-1node-periodic-gate-scale-stats-collection-daily-only-oxygen/13/

Do you want to cherry-pick to other branches now to test? or later when your patch is merged in master?

After it will be merged probably.

Created cherry-pick for stable/nitrogen: https://git.opendaylight.org/gerrit/#/c/63491/

Cherry-pick to stable/carbon will come after that.

stable/carbon: https://git.opendaylight.org/gerrit/#/c/63496/

Alright, updated the patch for stable/carbon, again I forgot to actually load the configuration flag to enable the preserialization.

Tomas, does this patch also cover what was in https://git.opendaylight.org/gerrit/#/c/62792/? that patch also had bug-8787 and was abandoned.

That abandoned patch was also supposed to fix https://bugs.opendaylight.org/show_bug.cgi?id=7826, but it didn't so also wondering if 7826 is fixed in this patch now?

No, the flow-related patch tried to improve performance of flow deletion, but it did not (and also caused more issues) so I abandoned it as current method for flow deletion what we are using seems more performant. Also, that patch was never supposed to solve OPNFLWPLUG-858 i just told Sunil about it because he was looking at device registries and I did some modifications to registries and deletion of marks there.

This current patch that solves this issue do not modifies the registry at all, so it will not help the OPNFLWPLUG-858 after it will be merged.

Last patch seems to work so we can close this bug after it is merged.

The patches was not working for extensions, so updated them to include real calls to serialization registry. Unfortunately that required changes in OFJ, and for stable/carbon it is stil separate project, so here is additional patch for OFJ for stable/carbon: https://git.opendaylight.org/gerrit/#/c/63843/

We have this issue in all branches because no patch has been merged yet.

Moving it to Arun.

Anil reviewing these set of patches:

https://git.opendaylight.org/gerrit/#/q/topic:bug/8787+status:open

From last test results:

• Carbon controller is not able to delete flows from switches: this is where we see switches disconnecting.
• Nitrogen and Oxygen controller delete flows from switches but stats are not correct (show flows in operational inventory).

This job started to work after adjusting 2 things:

• poll stats interval: 3 -> 10 sec.
• disable switch echo messages.

I think we still have issue of echo messages not responded when controller is busy, so I will leave this bug open for now reducing severity to Major.

Ideally this test should with without the above mentioned workaround. This bug is kept open to address this issue without the above workaround.

Given that we have a workaround for this issue, reducing the priority.