[NEUTRON-149] Security group and security rule events are not synced Created: 24/Mar/17  Updated: 04/Jul/18

Status: Open
Project: neutron
Component/s: General
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug Priority: High
Reporter: Tomas Cechvala Assignee: Tomas Cechvala
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Issue Links:
Blocks
is blocked by NEUTRON-158 Dependency checking for Northbound API In Progress
External issue ID: 8069
Priority: High

 Description   

In some cases, mostly when using scripts that use python API and perform stress tests, we've noticed that security groups and rules calls are not synced and so they are sometimes received disordered.

Specifically:

Security rules are created prior to security groups.
Security groups are deleted prior to security rules.

This can be seen in the log below:

  1. PLEASE NOTICE LINES STARTING WITH '#'
  1. PROBLEMS WHEN CREATING DATA
  1. A RULE IS CREATED EVEN THOUGH SECURITY GROUP DOES NOT EXIST YET
    2017-03-22 11:20:06,227 | TRACE | on-dispatcher-70 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | created securityRule - SecurityRule{getDirection=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.DirectionIngress, getEthertype=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.EthertypeV6, getRemoteGroupId=Uuid [_value=95d74f3d-d5d9-4250-a78b-b8d6b2b5f8a9], getSecurityGroupId=Uuid [_value=95d74f3d-d5d9-4250-a78b-b8d6b2b5f8a9], getTenantId=Uuid [_value=d25961e5-b2a3-4767-a46d-59ed3e0c647c], getUuid=Uuid [_value=26abf590-0383-4e24-90b3-ac7c3f581ec9], augmentations={}}
  1. GBP HAS TO CACHE THIS RULE AND PROCESS IT AFTER THE GROUP IS CREATED
    2017-03-22 11:20:06,227 | WARN | on-dispatcher-70 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | Security group of security rule SecurityRuleKey [_uuid=Uuid [_value=26abf590-0383-4e24-90b3-ac7c3f581ec9]] does not exist yet. The rule will be processedwhen the missing security group is created.
  1. NOW THE GROUP IS CREATED
    2017-03-22 11:20:06,242 | TRACE | on-dispatcher-63 | NeutronSecurityGroupAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | created securityGroup - SecurityGroup{getName=default, getTenantId=Uuid [_value=d25961e5-b2a3-4767-a46d-59ed3e0c647c], getUuid=Uuid [_value=95d74f3d-d5d9-4250-a78b-b8d6b2b5f8a9], augmentations={}}
  1. GBP PROCESSES CACHED SECURITY RULE
    2017-03-22 11:20:06,264 | TRACE | on-dispatcher-63 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | Flushing pending security rule SecurityRule{getDirection=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.DirectionIngress, getEthertype=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.EthertypeV6, getRemoteGroupId=Uuid [_value=95d74f3d-d5d9-4250-a78b-b8d6b2b5f8a9], getSecurityGroupId=Uuid [_value=95d74f3d-d5d9-4250-a78b-b8d6b2b5f8a9], getTenantId=Uuid [_value=d25961e5-b2a3-4767-a46d-59ed3e0c647c], getUuid=Uuid [_value=26abf590-0383-4e24-90b3-ac7c3f581ec9], augmentations={}}

########################################################

  1. PROBLEMS WHEN DELETING DATA
  1. CREATED RULE A IN GROUP G
    2017-03-22 11:12:28,294 | TRACE | on-dispatcher-40 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | created securityRule - SecurityRule{getDirection=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.DirectionEgress, getEthertype=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.EthertypeV4, getSecurityGroupId=Uuid [_value=8742b1d1-551d-46e8-96d4-62343beb2dbd], getTenantId=Uuid [_value=46eb8ea4-ef3a-4a05-a768-d0f53ac191fd], getUuid=Uuid [_value=1816dd6b-42c2-4cbf-ad6b-88fc301a0cde], augmentations={}}

...

  1. GROUP G IS REMOVED PRIOR TO RULE A
    2017-03-22 11:12:28,328 | TRACE | on-dispatcher-40 | NeutronSecurityGroupAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | deleted securityGroup - SecurityGroup{getName=CreateProjectUserTests-41f5c185-367f-43b-name, getTenantId=Uuid [_value=46eb8ea4-ef3a-4a05-a768-d0f53ac191fd], getUuid=Uuid [_value=8742b1d1-551d-46e8-96d4-62343beb2dbd], augmentations={}}
  1. GBP HAS TO CACHE GROUP G AND REMOVE IT LATER
    2017-03-22 11:12:28,329 | WARN | on-dispatcher-40 | NeutronSecurityGroupAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | Cannot remove security group SecurityGroupKey [_uuid=Uuid [_value=8742b1d1-551d-46e8-96d4-62343beb2dbd]] before removing last security rule.
    2017-03-22 11:12:28,329 | TRACE | on-dispatcher-40 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | Caching pending deleted security group SecurityGroupKey [_uuid=Uuid [_value=8742b1d1-551d-46e8-96d4-62343beb2dbd]]

#NOW THE RULE A IS DELETED, IT SHOULD HAVE BEEN BEFORE REMOVING SECURITY GROUP G
2017-03-22 11:12:28,329 | TRACE | on-dispatcher-40 | NeutronSecurityRuleAware | 327 - org.opendaylight.groupbasedpolicy.neutron-mapper - 0.5.0.SNAPSHOT | deleted securityRule - SecurityRule{getDirection=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.DirectionEgress, getEthertype=class org.opendaylight.yang.gen.v1.urn.opendaylight.neutron.constants.rev150712.EthertypeV4, getSecurityGroupId=Uuid [_value=8742b1d1-551d-46e8-96d4-62343beb2dbd], getTenantId=Uuid [_value=46eb8ea4-ef3a-4a05-a768-d0f53ac191fd], getUuid=Uuid [_value=1816dd6b-42c2-4cbf-ad6b-88fc301a0cde], augmentations={}}

 Comments   
Comment by Isaku Yamahata [ 24/Mar/17 ]

It looks like networking-odl problem.
several questions to clarify.
What version of networking-od? Is v2driver used?

creation case:
Does this happen only with default security group?
When neutron network is created, automatically default security group and rules are created. the name==default indicates that.

deletion case:
Does this happend when rule is not deleted explicity via neutron API?
Remaining rule under group is automatically deleted when group is deleted.
I suspect this is the case and the cause is the following line.
the logic of rule auto deletion needs to be done early.

https://review.openstack.org/gitweb?p=openstack/networking-odl.git;a=blob;f=networking_odl/ml2/mech_driver_v2.py;h=4622f28e0db262c8866322219673a42cc7ca19b8;hb=HEAD#l243

Comment by Isaku Yamahata [ 24/Mar/17 ]

Regarding to sg rule deletion, can you please give the following patch a try?
https://review.openstack.org/#/c/449727/

Comment by Isaku Yamahata [ 24/Mar/17 ]

Regarding creation, currently dependency validator for SG and SGRule is missing.
https://bugs.opendaylight.org/show_bug.cgi?id=8069

Comment by Isaku Yamahata [ 24/Mar/17 ]

https://bugs.launchpad.net/networking-odl/+bug/1660911

Comment by Isaku Yamahata [ 24/Mar/17 ]

https://review.openstack.org/#/c/449800/
this experimental patch is for sg/sgrule creation.

Comment by Tomas Cechvala [ 28/Mar/17 ]

Isaku,

thank you for looking at this in such short notice.

this was observed by running the following test script
/home/opnfv/repos/functest/functest/ci/run_tests.py -r
https://github.com/opnfv/functest/blob/master/functest/ci/run_tests.py
which runs multiple test tiers in our lab.
In this case we currently rely on karaf log when investigating, so we don't have clear answer how to replicate this yet.

We use v2driver and we rely on stable/newton. I was trying to cherry pick your patches manually, but the corresponding files look different in master vs stable/newton.

Could you please cherry-pick these patches to stable/newton? We will try to rerun tests accordingly.

Thanks

Comment by OpenDaylight Release [ 03/May/18 ]

Since the bug is unassigned I'm currently assigning it to you.

Please assign to the relevant person. 

Comment by Michael Vorburger [ 28/May/18 ]

tcechval NEUTRON-158 may address this - are you interested in helping to test it, when we have something ready there?

Comment by Michael Vorburger [ 07/Jun/18 ]

NEUTRON-158, specifically c/72372, should address this. I would be very intersted in testing feedback. Shall we close this as a dupe of that?

Generated at Wed Feb 07 20:25:41 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.