[NETVIRT-531] CSIT Sporadic failures - no connectivity from control node to instances Created: 14/Mar/17  Updated: 06/Jul/17  Resolved: 06/Jul/17

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: Carbon
Fix Version/s: None

Type: Bug
Reporter: Jamo Luhrsen Assignee: Alon Kochba
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File odl1_karaf.log    
External issue ID: 7982

 Description   

L3 connectivity suite

all instances received their dhcp lease

the test cases to ping from the control nodes dhcp namespace to each
instance is failing. However, the connectivity (ping) between instances
across compute nodes is fine. My first suspect was to check the flow
rules on the control node, but I didn't get very far (no time). here
is a direct link to those rules:

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-stateful-boron/212/archives/log.html.gz#s1-s1-s2-t9-k3-k1-k1-k1-k16-k4

 Comments   
Comment by Jamo Luhrsen [ 23/Mar/17 ]

L2 connectivity suite

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-transparent-boron/508/archives/log.html.gz#s1-s1-s1-t12

Comment by Koby Aizer [ 27/Mar/17 ]

Seems to be a duplicate of NETVIRT-551 - L2 connectivity issues are cause by a drop rule in table=51.

Comment by Jamo Luhrsen [ 05/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-learn-boron/240/archives/log.html.gz#s1-s1-s1-t12

Comment by Jamo Luhrsen [ 06/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-carbon/451/archives/log.html.gz#s1-s1-s2-t13

I don't think this is the table=19 or table=51,drop issue, but connectivity from
all instances in network1 could not reach network2. reverse connectivity was fine
as was all traffic to both network1 and network2 from network3. The traffic is
sourced from the control node dhcp namespace.

Comment by Jamo Luhrsen [ 10/Apr/17 ]

(In reply to Jamo Luhrsen from comment #4)
> https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-
> newton-nodl-v2-upstream-transparent-carbon/451/archives/log.html.gz#s1-s1-s2-
> t13
>
> I don't think this is the table=19 or table=51,drop issue, but connectivity
> from
> all instances in network1 could not reach network2. reverse connectivity was
> fine
> as was all traffic to both network1 and network2 from network3. The traffic
> is
> sourced from the control node dhcp namespace.

similar failures:

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-learn-boron/311/archives/log.html.gz

Comment by Jamo Luhrsen [ 10/Apr/17 ]

(In reply to Jamo Luhrsen from comment #5)
> (In reply to Jamo Luhrsen from comment #4)
> > https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-
> > newton-nodl-v2-upstream-transparent-carbon/451/archives/log.html.gz#s1-s1-s2-
> > t13
> >
> > I don't think this is the table=19 or table=51,drop issue, but connectivity
> > from
> > all instances in network1 could not reach network2. reverse connectivity was
> > fine
> > as was all traffic to both network1 and network2 from network3. The traffic
> > is
> > sourced from the control node dhcp namespace.
>

similar failures:

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-learn-boron/311/archives/log.html.gz

and

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-stateful-boron/371/archives/log.html.gz

Comment by Jamo Luhrsen [ 11/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-boron/373/archives/log.html.gz#s1-s1-s1-t13

Comment by Jamo Luhrsen [ 12/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-transparent-carbon/477/archives/log.html.gz#s1-s1-s3

Comment by Jamo Luhrsen [ 12/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-mitaka-upstream-stateful-boron/380/archives/log.html.gz#s1-s1-s2-t18

Comment by Jamo Luhrsen [ 12/Apr/17 ]

(In reply to Jamo Luhrsen from comment #8)
> https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-
> mitaka-upstream-transparent-carbon/477/archives/log.html.gz#s1-s1-s3

I tried to compare the compute nodes br-int flows in this failed run,
to a passing run and I noticed some things missing that may or may
not help with debugging.

I don't see a table 21 rule to catch the FIP that failed to goto table 22
which punts to controller (if my understanding is correct)

attached is the snippet of karaf log from when the FIP was created
until the end of the first failing test case in the robot log above.
hopefully it helps

Comment by Jamo Luhrsen [ 12/Apr/17 ]

Attachment odl1_karaf.log has been added with description: log that is referenced in comment 10

Comment by Jamo Luhrsen [ 24/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-carbon/358/archives/log.html.gz

Comment by Jamo Luhrsen [ 24/Apr/17 ]

(In reply to Jamo Luhrsen from comment #12)
> https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-
> newton-upstream-stateful-carbon/358/archives/log.html.gz

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-learn-boron/269/archives/log.html.gz#s1-s1-s1-t18

Comment by Jamo Luhrsen [ 24/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-transparent-boron/514/archives/log.html.gz#s1-s1-s2-t22

Comment by Jamo Luhrsen [ 28/Apr/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-upstream-stateful-snat-conntrack-carbon/8/archives/log.html.gz#s1-s4-t9

Comment by Jamo Luhrsen [ 05/May/17 ]

https://logs.opendaylight.org/releng/jenkins092/netvirt-csit-1node-openstack-newton-nodl-v2-upstream-stateful-carbon/487/archives/log.html.gz

Generated at Wed Feb 07 20:21:48 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.