Reproduction steps :
https://lists.opendaylight.org/pipermail/ovsdb-dev/2014-September/000720.html
Madhu,
Question... are you guys writing only to the config tree?
I ask, because writing to the operational tree is normally done by the statsmanager in response to the switch reporting back the flow. The only way off the top of my head I could think of that we would see the flow in the operational tree if it wasn't on the switch is if you guys are writing to both config and operational store (not accusing, just trying to eliminate possibilities).
OVSDB debug logs : https://gist.github.com/17f081e1dfe37c9ee56d
The above logs are from writeFlow() method in AbstractServiceInstance.java
CheckedFuture<Void, TransactionCommitFailedException> commitFuture = modification.submit();
try
catch (Exception e)
{ logger.error(e.getMessage(), e); modification.cancel(); }(In reply to Ed Warnicke from comment #2)
> Madhu,
>
> Question... are you guys writing only to the config tree?
Yes. we write ONLY to the config tree.
>
> I ask, because writing to the operational tree is normally done by the
> statsmanager in response to the switch reporting back the flow. The only
> way off the top of my head I could think of that we would see the flow in
> the operational tree if it wasn't on the switch is if you guys are writing
> to both config and operational store (not accusing, just trying to eliminate
> possibilities).
I know, that is why I don't know what to do with this issue 
.
Everything seems normal and appropriate from the state perspective, but the flows are missing.
Quick questions: Are you able to see all your flows in configuration store? IF yes, this is bug of Openflowplugin/FRM/Stats Manager and not MD-SAL Infrastructure.
This multiple threads are writing to datastore?
Or which multiple threads are you referencing?
E.g callback from datastore and LLDP discovery thread?
Reproduced again.. this time with pcap :
Flows : https://gist.github.com/1926c569ff46bfcfac74
(Missing DEFAULT_PIPELINE_FLOW_0 flow)
Config : https://gist.github.com/aaaa5d72a499fca19127
Operational : https://gist.github.com/592be004ee3fd21dc1c8
OVSDB Debug logs : https://gist.github.com/e31255c6ea9a82196e45
PCAP : https://www.dropbox.com/s/9rgolqu0x7enc9p/pipeline_flows_2.pcap?dl=0
[Filter with openflow_v4.type == 14 to see all the flowmods & you will see that the DEFAULT_PIPELINE_FLOW_0 (table =0 , goto_table=20) is missing.
(In reply to Tony Tkacik from comment #5)
> Quick questions: Are you able to see all your flows in configuration store?
Yes.
> IF yes, this is bug of Openflowplugin/FRM/Stats Manager and not MD-SAL
> Infrastructure.
Could be. But when am opening the bug, it is difficult for me to spot it and hence opened it against MD-SAL. Please move it to appropriate component once the
root-cause is identified.
>
> This multiple threads are writing to datastore?
> Or which multiple threads are you referencing?
Both these threads are internal to OVSDB codebase.
But you can think as 2 different threads both registering to the inventory update event which triggers these flow programming.
>
> E.g callback from datastore and LLDP discovery thread?
Was requested by Ed Warnicke to provide some data on this bug from the GroupBasedPolicy ProofOfConcept perspective, which we believe is hitting the same issue.
The net that we have discovered is that flows are missing in our Proof of Concept (POC) which involves two OVS instances (each on a separate VM).
Essentially, VM:odlgbp1 runs the OpenDaylight controller AND an OVS instance odlgbp2 runs an OVS instance (both in mininet).
OVA's of both these VMs are available in my DropBox and access can be granted on request and Thomas and I are also available to share our test environment and run additional tests.
Some of our debug steps have been captured in the following google doc (please note, this is our "scratchpad" so we can document some troubleshooting history, not a clean definitive document) :
https://docs.google.com/document/d/1cTnBBdQFaoR1hvsdHFuKlzcQKU1bvxvT8lCtBaVWTjQ/edit?usp=sharing
To simplify:
1. We see the requisite expected flows via REST. ASSUMPTION: Flows are correctly placed into the CONFIG datastore.
2. We see the requisite expected flows via Karaf debugging (see link above for specific debug levels) and have correlated the flowIDs between REST and Karaf logs via a python script. ASSUMPTION: Requisite flows are being passed to FRM.
3. Via jConsole, we see the correct number of messages being ENQUEUED/DEQUEUED. ASSUMPTION: Requisite flows are being passed through OpenFlowPlugin
4. Wireshark. Have attached wireshark trace, still parsing this.
Per Thomas' comments below: The net of this is that it appears that the flows make it to openflowjava, but get lost somewhere thereafter.
From Thomas Bachman:
*********************
TARBALL of data here:https://drive.google.com/a/noironetworks.com/#folders/0BztNICcppsJKa1B6WXlFYmFtWFE
The tarball has the following directories:
configstore/
jconsole/
karaf-logs/
ovs-logs
wireshark/
The configstore dir contains the result of REST queries to /restconf/config/opendaylight-inventory:nodes/node/openflow:<n>.
The jconsole directory contains a text file of the stats from the msg-spy-service-impl RuntimeBean.
The karaf-log directory contains the karaf log files obtained during the run (note: things to look for are "xid" for openflowjava and "Calling the FlowMod RPC method on MessageDispatchService AddFlow" for openflowplugin).
The ovs-logs dir contains the result of running ovs-ofctl
dump-flows [s1|s2] -O OpenFlow13 on each vSwitch.
The wireshark dir contains the pcapng file and a text file containing the openflow messages seen on the wire between the two vSwitches.
The net of this is that it appears that the flows make it to openflowjava, but get lost somewhere thereafter.
*********************
Madhu,
Could you attach:
a) Your config datastore output from RESTCONF
b) Your expected flows on the switch
c) Your actual flows on the switch
d) Your wireshark capture showing missing flows
to the bug so we can look at that case?
(In reply to Ed Warnicke from comment #9)
> Madhu,
>
> Could you attach:
>
> a) Your config datastore output from RESTCONF
> b) Your expected flows on the switch
> c) Your actual flows on the switch
> d) Your wireshark capture showing missing flows
>
> to the bug so we can look at that case?
I already did that yesterday :
Madhu,
Apologies You are correct, you did that yesterday
> The wireshark dir contains the pcapng file and a text file containing the
> openflow messages seen on the wire between the two vSwitches.
Applying following filter:
openflow_v4.type == 14 && (openflow_v4.flowmod.priority ==1 || openflow_v4.flowmod.priority ==100 || openflow_v4.flowmod.priority ==110 || openflow_v4.flowmod.priority ==111 || openflow_v4.flowmod.priority ==112 || openflow_v4.flowmod.priority ==120 || openflow_v4.flowmod.priority ==121 || openflow_v4.flowmod.priority ==132 || openflow_v4.flowmod.priority ==140 || openflow_v4.flowmod.priority ==150 || openflow_v4.flowmod.priority ==200 || openflow_v4.flowmod.priority ==50 || openflow_v4.flowmod.priority ==64999 || openflow_v4.flowmod.priority ==65000)
and manually counting the expanded OpenFlow 1.3 messages in each TCP message we get 116 across both switches.
ie. Filtering on FlowMODs (14) and also the priorities we set in the GBP project, we should have 114 flows (57 per OVS instances). We are still tracking down the extra 1 flow per switch but suspect it is administration outside of our project.
BUT
Executing <sudo ovs-ofctl dump-flows <switch> -O Openflow13 | wc -l > on each OVS we get:
host1 (s1): 55 (we discount the header as wc -l returns 56 which inc. a header)
host2 (s2): 57 (as above)
This is perplexing as it would appear we are getting 116 flowmods (as per filter above) on the wire but only 112 flows total installed.
moved barrier after message:
https://git.opendaylight.org/gerrit/#/c/11457/
Mentioned patch eliminates collisions during add session and remove session phases. Those can occur simultaneously and a collision usually results into NPE in remove session thread and registered but invalid session.
Those can cause message lost and failing rpcs.
Please retest and eventually mark as resolved.