Operating System: Mac OS
Platform: Macintosh

Hi Jan,

This issue is fixed now and code is pushed to controller repo through following gerrit:

https://git.opendaylight.org/gerrit/#/c/4583/

I locally verified the fix. Can you please also verify it once its get merged to the controller repo.

Thanks
Anil

Attachment ODL Node.json.postman_collection has been added with description: Postman collection to aid debugging

Anil

the problem is still there for a particular flow.

To reproduce, use mininet, Chrome Postman, and the 'ODL Node' Collection attached to this report as follows:

1. start mininet with 'sudo mn --topo single,3 --controller 'remote,ip=<your-ip-address>:6653' --switch ovsk,protocols=OpenFlow13'

2. Import the attached collection into Postman. You may have to edit the server IP address in the collection to reflect your controller's IP address.

3. Issue 'GET Node oper openflow:1' request to validate that the ovs node is up

4. Wait for stats collection to occur, and then issue the GET Table oper openflow:1/Table 2' request from the ODL Collection to validate that the table is empty. <flow-count> in <aggregate-flow-statistics> is reported as 0.

5. Add Flow 140: Issue the 'PUT FLOW openflow:1/Table 2/Flow 140' request from the ODL Node collection

6. Issue the 'GET Node oper openflow:1/Table 2' request to get Table 2 oper data. <flow-count> in <aggregate-flow-statistics> should be 1, but it is 0. The flow is reported in the table. You may also issue the 'sudo ovs-ofctl -O OpenFlow13 dump-flows s1' command on mininet to validate that the flow was programmed into the switch.

7. Delete Flow 140: Issue the 'DELEYE FLOW openflow:1/Table 2/Flow 140' request from the ODL Node collection

8. 6. Issue the 'GET Node oper openflow:1/Table 2' request to get Table 2 oper data. <flow-count> in <aggregate-flow-statistics> should be 0, but it is 2. The flow is reported in the table. You may also issue the 'sudo ovs-ofctl -O OpenFlow13 dump-flows s1' command on mininet to validate that the flow was deleted from the switch.

Hi Jan,

I fixed one bug related to transaction id management in statistics-manager. That was creating issue in entertaining all the multipart response from the switches. Hence you see that there are few node where you see stats update, but on few you don't. I pushed the code changes through

http://git.opendaylight.org/gerrit/4748

Once it get merge, can you please try with latest controller and latest plugin code. Because in my local setup, i am using the same steps and I see that its working as expected. ( I tried many time with flow 135 and i can see that operational data gets deleted when you remove the flow).

Tomorrow I will try exact step you mentioned and will update you with the results.

Thanks
Anil

Hi Anil,

flow 135 works fine, flow 140 does not. I don't know why. Also, Flow 140 is interesting in that the reported flow id is not the programmed flow id (140). On a freshly rebooted controller, the flow id in oper space is 1201.

I will retest with a freshly pulled code tomorrow.

Hi Jan,

When you restart the controller, your original flow 140 is not in config data store anymore. So when statistics-manager get flow stats for 140 and don't find it in config data store, it augment this data as 'unaccounted flow' and usage different id. That's why you are seeing this flow under id 1201. For unaccounted flows, id is created using the format [ 1-(table-number)0(incremental number starting from 1)].

I will also look at flow 140 and see if there is any issue.
Thanks
Anil

We just tried simple flow with OF10 mininet:

PUT http://127.0.0.1:8080/restconf/config/opendaylight-
inventory:nodes/node/openflow:1/table/2/flow/139

<flow xmlns="urn:opendaylight:flow:inventory">
<priority>2</priority>
<flow-name>Foo</flow-name>
<match>
<ethernet-match>
<ethernet-type>
<type>2048</type>
</ethernet-type>
</ethernet-match>
<ipv4-destination>10.0.10.1/32</ipv4-destination>
</match>
<id>139</id>
<table_id>2</table_id>
<instructions>
<instruction>
<order>0</order>
<apply-actions>
<action>
<order>0</order>
<dec-nw-ttl/>
</action>
</apply-actions>
</instruction>
</instructions>
</flow>

And after delete it, we see flow in operational space so I do not think this is fixed.

I am able to recreate this issue locally. Looks like some issue with IP address matching while augmenting stats data. Will push fix ASAP.

Hi Luis/Jan,

As I suspected, It was happening because of IP address comparison for flow from stats and flow stored in config data store was failing. I fixed the issue and pushed changes through following gerrits :

Controller repo : http://git.opendaylight.org/gerrit/4766
Plugin : http://git.opendaylight.org/gerrit/4765

Both are merged now.

Hi Luis,
I tested your scenario and now its working fine. Can you please verify.

Hi Jan,

I followed exactly same steps with the post man collection you provided. In step 5 of adding flow 140, i get following error from switch,
2014-01-25 19:28:52.178 IST [pool-23-thread-7] ERROR o.o.o.o.m.c.t.ErrorTranslator - Error Message received: type=2[BADACTION], code=0[BADTYPE], data=[04 0e 00 58 00 00 1a d0 00 00 00 00 00 00 00 0a 00 00 00 00 00 00 00 0a 02 03 0d 48 04 b0 00 02 ff ff ff ff ff ff ff ff ff ff ff ff 00 00 00 00 00 01 00 16 80 00 0a 02 08 00 80 00 19 08 0a 00 ]

And when i dumped the flow at switch it didn't show any installed flow as well, and because of that aggregate flows are still 0.

I did minor change (dec-mpls-ttl to dec-nw-ttl) and IP address from 10.0.0.1/24 to 10.0.0.0/24, and it worked fine. Probably some bug with dec-mpls-ttl action. I changed IP address from 10.0.0.1 to 10.0.0.0 because when i installed rule with ip=10.0.0.1/24, switch installed following rule

mininet> sh ovs-ofctl -O OpenFlow13 dump-flows s1
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0xa, duration=2.855s, table=2, n_packets=0, n_bytes=0, idle_timeout=3400, hard_timeout=1200, priority=2,ip,nw_dst=10.0.0.0/24 actions=dec_ttl

Please see nw_dst=10.0.0.0/24. So it ignores anything out of /24 mask. So in flow stats i get 10.0.0.0/24 rather then 10.0.0.1/24 and it was not augmenting the stats to the config flow, rather its augmenting it as a unaccounted flow, because we don;t see it in config data store. Unaccounted flows gets uninstalled if there is no stats update for them from switch for more then two cycles of flow collection.

I am trying to work out intelligent flow comparison mechanism to handle these scenarios. I will send mail about it to openflowplugin with more details and few point to take care of.

Thanks
Anil

Hi Jan,

I pushed code changes that fixes this issue.

http://git.opendaylight.org/gerrit/4822

I added netmask based ip matching for flow matching. I am augmenting statistics data to all the flows from config data store that are matching with the masked flow from switch. In my opinion, this will make things bit simpler, in term of maintaining the stats, updating and periodic cleanup. Also it will make life easier for the folks who wants to access stats for any specific flow, they won't have search through all the flows in operational data store and parsing the keys from flow id (we will have to if we use alternate approach of creating flow id from the keys of all matching flows).

Can you please verify with this latest fix.

Thanks
Anil

Hi Jan,

I forgot to mention one point, that whenever we update flows (add/delete), I am not sending aggregate flow statistics and flow table statistics request immediately. I only send individual flow statistics request at that moment to fetch the stats. Reason is, you can't send flow table statistics request for one specific table from the node, so with one request you receive stats update for all the tables in the switch. I felt this might create performance issue, at high load of flow updates. Although you can send aggregate flow stats request for specific table, but I am avoiding that as well, because it might create confusion when user see different number in flow table statistic and aggregate flow statisitcs.

Hope it make sense.

Thanks
Anil