karaf.log.gz
[OPNFLWPLUG-1115] Inconsistencies Deleting Elements in a Config Created: 16/Dec/20 Updated: 18/Nov/22 Resolved: 18/Nov/22 |
|
| Status: | Resolved |
| Project: | OpenFlowPlugin |
| Component/s: | None |
| Affects Version/s: | Aluminium-SR1 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | High |
| Reporter: | Eric Sender | Assignee: | Unassigned |
| Resolution: | Won't Do | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
karaf.log.gz
|
| Description |
|
I have found that when deleting an element in a config (specifically in my case, an OpenFlow action) a re-sequencing of keys occurs and I am unable to perform the delete again. This is best shown with an example: Original config: REST GET /restconf/operational/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101
<flow xmlns="urn:opendaylight:flow:inventory"> <id>101</id> <table_id>0</table_id> <flow-statistics xmlns="urn:opendaylight:flow:statistics"> <duration> <second>729</second> <nanosecond>245000000</nanosecond> </duration> <packet-count>249</packet-count> <byte-count>17059</byte-count> </flow-statistics> <priority>3</priority> <hard-timeout>0</hard-timeout> <match> <in-port>1</in-port> <ethernet-match> <ethernet-type> <type>2048</type> </ethernet-type> </ethernet-match> </match> <cookie_mask>0</cookie_mask> <cookie>0</cookie> <flags/> <instructions> <instruction> <order>0</order> <apply-actions> <action> <order>4</order> <set-field> <ipv4-destination>10.0.0.10/32</ipv4-destination> </set-field> </action> <action> <order>5</order> <output-action> <output-node-connector>3</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>6</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:56</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>7</order> <set-field> <ipv4-destination>10.0.3.10/32</ipv4-destination> </set-field> </action> <action> <order>0</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:54</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>1</order> <set-field> <ipv4-destination>10.0.1.10/32</ipv4-destination> </set-field> </action> <action> <order>2</order> <output-action> <output-node-connector>2</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>3</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:55</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>8</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>9</order> <set-nw-ttl-action> <nw-ttl>7</nw-ttl> </set-nw-ttl-action> </action> <action> <order>10</order> <set-field> <ipv4-destination>10.0.6.10/32</ipv4-destination> </set-field> </action> <action> <order>11</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> </apply-actions> </instruction> </instructions> <idle-timeout>0</idle-timeout> </flow>
Notice
Next I delete off action order 0: REST DELETE restconf/config/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101/instructions/instruction/0/apply-actions/action/0 Returns 200 Now the config looks like this: REST GET /restconf/operational/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101
<flow xmlns="urn:opendaylight:flow:inventory"> <id>101</id> <table_id>0</table_id> <flow-statistics xmlns="urn:opendaylight:flow:statistics"> <duration> <second>26</second> <nanosecond>701000000</nanosecond> </duration> <packet-count>249</packet-count> <byte-count>17059</byte-count> </flow-statistics> <priority>3</priority> <hard-timeout>0</hard-timeout> <match> <in-port>1</in-port> <ethernet-match> <ethernet-type> <type>2048</type> </ethernet-type> </ethernet-match> </match> <cookie_mask>0</cookie_mask> <cookie>0</cookie> <flags/> <instructions> <instruction> <order>0</order> <apply-actions> <action> <order>4</order> <output-action> <output-node-connector>3</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>5</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:56</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>6</order> <set-field> <ipv4-destination>10.0.3.10/32</ipv4-destination> </set-field> </action> <action> <order>7</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>0</order> <set-field> <ipv4-destination>10.0.1.10/32</ipv4-destination> </set-field> </action> <action> <order>1</order> <output-action> <output-node-connector>2</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>2</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:55</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>3</order> <set-field> <ipv4-destination>10.0.0.10/32</ipv4-destination> </set-field> </action> <action> <order>8</order> <set-nw-ttl-action> <nw-ttl>7</nw-ttl> </set-nw-ttl-action> </action> <action> <order>9</order> <set-field> <ipv4-destination>10.0.6.10/32</ipv4-destination> </set-field> </action> <action> <order>10</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> </apply-actions> </instruction> </instructions> <idle-timeout>0</idle-timeout> </flow>
First some notes:
Here is a bit of a diff to make it easier to compare the above: diff -r < <order>5</order> 43c37 < <order>6</order> --- > <order>5</order> 53c47 < <order>7</order> --- > <order>6</order> 59,66c53,57 < <set-field> < <ethernet-match> < <ethernet-destination> < <address>00:00:00:aa:00:54</address> < </ethernet-destination> < </ethernet-match> < </set-field> --- 99c89 < <order>9</order> --- > <order>8</order> 105c95 < <order>10</order> --- > <order>9</order> 111c101 < <order>11</order> --- > <order>10</order> Okay, so now what was originally action/1 is now action/0.
Here is the bug, I want to now delete this new action/0. Since the order value has changed to 0, re-run my REST DELETE command: REST DELETE restconf/config/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101/instructions/instruction/0/apply-actions/action/0 Returns 409:
{
"errors": {
"error": [
{
"error-type": "protocol",
"error-tag": "data-missing",
"error-message": "Data does not exist for path: /(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:257642806997568}]/AugmentationIdentifier{childNames=[(urn:opendaylight:flow:inventory?revision=2013-08-19)supported-match-types, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-instructions, (urn:opendaylight:flow:inventory?revision=2013-08-19)supported-actions, (urn:opendaylight:flow:inventory?revision=2013-08-19)switch-features, (urn:opendaylight:flow:inventory?revision=2013-08-19)manufacturer, (urn:opendaylight:flow:inventory?revision=2013-08-19)hardware, (urn:opendaylight:flow:inventory?revision=2013-08-19)software, (urn:opendaylight:flow:inventory?revision=2013-08-19)serial-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)description, (urn:opendaylight:flow:inventory?revision=2013-08-19)port-number, (urn:opendaylight:flow:inventory?revision=2013-08-19)ip-address, (urn:opendaylight:flow:inventory?revision=2013-08-19)meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-meter, (urn:opendaylight:flow:inventory?revision=2013-08-19)group, (urn:opendaylight:flow:inventory?revision=2013-08-19)stale-group, (urn:opendaylight:flow:inventory?revision=2013-08-19)table, (urn:opendaylight:flow:inventory?revision=2013-08-19)table-features]}/(urn:opendaylight:flow:inventory?revision=2013-08-19)table/table[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=0}]/flow/flow[{(urn:opendaylight:flow:inventory?revision=2013-08-19)id=101}]/instructions/instruction/instruction[{(urn:opendaylight:flow:inventory?revision=2013-08-19)order=0}]/instruction/apply-actions/action/action[{(urn:opendaylight:flow:inventory?revision=2013-08-19)order=0}]"
}
]
}
}
It appears internally to ODL and its internal config, it doesn't acknowledge action/0. Internally, it still sees it as action/1. When I retry that delete command but with action/1 it works: REST DELETE restconf/config/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101/instructions/instruction/0/apply-actions/action/1 Returns 200 And the resulting config is: REST GET /restconf/operational/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101 <flow xmlns="urn:opendaylight:flow:inventory"> <id>101</id> <table_id>0</table_id> <flow-statistics xmlns="urn:opendaylight:flow:statistics"> <duration> <second>2</second> <nanosecond>7000000</nanosecond> </duration> <packet-count>249</packet-count> <byte-count>17059</byte-count> </flow-statistics> <priority>3</priority> <hard-timeout>0</hard-timeout> <match> <in-port>1</in-port> <ethernet-match> <ethernet-type> <type>2048</type> </ethernet-type> </ethernet-match> </match> <cookie_mask>0</cookie_mask> <cookie>0</cookie> <flags/> <instructions> <instruction> <order>0</order> <apply-actions> <action> <order>4</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:56</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>5</order> <set-field> <ipv4-destination>10.0.3.10/32</ipv4-destination> </set-field> </action> <action> <order>6</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>7</order> <set-nw-ttl-action> <nw-ttl>7</nw-ttl> </set-nw-ttl-action> </action> <action> <order>0</order> <output-action> <output-node-connector>2</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>1</order> <set-field> <ethernet-match> <ethernet-destination> <address>00:00:00:aa:00:55</address> </ethernet-destination> </ethernet-match> </set-field> </action> <action> <order>2</order> <set-field> <ipv4-destination>10.0.0.10/32</ipv4-destination> </set-field> </action> <action> <order>3</order> <output-action> <output-node-connector>3</output-node-connector> <max-length>60</max-length> </output-action> </action> <action> <order>8</order> <set-field> <ipv4-destination>10.0.6.10/32</ipv4-destination> </set-field> </action> <action> <order>9</order> <output-action> <output-node-connector>4</output-node-connector> <max-length>60</max-length> </output-action> </action> </apply-actions> </instruction> </instructions> <idle-timeout>0</idle-timeout> </flow> Look close and you see that action/0 is what was originally action/2 and also notice that the highest action order number is 9 (it was 11 originally).
So internally, the action-order values keep their original value, even though when I get-config, the order numbers are sequential.
Goal: when running a get-config, I should be able to reference the action/order value as seen by the current get-config. I do not have a way to track the original order values and I believe maintaining the original order value would mean maintaining old states of the system.
Also, as mentioned earlier, the order numbers get overridden by ODL. I.e., if my order values were 123, 150, 155 it will get renumbered to 0, 1, 2. This doesn't bother me per-se, I can work around that, but I need to be able to reference the order values as given by the current get-config query.
This issue is a blocker for us in that we need to be able to do micro edits to our config without sweeping deletes and replaces of entire flows. |
| Comments |
| Comment by Robert Varga [ 16/Dec/20 ] |
|
This looks like a translation issue in OpenFlow – note neither RESTCONF nor datastore can perform data transformations (i.e. changing 'order'). 'order' is a construct defined by OFP YANG models. |
| Comment by Robert Varga [ 16/Dec/20 ] |
|
esender can you add the before and after contents of the config datastore, please?
|
| Comment by Eric Sender [ 16/Dec/20 ] |
|
I actually do have that pasted in the issue description, including a diff between the before and after. The full config would be huge, I narrowed the config to the specific flow in question. Did you want to see the full config? |
| Comment by Robert Varga [ 16/Dec/20 ] |
|
I was not sure what I was looking for exactly, now I think I know. What does REST GET restconf/config/opendaylight-inventory:nodes/node/openflow:123456789/table/0/flow/101/instructions/instruction/0/apply-actions/action/0 return? Also, there should be a stack trace in karaf.log to go with that 409, I think. Can you attach that, too? |
| Comment by Eric Sender [ 16/Dec/20 ] |
|
I don't see a stacktrace unforunatly, even in DEBUG logging mode. Look at:
https://jira.opendaylight.org/secure/attachment/16001/karaf.log.gz |
| Comment by Eric Sender [ 17/Dec/20 ] |
|
(If there is an alternative to DELETE that I could try, I'd like to. I am playing with the REST PATCH endpoint but I am having trouble getting it to work and I can't find any good examples of using REST PATCH) |
| Comment by Sangwook Ha [ 03/Dec/21 ] |
|
esender All the flow information above is from the operational datastore, which is populated by OpenFlow Plugin. Since OpenFlow itself does not have an explicit attribute order for ordering of actions, it's just an ordered list, the attribute is populated by the plugin incrementing from 0. So if you look at the configuration datastore, you will see different order values, and the values in the configuration datastore are not modified by the plugin. For example, I created a flow with 4 actions - note that the order values are 100, 101, 102, 103. Also, the URL is for the RFC8040 implementation, not for Bierman02 draft that starts with /restconf. GET /rests/data/opendaylight-inventory:nodes/node=openflow%3A1/flow-node-inventory:table=1/flow=1/instructions/instruction=0/apply-actions?content=config
{
"flow-node-inventory:apply-actions": {
"action": [
{
"order": 100,
"output-action": {
"output-node-connector": "LOCAL"
}
},
{
"order": 101,
"output-action": {
"output-node-connector": "1"
}
},
{
"order": 102,
"output-action": {
"output-node-connector": "2"
}
},
{
"order": 103,
"output-action": {
"output-node-connector": "3"
}
}
]
}
}
From the operational data store I get this - note that the order values are 0, 1, 2, 3. GET /rests/data/opendaylight-inventory:nodes/node=openflow%3A1/flow-node-inventory:table=1/flow=1/instructions/instruction=0/apply-actions?content=nonconfig
{
"flow-node-inventory:apply-actions": {
"action": [
{
"order": 0,
"output-action": {
"max-length": 0,
"output-node-connector": "LOCAL"
}
},
{
"order": 1,
"output-action": {
"max-length": 0,
"output-node-connector": "1"
}
},
{
"order": 2,
"output-action": {
"max-length": 0,
"output-node-connector": "2"
}
},
{
"order": 3,
"output-action": {
"max-length": 0,
"output-node-connector": "3"
}
}
]
}
}
|
| Comment by Sangwook Ha [ 03/Dec/21 ] |
|
However, in general the inventory data model is not very compatible with RFC8040 - which allows getting both configuration & operational data together. For example, the following REST call GET /rests/data/opendaylight-inventory:nodes/node=openflow%3A1/flow-node-inventory:table=1/flow=1/instructions/instruction=0/apply-actions or GET /rests/data/opendaylight-inventory:nodes/node=openflow%3A1/flow-node-inventory:table=1/flow=1/instructions/instruction=0/apply-actions?content=all returns the following combing both configuration & operational data. {
"flow-node-inventory:apply-actions": {
"action": [
{
"order": 0,
"output-action": {
"max-length": 0,
"output-node-connector": "LOCAL"
}
},
{
"order": 1,
"output-action": {
"max-length": 0,
"output-node-connector": "1"
}
},
{
"order": 2,
"output-action": {
"max-length": 0,
"output-node-connector": "2"
}
},
{
"order": 3,
"output-action": {
"max-length": 0,
"output-node-connector": "3"
}
},
{
"order": 100,
"output-action": {
"output-node-connector": "LOCAL"
}
},
{
"order": 101,
"output-action": {
"output-node-connector": "1"
}
},
{
"order": 102,
"output-action": {
"output-node-connector": "2"
}
},
{
"order": 103,
"output-action": {
"output-node-connector": "3"
}
}
]
}
}
And this obviously is really misleading at best. |
| Comment by Eric Sender [ 03/Dec/21 ] |
|
sangwookha - thank you for this. I did discover this distinction between the config and operational datastore (a year ago I was that much more newb to netconf). Our team engineered our solution with this understanding. We aren't quite able to micro-edit aspects of the operational flow, but we are okay doing full replacements instead. |