<!-- 
RSS generated by JIRA (8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d) at Wed Feb 07 20:32:04 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>OpenDaylight JIRA</title>
    <link>https://jira.opendaylight.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>8.20.10</version>
        <build-number>820010</build-number>
        <build-date>22-06-2022</build-date>
    </build-info>


<item>
            <title>[OPNFLWPLUG-290] Flows failed to get install in the same openflow table when programmed from different Threads simultaneously.</title>
                <link>https://jira.opendaylight.org/browse/OPNFLWPLUG-290</link>
                <project id="10155" key="OPNFLWPLUG">OpenFlowPlugin</project>
                    <description>&lt;p&gt;When an openflow node connects to the controller, ovsdb&apos;s net-virt-provider bundle programs the switch with a set of pipeline flows (From Table 0 -&amp;gt; 110 in increments of 10, except Table 10 and 70). These pipeline flows are vital for openstack operation.&lt;/p&gt;

&lt;p&gt;But, randomly some of the flows fails to get programmed. 1 of the cases that we narrowed down is on Table 0 flow. This randomness happens based on the timing of another flow being installed from another thread. This is the LLDP flow programmed by another thread. When these 2 threads happen to operate simultaneously (or almost), one of the flows gets lost somewhere in the system.&lt;/p&gt;


&lt;p&gt;Actual flows programmed in OVS : &lt;a href=&quot;https://gist.github.com/05b25649edd8ec6fa91c&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/05b25649edd8ec6fa91c&lt;/a&gt;&lt;br/&gt;
&lt;span class=&quot;error&quot;&gt;&amp;#91;Table 10 and 70 will be missing and that is intentional&amp;#93;&lt;/span&gt;. Table 0 pipeline flow is missing.&lt;/p&gt;

&lt;p&gt;The strangeness of the situation is that, the Table0 pipeline flow (DEFAULT_PIPELINE_FLOW_0) that is missing in the actual flow-table in OVS is seen in the config tree &amp;amp; operational tree.&lt;/p&gt;

&lt;p&gt;Config : &lt;a href=&quot;http://localhost:8181/restconf/config/opendaylight-inventory:nodes&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://localhost:8181/restconf/config/opendaylight-inventory:nodes&lt;/a&gt;&lt;br/&gt;
=&amp;gt; &lt;a href=&quot;https://gist.github.com/cef3a914ca18cb76e330&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/cef3a914ca18cb76e330&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Operational : &lt;a href=&quot;http://localhost:8181/restconf/operational/opendaylight-inventory:nodes&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://localhost:8181/restconf/operational/opendaylight-inventory:nodes&lt;/a&gt; =&amp;gt; &lt;a href=&quot;https://gist.github.com/bfc9e48b3d4fde91cb11&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/bfc9e48b3d4fde91cb11&lt;/a&gt;&lt;/p&gt;</description>
                <environment>&lt;p&gt;Operating System: All&lt;br/&gt;
Platform: All&lt;/p&gt;</environment>
        <key id="27558">OPNFLWPLUG-290</key>
            <summary>Flows failed to get install in the same openflow table when programmed from different Threads simultaneously.</summary>
                <type id="10104" iconUrl="https://jira.opendaylight.org/secure/viewavatar?size=xsmall&amp;avatarId=10303&amp;avatarType=issuetype">Bug</type>
                                                <status id="5" iconUrl="https://jira.opendaylight.org/images/icons/statuses/resolved.png" description="A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.">Resolved</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10000">Done</resolution>
                                        <assignee username="mavenugo@gmail.com">Madhu Venugopal</assignee>
                                    <reporter username="mavenugo@gmail.com">Madhu Venugopal</reporter>
                        <labels>
                    </labels>
                <created>Sat, 20 Sep 2014 11:13:52 +0000</created>
                <updated>Mon, 27 Sep 2021 09:01:20 +0000</updated>
                            <resolved>Thu, 19 Feb 2015 17:03:36 +0000</resolved>
                                                                    <component>General</component>
                        <due></due>
                            <votes>0</votes>
                                    <watches>7</watches>
                                                                                                                <comments>
                            <comment id="56203" author="mavenugo@gmail.com" created="Sat, 20 Sep 2014 11:18:01 +0000"  >&lt;p&gt;Reproduction steps : &lt;br/&gt;
&lt;a href=&quot;https://lists.opendaylight.org/pipermail/ovsdb-dev/2014-September/000720.html&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://lists.opendaylight.org/pipermail/ovsdb-dev/2014-September/000720.html&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="56204" author="hagbard@gmail.com" created="Sat, 20 Sep 2014 11:21:34 +0000"  >&lt;p&gt;Madhu,&lt;/p&gt;

&lt;p&gt;Question... are you guys writing &lt;b&gt;only&lt;/b&gt; to the config tree?&lt;/p&gt;

&lt;p&gt;I ask, because writing to the operational tree is normally done by the statsmanager in response to the switch reporting back the flow.  The only way off the top of my head I could think of that we would see the flow in the operational tree if it wasn&apos;t on the switch is if you guys are writing to both config and operational store (not accusing, just trying to eliminate possibilities).&lt;/p&gt;</comment>
                            <comment id="56205" author="mavenugo@gmail.com" created="Sat, 20 Sep 2014 12:20:30 +0000"  >&lt;p&gt;OVSDB debug logs : &lt;a href=&quot;https://gist.github.com/17f081e1dfe37c9ee56d&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/17f081e1dfe37c9ee56d&lt;/a&gt;&lt;br/&gt;
The above logs are from writeFlow() method in AbstractServiceInstance.java&lt;/p&gt;

&lt;p&gt;        CheckedFuture&amp;lt;Void, TransactionCommitFailedException&amp;gt; commitFuture = modification.submit();&lt;br/&gt;
        try &lt;/p&gt;
{
            commitFuture.get();  // TODO: Make it async (See CONTROLLER-624)
            logger.debug(&quot;Transaction success for write of Flow &quot;+flowBuilder.getFlowName());
            Thread.sleep(500);
        }
&lt;p&gt; catch (Exception e) &lt;/p&gt;
{
            logger.error(e.getMessage(), e);
            modification.cancel();
        }</comment>
                            <comment id="56206" author="mavenugo@gmail.com" created="Sat, 20 Sep 2014 12:24:16 +0000"  >&lt;p&gt;(In reply to Ed Warnicke from comment #2)&lt;br/&gt;
&amp;gt; Madhu,&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; Question... are you guys writing &lt;b&gt;only&lt;/b&gt; to the config tree?&lt;/p&gt;

&lt;p&gt;Yes. we write ONLY to the config tree.&lt;/p&gt;

&lt;p&gt;&amp;gt; &lt;br/&gt;
&amp;gt; I ask, because writing to the operational tree is normally done by the&lt;br/&gt;
&amp;gt; statsmanager in response to the switch reporting back the flow.  The only&lt;br/&gt;
&amp;gt; way off the top of my head I could think of that we would see the flow in&lt;br/&gt;
&amp;gt; the operational tree if it wasn&apos;t on the switch is if you guys are writing&lt;br/&gt;
&amp;gt; to both config and operational store (not accusing, just trying to eliminate&lt;br/&gt;
&amp;gt; possibilities).&lt;/p&gt;

&lt;p&gt;I know, that is why I don&apos;t know what to do with this issue &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.opendaylight.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;.&lt;br/&gt;
Everything seems normal and appropriate from the state perspective, but the flows are missing.&lt;/p&gt;</comment>
                            <comment id="56207" author="tony.tkacik@gmail.com" created="Sat, 20 Sep 2014 15:04:49 +0000"  >&lt;p&gt;Quick questions: Are you able to see all your flows in configuration store? IF yes, this is bug of Openflowplugin/FRM/Stats Manager and not MD-SAL Infrastructure.&lt;/p&gt;

&lt;p&gt;This multiple threads are writing to datastore? &lt;br/&gt;
Or which multiple threads are you referencing?&lt;/p&gt;

&lt;p&gt;E.g callback from datastore and LLDP discovery thread?&lt;/p&gt;</comment>
                            <comment id="56208" author="mavenugo@gmail.com" created="Sat, 20 Sep 2014 18:28:56 +0000"  >&lt;p&gt;Reproduced again.. this time with pcap :&lt;/p&gt;

&lt;p&gt;Flows : &lt;a href=&quot;https://gist.github.com/1926c569ff46bfcfac74&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/1926c569ff46bfcfac74&lt;/a&gt;&lt;br/&gt;
(Missing DEFAULT_PIPELINE_FLOW_0 flow)&lt;/p&gt;

&lt;p&gt;Config : &lt;a href=&quot;https://gist.github.com/aaaa5d72a499fca19127&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/aaaa5d72a499fca19127&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Operational : &lt;a href=&quot;https://gist.github.com/592be004ee3fd21dc1c8&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/592be004ee3fd21dc1c8&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;OVSDB Debug logs : &lt;a href=&quot;https://gist.github.com/e31255c6ea9a82196e45&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://gist.github.com/e31255c6ea9a82196e45&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;PCAP : &lt;a href=&quot;https://www.dropbox.com/s/9rgolqu0x7enc9p/pipeline_flows_2.pcap?dl=0&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://www.dropbox.com/s/9rgolqu0x7enc9p/pipeline_flows_2.pcap?dl=0&lt;/a&gt;&lt;br/&gt;
[Filter with openflow_v4.type == 14 to see all the flowmods &amp;amp; you will see that the DEFAULT_PIPELINE_FLOW_0 (table =0 , goto_table=20) is missing.&lt;/p&gt;</comment>
                            <comment id="56209" author="mavenugo@gmail.com" created="Sat, 20 Sep 2014 18:31:07 +0000"  >&lt;p&gt;(In reply to Tony Tkacik from comment #5)&lt;br/&gt;
&amp;gt; Quick questions: Are you able to see all your flows in configuration store?&lt;br/&gt;
Yes.&lt;/p&gt;

&lt;p&gt;&amp;gt; IF yes, this is bug of Openflowplugin/FRM/Stats Manager and not MD-SAL&lt;br/&gt;
&amp;gt; Infrastructure.&lt;/p&gt;

&lt;p&gt;Could be. But when am opening the bug, it is difficult for me to spot it and hence opened it against MD-SAL. Please move it to appropriate component once the&lt;br/&gt;
root-cause is identified.&lt;/p&gt;

&lt;p&gt;&amp;gt; &lt;br/&gt;
&amp;gt; This multiple threads are writing to datastore? &lt;br/&gt;
&amp;gt; Or which multiple threads are you referencing?&lt;/p&gt;

&lt;p&gt;Both these threads are internal to OVSDB codebase.&lt;br/&gt;
But you can think as 2 different threads both registering to the inventory update event which triggers these flow programming.&lt;/p&gt;

&lt;p&gt;&amp;gt; &lt;br/&gt;
&amp;gt; E.g callback from datastore and LLDP discovery thread?&lt;/p&gt;</comment>
                            <comment id="56210" author="alagalah" created="Sun, 21 Sep 2014 14:02:40 +0000"  >&lt;p&gt;Was requested by Ed Warnicke to provide some data on this bug from the GroupBasedPolicy ProofOfConcept perspective, which we believe is hitting the same issue.&lt;/p&gt;

&lt;p&gt;The net that we have discovered is that flows are missing in our Proof of Concept (POC) which involves two OVS instances (each on a separate VM).&lt;/p&gt;

&lt;p&gt;Essentially, VM:odlgbp1 runs the OpenDaylight controller AND an OVS instance odlgbp2 runs an OVS instance (both in mininet).&lt;/p&gt;

&lt;p&gt;OVA&apos;s of both these VMs are available in my DropBox and access can be granted on request and Thomas and I are also available to share our test environment and run additional tests.&lt;/p&gt;

&lt;p&gt;Some of our debug steps have been captured in the following google doc (please note, this is our &quot;scratchpad&quot; so we can document some troubleshooting history, not a clean definitive document) :&lt;br/&gt;
&lt;a href=&quot;https://docs.google.com/document/d/1cTnBBdQFaoR1hvsdHFuKlzcQKU1bvxvT8lCtBaVWTjQ/edit?usp=sharing&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://docs.google.com/document/d/1cTnBBdQFaoR1hvsdHFuKlzcQKU1bvxvT8lCtBaVWTjQ/edit?usp=sharing&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;To simplify:&lt;/p&gt;

&lt;p&gt;1. We see the requisite expected flows via REST. ASSUMPTION: Flows are correctly placed into the CONFIG datastore.&lt;br/&gt;
2. We see the requisite expected flows via Karaf debugging (see link above for specific debug levels) and have correlated the flowIDs between REST and Karaf logs via a python script. ASSUMPTION: Requisite flows are being passed to FRM.&lt;br/&gt;
3. Via jConsole, we see the correct number of messages being ENQUEUED/DEQUEUED. ASSUMPTION: Requisite flows are being passed through OpenFlowPlugin&lt;br/&gt;
4. Wireshark. Have attached wireshark trace, still parsing this.&lt;/p&gt;

&lt;p&gt;Per Thomas&apos; comments below: The net of this is that it appears that the flows make it to openflowjava, but get lost somewhere thereafter.&lt;/p&gt;

&lt;p&gt;From Thomas Bachman:&lt;br/&gt;
*********************&lt;br/&gt;
TARBALL of data here:&lt;a href=&quot;https://drive.google.com/a/noironetworks.com/#folders/0BztNICcppsJKa1B6WXlFYmFtWFE&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://drive.google.com/a/noironetworks.com/#folders/0BztNICcppsJKa1B6WXlFYmFtWFE&lt;/a&gt;&lt;br/&gt;
The tarball has the following directories:&lt;/p&gt;

&lt;p&gt;configstore/&lt;br/&gt;
jconsole/&lt;br/&gt;
karaf-logs/&lt;br/&gt;
ovs-logs&lt;br/&gt;
wireshark/&lt;/p&gt;

&lt;p&gt;The configstore dir contains the result of REST queries to /restconf/config/opendaylight-inventory:nodes/node/openflow:&amp;lt;n&amp;gt;.&lt;/p&gt;

&lt;p&gt;The jconsole directory contains a text file of the stats from the msg-spy-service-impl RuntimeBean.&lt;/p&gt;

&lt;p&gt;The karaf-log directory contains the karaf log files obtained during the run (note: things to look for are &quot;xid&quot; for openflowjava and &quot;Calling the FlowMod RPC method on MessageDispatchService AddFlow&quot; for openflowplugin).&lt;/p&gt;

&lt;p&gt;The ovs-logs dir contains the result of running ovs-ofctl&lt;br/&gt;
dump-flows &lt;span class=&quot;error&quot;&gt;&amp;#91;s1|s2&amp;#93;&lt;/span&gt; -O OpenFlow13 on each vSwitch.&lt;/p&gt;

&lt;p&gt;The wireshark dir contains the pcapng file and a text file containing the openflow messages seen on the wire between the two vSwitches.&lt;/p&gt;

&lt;p&gt;The net of this is that it appears that the flows make it to openflowjava, but get lost somewhere thereafter. &lt;br/&gt;
*********************&lt;/p&gt;</comment>
                            <comment id="56211" author="hagbard@gmail.com" created="Sun, 21 Sep 2014 14:11:38 +0000"  >&lt;p&gt;Madhu, &lt;/p&gt;

&lt;p&gt; Could you attach:&lt;/p&gt;

&lt;p&gt;a) Your config datastore output from RESTCONF&lt;br/&gt;
b) Your expected flows on the switch&lt;br/&gt;
c) Your actual flows on the switch&lt;br/&gt;
d) Your wireshark capture showing missing flows&lt;/p&gt;

&lt;p&gt;to the bug so we can look at that case?&lt;/p&gt;</comment>
                            <comment id="56212" author="mavenugo@gmail.com" created="Sun, 21 Sep 2014 14:14:17 +0000"  >&lt;p&gt;(In reply to Ed Warnicke from comment #9)&lt;br/&gt;
&amp;gt; Madhu, &lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt;  Could you attach:&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; a) Your config datastore output from RESTCONF&lt;br/&gt;
&amp;gt; b) Your expected flows on the switch&lt;br/&gt;
&amp;gt; c) Your actual flows on the switch&lt;br/&gt;
&amp;gt; d) Your wireshark capture showing missing flows&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; to the bug so we can look at that case?&lt;/p&gt;

&lt;p&gt;I already did that yesterday :&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://bugs.opendaylight.org/show_bug.cgi?id=1997#c6&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://bugs.opendaylight.org/show_bug.cgi?id=1997#c6&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="56213" author="hagbard@gmail.com" created="Sun, 21 Sep 2014 15:36:44 +0000"  >&lt;p&gt;Madhu,&lt;/p&gt;

&lt;p&gt;Apologies &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.opendaylight.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;  You are correct, you did that yesterday &lt;img class=&quot;emoticon&quot; src=&quot;https://jira.opendaylight.org/images/icons/emoticons/smile.png&quot; height=&quot;16&quot; width=&quot;16&quot; align=&quot;absmiddle&quot; alt=&quot;&quot; border=&quot;0&quot;/&gt;&lt;/p&gt;</comment>
                            <comment id="56214" author="alagalah" created="Sun, 21 Sep 2014 15:39:16 +0000"  >
&lt;p&gt;&amp;gt; The wireshark dir contains the pcapng file and a text file containing the&lt;br/&gt;
&amp;gt; openflow messages seen on the wire between the two vSwitches.&lt;/p&gt;

&lt;p&gt;Applying following filter:&lt;br/&gt;
openflow_v4.type == 14 &amp;amp;&amp;amp; (openflow_v4.flowmod.priority ==1 || 	openflow_v4.flowmod.priority ==100 || 	openflow_v4.flowmod.priority ==110 || 	openflow_v4.flowmod.priority ==111 || 	openflow_v4.flowmod.priority ==112 || 	openflow_v4.flowmod.priority ==120 || 	openflow_v4.flowmod.priority ==121 || 	openflow_v4.flowmod.priority ==132 || 	openflow_v4.flowmod.priority ==140 || 	openflow_v4.flowmod.priority ==150 || 	openflow_v4.flowmod.priority ==200 || 	openflow_v4.flowmod.priority ==50 || 	openflow_v4.flowmod.priority ==64999 || 	openflow_v4.flowmod.priority ==65000) &lt;/p&gt;

&lt;p&gt;and manually counting the expanded OpenFlow 1.3 messages in each TCP message we get 116 across both switches.&lt;/p&gt;

&lt;p&gt;ie. Filtering on FlowMODs (14) and also the priorities we set in the GBP project, we should have 114 flows (57 per OVS instances). We are still tracking down the extra 1 flow per switch but suspect it is administration outside of our project.&lt;/p&gt;

&lt;p&gt;BUT&lt;/p&gt;

&lt;p&gt;Executing &amp;lt;sudo ovs-ofctl dump-flows &amp;lt;switch&amp;gt; -O Openflow13 | wc -l &amp;gt; on each OVS we get:&lt;br/&gt;
host1 (s1): 55 (we discount the header as wc -l returns 56 which inc. a header)&lt;br/&gt;
host2 (s2): 57 (as above)&lt;/p&gt;

&lt;p&gt;This is perplexing as it would appear we are getting 116 flowmods (as per filter above) on the wire but only 112 flows total installed.&lt;/p&gt;</comment>
                            <comment id="56215" author="mirehak@cisco.com" created="Mon, 22 Sep 2014 21:01:02 +0000"  >&lt;p&gt;moved barrier after message:&lt;br/&gt;
&lt;a href=&quot;https://git.opendaylight.org/gerrit/#/c/11457/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/#/c/11457/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="56216" author="hagbard@gmail.com" created="Mon, 22 Sep 2014 22:22:13 +0000"  >&lt;p&gt;Fix: &lt;a href=&quot;https://git.opendaylight.org/gerrit/#/c/11457/&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/#/c/11457/&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="56217" author="mirehak@cisco.com" created="Tue, 23 Sep 2014 06:55:02 +0000"  >&lt;p&gt;Mentioned patch eliminates collisions during add session and remove session phases. Those can occur simultaneously and a collision usually results into NPE in remove session thread and registered but invalid session. &lt;/p&gt;

&lt;p&gt;Those can cause message lost and failing rpcs.&lt;/p&gt;

&lt;p&gt;Please retest and eventually mark as resolved.&lt;/p&gt;</comment>
                    </comments>
                <issuelinks>
                            <issuelinktype id="10002">
                    <name>Duplicate</name>
                                                                <inwardlinks description="is duplicated by">
                                        <issuelink>
            <issuekey id="27540">OPNFLWPLUG-272</issuekey>
        </issuelink>
                            </inwardlinks>
                                    </issuelinktype>
                    </issuelinks>
                <attachments>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                            <customfield id="customfield_11400" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10208" key="com.atlassian.jira.plugin.system.customfieldtypes:textfield">
                        <customfieldname>External issue ID</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>1997</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10201" key="com.atlassian.jira.plugin.system.customfieldtypes:url">
                        <customfieldname>External issue URL</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[https://bugs.opendaylight.org/show_bug.cgi?id=1997]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10204" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>ODL SR Target Milestone</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10372"><![CDATA[Helium-RC2]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10000" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i0308n:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>