<!-- 
RSS generated by JIRA (8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d) at Wed Feb 07 20:05:45 UTC 2024

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary append 'field=key&field=summary' to the URL of your request.
-->
<rss version="0.92" >
<channel>
    <title>OpenDaylight JIRA</title>
    <link>https://jira.opendaylight.org</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>8.20.10</version>
        <build-number>820010</build-number>
        <build-date>22-06-2022</build-date>
    </build-info>


<item>
            <title>[L2SWITCH-38] basic openflow network breaks after short cbench test with l2switch feature installed</title>
                <link>https://jira.opendaylight.org/browse/L2SWITCH-38</link>
                <project id="10134" key="L2SWITCH">l2switch</project>
                    <description>&lt;p&gt;TL;DR  using a larger number of switches in a cbench test breaks the openflow-ness such&lt;br/&gt;
that a simple mininet network will not work.&lt;/p&gt;




&lt;p&gt;Test Details:&lt;/p&gt;

&lt;p&gt;Setup:  install these features:&lt;/p&gt;
&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;odl-l2switch-all&lt;/li&gt;
	&lt;li&gt;odl-l2switch-switch-rest&lt;/li&gt;
	&lt;li&gt;odl-l2switch-switch-ui&lt;/li&gt;
	&lt;li&gt;odl-dlux-all&lt;/li&gt;
&lt;/ul&gt;


&lt;p&gt;Steps:&lt;br/&gt;
1) start small mininet topo (tree,1,3) and verify successful pingall, then quit&lt;br/&gt;
2) send brief cbench test (-m 5000 -M 10000 -s 64 -l 2 -t)&lt;br/&gt;
3) repeat step 1, but pingall fails and switches are not learned&lt;/p&gt;





&lt;p&gt;Notes:&lt;/p&gt;

&lt;p&gt;in step 1, the switch is learned and it is removed when mininet is done.&lt;/p&gt;

&lt;p&gt;after step 2, there are leftover switches in the operational store.&lt;/p&gt;

&lt;p&gt;once this state is seen, the logout (exiting karaf console) process takes a very long time&lt;br/&gt;
(or never exits), and I just kill the java process so I don&#8217;t have to wait.&lt;br/&gt;
see at the end of the logs here:  &lt;a href=&quot;http://pastebin.com/Kj4hdves&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;http://pastebin.com/Kj4hdves&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The cbench test simulates X openflow 1.0 switches all connecting.  When X == 32 this &lt;br/&gt;
problem doesn&#8217;t always happen, but maybe on the second iteration of steps 1-3.  With&lt;br/&gt;
64 it looks like it happens every time.&lt;/p&gt;

&lt;p&gt;I took a packet capture of the mininet network connecting in the problem state and the&lt;br/&gt;
handshake looks ok.  After the final ACK is given by the controller to the MULTIPART&lt;br/&gt;
REPLY the only traffic is originating from the switch (e.g. port status msg, or packet-in&#8217;s&lt;br/&gt;
triggered by data plane traffic &#8212; pingall).  In a working environment we&#8217;ll see LLDP&lt;br/&gt;
packet out learning, default flows being pushed, etc.  None of that is happening&lt;br/&gt;
here.&lt;/p&gt;</description>
                <environment>&lt;p&gt;Operating System: All&lt;br/&gt;
Platform: All&lt;/p&gt;</environment>
        <key id="21551">L2SWITCH-38</key>
            <summary>basic openflow network breaks after short cbench test with l2switch feature installed</summary>
                <type id="10104" iconUrl="https://jira.opendaylight.org/secure/viewavatar?size=xsmall&amp;avatarId=10303&amp;avatarType=issuetype">Bug</type>
                                                <status id="10004" iconUrl="https://jira.opendaylight.org/images/icons/status_generic.gif" description="">Verified</status>
                    <statusCategory id="3" key="done" colorName="green"/>
                                    <resolution id="10000">Done</resolution>
                                        <assignee username="ajayl.bro@gmail.com">Ajay L</assignee>
                                    <reporter username="jluhrsen">Jamo Luhrsen</reporter>
                        <labels>
                    </labels>
                <created>Wed, 25 Mar 2015 20:47:42 +0000</created>
                <updated>Mon, 30 Oct 2017 15:41:44 +0000</updated>
                            <resolved>Tue, 3 May 2016 17:24:43 +0000</resolved>
                                    <version>unspecified</version>
                                                    <component>General</component>
                        <due></due>
                            <votes>0</votes>
                                    <watches>5</watches>
                                                                                                                <comments>
                            <comment id="40334" author="jluhrsen" created="Wed, 25 Mar 2015 20:47:42 +0000"  >&lt;p&gt;Attachment mnWorks.pcap has been added with description: what mininet-to-controller traffic looks like in a working setup&lt;/p&gt;</comment>
                            <comment id="40335" author="jluhrsen" created="Wed, 25 Mar 2015 20:48:04 +0000"  >&lt;p&gt;Attachment mnBroken.pcap has been added with description: what mininet-to-controller traffic looks like in a broken setup&lt;/p&gt;</comment>
                            <comment id="40320" author="jluhrsen" created="Tue, 14 Apr 2015 17:51:25 +0000"  >&lt;p&gt;with Lithium build from 4/14/2015, this issue is there with throughput&lt;br/&gt;
mode (-t), but not there in latency mode (default mode)&lt;/p&gt;

&lt;p&gt;additionally, running this test twice resulted in an OOM:&lt;/p&gt;

&lt;p&gt;cbench -c 172.17.7.23 -m 30000 -M 10000 -s 64 -l 10 -t&lt;/p&gt;

&lt;p&gt;The OOM trace:&lt;/p&gt;

&lt;p&gt;Exception in thread &quot;Thread-2&quot; java.lang.OutOfMemoryError: GC overhead limit exceeded&lt;br/&gt;
	at sun.nio.cs.UTF_8.newEncoder(UTF_8.java:72)&lt;br/&gt;
	at java.lang.StringCoding$StringEncoder.&amp;lt;init&amp;gt;(StringCoding.java:282)&lt;br/&gt;
	at java.lang.StringCoding$StringEncoder.&amp;lt;init&amp;gt;(StringCoding.java:273)&lt;br/&gt;
	at java.lang.StringCoding.encode(StringCoding.java:338)&lt;br/&gt;
	at java.lang.String.getBytes(String.java:916)&lt;br/&gt;
	at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)&lt;br/&gt;
	at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)&lt;br/&gt;
	at java.io.File.isDirectory(File.java:843)&lt;br/&gt;
	at org.apache.karaf.main.Main.doMonitor(Main.java:283)&lt;br/&gt;
	at org.apache.karaf.main.Main.access$100(Main.java:69)&lt;br/&gt;
	at org.apache.karaf.main.Main$1.run(Main.java:271)&lt;br/&gt;
Exception in thread &quot;Thread-104&quot; java.util.concurrent.RejectedExecutionException: Task org.opendaylight.openflowplugin.openflow.md.core.HandshakeStepWrapper@29295cff rejected from org.opendaylight.openflowplugin.openflow.md.core.ThreadPoolLoggingExecutor@5a987e75&lt;span class=&quot;error&quot;&gt;&amp;#91;Shutting down, pool size = 0, active threads = 0, queued tasks = 1, completed tasks = 0&amp;#93;&lt;/span&gt;&lt;br/&gt;
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048)&lt;br/&gt;
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)&lt;br/&gt;
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)&lt;br/&gt;
	at org.opendaylight.openflowplugin.openflow.md.core.ConnectionConductorImpl.onConnectionReady(ConnectionConductorImpl.java:450)&lt;br/&gt;
	at org.opendaylight.openflowjava.protocol.impl.core.connection.ConnectionAdapterImpl$3.run(ConnectionAdapterImpl.java:449)&lt;br/&gt;
	at java.lang.Thread.run(Thread.java:744)&lt;/p&gt;</comment>
                            <comment id="40321" author="jluhrsen" created="Tue, 14 Apr 2015 17:52:45 +0000"  >&lt;p&gt;Also, in regards to the last comment, all four cores on the test system are running at 100% and the java processes running the controller needed to be forcefully killed.&lt;/p&gt;</comment>
                            <comment id="40322" author="abhijit2511" created="Mon, 1 Jun 2015 16:57:24 +0000"  >&lt;p&gt;Jamo,&lt;/p&gt;

&lt;p&gt;Can you update this bug after retest? Martin says this should not be an issue any more.&lt;/p&gt;

&lt;p&gt;Abhijit&lt;/p&gt;</comment>
                            <comment id="40323" author="jluhrsen" created="Mon, 1 Jun 2015 22:18:47 +0000"  >&lt;p&gt;I am still seeing this as described in a distribution built today (06/01/2015):&lt;/p&gt;

&lt;p&gt;distribution-karaf-0.3.0-20150601.211237-2105.zip&lt;/p&gt;

&lt;p&gt;looks like the cbench switches are not removed after that step is complete.&lt;/p&gt;

&lt;p&gt;let me know what deeper debugs I can provide, if any.&lt;/p&gt;</comment>
                            <comment id="40324" author="abhijit2511" created="Tue, 2 Jun 2015 01:37:00 +0000"  >&lt;p&gt;Assigning it to Martin - but feel free to reassign.&lt;/p&gt;</comment>
                            <comment id="40325" author="jluhrsen" created="Wed, 3 Jun 2015 16:19:29 +0000"  >&lt;p&gt;more details as we are deciding if this needs to be fixed for Lithium.&lt;/p&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;this is not an issue with Helium SR3&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;this is happening with the Helium plugin codebase with a distribution built 06/02/2015&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;some other symptoms:&lt;/li&gt;
&lt;/ul&gt;
&lt;ul&gt;
	&lt;li&gt;the mock cbench switches are not removed from operational after the short cbench test is run.&lt;/li&gt;
	&lt;li&gt;it doesn&apos;t appear that any new switch can connect at this point and I have tried with switches&lt;br/&gt;
     that will not conflict with the switch id.&lt;/li&gt;
	&lt;li&gt;one of the systems 4 CPUs is constantly pegged at 100%&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;the problem is not there if I only install odl-openflowplugin-flow-services-ui.  It requires&lt;br/&gt;
me to install odl-l2-switch-switch-ui.  I suppose that might need us to involve the l2switch team,&lt;br/&gt;
but obviously something openflow-ish is broken.&lt;/li&gt;
&lt;/ul&gt;


&lt;ul class=&quot;alternate&quot; type=&quot;square&quot;&gt;
	&lt;li&gt;because of the last point, I am not able to try this test with the Lithium redesign since enabling&lt;br/&gt;
l2switch will depend on, and install the Helium codebase.&lt;/li&gt;
&lt;/ul&gt;
</comment>
                            <comment id="40336" author="jluhrsen" created="Mon, 20 Jul 2015 17:12:48 +0000"  >&lt;p&gt;Attachment 1395_Lithium_data.ods has been added with description: re-test with Lithium&lt;/p&gt;</comment>
                            <comment id="40326" author="jluhrsen" created="Mon, 20 Jul 2015 17:13:45 +0000"  >&lt;p&gt;(In reply to Jamo Luhrsen from comment #8)&lt;br/&gt;
&amp;gt; Created attachment 573 &lt;span class=&quot;error&quot;&gt;&amp;#91;details&amp;#93;&lt;/span&gt;&lt;br/&gt;
&amp;gt; re-test with Lithium&lt;/p&gt;

&lt;p&gt;ignore this comment.  was meant for another bug.&lt;/p&gt;</comment>
                            <comment id="40327" author="colin@colindixon.com" created="Tue, 19 Jan 2016 19:20:22 +0000"  >&lt;p&gt;Do we know if this is still an issue in Beryllum?&lt;/p&gt;</comment>
                            <comment id="40328" author="jluhrsen" created="Wed, 20 Jan 2016 00:26:01 +0000"  >&lt;p&gt;(In reply to Colin Dixon from comment #10)&lt;br/&gt;
&amp;gt; Do we know if this is still an issue in Beryllum?&lt;/p&gt;

&lt;p&gt;yes, the high level issue does still seem to be there in Beryllium (used a &lt;br/&gt;
distro built 01/19/2016).  I did use a larger number of cbench switches than&lt;br/&gt;
in inital description.&lt;/p&gt;

&lt;p&gt;I also saw the same high level issue with Lithium SR3.&lt;/p&gt;

&lt;p&gt;I say &quot;high level&quot;, because I am not confident that the root cause is&lt;br/&gt;
the same.&lt;/p&gt;</comment>
                            <comment id="40329" author="ajayl.bro@gmail.com" created="Wed, 24 Feb 2016 00:44:01 +0000"  >&lt;p&gt;I am able to repro the high CPU issue in Beryllium using cbench commands mentioned. Threaddump shows that after the cbench test has run, below thread hogs the CPU&lt;/p&gt;

&lt;p&gt;&quot;pool-28-thread-2&quot; prio=10 tid=0x00007f6c9c566000 nid=0x2b38 runnable &lt;span class=&quot;error&quot;&gt;&amp;#91;0x00007f6c7c464000&amp;#93;&lt;/span&gt;&lt;br/&gt;
   java.lang.Thread.State: RUNNABLE&lt;br/&gt;
        at com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:56)&lt;br/&gt;
        at com.lmax.disruptor.PhasedBackoffWaitStrategy.waitFor(PhasedBackoffWaitStrategy.java:102)&lt;br/&gt;
        at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:55)&lt;br/&gt;
        at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)&lt;br/&gt;
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)&lt;br/&gt;
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)&lt;br/&gt;
        at java.lang.Thread.run(Thread.java:745)&lt;/p&gt;


&lt;p&gt;ODL uses LMAX disruptor library to implement DOM notifications. packet-handler decoder classes (e.g. ArpDecoder.java), consume the packet notifications, decode the packet, and publish the event as notification to be consumed by next decoder. All this happens in a single thread. Under high load, this may cause dead-lock as the disruptor ring-buffer will not move till the initial notification handler returns and it cannot return till it gets a slot to post event on the ring-buffer&lt;/p&gt;

&lt;p&gt;Proposed fix is to perform decode-and-publish on a separate thread to not block the initial notification handler. Unit-tested it and seems to work fine&lt;/p&gt;

&lt;p&gt;Jamo - can u pls try in your setup&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://git.opendaylight.org/gerrit/35300&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/35300&lt;/a&gt;&lt;/p&gt;</comment>
                            <comment id="40330" author="jluhrsen" created="Thu, 3 Mar 2016 05:33:57 +0000"  >&lt;p&gt;(In reply to Ajay L from comment #12)&lt;br/&gt;
&amp;gt; I am able to repro the high CPU issue in Beryllium using cbench commands&lt;br/&gt;
&amp;gt; mentioned. Threaddump shows that after the cbench test has run, below thread&lt;br/&gt;
&amp;gt; hogs the CPU&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; &quot;pool-28-thread-2&quot; prio=10 tid=0x00007f6c9c566000 nid=0x2b38 runnable&lt;br/&gt;
&amp;gt; &lt;span class=&quot;error&quot;&gt;&amp;#91;0x00007f6c7c464000&amp;#93;&lt;/span&gt;&lt;br/&gt;
&amp;gt;    java.lang.Thread.State: RUNNABLE&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; com.lmax.disruptor.BlockingWaitStrategy.waitFor(BlockingWaitStrategy.java:56)&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; com.lmax.disruptor.PhasedBackoffWaitStrategy.&lt;br/&gt;
&amp;gt; waitFor(PhasedBackoffWaitStrategy.java:102)&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; com.lmax.disruptor.ProcessingSequenceBarrier.&lt;br/&gt;
&amp;gt; waitFor(ProcessingSequenceBarrier.java:55)&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:123)&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:&lt;br/&gt;
&amp;gt; 1145)&lt;br/&gt;
&amp;gt;         at&lt;br/&gt;
&amp;gt; java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:&lt;br/&gt;
&amp;gt; 615)&lt;br/&gt;
&amp;gt;         at java.lang.Thread.run(Thread.java:745)&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; ODL uses LMAX disruptor library to implement DOM notifications.&lt;br/&gt;
&amp;gt; packet-handler decoder classes (e.g. ArpDecoder.java), consume the packet&lt;br/&gt;
&amp;gt; notifications, decode the packet, and publish the event as notification to&lt;br/&gt;
&amp;gt; be consumed by next decoder. All this happens in a single thread. Under high&lt;br/&gt;
&amp;gt; load, this may cause dead-lock as the disruptor ring-buffer will not move&lt;br/&gt;
&amp;gt; till the initial notification handler returns and it cannot return till it&lt;br/&gt;
&amp;gt; gets a slot to post event on the ring-buffer&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; Proposed fix is to perform decode-and-publish on a separate thread to not&lt;br/&gt;
&amp;gt; block the initial notification handler. Unit-tested it and seems to work fine&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; Jamo - can u pls try in your setup&lt;br/&gt;
&amp;gt; &lt;br/&gt;
&amp;gt; &lt;a href=&quot;https://git.opendaylight.org/gerrit/35300&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://git.opendaylight.org/gerrit/35300&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Ajay, I do intend to look at this, but I am going to need more time to &lt;br/&gt;
catch up on other things.  I will report back here when I am able to &lt;br/&gt;
check.&lt;/p&gt;</comment>
                            <comment id="40331" author="ajayl.bro@gmail.com" created="Tue, 8 Mar 2016 16:51:54 +0000"  >&lt;p&gt;Thx Jamo. Let us know once u get a chance to try&lt;/p&gt;</comment>
                            <comment id="40332" author="jluhrsen" created="Tue, 22 Mar 2016 03:57:27 +0000"  >&lt;p&gt;(In reply to Ajay L from comment #14)&lt;br/&gt;
&amp;gt; Thx Jamo. Let us know once u get a chance to try&lt;/p&gt;

&lt;p&gt;Ajay,  sorry for being slow on this one.  I was able to take the Boron&lt;br/&gt;
distribution created by your patch&apos;s verify job and compare it to the&lt;br/&gt;
latest boron distribution (which does not have your changes).&lt;/p&gt;

&lt;p&gt;good news!  The problem is fixed.  &lt;/p&gt;

&lt;p&gt;I could reproduce with current boron distro, and saw things fixed using&lt;br/&gt;
the distro from your patch.&lt;/p&gt;

&lt;p&gt;I did not test with the stable/beryllium patch, as I had trouble tracking&lt;br/&gt;
down the distro created by that patch.&lt;/p&gt;

&lt;p&gt;good work.&lt;/p&gt;</comment>
                            <comment id="40333" author="jluhrsen" created="Tue, 3 May 2016 17:24:43 +0000"  >&lt;p&gt;I&apos;ve also verified on Beryllium SR candidate distro:&lt;br/&gt;
&lt;a href=&quot;https://wiki.opendaylight.org/view/Simultaneous_Release:Beryllium_Release_Plan#Beryllium_SR2_Download&quot; class=&quot;external-link&quot; target=&quot;_blank&quot; rel=&quot;nofollow noopener&quot;&gt;https://wiki.opendaylight.org/view/Simultaneous_Release:Beryllium_Release_Plan#Beryllium_SR2_Download&lt;/a&gt;&lt;/p&gt;</comment>
                    </comments>
                    <attachments>
                            <attachment id="12747" name="1395_Lithium_data.ods" size="19159" author="jluhrsen" created="Mon, 20 Jul 2015 17:12:48 +0000"/>
                            <attachment id="12746" name="mnBroken.pcap" size="6070" author="jluhrsen" created="Wed, 25 Mar 2015 20:48:04 +0000"/>
                            <attachment id="12745" name="mnWorks.pcap" size="38200" author="jluhrsen" created="Wed, 25 Mar 2015 20:47:42 +0000"/>
                    </attachments>
                <subtasks>
                    </subtasks>
                <customfields>
                                                                            <customfield id="customfield_11400" key="com.atlassian.jira.plugins.jira-development-integration-plugin:devsummary">
                        <customfieldname>Development</customfieldname>
                        <customfieldvalues>
                            
                        </customfieldvalues>
                    </customfield>
                                                                                                                        <customfield id="customfield_10208" key="com.atlassian.jira.plugin.system.customfieldtypes:textfield">
                        <customfieldname>External issue ID</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>2897</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                <customfield id="customfield_10201" key="com.atlassian.jira.plugin.system.customfieldtypes:url">
                        <customfieldname>External issue URL</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue><![CDATA[https://bugs.opendaylight.org/show_bug.cgi?id=2897]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                                                                                                                                                                                                                                                <customfield id="customfield_10204" key="com.atlassian.jira.plugin.system.customfieldtypes:select">
                        <customfieldname>ODL SR Target Milestone</customfieldname>
                        <customfieldvalues>
                                <customfieldvalue key="10317"><![CDATA[Beryllium]]></customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                    <customfield id="customfield_10000" key="com.pyxis.greenhopper.jira:gh-lexo-rank">
                        <customfieldname>Rank</customfieldname>
                        <customfieldvalues>
                            <customfieldvalue>0|i01z5r:</customfieldvalue>

                        </customfieldvalues>
                    </customfield>
                                                                                                                                                                                </customfields>
    </item>
</channel>
</rss>