OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 [NETCONF-938] Cannot generate API docs for Junos device https://jira.opendaylight.org/browse/NETCONF-938 netconf <p>When we have device that contains Junos [common models|<a href="https://github.com/Juniper/yang/tree/master/22.3/22.3R1/common" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/Juniper/yang/tree/master/22.3/22.3R1/common</a>] and [junos family models|<a href="https://github.com/Juniper/yang/tree/master/22.3/22.3R1/junos" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/Juniper/yang/tree/master/22.3/22.3R1/junos</a>] its not possible to access [its mount point documentation|<a href="http://localhost:8181/apidoc/explorer/index.html?urls.primaryName=17830-sim-device%20resources%20-%20RestConf%20RFC%208040." class="external-link" target="_blank" rel="nofollow noopener">http://localhost:8181/apidoc/explorer/index.html?urls.primaryName=17830-sim-device%20resources%20-%20RestConf%20RFC%208040</a>].</p> <p>The docs are never loaded and we can see that Java VM is out of heap space. After some time the running ODL instance is killed.</p> NETCONF-938 Cannot generate API docs for Junos device Bug Highest In Review Unresolved Ivan Hrasko Ivan Hrasko Thu, 5 Jan 2023 11:43:33 +0000 Thu, 1 Feb 2024 08:59:03 +0000 7.0.0 restconf-openapi 0 3 <p>The DefinitionGenerator class is encountering a heap space issue while processing the <a href="https://git.opendaylight.org/gerrit/c/netconf/+/103968/7/restconf/sal-rest-docgen/src/test/resources/juniper/junos-conf-root%25402022-01-01.yang" class="external-link" target="_blank" rel="nofollow noopener">"junos-conf-root"</a> model.<br/> This issue arises when processing JSON body examples. Despite not being particularly large, this model only creates 100 JSON body examples when a simple test is created with the "junos-conf-root" model and its two import models.</p> <p>The problem lies in the fact that other models create approximately 62 augmentations to the configuration container within the "junos-conf-root" model. This results in the creation of a large number of JSON body examples, around 1 million, which contain a total of 10 million JSON elements.</p> <p>The Swagger API docs process each model and append all examples from every model to an object called "definition" inside <a href="https://github.com/opendaylight/netconf/blob/master/restconf/sal-rest-docgen/src/main/java/org/opendaylight/netconf/sal/rest/doc/swagger/SwaggerObject.java#L21" class="external-link" target="_blank" rel="nofollow noopener">SwaggerObject</a>. This involves appending 10 million elements, which can take up all available memory.</p> <p><b>It is likely that creating 1 million examples for augmentations is not correct and needs to be addressed to fix the issue.</b></p> <p>Code in master branch after creating configuration containers.<br/> <span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/18609/18609_image-2023-03-15-09-23-20-869.png" height="453" width="811" style="border: 0px solid black" /></span></p> <p>I have replaced ObjectNodes with DefinitionObjects and results in numbers for this model:<br/> <span class="image-wrap" style=""><img src="https://jira.opendaylight.org/secure/attachment/18610/18610_image-2023-03-15-09-47-49-518.png" height="392" width="809" style="border: 0px solid black" /></span></p> <p><a href="https://git.opendaylight.org/gerrit/c/netconf/+/104878/2" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/104878/2</a></p> <p>We assume that the high memory consumption is caused by:</p> <ol> <li>Generating unnecessary schema objects, for _XML, _TOP, etc.</li> <li>Unnecessary usage of many ArraNode(s) and ObjectNode(s).</li> <li>Recursion.</li> </ol> <p>Right, but at the end of the day the core problem is how JSON serialization is implemented.</p> <p>What we have is:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> @Override <span class="code-keyword">public</span> <span class="code-keyword">synchronized</span> Response getAllModulesDoc(<span class="code-keyword">final</span> UriInfo uriInfo) { <span class="code-keyword">final</span> DefinitionNames definitionNames = <span class="code-keyword">new</span> DefinitionNames(); <span class="code-keyword">final</span> OpenApiObject doc = openApiGeneratorRFC8040.getAllModulesDoc(uriInfo, definitionNames); <span class="code-keyword">return</span> Response.ok(doc).build(); } </pre> </div></div> <p>which then routes to JaxbContextResolver, which then uses ObjectMapper to turn OpenApiObject into JSON (and uses reflection to determine how to do that).</p> <p>The problem is the fact OpenApiObject exists. It is the moral equivalent of doing Stream.collect() before doing further processing of the stream contents. You have to realize that on input there is EffectiveModelContext and on output there are bytes (which happen to represent a JSON a document).</p> <p>Response.ok(doc) does this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> <span class="code-keyword">public</span> <span class="code-keyword">static</span> ResponseBuilder ok(<span class="code-object">Object</span> entity) { ResponseBuilder b = ok(); b.entity(entity); <span class="code-keyword">return</span> b; } </pre> </div></div> <p>and ResponseBuilder.entity() explicitly says:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Note that the entity can be also set as an {@link java.io.InputStream input stream}. </pre> </div></div> <p>So we really want to integrate at this level &#8211; e.g. provide an InputStream whose bytes are the JSON representation of what OpenApiObject models. At the end of the day, BaseYangOpenApiGenerator, needs to return an OpenApiInputStream from getFilledDoc() &#8211; the constructor of which just captures stuff from UriInfo and the EffectiveModelContext.</p> <p>As for the implementation of said OpenApiInputStream, think of it as a more complicated java.util.Iterator. Essentially it iterates over EffectiveModelContext (maybe multiple times) and each time it gets a piece of the JSON document, which it caches internally and gives out its bytes on InputStream.read(). Whenever it runs out of bytes to give out, it moves to the next bit of EffectiveModelContext, turns it into a JSON fragment, and hands that out &#8211; until it runs out of EffectiveModelContext, at which point it reports -1 to indicate it is all done.</p> <p>I think the core interface here ends up being <a href="https://www.javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.2/com/fasterxml/jackson/core/JsonGenerator.html" class="external-link" target="_blank" rel="nofollow noopener">https://www.javadoc.io/static/com.fasterxml.jackson.core/jackson-core/2.15.2/com/fasterxml/jackson/core/JsonGenerator.html</a>, where the generator is pointed to something that holds bytes for OpenApiInputStream to consume and the EffectiveModelContext-processing logic is provided with the JsonGenerator to use methods writeStartObject() and similar. This obviously needs further investigation, but the goal is clear I think.</p> <p>We already have stuff working on this general principle: see how <a href="https://github.com/opendaylight/yangtools/blob/master/model/yang-model-export/src/main/java/org/opendaylight/yangtools/yang/model/export/DeclaredStatementFormatter.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/yangtools/blob/master/model/yang-model-export/src/main/java/org/opendaylight/yangtools/yang/model/export/DeclaredStatementFormatter.java</a> works once it is configured. toYangTextSnippet() takes either a module, or a submodule and returns an Iterable &#8211; and that in turn defers to <a href="https://github.com/opendaylight/yangtools/blob/master/model/yang-model-export/src/main/java/org/opendaylight/yangtools/yang/model/export/YangTextSnippetIterator.java" class="external-link" target="_blank" rel="nofollow noopener">https://github.com/opendaylight/yangtools/blob/master/model/yang-model-export/src/main/java/org/opendaylight/yangtools/yang/model/export/YangTextSnippetIterator.java</a> &#8211; which converts a DeclaredStatement (and its subtree) to a sequence of Strings.</p> <p>We want to do a similar thing, except:</p> <ul class="alternate" type="square"> <li>it is not DeclaredStatement tree but an EffectiveModelContext tree on input</li> <li>it is not YANG text snippets buts JSON snippet bytes on output</li> <li>the machinery is driven by a request for a byte</li> <li>there is Jackson in between</li> </ul> <p>Perhaps a critical thing to understand: OpenApiInputStream here is consumed by framework is a similar fashion that YangTextSnippet users consume it. InputStream should be viewed as Interator&lt;Integer&gt;, where 'int read()' combines the function of hasNext() and next() (via the return of -1). Plus, obviously, it provides methods to consume bytes in bulk.</p> <p>At the end of the day, there must be no ObjectMapper or com.fasterxml.jackson.annotation in sight.</p> <p>Also, to drive this point home even more, think of EffectiveModelContext as a source of events (like "we have a module", "we have an RPC", "we have an RPC input", "we have a leaf", "we ended the RPC input", "we have ended the RPC", "we have ended the module") and we are turning them into JSON events (like beginObject(), endObject(), etc.) and then turning them into bytes. This is <a href="https://en.wikipedia.org/wiki/Stream_processing" class="external-link" target="_blank" rel="nofollow noopener">stream processing</a> at its <a href="https://en.wikipedia.org/wiki/Dataflow_programming" class="external-link" target="_blank" rel="nofollow noopener">purest</a>.</p> <p>Assuming JAX-RS, we can eliminate some of the complexity by shifing computation to a MessageBodyWriter, when we have the OutputStream available. <a href="https://git.opendaylight.org/gerrit/c/netconf/+/107149" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/107149</a> provides the basic things, but it highlights we really should think about how the model is actually laid out (centered around OpenApiEntity and its specializations).</p> <p>With <a href="https://git.opendaylight.org/gerrit/c/netconf/+/109803" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/netconf/+/109803</a> we can generate and download OpenAPI docs for junos device within 30 seconds with:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>curl -o /home/odl/Desktop/openapi.json --location 'http://localhost:8181/openapi/api/v3/mounts/1' --header 'Authorization: Basic YWRtaW46YWRtaW4='</pre> </div></div> <p>Anyway the whole junos model (1.3 GiB) is not parsable by browsers. We wil try to handle this in <a href="https://jira.opendaylight.org/browse/NETCONF-1225" title="OpenAPI: implement paging" class="issue-link" data-issue-key="NETCONF-1225">NETCONF-1225</a>.</p> Blocks NETCONF-1241 NETCONF-1242 Duplicate NETCONF-480 Relates NETCONF-1054 NETCONF-1056 NETCONF-1057 NETCONF-1234 NETCONF-982 NETCONF-997 NETCONF-1023 NETCONF-1024 NETCONF-1025 NETCONF-1036 NETCONF-1041 NETCONF-1050 NETCONF-939 NETCONF-982 Development Rank 0|i043wn: