https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/PLAS-7 plastic PLAS-7

Users need to specify unfolding/handling of nested array elements in schemas

New Feature Medium Done Done Unassigned Allan Clarke Mon, 27 Jan 2020 20:08:14 +0000 Thu, 30 Jul 2020 15:31:50 +0000 Thu, 30 Jul 2020 15:31:50 +0000 0 1 <p>Today Plastic allows binding on the input side and expansion on the output side for collections (JSon lists)<br/> embedded within a given payload. Today this feature has limitations. </p> <p> ${abc<span class="error">&#91;*&#93;</span>}</p> <p>First, it does not handle nested arrays withtout the schema designer breaking up the translation into <br/> multiple levels using multiple intput/output schemas. Second, it sort-of allows re-use of the asterisk <br/> for sibling arrays (and just hopes that the array lengths are the same). Conceptually there is only a <br/> single asterisk and it represents only a single length.</p> <p> Side note: the sibling array issue either results in a short output or an error, depending on<br/> which array Plastic encounters first. If array A is of size 10 and B is of 20, if Plastic encounters<br/> A first in expanding an output array that uses both A and B, then there will be 10 members of that<br/> output array. If it encounters B first, then when it gets to the 11th expansion, it will fail to <br/> find an 11th value for A and will error out.</p> <p>To handle an arbitrary number of collection sizes, the asterisk needs to represent an arbitrary number<br/> of values through two possible means. Plastic can generalize the current scheme by having an asterisk<br/> usage per level of nesting. You might see this say in a schema using this syntax</p> <p> ${abc<span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span>}</p> <p>This is simple to understand and can handle arbitrary nesting by adding more "<span class="error">&#91;*&#93;</span>" to match the nesting<br/> level.</p> <p> Side note: there is an interesting case of how to flatten. If the input payload has an array<br/> within an array so <span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span> is used and the output has only a single array, then Plastic needs<br/> to map two dimensions into one by flattening.</p> <p>Using only the nesting level and multiple <span class="error">&#91;*&#93;</span> does not fix the sibling array issue, because two sibling<br/> arrays sharing a parent array still might have different lengths.</p> <p>A new expansion feature can do better however.</p> <p>An iterator comes into existence when there are one or more <span class="error">&#91;*&#93;</span> as part of an arrayed variable when<br/> the payload is matched against an input schema. Iterators are not create on the output side, they are<br/> just used to expand arrays. Iterators will have a name that is a derivative of the variable name to<br/> which they were bound.</p> <p>Most importantly, iterators will be merged together so that mixing outputs together like the sibling<br/> array alluded to above are handled intuitively.</p> <p>Lets look at examples. </p> <p>1 - Introducing Iterators<br/> -------------------------</p> <p>Consider this in the INPUT schema...</p> <p> ${abc<span class="error">&#91;*&#93;</span>}</p> <p>Consider this on the OUTPUT schema...</p> <p> ${abc<span class="error">&#91;*&#93;</span>}</p> <p>The new feature would create an iterator with the name 'abc' and the length L (from the payload) when<br/> the payload is bound to the input schema. On the output side, the abc variable will automatically be<br/> using the abc iterator from the input side.</p> <p>So right now our working definition of an iterator is a name and a range 0..(L-1). Later on when we<br/> cover nested arrays, we'll expand the definition of an iterator to be more complex.</p> <p>2 - Independent Sibling Arrays<br/> ------------------------------</p> <p>Payload...</p> { "abc": [ 'aa', 'bb', ... 'zz' ], "colors": [ "red", "green", "blue" ] } <p>Input schema...<br/> {<br/> "abc": [ "${abc<span class="error">&#91;*&#93;</span>}" ],<br/> "colors": [ "${colors<span class="error">&#91;*&#93;</span>}" ]<br/> }</p> <p>Internal bindings...</p> <p> abc<span class="error">&#91;0&#93;</span> = aa<br/> abc<span class="error">&#91;1&#93;</span> = bb<br/> ...<br/> abc<span class="error">&#91;25&#93;</span> = zz</p> <p> _<span class="error">&#91;abc&#93;</span> = (26)</p> <p> colors<span class="error">&#91;0&#93;</span> = red<br/> colors<span class="error">&#91;1&#93;</span> = green<br/> colors<span class="error">&#91;2&#93;</span> = blue</p> <p> _<span class="error">&#91;colors&#93;</span> = (3)</p> <p>Output schema...<br/> {<br/> "foo": [ "${abc<span class="error">&#91;*&#93;</span>}" ],<br/> "bar": [ "${colors<span class="error">&#91;*&#93;</span>}" ]<br/> }</p> <p>Output...</p> { "foo": [ 'aa', 'bb', ... 'zz' ], "bar": [ "red", "green", "blue" ] } <p>We introduce the internal naming for the iterator (_<span class="error">&#91;name&#93;</span>). We introduced the (n,m,...) as a way to <br/> represent the ranges of the indices. They are a list of ints with one per dimension. So (26) above <br/> is a list of one number.</p> <p>Output Steps</p> <ul> <li>Encounter owning collection and push on parent stack with empty iterator</li> <li>Encounter array variable</li> <li>Verify iterator exists and dimensionality matches else ERROR</li> <li>Impose iterator on iterator stack (effectively replaces empty iterator)</li> <li>Start iterate, clone, recurse</li> <li>Pop parent stack including iterator</li> </ul> <p>4 - Mixing Input Arrays into a Single Output Array<br/> --------------------------------------------------</p> <p>Payload...</p> { "aaa": [ 'aa', 'bb', 'cc' ], "colors": [ "red", "green", "blue", "purple" ] } <p>Input schema...<br/> {<br/> "aaa": [ "${aaa<span class="error">&#91;*&#93;</span>}" ],<br/> "colors" [ "${colors<span class="error">&#91;*&#93;</span>}" ]<br/> }</p> <p>Internal bindings...</p> <p> aaa<span class="error">&#91;0&#93;</span> = aa<br/> aaa<span class="error">&#91;1&#93;</span> = bb<br/> aaa<span class="error">&#91;2&#93;</span> = cc</p> <p> _<span class="error">&#91;aaa&#93;</span> = (3)</p> <p> colors<span class="error">&#91;0&#93;</span> = red<br/> colors<span class="error">&#91;1&#93;</span> = green<br/> colors<span class="error">&#91;2&#93;</span> = blue<br/> colors<span class="error">&#91;3&#93;</span> = purple</p> <p> _<span class="error">&#91;colors&#93;</span> = (4)</p> <p>Output schema...<br/> {<br/> "foobar": [ "${aaa<span class="error">&#91;*&#93;</span>=xx}/${colors<span class="error">&#91;*&#93;</span>=gray}" ]<br/> }</p> <p>Output...</p> { "foobar": [ "aa/red", "bb/green", "cc/blue", "xx/purple" ] } <p>The "foobar" array in the output schema is constructed from two arrayed variables "aaa" and<br/> "colors". If the length for aaa is not the same as the length for colors, then this would<br/> result in an error, unless there are defaults present.</p> <p>Output Steps</p> <ul> <li>Encounter owning collection and push on parent stack with empty iterator</li> <li>Encounter array variables</li> <li>Verify iterators exists and dimensionality matches else ERROR</li> <li>Impose iterator aaa on iterator stack (effectively replacing empty iterator)</li> <li>Impose iterator colors</li> </ul> <ul class="alternate" type="square"> <li>existing iterator, so need to merge</li> <li>for each dimension, take the larger range</li> <li>if dimension is missing, then it wins</li> <li>merged iterator now has two names aaa and colors</li> <li>no worry yet about imposing up the parent stack because dimension = 1</li> </ul> <ul> <li>Start iterate, clone, recurse</li> <li>Pop iterator stack when popping parent stack</li> </ul> <p>5 - Multivariable Example<br/> -------------------------<br/> Payload...<br/> {<br/> "deviceName": "CXP-2501",<br/> "interfaceName": "TCP/1/0/24",<br/> "addresses": [<br/> </p> { "address": "10.10.100.100", "prefix-length": "24" } <p>,<br/> </p> { "address": "10.10.100.221", "prefix-length": "24" } <p>,</p> { "address": "10.10.100.168", "prefix-length": "24" } <p> ]<br/> }</p> <p>Input schema...<br/> {<br/> "deviceName": "${dName}",<br/> "interfaceName": "${iName}",<br/> "addresses": [<br/> {<br/> "address": "${addr<span class="error">&#91;*&#93;</span>}",<br/> "prefix-length": "${pref-len<span class="error">&#91;*&#93;</span>}"<br/> }<br/> ]<br/> }</p> <p> addr<span class="error">&#91;0&#93;</span> = 10.10.100.100<br/> addr<span class="error">&#91;1&#93;</span> = 10.10.100.221<br/> addr<span class="error">&#91;2&#93;</span> = 10.10.100.168</p> <p> _<span class="error">&#91;addr&#93;</span> = (3)</p> <p> pref<span class="error">&#91;0&#93;</span> = 24<br/> pref<span class="error">&#91;1&#93;</span> = 24<br/> pref<span class="error">&#91;2&#93;</span> = 24</p> <p> _<span class="error">&#91;pref-len&#93;</span> = (3)</p> <p>Output schema...<br/> {<br/> "dev-name": "${dName}",<br/> "interface-name": "${iName}",<br/> "objects": [<br/> {<br/> "subnet": "${addr<span class="error">&#91;*&#93;</span>}/${pref-len<span class="error">&#91;*&#93;</span>}"<br/> }<br/> ]<br/> }</p> <p>There are two iterators named 'addr' and 'pref-len' that are associated with the parent array.<br/> It turns out that they have the same range.</p> <p>Output Steps</p> <ul> <li>very similar to the previous example</li> </ul> <p>6 - Multi-dimensional Example (two deep)<br/> ----------------------------------------</p> <p>Payload...<br/> {<br/> "aaa": [ <br/> </p> { "id": "1", "bbb": [ 'a', 'b', 'c' ] } <p>,<br/> </p> { "id": "2", "bbb": [ 'd', 'e' ] } <p>,<br/> </p> { "id": "3", "bbb": [ 'f' ] } <p>,</p> { "id": "4", "bbb": [ ] } <p> ]<br/> }</p> <p>Input schema...<br/> {<br/> "aaa": [ &lt;------------ parent/owning array<br/> {<br/> "id": "${id<span class="error">&#91;*&#93;</span>},<br/> "bbb": [ &lt;-------------- child/nested array<br/> "${bbb<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ],<br/> }<br/> ]<br/> }</p> <p>Rule: if "id" or "bbb" is missing, then its an error (that might be resolved through early classifier normalization)</p> <p> id<span class="error">&#91;0&#93;</span> = 1<br/> id<span class="error">&#91;1&#93;</span> = 2<br/> id<span class="error">&#91;2&#93;</span> = 3<br/> id<span class="error">&#91;3&#93;</span> = 4</p> <p> _<span class="error">&#91;id&#93;</span> = (4)</p> <p> bbb<span class="error">&#91;0&#93;</span><span class="error">&#91;0&#93;</span> = a<br/> bbb<span class="error">&#91;0&#93;</span><span class="error">&#91;1&#93;</span> = b<br/> bbb<span class="error">&#91;0&#93;</span><span class="error">&#91;2&#93;</span> = c<br/> bbb<span class="error">&#91;1&#93;</span><span class="error">&#91;0&#93;</span> = d<br/> bbb<span class="error">&#91;1&#93;</span><span class="error">&#91;1&#93;</span> = e<br/> bbb<span class="error">&#91;2&#93;</span><span class="error">&#91;0&#93;</span> = f</p> <p> _<span class="error">&#91;bbb&#93;</span> = (4,3)</p> <p>Note: this example shows that you cannot even count on rectangular matrices (ie, bbb is not 4x3=12)!<br/> And because of this, we may have shortened ranges (they must be dealt with). Worse, they actually<br/> can be real holes and can be at any level of dimension. So even though there are only 6 entries <br/> for bbb, we have to iterate over all 12 possibilities to be sure to cover everything.</p> <p>Note that had bbb been given default values, it would have "filled in the holes" so that it is a<br/> squared off 4x3</p> <p>6A - Flattening<br/> ---------------</p> <p>Output schema...<br/> {<br/> "foo": [<br/> "${id<span class="error">&#91;*&#93;</span>}"<br/> ],<br/> "bar": [<br/> "${bbb<span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ]<br/> }</p> <p>Output...</p> { "foo": [ 1, 2, 3, 4 ], "bar": [ 'a', 'b', 'c', 'd', 'e', 'f' ] } <p>Flattening occurs when the number asterisks is the same as the number of dimensions on the data.</p> <p>Rule: silently ignore indices in bbb that don't exist (to allow for holes in data)</p> <p>Output Steps</p> <ul> <li>Encounter owning collection and push on parent stack with empty iterator</li> <li>Encounter array variable called id</li> <li>Verify iterator exists and dimensionality matches else ERROR</li> <li>Impose iterator id on iterator stack (effectively replacing empty iterator)</li> <li>Start iterate, clone, recurse</li> <li>Pop iterator stack when popping parent stack</li> </ul> <ul> <li>Encounter bar collection and push on parent stack with empty iterator</li> <li>Encounter array variable called bbb</li> <li>Verify iterator exists and dimensionality matches else ERROR</li> <li>Impose iterator id on iterator stack</li> </ul> <ul class="alternate" type="square"> <li>empty iterator is there</li> <li>parent stack is 1 deep (argument to make iterator stack and parent stack be the same)</li> <li>dimensionality is 2</li> <li>all of dimensionality is handled at this depth</li> </ul> <ul> <li>Start iterate, clone, recurse</li> <li>Pop iterator stack when popping parent stack</li> </ul> <p>Similar output schema but top level array and using new ^ ...<br/> [<br/> {<br/> "foo": [<br/> "${id<span class="error">&#91;*&#93;</span>}"<br/> ],<br/> "bar": [<br/> "${bbb<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ]<br/> }<br/> ]</p> <p>Output...<br/> [<br/> </p> { "foo": [ 1, 2, 3, 4 ], "bar": [ 'a', 'b', 'c' ] } <p>,<br/> </p> { "foo": [ 1, 2, 3, 4 ], "bar": [ 'd', 'e' ] } <p>,<br/> </p> { "foo": [ 1, 2, 3, 4 ], "bar": [ 'f' ] } <p>,<br/> </p> { "foo": [ 1, 2, 3, 4 ], "bar": [ ] } <p>,<br/> ]</p> <p>New syntax is <span class="error">&#91;^&#93;</span> which means that this dimension is going to come from a parent array. A<br/> valid syntax is now extended to be zero or more <span class="error">&#91;^&#93;</span> followed by one or more <span class="error">&#91;*&#93;</span>. No alternating<br/> of ^ and *.</p> <p>For inputs, the only acceptable syntax is <span class="error">&#91;^&#93;</span>...<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span>. For no other reason, this is just to be <br/> consistent with the output usage. Syntax of <span class="error">&#91;*&#93;</span>...<span class="error">&#91;*&#93;</span> for two or more dimensions would be an <br/> error.</p> <p>So if there are multiple dimensions of <span class="error">&#91;*&#93;</span> in the output schema, then some kind of flattening<br/> is happening. It normally would be <span class="error">&#91;^&#93;</span>...<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span> for conventional usage.</p> <p>Algorithmic implication is that you don't know how many iterations there are without<br/> recursing completely into your children. If their dimensionality is large enough, it<br/> could reach back and affect your size.</p> <p>The fixed relationship between parent collection and expansion is just how it is. </p> <p>In today's Plastic, the logic looks for the immediate parent as the expansion point. If <br/> you didn't like this simplification, say you wanted expansion to occur at a parent's parent,<br/> then you needed to code.</p> <p>After this implementation, the logic will look up the parent stack to handle each dimension of the <br/> iteration. If there are enough parents, then flattening occurs.</p> <p>If you don't like these simplifications, then you need to code.</p> <p>6B - Mixing Input Arrays into a Flattened Output Array<br/> ------------------------------------------------------</p> <p>What if I want to leverage the original arrays?</p> <p>Desired output...<br/> [<br/> '1/a', <br/> '1/b', <br/> '1/c',<br/> '2/d', <br/> '2/e',<br/> '3/f'<br/> ]</p> <p>Output schema...<br/> [<br/> "${id<span class="error">&#91;*&#93;</span>}"/"${bbb<span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ]</p> <p>This is a flattening because of <span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span>. That dimension wins out so the flattening is 2 dimensional.<br/> The iterators for id and bbb will be merged into a single iterator. The first <span class="error">&#91;*&#93;</span> will have a range of <br/> the larger of id and bbb. Then second <span class="error">&#91;*&#93;</span> will have the range of the second dimension from bbb. This<br/> means that the leftmost <span class="error">&#91;*&#93;</span> in both iterators really need to have the same range to avoid issues.</p> <p>Note that i=(4) and bbb=(4,3) and there are no entries for bbb<span class="error">&#91;3&#93;</span><span class="error">&#91;*&#93;</span> so the only choice is to error<br/> out or to skip that expansion. If there are default values associated with id and/or bbb, they can<br/> be used to fill in holes. If not, then it should be an error. This is a mixed hole because it takes<br/> values from two variables to create a single output value. Had the output value just been from a single<br/> variable, then this would have been a simple hole and skipped.</p> <p>Rule: holes in simple output values will be skipped in interations, unless there is a default value, <br/> but holes in mixed output values result in an error.</p> <p>Had the output schema been...<br/> [<br/> "${id<span class="error">&#91;*&#93;</span>}"/"${bbb<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ]</p> <p>then it would have been an error because there is no parent array to supply the missing dimension.</p> <p>Rule: using ^ and not having a parent array to supply the missing dimension is an error.</p> <p>6C - Naive Isomorphic Conversion<br/> --------------------------------</p> <p>Output schema...<br/> {<br/> "AAA": [<br/> {<br/> "ID": "${id<span class="error">&#91;*&#93;</span>},<br/> "BBB": [<br/> "${bbb<span class="error">&#91;*&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ]<br/> }<br/> ]<br/> }</p> <p>Output...<br/> {<br/> "aaa": [ <br/> </p> { "id": "1", "bbb": [ 'a', 'b', 'c', 'd', 'e', 'f' ] } <p>,<br/> </p> { "id": "2", "bbb": [ 'a', 'b', 'c', 'd', 'e', 'f' ] } <p>,</p> { "id": "3", "bbb": [ 'a', 'b', 'c', 'd', 'e', 'f' ] } { "id": "4", "bbb": [ 'a', 'b', 'c', 'd', 'e', 'f' ] } <p> ]<br/> }</p> <p>6D - Isomorphic Conversion (as a sanity check)<br/> -----------------------------------------------------</p> <p>Output schema...<br/> {<br/> "AAA": [<br/> {<br/> "ID": "${id<span class="error">&#91;*&#93;</span>},<br/> "BBB": [<br/> "${bbb<span class="error">&#91;^&#93;</span><span class="error">&#91;*&#93;</span>}"<br/> ],<br/> }<br/> ]<br/> }<br/> Output...<br/> {<br/> "aaa": [ <br/> </p> { "id": "1", "bbb": [ 'a', 'b', 'c' ] } <p>,<br/> </p> { "id": "2", "bbb": [ 'd', 'e' ] } <p>,</p> { "id": "3", "bbb": [ 'f' ] } { "id": "4", "bbb": [ ] } <p> ]<br/> }</p> <p>Design Notes<br/> ------------</p> <p>MoArray logic </p> <ul> <li>will need to create iterator</li> <li>dump an array for debug including iterator</li> <li>copy iterator from one array to another</li> <li>Need way to create missing iterator for legacy morpher logic</li> </ul> <p>Reserve for internal use variable names with</p> <ul> <li>leading: _</li> </ul> <p>Illegal variable names</p> <ul> <li>containing: . [ ] * ~ ^ ? {} &amp; | ( ) @</li> </ul> <p>Iterator is special attribute "_<span class="error">&#91;name&#93;</span>" and is an array of int[]<br/> Error if dimensionality of variable is not equal to the dimensionality of the bound values</p> <p>Iterator (excluding empty iterator) has</p> <ul> <li>one or more names</li> <li>one or more dimensions</li> <li>each dimension has a range 0 ... L-1</li> <li>merging where larger ranges and bigger dimensions win</li> <li>incrementable with a current value</li> <li>incrementing only applies to non-parent dimensions</li> <li>a concept of "is done", which needs to be checked FIRST to allow for zero length arrays</li> <li>"is done" only applies to non-parent dimension</li> </ul> <p>Algorithm<br/> ---------</p> <p>Recurse through schema<br/> When you hit an array then push onto parent stack with empty iterator<br/> Recurse through nearby children and collect iterators specifications (don't drill into arrays)<br/> For each iterator specification<br/> if input schema then<br/> validate iterator syntax<br/> create iterator<br/> if output schema then<br/> validate iterator syntax<br/> validate the iterator is bound<br/> valid dimensions of bound values matches specification<br/> if iterator has parent references then<br/> link all parent references up the iterator inheritance chain<br/> impose/merge iterator on TOS</p> <p>loop if iteration is not done then<br/> increment iterator<br/> clone<br/> walk cloned output<br/> if hit array, recurse and push<br/> when resolving arrayed variable, need to look up stack of pushed iterators for resolution</p> <p>QUESTIONS</p> <p>All ^^^^^ should be semantically valid? <br/> JW TMO example</p> Development Rank 0|i03qy7: