[YANGTOOLS-1128] Add a dedicated ANTLR token factory Created: 10/Aug/20 Updated: 26/Aug/20 Resolved: 20/Aug/20 |
|
| Status: | Resolved |
| Project: | yangtools |
| Component/s: | parser |
| Affects Version/s: | None |
| Fix Version/s: | 6.0.0, 4.0.14, 5.0.6, 3.0.17 |
| Type: | Improvement | Priority: | Medium |
| Reporter: | Robert Varga | Assignee: | Robert Varga |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Description |
|
Examining a heap dump for YANG parser before actual parsing reveals ANTLR can end up eating quite a bit of memory for its CommonToken instances. As it turns out, we can probably do much better by providing a customized token factory. |
| Comments |
| Comment by Robert Varga [ 15/Aug/20 ] |
|
After a bit of experimentation, this only scratches the surface. Nominally we can reduce the footprint by about 15% through use of memory-optimized Token implementation. Since ANTLR needs to track location for each token, further improvement can be made by post-processing the parse tree and replacing tokens with location-independent implementation. |