[TSDR-21] HBase Query and NBI only looks into the first 1000 records for Query output Created: 05/Feb/16  Updated: 10/Feb/16  Resolved: 10/Feb/16

Status: Verified
Project: tsdr
Component/s: General
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: premnath sundarababu Assignee: YuLing Chen
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 5244
Priority: Normal

 Description   

Connect 200 openflow nodes .

openflow:2 records are not present in tsdr:list [ which is top 1000 records]

opendaylight-user@root>tsdr:list FLOWSTATS | grep openflow:2
opendaylight-user@root>

openflow:2 records is present in HBase table using query scan 'FLOWSTATS',

{FILTER =>"(RowFilter(=,'regexstring:openflow:2')) AND (RowFilter(=,'regexstring:openflow:2'))"}

:

[NID=openflow:2][DC=FLOWSTATS column=c1:raw, timestamp=1454489829356, value=864
][MN=PacketCount][RK=Node:ope
nflow:2,Table:0,Flow:#UF$TABL
E*0-282][TS=1454489829356]
[NID=openflow:2][DC=FLOWSTATS column=c1:raw, timestamp=1454489955263, value=912
][MN=PacketCount][RK=Node:ope
nflow:2,Table:0,Flow:#UF$TABL
E*0-282][TS=1454489955263]
[NID=openflow:2][DC=FLOWSTATS column=c1:raw, timestamp=1454490095678, value=970
][MN=PacketCount][RK=Node:ope
nflow:2,Table:0,Flow:#UF$TABL
E*0-282][TS=1454490095678]
[NID=openflow:2][DC=FLOWSTATS column=c1:raw, timestamp=1454490220987, value=1020
][MN=PacketCount][RK=Node:ope
nflow:2,Table:0,Flow:#UF$TABL
E*0-282][TS=1454490220987]
[NID=openflow:2][DC=FLOWSTATS column=c1:raw, timestamp=1454490347397, value=1070
][MN=PacketCount][RK=Node:ope
nflow:2,Table:0,Flow:#UF$TABL
E*0-282][TS=1454490347397]

Query returns 0 values

[ WARN ] {
"metricRecords": [],
"recordCount": 0
}
[ WARN ] {}

Query returns output for openflow:102 which is present in tsdr:list

opendaylight-user@root>tsdr:list FLOWSTATS | grep openflow:102
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454487809960][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454487920020][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488061334][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488186115][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488295676][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488436026][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488560916][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488685684][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488811623][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454488937212][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489078098][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489187799][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489329182][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489469966][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489595061][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489751937][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454489861357][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454490002550][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454490127392][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-93[TS=1454490268255][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-94[TS=1454487683556][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-95[TS=1454487683556][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount]RK=Node:openflow:102,Table:0,Flow:#UF$TABLE*0-96[TS=1454487683556][0]
[NID=openflow:102][DC=FLOWSTATS][MN=ByteCount][RK=Node:openflow:102,Table:0,Flow:#
<output truncated>

${resp}= RequestsLibrary.Get Request session /tsdr/metrics/query?tsdrkey="[NID=openflow:102][DC=FLOWSTATS][MN=][RK=]"&from=${from}&until=${until} headers=${HEADERS_QUERY}
${resp1}= RequestsLibrary.Get Request session /tsdr/nbi/render?target="[NID=openflow:102][DC=FLOWSTATS][MN=][RK=]"&from=0&until=now&maxDataPoints=1000 headers=${HEADERS_QUERY}

{
"metricName": "PacketCount",
"metricValue": 406,
"timeStamp": "Wed Feb 03 14:30:33 IST 2016",
"nodeID": "openflow:102",
"tsdrDataCategory": "FLOWSTATS",
"recordKeys": [

{ "keyName": "Node", "keyValue": "openflow:102" }

,

{ "keyName": "Table", "keyValue": "0" }

,

{ "keyName": "Flow", "keyValue": "#UF$TABLE*0-984" }

]
},
{
"metricName": "PacketCount",
"metricValue": 406,
"timeStamp": "Wed Feb 03 14:32:38 IST 2016",
"nodeID": "openflow:102",
"tsdrDataCategory": "FLOWSTATS",
"recordKeys": [

{ "keyName": "Node", "keyValue": "openflow:102" }

,

{ "keyName": "Table", "keyValue": "0" }

,

{ "keyName": "Flow", "keyValue": "#UF$TABLE*0-984" }

]
},
{
"metricName": "PacketCount",
"metricValue": 406,
"timeStamp": "Wed Feb 03 14:34:43 IST 2016",
"nodeID": "openflow:102",
"tsdrDataCategory": "FLOWSTATS",
"recordKeys": [

{ "keyName": "Node", "keyValue": "openflow:102" }

,

{ "keyName": "Table", "keyValue": "0" }

,

{ "keyName": "Flow", "keyValue": "#UF$TABLE*0-984" }

]
}
],
"recordCount": 206
}
[ WARN ] [

{"target":"\"[NID\u003dopenflow:102][DC\u003dFLOWSTATS][MN\u003d][RK\u003d]\"","datapoints":[[0.0,1454487809],[0.0,1454487920],[0.0,1454488061],[0.0,1454488186],[0.0,1454488295],[0.0,1454488436],[0.0,1454488560],[0.0,1454488685],[0.0,1454488811],[0.0,1454488937],[0.0,1454489078],[0.0,1454489187],[0.0,1454489329],[0.0,1454489469],[0.0,1454489595],[0.0,1454489751],[0.0,1454489861],[0.0,1454490002],[0.0,1454490127],[0.0,1454490268],[0.0,1454487683],[0.0,1454487683],[0.0,1454487683],[5162.0,1454487809],[9078.0,1454487920],[13884.0,1454488061],[18512.0,1454488186],[22428.0,1454488295],[27412.0,1454488436],[32040.0,1454488560],[36312.0,1454488685],[40762.0,1454488811],[45212.0,1454488937],[50374.0,1454489078],[54112.0,1454489187],[59274.0,1454489329],[64258.0,1454489469],[68708.0,1454489595],[74226.0,1454489751],[78142.0,1454489861],[83126.0,1454490002],[87576.0,1454490127],[92560.0,1454490268],[0.0,1454487825],[0.0,1454487951],[0.0,1454488076],[0.0,1454488201],[0.0,1454488326],[0.0,1454488451],[0.0,1454488576],[0.0,1454488701],[0.0,1454488842],[0.0,1454488968],[0.0,1454489093],[0.0,1454489219],[0.0,1454489360],[0.0,1454489485],[0.0,1454489626],[0.0,1454489767],[0.0,1454489892],[0.0,1454490033],[0.0,1454490158],[0.0,1454490283],[0.0,1454487825],[0.0,1454487951],[0.0,1454488076],[0.0,1454488201],[0.0,1454488326],[0.0,1454488451],[0.0,1454488576],[0.0,1454488701],[0.0,1454488842],[0.0,1454488968],[0.0,1454489093],[0.0,1454489219],[0.0,1454489360],[0.0,1454489485],[0.0,1454489626],[0.0,1454489767],[0.0,1454489892],[0.0,1454490033],[0.0,1454490158],[0.0,1454490283],[18618.0,1454487825],[24824.0,1454487951],[24824.0,1454488076],[31030.0,1454488201],[31030.0,1454488326],[31030.0,1454488451],[31030.0,1454488576],[37236.0,1454488701],[37236.0,1454488842],[37236.0,1454488968],[37236.0,1454489093],[37236.0,1454489219],[37236.0,1454489360],[37236.0,1454489485],[37236.0,1454489626],[43442.0,1454489767],[43442.0,1454489892],[43442.0,1454490033],[43442.0,1454490158],[43442.0,1454490283],[0.0,1454487809],[0.0,1454487920],[0.0,1454488061],[0.0,1454488186],[0.0,1454488295],[0.0,1454488436],[0.0,1454488560],[0.0,1454488685],[0.0,1454488811],[0.0,1454488937],[0.0,1454489078],[0.0,1454489187],[0.0,1454489329],[0.0,1454489469],[0.0,1454489595],[0.0,1454489751],[0.0,1454489861],[0.0,1454490002],[0.0,1454490127],[0.0,1454490268],[0.0,1454487683],[0.0,1454487683],[0.0,1454487683],[58.0,1454487809],[102.0,1454487920],[156.0,1454488061],[208.0,1454488186],[252.0,1454488295],[308.0,1454488436],[360.0,1454488560],[408.0,1454488685],[458.0,1454488811],[508.0,1454488937],[566.0,1454489078],[608.0,1454489187],[666.0,1454489329],[722.0,1454489469],[772.0,1454489595],[834.0,1454489751],[878.0,1454489861],[934.0,1454490002],[984.0,1454490127],[1040.0,1454490268],[0.0,1454487825],[0.0,1454487951],[0.0,1454488076],[0.0,1454488201],[0.0,1454488326],[0.0,1454488451],[0.0,1454488576],[0.0,1454488701],[0.0,1454488842],[0.0,1454488968],[0.0,1454489093],[0.0,1454489219],[0.0,1454489360],[0.0,1454489485],[0.0,1454489626],[0.0,1454489767],[0.0,1454489892],[0.0,1454490033],[0.0,1454490158],[0.0,1454490283],[0.0,1454487825],[0.0,1454487951],[0.0,1454488076],[0.0,1454488201],[0.0,1454488326],[0.0,1454488451],[0.0,1454488576],[0.0,1454488701],[0.0,1454488842],[0.0,1454488968],[0.0,1454489093],[0.0,1454489219],[0.0,1454489360],[0.0,1454489485],[0.0,1454489626],[0.0,1454489767],[0.0,1454489892],[0.0,1454490033],[0.0,1454490158],[0.0,1454490283],[174.0,1454487825],[232.0,1454487951],[232.0,1454488076],[290.0,1454488201],[290.0,1454488326],[290.0,1454488451],[290.0,1454488576],[348.0,1454488701],[348.0,1454488842],[348.0,1454488968],[348.0,1454489093],[348.0,1454489219],[348.0,1454489360],[348.0,1454489485],[348.0,1454489626],[406.0,1454489767],[406.0,1454489892],[406.0,1454490033],[406.0,1454490158],[406.0,1454490283]]}

]

 Comments   
Comment by premnath sundarababu [ 05/Feb/16 ]

Prefix filter issue is seen in HBase

Comment by YuLing Chen [ 06/Feb/16 ]

Hi Premnath,

I'm having trouble understanding this bug.

From your output, it seems that the data in hbase is large enough that the tsdr:list would not be able to return records that matches "openflow:2", which is normal since tsdr:list does only return the first 1000 records from the data store.

Are you complaining that using Query API with filter of "NID=openflow:2" you would not be able to retrieve matches record as well? Please confirm. If so, could you let me know your curl command for the query API?

Please also explain in more detail about "prefix filter issue is seen in hbase".

Thanks,

YuLing

Comment by yuling chen [ 09/Feb/16 ]

fixed with patch set https://git.opendaylight.org/gerrit/34229

Comment by premnath sundarababu [ 10/Feb/16 ]

sudo mn --controller=remote,ip=127.0.0.1 --topo=linear,22 --switch ovsk,protocols=OpenFlow13

Part of the issue is solved..But still see issue in some of the scenarios like one given below.

Collect the data for 1 hour

Try to Query

${resp}= RequestsLibrary.Get Request session /tsdr/metrics/query?tsdrkey="[NID=openflow:1][DC=FLOWTABLESTATS][MN=][RK=Table:0]"&from=${from}&until=${until} headers=${HEADERS_QUERY}

It does not return any value

Generated at Wed Feb 07 20:46:09 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.