From 55310a5309e944ac3d0bcf5668f35b9ab35824cd Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Thu, 13 Jul 2017 09:32:47 -0500 Subject: [PATCH 1/4] frame cache caveats documentation --- docs/query-language.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/query-language.md b/docs/query-language.md index 5400c8ead..106bfb08f 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -341,6 +341,14 @@ have the attribute specified by `field` with one of the values specified in **Result Type:** array of key/count objects +**Caveats:** +* TopN will return the top bitmaps by count only on frames with a ranked cache. +* The frame's cache size determines the limit of sorted bitmaps. +* Non-ranked frames maintain a LRU (least recently used) cache, thus a TopN query on this frame will return bitmaps in most recently setbit order. +* The ranked cache maintains a sorted set of bitmaps by count up to the cache size. +* Once full, the cache will truncate the set of bitmaps at the frame options CacheSize limit. Bitmaps with a similar count that straddle the limit will be truncated in no particular order. +* The TopN filter operation only applies to data in the cache, and not all bitmaps in the frame. + **Examples:** ``` From b313902eddc439eb25ddb829336a2def01b6d836 Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Thu, 13 Jul 2017 14:24:20 -0500 Subject: [PATCH 2/4] clarify sort order in TopN doc --- docs/query-language.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/query-language.md b/docs/query-language.md index 106bfb08f..cf551b3b5 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -342,9 +342,9 @@ have the attribute specified by `field` with one of the values specified in **Result Type:** array of key/count objects **Caveats:** -* TopN will return the top bitmaps by count only on frames with a ranked cache. +* TopN on a frame with ranked cache will return the top bitmaps sorted by count. +* Non-ranked frames maintain a LRU (least recently used) cache, thus a TopN query on this frame will return bitmaps sorted in most recently setbit order. * The frame's cache size determines the limit of sorted bitmaps. -* Non-ranked frames maintain a LRU (least recently used) cache, thus a TopN query on this frame will return bitmaps in most recently setbit order. * The ranked cache maintains a sorted set of bitmaps by count up to the cache size. * Once full, the cache will truncate the set of bitmaps at the frame options CacheSize limit. Bitmaps with a similar count that straddle the limit will be truncated in no particular order. * The TopN filter operation only applies to data in the cache, and not all bitmaps in the frame. From 0feec9be62447342041b4db7c58eeec384c1c49b Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Mon, 17 Jul 2017 10:46:40 -0500 Subject: [PATCH 3/4] TopN Cache clarification. --- docs/query-language.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/query-language.md b/docs/query-language.md index cf551b3b5..b5d88fd1a 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -342,12 +342,11 @@ have the attribute specified by `field` with one of the values specified in **Result Type:** array of key/count objects **Caveats:** -* TopN on a frame with ranked cache will return the top bitmaps sorted by count. -* Non-ranked frames maintain a LRU (least recently used) cache, thus a TopN query on this frame will return bitmaps sorted in most recently setbit order. -* The frame's cache size determines the limit of sorted bitmaps. -* The ranked cache maintains a sorted set of bitmaps by count up to the cache size. -* Once full, the cache will truncate the set of bitmaps at the frame options CacheSize limit. Bitmaps with a similar count that straddle the limit will be truncated in no particular order. -* The TopN filter operation only applies to data in the cache, and not all bitmaps in the frame. +* Performing a TopN() query on a frame with cache type ranked will return the top bitmaps sorted by count in descending order. +* Frames with cache type lru will maintain an LRU (Least Recently Used) cache, thus a TopN() query on this type of frame will return bitmaps sorted in order of most recently set bit. +* The frame's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN() queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. +* Once full, the cache will truncate the set of bitmaps according to the frame option CacheSize. Bitmaps that straddle the limit and have the same count will be truncated in no particular order. +* The TopN() query's attribute filter is applied the the existing sorted cached of bitmaps. Bitmaps that fall outside of the sorted cache range, but would pass the filter are ignored. **Examples:** From dd7e159f8337e6bab67f71aa9ed9e28b81f75d10 Mon Sep 17 00:00:00 2001 From: Michael Baird Date: Mon, 17 Jul 2017 12:43:15 -0500 Subject: [PATCH 4/4] TopN attribute filter doc cleanup. --- docs/query-language.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/query-language.md b/docs/query-language.md index b5d88fd1a..93ddb846a 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -346,7 +346,7 @@ have the attribute specified by `field` with one of the values specified in * Frames with cache type lru will maintain an LRU (Least Recently Used) cache, thus a TopN() query on this type of frame will return bitmaps sorted in order of most recently set bit. * The frame's cache size determines the number of sorted bitmaps to maintain in the cache for purposes of TopN() queries. There is a tradeoff between performance and accuracy; increasing the cache size will improve accuracy of results at the cost of performance. * Once full, the cache will truncate the set of bitmaps according to the frame option CacheSize. Bitmaps that straddle the limit and have the same count will be truncated in no particular order. -* The TopN() query's attribute filter is applied the the existing sorted cached of bitmaps. Bitmaps that fall outside of the sorted cache range, but would pass the filter are ignored. +* The TopN() query's attribute filter is applied to the existing sorted cache of bitmaps. Bitmaps that fall outside of the sorted cache range, even if they would normally pass the filter, are ignored. **Examples:**