Primary message interface is the MessageBroker which is an attribute of the Messenger.
MessageBroker implementations:
- Gossip (memberlist)
- Broadcast (uses HTTP, received by existing Handler)
- Static (no-ops)
Changes CacheSize from `int` to `uint32` for consistency with protobuf.
Removes unnecessary dependencies in glide:
- `github.com/aws/aws-sdk-go`
- `golang.org/x/net` (although this gets included by memberlist)
TODO:
- [ ] Add tests around the Messenger and MessageBroker objects.
- [ ] Refactor CreateSliceMessage to work with views.
- [ ] Support propogation of meta data on PATCH calls.
fixing some issues from last rebase
Previously the cache type (`lru` or `ranked`) was determined by
checking the ending suffix of the frame. If it ended with a `.n`
then it was ranked. Otherwise it was an LRU.
This commit changes frames so that a `cacheType` option can be
passed in during creation to set either `lru` or `ranked`.
Previously, multiple frames with different prefixes were used to separate
different data layouts. This included separating standard row/column
layouts from inverted column/row layouts as well as storing aggregate
information for timestamp data.
Unfortunately, this caused frame meta data to be copied between multiple
frames and it made it difficult to keep these frames in sync.
This commit separates these different physical layouts into `Views`.
A `Frame` now has one or more views which represent each layout.
Fragments have been moved from under the `Frame` to be contained
within the `View`.
There are two primary views:
- `standard`
- `inverse`
If a frame has a time quantum, then views are generated for these
each of the standard/inverse views. For example a time quantum
of `YMDH` for the date `2000-01-02T00:00:00Z` would create the
following views:
- `standard_2000`
- `inverse_2000`
- `standard_200001`
- `inverse_200001`
- `standard_20000102`
- `inverse_20000102`
From the user's perspective, nothing should change in PQL. Different
PQL statements will handle the appropriate view automatically. For
example, `Bitmap()` and `Profile()` will fetch using the `standard`
view or the `inverse` view, respectively. The `Range()` statement
will lookup the appropriate time-based views automatically.
it's no longer reponsible for updating the count cache.
This commit also helps SetBit/ClearBit performance by allowing them
to work against data from `bitmapCache` instead of loading bitmaps
from fragment.storage every time.
heap sort order backwards
WIP TopN accuracy
adjusted first phase topn to collect all slices id's
incorrect handling of large topns
cache performance enhancement
fix for failed test TestMain_FrameRestore
remove unused code and fix some variable names
- Ingore asserts in fragment container.
- Only log queries that take longer than 90 seconds.
TODO:
- address the TODOs that make the asserts configurable.
This changes the API of the types but does not alter the functionality
since only the `context.Background()` is currently being used. Adding
`Context` will help handle fault tolerance in the future by allowing
timeouts to be propagated across calls to different nodes.
A `StatsClient` for `expvar` is added so we can track Stats through
the `/debug/vars` endpoint. Tags are nested inside maps so that we
can see stats for db, frame & slice.
Also added a `MultiStatsClient` for chaining multiple `StatsClient`
implementations together (e.g. `expvar` and DataDog).
Introduces new `pilosactl sort` to sort import files by bit
position so they can be inserted faster. Also optimizes container
scanning and adds a `-buffer-size` flag to `import`.
