GossipNodeSet and GossipBroadcaster had a lot of cross dependency - I made them
the same object and had it implement all three interfaces (NodeSet, Broadcaster,
BroadcastReceiver).
I implmented an HTTPBroadcastReceiver which runs as as separate server.
BroadcastReceiver is a separate entity on the pilosa.Server object and is
started separately from broadcaster and nodeset. In the case of GossipNodeSet,
the broadcast receiver must be started before Open()ing the Nodeset, because it
doesn't actually start listening until Open() is called, but it needs the
handler set up before then.
server/server.go was heavily refactored to configure the new structs and
interfaces on the pilosa.Server object.
greatly simplifies Messenger, to the point of making it basically shell around
MessageBroker. Next stesp are to make the receiving of internal messages more
well-defined and behind an interface, remove/merge Messenger and MessageBroker,
and have the implementations of MessageBroker in separate packages.
Primary message interface is the MessageBroker which is an attribute of the Messenger.
MessageBroker implementations:
- Gossip (memberlist)
- Broadcast (uses HTTP, received by existing Handler)
- Static (no-ops)
Changes CacheSize from `int` to `uint32` for consistency with protobuf.
Removes unnecessary dependencies in glide:
- `github.com/aws/aws-sdk-go`
- `golang.org/x/net` (although this gets included by memberlist)
TODO:
- [ ] Add tests around the Messenger and MessageBroker objects.
- [ ] Refactor CreateSliceMessage to work with views.
- [ ] Support propogation of meta data on PATCH calls.
fixing some issues from last rebase
Modified the gossip implementation to optionally send messages directly (instead of over gossip).
Added support for local and remote NodeState, as well as function to merge remote into local state.
Refactored Protobuf models to include `DBMeta` and `FrameMeta` as well as support for NodeState.
Changes Gossiper to NodeSet.
Adds StaticNodeSet (for testing) and GossipNodeSet (for memberlist) implementations.
Changes NodeSet interface to return a pilosa-specific generic instead of `NodeSet interface`.
Removes NumMembers from interface (which is specific to memberlist).
Implements a Messenger interface with which to send inter-node messages via NodeSet.
The Pilosa implementation occurs in the GossipNodeSet.
Implements the Messenger as an object on Server, Handler, and Index.
Uses HealthStatus constants.
Removes commented-out code.
for gossip, make sure to bind to both host and port, and advertise those as well
adjust messenger to work with the db schema logic
add dependencies: hashicorp/memberlist, golang.org/x/sync
add dependency: golang.org/x/net
Uses `errgroup` to handle errors from broadcast messages.
Marshals message one time instead of once for every node.
Adds error handling for some errors that were being swallowed.
Previously the cache type (`lru` or `ranked`) was determined by
checking the ending suffix of the frame. If it ended with a `.n`
then it was ranked. Otherwise it was an LRU.
This commit changes frames so that a `cacheType` option can be
passed in during creation to set either `lru` or `ranked`.
Previously only the standard max slice was available. This commit
changes it so that the inverse max can be retrieved separately
through the `DB` and `Frame` types as well as through the
`HTTP` API and `Client`.
Previously, multiple frames with different prefixes were used to separate
different data layouts. This included separating standard row/column
layouts from inverted column/row layouts as well as storing aggregate
information for timestamp data.
Unfortunately, this caused frame meta data to be copied between multiple
frames and it made it difficult to keep these frames in sync.
This commit separates these different physical layouts into `Views`.
A `Frame` now has one or more views which represent each layout.
Fragments have been moved from under the `Frame` to be contained
within the `View`.
There are two primary views:
- `standard`
- `inverse`
If a frame has a time quantum, then views are generated for these
each of the standard/inverse views. For example a time quantum
of `YMDH` for the date `2000-01-02T00:00:00Z` would create the
following views:
- `standard_2000`
- `inverse_2000`
- `standard_200001`
- `inverse_200001`
- `standard_20000102`
- `inverse_20000102`
From the user's perspective, nothing should change in PQL. Different
PQL statements will handle the appropriate view automatically. For
example, `Bitmap()` and `Profile()` will fetch using the `standard`
view or the `inverse` view, respectively. The `Range()` statement
will lookup the appropriate time-based views automatically.
Databases and frames now require explicit creation and have the
option of setting row & column labels. If no labels are provided
then the default `id` and `profileID` labels are used.
- Ingore asserts in fragment container.
- Only log queries that take longer than 90 seconds.
TODO:
- address the TODOs that make the asserts configurable.
This changes the API of the types but does not alter the functionality
since only the `context.Background()` is currently being used. Adding
`Context` will help handle fault tolerance in the future by allowing
timeouts to be propagated across calls to different nodes.
CSV exports can now be done with the pilosactl application:
$ pilosactl -d mydb -f myframe -o MYFILE.csv
If `-o` is not specified then the CSV is written to STDOUT. The
exporter combines all slices for the db/frame to into a single
concatenated CSV file.
