Commit graph

169 commits

Author SHA1 Message Date
Matt Jaffee
d2621709f1 move NodeSet out of cluster.go into broadcast and messenger
HTTP implementation will be split into separate package.
2017-04-19 10:36:27 -05:00
Matt Jaffee
4b8c2630e3 split MessageBroker.Send into SendSync and SendAsync 2017-04-19 10:34:30 -05:00
Matt Jaffee
5079ee7a22 remove messenger "wrapper" around messagebroker
all the functionality aside from the broker has been removed
2017-04-19 10:33:59 -05:00
Matt Jaffee
10a6cead17 remove unused methods on StaticNodeSet
these used to implement the NodeSet interface, but they've since been removed
from NodeSet
2017-04-19 10:33:26 -05:00
Matt Jaffee
340cbc0cd0 fix comment s/list/map 2017-04-19 10:33:26 -05:00
Travis
b0f1fc7523 Makes Messenger a first-class object under Server (with pointers in Handler and Index).
Primary message interface is the MessageBroker which is an attribute of the Messenger.
MessageBroker implementations:
- Gossip (memberlist)
- Broadcast (uses HTTP, received by existing Handler)
- Static (no-ops)

Changes CacheSize from `int` to `uint32` for consistency with protobuf.
Removes unnecessary dependencies in glide:
- `github.com/aws/aws-sdk-go`
- `golang.org/x/net` (although this gets included by memberlist)

TODO:
- [ ] Add tests around the Messenger and MessageBroker objects.
- [ ] Refactor CreateSliceMessage to work with views.
- [ ] Support propogation of meta data on PATCH calls.

fixing some issues from last rebase
2017-04-19 10:33:26 -05:00
Travis
13e08ac3a5 add tests for HTTPNodeSet 2017-04-19 10:24:00 -05:00
Travis
3abfea140b placeholder for localNode in HTTPNodeSet 2017-04-19 10:24:00 -05:00
Travis
8fa45649b6 WIP: Added support for NodeState.
Modified the gossip implementation to optionally send messages directly (instead of over gossip).
Added support for local and remote NodeState, as well as function to merge remote into local state.
Refactored Protobuf models to include `DBMeta` and `FrameMeta` as well as support for NodeState.
2017-04-19 10:19:17 -05:00
Travis
2b35df879c WIP: first pass at sending CreateDB() through Messenger 2017-04-19 10:15:57 -05:00
Travis
7c2f6a9bae adds a basic gossip implementation using hashicorp/memberlist
Changes Gossiper to NodeSet.
Adds StaticNodeSet (for testing) and GossipNodeSet (for memberlist) implementations.
Changes NodeSet interface to return a pilosa-specific generic instead of `NodeSet interface`.
Removes NumMembers from interface (which is specific to memberlist).

Implements a Messenger interface with which to send inter-node messages via NodeSet.
The Pilosa implementation occurs in the GossipNodeSet.

Implements the Messenger as an object on Server, Handler, and Index.

Uses HealthStatus constants.
Removes commented-out code.

for gossip, make sure to bind to both host and port, and advertise those as well

adjust messenger to work with the db schema logic

add dependencies: hashicorp/memberlist, golang.org/x/sync
add dependency: golang.org/x/net

Uses `errgroup` to handle errors from broadcast messages.
Marshals message one time instead of once for every node.
Adds error handling for some errors that were being swallowed.
2017-04-19 10:13:38 -05:00
Travis
b6f0a62fdd changes MaxSlices logic to be DB-specific 2016-11-21 16:12:30 -06:00
Ben Johnson
94395b4fce Add fault tolerance to executor. 2016-11-17 12:02:53 -07:00
Ben Johnson
c54047041d forward bitmap & profile attributes
This commit adds forwarding of bitmap and profile attributes to
other servers in the cluster.
2016-07-19 13:54:01 -06:00
Ben Johnson
14627a2152 use db in fragment allocation
This commit changes the fragment allocation algorithm in the cluster
to make use of the `DB` name. This allows each database to use a
different slice distribution.

Initially, the `frame` was going to be used for allocation, however,
this was problematic since queries can span multiple frames so it's
impossible to choose a single frame to use.
2016-06-07 14:01:13 -06:00
Ben Johnson
0ab4ee6f07 add frame restore for cluster transfer
This commit adds an endpoint to restore an entire frame from
another cluster. Multiple hosts can use this endpoint to copy
and rebalance a cluster to a new cluster.
2016-03-15 14:42:42 -06:00
Ben Johnson
c12c186a73 cluster-wide import
This commit fixes the `pilosactl import` command to import data files
across an entire cluster of machines.

A `pilosactl config` command is also added to provide a default config.
2016-01-05 20:09:17 -07:00
Ben Johnson
320e9811d8 bulk importer
This commit adds the pilosactl binary with a single `import` command.
The import allows users to bulk import data from a CSV file.
2016-01-02 15:10:17 -07:00
Ben Johnson
a632d08629 refactor
This commit refactors the pilosa codebase. It makes several major
changes:

* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
2015-12-02 15:34:49 -07:00