Commit graph

73 commits

Author SHA1 Message Date
Ben Johnson
60d534c505 Limit translation memory & add max query memory config 2021-08-02 15:28:00 -06:00
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Travis
4b494c3ec3
Update metric names to use "featurebase" prefix
If the `--future.rename` flag is set (to true), this commit will cause
metric names to be prefixed with "featurebase" instead of "pilosa".
2021-07-12 17:51:59 -05:00
Travis
3c7ee11ff3
Add feature flag --future.rename to support rename to FeatureBase
This commit adds a `Future` scope to the configuration options, and for
the time being includes a single flag within that scope: `rename`.

Usage:
--future.rename

The value is a boolean available internally at: m.Config.Future.Rename
2021-07-08 13:18:43 -05:00
Samir Patel
8bf472bb75 add duty cycle config flag 2021-06-14 10:48:52 -05:00
Samir Patel
b46b5fc134 remove usage-interval flag 2021-06-11 11:16:32 -05:00
Samir Patel
ae6687e71b change flag name to usage-interval 2021-06-01 10:47:09 -05:00
Samir Patel
f5cc179893 change flag from interval to duration 2021-06-01 10:47:09 -05:00
Samir Patel
8ce17f405b rename flag to disk-usage-interval 2021-06-01 10:47:09 -05:00
Samir Patel
827b125c3c rename and set default 2021-06-01 10:47:09 -05:00
Samir Patel
835c63011a add server flag 2021-06-01 10:47:08 -05:00
Nia Weiss
f4ba34247f
remove attributes
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
2021-05-14 10:28:08 -04:00
Nia Weiss
9bc1b23b7e
change "External" DB to "Lookup" DB 2021-03-29 12:54:29 -04:00
Nia Weiss
c4aad290ed
address ExternalLookup review comments 2021-03-29 08:22:41 -04:00
Nia Weiss
e9b92e1cd4
add ExternalLookup query 2021-03-25 13:21:28 -04:00
Todd Gruben
f52b88a962 etcd tls configuration support 2021-03-22 09:10:32 -05:00
Travis
42d69d6c5d
Remove gossip config 2021-03-04 13:51:18 -06:00
Travis
ea8b07d380
Merge branch 'master' into disco 2021-03-02 22:11:04 -06:00
Travis
ef40bbb617
renew heartbeat lease if the lease expires while a node is unavailable 2021-02-26 16:15:06 -06:00
Cody Soyland
dca19f7e94
Add new Docker build process and continuous delivery 2021-02-26 14:34:14 -06:00
Seebs
4f5f3e30ea remove port_mapper because it can't work with our unrestartable server
Long story short: Once we create a server and start it, we can't start
it again. We can't close it and restart it, and we can't just start
it without closing it.

Unfortunately, if the server's config needs to change, we have a Problem
here.

This ultimately means that the retry logic for GetListeners can't actually
retry successfully; if we fail on the first attempt, we necessarily fail
on any later attempts also, and if we try to fix that, we get panics.

But!

We don't actually NEED to retry. We just need to ensure that we can
open a :0 port, extract the actual port number, and use that in places
where the port number mattered, without having to rebind it.

The only actual place we needed to rebind things was opening gRPC
servers, so we introduce a gRPC Listener that can be used instead of
trying to bind to a specified port.

In a bunch of other cases where we had similar logic to try to allocate
and then use a port, we can switch to just using a provided listener.
For instance, net/http has `Serve(net.Listener, handler)`, not just
ListenAndServe(addr, handler).

This should eliminate the weird CI failures from eaddrinuse.

NOT fixed: server/cluster_test.go/TestClusterResize_AddNode isn't working
right now. The new node isn't actually being added to the existing cluster.
I attempted this but was outsmarted by it, and I think fixing the
rest of this is worth it as a separate thing.
2021-02-24 11:25:46 -06:00
Antonio Navarro Perez
c45e21640c
Change coordinator to primary
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-02 15:56:12 -06:00
Travis
457194f6a8
update config to support etcd arguments 2021-01-29 19:43:48 -06:00
Travis
e459c9a77b
change Config.DisCo to Config.Etcd 2021-01-29 14:11:51 -06:00
Travis
13984353e4
remove instances of os.Getenv("PILOSA_TXSRC") 2021-01-21 21:18:46 -06:00
Travis
08fae2be4c
introduce storage.Config 2021-01-20 22:05:38 -06:00
Travis
bc13834343
disco/etcd work: fix lots of races, start all cluster nodes at once.
port mapper gives out ports from 63000-65000 for the tests

fix another race

http test uses port.MustGetPort

rbf: remove :0 port request

ocd happy

test fix for grpc listener address already in use

test/disco allocates BindGRPC port from the port mapper

dump stack on each GetPort

verify each port is usable right away

server/config.go has Config.Validate() now

panic if gossip port is 0. validate server.Config

fix another gossip port 0

builds

quiet, don't dump stack on each port alloc

builds

happy linter

even gossip fallback should not be zero but rather use the port mapper
2021-01-12 21:06:12 -06:00
Maxton Huff
dee700741a deprecate cluster.long-query-time and create long-query-time
moved lonquerytime from cluster into server and moved cluster.longquerytime into top level config
kept cluster.longquerytime for backwards compatibility, favored if both longquerytime options are present
2020-12-10 10:51:31 -06:00
Jason E. Aten
03a54c6d5f rowcache off by default. pilosa server --rowcache-on turns it back on. 2020-12-08 23:18:45 +00:00
Alan Bernstein
2bc2a32263 Make query history length configurable 2020-11-23 20:23:27 -06:00
Jason E. Aten
957cba1768 performance tuning: rbfcfg package, binary search for wal segment
- rbfcfg package holds Config for --rbf- command line flags
- wal.go: replace linear search with bisection for wal segment
2020-10-27 00:49:56 +00:00
Jason E. Aten
8d2ad048fa pilosa server --norowcache disables the row cache
- this can lessen memory pressure
- certain backends may not need it
- enables performance benchmarking and tuning
2020-10-22 22:20:48 +00:00
Alan Bernstein
5705b45864 Add a CLI flag for cluster name and include it in /status response 2020-10-01 10:43:49 -05:00
Nia Weiss
f0f8336f57
rename postgres.addr to postgres.bind for consistency 2020-08-27 11:35:54 -04:00
Nia Weiss
288593231f
require TLS when configured 2020-08-27 10:59:40 -04:00
Nia Weiss
c2823a933b
add a config option to require TLS on postgres 2020-08-27 10:49:43 -04:00
Nia Weiss
128e02046a
add a postgres endpoint to pilosa 2020-08-20 11:29:16 -04:00
Jason Aten
f5688fa700 remove premature configuration of Txsrc in test config 2020-07-20 20:22:47 -04:00
Jason Aten
97b530ca78 integration of Tx, RoaringTx and BadgerTx implementations.
- all tests green on RoaringTx
  - RoaringTx on by default
  - blueGreenTx testing framework available for A-vs-B comparison
    of Tx implementations
  - flag -tx added to server command line but not wired to
    change NewIndex() selection yet.
  - 918 green tests, 14 tests red on BadgerTx.

    A full list of the 14 red tests on BadgerTx follows.
    Note that these red tests represent not defects in BadgerDB
    or BadgerTx but rather failures of the pre-existing pilosa infrastructure to yet
    be fully adapted from files to using a transactional storage engine.

    As such these are tests that RBF should not be expected to
    pass yet either.

    Fixing the pilosa infrastructure to allow these tests
    to go green under Badger is the next and highest priority
    order of business, but RBF can get much testing benefit
    from the 918 green tests we do have, and hence we merge
    as much as we have today.

    The 14 red tests when NewIndex() is set to use
    BadgerTx are as follows. Note in particular
    that pilosa cluster resizing is not working yet under a
    transactional store.

     TestCluster_ResizeStates/Multiple_nodes,_with_data
     TestImportClearRestart/0MaxOpN10000
     TestImportClearRestart/1MaxOpN10000
     TestImportClearRestart/2MaxOpN10000
     TestImportClearRestart/3MaxOpN10000
     TestExecutor_Execute_Existence/Row
     TestExecutor_ForeignIndex
     TestExecutor_Execute_CountDistinct/Distinct
     TestExecutor_Execute_CountDistinct/Count(Distinct)
     TestExecutor_Execute_CountDistinct/GroupBy(Distinct)
     TestExecutor_BareDistinct
     TestExecutor_Execute_TopNDistinct/TopN
     TestHolderSyncer_IntField/BasicSync
     TestHolderSyncer_IntField/MultiShard
2020-07-20 15:50:08 -04:00
Travis
944a6dca78
add --advertise-grpc configuration option 2020-07-09 11:17:12 -05:00
Kuba Podgórski
7ee8f80012
Merge branch 'master' into opt-indiagnostics/310 2020-05-15 00:53:33 +02:00
Kuba Podgórski
fd27cbb886
Update server/config.go
Co-authored-by: Matthew Jaffee <matthew.jaffee@gmail.com>
2020-05-12 18:51:53 +02:00
Kuba Podgórski
889f79d11a Turn anti-entropy off by default 2020-05-12 16:49:43 +02:00
Kuba Podgórski
368cb8cb53 opt-in diagnostics 2020-05-12 15:37:47 +02:00
Matt Jaffee
8aa7a76d31
change default tracing config to 'off'
also fix a typo
2020-04-30 15:02:15 -05:00
Travis
9405463911 suppress test config arguments from pilosa config output 2020-04-24 11:02:42 -05:00
Alan Bernstein
8b405c226c
Add 'prometheus' option in other help text/comments 2020-04-10 20:59:18 -05:00
Travis
a7bb90fcd0 initial gRPC server implementation
add makeRows() tests
register the gRPC server
use api.Index() instead of api.Schema()

support most field types in Inspect() query

currently, there's no support for `time` fields.
those will be dependent upon the output format
and the ability to materialize the timestamp from
the time views.

this commit also changes the response type of the
`Inspect()` query to be a tabular `RowResponse`.
2019-10-29 14:27:07 -05:00
Ben Johnson
c7c9c1e1d7
v2.0.0
Co-authored-by: Cody Soyland <codysoyland@gmail.com>
2019-10-08 14:56:17 -06:00
Cody Soyland
0ec9d3b7e6 Add support for TLS client certificate verification and custom CA 2019-09-24 11:09:58 -05:00