Commit graph

400 commits

Author SHA1 Message Date
tgruben
0aa5efcc51
staticcheck fixes (#2278) 2022-11-07 10:51:55 -06:00
Garrison Davis
0f5a56c958 Stop using string keys in contexts
This fixes the OriginalIP and RequestUserID in the main featurebase
package, and the Access and Refresh tokens, the UserInfo, and the
[]string of Indexes passed with context.Context(s) in the authn package.

An empty struct was used for all of these keys (and relevant helper
functions we added) to avoid allocations where possible while still
using the context functionality.

Some of the logic in the server.GetIndexes function was fixed.
2022-11-04 15:01:40 -06:00
tgruben
f987009406
add flag to bypass space check (#2265) 2022-10-31 11:08:01 -05:00
Seebs
c3b032d5cb drop ioutil
The ioutil package is deprecated, with all of its functions having
moved into os or io. Do the replacements so we stop having this
impending.
2022-09-23 16:56:27 -05:00
Matthew Jaffee
5be85a0d47 add persistent history and combination of multi-line commands to CLI
"featurebase cli" will now save command history to
$HOME/.featurebase/cli_history by default. Additionally if a command
is entered across multiple lines, the newlines will be removed when
the command is saved in the history. Previously each line was saved
separately which was a bit annoying.
2022-09-07 13:10:29 -05:00
pokeeffe-molecula
7c936471cc
sql3 changes (#2211)
* first cut of working (slowly) bulk insert; table valued functions and a tuple data type to support time quantums

* oversight

* filter pushdown implementation; bulk insert

* addressed some linter issues
2022-09-06 11:16:11 -05:00
Seebs
2052bb01d8 refactor testing to share clusters more often
When doing tests, we create a ton of one-off clusters. This
turns out to be expensive and slow. Fixing it is surprisingly hard.

Fundamentally: If we're sharing clusters, we need to use different
indexes for each test, to avoid clashes. This changes index names.
As a side-effect, this reorders many partition-based things, like
the order keys are returned in. Thus, to fix this, we change a lot
of tests to no longer depend on the *order* in which strings are
returned.

Having done that, we can also discard the ModHasher behavior, since
that only existed to allow us to reliably predict partitioning.

The basic design is as follows: Instead of a cluster being a
[]*Command, a "shareable" cluster is now a []*Command plus some
flags, and a "cluster" is a pointer to a possibly-shared cluster,
plus a link to the specific test using this specific cluster,
and correspondingly, its test name suitably coerced to be a valid
index name prefix.

The "test.Cluster" object now has methods to allow retrieving an
index name, and also implemnts fmt.Formatter to let you use,
e.g., `%i` with it in Sprintf to get "the index name, plus an i".
(This works for everything but %p and %T.)

This allows us to consistently rework all the many things that
use index names in a persistent way.

We also have `MustUnshared` and `MustRunUnsharedCluster` methods
which allow us to specify that a given test needs its own cluster
for some reason. For instance, the tests that want to run backups
need their own isolated cluster, and the tests that want to close
or reopen nodes need their own cluster because a reopened cluster
won't have working GRPC for some reason.

On "closing" a shared cluster (actually the test-specific wrapper
that reflects a given sharing), we delete any indexes starting with
that test's index name prefix. Otherwise, the huge pile of open
indexes prevents `go test -race` from working on MacOS, where we
run out of address space too quickly.

This is fairly enormous but most of the individual changes are
fairly trivial things like replacing the string "i" with "c.Idx()".

We also tweaked a test that failed for me a couple of times to
not depend on sort order.
2022-09-02 11:40:37 -05:00
Seebs
397badbccb reduce size of KeyReplication test
This test used to be large, because it was testing some features that
were refactored out in October of 2019. Since we no longer have the
"buffer growth" to check, let's check a much smaller file.
2022-09-02 11:40:37 -05:00
pokeeffe-molecula
e4a4a06af0
added /sql endpoint; implemented SHOW TABLES (#1935)
* squashed 45 commits into one :)

* tlt/sql experiment (#2035)

* Move PlanOperator to sql3/planner/types package

includes:
type PlanOperatorColumn struct
type PlanOperator interface

* Remove planner dependencies from pilosa package

The goal after this is to prevent the planner package (which doesn't exist yet)
from being imported by the pilosa package; we just want it injected into the server
in server/server.go. This is because the planner package uses pilosa types, so we need
to avoid circular dependencies.

Added ExecutionPlannerFn
Make public: pilosa.ExecOptions
Added a pilosa.Executor interface
Added a planner.types.CompilePlanner interface
Isolated the planner calls to:
- Executor.Execute()
- *API.[method]()

* Move executionplanner files into the sql3/planner package.

This required a bit of gymnastics, and there are some things around
FieldOptions which need to be addressed soon.

* Remove the hacky FieldOptions stuff I added earlier

This implementation just uses the pilosa.FieldOption functional options
provided by the API (as opposed to trying to build a FieldOptions
object.

It also changes field types to constants. These are private for now, but
if we need to make them public, we should put them in the planner/types
package.

* Implement the "scale" value from Decimal(scale)

Also, precision and scale were currently reversed in the parser. This
fixes that.

* Modify the parser to handle CACHETYPE <type> SIZE <size>

It's a little odd to me that the cache type values are Tokens, but I
guess it's ok. One thing to keep in mind is that FeatureBase expects
lowercase values, so this commit changes the parser to set the value to
the lowercase version of the type.

* Fix the /sql2 tests

This entailed a combination of commenting out or t.Skip()-ing tests
which covered code in the parser that has been commented out or removed
as not currently supported in sql3.

It also adds some coverage for the sql.Contraint stringers.

* Prevent JSON sql results from containing closing commas

This commit just re-works the existing output code to avoid inserting
closing commas (which results in invalid JSON).

* Enhance the CREATE TABLE test coverage.

In particular, ensure that the fields which get created in FeatureBase
are what we expect based on the fields defined in the CREATE TABLE
statement.

This also ensures that the TIMEQUANTUM and CACHETYPE contraints are not
provided for the same field (since those constraints are not supported
together).

* Adjust the EBNF file to indicate SIZE contraint is optional

A CACHETYPE can be provided without a SIZE. This change indicates that
SIZE is optional.

* Remove `executionplanner_` from file names (#2040)

* implementation of ALTER TABLE (sans column RENAME)

* refactored expression analysis; added more robust type checking; all unary and bin ops function on ints

* added type support for expressions; full bin/unary op support; added cast; more literal support

* cast int to all other types

* all literals (except idset, stringset & timestamp) make it thru; cast to all types with int as source now works

* implemented LIKE/NOT LIKE

* Implemented IS [NOT] NULL

* Move sql2 files into sql3/parser package (#2045)

* Move sql2 files into sql3/parser package

This also removes the sql2 package.

* Fix tests which were typing _id fields as INT intead of ID

* implemented BETWEEN, NOT BETWEEN

* Add featurebase/error package (#2046)

* Add featurebase/error package

I copied the `dax/errors` package which I am starting to use in the DAX
prototype into `featurebase/errors` in order to start using it with the
sql3 package. It's basically a wrapper around `github.com/pkg/errors`,
but it uses a customer coded error.

The sql package can define its own errors based on the
`featurebase/errors` types. Then do things like `Wrap()` and `Is()`.

* Address the linter complaints: shadowed variables, unreachable code

* implemented IN & NOT IN with expression lists

* first cut of CASE

* Fixed some errors from rebase

* updated bnf; removed unused code; tightened up error handling

* first crack at basic CLI for SQL3

Use: `featurebase cli`

Still lots to do here, but for example:

> select count(*) from tremor
+--------------+
|        COUNT |
+--------------+
| 1.158321e+06 |
+--------------+

* Iterate on the CLI (#2057)

Handle the errors.
Add an "exit" command.
Add some general formatting and white space.

Add termination character: ";" (semicolon)

This commit allows a user to provide multiple or partial SQL statements.

Example of multiple statements:
```
show tables; select * from foo;
```

Example of partial (multi-line) statements:
```
select *
from foo;
```

Don't uppercase the header values

* error refactoring; first cut of TOP; remove unused code; use log.Printf instead of fmt.Printf

* fixed a bug with QualifiedRef from refactoring; added bones of INSERT; removal of unused code; tightened up errors more; fixed failing tests

* single value list for INSERT

* Update bnf per discussion with Travis; INSERT now doing the requisite stuff

* Pat's eyes went square - nothing wrong with TOP, Pat needed to learn arrays again.

* improved some errors; fixed tests to suit

* send warnings back in the api; update CLI to display warnings

* start warning on stuff not implemented so we don't get bugged about it

* Tlt/sql experiment (#2063)

* Expresssion -> Expression

* Add SQL planner test

- adds a test to which it is easier to add tables and SQL statments
- un-exports all of the expression types
- removes the planner pointer from the expression types (it can be added
  back later if need be)

* Fix where clause on a string field

Prior to this commit, the binary expression for a where clause on a
string field was building the call by providing a range operator which
is typically used for BSI fields. This changes it to use the call.Args
for string values.

* Update planner tests to handle multiple sql for the same results

* Reorganize SQL tests

Introduce a test/helpers package and move shared MustQueryRows into that
package.

* Add a compatibility map for field types. (#2064)

This is primarily to address the fact that ID fields were previously
incompatible with INT literals.

We should probably consider introducing a custom type for FieldType
which can be used to define compatibilities.

* significantly refactored type checking

* Handle nil (NULL) values in the sql CLI. (#2067)

go-pretty panics if the interface{} field value is nil. This replaces
nil values with a "NULL" string.

* Squash some commits

fixed a still failing test

added line, col to all error messages

refactored source handling to enable table aliases

fixed some copypasta per review

warnings for order by & topn; implemented select as a source

starting to handle in (select...); added stub for optimizer

JSON-encode the sql error and warning strings (#2069)

Error strings with unencoded characters (like double quotes) were
resulting in invalid json.

got insert working; added symbol table; added concrete optimizer; added nascent NestedLoopsOperator; rewrite "where foo in (select..." as inner join

* all about the sets (#2085)

* implemented setcontains()

* implemented set literal; insert set column values; setcontains/all/any both in expr eval and pql filters

* Convert test to use latest framework. (#2086)

* fixed some comments

* removed refactored tests

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add support for Decimal fields to the sql test. (#2090)

* dates (#2094)

* return dates as strings in output; tightened up decimal type checking

* return dates as strings in output; tightened up decimal type checking

* fixed failing tests after decimal changes

* can now insert decimal values

* implemented insert for timestamp data type; implemented current_date, current_timestamp constants

* fixed some failing tests

* handle date literals from strings in insert statements

* changes from feedback

* Fix pointer method error

* sql3 API interface (#2110)

* Introduce API-related interfaces: SchemaAPI, ComputeAPI

The sql3 code was relying on the pointer: *pilosa.API in order to call
API methods directly on the local node. If we want to import and use the
sql3 package in another service (the DAX queryer, for example), we need
to be able to use an implementation of an interface for those API method
calls.

This commit introduces two interfaces, both automatically implemented by
pilosa.API:
- SchemaAPI
- ComputeAPI

* Convert sql3 code to use IndexInfo instead of Index

The sql3 code was relying on a *pilosa.Index and its methods to get
general information like index and field name, type, etc. This commit
converts everything to use a *pilosa.IndexInfo instead.

This allows us to modify the SchemaAPI interface to also return
IndexInfo instead of Index, which will be a lot easier to implement in a
non-pilosa package (like DAX); creating a *pilosa.Index requires
providing things like data directory paths and holders, which are not
necessary for these use cases.

* Unary and Binary Ops R US plus CAST (#2111)

* implemented string literal for timestamp epoch

* fixed failing test

* fixed the failing test again

* refactored tests; implemented unary op tests for all datatypes; implemented binop tests for int/int, int/id, int/decimal & ID/int

* implemented all binary ops for INT & all other types, ID & all other types

* implemented binary ops for DECIMAL types & all other types

* added STRING & BOOL to various tests; implemented all remaining binOp tests

* fix up some stuff after rebasing

* refactored test defs into multiple files; implemented CAST for every datatype

* added tests for like/not like

* addressed review feedback

* addressed type review feedback

* tightened up IS [NOT] NULL behavior plus tests (#2118)

* tightened up IS [NOT] NULL behavior plus tests

* BETWEEN/NOT BETWEEN with all data types

* addressed review feedback

* Handle negative integers in column min/max constraints (#2120)

This commit parses the min/max contraint as an expression, as opposed to
an int literal, so that negative values are treated as Unary
expressions.

There currently isn't support for min/max constraints on `decimal`
fiels, so for now this change only expects +/- integer values.

* Implement the CREATE TABLE keypartitions logic (#2123)

* Execution time, IN/NOT IN & multiple aggregates (#2124)

* added display of execution time

* IN/NOT IN tests for all data types

* fixed date parsing

* removed duplicative tests

* refactoring aggregates

* suport multiple aggregates

* Address review feedback

* final round of feedback

* Add method SchemaAPI.CreateIndexAndFields() (#2127)

In order to support a CREATE TABLE statement as a single command, this
commit alters the SchemaAPI interface to contain a single method which
handles both the index and its fields. It also updates the sql3 code to
use this interface instead of CreateIndex() and CreateField()
indepedently.

* Symbol Handling (Again) (#2129)

* Refactored symbol handling in the planner; re-instated the select as source tests

* removed commented out code

* addressing review feedback

* Move hard-coded _id field out of planner and into interface implementation (#2130)

This commit moves the hard-coded addition of the `_id` field from the
planner to the SchemaAPI.IndexInfo() implementation method.

NOTE: If anything was expecting SchemaAPI.Schema() to also return the
`_id` field as part of its field list in each table, then it would not
be there because the `_id` field is only added in the IndexInfo() method
for now. Currently that's not a problem because nothing is expecting the
`_id` field for `Schema()`.

* Multiple aggregates, all aggregates stand alone and in GROUP BY (#2132)

* handle multiple aggregates in group by queries

* added handling for avg() aggregate both stand alone and in group by

* tightened up sum & avg outside of group by

* added min, max & percentile

* added warnings

* Make MaterializedRowSet implement the PlanOperator interface. (#2133)

This commit refactors the PQLMultiGroupByOperator to have a PlanOperator
as its output. Then, when it initializes, it sets up a
MaterializedRowSet and populates that with the values from the multiple
group by operations.

* added explicit min/max pql operators

* saved a file I forgot to save

* per review

* Un-indent some if/else nesting (#2136)

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add optional `name` argument to test structs.

This commit adds the `name` argument to `tableTest` and `sqlTest` so
that a test can be optionally named. This allows a developer to more
easily run/identify a particular test by name.

* Inbuilt functions (redux) (#2141)

* set functions type parameter type checking

* implemented datepart

* include SQL3 type in SHOW COLUMNS output

* fixed select as source; failing SHOW COLUMNS test

* select in select list

* dump output columns; handle optimization for select list subqueries

* make it an error to return multiple rows for a select list subquery

* added description

* contants and test coverage for datepart function

* SQL3 Refactor-palooza (#2182)

* removed unneeded IsAggregate()

* first cut of working nested loops operator aka INNER JOIN

* remove selectListItemPlanExpression

* added some warnings

* all the tests are passing again!

* addressed some linter complaints

* added basic order by

* bug fixes; added 'or replace'/'replace' to insert

* for insert references should return appropriately

* added back ability to use subquery singleton expressions

* removed dead code; fixed test

* json-able plan, Schema() plus refactoring

* fixed dumb code

* add some tests for time quantum behavior

* Code cleanup during review. Also fixed INSERT to keyed table bug.

This commit contains a lot of minor adjustments made during code review.

It also contains a bug fix that was preventing INSERT into a keyed table
(i.e. _id type STRING) from working.

Co-authored-by: Travis Turner <travis@molecula.com>

* Fix expected min/max on timestamp column test (decimal field)

I don't know why this changed, but presumably something to do with
decimal related work that happened on master.

* Fix compile problem after rebase

* review feedback

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
2022-08-31 13:01:05 -07:00
Hoang Pham
650b7eaa45 FB-1627 - Added backup/restore tar. Purpose: for cloud team to backup and restore directly through stdout, stdin 2022-08-29 11:03:10 -05:00
reesporte
b9865ee99a add row-col-mode to featurebase import
most types are imported in the format `<row>,<col>`, but ints and decimals
aren't. with this new flag, ints and decimals are imported using the
`<row>,<col>` format, instead of `<col>,<row>`.
2022-08-02 12:12:53 -05:00
tgruben
0300cb071f
[FB-1598] Phase 0 of etcd extraction (#2169)
* Phase 0 of etcd extraction

* added test shell

* bare basic test

* address comments
2022-07-29 15:27:04 -05:00
Samir Patel
6f1514933b verify available space before backup
compares free space in output directory to
the usage of either the data directory or
index depending on what is being backed up.

- adds an http_handler endpoint to get usage
of a particular index
- adds InternalClient methods to get DiskUsage and
IndexUsage
2022-07-29 11:30:36 -05:00
reesporte
7fbe96628d change perms: 0666 -> 0600; 0777 -> 0750 2022-07-21 13:34:46 -05:00
Seebs
b3a4e52a13 simplify, streamline, and possibly debug embedded etcd
The root problem this is attempting to address is sporadic
weird cases in which etcd mistakenly thinks it's down even when
it's up. I am not confident that this is addressed, but there's
a reasonable chance that it is, and I can't trigger it at the
moment, but it was always sporadic, so that doesn't prove much.

There's a lot going on here, and it comes into roughly three
categories.

First: Dropping unused/unneeded code. There's a lot of leftover
bits from the initial development and refactoring of this.

Second: Unifying and shuffling some of the design. We had
multiple interfaces which are functionally impossible to
usefully implement separately, so they're combined together,
and in some cases, moved.

Third: Streamlining logic and simplifying design choices.

This is combined into one commit because the changes are
thoroughly entertwined with each other and you can't usefully
break most of them out.

Also, a bunch of test coverage for most of these changes.

Big changes:

We merge the topology and disco packages.  The topology and disco
packages being separate creates a complicated tangle of problems
and dependencies.  The fundamental problem, approximately, is that
topology.Node has to track disco.NodeState.

There's three core interfaces interacting here:
	topology.Noder (maintains list of nodes)
	disco.Stator (maintains the state of a node)
	disco.Metadator (stores, possibly retrieves, node metadata)
But the node state mantained by the Noder *is* the set of node
metadata, plus state updates produced by Stators. The only actual
non-trivial and usable implementation of these interfaces is a single
thing which implements all three, and in which the implementations
share a single backend data source which they are all modifying.

But you can't move Noder into disco, because Noder has to refer
to topology.Node, but topology.Node refers to disco.

Solution: First, merge these two packages. Second, merge these
three interfaces, to provide a single interface which is more
clear about the fact that (metadator.)SetMetadata() and
(stator.)Started() are both changing the output we'll get from
(noder.)Nodes().

We rework the node state tracking.

We have this nodeStates map which is almost unused. Really, we
don't need it at all. Every node's state is either its last heartbeat
state or "Unknown", so we simplify this a bit. Also, we ensure that
the populateNodeStates function itself is yielding the sorted nodes
list, so we don't have to be as worried about possible later lookups
of sortedNodes happening outside a lock. We also add diagnostics
for deleting nodes from the metadata list (this should never happen),
and try to track heartbeat state more closely.

This is *probably* what fixes the underlying reported problem,
if anything did.

Still an open issue: Make heartbeat state changes aware of when
they're talking about *this* node and possibly not try to
mark it down? Except this may have a flaw: That would result in
each node disagreeing with other nodes in etcd about the state
of that node in the failure cases, and undermine the point of
using etcd to keep these states consistent.

We reduce the number of contexts and cancelfuncs in the etcd wrapper.

We create a shared context for the non-etcd.embed children of our
etcd wrapper, the heartbeat/keepalive and the node watcher, so we
can cancel that one context and cancel all of those at once, so
we don't need to separately track a function to call to cancel
the watch, AND be closing another channel. Also, our shutdown
now propagates automatically to the various etcd API calls we've
made for things like the node watcher and keepalive calls.

We still need to watch that channel in watchNodesOnce, though,
because apparently the watch doesn't yield an error even if the
context calling it is canceled. Whee.

This should reduce the risk of ending up in an inconsistent state,
and also the Close() function is probably idempotent now.

Smaller changes:

* Remove config-generators that existed to generate etcd
  configs but were used only for tests that no longer exist
  or make sense.
* Move the logic to generate etcd configs into the etcd
  package, instead of the "testing" subpackage. This allows
  us to write a self-contained config generator for
  clusters where the nodes know about each other, but do
  this just with etcd, not with full featurebase servers.
* Move the thing generating `fake:%d` socket names into
  the etcd package, which is the only place we use it.
  Also simplify it slightly.
* Don't panic on invalid URLs, report errors from them.
* At least try to use etcd's config.Validate functionality.
  It's underdocumented, so we're not sure what it will report,
  but at least if it does we'll get reports from it and
  know what they are?
* Try to handle CompactRevision errors from watches more
  correctly -- after a CompactRevision, any future attempt
  to watch from a lower revision will necessarily fail, so
  we adjust our target revision up. We don't have good
  testing for this.
* Drop the Metadata() method (that used to be in Metadator)
  because nothing ever used it and it didn't make much sense
  to try.
* Convert SetMetadata from taking an arbitrary json blob
  to taking the only data that would ever be valid since
  we always use it to extract node information anyway.
* Drop several unused functions, unexport things only used
  internally.
* Replace Started() with SetState("STARTED"), allowing us
  to write tests that mess with states. We weren't really thinking
  carefully about state transitions sometimes and now it's much
  easier to do that thinking.
* Stop leaving stray localhost:2380 and localhost:2379 in
  our embed config. We still sometimes see peer requests from
  those and I honestly don't know why, but at least it should
  be rarer.
2022-07-21 11:42:35 -05:00
souhailanoor
91e59e4376
Create a pool of goroutines bounded to hardware limits. This should resolve "connection refused" error from spawning too many goroutines at once. (#2138) 2022-06-29 16:22:37 -05:00
reesporte
0d91e2991d Revert "FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)"
This reverts commit acbcec54c8.
2022-06-17 14:58:30 -05:00
rachithrr
acbcec54c8
FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)
* Revert "make pql.Decimal.Value a private big.Int field"

This reverts commit eab6174388.

* Revert "pql.Decimal for DecimalVal in ValCount&GroupCount"

This reverts commit a0c9eec410.

* Revert "Add AddDecimal support to pql"

This reverts commit 50787fd37a.
2022-06-15 09:15:37 -07:00
reesporte
eab6174388 make pql.Decimal.Value a private big.Int field
This ensures that we can't overflow when adding `pql.Decimal`s together. The
only place we can possibly overflow is when converting pql.Decimal to an Int64,
but that is a risk we have to take. Also, the only place we do this is in our
ToRowser. We could maybe change that to strings, so the presentation of data
doesn't indicate an overflow, but that is a later decision to make. It will
also involve fixing the generate-proto-grpc make command, because that's broken
rn.
2022-06-06 16:44:01 -05:00
Matthew Jaffee
991d3d7780 remove experimental RBF Viz stuff 2022-05-27 11:25:17 -05:00
Matthew Jaffee
772496b440 "all bitmap" multi-field, single-shard ingest
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.

It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
2022-05-27 11:25:17 -05:00
reesporte
8ba81643d2
[FB-1379] Create a featurebase subcommand to obtain an auth token (#2079)
* Add CleanOAuthConfig endpoint

We will use this to get the OAuthConfig information, without the client secret, from
FeatureBase without having to have access to the config file. This will be useful
for the auth-token subcommand.

* Add string manipulation utility functions

Go doesn't have native support for these kind of things, so I added this to make it
easier to do string reversal, and replacing the first string encountered from the
end of the string to the front.

* Add auth-token subcommand

This is for work on [FB-1379](https://molecula.atlassian.net/browse/FB-1379).

We need this new auth-token subcommand to allow users to get access and refresh
tokens without having to login to featurebase via the UI. This commit adds that
functionality.

* error on oauth endpoint if auth isn't on

* https as default scheme in cmd, not internalclient
2022-05-26 11:35:51 -05:00
reesporte
60e6900c2e
Add refresh token header/cookie (#2071)
* Add refresh token header/cookie

As part of work on automatic refreshing of access tokens in the grafana plugin
(FB-1377), we will now accept a refresh token in the "X-Molecula-Refresh-Token"
header or the "refresh-molecula-chip" cookie.

This refresh token will be used if the access token is expired. To achieve this,
there was a lot of plumbing that had to be done. Here is a list of some of it:

* Added lots of constants for the new values.
* Removed token cache, since we will be keeping state on the clients.
* We now only refresh tokens when they are expired, which is more inline with the
  OAuth spec.
* Refactored SetGRPCMetadata to be simpler to read.
* Refactored AddAuthToken.
* Update failing tests.
* We now don't split GRPC cookies on ";". Not sure why we did that before tbh.

I also added TODOs to add the refresh token to other subcommands. This is out of
scope for my current ticket, but it would be nice to have in the future.

* remove unnecessary context from Authenticate

* Add comments on why we check both cases for headers

It's because some GRPC clients lowercase metadata names. I've run into issues with
this enough that I think it's worth the extra checks. We prefer lowercase though,
because that's "standard".

* Fix test that broke during rebase
2022-05-20 16:12:27 -05:00
souhailanoor
3986e202bf
FB-1378: Use IP whitelisting for ingest authentication and authorization (#2070)
* Use IP whitelisting for ingest
For ingest, use configured IPs to authenticate the requests.
Auth-token will no longer be used for requests from ingest consumers.
If IP in request is in configured IPs, authenticate and authorize as an admin.
If IP in request is not in configured IPs, proceed with the standard authentication/authorization using ADD.

* need to remove port from client IP

* addressed review comments
2022-05-20 14:57:05 -05:00
tgruben
09b4eed348
[FB-1388] Add optional support for continuous profiling in datadog (#2037)
Enable ability to use DataDog continuous profiling
2022-05-12 12:33:46 -05:00
Matthew Jaffee
e0e01f9f65 expose --header-timeout option on featurebase backup 2022-04-29 16:37:40 -05:00
Matthew Jaffee
dee46d4423 add PartitionToNodeAssignment as a new option
We default to the jmp-hash method which we had previously, and allow a
user to set the "modulus" option which uses a simple mod operation to
ensure an even spread of partitions across nodes.

I think that ideally we would have new indexes uses modulus and
existing indexes use jmp-hash which implies supporting this
configuration on a per-index basis.

If we don't do per index, we should probably run the whole test suite
both ways.
2022-04-27 17:21:47 -05:00
reesporte
ea33a2b976 fix broken test 2022-04-14 09:37:33 -05:00
Todd Gruben
74ab06f5d3 expose etcd ttl timeout period 2022-04-11 08:58:39 -05:00
Seebs
a6aa1097f1 listen on localhost:0 instead of :0
MacOS's firewall complains about a previously unknown app trying to
listen for network connections whenever we run go test. That's because
we *are* listening for network connections on arbitrary interfaces, not
just on localhost as we probably intended. Fix that.
2022-03-18 13:38:30 -05:00
reesporte
17203e3441 remove cardinality calculation from schema/details
this is related to work for [fb-1127](https://molecula.atlassian.net/browse/FB-1127)

cardinality reporting has caused no shortage of issues such that we recommend
disabling them almost everywhere.

this commit removes the cardinality calculation for right now, as well as the option
to enable/disable schema details.
2022-03-02 14:09:09 -06:00
reesporte
248dc4fe85 rip out ui/usage
addresses concerns in [fb-1127](https://molecula.atlassian.net/browse/FB-1127)

TLDR;
/ui/usage was a hotbed for issues and SEs have been turning it off anyway for ages
2022-02-28 12:05:44 -06:00
reesporte
88d2914b15 fb1172: enable refresh tokens
- rip out gobby stuff
- add tokenCache, groupsCache
- refresh the token if needed
- set cookies after authenticate
- remove signature validation, the IDP does that for us
- added way more unit tests
- update older tests to use new API
- add fake idp to authcluster tests
2022-02-07 13:42:11 -06:00
Matthew Jaffee
254bacc40c remove http subpackage and bring implementations into core
remove interfaces as necessary
2022-02-03 21:04:04 -06:00
Matthew Jaffee
d1f3b58861 remove inspect command 2022-02-03 11:25:31 -06:00
Matthew Jaffee
bff6b17a8e remove check command (was for roaring backend files) 2022-02-02 20:56:18 -06:00
Matthew Jaffee
70ea784d41 don't mind me, just submitting stuff that doesn't even compile and
then getting confused by linter errors
2022-02-02 20:56:18 -06:00
Matthew Jaffee
69c00a92ad remove a bunch of roaring backend stuff
snapshotQueue, op tracking, roaring-only tests
2022-02-02 20:56:18 -06:00
Matthew Jaffee
61783e5827 add option to set ResponseHeaderTimeout per client
this is necessary as in some cases we want a low timeout (when we
expect a quick response, e.g. with backup), but in others we may want
a very long timeout (long running query).

Now we have more granular control over timeouts so we can get things
to fail more predictably in tests.
2022-02-02 14:04:40 -06:00
Souhaila Noor
0e1cf5bbbd Enable authentication/authorization for featurebase tools
- Add auth-token for featurebase import, backup and restore
- Add auth-token to http request
- Create a cluster tests with auth enabled
- Add test for import with auth enabled
2022-01-26 17:30:26 -06:00
Ben Johnson
9ebf0e2119 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
Seebs
375aaf8fbc don't hardcode local port for backup and restore pprof service
If we hardcode a port, we can't run on a crowded machine, like in
CI. If we use :0, we can print the value actually picked.
2022-01-21 11:12:10 -06:00
reese
2e0aaa27a9
Merge branch 'master' into redirect-url 2022-01-20 13:08:16 -06:00
reesporte
87bbca938c add a redirect-base-url config option
this allows the user to configure a url for their IDP to redirect to, rather
than relying on the bind address of the featurebase server itself
2022-01-20 12:09:18 -06:00
Samir Patel
e1d7389893
Update ctl/keygen.go
Co-authored-by: reese <45641995+reesporte@users.noreply.github.com>
2022-01-19 11:23:02 -05:00
Samir Patel
950e62aae9 update keygen subcommand
this updates the subcommand to output a single secret key
instead of two reflecting changes made to AuthN/authZ
2022-01-19 10:11:19 -06:00
reesporte
cf2410fea6 addresses multiple authn/z tickets
* fb-998 - authn/z enabled in handlers (kitchen-sink ticket)
    - authorization is enabled through the use of a bearer token (using header "Authorization")
    - authorization may occur through the use of an "Authorization" header or "molecula-chip" cookie
    - ui is updated for changes to handler
* fb-1131 - protect grpc endpoints
    - GRPC endpoints now check authorization if auth is enabled
* fb-1129 - inter-node communication
    - the following endpoints use the secretKey for authentication:
        - /internal/cluster/message: POST
        - /internal/translate/data: GET, POST

* added test to api_test.go (TestAuth_MultiNode) testing various auth/permissions stuff on a multi-node cluster

not included:
    - fb-1130 - filter response of endpoints
    - fb-1109 - improved audit logging

@jaffee [are you not entertained](https://www.youtube.com/watch?v=mutgotxrcqg)

Co-authored-by: souhailanoor <90720110+souhailanoor@users.noreply.github.com>
Co-authored-by: tgruben <tgruben@gmail.com>
Co-authored-by: 54mir <48686912+54mir@users.noreply.github.com>
Co-authored-by: kcrodgers24 <49999391+kcrodgers24@users.noreply.github.com>
2022-01-14 12:31:32 -06:00
Matthew Jaffee
34393dee09 rip out rowcache
not strictly backward compatible... hopefully no one is actually using
the rowcache config option
2022-01-11 13:49:09 -06:00
Matthew Jaffee
6335b9c801 disable retryablehttp logger because *wow* that's a lot of output 2022-01-11 10:15:58 -06:00
reesporte
8d6490329b Merge branch 'master' into protect-endpoints 2022-01-04 16:18:18 -06:00