* got single node working; refining error messaging and version info to follow
* better implementation that separates the build condition into etcd/enterprise_cluster.go and etcd/plg_cluster.go. go build will default to a clustering version and 'go build -tags plg' will build the non-clustering version
* added Makefile target for 'make plg'
* additional comments, CI/CD update
Co-authored-by: Kasey Rodgers <kaseyrodgers@Kaseys-MBP.attlocal.net>
We only actually check presence/absence in this map, we never
set the value stored to false, and we assume in some places that
any value present is equivalent to true, so we might as well
use a map of struct{} and save the several whole bytes of memory.
add sentry for error monitoring and performance tracking. Must call the init function to actually turn on the feature. This is expected to be used in the PLG binary and not the enterprise binary.
This ensures that we can't overflow when adding `pql.Decimal`s together. The
only place we can possibly overflow is when converting pql.Decimal to an Int64,
but that is a risk we have to take. Also, the only place we do this is in our
ToRowser. We could maybe change that to strings, so the presentation of data
doesn't indicate an overflow, but that is a later decision to make. It will
also involve fixing the generate-proto-grpc make command, because that's broken
rn.
This way we can avoid annoying floating point rounding errors.
Check out FB-1359 for an example:
```
--- FAIL: TestExecutor_GroupByStrings (0.55s)
--- FAIL: TestExecutor_GroupByStrings/3 (0.00s)
executor_test.go:5433: unexpected result at 0:
got:{Group:[generals.1.r1] Count:5 Agg:2775
DecimalAgg:27.749999999999996}
want:{Group:[generals.1.r1] Count:5 Agg:2775 DecimalAgg:27.75}
```
* mutex clear on nil support with test
* Update client/batch.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
* spot instance test
* update outputs.tf to provide spot instances
* uptate outputs.tf data_node_ips
* propogate spot instance request tags to the instances
it's not something that we can send to the server (it calculates base
off of min and max), and we don't need it in the json string in order
to use it client side when building import requests.
There are 16 containers in a row in a shard with the default shard
width of 2^20, but since you can change the shard width at compile
time, everything should be computed off that.
if the same ID is added multiple times with different values, only the
last value should get set. Without this change, if the multiple
records weren't immediately next to each other, all the bits from all
the values would get set.
the ranked cache must be fully rebuilt as the BitmapRewriter doesn't
have an easy way to track which rows had how many bits changed which
we would need to update the cache.
We also needed to add a Clear method to topn cache to completely
remove old values before the rebuild as otherwise they'd sit there and
pollute the cache after it was rebuilt.
This also includes fixing a strange idiosyncrasy where the _exists
field was a set field, but didn't have its type explicitly set. This
was causing it to have a ranked cache even though that option was
turned off. Hoping this doesn't have any weird follow-on effects... or
if it does the tests catch them.
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.
It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
* Add CleanOAuthConfig endpoint
We will use this to get the OAuthConfig information, without the client secret, from
FeatureBase without having to have access to the config file. This will be useful
for the auth-token subcommand.
* Add string manipulation utility functions
Go doesn't have native support for these kind of things, so I added this to make it
easier to do string reversal, and replacing the first string encountered from the
end of the string to the front.
* Add auth-token subcommand
This is for work on [FB-1379](https://molecula.atlassian.net/browse/FB-1379).
We need this new auth-token subcommand to allow users to get access and refresh
tokens without having to login to featurebase via the UI. This commit adds that
functionality.
* error on oauth endpoint if auth isn't on
* https as default scheme in cmd, not internalclient
* ugly first cut at supportings Rows(in=[...])
need tests, better handling of various combinations of arguments and
error cases
* explicitly error when other arguments passed with 'in' to Rows
* first cut at supporting Rows(in=[...])
'in' is explicitly not supported with any other arguments (except the
field of course), and will error. It works both as a standalone Rows
call and in GroupBy.
* bitmapfilter require ordered rowids
* remove log message
Co-authored-by: Todd Gruben <todd@molecula.com>
This fixes a bug where the frontend UI sends GRPC cookies via a single semi-colon
separated string, which our new token parsing algorithm did not recognize as valid.
Now we account for that special case, at the small expense of greater overall
computational complexity.
* FB-1403 - fixed TTL to not allowed negative values
* FB-1403 - TTL - fixed comment
* FB-1403 - TTL - added comments for unit test, removed extra if statement, removed extra unit test
* Add refresh token header/cookie
As part of work on automatic refreshing of access tokens in the grafana plugin
(FB-1377), we will now accept a refresh token in the "X-Molecula-Refresh-Token"
header or the "refresh-molecula-chip" cookie.
This refresh token will be used if the access token is expired. To achieve this,
there was a lot of plumbing that had to be done. Here is a list of some of it:
* Added lots of constants for the new values.
* Removed token cache, since we will be keeping state on the clients.
* We now only refresh tokens when they are expired, which is more inline with the
OAuth spec.
* Refactored SetGRPCMetadata to be simpler to read.
* Refactored AddAuthToken.
* Update failing tests.
* We now don't split GRPC cookies on ";". Not sure why we did that before tbh.
I also added TODOs to add the refresh token to other subcommands. This is out of
scope for my current ticket, but it would be nice to have in the future.
* remove unnecessary context from Authenticate
* Add comments on why we check both cases for headers
It's because some GRPC clients lowercase metadata names. I've run into issues with
this enough that I think it's worth the extra checks. We prefer lowercase though,
because that's "standard".
* Fix test that broke during rebase
* Use IP whitelisting for ingest
For ingest, use configured IPs to authenticate the requests.
Auth-token will no longer be used for requests from ingest consumers.
If IP in request is in configured IPs, authenticate and authorize as an admin.
If IP in request is not in configured IPs, proceed with the standard authentication/authorization using ADD.
* need to remove port from client IP
* addressed review comments
swallowing this error was a mistake... caller would not know that
ingest failed and might incorrectly report success
upstream (e.g. committing offsets to Kafka)
* Changed several gauntlet tests to be manual due to cost emergency
* Made perf_able manual
Co-authored-by: Fletcher Haynes <fletcher.haynes@molecula.com>
out of range. Previously internal server error was returned.
This is to allow for ingest to continue while logging bad values
instead of stopping ingest as we do when there is a server error.
We've been seeing weird retention of Tx that shouldn't still be open, and
one possible explanation is that, until a Tx actually uses the freelist
cursor (either to allocate a page or to release it back to the freelist),
the freelistCursor statically stored in the Db object continues to have a
pointer to the previous Tx which used it, which allows a Tx, and thus its
dirty page map, to be retained forever.
I previously thought this should also nil out the page maps in the Tx, but
the more I think about it, the less I think that's a good idea. The actual
lifespan of a committed Tx should be quite short. If it *does* stick around,
it's beneficial to us as debuggers to see those large maps of dirty pages
sticking around. So after thinking about it a lot I decided not to do
that.
Similarly, when closing out a container filter (whether a filter or
a rewriter), zero out the Cursor, Tx, and filter and rewriter functions.
(We don't have to worry about the cursor's Tx, because the cursor gets
closed, which zeros its Tx and returns the cursor to the cursor pool,
too.) This likely matters a lot less, as the filters in the pool
get garbage collected "soon", but it still reduces the amount of
stuff being retained.
* make CookieName an exported constant
* fix SetGRPCMetadata
this will actually set the grpc metadata even if there are no cookies in the
metadata already.
* gofmt yourself
* FB- 1456 - TTL - fixed views not returning correct results when least precise quantum are deleted
* FB-1456 - TTL - PR - fixed comment
* FB-1456 - TTL - changed getQuantum to getLowestGranularityQuantum since we only care about the least precise quantum that is available
* FB-1456 - TTL - removed unit test used for debug
* FB-1456 - TTL - fixed comments
* [FB-1435] BSI Base Fix (#2056)
* add bsi base back to int value
* test bsi base/min/max for IntFields
motivated by bsi base not being added back to values
in extract calls when min was a positive integer.
* FB-1456 - TTL - fixed comments
Co-authored-by: Samir Patel <48686912+54mir@users.noreply.github.com>
* add bsi base back to int value
* test bsi base/min/max for IntFields
motivated by bsi base not being added back to values
in extract calls when min was a positive integer.
