In this commit, the Directive is mocked; it doesn't actually reach out
to a controller.
Limits key translation to only those partitions (per index) specified in
the Directive. Attempting to create or find a key (or ID) for a
partition which is not handled by this node will result in an error;
translation requests are no longer forwarded to other nodes.
Limits import into only those shards specified, per index, by the
Directive. Attempting to import into a shard which is not handled by
this node will result in an error.
Stub out /directive endpoint
The `applyDirective()` method still needs to be implemented.
Update mds references to use the new /mds/types structure
In mds, we moved the shared types to mds/types. FeatureBase needs to
reference those instead.
This also bumps the mds version in go.mod.
Implement the Add/Remove Index part of Holder.ApplyDirective()
This adds functionality to `Holder.ApplyDirective()` which adds or
removes indexes (tables) based on those provided in the Directive. Still
to be implemented here: shards and partitions.
WIP: remove client from Batch
Move Batch into its own package: batch
Also, in order to avoid import loops, this introduces packages:
/batch/types
/client/types
Reorganize the Importer-related code
Moved the Importer interface to package: batch
Move the "pilosa client" implementation of the Importer interface to
package: client
Modify batch.NewBatch to take an Importer (not client)
This commit modifies the batch.NewBatch() function to use a functinal
option on Batch to inject an Importer into the Batch. Prior to this,
NewBatch() took a pointer to a client, which was a little too
restrictive. Now, MDS can implement an Importer which uses information
from MDS to determine to which node(s) the import calls should be directed.
Add client.SetAuthToken() method to satisfy SchemaManager interface
Update ApplyDirective logic to include fields.
This needs more work, but it was enough to get a basic test passing.
Move Transaction type into /types package.
Add interface check on batch.Importer no-op implementation
Updated ApplyDirective to create all currently support Field types
There are still the following TODOs:
- [ ] impolement field options (ex: decimal scale, int min/max, etc).
- [ ] `time` fields
Added support for Decimal.Scale in ApplyDirective
Update mds dependency
Add /health endpoint
Update to use dax (dax/mds) instead of mds.
After moving the mds repository into the dax repository as a
sub-package, this commit changes everything in FeatureBase to use the
dax repo instead of the now abandoned mds repo.
Introduce and implment the WriteLogger interfaces.
This adds both a `WriteLogReader` and `WriteLogWriter` interface. They
are both implemented by the implementation: `fileWriteLogger`. The
`fileWriteLogger` uses the dax/writelogger API to append log messages to
files on disk.
Add WriteLogWriter.ImportRoaring method to interface
This commit adds the `ImportRoaring` method to the `WriteLogWriter`
interface. Still to implement are the `Import` and `ImportValue`
methods.
Reorganize the ApplyDirective code
The primary goal was to cache the incoming Directive on the Holder prior
to applying all of the changes in the directive (i.e. loading data from
the WriteLogger) because applying those changes often validated against
the accepted state of the node.
Implement all of the WriteLogger read/write methods
Implement the HTTP WriteLogger implementation
WIP: Introduce shard.Version. Implement snapshotter.
Add HTTP Snapshotter implementation
This also recofigures server to use the HTTPSnapshotter instead of the
FileSnapshotter.
Implement snapshotter: TableKeys
Implement snapshotter: FieldKeys
Dependency dance
last of the dependency dance
Add support for prototype
This adds the Makefile targets to build the docker container and push it
to ECR.
SQL3 changes which break with dax changes
Missed TODO: implement FieldVersion version to WriteLogger
Address bug causing missing TranslateStores to error
Originally, we tried to limit the TranslateStores which get allocated to
only those for which the node is responsible. This works when adding a
new table. But if a table already exists, there's no logic to start
missing TranslateStores.
This reverts back to the old FeatureBase logic which brutishly allocates
a TranslateStore for every partition, even if one is not needed.
We need to address this by allowing the ApplyDirective logic to
initialize TranslateStores when they don't yet exist.
Move the ImportRoaringShardRequest type to the types package
Since the ImportRoaringShardRequest object is part of the Importer
interface, we need to move it to a non-root (i.e. pilosa) package. All
the other interface types are either concrete types or part of a
sub-package (such as roaring). We do this to prevent an implementer of
the interface from having to import the entire pilosa package and risk
circular imports.
buncha changes to support latest dax stuff
Move dax related types to /dax sub-package
This commit moves all the common "dax" types into the /dax sub-package.
The idea is to ensure that featurebase does not import dax at all.
It's ok if dax imports featurebase.
In the future, we might need to split the dax sub-package (common data
types used by muliple molecula data-plan services) into it's own repo.
Add type: dax.Schema
This isn't currently being used; I started to use is as a replacement
for pilosa_client.Schema, but then deferred that. But we'll need to do
it eventually, so it doesn't hurt to have this type in place.
Export RowIDs.Merge() method for use in orchestrator.
Add CreateSQL method to dax.Table type
The CreateSQL() method will return the "CREATE TABLE" statement required
to create the dax.Table.
Comment out confusing writelogger log message.
We need to revisit this, but for now, this log message is confusing.
Also, rename daxSharder to versionStore.
Remove hard-coded AWS account
Implement more FieldOptions such as Epoch
Some of the FieldOption logic was stubbed out in the dax package. This
commit fills that out more; specifically, it adds the
dax.Field.Options.Epoch parameter.
export stuff needed for TopK in orchestrator
export ValCount stuff to implement Percentile in orchestrator
export more stuff to support less code in orchestrator, shared objs
Port dax repo over to featurebase/dax (run all as sub-services)
This commit does ALOT. Sorry.
It introduces a `featurebase dax` sub-command which can be configured to
run the various dax services as sub-services within the same process, or
individually as the lone service in process.
It also changes all the URL paths to be prefixed with the service name.
So for example, instead of calling localhost:8080/status, you would now
call localhost:8080/featurebase/status.
Also, note that all services provide a /health endpoint to confirm they
are running in process.
Clean up integration tests. Remove PILOSA_ config prefix.
Remove duplicate clients (mistake from porting dax to featurebase)
Rename sub-service "featurebase" to "computer"
In the places where we have hard-coded the sub-service name into a URI
path, I've tried to tag the line with a comment containing:
`// #SERVICEPATHPREFIX`
Update copilot manifest files to reference "computer"
Port dax/README.md from dax repository
Separate (toml) Queryer Config from Injections
We needed to separate the toml config from the configuration required to
inject sub-services into the Queryer. I'm not sure this is the best
solution, but it's *a* solution. So here we are.
Clean up (i.e. remove) the queryer "implementations" package
Remove old test file
Run WriteLogger and Snapshotter as local sub-services.
Prior to this commit, the writelogger and snapshotter services only
worked when run as separate services. This allows them to be run in the
same process as all the other dax services.
There is still some naming issues that we should address, but it's
functional for now.
Clean up (i.e. organize) the intra-service interfaces.
Implement alpha Director for local messages from MDS to Computer
Prior to this commit, messages from MDS to the computer service were
still going over http. This commit introduces an interface
implementation which registers the local computer command, and use that
command's API to directly reference methods used by the Director.
Clean up a few more interface names
Add Queryer OpenAPI document.
Update copilot manifests to reflect latest changes
Add OpenAPI documents for WriteLogger and Snapshotter
Add OpenAPI document for MDS service
Add OpenAPI document for Computer service
Consolidate errors to use fb/errors package.
This commit is a first pass at trying to ensure that all of the DAX code
uses:
"github.com/molecula/featurebase/v3/errors"
This package is a wrapper for "github.com/pkg/errors", so going forward
we want to avoid importing that package.
The only method which isn't backward-compatible is `New()`; the
New() method in the featurebase/errors package takes an errors.Code. If
this becomes a problem, we could change this by reverting New() and then
introducing something like NewCoded(). But for now I think it might
actually discourage someone from just creating a New() error without
thinking about how it should be coded.
Introduce VersionStore interface
Move the existing VersionStore code to the `inmem` package as the
in-memory implementation of the new dax.VersionStore interface.
Introduce NodeService interface
With this, the Controller can maintain a registry of nodes by using this
NodeService interface as opposed to an in-memory map of nodes on the
Controller struct.
This also adds an inmem implementation of the NodeService interface.
Introduce controller.Balancer interface
This moves the existing balancer package to controller/naive package.
The idea is to allow us to add a different Balancer implementation in
the future.
Introduce DirectiveVersion interface
This commit also includes *A LOT* of refactoring to use dax.Worker and
dax.Job types everywhere instead of strings.
Introduce Schemar interface
The previous `Schemar` struct was moved to the `schemar/inmem` package,
and `Schemar` is now an interface implemented by that inmem package.
Remove unused type `nUnit`
Add boltdb implementation of VersionStore interface.
This removed the previous sqlite implementation; we decided not to use
sqlite for now (as a basic, local disk implementation) because it
requires CGO.
--------------------------------------------
No longer applicable:
Add sqlite implementation of VersionStore interface.
This commit implements the VersionStore interface using sqlite. Sqlite
requires CGO, so this may not be something we want to include, but it's
implemented here to get a feel for how an external implementation might
be used; the next step will be to determine how the user configured
FeatureBase to run using sqlite as a backing store for services like
MDS.
Add boltdb implementation of NodeService and DirectiveVersion interfaces.
Add boltdb implementation of naive Balancer interfaces.
This includes the two interfaces defined in `naive/balancer.go`:
- WorkerJobService
- FreeJobService
Add boltdb implementation of Schemar interface.
clean up a linter issue
Thread context.Context through all the interfaces.
Some of the interface implementations are going to use context, so we
need to make that part of the interface. The boltdb implementations, for
example, take a context. This is probably so we can do things like
cancel or timeout operations.
Update interfaces to return error; remove `panic(err)` everywhere.
Down-rev grpc version to 1.38.0
Later versions (after 1.42.0?) cause MustRunCluster.Close() in tests to
deadlock.
This commit also adds an `isComputeNode` feature flag around some of the
write log and shard/partition check functionality so that it doesn't run
under normal conditions (this is excercised by running the sql3 tests
for example).
Add MDS_Persistence test to cover meta data persistence
This adds a basic test which configures the MDS container to use boltdb
as its persistence storage, saved on a docker volume. Then, the mds
container is stopped/replaced, and we confirm that the data stored on
the volume is availble to the new MDS container.
Fix a few things after rebase with sql-experiment branch
The lastest version of sql-experiment contains a fairly significan
refactor of the way query iteration works. This commit adjusts for those
changes.
pull dax IDK changes in to FB IDK (#2177)
* pull dax IDK changes in to FB IDK
* Move docker-related IDK build stuff to featurebase root
Building the docker image required the root level go.mod and vendor
directory. This change moves the make targets to the root level
Makefile, and the Dockerfiles now copy the root level vendor directory
(and everything else in the root for that matter).
* Fix batch- and client-related tests
* InitializePoller on MDS restart/replacement
Prior to this change, if MDS was restarted, its internal poller (which
maintains an in-memory list of nodes to poll) is empty. This is bad,
because it doesn't know about nodes that it should be polling.
This change fixes that. Upon MDS startup, it intializes the poller with
the list of nodes that MDS keeps in persistent storage (currently:
boltdb).
* Add EFS volume to MDS Copilot manifest
This allows us to use MDS's persistent storage (via boltdb) in the
Copilot demo by saving metadata in a boltdb file on EFS.
* Thread logger.Logger through all dax components
* Revert some of the breaking changes from DAX development.
When we first started prototyping DAX, we made changes to the
featurebase core code which would have broken the existing featurebase
functionality. This commit reverts some of those changes. Anywhere that
we need to modify core featurebase functionilty, we put it behind some
kind of feature flag. This flag is typically determined by whether the
running node is a "compute" node (i.e. DAX.COMPUTER.RUN = true).
Co-authored-by: Travis Turner <travis@molecula.com>
add packaging for DAX
need cgo for datagen build
bind to 0.0.0.0, pass GOOS and GOARCH explicitly
not sure if the explicit GOOS/GOARCH is actually necessary...
Get INSERT INTO (aka ingest) working through SQL3
This commit does a few things which I'll try do describe here.
- Introduces a Qctx interface. The existing Qcx is an implementation of
this interface, and can be used exactly how it has been. But this
allows us to abstract away the notion of Qcx in the Queryer (which is
handling SQL3) until we're ready to address that. As an example, the
Qcx has a notion of a featurebase Holder, but that doesn't make sense
when we're at the Queryer layer. For now, the Qctx used in the Queryer
is a no-op.
- Adds a ComputeAPI interface implementation for the Queryer. This is
effectively the Import() and ImportValues() methods used for ingest.
The logic here handles the incoming ImportRequest by first doing any
necessary column and row translation for the entire request, then it
splits the records by shard, and generates a new ImportRequest per
shard with only the shard-appropriate records.
- Changes the mds.Importer to take an MDS interface implementation
(which can be an mds client) instead of an mdsAddress. This allows us
to use a localy MDS implementation rather than assuming we need a
client to make calls over a network.
Add queryer.Importer interface to handle ingest via SQL (#2203)
* Add queryer.Importer interface to handle ingest via SQL
This is meant to support ingest through SQL when the queryer and the
compute services are running in the same process, or when they are on
seperate processes and need to talk via http client.
* remove datagen from RPM
was originally added as a convenience to generate test data, but is
unused and annoying because datagen doesn't easily cross-compile due
to cgo
* add marshalUnmarshal to controller to avoid passing pointers
passing pointers across API boundaries can cause unpredictable things
in local vs remote configurations.
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
"fix" a few issues with wrong default partition numbers
these still need to be properly fixed and actually get the correct
data from MDS
go mod tidy
Introduce TableQualifier (OrganizationID/DatabaseID) (#2220)
* add check in ApplyDirective that version is increasing
fix TestAPIDirective to make version always increasing
* fix docker image build and break out dax test in CI
We have to run the DAX integration tests separately as they call out
to Docker, and so it isn't easy to run them in a Docker container as
the other tests do. So we run them directly on the CI runner which has
Docker and Go installed.
We also explicitly exclude these tests from running during the other
tests.
Also my editor was automatically reformatting some comments badly
which is why I added the "data" thing in those two places
* add timeout to poller
* give Poller a default Logger
apparently we can NPE sometimes, seen in CI: https://gitlab.com/molecula/featurebase/-/jobs/3028286364
* bunch of testing fixes, mostly IDK/DAX related
make MDS error if sendDirectives errors, don't just
log. sendDirectives can error if computer nodes disagree about the
validity of a schema (for example), in which case it might need to get
deleted and user notified somehow. very messy, needs more thought.
re-introduce old env prefix to maintain compatibility with master
branch
make self-contained dax container for IDK testing
build IDK images from source (now that all the source is available
since it's in the same repo)
catch errors in DoExtractQuery in idktest.go
fix IDK bug where prefix path was hardcoded in all cases rather than
only when useMDS was true
fix TestBatchTargetMDS... needed to add field options and catch error
when creating table. also needed an _id field
* fix env prefix in tests
* WIP getting tests to pass, wanna see CI
* don't error if we get a zero version directive and we don't have a
directive yet
* cleanup debugging junk
* "fix" future.rename thing, run IDK tests
* Introduce TableQualifier (OrganizationID/DatabaseID)
This commit introduces a lot of new types (in dax/table.go) related to
TableQualifer (which is made up of OrganizationID and DatabaseID), as
well as things like TableID and TableKey.
For the most part, we try to thread a QualifiedTableID through the
entirety of DAX. There are some places (for example in the Balancers,
which are just aware of string keys) which use a string TableKey
(tbl__org__db__tableid).
* Remove some debugging comments
* Add Org/DB support to CLI.
This commit adds support for special commands:
SET
SET ORG acme
SET DB db1
USE db1
* remove ".pulled" from IDK Makefile
I don't think we need it any more as most things can be built
locally. I think it was only there to refresh the FeatureBase images
that were tagged as master, but we don't need to do that any more.
* Change DAX json tags to kebab-case (i.e. hyphenated)
This commit also renames some struct arguments to more accurately
reflect their type: for example, renaming `Table` to `TableKey` when the
type is TableKey.
* Return DAX TableName in SHOW TABLES (instead of Index.Name)
There are cases where SchemaAPI is used to return DAX friendly table
names (as opposed to featurebase index names, which are DAX TableKey).
This is an attempt to do that. With that said, it's not ideal because
anything could call those API methods and expect the other type.
* Fix a bug which wasn't completely dropping a table.
When using boltdb as a backend, DROP TABLE wasn't removing the
reverse-lookup key for the table in boltdb.
* Remove idk/testenv/certs which got accidentally committed.
also update .gitignore to include those.
* Fix IDK ingest tests to be TableQualifier aware.
* Add example Table types to dax/table.com godoc.
* ignore idk.Main fields for flags, upgrade commandeer
* go mod tidy
* Fix DAX integration tests: ingester using wrong ENV VARs
We change from ORGANIZATION_ID to ORG_ID
and from DATABASE_ID to DB_ID
* Clarify things around idk (docker) tests
* Stop running TestKafkaSourceIntegration with t.Parallel()
This test can't be run in parallel as it's currently written. Doing so
allows for interleaving of messages to the same kafka topic between
tests.
I didn't attempt to modify the test so it could be run in parallel. That
could be done, but left for someone more ambitious.
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Require Directive.Version be a non-zero value. (#2227)
Because the directive cached on the holder is not a pointer, its default
version is 0. In order to avoid having to compare against that, we just
require that Directive.Version start at 1.
General, non-invasive code cleanup and comment adjustment.
Move ImportRoaringShardRequest out of the types package
Early on in the DAX development, I moved ImportRoaringShardRequest into
a types package. There must have been some import loop going on, but
since that is not longer the case, it's safe to move this back into the
core featurebase (er... pilosa) package.
Move Transaction struct back into the pilosa package (from types)
Revert some name changes (cli -> client)
Add DAX Handler CloseTimeout
This was implemented in htt_handler.go, but it had been commented out in
the DAX handler. This just uncomments that and finishes the
implementation.
Remove Qcx from queryer.Importer interface
This sets us up to revert the Qctx interface that was initially
introduced to allow us to abstract away the need for a Qcx when calling
the ComputeAPI from a remote service (i.e. the queryer).
Add some go-doc comments and remove unused code.
Move SchemaManager setup from datagen to idk.Main (#2233)
The set for idk.SchemaManager (for dax implementations) was previously
in datagen. This may have been because of some import loop problem
during development, but that's no longer an issue.
The setup for this should be in idk.Main so anything using that can
leverage the MDS-specific SchemaManager setup.
Fix issues around nil TxFactory
First, don't return a nil. Rather return a new *TxFactory (with no
holder).
Second, don't call `f.holder` in the testhook outside of checking if
`f.holder` is nil.
Wrap all bare errors
Make service prefixes constants
Instead of having `"computer"` throughout the code, use instead a
constant: `dax.ServicePrefixComputer`.
MDS skip errors when sending empty directives
also add in the docker-login and ecr-push changes for serverless DAX
Fix the logic in Directive.IsEmpty() (#2236)
Update the cached value for Index.translatePartitions
In the case where a node already knows about an index, but its
assignment of partitions for that index changes (for example, when
another node goes down and the node in question is now responsible for
more partitions than it previously was), then we need to update the
cached value of Index.translatePartitions because that's used in
translation checks.
minor fixes for IDK-related bugs
WIP: tokenize CLI to access cloud
FB CLI cloud support with automatic token refresh
Also adds support for a GET command which allows making HTTP GET
queries to cloud CP which can be handy for debugging stuff. E.g. GET /v2/databases
buncha little fixes working on writelogger stuff
fix writelogger/snapshotter setup bugs
implement writelogging for importRoaringShard
add debug endpoint to MDS
use shard transactional endpoint in MDS datagen
add debugging to API related to writelogger
revert handleroption change
clean up big PR
remove "GET" command from CLI for making arbitrary HTTP request to
cloud control plane (was a messy hack and not that useful)
remove json tags from FB objects where we had to duplicate the object
elsewhere due to import loops and weren't actually json encoding it
unexport handlerOption which was exported to try to avoid doing
certain things if we're in DAX mode, but I didn't end up merging that code.
remove (hopefully) unecessary extra call to api.indexField
fix some formatting, unexport some vars, godoc, etc
oops, fix build failure
Update FeatureBase CLI to support a standard deployment
The standard deployment uses a different endpoint and request payload.
This commit tries to detect is the standard deployment is being used,
and if so, it uses a standard-specific FBQueryer.
It also modifies the auto-detection logic to try standard featurebase
and dax ports in the case where a port was not provided.
MDS API refactor (#2259)
* MDS API refactor
table IDs are exposed but only created server side
also cleaned up dax Makefile
* clean up review feedback
Co-authored-by: Travis Turner <travis@pilosa.com>
* remove TablesByName
* rip out inmem implementations and use boltdb everywhere
* remove inmem balancer, create bolt tempfile by default on startup
* WIP on snapshot table impl and test
* Minor comment and code layout adjustments.
This commit also adds the `Equals` method to `QualifiedTableID` for
equality comparisons. It's no longer safe to compare struct (two structs
might still be equal even if one of the structs doesn't have a `Name`
value.
* Use a unique docker network for each dax test
Ocassionally we would see some test failures due to a network already
existing. This shouldn't happen, but to avoid that, this commit
generates a unique name for each sub test (which gets deleted at the end
of every test).
* Fix one instance of NewQualifiedTableID losing Name
We should probably check the other instances and see if Name is getting
lost.
* simplify unique network stuff and fix api directive tests
* Remove TableIDRequest and TableIDResponse types for /table-id (#2267)
For the mds/table-id http requests, just use dax.QualifiedTableID as
both the request and response types.
* remove lattice from dax, no error on node re-reg, dax docker-compose
* various updates
* WIP: mds-refactor branch review
* no-op on SnapshotTableKeys if table is not keyed
* Makefile helpers
* add doWeCare so controller doesn't fail unnecessarily
* clean up table creation (#2272)
* Strip underscores from TableID stub name
* fix boltdb versionstore tests: generate unique, sorted tables
* fix controller test related to reregistering a node
* JobSet -> generic Set
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Cleanup after rebase on master
The latest rebase on master entailed all the client/batch changes as
well as some of the qcx refactoring. It made for a hairy rebase. This
commit fixes some of the tests that were failing after that rebase.
Fix batch/client import loop missed during rebase (#2280)
It's not surprising that `batch` can't import `client`. It was doing
that here (importing an error type from the `client` package). What is
surprising is that it's okay for `batch_test.go` to import `client` even
though `batch_test.go` is an internal test and therefore part of the
`batch` package.
different boltDB's for schemar/controller, explicit balancers
nice helpers for dax docker-compose, make build really fast
build FB binary outside of docker, then create Docker image with its
working dir in an empty subdirectory so it doesn't send a GB of
context to the daemon.
error on unassigned jobs and use client with timeout
fix CR feedback
deregister batch of nodes
also make removal faster via director dial timeout
implement WorkersForJobPrefix so orchestrator doesn't make up shards
also fix some godocs and remove unused method
Run sub-tasks of a Directive concurrently in a worker pool. (#2275)
* Run sub-tasks of a Directive concurrently in a worker pool.
This allows the compute node to concurrently load shapshot and writelog
data concurrently, instead of one keyset/partition/shard at a time.
It introduces a config parameter called `DirectiveWorkerPoolSize`.
* code review cleanup
* Use unique container names in DAX integration tests
We were seeing "container already exists" errors in CI, so just to be
safe, this commit constructs a unique container name for every container
in the DAX integration test run.
Stub in SystemAPI to Queryer (note: will not work if used)
This just makes is so that dax can compile. Actually implementing
system-table functionality for dax will take some planning.
Tlt/dax merge prep (#2282)
* Remove copilot directory
* Remove Dockerfile-datagen-long
* Remove orphaned RegisterNodeRequest
This type is not defined in the dax/mds/http package.
* implement TIMEQUANTUM and TTL in Table.Field type
* Remove the "service" misdirection in queryer/writelogger/snapshotter.
We had originally used an additional layer, er.. package, for a "service".
The main distinction was that the Config differed in that it was
internal, unlike the Config that we need to provide for the top-level
server config (i.e. toml). Having that additional layer just to support
a different Config seemed premature at best. So I'm removing it.
* Remove dax docker containers no longer used in tests
Since we run everything as "featurebase", we don't have multiple
container types anymore.
* Some minor comment updates
* Remove nfpm stuff related to dax
* Fix linter issues
Fix "duplicate" issues raised by sonarcloud.
run docker components of dax integration tests with coverage
trying to get dax integration coverage
add coverate volume mounts throughout dax integration tests
add a lock, tweak dax Makefile, remote flag on query handler
remove some unused code
convert batch tests to use clustertests to get coverage
maybe fix clustertests
more authclustertests fixes, test is failing locally
but also seems to have been silently failing in CI prior to these
changes... let's see if it's still silent
fix some lint to kick CI
just re-running the job wasn't working... strange behavior
remove RetryLogic test and pipe which don't work
RetryLogic test removed due to etcd changes. Seebs thinks we shouldn't
test this here.
Pipe was being ignored since we're no longer using "bash -c" to
execute the command. If we need to generate that output file we'll
either have to reintroduce bash -c and set -o pipefail so that it
actually fails properly, or figure out some other solution.
shooting into the dark...
first cut at bulk node registration
remove unused stuff from batch tests, set coverpkg to ../...
batch registration timeout and fix tests
disable most tests and don't run fb background batch test
debuggin!!!!!!!!!
and then he tried this....
Implement importer (for INSERT INTO) in the Queryer
Prior to this, we we passing a nil value in for the importer to the
planner.NewExecutionPlanner in the Queryer. This meant that INSERT INTO
statements didn't work. Now they should.
It uses the importer that we build for IDK in /idk/mds/importer.go, and
wrapps that with a type that can determine if the provided string
"index" is of the form indexName or TableKey.
turn off debug mode, fix log saving
Run sql3 test definitions in a dax integration test
There are currently 22 tests which are not passing. They are skipped in
the "skips" slice.
WIP, not working, pql queries to tests
Add TableQualifier to PQL query logic in the Queryer
Add more PQL tests to the keyed table
Allow instant node registration if registration-batch-timeout=0
When running dax services in process, we don't want to wait 3s for the
compute node to register; we know it's there because it's in the same
process.
Fixes related to IncludesColumn PQL test.
Tests for ConstRow and FieldValue
cleanup
add UnionRows and Options, better error reporting on bad queries
delete unused schemar client.go, clean up unused in batch test CI
move test timeouts into more reasonable territory
apparently this had already been done, but got merge-stommped at some point
move dax bolt test helpers into dax package
Add computer CheckIn routine (#2296)
* Add computer CheckIn routine
This adds a background routine which sends a "check-in" request to MDS
every <interval>. This is to address the case where the poller has
removed a computer node from the node list (due to a network fault, for
example), but the node is still healthy and becomes available again. In
that case, the node needs to "check-in" to tell MDS it is still there.
MDS will likely send the node a new directive with Method=reset telling
the node to delete all of its data an apply the latest directive.
* Don't send directives to Deregistered (i.e. removed) nodes
We have an issue where we're locking on sendDirective in the
controller, and when the node is unavailable, the send hangs and never
releases the lock. This is a temporary fix for that until we address the
real problem.
Fix .gitlab-ci.yml after rebase
fix some indentation shenanigans
There's an obvious bug, plus another bug that I hit trying to reproduce
the first bug, plus another... it's a long story.
Basically: If you get nothing back from executeDistinctShardBSI on a
Timestamp field, the request for a large enough pool of strings to hold
timestamp conversions of the nothing segfaults because r.Columns() on
a nil row segfaults.
To try to test this better, I added a filter to the executor test that
we use for this case, which got me a different result complaining about
a DistinctTimestamp result not being a SignedRow.
So, there's a couple of issues. One is that, in the case where a filter
is present, if the filter comes up with nothing, we can bail early
and return a result of the SignedRow type, which then breaks the reduce
part of our map/reduce when we try to reduce DistinctTimestamp values
into a SignedRow. To fix this, we make sure that we return the expected
type even in the case where we're bailing early.
A simpler way to see the actual original bug is, rather than having
a filter, just have a shard that has a value in *some other field*
but not in the timestamp field. So we add that to the test, too.
But also, really, since this is a problem that's happened more than
once, I propose that we also just make nil rows allow you to request
their columns and get back nil, so things like this don't bite us as
much. This wouldn't be a sufficient fix for the filter case, and I
still have the short-circuit for the nil row case explicitly in this
particular case because relying on the nil behavior bugs me, but I
think it's safer to allow .Columns on nil rows.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
I used a "paranoia" check to find these, but then realized the check
had a ton of false positives and doing it properly wasn't going to be
straightforward. I'm leaving the paranoia stuff in unless there are
objections, because I've wanted it before and not had it.
I also removed a log line that is very verbose and I don't think helps
anyone.
This commit changes executeDistinct to return either a *Row or a
SignedRow (instead of only being able to return a SignedRow). Distinct
on a set field will return a *Row while an int field will still return
a signed row.
We then add Field and Index fields to the Row object so that we can
determine how to translate the rows IDs to keys (if needed). This adds
a lot of logic around the translation which fixes bugs where Distinct
calls would fail to get translated.
There are, I think, still issues if you were to try to join a keyed
field to a keyed index which wasn't explicitly specified as the
field's foreign index. The IDs in the field wouldn't be using the same
translation as the IDs in the index, so the query might appear to work
but give incorrect results.
- Atomic record contains multiple ImportRequest and
ImportValueRequest, plus ability to Clear individual requests.
- adds http handlers for importing AtomicRecord.
a) All tests green under -race for both PILOSA_TXSRC=roaring and PILOSA_TXSRC=badger.
b) Distinct is merged back into mainline pilosa.
Seebs notes on the Distinct work:
merge Distinct plugin back into main source tree, convert to Tx
We drop all references to the Preemptively Deprecated Don't You Dare
Use This extension interface, and move the one and only extension we had
(Distinct) into the main executor.
Also this fixes an arguable bug, which is that Container.AsBitmap()
would panic on a nil parameter, but it should have returned an empty
bitmap, because a nil *Ccontainer is a valid empty container. This
simplifies logic significantly in Distinct.
Fixes#569#570#571#572#573#584#585
- all tests green on RoaringTx
- RoaringTx on by default
- blueGreenTx testing framework available for A-vs-B comparison
of Tx implementations
- flag -tx added to server command line but not wired to
change NewIndex() selection yet.
- 918 green tests, 14 tests red on BadgerTx.
A full list of the 14 red tests on BadgerTx follows.
Note that these red tests represent not defects in BadgerDB
or BadgerTx but rather failures of the pre-existing pilosa infrastructure to yet
be fully adapted from files to using a transactional storage engine.
As such these are tests that RBF should not be expected to
pass yet either.
Fixing the pilosa infrastructure to allow these tests
to go green under Badger is the next and highest priority
order of business, but RBF can get much testing benefit
from the 918 green tests we do have, and hence we merge
as much as we have today.
The 14 red tests when NewIndex() is set to use
BadgerTx are as follows. Note in particular
that pilosa cluster resizing is not working yet under a
transactional store.
TestCluster_ResizeStates/Multiple_nodes,_with_data
TestImportClearRestart/0MaxOpN10000
TestImportClearRestart/1MaxOpN10000
TestImportClearRestart/2MaxOpN10000
TestImportClearRestart/3MaxOpN10000
TestExecutor_Execute_Existence/Row
TestExecutor_ForeignIndex
TestExecutor_Execute_CountDistinct/Distinct
TestExecutor_Execute_CountDistinct/Count(Distinct)
TestExecutor_Execute_CountDistinct/GroupBy(Distinct)
TestExecutor_BareDistinct
TestExecutor_Execute_TopNDistinct/TopN
TestHolderSyncer_IntField/BasicSync
TestHolderSyncer_IntField/MultiShard
I think when this code was written, I thought "freeze" would be
really cheap. It's not actually that cheap. As a result, freezing
things preemptively when it may be that nothing ever tries to write
to them anyway is possibly disadvantageous, to the tune of being
roughly 20% of a sample profile we were shown. Instead, we don't
mark the components "writable", so if anything wants to write to
them, it'll end up freezing itself new copies of their bitmaps
later. But in practice that probably doesn't happen.
After a few experiments with pkg/plugin, I'm ready to concede that the
people warning me it was unsuitable for production use were in fact
correct.
In the brave new world, the "ext" package is moved to its own module
outside pilosa. This means that importing it doesn't imply any need to
version-check against pilosa; we can just use versioned copies of the
ext package, which can be public because it doesn't contain anything
we need to care about keeping proprietary.
Then we can, conditional on build tags, import modules from a
neighboring repo which contains the actual implementations, and if
they're imported, their init functions register them.
So in some cases, when we do a query, the results of one
part of the query are innately shared-across-nodes; for
instance, a hypothetical Distinct query. More generally,
we allow cross-index queries; calls can have "index=foo"
in them.
This patch lets us handle that without duplicating that
query all over. Before we actually start doing the
separate calls, we run the query once from the coordinating
node, then patch the results in, and send relevant subsets
over to each client, etcetera. Also provides slightly
friendlier (and I hope faster) support for converting
bitmaps to/from sets of rows.
We also add an extension interface, and some fancy stuff
to let us define new calls, which use this. They're sort
of tied together because the first extension I wanted to
implement needed precomputed calls. The extension API
lets us create extensions using `pkg/plugin` (with all its
associated limitations, unfortunately), then query them
at load time for functionality.
This also implies some revamping of the argument
validation for PQL, like verifying that functions exist
and knowing things about their argument types.
So basically this is an overly intrusive patch, and would
be better as separate patches, but they're hard to detangle.
add trivial execution-time profiling
What if you could ?profile=true on a query and get some
numbers back? That'd be really cool.
We already have tracing/spans, but right now, those only generate
any data if you have something set up for them to trace to. Add a
fancy wrapper that lets us generate our own tracing data, and dump
it into the request response, if ?profile=true.
add a sample extension, add missing features to extension interface
Implement a naive probabilistic filter extension as an example of
what an extension looks like. In the process, discover multiple
omissions in the bitmap API. Well, I did *say* it was experimental.
This code represents an attempt at providing reliable tracking
of whether any bitmaps still in use have access to a given block
of mmapped data, allowing us to unmap the data when nothing is using
it anymore.
The basic approach is as follows: Each mmap is associated with
a new object, called a "generation". A generation reflects
a particular instance of a given file being mapped. When a
bitmap is built from an mmapped data source, the bitmap is
given a pointer to the generation as its Source. When bitmap
operations combine containers from other bitmaps, they
produce new bitmaps that are tagged with the combined set of
sources.
When we snapshot a file, or for some other reason wish to remap
it, the corresponding bitmap has all its containers updated to
use the new storage, and the bitmap's source is changed. However,
previously-handed-out containers might still have references to the
old storage. Those containers would be in bitmaps with the old
source.
After a bunch of study of trying to reference-count and track
this, I realized: We don't actually need to do that, because we
already have something suitable for determining whether anything
can reach a given object. It's the garbage collector.
So we set a finalizer on the generation object, which handles
unmapping. There's additional sanity-checks here to confirm things
like "we thought this generation should be expiring", and we
track timestamps. We could also have things check whether a
given bitmap's source was marked as obsolete "a while ago", but
that isn't implemented yet.
There's a debug version of this which tracks finalization, creation,
and ending timestamps, and has a call to provide diagnostics for
this. Identical generation IDs get separated out with random
suffixes in this case -- there's sometimes a second or third
instance of the same name due to a holder closing and reopening,
but this basically only happens in testing.
Note that generations are still used even when there's no mmapping,
but unless debugging is turned on, they shouldn't propagate much --
we don't consider a generation to be the source of a bitmap unless
the bitmap actually mapped things from that generation's mmapped
storage, or debugging is on.
There's a couple of other, possibly more subtle, changes and
bug fixes that got caught by the testing on this:
* If a fragment is partially opened and then opening some later
part fails, we close the earlier parts before returning the
error so we aren't leaving it partially open.
* Several operations on segments which were requesting that a
frozen copy of a bitmap be created are now actually *replacing*
their bitmap with the frozen bitmap, rather than discarding it.
* intersectRunRun, if it decides to create an array or bitmap,
will yield that container instead of discarding it.
And why all of this? Why, so we can actually implement the thing
where when a fragment has a valid roaring bitmap, but the ops log
is corrupt, we can truncate the corrupt part of the ops log and
reopen it. Which I did.
When the generationdebug build tag is in use, every generation
has a finalizer all the time. When it's not, they only get finalizers
when we expect them to be done -- say, when closing a fragment.
This is because finalizers appear to be possibly-expensive.
There's some logical cleanup to openStorage here, dividing part
of its work into applyStorage and importStorage, which have a common
case for handling "there's no data in this file".
Usage:
`IncludesColumn(Intersect(Row(a=1), Row(b=2)), column=10)`
The above query will return a `bool` indicating whether the
intersection of rows a-1 and b-2 contains column 10. Because
a single column is specified, this executes on a single shard
(shard=0 in this example).
add makeRows() tests
register the gRPC server
use api.Index() instead of api.Schema()
support most field types in Inspect() query
currently, there's no support for `time` fields.
those will be dependent upon the output format
and the ability to materialize the timestamp from
the time views.
this commit also changes the response type of the
`Inspect()` query to be a tabular `RowResponse`.
This patch replaces a lot of circumstances in which containers
were being copied with circumstances in which they are shared,
using copy-on-write semantics.
To achieve this, we emulate somewhat the design of go's
native `append` function. Operations on a container may optionally
yield a new container. A container can be marked "frozen",
after which no operation should ever write to it in any way;
that applies both to the container itself and the backing store
it refers to, if any. So for instance, instead of:
c.arrayToBitmap()
we now write:
c = c.arrayToBitmap()
Operations which need to modify a container in any way
need to be able to return a new container, which is a modified
copy of the previous container. This applies to operations
like add/remove, but also to things like unmapping memory-mapped
storage, or changing a container's type.
Bitmaps do not support the same copy-on-write semantics,
currently, but "copying" a bitmap and sharing the containers
instead of duplicating them is *much* cheaper than copying
the containers.
Bitmaps do support a .Freeze method, which currently copies
the previous bitmap, making a new one with the same container
pointers, and freezes the individual containers. Use this
if you need a writeable copy of a bitmap -- the resulting
bitmap can safely have its set of containers modified, and
bitmap operators that would want to modify the containers
will use copy-on-write for that.
The primary motivation of this is to reduce the cost of the
row cache used by fragments. As a secondary issue, the row cache
is no longer updated on writes -- that update was actually a
race condition waiting to happen. Rather, writes to a row
invalidate the cache entry for that row. The row cache is
created by creating a new bitmap, and freezing the relevant
containers from the fragment's storage. In the case where
nothing is being written, the row cache grows to contain
bitmaps containing all those containers, but never copies
any containers. If nothing's being read, the row cache is
never created, and the containers are in general not getting
frozen. The only circumstance where copies have to happen is
when things are read (and thus stored in the row cache) and
later modified. In that case, each read freezes objects, and
the first write to a container after it's been frozen will
create a new copy.
We drop the enterprise/b btree implementation, because we
don't really need it anymore -- we now provide that
implementation by default in the open source product anyway.
Along with this, there's a lot of other changes which
improve support for nil containers, as a cheaper representation
for empty containers. Operations which we know will provide
an empty container can always short-circuit and just yield
a nil *Container. Similarly, operations which would provide
a full container can return a single shared full container
object (which is frozen). The higher-level (non type-specific)
container ops are now using that logic to short-circuit
operations for empty and full containers. (For instance,
difference of anything minus an empty container is the
original thing, union of anything and empty is the original
thing, and so on.)
The Containers interface adds "Update" and "UpdateEvery"
methods, based in part on the "Put" interface provided
by the underlying btree implementation; Update performs
a possible update in-place of a container for a given
key, bypassing the need to replicate the search for that
key in the container. UpdateEvery loops through all the
containers.
Containers do not strictly guarantee that they won't
return nil `*Container` objects. However, the container
iterators won't return those -- empty containers aren't
interesting. Some tests are updated to reflect this.
Some of the container internals, like N(), or the isArray()
and related functions, accept nil container pointers. Some,
like Thaw(), do not. For the array(), bitmap(), and runs()
methods, roaringparanoia enables an explicit panic on a nil
container explaining the problem, but the intent is that those
should never be called unless you already know you have the
right kind of container, so by default they don't perform
the extra checks. In most cases, this is already covered
because a nil container is empty, and there's no operation
we can perform that requires us to inspect the contents of
an empty container. This is passing a fair amount of testing,
but the testing may not be comprehensive enough.
The overall impact of this is pretty trivial performance-wise.
In our default roaring/ benchmarks, a few things get a few
percent faster, or slower. The advantage is that, with
read-heavy workloads, the row cache no longer eats up incredible
amounts of memory.
For a smallish test case, pilosa's memory usage (RES in top) after
startup was ~2.5GB. Without this patch, simply reading every
row a few times got memory usage to about 9GB, which seemed
reasonably stable. With this patch, memory usage went to about
3GB. This will be less noticeable in mixed read/write loads,
but it should be consistently significantly lower.
In addition to dropping things from the rowCache on modifications,
we also stopped performing a full count on a modified row when
not using a cache of a kind that would use that count, and don't
repopulate the rowCache regardless. We don't want every write
to imply a corresponding read after it.
There's a lot of room for possible future optimizations in
terms of things like in-place operations, and some of the
row/rowSegment code is a little suspicious to me, but I don't
think it should be *worse* in any cases.
This commit implements BSI with variable bit depth using a
sign magnitudeto indicate whether a value is positive or negative.
This also rearranges the existence bit to be the first bit instead
of the last bit.
