* Remove Node from data model; standardize on Worker
This commit does a lot of things, but in general it attempts to simplify
the data model by getting rid of the Node and NodeRole models. Instead,
these will use the Worker model, which itself has individual boolean
fields for role types.
Get rid of roleType in some FreeWorker methods
rename NodeService to WorkerRegistry
simplify the freeworker interface
fix the tests
* Remove DeleteWorker method from workerJobService
* First pass at RetryWithTx
* Refactor RetryWithTx to take a writable bool (instead of reads, writes)
* Implment DirectiveMethodDiff
This commit adds support for a Directive to contain only the diffs (as
opposed to the full Directive).
* Update controller tests to allow for DirectiveMethodDiff (over Full)
* Update RetryWithTx to retry on duplicate key constraint.
If two concurrent processes call IngestShard() for the same shard, both
were trying to insert the same job into the jobs table. That resulted in
a duplicate key error from the database. We want to include that error
in the list of errors for which RetryWithTx should retry.
* Remove unused method: Directive.TranslatePartitions()
* Replace query in a loop with a single query
We had a query which was looking to see if a job already existed. That
query was inside a loop, and could potentially generate 256 queries (for
example). This commit replaces that logic so that we use a single query
wiht an `IN ()` clause.
* Convert to directive version-by-address
This commit uses a separate directive version per address. It moves the
version get/increment back inside the buildDirective method so that if
two concurrent processes are building a directive for the same address,
one of them will get rolled back trying to commit the version update.
* Migration for directive version by address
* Add a comment about DirectiveVersion lock/unlock logic
* Remove AddLastWins
* fix linter
* handle error in walkdir
* fix test failures from removing AddLastWins
* serverless sqldb use same env for test config as normal
* rip boltdb implementation of controller backend out
it was replaced by postgres and no longer works properly.
This involved migrating a number of tests which only worked with
boltdb, which exposed several ways in which the postgres
implementation had slightly different behavior from the bolt
one:
1. ordering of results in some cases, and
2. (more importantly) erroring when a record to delete was not
found. The bolt implementation silently ignored it when things to
delete weren't found, so we make some changes to match that behavior.
Also stopped propagating CreatedAt and UpdatedAt from DB tables into
dax types. These were breaking existing tests. Perhaps it would be
better to actually use them, but for now they will only exist at the
DB level.
This change set also moves the insertion of the directive_versions
record out of migrations and into the startup/connection code. Having
this in the migrations was a bit ugly because you couldn't just
truncate all the tables and have everything work from
scratch. Inserting it during startup is fairly innocuous, and will
just continue on if it already exists.
* update directive_version test
I changed the initial value to 0 so that the first version that gets
sent out is 1
Switch Serverless from using BoltDB to Postgres as metadata store.
Previously, the controller stored all metadata to BoltDB. This implements SQLDB (currently Postgres flavored) as the backing store for metadata. This will allow us to have multiple instances of the controller running for HA, and to easily inspect and repair the contents of the metadata store.
Unfortunately, it was not straightforward to keep the BoltDB implementation working alongside the SQL one, so it will be removed in a later patch. Once that's done, the SQL implementation should allow for a number of simplifications of the schemar and balancer interfaces.
Database migration is built directly into the application by embedding the migration files and logic from the `soda` command line tool. When connecting to the RDBMS, the app will always attempt to create the necessary database and apply any outstanding migrations.
Integration tests truncate all tables upon start, but *not* at the end, so the state of the database can be inspected after integration tests.
Had to refactor some of the controller's background tasks to make sure they get properly shut down on controller exit.
