This adopts the task pool functionality to let us spawn new worker
threads when worker threads are blocked. The underlying reason for
this is the same as the reason for the previous worker-pool-growing
strategy; while our design persistently has at least one thing which
can proceed, it can be the case that there are N things blocked,
where N is the size of our worker pool. Blocked workers shouldn't
count against our desired number of workers.
Originally, the intent was to thread this into RBF, and provide
backpressure from RBF on the pool when blocking on writes. Unfortunately,
that's not good enough, because while a write is blocked, the Qcx
calling it is *also* holding the Qcx's mutex, which means that any other
NewTx on that Qcx will *also* block. So we need to block for the
entire time of the NewTx.
Removing the existing worker spawning code resulted in a subtle
and maybe-harmless change; prior to this, each invocation of `mapperLocal`
would hold a lock, which meant that all the tasks for a given local mapper
would be put in the queue *sequentially*, ensuring that they'd all be
picked up by workers before things from later workers.
With the new pushback, that's not, strictly, necessary. Also, if you
disable it, you can end up with 300,000 goroutines at once, most of them
blocked.
A smallish run does, in fact, eventually complete anyway -- it will
indeed keep making workers until everything gets one. However, while
it's *correct*, it's also noticably *slower*. The same test workload
goes from around 33 seconds to a bit over 40 seconds when that lock
isn't present. (But that's with an extremely small WAL write cap
introduced to make the previous deadlock possible.)
With large numbers of shards, the practical impact is that you can
have quite a lot of things in process, with hundreds of goroutines
each, all blocked waiting for one writer. If we force them to all be
processed at the same time, all the reads that are connected to
each other are much more likely to get all processed at once, before
something new comes along.
In short, that lock isn't strictly necessary but it seems to help
noticably with performance and reduce simultaneous goroutines
significantly.
This implements a task pool which can handle backpressure; the
idea is, you have a target number of workers, but when a worker
blocks, you can tell it that it's blocking, and it can spawn
another worker in the mean time. This reduces the bounding provided
by the worker pool, and can significantly overshoot the intended size
of the pool in some cases, but it provides quick scaling up when
part of a workload gets blocked.
There's also a simulator attached to it. The simulator's job is
to act similarly to the executor's worker pool working on RBF
databases, including the weird semantics of writes and reads;
specifically, that reads aren't blocked by writes, but a write
can't terminate until every read that started before it has exited.
(This is an oversimplification; actually, writes can complete,
but they still hold the write lock until any WAL merge completes,
and the WAL merge can't complete until old reads are done.)
The simulator is significantly more complicated than the pool.
- For server side, used an instrumented binary with a test that wraps around the main entrypoint for featurebase
- Every time, the binary is called, a new coverage file is generated.
- For the client side, used the standard -coverprofile flag for go test to generate code coverage
- For backup test that's expected to fail, needed to call Run call in backup.go directly. The code coverage is not written to disk for an instrumented binary if there is an error.
We have pipelines that get to the gauntlet stage then get failed because
the ASG scales-in before the gauntlet stage finishes. (4/6 of the last
gauntlet failures were from this failure.)
There are a few ways to fix this, but my proposal is to turn on scale-in
protection to stop scaling in the instance running the gauntlet job
(scale in other instances instead), then turn off the scale-in
protection after the gauntlet test is run.
