We switch everything to use QueryContext/QueryRead/etc instead
of Qcx/Tx.
We drop the short_txkey subpackage (it's now handled by either
keys or querycontext).
We drop all the dbshard stuff, and all the tx/txfactory stuff.
We remove all the things that related to the old "Block" concept,
which was mostly used by the anti-entropy code, but had one
fragmentary usage left in the ImportRoaringOverwrite case of
ImportRoaring. That's replaced by using a rewriter that deletes
all bits (not just bits in specific columns) from an existing
thing, but writes in new bits. Actually we could probably do that
better with a custom "eradicate-rewriter" that doesn't try to
be clever, and just eliminates things.
This includes a number of minor bug fixes that were
exposed by getting the testing to work. For example:
* When checking whether an operation "requires write", we
now consider a Delete a kind of a Write, because it is.
* Several tests were relying on the fact that writes through
Qcx were being committed whether or not the Qcx was ever
told to finish. With QueryContext, you actually have to
reach a Commit() or the writes don't happen (except for
special cases in Delete).
* Replaced a lot of panics with t.Fatalf in tests.
There's also some minor staticcheck fixes, like deleting the
unused "db" member of a boltdb transaction wrapper.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
In nearly all cases, we can just switch ioutil.TempDir->testhook.TempDir
and similarly for TempFile. There's one case where we can't because we
need files to be removed before tests are over.
Also in the process give identifiable names to a lot of temporary files
and make sure they're being cleaned up, and don't use "/tmp/foo" as a
file name in a test that could be running in more than one test process
at once. :)
- Previously, on timequantum schemas, we would
create and open a view for the cartesian
product of every possible view and shard.
- This caused us to be very slow on re-open,
and to use lots of memory for views that
held nothing.
- This change makes startup faster, memory
use much lower, and should speed migration.
- use short_txkey for rbf
- short_txkey breaks a bunch of bolt_test.go, so leave it on (long) txkey for now.
- remove SliceOfShards method from Tx interface
- the sync.Pool default uses little memory under CI.
- arena approach provides ability to control the maximum memory
used by rbf Cursors.
- cursor caching is adjustable with --rbf-cursor-cache
currently 0 by default (meaning use sync.Pool), and
larger than 0 meaning use an arena of this size.
With the arena, 20 or less is needed to pass CI.
- rbf test suite runs ~ 4x faster
- kitchen sink ingest test runs 16% faster.
- report TotalAlloc in CALLSTATs
fixes#1105
- fix a bug in computing leafCell.BitN in a run after a bit Remove
- shrink bitmaps on remove
- util_test.go has Cursor.DebugSlowCheckAllPages to verify;
used by cursor_test.go
