First, it is possible for us to end up allocating *or freeing* pages during
a modification of the free list, in a way such that the change to the free list
means that when we finish the modification which caused the allocate or free,
we've overwritten the inner change.
Second, when deallocating trees, we don't actually deallocate the branch nodes
themselves.
The former causes potentially severe data corruption. The latter causes us
to gradually leak pages in a way that we don't notice because we only run those
tests during the RBF tests.
The fix for this is surprisingly intricate, because of the counterintuitive
fact that *allocating* a page means *removing* things from the free list
(and thus potentially deallocating free list pages), while *freeing* a page
means *adding* things to the free list (and thus potentially needing to
allocate pages for the free list).
While modifying the free list, any allocations we need always just come from
the end of the file; we don't try to reuse free pages. If a page becomes
*deallocated* by a free list modification, we don't annotate it in the free
list at the instant that it happens; we stash that information until the
current modification of the free list happens, then iterate through any
such pages.
I am pretty sure there's virtually never more than one, and I don't actually
know that I can create a case wherein we'd end up with the nested case
firing, wherein removing a page from the free list causes us to remove another
page, but I think if the free list got large and cluttered and needed
rebalancing or something it could maybe happen.
try to fix yml syntax
same
same
same
same2
same3
same4
same5
try with shell runner instead of dind
remove lattice from dockerfile
change path to bin
runs after linux arm64 build
change dockerfile path
same
same
add dir
better test coverage
This is targeted at reducing startup times, especially on OSX where
the fsync calls seem to be taking an egregiously long time. I got one
index to go from ~1min to open to ~1sec. This looks safe to me, but
will get opinions from RBF experts.
These commits are hard to disentagle, and doing them separately means
re-modifying the same chunks of code several times before removing it,
and similar things.
Basically:
(1) Drop the bolt backend storage.
(2) Drop the blue-green wrapper that compares two backends.
(3) Drop unused or barely-used Tx API components from all the
remaining backends.
(4) Minor related cleanup to simplify things related to these.
The boltdb backend existed only to verify RBF. The blue-green wrapper
was mostly used to verify RBF, but in practice we had to do a lot
of working around that, and it introduced a lot of special cases.
Types removed:
IteratorFinder: Used only to implement the roaring iterator
on top of boltdb, and to complicate the way it worked in roaring.
Reverted the complications. Also unexport NewSliceContainers
which is used only for that outside of roaring's internals.
PortMapper from cluster_internal_test.go: Used only for a test
we removed early this year. Never used for anything else.
RawRoaringData: Totally unused.
TxStore: Totally unused.
Functions removed from Tx API, and sometimes corresponding
members were removed from structs:
* Dump: debugging code, I don't think I found any actually reachable
paths to it.
* Group: only used for debugging TxGroup stuff
* IncrementOpN: only used by fragment, fragment can increment its
own opN.
* Options: unused?
* Pointer: debugging only
* Readonly: used only to decide how to handle Tx in a TxGrp,
but we never add a non-readonly Tx to a TxGrp. Removed also all
the corresponding write-aware stuff.
* RoaringBitmapReader: Used exactly once, can just be a bm.WriteTo.
* Sn (and OpenSnList): Unused
* UnionInPlace: unused and conceptually-invalid; it didn't write
to storage and shouldn't have, and was just "create a bitmap
then call union-in-place", which we can do directly.
* UseRowCache: just checked storage.UseRowCache.
Other things removed:
The SetRequiredForAtomicWriteTx and ClearRequiredForAtomicWriteTx
functions go away, since nothing now seems to be using them? Same
for holder_internal_test's `testHasBit` and `testMustNotHaveBit`,
which were unused.
The DBPerShard "DeleteDBPath" and "HasData" functions and related
parts were mostly unused; took out the parts that were never
actually being reached.
Changed the API of one function to simplify special cases and
remove things:
* ImportRoaringBits had a special "data" argument which gave it
subtly different semantics for RBF and roaring (for roaring, it
could produce a roaring bitmap *with ops log*), didn't seem to
be adding much. Removed corresponding "readStorageFromArchive"
which is not otherwise used.
Also took out various debugging/dumping functions that were unused
and may have bitrotted.
Dropped a test from txfactory_internal_test, and the "pjobs"
code, because those two were the only things that needed Barrier
and thus idem, which lets us drop two more dependencies. We already
have errgroup for grouping things which want to terminate as
soon as one of them errors, approximately. To do better we'd have
to have context-threading, really.
Unbroke the WriteFragment test for non-roaring tests and made it
not roaring-only.
In test runs, we open, and close, *huge* numbers of databases. Even
the single fsync on close for these ends up being expensive on some
hosts. *cough* Apple. At least in theory, writes delivered to the
disk are just as written whether or not you've hit fsync, as long
as the machine doesn't power off before getting to them. In the
circumstances where we disable fsync, that's fine.
Since we already have an fsync function for "fsync if it's
not disabled", use that.
This commit fixes an issue where the root record cache is only
built when a write transaction successfully commits. However, if
no write transactions are occurring then the the cache is never
built and saved so it is recomputed on every read tx.
The remake container logic (used to avoid allocating extra containers while
applying filters) relied on roaring recomputing N, which it did for bitmaps
but didn't do for runs. Fix this both ways; it would now do that for runs,
but also we add "with explicit N" variants and use those since we have a
correct count already, and don't need it. This means fewer popcounts on
bitmaps, and working at all on runs.
Due to lack of synchronization, this test would sometimes close the DB before terminating a transaction:
=== RUN TestTx_CommitRollback/SingleWriter
tx_test.go:132: db still has 1 active transactions; must closed before closing db
The test now waits for the goroutines to terminate before closing the DB.
In nearly all cases, we can just switch ioutil.TempDir->testhook.TempDir
and similarly for TempFile. There's one case where we can't because we
need files to be removed before tests are over.
Also in the process give identifiable names to a lot of temporary files
and make sure they're being cleaned up, and don't use "/tmp/foo" as a
file name in a test that could be running in more than one test process
at once. :)
Long story short: Once we create a server and start it, we can't start
it again. We can't close it and restart it, and we can't just start
it without closing it.
Unfortunately, if the server's config needs to change, we have a Problem
here.
This ultimately means that the retry logic for GetListeners can't actually
retry successfully; if we fail on the first attempt, we necessarily fail
on any later attempts also, and if we try to fix that, we get panics.
But!
We don't actually NEED to retry. We just need to ensure that we can
open a :0 port, extract the actual port number, and use that in places
where the port number mattered, without having to rebind it.
The only actual place we needed to rebind things was opening gRPC
servers, so we introduce a gRPC Listener that can be used instead of
trying to bind to a specified port.
In a bunch of other cases where we had similar logic to try to allocate
and then use a port, we can switch to just using a provided listener.
For instance, net/http has `Serve(net.Listener, handler)`, not just
ListenAndServe(addr, handler).
This should eliminate the weird CI failures from eaddrinuse.
NOT fixed: server/cluster_test.go/TestClusterResize_AddNode isn't working
right now. The new node isn't actually being added to the existing cluster.
I attempted this but was outsmarted by it, and I think fixing the
rest of this is worth it as a separate thing.
We need to be able to count bits in BitmapPtr containers. This only
comes up if you have a non-container-aligned range count, which we
never do in real production yet, but the API allows it so it should
work. In order to do this, we need to provide the tx to countRange
so it can grab pages as needed. Arguably, we should be able to avoid
actually creating/copying that page since we're only using it
internally, never returning it, but this is a pretty rare case
and probably not performance-critical.
port mapper gives out ports from 63000-65000 for the tests
fix another race
http test uses port.MustGetPort
rbf: remove :0 port request
ocd happy
test fix for grpc listener address already in use
test/disco allocates BindGRPC port from the port mapper
dump stack on each GetPort
verify each port is usable right away
server/config.go has Config.Validate() now
panic if gossip port is 0. validate server.Config
fix another gossip port 0
builds
quiet, don't dump stack on each port alloc
builds
happy linter
even gossip fallback should not be zero but rather use the port mapper
- Previously, on timequantum schemas, we would
create and open a view for the cartesian
product of every possible view and shard.
- This caused us to be very slow on re-open,
and to use lots of memory for views that
held nothing.
- This change makes startup faster, memory
use much lower, and should speed migration.
CountRange for RBF had a subtle bug which wasn't noticed, so, let's
have some CountRange testing and also a benchmark.
We also fix a couple of subtle bugs caught in the process of developing
and testing this.
SliceContainers will allow nil containers, but doesn't return them when
iterating because there's various things that can panic if called on a nil
container. Since countEmptyContainers() has to traverse the whole bitmap
anyway, it doesn't matter which it counts, so we replace it with
countNonEmptyContainers(), and adjust test cases accordingly. This fixes
an issue where if roaring is smart enough to insert a nil container
into a SliceContainers, trying to write it to a file produces an invalid
bitmap with offsets off by 16 and one container fewer than its header predicts.
RBF: don't try to count 0 bits in a container
If we're to the "last container", and we'd be counting all the bits less than
zero, we can skip that. This avoids hitting a bug, which is that c.countRange
doesn't handle BitmapPtr.
This reduces noticably the cost of reading leaf cells, by passing
a single pointer down the stack instead of the entire data structure
up the stack. It's only a few percent overall, but it's noticeable.
This gives RBF an ApplyFilter that can run without instantiating containers
when the filter it's using doesn't need them instantiated. We can also seek
ahead in cases where we know the next key we care about is not just the next
key numerically.
- use short_txkey for rbf
- short_txkey breaks a bunch of bolt_test.go, so leave it on (long) txkey for now.
- remove SliceOfShards method from Tx interface
