If the min/max provided are already on the boundary of int64,
then we don't want to operate on them and cause overflow.
there are still overflow scenarios where a user provides a
min/max which is not on the boundary, but overflow once the
scale is applied. This does not address those cases, but at
least it addresses the default case (where a min/max is not
provided)
When the interval is a proper superset of the range with start equal to
interval start, the range must be considered a superset or it will be
completly ignored (since it neither a subset nor it overlaps)
Co-authored-by: Pierre Fersing <pierre.fersing@bleemeo.com>
It turns out that it's not very useful to keep the sign
value as a separate argument in the pql.Decimal struct.
This commit incorporates it into Value, and makes Value
an `int64` (for some bone-headed reason I had made it a
`uint32` before which is just dumb).
This commit introduces a new type: pql.Decimal
We use that instead of float64 in order to ensure
that the string representation is consistent.
One unfortunate discovery during implementation is
that the RowAttrs and ColAttrs support floats, and
the PEG file was treating them as such. So I had
to split the PEG definitions into float-specific
items and decimal-specific items.
This PR adds support for anti-entropy syncing for integer
and decimal fields. It differs from the logic for other
field types in that it does not rely on a consensus to determine
what the value should be; instead, it considers the correct
values to be those of the primary replica. From there, data
is pushed to all non-primary replicas.
Two changes:
1. Don't write batch/roaring adds or removes when N is 0, because
a write of no bits is not a meaningful write.
2. When unmarshalling roaring things, if a roaring bitmap didn't
change many bits, treat it as having changed at least 1 bit per 8 bytes,
so an 8KB hunk of roaring data counts as 1K changes, which will
nudge us towards snapshotting. This should keep us from having
Large Files show up so much.
This was particularly noticeable on the existence field, which
tends to a steady state of "completely full" very quickly in a lot
of cases.
This PR adds a translationSyncer interface; I tried to include
comments in the code explaining what's going on. This is taken
from those comments:
translationSyncer provides an interface allowing a function
to notify the server that an action has occurred which requires
the translation sync process to be reset. In general, this
includes anything which modifies schema (add/remove index, etc),
or anything that changes the cluster topology (add/remove node).
I originally considered leveraging the broadcaster since that was
already in place and provides similar event messages, but the
broadcaster is really meant for notifiying other nodes, while
this is more akin to an internal message bus. In fact, I think
a future iteration on this may be to make it more generic so
it can act as an internal message bus where one of the messages
being published is "translationSyncReset".
This PR forces the non-coordinator nodes to reset their translation
sync (and therefore their own cosideration of read-only partitions)
any time they receive a `ClusterStatus` message. So basically, as the
cluster grows during the startup process, each node will reset their
translation sync.
This is NOT a good solution log term, but it should address the
immediate problem.
Things to note:
- the coordinator sync isn't getting reset, but that's ok, because the
immediate problem is a partition marked as read-only when it shouldn't
be; i.e. it's ok to have the inverse (a partition not marked as
read-only when it should be) because that partition won't receive
requests anyway.
- the last node to start is already correct and doesn't really need to
reset its sync.
- there are many other scenarios not covered by this fix.
Based on this theory:
```
i have another theory that i’m going to try to test.
this one would only apply in the case where a multi-node cluster is restarted with an existing, keyed index.
- start node0: it thinks it’s responsible for all partitions (nothing is read-only)
- start node1: it thinks it’s responsible for ~1/2 of the partitions and marks the other 1/2 as read-only
- start node2: it thinks it’s responsible for ~1/3 of the partitions and marks the other 2/3 as read-only
now if node0 is the coordinator receiving all translation requests, that still might not explain what’s happening, because in that case it would just do all the translating. i think. but either way, i should make sure that scenario is not happening, but i think it may be.
actually, that might explain it, because what would happen when the coordinator received a translation request, is that it would handle the 1/3 that it owned (now that the cluster is 3 nodes), and it would send the other 2/3 out to the other 2 nodes. but where it sent the requests wouldn’t line up with what the nodes thought they were responsible for based on the restart order
in this example, node 1 would receive requests for the wrong partitions
```
In the old unmarshal code, the decision to mark a thing as mapped (always
yes) happens separately from setting the mapping. What if this could ever
somehow possibly go wrong? Let's sanity-check that to be extra careful.
