We switch everything to use QueryContext/QueryRead/etc instead
of Qcx/Tx.
We drop the short_txkey subpackage (it's now handled by either
keys or querycontext).
We drop all the dbshard stuff, and all the tx/txfactory stuff.
We remove all the things that related to the old "Block" concept,
which was mostly used by the anti-entropy code, but had one
fragmentary usage left in the ImportRoaringOverwrite case of
ImportRoaring. That's replaced by using a rewriter that deletes
all bits (not just bits in specific columns) from an existing
thing, but writes in new bits. Actually we could probably do that
better with a custom "eradicate-rewriter" that doesn't try to
be clever, and just eliminates things.
This includes a number of minor bug fixes that were
exposed by getting the testing to work. For example:
* When checking whether an operation "requires write", we
now consider a Delete a kind of a Write, because it is.
* Several tests were relying on the fact that writes through
Qcx were being committed whether or not the Qcx was ever
told to finish. With QueryContext, you actually have to
reach a Commit() or the writes don't happen (except for
special cases in Delete).
* Replaced a lot of panics with t.Fatalf in tests.
There's also some minor staticcheck fixes, like deleting the
unused "db" member of a boltdb transaction wrapper.
* FB-1251: Add ability to sort Extract queries by some field
There is a sort call which takes a row call and the field, and
based on the field type, the corresponding rows are read. both key and
value are stored in RowKV{}. The value is stored since its required to
merge data from shards. values are sorted in each shard and these sorted
listes are merged in the reduce.
sort-desc flag is sent to comparator to decide the sorting order. ok
flag is added to the compare function to track any error in the
sort.Slice anonymous function
Sorting over set field was removed, since there would be multiple values
for each ids and there would be no right sorting order there.
This is fairly experimental, but basically, we make a fragment-level
op which, given a []uint64, can produce a union of all those rows in
the fragment, with a single scan through the fragment and not needing
a ton of additional space to reify all the rows at once.
Now with the Repair calls happening in the Results assembly rather than
on the intermediate data.
* ugly first cut at supportings Rows(in=[...])
need tests, better handling of various combinations of arguments and
error cases
* explicitly error when other arguments passed with 'in' to Rows
* first cut at supporting Rows(in=[...])
'in' is explicitly not supported with any other arguments (except the
field of course), and will error. It works both as a standalone Rows
call and in GroupBy.
* bitmapfilter require ordered rowids
* remove log message
Co-authored-by: Todd Gruben <todd@molecula.com>
Fixed the issue where deselecting variables didn't work on quries like
groupby and extract.
The empty variables list returns All() when the variables are empty.
queries can become arbitrarily long when variables are used.
This is especially the case when a variable is defined as
'select distinct field from table'
and a user wants to use all the values in a Row call
(which is effectively disabling any condition on the field).
This change allows users to select no values for a variable
associated with a Row call to disable the condition. If that
variable is the only condition (query expands to nothing)
then it interprets it as an All call.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
This changes Count(Precall()) operations to execute the precall directly inside of the count operation, bypassing the transformation to a Precomputed() call.
Eliminating the Precomputed() step causes Count(Distinct()) to work properly on negative integers.
Back out support for sorting on fields (only count and aggregate
supported for now).
Fix bug where default return of "true" caused sort to be unstable. (If
they are equal, Less should return false)
Fix bug where limit was being applied before sorting.
Fix bug where offset was not actually allowed to be an argument to
GroupBy (weird! guess we weren't testing that very well)
Apply "having" after calculating Count(Distinct) aggregate so that
having can apply to that.
Switch to stable sort to make testing easier.
This replaces the former TopK BSI building algorithm, as the row cache was too expensive.
Additionally, BSI addition has been optimized with specialized adders inside of roaring.
Use "" strings for fixed string names. In startCall(), look up the
lowercase conversion of a call name in a table mapping all-lowercase
representations to canonical case, so we don't have to chase down
everyplace in the rest of the code base that assumes "Row" is
capitalized exactly like that.
PQL always produces decimals, which have effectively-arbitrary range,
but can convert them to floats when required; the executor then requests
this conversion in the handful of cases (SetRowAttrs and SetColumnAttrs)
where it wants floats rather than decimals.
Not yet fixed: The "Range" call may also be wrong now. It was specifying
an "fvalue" but is now effectively getting what used to be called a
"dvalue". However, so far as I can tell, that didn't work before either.
