This ensures that we can't overflow when adding `pql.Decimal`s together. The
only place we can possibly overflow is when converting pql.Decimal to an Int64,
but that is a risk we have to take. Also, the only place we do this is in our
ToRowser. We could maybe change that to strings, so the presentation of data
doesn't indicate an overflow, but that is a later decision to make. It will
also involve fixing the generate-proto-grpc make command, because that's broken
rn.
* mutex clear on nil support with test
* Update client/batch.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
it's not something that we can send to the server (it calculates base
off of min and max), and we don't need it in the json string in order
to use it client side when building import requests.
if the same ID is added multiple times with different values, only the
last value should get set. Without this change, if the multiple
records weren't immediately next to each other, all the bits from all
the values would get set.
the ranked cache must be fully rebuilt as the BitmapRewriter doesn't
have an easy way to track which rows had how many bits changed which
we would need to update the cache.
We also needed to add a Clear method to topn cache to completely
remove old values before the rebuild as otherwise they'd sit there and
pollute the cache after it was rebuilt.
This also includes fixing a strange idiosyncrasy where the _exists
field was a set field, but didn't have its type explicitly set. This
was causing it to have a ranked cache even though that option was
turned off. Hoping this doesn't have any weird follow-on effects... or
if it does the tests catch them.
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.
It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
swallowing this error was a mistake... caller would not know that
ingest failed and might incorrectly report success
upstream (e.g. committing offsets to Kafka)
out of range. Previously internal server error was returned.
This is to allow for ingest to continue while logging bad values
instead of stopping ingest as we do when there is a server error.
* Change Ttl to TTL
Following go convention, acronyms should have a consistent case.
See
[Initialisms](https://github.com/golang/go/wiki/CodeReviewComments#initialisms)
This commit changes some public-facing methods, so any code importing
this package and using these methods will need to be updated.
* rewrite Ttl -> TTL
Co-authored-by: reesporte <reesedporter@gmail.com>
If non-primary host fails to process a request, retry on primary node.
conditions when we should not do this:
- no error
- we've aleady tried the primary
- we're making a status request to get the primary node...this
could lead to lock contention if we allow it to happen as we are
making an http request within an on going http request to discover
the primary node.
This also deletes the RemoveHost method and the associated test b/c
it is not used anywhere anymore and updates the returned error type.
- Add auth-token for featurebase import, backup and restore
- Add auth-token to http request
- Create a cluster tests with auth enabled
- Add test for import with auth enabled
the shardwidth22 tests were broken client side, but we didn't realize
this because we weren't running the client side tests since moving the
client code into the main FB repo until recently (woops), and more
recently, we'd stopped running the shardwidth22 tests in the move to
Gitlab, so when we re-enabled them we finally noticed that they were
broken in the client.
All this change does is takes the shardWidth value from the core
featurebase package instead of using a hardcoded value in the client package.
also use a single cluster with each test creating a different index
rather than each test creating a whole new cluster.
runtime went from 38s to 30s in my informal tests
discovered that client tests weren't running due to integration build
tag. Fixed the file I needed to get through SonarCloud and documented
rest of what needs to be done in FB-1152 https://molecula.atlassian.net/browse/FB-1152
fixes pathological case where imports with randomly ordered IDs which
spanned multiple shards and included ints or mutex fields could be
incredibly slow due to making 1000s of requests.
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
Time fields were not listed as a type which could accept key translation, causing the translation code to fail.
This also adds tests for keyed time and mutex fields.
This migrates existing code from the old TranslateKey(s) endpoints to the newer CreateKeys and FindKeys endpoints.
The CreateKeys and FindKeys endpoints were created previously as the TranslateKeys endpoint had no way to behave sanely when the looked-up key did not exist (the parallel-arrays representation did not have a good way to represent a missing key).
This change also removes the old TranslateKey(s) functions from the translation stores.
It leaves a wrapper emulating the TranslateKey(s) endpoints so that old idk still works for now.
