Commit graph

23 commits

Author SHA1 Message Date
tgruben
04b19c066a
[FB-1776] support for marshaling ExtractedIDMatrixSorted (#2319)
support for marshaling ExtractedIDMatrixSorted
2022-11-30 13:47:26 -06:00
pokeeffe-molecula
158cc669d9
Tighten up ORDER BY (fb 507) (#2318)
* tighten up checks for order by expressions fixed ordering by expressions

* added testing to cover order by cases

* Add DecimalAgg member to proto GroupCount definition

In DAX, where we have split the orchestrator from the executor, and the
orchestrator can run on a different host, there are cases where
`GroupCount`s can travel over the wire via the Internal Client. In these
cases, when the group count contains a decimal aggregate, we need to
send that value as the appropriate type.

* fixed missing cases in order by and case block eval

Co-authored-by: Travis Turner <travis@molecula.com>
2022-11-30 11:32:37 -06:00
tgruben
2f1beaf119
Dataframe (#2241)
* Dataframe
2022-11-21 17:38:48 -06:00
pokeeffe-molecula
fb40cdc2cb
fb-1729 Enriched Table Metadata (#2255)
enriched metadata for tables

added support for the concept of a table and field owners in metadata; mechanism to derive owner from http request metadata; metadata for table description
2022-10-26 11:23:40 -05:00
Seebs
c3b032d5cb drop ioutil
The ioutil package is deprecated, with all of its functions having
moved into os or io. Do the replacements so we stop having this
impending.
2022-09-23 16:56:27 -05:00
reesporte
f3f1ea41b7 indicate sign with a bool 2022-06-17 14:58:30 -05:00
reesporte
51fb34e657 make Decimal proto definition backwards compatible 2022-06-17 14:58:30 -05:00
reesporte
0d91e2991d Revert "FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)"
This reverts commit acbcec54c8.
2022-06-17 14:58:30 -05:00
rachithrr
acbcec54c8
FB-1491: Revert pql.Decimal commit for the 4.8.5 release (#2117)
* Revert "make pql.Decimal.Value a private big.Int field"

This reverts commit eab6174388.

* Revert "pql.Decimal for DecimalVal in ValCount&GroupCount"

This reverts commit a0c9eec410.

* Revert "Add AddDecimal support to pql"

This reverts commit 50787fd37a.
2022-06-15 09:15:37 -07:00
reesporte
eab6174388 make pql.Decimal.Value a private big.Int field
This ensures that we can't overflow when adding `pql.Decimal`s together. The
only place we can possibly overflow is when converting pql.Decimal to an Int64,
but that is a risk we have to take. Also, the only place we do this is in our
ToRowser. We could maybe change that to strings, so the presentation of data
doesn't indicate an overflow, but that is a later decision to make. It will
also involve fixing the generate-proto-grpc make command, because that's broken
rn.
2022-06-06 16:44:01 -05:00
Matthew Jaffee
772496b440 "all bitmap" multi-field, single-shard ingest
This adds a shard-based import endpoint which takes bitmap data for
all field types and imports data for the whole shard transactionally.

It uses the BitmapRewriter interface to try to intelligently allow for
setting and clearing bits simultaneously without multiple writes which
is especially helpful when ingesting into int-like fields, but also
allows clear-and-then-set behavior for set fields.
2022-05-27 11:25:17 -05:00
Matthew Jaffee
0d50bd2890 implement ability to update TTL on time fields 2022-05-04 11:04:38 -05:00
Travis Turner
7ccc845aac
Change Ttl to TTL (#2038)
* Change Ttl to TTL

Following go convention, acronyms should have a consistent case.
See
[Initialisms](https://github.com/golang/go/wiki/CodeReviewComments#initialisms)

This commit changes some public-facing methods, so any code importing
this package and using these methods will need to be updated.

* rewrite Ttl -> TTL

Co-authored-by: reesporte <reesedporter@gmail.com>
2022-04-27 11:20:39 -05:00
Hoang Pham
e16171cb4b FB-1188 - Added TTL field option 2022-02-25 11:39:31 -06:00
reesporte
48aef0c8a4 add copyright notice back in
```bash
for file in `cat diffys`; do
   printf '%s\n%s\n' "// Copyright 2021 Molecula Corp. All rights reserved." "$(cat $file)" >$file;
done
```
2021-12-10 11:01:04 -06:00
reesporte
4c53f86e82 removed license from each go file
i used this script, a little clunky but it got the job done

```bash
for file in `find . -type f -print | grep '\.go'`; do
    sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
    result=`cat $file.tmp`
    if [[ result != "" ]]; then
        gofmt $file.tmp &> /dev/null;
        if [[ $? == 0 ]]; then
            mv $file.tmp $file && gofmt -w $file;
        else
            rm $file.tmp;
        fi
    else
        rm $file.tmp;
    fi
done
```
2021-12-10 09:17:17 -06:00
reesporte
af3c5809e2 add support for multi-node queries 2021-12-06 14:57:05 -06:00
reesporte
fa635874d4 encode/decode timestamp val appropriately 2021-11-22 10:08:18 -06:00
Seebs
610c4ed6cb introduce protobuf types for ingest ops
We add a new protobuf type. Also, protoc changed slightly and remade
some tests, in a way which should have no effects but makes the code
*very* slightly cleaner.

This introduces the first testing code in encoding/proto (whoops)
so that scaffolding is a first draft; if you're looking at this code
and the design is a problem go ahead and fix it.

The purpose of this is to verify that we're actually covering all
the branches in the ingest.ShardedRequest and pb.ShardedIngestRequest
message conversions. (Except the top-level one for a nil request,
which isn't checked by this.)

The coverage report doesn't actually include coverage for the ingest
code, though, so we haven't actually properly tested Compare.
Baby steps!
2021-09-27 12:05:57 -05:00
Ben Johnson
d16978f5dc Add max memory limit to Extract() to prevent OOM
This commit changes the Extract() query to return an error if the
result set gets too large in order to prevent out-of-memory (OOM)
panics.
2021-08-02 08:20:12 -06:00
Nia Weiss
f4ba34247f
remove attributes
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
2021-05-14 10:28:08 -04:00
Ben Johnson
d70eb737cb Fix timestamp field issues 2021-04-09 08:26:12 -06:00
Kuba Podgórski
1e5388dbe9 Move internals proto files into separate package (pb) 2021-03-22 20:28:00 +01:00