Commit graph

7698 commits

Author SHA1 Message Date
Seebs
f019cc7409 make import correctly reflect that it needs a single shard always
In fact, we have a number of things assuming that values passed to Import
always fit within a single known shard, so, drop all the extra complexity
around this, drop the computation of fancy view/shard keys, and so on.

There's a lot of room left to improve this probably but it's at least
better, I think.

Unfortunately, there's a handful of things, basically all of which are
test cases, which were relying on this, so, we also add functionality
for splitting import requests by shards. But this allows us to stop
duplicating each shard's inputs one at a time... which turns out to
mean that we now care that the import operation can write back to the
import request. This only affects test cases, so we adopt a crufty
hack involving cloning import requests in those rare cases, and also
when reusing the same column IDs to write to the existence field that
we'd be using later to write to another field.

Note that even if we weren't overwriting the column IDs with positions,
we'd be sorting the column/row ID lists by row-then-column, which means
we'd still be corrupting the column ID lists. This may want to change
at some point.

We also reuse a single Tx for all the views, because DB-per-shard
means that should work fine, and reduces the cost of doing these
updates, probably.
2021-08-18 13:45:36 -05:00
Seebs
6af987a5ba fix error formatting/spelling
Go convention is that error messages don't end with periods and
don't start with capital letters.
2021-08-18 13:45:36 -05:00
Seebs
ab9b70d7e8 mapperLocal: actually leave loop on read from done channel
staticcheck points out that the break is otherwise an ineffective
break because it just ends the current case clause of the switch
it's in, which is true.
2021-08-18 13:45:36 -05:00
Seebs
1745a93aee allow "us" for microseconds in timestamp units
The convention of using a "u" for "micro" is pretty well-established and some
people will have trouble typing the Greek letter, accept that as a synonym.
2021-08-18 13:45:36 -05:00
Seebs
e768fc89ea stop using pointers to time.Time
We're reading timestamps as []int64, instead of allocating a time.Time
for each timestamp, just use the same logic to determine whether to use the
int64 timestamp that we would have used to decide whether to allocate it.
We still have to check the whole run, though, because we're providing a large
list of 0s instead of "no timestamps", for Reasons.
2021-08-18 13:45:36 -05:00
Seebs
35faa39b20 don't use nil qcx
A nil Qcx is a crime against existence and makes baby pandas cry.

Having taken out the hack that tried to accommodate this when tests did it,
we now have to fix the tests. Oh no.
2021-08-18 13:45:36 -05:00
Seebs
dab8dff9c2 make "subdivide list by shardwidth" available for reuse
We keep wanting this, and it's shardwidth-dependent code, and we keep rewriting it.
It should be in the shardwidth package.
2021-08-18 13:45:20 -05:00
Seebs
7b27a48d7c allow Molecula copyrights 2021-08-17 15:23:31 -05:00
Ben Johnson
409d59e297
Merge pull request #1675 from molecula/sql-count
CORE-806: Implement basic SQL COUNT(*) query
2021-08-17 14:12:40 -06:00
Ben Johnson
702f55964f Add apt-get update flag 2021-08-17 07:21:40 -06:00
Ben Johnson
5122c2decc Refactor SQL planner to inside Server 2021-08-16 13:12:16 -06:00
Ben Johnson
9b8dc3d7e6 Implement basic SQL COUNT(*) query 2021-08-13 10:41:39 -06:00
Matthew Jaffee
dac6234d0d
Merge pull request #1671 from jaffee/differentiate-map-error
differentiate error messages between opening file and mapping it
2021-08-09 12:27:22 -05:00
Matthew Jaffee
207634aea3 differentiate error messages between opening file and mapping it 2021-08-09 11:42:25 -05:00
Matthew Jaffee
54a4c2a587
Merge pull request #1663 from molecula/fully-disable-usage-endpoint
CORE-800 Disable /ui/usage endpoint completely when usage-duty-cycle is set to 0
2021-08-05 09:37:10 -05:00
Alan Bernstein
9565620f4d Disable /ui/usage endpoint completely when usage-duty-cycle is set to 0 2021-08-04 16:22:55 -05:00
Alan Bernstein
51cc29364c
Merge pull request #1669 from molecula/feature/syang/aggregate-sort
CLOUD-61: Add aggregate sort option to query builder
2021-08-02 20:46:58 -05:00
Stephanie Yang
1609582ba1 Update aggregate label to cardinality 2021-08-02 19:01:37 -05:00
Stephanie Yang
97df326ac6 update test and add additional test for aggregate sort 2021-08-02 19:01:37 -05:00
Stephanie Yang
b0633898a3 add ability to set an aggregate sort on int fields 2021-08-02 19:01:37 -05:00
Ben Johnson
009f4d6d71
Merge pull request #1668 from molecula/extract-max-memory
CORE-473: Add max memory limit to Extract() to prevent OOM
2021-08-02 15:44:27 -06:00
Ben Johnson
60d534c505 Limit translation memory & add max query memory config 2021-08-02 15:28:00 -06:00
Ben Johnson
d16978f5dc Add max memory limit to Extract() to prevent OOM
This commit changes the Extract() query to return an error if the
result set gets too large in order to prevent out-of-memory (OOM)
panics.
2021-08-02 08:20:12 -06:00
Stephanie Yang
317f1e0145
Merge pull request #1665 from molecula/refactor/syang/query-builder
CLOUD-136: Refactor query builder
2021-07-30 09:52:45 -05:00
Stephanie Yang
2947583a95 fix issue with editing saved queries 2021-07-29 12:03:32 -05:00
Stephanie Yang
8dc2105b63 add clear link for sort 2021-07-28 11:38:13 -05:00
Stephanie Yang
d8ac3251f6 replace old query builder 2021-07-28 11:21:54 -05:00
Stephanie Yang
fa353faf72 initial refactor commit 2021-07-28 11:21:54 -05:00
Alan Bernstein
6935ac2667
Merge pull request #1664 from molecula/fix-apt-404
Add apt-get update to dockerfile
2021-07-27 13:52:31 -05:00
Alan Bernstein
a7654ee750 Use -y for apt-get command 2021-07-27 10:21:08 -05:00
Alan Bernstein
9e56335ff9 Add apt-get update to dockerfile 2021-07-26 17:37:39 -05:00
Mahesh Arumugam
abb9b930bf
Merge pull request #1662 from molecula/rowcache-race-again
need to create rowCache lock at the top... otherwise it does nothing
2021-07-20 11:06:38 -07:00
Matthew Jaffee
0d7f1c49c4 need to create rowCache lock at the top... otherwise it does nothing 2021-07-20 12:15:35 -05:00
Matthew Jaffee
e42c96e996
Merge pull request #1661 from molecula/other-client-rowCache-race
fix other batch client rowCache race
2021-07-20 11:06:05 -05:00
Matthew Jaffee
d2e8c5f773 fix other batch client rowCache race 2021-07-20 10:36:17 -05:00
Matthew Jaffee
e450099579
Merge pull request #1660 from molecula/client-batch-race
Fix data race in batch import client by adding locking around rowCache
2021-07-20 09:40:28 -05:00
Matthew Jaffee
11686cac97 add locking around rowCache to avoid data race 2021-07-20 09:00:49 -05:00
Mahesh Arumugam
547d74b0fc
Merge pull request #1658 from molecula/ma/cloud-109
CLOUD-109 FeatureBase Renaming: changing go.mod module name for featurebase
2021-07-19 15:33:10 -07:00
Mahesh Arumugam
bce6d91618 Merge branch 'master' into ma/cloud-109 2021-07-19 14:42:36 -07:00
Travis Turner
695cb10986
Merge pull request #1656 from travisturner/rename-binary
Rename pilosa binary to featurebase
2021-07-19 16:15:59 -05:00
Mahesh Arumugam
44a5e68b3e fix build issue 2021-07-19 13:47:47 -07:00
Mahesh Arumugam
c14c6afbd3 Merge branch 'master' into ma/cloud-109 2021-07-19 13:27:17 -07:00
Travis
7e718743d2
Rename pilosa binary to featurebase 2021-07-19 15:27:14 -05:00
tgruben
eb9adcdec3
Merge pull request #1657 from tgruben/migrate-to-rbf
[ CORE-733] Migration tool for rbf
2021-07-19 15:08:50 -05:00
Mahesh Arumugam
f51fd2351c Merge branch 'master' into ma/cloud-109 2021-07-19 12:40:35 -07:00
Mahesh Arumugam
faf1f93071 fix clustertests 2021-07-19 12:33:22 -07:00
Mahesh Arumugam
a021412d00 fix clustertests 2021-07-19 12:17:59 -07:00
tgruben
fb7f8a69ea
Merge branch 'master' into migrate-to-rbf 2021-07-19 13:05:16 -05:00
Todd Gruben
be0789beb3 use filepath.join 2021-07-19 12:55:39 -05:00
Todd Gruben
bc0c98cca7 applied ben's suggestions 2021-07-19 12:34:43 -05:00