Commit graph

418 commits

Author SHA1 Message Date
Seebs
f019cc7409 make import correctly reflect that it needs a single shard always
In fact, we have a number of things assuming that values passed to Import
always fit within a single known shard, so, drop all the extra complexity
around this, drop the computation of fancy view/shard keys, and so on.

There's a lot of room left to improve this probably but it's at least
better, I think.

Unfortunately, there's a handful of things, basically all of which are
test cases, which were relying on this, so, we also add functionality
for splitting import requests by shards. But this allows us to stop
duplicating each shard's inputs one at a time... which turns out to
mean that we now care that the import operation can write back to the
import request. This only affects test cases, so we adopt a crufty
hack involving cloning import requests in those rare cases, and also
when reusing the same column IDs to write to the existence field that
we'd be using later to write to another field.

Note that even if we weren't overwriting the column IDs with positions,
we'd be sorting the column/row ID lists by row-then-column, which means
we'd still be corrupting the column ID lists. This may want to change
at some point.

We also reuse a single Tx for all the views, because DB-per-shard
means that should work fine, and reduces the cost of doing these
updates, probably.
2021-08-18 13:45:36 -05:00
Seebs
e768fc89ea stop using pointers to time.Time
We're reading timestamps as []int64, instead of allocating a time.Time
for each timestamp, just use the same logic to determine whether to use the
int64 timestamp that we would have used to decide whether to allocate it.
We still have to check the whole run, though, because we're providing a large
list of 0s instead of "no timestamps", for Reasons.
2021-08-18 13:45:36 -05:00
Seebs
35faa39b20 don't use nil qcx
A nil Qcx is a crime against existence and makes baby pandas cry.

Having taken out the hack that tried to accommodate this when tests did it,
we now have to fix the tests. Oh no.
2021-08-18 13:45:36 -05:00
Alan Bernstein
9565620f4d Disable /ui/usage endpoint completely when usage-duty-cycle is set to 0 2021-08-04 16:22:55 -05:00
Ben Johnson
d16978f5dc Add max memory limit to Extract() to prevent OOM
This commit changes the Extract() query to return an error if the
result set gets too large in order to prevent out-of-memory (OOM)
panics.
2021-08-02 08:20:12 -06:00
Mahesh Arumugam
f51fd2351c Merge branch 'master' into ma/cloud-109 2021-07-19 12:40:35 -07:00
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Alan Bernstein
6415067200 Make duration print format more readable 2021-07-15 14:22:17 -05:00
Alan Bernstein
60f2893152 Add usageCache logging 2021-07-15 14:22:17 -05:00
Alan Bernstein
46df93c521 Unindent 2021-07-15 14:22:17 -05:00
Alan Bernstein
f1e12567a3 Define some constants for usageCache 2021-07-15 14:22:17 -05:00
Alan Bernstein
b7898b0a22 Allow usage-duty-cycle < 20%, and 0 disables 2021-07-15 14:22:17 -05:00
Samir Patel
6b45325ca7 expand comment on duty cycle 2021-06-15 09:37:42 -05:00
Samir Patel
fb11d7f656
Update api.go
Co-authored-by: Alan Bernstein <alanaaronbernstein@gmail.com>
2021-06-15 09:17:53 -05:00
Samir Patel
d83a502657 change comment 2021-06-14 11:55:23 -05:00
Samir Patel
bfc90d148d Merge branch 'reset-cache-on-schema' of github.com:54mir/pilosa into reset-cache-on-schema 2021-06-14 11:53:46 -05:00
Samir Patel
6bdf685471 change waitMultiplier to float64 2021-06-14 11:43:49 -05:00
Samir Patel
11207fc589
Update api.go
Co-authored-by: Alan Bernstein <alanaaronbernstein@gmail.com>
2021-06-14 11:12:12 -05:00
Samir Patel
8bf472bb75 add duty cycle config flag 2021-06-14 10:48:52 -05:00
Samir Patel
011291f19b update comments 2021-06-11 11:28:39 -05:00
Samir Patel
b46b5fc134 remove usage-interval flag 2021-06-11 11:16:32 -05:00
Samir Patel
268e710e69 update comment 2021-06-10 17:36:03 -05:00
Samir Patel
9b11ded9e3 scale refresh rate by last calculation time 2021-06-10 16:30:22 -05:00
Samir Patel
a7c24c6745 lock read of lastUpdated 2021-06-10 11:28:23 -05:00
Samir Patel
486244ab58 reset cache if outdated on call 2021-06-10 10:48:44 -05:00
Samir Patel
5ef218977f
Merge branch 'master' into lattice-fix 2021-06-04 10:16:46 -05:00
nagamocha3000
a904d9e68e Delete ids allocated for index if any present 2021-06-04 14:34:48 +03:00
Samir Patel
883fa9ac0b replace sleep with after in select 2021-06-01 21:24:08 -05:00
Samir Patel
15ef157779 remove comment 2021-06-01 20:50:55 -05:00
Samir Patel
2e08b15620 remove lastupdate log to console 2021-06-01 20:48:44 -05:00
Samir Patel
041ae01da1 poll places in usage calculation to check for closing 2021-06-01 20:46:44 -05:00
Samir Patel
a9079815b2 remove prints 2021-06-01 14:45:52 -05:00
Samir Patel
6a8bf3acd6 stops calculation on server close 2021-06-01 14:34:51 -05:00
Samir Patel
660c6d10ca rename locks and remove prints 2021-06-01 10:47:10 -05:00
Samir Patel
017d012d51 change lock order 2021-06-01 10:47:10 -05:00
Samir Patel
20b721ad9d add wait group 2021-06-01 10:47:10 -05:00
Samir Patel
81413c4f0a Remove recalculation on new index 2021-06-01 10:47:10 -05:00
Samir Patel
a5ae6c15fc add locks 2021-06-01 10:47:10 -05:00
Samir Patel
fcdefe8bb6 add ability to reset cache 2021-06-01 10:47:10 -05:00
Samir Patel
a7620012fa adjust locking 2021-06-01 10:47:10 -05:00
Samir Patel
4b7e0e9194 move where cache updates lastUpdated val 2021-06-01 10:47:10 -05:00
Samir Patel
abcd90b60a remove print statements 2021-06-01 10:47:10 -05:00
Samir Patel
53b0e98bb1 add while loop to wait for holder to populate 2021-06-01 10:47:09 -05:00
Samir Patel
5cbf828588 adjust timing 2021-06-01 10:47:09 -05:00
Samir Patel
8824dddc3a add debug statements 2021-06-01 10:47:09 -05:00
Samir Patel
a0ba9327f7 play with timing 2021-06-01 10:47:09 -05:00
Samir Patel
dbdd3c4998 see if this is the only test failing 2021-06-01 10:47:09 -05:00
Samir Patel
fd58fe1a7d test stuff 2021-06-01 10:47:09 -05:00
Samir Patel
63300db1bd try test with usage always blocking on calc 2021-06-01 10:47:09 -05:00
Samir Patel
108da005b7 test stuff 2021-06-01 10:47:09 -05:00