Commit graph

90 commits

Author SHA1 Message Date
Seebs
6af987a5ba fix error formatting/spelling
Go convention is that error messages don't end with periods and
don't start with capital letters.
2021-08-18 13:45:36 -05:00
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Samir Patel
81f1606242 poll places in indexDetails to check for closing 2021-06-01 20:47:53 -05:00
Samir Patel
4e2727c255 revert txfactory to original 2021-06-01 10:47:10 -05:00
Samir Patel
eff3b25b97 change test case to reflect cache loading before test 2021-06-01 10:47:10 -05:00
Samir Patel
53b0e98bb1 add while loop to wait for holder to populate 2021-06-01 10:47:09 -05:00
Samir Patel
8824dddc3a add debug statements 2021-06-01 10:47:09 -05:00
Samir Patel
a0ba9327f7 play with timing 2021-06-01 10:47:09 -05:00
Samir Patel
e7f1c8b66a Reverted directoryUsage back to using old Readdir() 2021-06-01 10:47:09 -05:00
Samir Patel
11d1859830 undo changes to IndexUsageDetails 2021-06-01 10:47:08 -05:00
Samir Patel
4842d93850 adds logic to remove old items from cache 2021-06-01 10:47:08 -05:00
Samir Patel
d71da09db2 add cache update on time 2021-06-01 10:47:08 -05:00
Samir Patel
713ffb723b replace ReadDir 2021-06-01 10:47:08 -05:00
Samir Patel
155003122b replace ReadDir syscall with new one from 1.16 2021-06-01 10:47:08 -05:00
Samir Patel
c812cca58e add cache 2021-06-01 10:47:08 -05:00
Nia Weiss
f4ba34247f
remove attributes
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
2021-05-14 10:28:08 -04:00
Alan Bernstein
285d0a0af8 Add log prefix levels 2021-04-12 20:33:39 -05:00
Kuba Podgórski
9a02004a4f Move vprint to separate pakage 2021-03-29 14:29:19 +02:00
Travis
f78a6508f0
Remove stutter and suffix from backend files 2021-03-07 22:11:51 -06:00
Travis
cad78f1d34
remove "Default*" from const names 2021-03-05 16:56:44 -06:00
Travis
4661f3ac08
reorganize the storage backends directory 2021-03-05 16:56:44 -06:00
Travis
345d076fdf
introduce "fields" directory between index and field 2021-03-05 16:56:44 -06:00
Travis
7789e24965
introduce "indexes" directory between datadir and index 2021-03-05 16:56:43 -06:00
Travis
d192c1f24f
Merge branch 'master' into disco 2021-02-05 15:58:36 -06:00
Alan Bernstein
1e7c4d7e8e Include metadata AKA 'other' in response 2021-01-29 17:48:24 -06:00
Alan Bernstein
c81ea88847 Fix total summation 2021-01-29 17:48:24 -06:00
Alan Bernstein
703bd14048 Fix errors in usage check 2021-01-29 17:48:24 -06:00
Alan Bernstein
85fad859e2 Correct some disk usage computations 2021-01-29 17:48:24 -06:00
Ben Johnson
32a35805a4 Add RBF index/field usage stats 2021-01-29 17:48:24 -06:00
Alan Bernstein
e397d35ed5 Include roaring field and key details in usage endpoint 2021-01-29 17:48:24 -06:00
Travis
19f91782e7
remove all instances of txsrc 2021-01-21 21:59:51 -06:00
Travis
13984353e4
remove instances of os.Getenv("PILOSA_TXSRC") 2021-01-21 21:18:46 -06:00
Travis
f292d6061a
replace pilosa.DefaultTxsrc with storage.DefaultBackend 2021-01-20 22:06:13 -06:00
Travis
08fae2be4c
introduce storage.Config 2021-01-20 22:05:38 -06:00
Jason E. Aten
035073555a pilosa: only open views with data
- Previously, on timequantum schemas, we would
create and open a view for the cartesian
product of every possible view and shard.

- This caused us to be very slow on re-open,
and to use lots of memory for views that
held nothing.

- This change makes startup faster, memory
use much lower, and should speed migration.
2020-12-19 00:03:31 +00:00
Cody Soyland
4cb21da6e9 Fix disk usage calculation in RBF backend 2020-12-15 13:39:27 -06:00
Todd Gruben
8ad7afbe43 FragProxy reduces string memory consumption drastically
for datasets with lots of views, because we don't
replicate path, index, field, view strings so often.
2020-12-11 21:01:15 +00:00
Cody Soyland
c1fe8b214a Default TxSrc to roaring 2020-12-09 09:31:40 -06:00
Jason E. Aten
7a9e0969cb remove lmdb as a Tx backend
- test only still uses a Barrier utility from the go-lmdb package;
  it could be ported in at some point.
2020-12-04 23:17:59 +00:00
Ben Johnson
8de1959938 Remove RBF exclusive/direct write. 2020-11-09 08:23:09 -07:00
Jason E. Aten
7e22994a6d run migration in parallel
- migration can be slow. parallelize it.
2020-11-05 18:31:27 +00:00
Jason E. Aten
458095a707 rbf default. Add TODO comments, slurp -profile returns a cpu profile
- default Tx is once again RBF, changed from bolt.
- document the RBF code review comments that were not addressed
  before #1052 was merged, so they don't get lost.
- they should be easily addressed by replaying the entire WAL file
  rather than from the DB meta page 0 notion of the last WalID
- cleanup rbf/cfg/cfg.go stale comments, ensure default0 respected.
  1 msec checkpoint time, 1MB wal segment defaults.
- return a specific error, ErrNoMetaFound, from findNextWALMetaPage()
  rather than io.EOF, since there actually wasn't any file IO involved.
- add http handlers for /cpu-profile/start and /cpu-profile/stop
  in http/handler.go enable CPU profiling at specific time points
  during an ingest or other operation.
2020-11-04 01:36:17 +00:00
Jason E. Aten
bf36f54b8d take a mutex off the fast path of TxFactory.NewTx 2020-11-04 01:04:04 +00:00
Jason E. Aten
6aadd13095 qcx.GetTx returns an error
- to indicate that the query context is already
   done.
 - handles the case where the import worker is
   interrupted early by a ctx cancellation,
   thus avoiding a panic.
2020-11-04 00:01:02 +00:00
Matt Jaffee
83dd8026d5
default to Bolt txn due to CI failures blocking other team
Our nightly CI has been failing for a week due to WAL issues and it's
making it difficult for Kuba and Antonio to do things on the
integration repo. Hoping bolt backend will solve that in the short
term. I think the issue is Pilosa #1046 (that's from memory though)
2020-10-30 13:55:36 -05:00
Jason Aten
016e5e0774 no automatic reuse of Qcx
Per slack discussion with Seebs and Nia,
we'll try not automatically resetting the Qcx.

The worry was that our goroutine shutdown
management is so poor that we are asking for
GetTx on a goroutine that still has a Qcx
from a query that was cancelled.

If this is the case, we will now panic instead of
issuing a new Tx. Then we can fix the poor
goroutine management.

 - also require Qcx.Finish or Abort before Reset
2020-10-30 00:36:46 +00:00
Jason E. Aten
f8e7b27a6f don't apply startup shard cache to roaring with a specified view path
- avoids creating a new empty 8 byte shard file under all the
   views that don't have them already.
2020-10-29 22:09:55 +00:00
Jason E. Aten
2d55ffbd28 short circuit if no data to migrate 2020-10-29 02:44:22 +00:00
Jason E. Aten
d4df15721d allow migration of empty to empty 2020-10-29 02:41:09 +00:00
Jason E. Aten
f9f94aab0d fix typo 2020-10-29 02:15:43 +00:00