Commit graph

429 commits

Author SHA1 Message Date
Todd Gruben
7de69df424 added protection against trailing slash 2021-11-15 10:54:13 -06:00
Todd Gruben
2ddcbce8ad fix govet and gofmt errors in existing code 2021-10-29 13:14:27 -05:00
Seebs
ad30a926f4 Giant Commit: drop a bunch of stuff we don't use.
These commits are hard to disentagle, and doing them separately means
re-modifying the same chunks of code several times before removing it,
and similar things.

Basically:
(1) Drop the bolt backend storage.
(2) Drop the blue-green wrapper that compares two backends.
(3) Drop unused or barely-used Tx API components from all the
remaining backends.
(4) Minor related cleanup to simplify things related to these.

The boltdb backend existed only to verify RBF. The blue-green wrapper
was mostly used to verify RBF, but in practice we had to do a lot
of working around that, and it introduced a lot of special cases.

Types removed:

IteratorFinder: Used only to implement the roaring iterator
on top of boltdb, and to complicate the way it worked in roaring.
Reverted the complications. Also unexport NewSliceContainers
which is used only for that outside of roaring's internals.

PortMapper from cluster_internal_test.go: Used only for a test
we removed early this year. Never used for anything else.

RawRoaringData: Totally unused.

TxStore: Totally unused.

Functions removed from Tx API, and sometimes corresponding
members were removed from structs:

* Dump: debugging code, I don't think I found any actually reachable
  paths to it.
* Group: only used for debugging TxGroup stuff
* IncrementOpN: only used by fragment, fragment can increment its
  own opN.
* Options: unused?
* Pointer: debugging only
* Readonly: used only to decide how to handle Tx in a TxGrp,
  but we never add a non-readonly Tx to a TxGrp. Removed also all
  the corresponding write-aware stuff.
* RoaringBitmapReader: Used exactly once, can just be a bm.WriteTo.
* Sn (and OpenSnList): Unused
* UnionInPlace: unused and conceptually-invalid; it didn't write
  to storage and shouldn't have, and was just "create a bitmap
  then call union-in-place", which we can do directly.
* UseRowCache: just checked storage.UseRowCache.

Other things removed:

The SetRequiredForAtomicWriteTx and ClearRequiredForAtomicWriteTx
functions go away, since nothing now seems to be using them? Same
for holder_internal_test's `testHasBit` and `testMustNotHaveBit`,
which were unused.

The DBPerShard "DeleteDBPath" and "HasData" functions and related
parts were mostly unused; took out the parts that were never
actually being reached.

Changed the API of one function to simplify special cases and
remove things:
* ImportRoaringBits had a special "data" argument which gave it
  subtly different semantics for RBF and roaring (for roaring, it
  could produce a roaring bitmap *with ops log*), didn't seem to
  be adding much. Removed corresponding "readStorageFromArchive"
  which is not otherwise used.

Also took out various debugging/dumping functions that were unused
and may have bitrotted.

Dropped a test from txfactory_internal_test, and the "pjobs"
code, because those two were the only things that needed Barrier
and thus idem, which lets us drop two more dependencies. We already
have errgroup for grouping things which want to terminate as
soon as one of them errors, approximately. To do better we'd have
to have context-threading, really.

Unbroke the WriteFragment test for non-roaring tests and made it
not roaring-only.
2021-10-26 12:30:25 -05:00
Todd Gruben
786bebe58b partial backup/restore 2021-10-08 10:50:52 -05:00
Samir Patel
a845d93a25 Add license headers 2021-09-30 09:28:33 -05:00
Samir Patel
5d070b47bb clean up 2021-09-29 15:57:31 -05:00
Samir Patel
feb8997ca8 Add darwin build to roaring-migrate-tool 2021-09-29 15:42:50 -05:00
Ben Johnson
92646cbdf7 Implement non-aggregate SELECT query 2021-09-08 08:09:54 -06:00
Mahesh Arumugam
bce6d91618 Merge branch 'master' into ma/cloud-109 2021-07-19 14:42:36 -07:00
Mahesh Arumugam
44a5e68b3e fix build issue 2021-07-19 13:47:47 -07:00
Mahesh Arumugam
c14c6afbd3 Merge branch 'master' into ma/cloud-109 2021-07-19 13:27:17 -07:00
Travis
7e718743d2
Rename pilosa binary to featurebase 2021-07-19 15:27:14 -05:00
Todd Gruben
be0789beb3 use filepath.join 2021-07-19 12:55:39 -05:00
Todd Gruben
bc0c98cca7 applied ben's suggestions 2021-07-19 12:34:43 -05:00
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Todd Gruben
023d05aaf9 linter fix 2021-07-16 14:22:14 -05:00
Todd Gruben
3c9e1c74af phase 1 complete all data migrated 2021-07-16 13:44:18 -05:00
Mahesh Arumugam
48fcb3a198 FeatureBase renaming: version info in server log 2021-07-12 12:25:43 -07:00
Mahesh Arumugam
eb1cc304cf FeatureBase Renaming: EnvPrefix 2021-07-09 14:52:40 -07:00
Todd Gruben
44281fa5c2 add chksum command for easier data validation 2021-06-23 16:10:52 -05:00
Ben Johnson
dad2372a8c CORE-640: Parallelize restore command 2021-06-18 16:03:38 -06:00
Ben Johnson
3df1f2d094 CORE-639: Parallelize backup 2021-06-15 16:55:05 -06:00
Ben Johnson
73f0668fe1 Refactor backup/restore to use directory archive 2021-06-14 09:43:20 -06:00
Todd Gruben
94e66b74b5 tls config bug and arge handling 2021-05-24 11:00:37 -05:00
Todd Gruben
42b465b80c load schema 2021-05-21 09:27:08 -05:00
Todd Gruben
9d24fb07b7 wired in restore command 2021-05-21 09:27:08 -05:00
Todd Gruben
7c6423587a skeleton restore 2021-05-21 09:27:08 -05:00
Ben Johnson
a4f282c8e8 Allow backup to stdout 2021-05-19 15:04:36 -06:00
Ben Johnson
776b43a3cd Backup CLI 2021-05-07 10:59:01 -06:00
Alan Bernstein
285d0a0af8 Add log prefix levels 2021-04-12 20:33:39 -05:00
Kuba Podgórski
9a02004a4f Move vprint to separate pakage 2021-03-29 14:29:19 +02:00
Matt Jaffee
11288c2ae8
Revert "Basic pilosa changes for oracle support"
This reverts commit 47e74f2603.
2021-03-14 22:17:26 -05:00
Seebs
1045268f01 use testhook to ensure temporary files and directories are cleaned up
In nearly all cases, we can just switch ioutil.TempDir->testhook.TempDir
and similarly for TempFile. There's one case where we can't because we
need files to be removed before tests are over.

Also in the process give identifiable names to a lot of temporary files
and make sure they're being cleaned up, and don't use "/tmp/foo" as a
file name in a test that could be running in more than one test process
at once. :)
2021-03-11 19:42:10 -06:00
tgruben
21910ef321
Update main.go 2021-03-11 11:47:57 -06:00
tgruben
a5dceaae11
Update main.go 2021-03-11 11:26:58 -06:00
Todd Gruben
47e74f2603 Basic pilosa changes for oracle support 2021-03-11 11:16:53 -06:00
Travis
2bbe1fdde0
remove remaining references to "coordinator" 2021-02-23 17:23:09 -06:00
Travis
a2a6e91f6d
remove old, now conflicting test value 2021-02-12 21:21:56 -06:00
Travis
114f6a8751
add withViews argument to api.Schema() method 2021-02-08 10:42:55 -06:00
Travis
4dd6bddf7f
remove pilosa-chk 2021-02-05 16:13:43 -06:00
Travis
c8c59b649d
remove pilosa-fsck 2021-02-04 21:34:15 -06:00
Travis
a4b37273ea
remove the rest of the gossip code (except config) 2021-02-04 13:03:02 -06:00
Antonio Navarro Perez
87ba73fa16 Stop writes on DEGRADED state
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-04 17:30:14 +01:00
Travis
457194f6a8
update config to support etcd arguments 2021-01-29 19:43:48 -06:00
Travis
1c0b926eae
Merge master into disco 2021-01-28 18:03:38 -06:00
Travis
4380a05bbd
address some coord/node0 test issues 2021-01-26 22:46:37 -06:00
Travis
ace4dea46f
address some test failures due to random ordered etcd ID 2021-01-25 00:52:49 -06:00
Travis
1d55e671a2
go mod tidy and linter
fix race

cleanup
2021-01-15 17:48:44 -06:00
Travis
9855f4d0a0
Merge branch 'travis-test-ci' into disco-try 2021-01-15 14:31:19 -06:00
Travis
10380a1da1
Implement snap := ClusterSnapshot()
Below is the list of instance of `ClusterSnapshot()` in the latest
`with-etcd` code. Some of these may not yet exist in the `disco` branch,
but this commit is implementing any that currently apply.

==========================
Done:
==========================
index.go
930:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1072:	snap := NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

cmd/pilosa-fsck/fsck.go
786:	snap := pilosa.NewClusterSnapshot(cfg.topo, cfg.topo.Hasher, cfg.topo.ReplicaN)

boltdb/translate.go
558:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)
1264:	snap := pilosa.NewClusterSnapshot(topo, topo.Hasher, topo.ReplicaN)

fragment.go
3448:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3568:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
3620:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)

==========================
Remaining:
==========================

cluster.go
371:	snap := NewClusterSnapshot(NewLocalNoder(nodes), c.Hasher, c.ReplicaN)
474:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
639:	fSnap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
640:	toSnap := NewClusterSnapshot(to.noder, c.Hasher, to.ReplicaN)
703:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1475:		snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1502:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1941:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
1986:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2049:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)
2126:	snap := NewClusterSnapshot(c.noder, c.Hasher, c.ReplicaN)

api.go
475:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
604:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
690:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1684:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)
1946:	snap := NewClusterSnapshot(api.cluster.noder, api.cluster.Hasher, api.cluster.ReplicaN)

executor.go
3781:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4157:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4200:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4243:	snap := NewClusterSnapshot(e.Cluster.noder, e.Cluster.Hasher, e.Cluster.ReplicaN)
4517:	snap := NewClusterSnapshot(NewLocalNoder(e.Cluster.Nodes()), e.Cluster.Hasher, e.Cluster.ReplicaN)

holder.go
1465:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1668:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1889:	snap := NewClusterSnapshot(s.Cluster.noder, s.Cluster.Hasher, s.Cluster.ReplicaN)
1963:	snap := NewClusterSnapshot(c.Cluster.noder, c.Cluster.Hasher, c.Cluster.ReplicaN)
2021-01-15 11:47:37 -06:00