Commit graph

25 commits

Author SHA1 Message Date
Todd Gruben
6c512359a1 missed a fmt statement 2022-02-14 14:53:47 -06:00
Todd Gruben
cbc9bf71a1 logging 2022-02-14 14:41:22 -06:00
Todd Gruben
8a48c1b67a standard logger 2022-02-14 14:09:05 -06:00
tgruben
c4eebc6885
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:55 -06:00
tgruben
89598a7788
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:42 -06:00
tgruben
e1e968ef97
Update cmd/roaring-migrate/main.go
Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2022-02-14 12:17:24 -06:00
Todd Gruben
9bc28091c9 better testing 2022-02-14 12:11:28 -06:00
Todd Gruben
8d5cdbdd77 roaring-migrate bug;performance improvements 2022-02-11 11:19:44 -06:00
Ben Johnson
9ebf0e2119 Upgrade go.mod to featurebase/v3 2022-01-21 10:57:05 -07:00
Todd Gruben
c25ab78b03 use higherlevel iterator in order to account for ops log 2021-12-13 13:10:16 -06:00
reesporte
48aef0c8a4 add copyright notice back in
```bash
for file in `cat diffys`; do
   printf '%s\n%s\n' "// Copyright 2021 Molecula Corp. All rights reserved." "$(cat $file)" >$file;
done
```
2021-12-10 11:01:04 -06:00
reesporte
4c53f86e82 removed license from each go file
i used this script, a little clunky but it got the job done

```bash
for file in `find . -type f -print | grep '\.go'`; do
    sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
    result=`cat $file.tmp`
    if [[ result != "" ]]; then
        gofmt $file.tmp &> /dev/null;
        if [[ $? == 0 ]]; then
            mv $file.tmp $file && gofmt -w $file;
        else
            rm $file.tmp;
        fi
    else
        rm $file.tmp;
    fi
done
```
2021-12-10 09:17:17 -06:00
reesporte
76c1d237ac translaste -> translate
also remove meaingless comments
2021-12-08 16:35:26 -06:00
reesporte
b046ad5e8f fixes some more staticcheck errors 2021-12-03 16:50:02 -06:00
Todd Gruben
7de69df424 added protection against trailing slash 2021-11-15 10:54:13 -06:00
Todd Gruben
2ddcbce8ad fix govet and gofmt errors in existing code 2021-10-29 13:14:27 -05:00
Seebs
ad30a926f4 Giant Commit: drop a bunch of stuff we don't use.
These commits are hard to disentagle, and doing them separately means
re-modifying the same chunks of code several times before removing it,
and similar things.

Basically:
(1) Drop the bolt backend storage.
(2) Drop the blue-green wrapper that compares two backends.
(3) Drop unused or barely-used Tx API components from all the
remaining backends.
(4) Minor related cleanup to simplify things related to these.

The boltdb backend existed only to verify RBF. The blue-green wrapper
was mostly used to verify RBF, but in practice we had to do a lot
of working around that, and it introduced a lot of special cases.

Types removed:

IteratorFinder: Used only to implement the roaring iterator
on top of boltdb, and to complicate the way it worked in roaring.
Reverted the complications. Also unexport NewSliceContainers
which is used only for that outside of roaring's internals.

PortMapper from cluster_internal_test.go: Used only for a test
we removed early this year. Never used for anything else.

RawRoaringData: Totally unused.

TxStore: Totally unused.

Functions removed from Tx API, and sometimes corresponding
members were removed from structs:

* Dump: debugging code, I don't think I found any actually reachable
  paths to it.
* Group: only used for debugging TxGroup stuff
* IncrementOpN: only used by fragment, fragment can increment its
  own opN.
* Options: unused?
* Pointer: debugging only
* Readonly: used only to decide how to handle Tx in a TxGrp,
  but we never add a non-readonly Tx to a TxGrp. Removed also all
  the corresponding write-aware stuff.
* RoaringBitmapReader: Used exactly once, can just be a bm.WriteTo.
* Sn (and OpenSnList): Unused
* UnionInPlace: unused and conceptually-invalid; it didn't write
  to storage and shouldn't have, and was just "create a bitmap
  then call union-in-place", which we can do directly.
* UseRowCache: just checked storage.UseRowCache.

Other things removed:

The SetRequiredForAtomicWriteTx and ClearRequiredForAtomicWriteTx
functions go away, since nothing now seems to be using them? Same
for holder_internal_test's `testHasBit` and `testMustNotHaveBit`,
which were unused.

The DBPerShard "DeleteDBPath" and "HasData" functions and related
parts were mostly unused; took out the parts that were never
actually being reached.

Changed the API of one function to simplify special cases and
remove things:
* ImportRoaringBits had a special "data" argument which gave it
  subtly different semantics for RBF and roaring (for roaring, it
  could produce a roaring bitmap *with ops log*), didn't seem to
  be adding much. Removed corresponding "readStorageFromArchive"
  which is not otherwise used.

Also took out various debugging/dumping functions that were unused
and may have bitrotted.

Dropped a test from txfactory_internal_test, and the "pjobs"
code, because those two were the only things that needed Barrier
and thus idem, which lets us drop two more dependencies. We already
have errgroup for grouping things which want to terminate as
soon as one of them errors, approximately. To do better we'd have
to have context-threading, really.

Unbroke the WriteFragment test for non-roaring tests and made it
not roaring-only.
2021-10-26 12:30:25 -05:00
Samir Patel
a845d93a25 Add license headers 2021-09-30 09:28:33 -05:00
Samir Patel
5d070b47bb clean up 2021-09-29 15:57:31 -05:00
Samir Patel
feb8997ca8 Add darwin build to roaring-migrate-tool 2021-09-29 15:42:50 -05:00
Mahesh Arumugam
44a5e68b3e fix build issue 2021-07-19 13:47:47 -07:00
Todd Gruben
be0789beb3 use filepath.join 2021-07-19 12:55:39 -05:00
Todd Gruben
bc0c98cca7 applied ben's suggestions 2021-07-19 12:34:43 -05:00
Todd Gruben
023d05aaf9 linter fix 2021-07-16 14:22:14 -05:00
Todd Gruben
3c9e1c74af phase 1 complete all data migrated 2021-07-16 13:44:18 -05:00