No description
Find a file
Seebs 97486f410b WIP: Union/UnionInPlace performance improvements
This consolidates a number of changes. The first is significant
reductions in allocation and copying during UnionInPlace
operations on very sparse containers -- for instance, combining
two array containers with one item each.

We fix up the logic for identifying and handling cases where
only one of the containers being unioned together has a given
key.

We generally favor cloning an existing container over unioning
it into a new empty container.

When unioning two containers, we were using unionIntoTargetSingle
on those two containers, into an empty bitmap. For more, we were
creating an empty bitmap, then unioning all the others into
it; it's faster to clone the first, then union the others into
it.

The overall logic for UnionInPlace is cleaned up and simplified
a bit. However, it's then complexified a bit, because it turns
out that while it's a bad idea to convert single-item arrays to
bitmaps to union them, by a few hundred items, the bitmap
conversion saves a lot of time even if it costs an allocation.

The value of N picked here is sort of arbitrary, but
512 seems to be about right. The big problem is a massive
performance hit in cases where, say, there's only a
couple of items per container, and the bitmap conversion
is extremely expensive. If you wait until N reaches
the array size cap, though, you take a very noticeable
performance hit (can be a factor of 2.5-3 in simple
testing).

We also add some stat counters, and rename an internal
method on the `handledIters` type.
2019-03-14 15:17:38 -05:00
.circleci Ensure Go modules disabled during installation of tools 2019-02-07 10:44:46 -06:00
.github provide friendlier prompts 2019-01-10 08:07:48 -06:00
boltdb Merge branch 'master' into 1517-metalinter-unconvert 2018-08-02 11:45:24 -05:00
cmd Add distributed tracing. 2018-11-21 15:08:33 -06:00
ctl improve help for metrics options 2019-03-04 17:36:01 -06:00
docs add Group By heading to query language docs 2019-02-06 12:49:57 -06:00
encoding/proto trivial 2019-01-04 01:54:05 +03:00
enterprise update licensing and NOTICE to reflect btree being moved to roaring 2019-03-05 15:52:32 -06:00
gcnotify Unexport gcnotify.ActiveGCNotifier 2018-07-05 23:11:56 -05:00
gopsutil Adds diagnostics CPUArch field 2018-10-01 15:31:13 +03:00
gossip Advertise address and listen on 0.0.0.0 2019-01-21 22:36:55 -06:00
http Advertise address and listen on 0.0.0.0 2019-01-21 22:36:55 -06:00
inmem Unexport inmem.TranslateStore 2018-07-05 23:11:56 -05:00
internal Add go module support 2019-01-29 14:53:03 -06:00
logger move Logger and Stats to their own packages 2018-11-15 15:10:44 -06:00
lru fix staticcheck warnings 2019-01-21 14:24:11 -06:00
mock ignore remote translation 2018-07-30 11:45:34 -05:00
pql fix staticcheck warnings 2019-01-21 14:24:11 -06:00
roaring WIP: Union/UnionInPlace performance improvements 2019-03-14 15:17:38 -05:00
server use UnionInPlace in import-roaring 2019-03-11 17:43:55 -05:00
stats move Logger and Stats to their own packages 2018-11-15 15:10:44 -06:00
statsd move Logger and Stats to their own packages 2018-11-15 15:10:44 -06:00
test trivial 2019-01-04 01:56:10 +03:00
testdata/sample_view Adds fragment test data, implements #1090 2018-05-01 17:28:10 +03:00
toml move pilosa.Config to pilosa/server.Config 2018-04-19 14:51:42 -05:00
tracing Add distributed tracing. 2018-11-21 15:08:33 -06:00
.gitignore Hide build directory in .gitignore 2017-05-19 14:17:57 -05:00
api.go comment adjustments 2019-02-19 10:54:09 -06:00
api_test.go Merge Range() into Row() call. 2019-01-09 15:10:49 -07:00
apimethod_string.go Merge branch 'master' into cluster-tests 2018-11-09 12:23:21 -06:00
attr.go Add distributed tracing. 2018-11-21 15:08:33 -06:00
attr_test.go Fix linter issues: staticcheck (covered by megacheck, along with gosimple and unused) 2018-07-20 11:19:24 -05:00
broadcast.go Revert "msg type stringer" 2018-11-09 11:34:32 -06:00
cache.go move Logger and Stats to their own packages 2018-11-15 15:10:44 -06:00
cache_test.go account for all same values in rank 2017-08-28 13:46:18 -05:00
CHANGELOG.md code review fixup for changelog 2018-12-20 11:13:00 -06:00
client.go Replace the /fragment/data endpoint to support cluster resizing 2018-12-18 08:34:47 -06:00
cluster.go use UnionInPlace in import-roaring 2019-03-11 17:43:55 -05:00
cluster_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
cmd.go Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
CONTRIBUTING.md Address code review feedback 2019-02-04 08:56:32 -06:00
diagnostics.go fix staticcheck warnings 2019-01-21 14:24:11 -06:00
diagnostics_internal_test.go fix failing tests due to staticcheck fixes 2019-01-21 14:38:58 -06:00
doc.go Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
Dockerfile Add go module support 2019-01-29 14:53:03 -06:00
Dockerfile-clustertests Add go module support 2019-01-29 14:53:03 -06:00
event.go export NodeEvent 2018-07-05 09:26:58 -05:00
executor.go remove extra nesting 2019-02-01 16:46:45 -06:00
executor_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
executor_test.go add logic to restrict time range to available views 2019-02-01 15:30:10 -06:00
field.go set cache size to 0 if cache type is none 2019-01-25 14:51:00 -06:00
field_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
field_test.go Add DeleteAvailableShard() 2018-10-17 08:50:06 -06:00
fragment.go improve union-related benchmarking 2019-03-14 15:16:49 -05:00
fragment_internal_test.go add importRoaring small updates benchmark 2019-03-11 17:43:55 -05:00
gc.go Fix linter issues: gochecknoinits 2018-07-18 11:58:27 -05:00
go.mod Regenerate go mod files 2019-02-07 11:46:30 -06:00
go.sum Regenerate go mod files 2019-02-07 11:46:30 -06:00
handler.go Merged with master 2018-11-26 17:03:38 +03:00
holder.go Adds shardWidth to index info in schema 2019-03-01 14:57:46 +03:00
holder_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
holder_test.go fixed a mistake in the test from PR 1780 2018-12-11 16:20:10 -06:00
index.go added missing index.go changes 2019-03-01 15:02:49 +03:00
index_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
index_test.go ditch OptFieldTypeTimeWithOptions 2018-11-20 23:21:09 +03:00
iterator.go Fix linter issues: unparam 2018-07-17 12:05:07 -05:00
iterator_internal_test.go rename internal_tests. fix license header 2018-07-04 10:59:08 -05:00
LICENSE Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
Makefile Go mod should be enabled here 2019-02-07 10:49:40 -06:00
NOTES rename slice to shard 2018-06-28 14:07:07 -05:00
NOTICE update licensing and NOTICE to reflect btree being moved to roaring 2019-03-05 15:52:32 -06:00
pilosa.go prevent omitting zero ids on columnattrs 2019-01-28 13:16:43 -06:00
pilosa_internal_test.go change exists field to _exists 2018-12-17 15:44:50 -06:00
pilosa_test.go un-export some top-level functions 2018-06-13 16:44:24 -05:00
README.md Remove TravisCI, add CircleCI shield 2018-11-13 12:34:44 -06:00
row.go Modifying some of the logic around Shift() 2019-01-25 12:54:44 -06:00
row_test.go a quicker empty check for group by 2019-01-02 14:32:59 -06:00
server.go fix staticcheck warnings 2019-01-21 14:24:11 -06:00
server_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
time.go add logic to restrict time range to available views 2019-02-01 15:30:10 -06:00
time_internal_test.go add logic to restrict time range to available views 2019-02-01 15:30:10 -06:00
translate.go prevent deadlock in replication logic on reopening a store 2019-01-18 13:24:24 -06:00
translate_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
uri.go Require a valid port that isn't greater than 65,535 2018-08-16 21:05:41 -04:00
uri_internal_test.go Require a valid port that isn't greater than 65,535 2018-08-16 21:05:41 -04:00
utils_internal_test.go make sure more tests and benchmarks can have their temp dir set by flag 2019-01-21 16:23:27 -06:00
version.go Fix linter issues: gochecknoinits 2018-07-18 11:58:27 -05:00
view.go view.go: deal with races in fragment creation 2019-02-21 16:43:19 -06:00
view_internal_test.go view.go: deal with races in fragment creation 2019-02-21 16:43:19 -06:00

CircleCI GoDoc Go Report Card license CLA Assistant GitHub release

An open source, distributed bitmap index.

Want to contribute? One of the easiest ways is to tell us how you're using (or want to use) Pilosa. We learn from every discussion!

Docs

See our Documentation for information about installing and working with Pilosa.

Getting Started

  1. Install Pilosa.

  2. Start Pilosa with the default configuration:

    pilosa server
    

    and verify that it's running:

    curl localhost:10101/nodes
    
  3. Follow along with the Sample Project to get a better understanding of Pilosa's capabilities.

Data Model

Check out how the Pilosa Data Model works.

Query Language

You can interact with Pilosa directly in the console using the Pilosa Query Language (PQL).

Client Libraries

There are supported libraries for the following languages:

Licenses

The core Pilosa code base and all default builds (referred to as Pilosa Community Edition) are licensed completely under the Apache License, Version 2.0. If you build Pilosa with the enterprise build tag (Pilosa Enterprise Edition), then that build will include features licensed under the GNU Affero General Public License (AGPL). Enterprise code is located entirely in the github.com/pilosa/pilosa/enterprise directory. See github.com/pilosa/pilosa/NOTICE and github.com/pilosa/pilosa/LICENSE for more information about Pilosa licenses.

Get Support

There are several channels available for you to reach out to us for support.

Contributing

Pilosa is an open source project. Please see our Contributing Guide for information about how to get involved.