pilosa

Pilosa is a bitmap index database.

Getting Started

Pilosa requires Go 1.7 or greater.

You can download the source by running go get:

$ go get github.com/pilosa/pilosa

Now you can install the pilosa binary:

$ go install github.com/pilosa/pilosa/cmd/...

Now run pilosa with the default configuration:

pilosa

Configuration

You can specify a configuration by setting the -config flag when running pilosa.

pilosa -config custom-config-file.cfg

The config file uses the TOML configuration file format, and should look like:

data-dir = "/tmp/pil0"
host = "127.0.0.1:15000"

[cluster]
replicas = 2

[[cluster.node]]
host = "127.0.0.1:15000"

[[cluster.node]]
host = "127.0.0.1:15001"

The first two configuration options will be unique to each node in the cluster:

data-dir: directory in which data is stored to disk

host: IP and port of the pilosa node

The remaining configuration options should be the same on every node in the cluster.

replicas: the number of replicas within the cluster

[[cluster.node]]: specifies each node within the cluster

Usage

You can interact with Pilosa via HTTP requests to the host:port on which you have Pilosa running. The following examples illustrate how to do this using curl with a Pilosa cluster running on 127.0.0.1 port 15000.

Return the version of Pilosa:

$ curl "http://127.0.0.1:15000/version"

Return a list of all databases and frames in the index:

$ curl "http://127.0.0.1:15000/schema"

Queries

Queries to Pilosa require sending a POST request where the query itself is sent as POST data. You specify the database on which to perform the query with a URL argument db=database-name.

A query sent to database exampleDB will have the following format:

$ curl -X POST "http://127.0.0.1:15000/query?db=exampleDB" -d 'Query()'

The Query() object referenced above should be made up of one or more of the query types listed below. So for example, a SetBit() query would look like this:

$ curl -X POST "http://127.0.0.1:15000/query?db=exampleDB" -d 'SetBit(id=10, frame="foo", profileID=1)'

Query results have the format {"results":[]}, where results is a list of results for each Query(). This means that you can provide multiple Query() objects with each HTTP request and results will contain the results of all of the queries.

$ curl -X POST "http://127.0.0.1:15000/query?db=exampleDB" -d 'Query() Query() Query()'

---

SetBit()

SetBit(id=10, frame="foo", profileID=1)

A return value of {"results":[true]} indicates that the bit was toggled from 0 to 1. A return value of {"results":[false]} indicates that the bit was already set to 1 and therefore nothing changed.

---

ClearBit()

ClearBit(id=10, frame="foo", profileID=1)

A return value of {"results":[true]} indicates that the bit was toggled from 1 to 0. A return value of {"results":[false]} indicates that the bit was already set to 0 and therefore nothing changed.

---

SetBitmapAttrs()

SetBitmapAttrs(id=10, frame="foo", category=123, color="blue", happy=true)

Returns {"results":[null]}

---

Bitmap()

Bitmap(id=10, frame="foo")

Returns {"results":[{"attrs":{"category":123,"color":"blue","happy":true},"bits":[1,2]}]} where attrs are the attributes set using SetBitmapAttrs() and bits are the bits set using SetBit().

---

Union()

Union(Bitmap(id=10, frame="foo"), Bitmap(id=20, frame="foo")))

Returns a result set similar to that of a Bitmap() query, only the attrs dictionary will be empty: {"results":[{"attrs":{},"bits":[1,2]}]}. Note that a Union() query can be nested within other queries anywhere that you would otherwise provide a Bitmap().

---

Intersect()

Intersect(Bitmap(id=10, frame="foo"), Bitmap(id=20, frame="foo")))

Returns a result set similar to that of a Bitmap() query, only the attrs dictionary will be empty: {"results":[{"attrs":{},"bits":[1]}]}. Note that an Intersect() query can be nested within other queries anywhere that you would otherwise provide a Bitmap().

---

Difference()

Difference(Bitmap(id=10, frame="foo"), Bitmap(id=20, frame="foo")))

Difference() represents all of the bits that are set in the first Bitmap() but are not set in the second Bitmap(). It returns a result set similar to that of a Bitmap() query, only the attrs dictionary will be empty: {"results":[{"attrs":{},"bits":[2]}]}. Note that a Difference() query can be nested within other queries anywhere that you would otherwise provide a Bitmap().

---

Count()

Count(Bitmap(id=10, frame="foo"))

Returns the count of the number of bits set in Bitmap(): {"results":[28]}

---

Range()

Range(id=10, frame="foo", start="1970-01-01T00:00", end="2000-01-02T03:04")

---

TopN()

TopN(frame="bar", n=20)

Returns the top 20 Bitmaps from frame bar.

TopN(Bitmap(id=10, frame="foo"), frame="bar", n=20)

Returns the top 20 Bitmaps from bar sorted by the count of bits in the intersection with Bitmap(id=10).

TopN(Bitmap(id=10, frame="foo"), frame="bar", n=20, field="category", [81,82])

Returns the top 20 Bitmaps from barin attribute category with values 81 or 82 sorted by the count of bits in the intersection with Bitmap(id=10).

Development

Updating dependencies

To update dependencies, you'll need to install Glide.

Then add the new dependencies in your project:

$ glide get github.com/foo/bar

Protobuf

If you update protobuf (pilosa/internal/internal.proto), then you need to run go generate

$ go generate

Version

In order to set the version number, compile Pilosa with the following argument:

$ go install --ldflags="-X main.Version=1.0.0"

Benchmarks

The usual interface for running benchmarks is:

pilosactl bspawn benchmark-file.json

There are several example json config files in cmd/pilosactl

The bspawn command calls other pilosactl subcommands such as create and bagent to perform the benchmarks. These commands can also be used directly if one wishes e.g. to just create a cluster, or locally run a benchmarks against an existing cluster. Pass the -help flag to either to get more information about its usage.

Configuration Format

bspawn uses a json config format that has 5 top level items - an example is below.

{
    "CreatorArgs": ["-type", "local", "-serverN", "1", "-replicaN", "1"],
    "PilosaHosts": ["localhost:19327"],
    "AgentHosts": ["agent.example.com"],
    "Benchmarks": [
        {
            "Num": 1,
            "Args": ["import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width"]
        },
        {
            "Num": 1,
            "Args": ["import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width", "-random-bitmap-order", "-db", "randoload"]
        }
    ]
}

CreatorArgs

Specifies the pilosa cluster that should be created to run benchmarks against. For more information about the configuration for this option, see the pilosactl create -help

PilosaHosts

If PilosaHosts is set, CreatorArgs will be ignored, and an existing pilosa cluster specified by the list of hosts will be used.

AgentHosts

If AgentHosts is not empty, the agents specified here are used; if it is empty, agents will be run locally.

Benchmarks

Benchmarks is where the actual benchmarks to run are specified - each contains a Num which is the number of agents that should run that benchmark, and Args which specifies the benchmark. The benchmarks in the Benchmarks list will be run concurrently. For more information about Args, see the pilosactl bagent -help.

For documentation on a specific bagent subcommand do pilosactl bagent <subcommand> -help