scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-06 19:07:50 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'master' into performance-enhancements

Browse source
This commit is contained in:
Travis 2017-03-07 15:07:21 -06:00
commit 75e12740d3
81 changed files with 2366 additions and 5003 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.travis.yml
View file

@ -2,6 +2,12 @@ language: go
go:
- 1.7.x
- master
addons:
apt:
sources:
- sourceline: 'ppa:masterminds/glide'
packages:
- glide
script: make test
notifications:
slack:

14
Dockerfile
View file

@ -7,8 +7,20 @@ VOLUME /data
RUN echo 'data-dir = "/data"' > /config
RUN git clone --depth 1 https://github.com/Masterminds/glide.git /go/src/github.com/Masterminds/glide \
&& cd /go/src/github.com/Masterminds/glide \
&& git fetch --tags --depth 1 \
&& git checkout tags/v0.12.3 -b build \
&& make build \
&& mv ./glide /go/bin \
&& cd / \
&& rm -r /go/src/github.com/Masterminds/glide
COPY . /go/src/github.com/pilosa/pilosa
RUN CGO_ENABLED=0 go install -a github.com/pilosa/pilosa/cmd/pilosa
RUN cd /go/src/github.com/pilosa/pilosa \
&& make vendor \
&& CGO_ENABLED=0 go install -a github.com/pilosa/pilosa/cmd/pilosa
ENTRYPOINT ["/go/bin/pilosa"]
CMD ["-config", "/config"]

2
Makefile
View file

@ -43,5 +43,5 @@ install: vendor
go install $(LDFLAGS) $(FLAGS) $(CLONE_URL)/cmd/pilosa
go install $(LDFLAGS) $(FLAGS) $(CLONE_URL)/cmd/pilosactl
docker: vendor
docker:
docker build -t pilosa:latest .

107
README.md
View file

@ -2,6 +2,7 @@
Pilosa is a bitmap index database.
[![Build Status](https://travis-ci.com/pilosa/pilosa.svg?token=Peb4jvQ3kLbjUEhpU5aR&branch=master)](https://travis-ci.com/pilosa/pilosa)
## Getting Started
@ -22,18 +23,16 @@ $ go install github.com/pilosa/pilosa/cmd/...
Now run a single pilosa node with the default configuration:
```sh
pilosa
pilosa server
```
If you would like to quickly create a multi-node pilosa cluster, see the `pilosactl create` documentation.
## Configuration
You can specify a configuration by setting the `-config` flag when running `pilosa`.
```sh
pilosa -config custom-config-file.cfg
pilosa server --config custom-config-file.cfg
```
The config file uses the [TOML](https://github.com/toml-lang/toml) configuration file format,
@ -53,6 +52,12 @@ host = "127.0.0.1:15000"
host = "127.0.0.1:15001"
```
You can generate a template config file with default values with:
```sh
pilosa config
```
The first two configuration options will be unique to each node in the cluster:
`data-dir`: directory in which data is stored to disk
@ -65,6 +70,20 @@ The remaining configuration options should be the same on every node in the clus
`[[cluster.node]]`: specifies each node within the cluster
## Docker
You can create a Pilosa container using `make docker` or equivalently:
```
docker build -t pilosa:latest .
```
You can run a temporary container using:
```
docker run -it --rm --name pilosa -p 15000:15000 pilosa:latest
```
When you click `Ctrl+C` to stop the container, the container and the data in the container will be erased. You can leave out `--rm` flag to keep the data in the container. See [Docker documentation](https://docs.docker.com) for other options.
## Usage
You can interact with Pilosa via HTTP requests to the host:port on which you have Pilosa running.
@ -241,83 +260,3 @@ $ go install --ldflags="-X main.Version=1.0.0"
```
[Glide]: http://glide.sh/
## Pilosactl
Pilosactl contains a suite of tools for interacting with pilosa. Run `pilosactl` for an overview of commands, and `pilosactl <command> -h` for specific information on that command.
### Create
`pilosactl create` is used to create pilosa clusters. It has a number of options for controlling how the cluster is configured, what hosts it is on, and even the ability to build the pilosa binary locally and copy it to each cluster node automatically. To start pilosa on remote hosts, you only need `ssh` access to those hosts. See `pilosactl create -h` for a full list of options.
Examples:
Create a 5 node cluster locally (using 5 different ports), with a replication factor of 2.
```
pilosactl create \
-serverN 5 \
-replicaN 2
```
Create a cluster on 3 remote hosts - all logs will come to local stderr, pilosa binary must be available on remote hosts. The ssh user on the remote hosts needs to be the same as your local user. Otherwise use the `ssh-user` option.
```
pilosactl create \
-hosts="node1.example.com:15000,node2.example.com:15000,node3.example.com:15000"
```
Create a cluster on 3 remote hosts running OSX, but build the binary locally and copy it up. Stream the stderr of each node to a separate local log file.
```
pilosactl create \
-hosts="mac1.example.com:15000,mac2.example.com:15000,mac3.example.com:15000" \
-copy-binary \
-goos=darwin \
-goarch=amd64 \
-log-file-prefix=clusterlogs
```
### Bagent
`pilosactl bagent` is what you want if you just want to run a simple benchmark against an existing cluster. Running it with no arguments will print some help, including the set of subcommands that it may be passed. Calling a subcommand with `-h'` will print the options for that subcommand. The `agent-num` flag can be passed an integer which can change the behavior the benchmarks that are run. This is useful when multiple invocations of the same benchmark are made by the `bspawn` command - they can each (for example) set different bits even though they all have the same arguments.
E.G.
```
pilosactl bagent \
-hosts="localhost:15000,localhost:15001" \
import -h
```
Multiple subcommands and their arguments may be concatenated at the command line and they will be run serially. This is useful (i.e.) for importing a bunch of data, and then executing queries against it.
This will generate and import a bunch of data, and then execute random queries against it.
```
pilosactl bagent \
-hosts="localhost:15000,localhost:15001" \
import -max-bits-per-map=10000 \
random-query -iterations 100
```
### Bspawn
`pilosactl bspawn` allows you to automate the creation of clusters and the running of complex benchmarks which span multiple benchmark agents against them. It has a number of options which are described by `pilosactl bspawn` with no arguments, and also takes a config file which describes the Benchmark itself - this file is described below.
#### Configuration Format
The configuration file is a json object with the top level key `benchmarks`. This contains a list of objects each of which represents a `bagent` command (the `args` key) that will be run some number of times concurrently (the `num` key), and a `name` which should describe the overall effect that command. An example is below.
```json
{
"benchmarks": [
{
"num": 3,
"name": "set-diags",
"args": ["diagonal-set-bits", "-iterations", "30000", "-client-type", "round_robin"]
},
{
"num": 2,
"name": "rand-plus-zipf",
"args": ["random-set-bits", "-iterations", "20000", "zipf", "-iterations", "100"]
}
]
}
```
All of the benchmarks, and agents are run concurrently. Each agent will be passed an `agent-num` which can modify the behavior in a way that is benchmark specific. See the documentation for each benchmark to see how `agent-num` changes its behavior.

30
bench/LICENSE
View file

@ -1,30 +0,0 @@
The following applies to the files errgroup.go and errgroup_test.go
in the same directory this file resides:
Copyright (c) 2009 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

41
bench/bench.go
View file

@ -1,41 +0,0 @@
package bench
import "context"
// Benchmark is an interface to guide the creation of new pilosa benchmarks or
// benchmark components. It defines 2 methods, Init, and Run. These are separate
// methods so that benchmark running code can time only the running of the
// benchmark, and not any setup.
type Benchmark interface {
// Init takes a list of hosts and an agent number. It is generally expected
// to set up a connection to pilosa using whatever client it chooses. The
// agentNum should be used to parameterize the benchmark's configuration if
// it is being run simultaneously on multiple "agents". E.G. the agentNum
// might be used to make a random seed different for each agent, or have
// each agent set a different set of bits. Init's doc string should document
// how the agentNum affects it. Every benchmark should have a 'Name' field
// set by init, which appears when the benchmark is marshalled to json as
// "name".
Init(hosts []string, agentNum int) error
// Run runs the benchmark. The return value of Run is kept generic so that
// any relevant statistics or metrics that may be specific to the benchmark
// in question can be reported. TODO guidelines for what gets included in
// results and what will get added by other stuff. Run does not need to
// report total run time in `results`, as that will be added by calling
// code.
Run(ctx context.Context) map[string]interface{}
}
// Command extends Benchmark by adding methods for configuring via command line flags and returning usage information.
type Command interface {
Benchmark
// ConsumeFlags sets and parses flags, and then returns flagSet.Args(). This
// is so that multiple benchmarks can be specified at the command line.
ConsumeFlags(args []string) ([]string, error)
// Usage returns information on how to use this benchmark. The usage string
// should explain how the agent num affects the benchmark's operation.
Usage() string
}

67
bench/client.go
View file

@ -1,67 +0,0 @@
package bench
import (
"context"
"errors"
"fmt"
"github.com/pilosa/pilosa"
)
func firstHostClient(hosts []string) (*pilosa.Client, error) {
client, err := pilosa.NewClient(hosts[0])
if err != nil {
return nil, err
}
return client, nil
}
func roundRobinClient(hosts []string, agentNum int) (*pilosa.Client, error) {
clientNum := agentNum % len(hosts)
return firstHostClient(hosts[clientNum:])
}
// HasClient provides a reusable component for Benchmark implementations which
// provides the Init method, a ClientType argument and a cli internal variable.
type HasClient struct {
client *pilosa.Client
ClientType string `json:"client-type"`
ContentType string `json:"content-type"`
}
// Init for HasClient looks at the ClientType field and creates a pilosa client
// either using the first host in the list of hosts or based on the agent
// number mod len(hosts)
func (h *HasClient) Init(hosts []string, agentNum int) error {
var err error
switch h.ClientType {
case "single":
h.client, err = firstHostClient(hosts)
case "round_robin":
h.client, err = roundRobinClient(hosts, agentNum)
default:
err = fmt.Errorf("Unsupported ClientType: %v", h.ClientType)
}
if err != nil {
return err
}
switch h.ContentType {
case "protobuf":
return nil
case "pql":
return nil
default:
return fmt.Errorf("Unsupported ContentType: %v", h.ContentType)
}
}
func (h *HasClient) ExecuteQuery(contentType, db, query string, ctx context.Context) (interface{}, error) {
if contentType == "protobuf" {
return h.client.ExecuteQuery(ctx, db, query, true)
} else if contentType == "pql" {
return h.client.ExecutePQL(ctx, db, query)
} else {
return nil, errors.New("unsupport content type")
}
}

105
bench/diagonal.go
View file

@ -1,105 +0,0 @@
package bench
import (
"fmt"
"flag"
"io/ioutil"
"context"
"time"
)
// DiagonalSetBits sets bits with increasing profile id and bitmap id.
type DiagonalSetBits struct {
HasClient
Name string `json:"name"`
BaseBitmapID int `json:"base-bitmap-id"`
BaseProfileID int `json:"base-profile-id"`
Iterations int `json:"iterations"`
DB string `json:"db"`
}
// Init sets up the pilosa client and modifies the configured values based on
// the agent num.
func (b *DiagonalSetBits) Init(hosts []string, agentNum int) error {
b.Name = "diagonal-set-bits"
b.BaseBitmapID = b.BaseBitmapID + (agentNum * b.Iterations)
b.BaseProfileID = b.BaseProfileID + (agentNum * b.Iterations)
return b.HasClient.Init(hosts, agentNum)
}
// Usage returns the usage message to be printed.
func (b *DiagonalSetBits) Usage() string {
return `
diagonal-set-bits sets bits with increasing profile id and bitmap id.
Agent num offsets both the base profile id and base bitmap id by the number of
iterations, so that only bits on the main diagonal are set, and agents don't
overlap at all.
Usage: diagonal-set-bits [arguments]
The following arguments are available:
-base-bitmap-id int
bits being set will all be greater than BaseBitmapID
-base-profile-id int
profile id num to start from
-iterations int
number of bits to set
-db string
pilosa db to use
-client-type string
Can be 'single' (all agents hitting one host) or 'round_robin'
-content-type string
protobuf or pql
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *DiagonalSetBits) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("DiagonalSetBits", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.IntVar(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.IntVar(&b.BaseProfileID, "base-profile-id", 0, "")
fs.IntVar(&b.Iterations, "iterations", 100, "")
fs.StringVar(&b.DB, "db", "benchdb", "")
fs.StringVar(&b.ClientType, "client-type", "single", "")
fs.StringVar(&b.ContentType, "content-type", "protobuf", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
return fs.Args(), nil
}
// Run runs the DiagonalSetBits benchmark
func (b *DiagonalSetBits) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
if b.client == nil {
results["error"] = fmt.Errorf("No client set for DiagonalSetBits")
return results
}
s := NewStats()
var start time.Time
for n := 0; n < b.Iterations; n++ {
query := fmt.Sprintf("SetBit(%d, 'frame.n', %d)", b.BaseBitmapID+n, b.BaseProfileID+n)
start = time.Now()
_, err := b.client.ExecuteQuery(ctx, b.DB, query, true)
if err != nil {
results["error"] = err
return results
}
s.Add(time.Now().Sub(start))
}
AddToResults(s, results)
return results
}

39
bench/doc.go
View file

@ -1,39 +0,0 @@
// bench contains benchmarks and common utilities useful to benchmarks
//
// In order to write new benchmarks, one must satisfy the Benchmark and Command
// interfaces in bench.go. In order to use the benchmark from pilosactl, it
// needs to be wired in in two places. The first is BagentCommand.ParseFlags,
// where a case statement needs to be added, and the second is just adding the
// benchmark to the BagentCommand.Usage usage string.
//
// When writing a new benchmark, there are a few things to keep in mind other
// than just implementing the interface:
//
// 1. The benchmark should modify its own configuration in its Init method based
// on the agentNum it is given. How it modifies is specific to the benchmark,
// but the idea is that it should make sense to call the benchmark with the same
// configuration, but multiple different agent numbers, and it should do useful
// work each time (i.e. not just setting the same bits, or running the same
// queries).
//
// 2. The Init method should do everything that needs to be done to get the
// benchmark to a runnable state - all code in run should be the stuff that we
// actually want to time.
//
// 3. The Run method does not need to report the total runtime - that is collected
// by calling code.
//
// 4. Usage should follow the format in other benchmarks, and explain how the
// benchmark uses agentNum to modify its behavior
//
//
// Files:
//
// 1. client.go contains pilosa client code which is shared by many benchmarks
//
// 2. errgroup.go contains the ErrGroup implementation copied from golang.org/x/
// so as not to pull in a bunch of useless deps.
//
// 3. stats.go contains useful code for gathering stats about a series of timed
// operations.
package bench

67
bench/errgroup.go
View file

@ -1,67 +0,0 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package errgroup provides synchronization, error propagation, and Context
// cancelation for groups of goroutines working on subtasks of a common task.
package bench
import (
"sync"
"context"
)
// An ErrGroup is a collection of goroutines working on subtasks that are part of
// the same overall task.
//
// A zero ErrGroup is valid and does not cancel on error.
type ErrGroup struct {
cancel func()
wg sync.WaitGroup
errOnce sync.Once
err error
}
// WithContext returns a new ErrGroup and an associated Context derived from ctx.
//
// The derived Context is canceled the first time a function passed to Go
// returns a non-nil error or the first time Wait returns, whichever occurs
// first.
func WithContext(ctx context.Context) (*ErrGroup, context.Context) {
ctx, cancel := context.WithCancel(ctx)
return &ErrGroup{cancel: cancel}, ctx
}
// Wait blocks until all function calls from the Go method have returned, then
// returns the first non-nil error (if any) from them.
func (g *ErrGroup) Wait() error {
g.wg.Wait()
if g.cancel != nil {
g.cancel()
}
return g.err
}
// Go calls the given function in a new goroutine.
//
// The first call to return a non-nil error cancels the group; its error will be
// returned by Wait.
func (g *ErrGroup) Go(f func() error) {
g.wg.Add(1)
go func() {
defer g.wg.Done()
if err := f(); err != nil {
g.errOnce.Do(func() {
g.err = err
if g.cancel != nil {
g.cancel()
}
})
}
}()
}

176
bench/errgroup_test.go
View file

@ -1,176 +0,0 @@
// Copyright 2016 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package bench_test
import (
"errors"
"fmt"
"net/http"
"os"
"testing"
"context"
"github.com/pilosa/pilosa/bench"
)
var (
Web = fakeSearch("web")
Image = fakeSearch("image")
Video = fakeSearch("video")
)
type Result string
type Search func(ctx context.Context, query string) (Result, error)
func fakeSearch(kind string) Search {
return func(_ context.Context, query string) (Result, error) {
return Result(fmt.Sprintf("%s result for %q", kind, query)), nil
}
}
// JustErrors illustrates the use of a ErrGroup in place of a sync.WaitGroup to
// simplify goroutine counting and error handling. This example is derived from
// the sync.WaitGroup example at https://golang.org/pkg/sync/#example_WaitGroup.
func ExampleGroup_justErrors() {
var g bench.ErrGroup
var urls = []string{
"http://www.golang.org/",
"http://www.google.com/",
"http://www.somestupidname.com/",
}
for _, url := range urls {
// Launch a goroutine to fetch the URL.
url := url // https://golang.org/doc/faq#closures_and_goroutines
g.Go(func() error {
// Fetch the URL.
resp, err := http.Get(url)
if err == nil {
resp.Body.Close()
}
return err
})
}
// Wait for all HTTP fetches to complete.
if err := g.Wait(); err == nil {
fmt.Println("Successfully fetched all URLs.")
}
}
// Parallel illustrates the use of a ErrGroup for synchronizing a simple parallel
// task: the "Google Search 2.0" function from
// https://talks.golang.org/2012/concurrency.slide#46, augmented with a Context
// and error-handling.
func ExampleGroup_parallel() {
Google := func(ctx context.Context, query string) ([]Result, error) {
g, ctx := bench.WithContext(ctx)
searches := []Search{Web, Image, Video}
results := make([]Result, len(searches))
for i, search := range searches {
i, search := i, search // https://golang.org/doc/faq#closures_and_goroutines
g.Go(func() error {
result, err := search(ctx, query)
if err == nil {
results[i] = result
}
return err
})
}
if err := g.Wait(); err != nil {
return nil, err
}
return results, nil
}
results, err := Google(context.Background(), "golang")
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
for _, result := range results {
fmt.Println(result)
}
// Output:
// web result for "golang"
// image result for "golang"
// video result for "golang"
}
func TestZeroGroup(t *testing.T) {
err1 := errors.New("errgroup_test: 1")
err2 := errors.New("errgroup_test: 2")
cases := []struct {
errs []error
}{
{errs: []error{}},
{errs: []error{nil}},
{errs: []error{err1}},
{errs: []error{err1, nil}},
{errs: []error{err1, nil, err2}},
}
for _, tc := range cases {
var g bench.ErrGroup
var firstErr error
for i, err := range tc.errs {
err := err
g.Go(func() error { return err })
if firstErr == nil && err != nil {
firstErr = err
}
if gErr := g.Wait(); gErr != firstErr {
t.Errorf("after %T.Go(func() error { return err }) for err in %v\n"+
"g.Wait() = %v; want %v",
g, tc.errs[:i+1], err, firstErr)
}
}
}
}
func TestWithContext(t *testing.T) {
errDoom := errors.New("group_test: doomed")
cases := []struct {
errs []error
want error
}{
{want: nil},
{errs: []error{nil}, want: nil},
{errs: []error{errDoom}, want: errDoom},
{errs: []error{errDoom, nil}, want: errDoom},
}
for _, tc := range cases {
g, ctx := bench.WithContext(context.Background())
for _, err := range tc.errs {
err := err
g.Go(func() error { return err })
}
if err := g.Wait(); err != tc.want {
t.Errorf("after %T.Go(func() error { return err }) for err in %v\n"+
"g.Wait() = %v; want %v",
g, tc.errs, err, tc.want)
}
canceled := false
select {
case <-ctx.Done():
canceled = true
default:
}
if !canceled {
t.Errorf("after %T.Go(func() error { return err }) for err in %v\n"+
"ctx.Done() was not closed",
g, tc.errs)
}
}
}

203
bench/import.go
View file

@ -1,203 +0,0 @@
package bench
import (
"context"
"flag"
"fmt"
"io"
"io/ioutil"
"math/rand"
"sort"
"github.com/pilosa/pilosa/pilosactl"
)
// NewImport returns an Import Benchmark which pilosactl importer configured.
func NewImport(stdin io.Reader, stdout, stderr io.Writer) *Import {
return &Import{
ImportCommand: pilosactl.NewImportCommand(stdin, stdout, stderr),
}
}
// Import sets bits with increasing profile id and bitmap id.
type Import struct {
Name string `json:"name"`
BaseBitmapID int64 `json:"base-bitmap-id"`
MaxBitmapID int64 `json:"max-bitmap-id"`
BaseProfileID int64 `json:"base-profile-id"`
MaxProfileID int64 `json:"max-profile-id"`
RandomBitmapOrder bool `json:"random-bitmap-order"`
MinBitsPerMap int64 `json:"min-bits-per-map"`
MaxBitsPerMap int64 `json:"max-bits-per-map"`
AgentControls string `json:"agent-controls"`
Seed int64 `json:"seed"`
numbits int
*pilosactl.ImportCommand
}
// Usage returns the usage message to be printed.
func (b *Import) Usage() string {
return `
import generates an import file and imports using pilosa's bulk import interface
Agent num can have various effects - see -agent-controls flag.
Usage: import [arguments]
The following arguments are available:
-base-bitmap-id int
bits being set will all be greater than this
-max-bitmap-id int
bits being set will all be less than this
-base-profile-id int
profile id num to start from
-max-profile-id int
maximum profile id to generate
-random-bitmap-order
if this option is set, the import file will not be sorted by bitmap id
-min-bits-per-map int
minimum number of bits set per bitmap
-max-bits-per-map int
maximum number of bits set per bitmap
-agent-controls string
can be 'height', 'width', or empty (TODO or square?)- increasing
number of agents modulates bitmap id range, profile id range,
or just sets more bits in the same range.
-seed int
seed for RNG
-db string
pilosa db to use
-frame string
frame to import into
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *Import) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("Import", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.Int64Var(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.Int64Var(&b.MaxBitmapID, "max-bitmap-id", 1000, "")
fs.Int64Var(&b.BaseProfileID, "base-profile-id", 0, "")
fs.Int64Var(&b.MaxProfileID, "max-profile-id", 1000, "")
fs.BoolVar(&b.RandomBitmapOrder, "random-bitmap-order", false, "")
fs.Int64Var(&b.MinBitsPerMap, "min-bits-per-map", 0, "")
fs.Int64Var(&b.MaxBitsPerMap, "max-bits-per-map", 10, "")
fs.StringVar(&b.AgentControls, "agent-controls", "", "")
fs.Int64Var(&b.Seed, "seed", 0, "")
fs.StringVar(&b.Database, "db", "benchdb", "")
fs.StringVar(&b.Frame, "frame", "testframe", "")
fs.IntVar(&b.BufferSize, "buffer-size", 10000000, "")
if err := fs.Parse(args); err != nil {
return nil, err
}
return fs.Args(), nil
}
// Init generates import data based on the agent num and fields of 'b'.
func (b *Import) Init(hosts []string, agentNum int) error {
if len(hosts) == 0 {
return fmt.Errorf("Need at least one host")
}
b.Name = "import"
b.Host = hosts[0]
// generate csv data
b.Seed = b.Seed + int64(agentNum)
switch b.AgentControls {
case "height":
numBitmapIDs := (b.MaxBitmapID - b.BaseBitmapID)
b.BaseBitmapID = b.BaseBitmapID + (numBitmapIDs * int64(agentNum))
b.MaxBitmapID = b.BaseBitmapID + numBitmapIDs
case "width":
numProfileIDs := (b.MaxProfileID - b.BaseProfileID)
b.BaseProfileID = b.BaseProfileID + (numProfileIDs * int64(agentNum))
b.MaxProfileID = b.BaseProfileID + numProfileIDs
case "":
break
default:
return fmt.Errorf("agent-controls: '%v' is not supported", b.AgentControls)
}
f, err := ioutil.TempFile("", "")
if err != nil {
return err
}
// set b.Paths)
num := GenerateImportCSV(f, b.BaseBitmapID, b.MaxBitmapID, b.BaseProfileID, b.MaxProfileID,
b.MinBitsPerMap, b.MaxBitsPerMap, b.Seed, b.RandomBitmapOrder)
b.numbits = num
// set b.Paths
b.Paths = []string{f.Name()}
return f.Close()
}
// Run runs the Import benchmark
func (b *Import) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
results["numbits"] = b.numbits
results["db"] = b.Database
err := b.ImportCommand.Run(ctx)
if err != nil {
results["error"] = err.Error()
}
return results
}
// Int64Slice is a sortable slice of 64 bit signed ints
type Int64Slice []int64
func (s Int64Slice) Len() int { return len(s) }
func (s Int64Slice) Less(i, j int) bool { return s[i] < s[j] }
func (s Int64Slice) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
// GenerateImportCSV writes a generated csv to 'w' which is in the form pilosactl expects for imports.
func GenerateImportCSV(w io.Writer, baseBitmapID, maxBitmapID, baseProfileID, maxProfileID, minBitsPerMap, maxBitsPerMap, seed int64, randomOrder bool) int {
src := rand.NewSource(seed)
rng := rand.New(src)
var bitmapIDs []int
if randomOrder {
bitmapIDs = rng.Perm(int(maxBitmapID - baseBitmapID))
}
numrows := 0
profileIDs := make(Int64Slice, maxBitsPerMap)
for i := baseBitmapID; i < maxBitmapID; i++ {
var bitmapID int64
if randomOrder {
bitmapID = int64(bitmapIDs[i-baseBitmapID])
} else {
bitmapID = int64(i)
}
numBitsToSet := rng.Int63n(maxBitsPerMap-minBitsPerMap) + minBitsPerMap
numrows += int(numBitsToSet)
for j := int64(0); j < numBitsToSet; j++ {
profileIDs[j] = rng.Int63n(maxProfileID-baseProfileID) + baseProfileID
}
profIDs := profileIDs[:numBitsToSet]
if !randomOrder {
sort.Sort(profIDs)
}
for j := int64(0); j < numBitsToSet; j++ {
fmt.Fprintf(w, "%d,%d\n", bitmapID, profIDs[j])
}
}
return numrows
}

136
bench/import_test.go
View file

@ -1,136 +0,0 @@
package bench_test
import (
"bytes"
"testing"
"io/ioutil"
"os"
"github.com/pilosa/pilosa/bench"
)
func TestImportInit(t *testing.T) {
imp := bench.NewImport(os.Stdin, os.Stdout, os.Stderr)
imp.BaseBitmapID = 0
imp.MaxBitmapID = 10
imp.BaseProfileID = 0
imp.MaxProfileID = 10
imp.RandomBitmapOrder = false
imp.MinBitsPerMap = 2
imp.MaxBitsPerMap = 3
imp.AgentControls = "width"
imp.Seed = 0
imp.Init([]string{"blah"}, 2)
f, err := os.Open(imp.Paths[0])
if err != nil {
t.Fatalf("Couldn't open file: %v, err: %v", imp.Paths[0], err)
}
bytes, err := ioutil.ReadAll(f)
if err != nil {
t.Fatalf("error reading file: %v", err)
}
expected := `
0,21
0,22
1,20
1,22
2,22
2,26
3,21
3,23
4,21
4,22
5,20
5,28
6,23
6,27
7,20
7,20
8,23
8,29
9,23
9,29
`[1:]
if string(bytes) != expected {
t.Fatalf("unexpected result: %v", string(bytes))
}
}
func TestGenerateImportCSVNonRand(t *testing.T) {
b := bytes.NewBuffer(make([]byte, 0))
bench.GenerateImportCSV(b, 0, 10, 20, 30, 2, 3, 2, false)
bytes, err := ioutil.ReadAll(b)
if err != nil {
t.Fatalf("Error reading buffer: %v", err)
}
expected := `
0,21
0,22
1,20
1,22
2,22
2,26
3,21
3,23
4,21
4,22
5,20
5,28
6,23
6,27
7,20
7,20
8,23
8,29
9,23
9,29
`[1:]
if string(bytes) != expected {
t.Fatalf("unexpected value for generated csv: \n%v", string(bytes))
}
}
func TestGenerateImportCSVRand(t *testing.T) {
b := bytes.NewBuffer(make([]byte, 0))
bench.GenerateImportCSV(b, 0, 10, 21, 29, 1, 4, 0, true)
bytes, err := ioutil.ReadAll(b)
if err != nil {
t.Fatalf("Error reading buffer: %v", err)
}
expected := `
8,25
2,23
3,22
3,28
3,28
0,25
0,23
5,26
5,23
7,21
7,23
1,25
6,23
6,27
6,25
9,26
4,22
4,24
`[1:]
if string(bytes) != expected {
t.Fatalf("unexpected value for generated csv: \n%v", string(bytes))
}
}

97
bench/multidb.go
View file

@ -1,97 +0,0 @@
package bench
import (
"context"
"flag"
"fmt"
"io/ioutil"
"strconv"
"time"
)
// MultiDBSetBits sets bits with increasing profile id and bitmap id.
type MultiDBSetBits struct {
HasClient
Name string `json:"name"`
BaseBitmapID int `json:"base-bitmap-id"`
BaseProfileID int `json:"base-profile-id"`
Iterations int `json:"iterations"`
Database string `json:"database"`
}
// Init sets up the db name based on the agentNum and sets up the pilosa client.
func (b *MultiDBSetBits) Init(hosts []string, agentNum int) error {
b.Name = "multi-db-set-bits"
b.Database = b.Database + strconv.Itoa(agentNum)
return b.HasClient.Init(hosts, agentNum)
}
// Usage returns the usage message to be printed.
func (b *MultiDBSetBits) Usage() string {
return `
multi-db-set-bits sets bits with increasing profile id and bitmap id using a different DB for each agent.
Agent num changes the database being written to.
Usage: multi-db-set-bits [arguments]
The following arguments are available:
-base-bitmap-id int
bits being set will all be greater than base-bitmap-id
-base-profile-id int
profile id num to start from
-iterations int
number of bits to set
-client-type string
Can be 'single' (all agents hitting one host) or 'round_robin'
-content-type string
protobuf or pql
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *MultiDBSetBits) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("MultiDBSetBits", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.IntVar(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.IntVar(&b.BaseProfileID, "base-profile-id", 0, "")
fs.IntVar(&b.Iterations, "iterations", 100, "")
fs.StringVar(&b.ClientType, "client-type", "single", "")
fs.StringVar(&b.ContentType, "content-type", "protobuf", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
return fs.Args(), nil
}
// Run runs the MultiDBSetBits benchmark
func (b *MultiDBSetBits) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
if b.client == nil {
results["error"] = fmt.Errorf("No client set for MultiDBSetBits")
return results
}
s := NewStats()
var start time.Time
for n := 0; n < b.Iterations; n++ {
query := fmt.Sprintf("SetBit(%d, 'frame.n', %d)", b.BaseBitmapID+n, b.BaseProfileID+n)
start = time.Now()
_, err := b.client.ExecuteQuery(ctx, b.Database, query, true)
if err != nil {
results["error"] = err
return results
}
s.Add(time.Now().Sub(start))
}
AddToResults(s, results)
return results
}

64
bench/permutations.go
View file

@ -1,64 +0,0 @@
package bench
// PermutationGenerator provides a way to pass integer IDs through a permutation
// map that is pseudorandom but repeatable. This could be done with rand.Perm,
// but that would require storing a [Iterations]int64 array, which we want to avoid
// for large values of Iterations.
// It works by using a Linear Congruence Generator (https://en.wikipedia.org/wiki/Linear_congruential_generator)
// with modulus m = Iterations,
// c = an arbitrary prime,
// a = computed to ensure the full period.
// relevant stackoverflow: http://cs.stackexchange.com/questions/29822/lazily-computing-a-random-permutation-of-the-positive-integers
type PermutationGenerator struct {
a int64
c int64
m int64
}
func NewPermutationGenerator(m int64, seed int64) *PermutationGenerator {
// figure out 'a' and 'c', return PermutationGenerator
a := LCGmultiplierFromModulus(m, seed)
c := int64(22695479)
return &PermutationGenerator{a, c, m}
}
func (p *PermutationGenerator) Next(n int64) int64 {
// run one step of the LCG
return (n*p.a + p.c) % p.m
}
// LCG parameters must satisfy three conditions:
// 1. m and c are relatively prime (satisfied for prime c != m)
// 2. a-1 is divisible by all prime factors of m
// 3. a-1 is divisible by 4 if m is divisible by 4
// Additionally, a seed can be used to select between different permutations
func LCGmultiplierFromModulus(m int64, seed int64) int64 {
factors := primeFactors(m)
product := int64(1)
for p := range factors {
// satisfy condition 2
product *= p
}
if m%4 == 0 {
// satisfy condition 3
product *= 2
}
return product*seed + 1
}
// Returns map of {integerFactor: count, ...}
// This is a naive algorithm that will not work well for large prime n.
func primeFactors(n int64) map[int64]int {
factors := make(map[int64]int)
for i := int64(2); i <= n; i++ {
div, mod := n/i, n%i
for mod == 0 {
factors[i] += 1
n = div
div, mod = n/i, n%i
}
}
return factors
}

41
bench/prettify.go
View file

@ -1,41 +0,0 @@
package bench
import "time"
// wrapper type to force human-readable JSON output
type PrettyDuration time.Duration
// MarshalJSON returns a nicely formatted duration, instead of it just being
// treated like an int.
func (d PrettyDuration) MarshalJSON() ([]byte, error) {
s := time.Duration(d).String()
return []byte("\"" + s + "\""), nil
}
// Recursively replaces elements of ugly types with their pretty wrappers
func Prettify(m map[string]interface{}) map[string]interface{} {
newmap := make(map[string]interface{})
for k, v := range m {
switch v.(type) {
case map[string]interface{}:
newmap[k] = Prettify(v.(map[string]interface{}))
case []time.Duration:
newslice := make([]PrettyDuration, len(v.([]time.Duration)))
slice := v.([]time.Duration)
for n, e := range slice {
newslice[n] = PrettyDuration(e)
}
newmap[k] = newslice
case time.Duration:
newmap[k] = PrettyDuration(v.(time.Duration))
default:
if interv, ok := v.([]map[string]interface{}); ok {
for i, iv := range interv {
interv[i] = Prettify(iv)
}
}
newmap[k] = v
}
}
return newmap
}

93
bench/prettify_test.go
View file

@ -1,93 +0,0 @@
package bench_test
import (
"bytes"
"encoding/json"
"fmt"
"os"
"testing"
"time"
"github.com/pilosa/pilosa/bench"
)
func prettyEncode(data map[string]interface{}) string {
pretty := bench.Prettify(data)
jsonString := new(bytes.Buffer)
enc := json.NewEncoder(jsonString)
enc.SetIndent("", " ")
err := enc.Encode(pretty)
if err != nil {
fmt.Fprintln(os.Stderr, err)
}
return jsonString.String()
}
func TestPrettifyString(t *testing.T) {
res := make(map[string]interface{}, 1)
res["0"] = "foobar"
pretty := prettyEncode(res)
expected := `
{
"0": "foobar"
}
`[1:]
if pretty != expected {
t.Fatalf("Pretty string doesn't match")
}
}
func TestPrettifyInt(t *testing.T) {
res := make(map[string]interface{}, 1)
res["0"] = 234567
pretty := prettyEncode(res)
expected := `
{
"0": 234567
}
`[1:]
if pretty != expected {
t.Fatalf("Pretty int doesn't match")
}
}
func TestPrettifyDuration(t *testing.T) {
res := make(map[string]interface{}, 1)
res["0"] = time.Duration(234567)
pretty := prettyEncode(res)
expected := `
{
"0": "234.567µs"
}
`[1:]
if pretty != expected {
t.Fatalf("Pretty duration doesn't match")
}
}
func TestPrettifyDurationSlice(t *testing.T) {
res := make(map[string]interface{}, 1)
res["0"] = []time.Duration{123, 234567, 34567890}
pretty := prettyEncode(res)
expected := `
{
"0": [
"123ns",
"234.567µs",
"34.56789ms"
]
}
`[1:]
if pretty != expected {
t.Fatalf("Pretty duration slice doesn't match")
}
}

197
bench/query.go
View file

@ -1,197 +0,0 @@
package bench
import (
"math/rand"
"github.com/pilosa/pilosa/pql"
)
// NewQueryGenerator initializes a new QueryGenerator
func NewQueryGenerator(seed int64) *QueryGenerator {
return &QueryGenerator{
IDToFrameFn: func(id uint64) string { return "frame.n" },
R: rand.New(rand.NewSource(seed)),
Frames: []string{"frame.n"},
}
}
// QueryGenerator holds the configuration and state for randomly generating
// queries.
type QueryGenerator struct {
IDToFrameFn func(id uint64) string
R *rand.Rand
Frames []string
}
// Random returns a randomly generated query.
func (q *QueryGenerator) Random(maxN, depth, maxargs int, idmin, idmax uint64) *pql.Call {
// TODO: handle depth==1 or 0
val := q.R.Intn(5)
switch val {
case 0:
return q.RandomTopN(maxN, depth, maxargs, idmin, idmax)
default:
return q.RandomBitmapCall(depth, maxargs, idmin, idmax)
}
}
// RandomTopN returns a randomly generated TopN query.
func (q *QueryGenerator) RandomTopN(maxN, depth, maxargs int, idmin, idmax uint64) *pql.Call {
frameIdx := q.R.Intn(len(q.Frames))
return &pql.Call{
Args: map[string]interface{}{
"frame": q.Frames[frameIdx],
"n": uint64(q.R.Intn(maxN-1) + 1),
},
Children: []*pql.Call{q.RandomBitmapCall(depth, maxargs, idmin, idmax)},
}
}
// RandomBitmapCall returns a randomly generate query which returns a bitmap.
func (q *QueryGenerator) RandomBitmapCall(depth, maxargs int, idmin, idmax uint64) *pql.Call {
if depth <= 1 {
bitmapID := q.R.Int63n(int64(idmax)-int64(idmin)) + int64(idmin)
return Bitmap(uint64(bitmapID), q.IDToFrameFn(uint64(bitmapID)))
}
call := q.R.Intn(4)
if call == 0 {
return q.RandomBitmapCall(1, 0, idmin, idmax)
}
var numargs int
if maxargs <= 2 {
numargs = 2
} else {
numargs = q.R.Intn(maxargs-2) + 2
}
calls := make([]*pql.Call, numargs)
for i := 0; i < numargs; i++ {
calls[i] = q.RandomBitmapCall(depth-1, maxargs, idmin, idmax)
}
switch call {
case 1:
return Difference(calls...)
case 2:
return Intersect(calls...)
case 3:
return Union(calls...)
}
return nil
}
///////////////////////////////////////////////////
// Helpers TODO: move elsewhere
///////////////////////////////////////////////////
func ClearBit(id uint64, frame string, profileID uint64) *pql.Call {
return &pql.Call{
Name: "ClearBit",
Args: map[string]interface{}{
"id": id,
"frame": frame,
"profileID": profileID,
},
}
}
func Count(child *pql.Call) *pql.Call {
return &pql.Call{
Name: "Count",
Children: []*pql.Call{child},
}
}
func Profile(id uint64) *pql.Call {
return &pql.Call{
Name: "Profile",
Args: map[string]interface{}{"id": id},
}
}
func SetBit(id uint64, frame string, profileID uint64) *pql.Call {
return &pql.Call{
Name: "SetBit",
Args: map[string]interface{}{
"id": id,
"frame": frame,
"profileID": profileID,
},
}
}
func SetBitmapAttrs(id uint64, frame string, attrs map[string]interface{}) *pql.Call {
args := copyArgs(attrs)
args["id"] = id
args["profileID"] = frame
return &pql.Call{
Name: "SetBitmapAttrs",
Args: args,
}
}
func SetProfileAttrs(id uint64, attrs map[string]interface{}) *pql.Call {
args := copyArgs(attrs)
args["id"] = id
return &pql.Call{
Name: "SetProfileAttrs",
Args: args,
}
}
func TopN(frame string, n int, src *pql.Call, bmids []uint64, field string, filters []interface{}) *pql.Call {
return &pql.Call{
Name: "TopN",
Children: []*pql.Call{src},
Args: map[string]interface{}{
"frame": frame,
"n": n,
"ids": bmids,
"field": field,
"filters": filters,
},
}
}
func Difference(bms ...*pql.Call) *pql.Call {
// TODO does this need to be limited to two inputs?
return &pql.Call{
Name: "Difference",
Children: bms,
}
}
func Intersect(bms ...*pql.Call) *pql.Call {
return &pql.Call{
Name: "Intersect",
Children: bms,
}
}
func Union(bms ...*pql.Call) *pql.Call {
return &pql.Call{
Name: "Union",
Children: bms,
}
}
func Bitmap(id uint64, frame string) *pql.Call {
return &pql.Call{
Name: "Bitmap",
Args: map[string]interface{}{
"id": id,
"frame": frame,
},
}
}
// copyArgs returns a shallow copy of m.
func copyArgs(m map[string]interface{}) map[string]interface{} {
other := make(map[string]interface{}, len(m))
for k, v := range m {
other[k] = v
}
return other
}

20
bench/query_test.go
View file

@ -1,20 +0,0 @@
package bench_test
import (
"github.com/pilosa/pilosa/bench"
"testing"
)
func TestRandomBitmapCall(t *testing.T) {
qm := bench.NewQueryGenerator(5)
bmc := qm.RandomBitmapCall(4, 3, 0, 1000)
t.Log(bmc.String())
}
func TestRandom(t *testing.T) {
for i := 99; i < 120; i++ {
qm := bench.NewQueryGenerator(int64(i))
call := qm.Random(10, 4, 3, 0, 1000)
t.Log(call)
}
}

117
bench/random.go
View file

@ -1,117 +0,0 @@
package bench
import (
"fmt"
"flag"
"io/ioutil"
"context"
"math/rand"
"time"
)
// RandomSetBits sets bits randomly and deterministically based on a seed.
type RandomSetBits struct {
HasClient
Name string `json:"name"`
BaseBitmapID int64 `json:"base-bitmap-id"`
BaseProfileID int64 `json:"base-profile-id"`
BitmapIDRange int64 `json:"bitmap-id-range"`
ProfileIDRange int64 `json:"profile-id-range"`
Iterations int `json:"iterations"`
Seed int64 `json:"seed"`
DB string `json:"db"`
}
// Init adds the agent num to the random seed and initializes the client.
func (b *RandomSetBits) Init(hosts []string, agentNum int) error {
b.Name = "random-set-bits"
b.Seed = b.Seed + int64(agentNum)
return b.HasClient.Init(hosts, agentNum)
}
// Usage returns the usage message to be printed.
func (b *RandomSetBits) Usage() string {
return `
random-set-bits sets random bits
Agent number modifies the random seed.
Usage: random-set-bits [arguments]
The following arguments are available:
-base-bitmap-id int
bits being set will all be greater than BaseBitmapID
-bitmap-id-range int
number of possible bitmap ids that can be set
-base-profile-id int
profile id num to start from
-profile-id-range int
number of possible profile ids that can be set
-iterations int
number of bits to set
-seed int
Seed for RNG
-db string
pilosa db to use
-client-type string
Can be 'single' (all agents hitting one host) or 'round_robin'
-content-type string
protobuf or pql
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *RandomSetBits) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("RandomSetBits", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.Int64Var(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.Int64Var(&b.BitmapIDRange, "bitmap-id-range", 100000, "")
fs.Int64Var(&b.BaseProfileID, "base-profile-id", 0, "")
fs.Int64Var(&b.ProfileIDRange, "profile-id-range", 100000, "")
fs.Int64Var(&b.Seed, "seed", 1, "")
fs.IntVar(&b.Iterations, "iterations", 100, "")
fs.StringVar(&b.DB, "db", "benchdb", "")
fs.StringVar(&b.ClientType, "client-type", "single", "")
fs.StringVar(&b.ContentType, "content-type", "protobuf", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
return fs.Args(), nil
}
// Run runs the RandomSetBits benchmark
func (b *RandomSetBits) Run(ctx context.Context) map[string]interface{} {
src := rand.NewSource(b.Seed)
rng := rand.New(src)
results := make(map[string]interface{})
if b.client == nil {
results["error"] = fmt.Errorf("No client set for RandomSetBits")
return results
}
s := NewStats()
var start time.Time
for n := 0; n < b.Iterations; n++ {
bitmapID := rng.Int63n(b.BitmapIDRange)
profID := rng.Int63n(b.ProfileIDRange)
query := fmt.Sprintf("SetBit(%d, 'frame.n', %d)", b.BaseBitmapID+bitmapID, b.BaseProfileID+profID)
start = time.Now()
b.client.ExecuteQuery(ctx, b.DB, query, true)
s.Add(time.Now().Sub(start))
}
AddToResults(s, results)
return results
}

119
bench/randquery.go
View file

@ -1,119 +0,0 @@
package bench
import (
"context"
"flag"
"fmt"
"io/ioutil"
"strings"
"time"
)
// RandomQuery queries randomly and deterministically based on a seed.
type RandomQuery struct {
HasClient
Name string `json:"name"`
MaxDepth int `json:"max-depth"`
MaxArgs int `json:"max-args"`
MaxN int `json:"max-n"`
BaseBitmapID int64 `json:"base-bitmap-id"`
BitmapIDRange int64 `json:"bitmap-id-range"`
Iterations int `json:"iterations"`
Seed int64 `json:"seed"`
DBs []string `json:"dbs"`
}
// Init adds the agent num to the random seed and initializes the client.
func (b *RandomQuery) Init(hosts []string, agentNum int) error {
b.Name = "random-query"
b.Seed = b.Seed + int64(agentNum)
return b.HasClient.Init(hosts, agentNum)
}
// Usage returns the usage message to be printed.
func (b *RandomQuery) Usage() string {
return `
random-query constructs random queries
Agent number modifies the random seed.
Usage: random-query [arguments]
The following arguments are available:
-max-depth int
Maximum nesting depth of queries
-max-args int
Maximum number of args for Union/Intersect/Difference Queries
-max-n int
Maximum N value for TopN queries.
-base-bitmap-id int
bitmap id to start from
-bitmap-id-range int
number of possible bitmap ids that can be set
-iterations int
number of bits to set
-seed int
Seed for RNG
-dbs string
Comma separated list of DBs to query against
-client-type string
Can be 'single' (all agents hitting one host) or 'round_robin'
-content-type string
protobuf or pql
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *RandomQuery) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("RandomQuery", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.IntVar(&b.MaxDepth, "max-depth", 4, "")
fs.IntVar(&b.MaxArgs, "max-args", 4, "")
fs.IntVar(&b.MaxN, "max-n", 4, "")
fs.Int64Var(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.Int64Var(&b.BitmapIDRange, "bitmap-id-range", 100000, "")
fs.Int64Var(&b.Seed, "seed", 1, "")
fs.IntVar(&b.Iterations, "iterations", 100, "")
var dbs string
fs.StringVar(&dbs, "dbs", "benchdb", "")
fs.StringVar(&b.ClientType, "client-type", "single", "")
fs.StringVar(&b.ContentType, "content-type", "protobuf", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
b.DBs = strings.Split(dbs, ",")
return fs.Args(), nil
}
// Run runs the RandomQuery benchmark
func (b *RandomQuery) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
if b.client == nil {
results["error"] = fmt.Errorf("No client set for RandomQuery")
return results
}
qm := NewQueryGenerator(b.Seed)
s := NewStats()
var start time.Time
for n := 0; n < b.Iterations; n++ {
call := qm.Random(b.MaxN, b.MaxDepth, b.MaxArgs, uint64(b.BaseBitmapID), uint64(b.BitmapIDRange))
start = time.Now()
b.ExecuteQuery(b.ContentType, b.DBs[n%len(b.DBs)], call.String(), ctx)
s.Add(time.Now().Sub(start))
}
AddToResults(s, results)
return results
}

151
bench/sliceheight.go
View file

@ -1,151 +0,0 @@
package bench
import (
"context"
"flag"
"io"
"io/ioutil"
"strconv"
"time"
"github.com/pilosa/pilosa"
"github.com/pilosa/pilosa/pql"
)
// NewSliceHeight creates a new slice height benchmark with stdin/out/err
// initialized.
func NewSliceHeight(stdin io.Reader, stdout, stderr io.Writer) *SliceHeight {
return &SliceHeight{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// SliceHeight benchmark tests the effect of an increasing number of bitmaps in
// a single slice on query time.
type SliceHeight struct {
MaxTime time.Duration `json:"max-time"`
hosts []string
Name string `json:"name"`
MinBitsPerMap int64 `json:"min-bits-per-map"`
MaxBitsPerMap int64 `json:"max-bits-per-map"`
Seed int64 `json:"seed"`
Database string `json:"database"`
Frame string `json:"frame"`
Stdin io.Reader `json:"-"`
Stdout io.Writer `json:"-"`
Stderr io.Writer `json:"-"`
}
// Usage returns the usage message to be printed.
func (b *SliceHeight) Usage() string {
return `
slice-height repeatedly imports more bitmaps into a single slice and tests query times in between.
Agent number has no effect on this benchmark.
Usage: slice-height [arguments]
The following arguments are available:
-max-time int
stop benchmark after this many seconds
-min-bits-per-map int
minimum number of bits set per bitmap
-max-bits-per-map int
maximum number of bits set per bitmap
-seed int
seed for RNG
-db string
pilosa db to use
-frame string
frame to import into
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *SliceHeight) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("SliceHeight", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
maxTime := fs.Int("max-time", 30, "")
fs.Int64Var(&b.MinBitsPerMap, "min-bits-per-map", 0, "")
fs.Int64Var(&b.MaxBitsPerMap, "max-bits-per-map", 10, "")
fs.Int64Var(&b.Seed, "seed", 0, "")
fs.StringVar(&b.Database, "db", "benchdb", "")
fs.StringVar(&b.Frame, "frame", "testframe", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
b.MaxTime = time.Duration(*maxTime) * time.Second
return fs.Args(), nil
}
// Init sets up the slice height benchmark.
func (b *SliceHeight) Init(hosts []string, agentNum int) error {
b.Name = "slice-height"
b.hosts = hosts
return nil
}
// Run runs the SliceHeight benchmark
func (b *SliceHeight) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
imp := NewImport(b.Stdin, b.Stdout, b.Stderr)
imp.MaxBitmapID = 100
imp.MaxProfileID = pilosa.SliceWidth
imp.MinBitsPerMap = b.MinBitsPerMap
imp.MaxBitsPerMap = b.MaxBitsPerMap
imp.Database = b.Database
imp.Frame = b.Frame
start := time.Now()
for i := 0; i > -1; i++ {
iresults := make(map[string]interface{})
results["iteration"+strconv.Itoa(i)] = iresults
genstart := time.Now()
imp.Init(b.hosts, 0)
gendur := time.Now().Sub(genstart)
iresults["csvgen"] = gendur
iresults["import"] = imp.Run(ctx)
qstart := time.Now()
q := &pql.Call{
Name: "TopN",
Args: map[string]interface{}{
"frame": b.Frame,
"n": 50,
},
}
_, err := imp.Client.ExecuteQuery(ctx, b.Database, q.String(), true)
if err != nil {
iresults["query_error"] = err.Error()
} else {
qdur := time.Now().Sub(qstart)
iresults["query"] = qdur
}
imp.BaseBitmapID = imp.MaxBitmapID
imp.MaxBitmapID = imp.MaxBitmapID * 10
if time.Now().Sub(start) > b.MaxTime {
break
}
}
return results
}

65
bench/stats.go
View file

@ -1,65 +0,0 @@
package bench
import (
"math"
"time"
)
// Stats object helps track timing stats.
type Stats struct {
Min time.Duration
Max time.Duration
Mean time.Duration
sumSquareDelta float64
Total time.Duration
Num int64
All []time.Duration
SaveAll bool
}
// NewStats gets a Stats object.
func NewStats() *Stats {
return &Stats{
Min: 1<<63 - 1,
All: make([]time.Duration, 0),
}
}
// Add adds a new time to the stats object.
func (s *Stats) Add(td time.Duration) {
if s.SaveAll {
s.All = append(s.All, td)
}
s.Num += 1
s.Total += td
if td < s.Min {
s.Min = td
}
if td > s.Max {
s.Max = td
}
// online variance calculation
// https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Online_algorithm
delta := td - s.Mean
s.Mean += delta / time.Duration(s.Num)
s.sumSquareDelta += float64(delta * (td - s.Mean))
}
// Avg returns the average of all durations Added to the Stats object.
func (s *Stats) Avg() time.Duration {
return s.Total / time.Duration(s.Num)
}
// AddToResults serializes the summary of Stats and adds them to the results
// map.
func AddToResults(s *Stats, results map[string]interface{}) {
results["min"] = s.Min
results["max"] = s.Max
results["avg"] = s.Mean
variance := s.sumSquareDelta / float64(s.Num)
results["std"] = time.Duration(math.Sqrt(variance))
if s.SaveAll {
results["all"] = s.All
}
}

39
bench/store.go
View file

@ -1,39 +0,0 @@
package bench
import (
"fmt"
"strings"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
)
// NewS3Uploader creates an S3Uploader with specified bucket and key
func NewS3Uploader(bucket string, key string) *S3Uploader {
return &S3Uploader{
bucket,
s3.New(session.New(&aws.Config{})),
key,
}
}
// S3Uploader is an io.Writer for sending output to AWS S3 storage.
type S3Uploader struct {
bucket string
service *s3.S3
key string
}
// Write writes data to the uploader's bucket/key
func (u *S3Uploader) Write(data []byte) (int, error) {
// first return value of PutObject contains an ETag (hash) of the uploaded object, not needed here
fmt.Println(string(data))
_, err := u.service.PutObject(&s3.PutObjectInput{
Body: strings.NewReader(string(data)),
Bucket: &u.bucket,
Key: &u.key,
})
return 0, err
}

191
bench/zipf.go
View file

@ -1,191 +0,0 @@
package bench
import (
"fmt"
"flag"
"io/ioutil"
"context"
"math"
"math/rand"
"time"
)
// Zipf sets random bits according to the Zipf-Mandelbrot distribution.
// This distribution accepts two parameters, Exponent and Ratio, for both bitmaps and profiles.
// It also uses PermutationGenerator to permute IDs randomly.
type Zipf struct {
HasClient
Name string `json:"name"`
BaseBitmapID int64 `json:"base-bitmap-id"`
BaseProfileID int64 `json:"base-profile-id"`
BitmapIDRange int64 `json:"bitmap-id-range"`
ProfileIDRange int64 `json:"profile-id-range"`
Iterations int `json:"iterations"`
Seed int64 `json:"seed"`
DB string `json:"db"`
BitmapExponent float64 `json:"bitmap-exponent"`
BitmapRatio float64 `json:"bitmap-ratio"`
ProfileExponent float64 `json:"profile-exponent"`
ProfileRatio float64 `json:"profile-ratio"`
Operation string `json:"operation"`
bitmapRng *rand.Zipf
profileRng *rand.Zipf
bitmapPerm *PermutationGenerator
profilePerm *PermutationGenerator
}
// Usage returns the usage message to be printed.
func (b *Zipf) Usage() string {
return `
zipf sets random bits according to the Zipf distribution.
This is a power-law distribution controlled by two parameters.
Exponent, in the range (1, inf), with a default value of 1.001, controls
the "sharpness" of the distribution, with higher exponent being sharper.
Ratio, in the range (0, 1), with a default value of 0.25, controls the
maximum variation of the distribution, with higher ratio being more uniform.
Agent number modifies random seed.
Usage: zipf [arguments]
The following arguments are available:
-base-bitmap-id int
bits being set will all be greater than BaseBitmapID
-bitmap-id-range int
number of possible bitmap ids that can be set
-base-profile-id int
profile id num to start from
-profile-id-range int
number of possible profile ids that can be set
-iterations int
number of bits to set
-seed int
Seed for RNG
-db string
pilosa db to use
-bitmap-exponent float64
zipf exponent parameter for bitmap IDs
-bitmap-ratio float64
zipf probability ratio parameter for bitmap IDs
-profile-exponent float64
zipf exponent parameter for profile IDs
-profile-ratio float64
zipf probability ratio parameter for profile IDs
-client-type string
Can be 'single' (all agents hitting one host) or 'round_robin'
-operation string
Can be 'set' or 'clear'
-content-type string
protobuf or pql
`[1:]
}
// ConsumeFlags parses all flags up to the next non flag argument (argument does
// not start with "-" and isn't the value of a flag). It returns the remaining
// args.
func (b *Zipf) ConsumeFlags(args []string) ([]string, error) {
fs := flag.NewFlagSet("Zipf", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.Int64Var(&b.BaseBitmapID, "base-bitmap-id", 0, "")
fs.Int64Var(&b.BitmapIDRange, "bitmap-id-range", 100000, "")
fs.Int64Var(&b.BaseProfileID, "base-profile-id", 0, "")
fs.Int64Var(&b.ProfileIDRange, "profile-id-range", 100000, "")
fs.Int64Var(&b.Seed, "seed", 1, "")
fs.IntVar(&b.Iterations, "iterations", 100, "")
fs.StringVar(&b.DB, "db", "benchdb", "")
fs.Float64Var(&b.BitmapExponent, "bitmap-exponent", 1.01, "")
fs.Float64Var(&b.BitmapRatio, "bitmap-ratio", 0.25, "")
fs.Float64Var(&b.ProfileExponent, "profile-exponent", 1.01, "")
fs.Float64Var(&b.ProfileRatio, "profile-ratio", 0.25, "")
fs.StringVar(&b.ClientType, "client-type", "single", "")
fs.StringVar(&b.Operation, "operation", "set", "")
fs.StringVar(&b.ContentType, "content-type", "protobuf", "")
if err := fs.Parse(args); err != nil {
return nil, err
}
return fs.Args(), nil
}
// Offset is the true parameter used by the Zipf distribution, but the ratio,
// as defined here, is a simpler, readable way to define the distribution.
// Offset is in [1, inf), and its meaning depends on N (a pain for updating benchmark configs)
// ratio is in (0, 1), and its meaning does not depend on N.
// it is the ratio of the lowest probability in the distribution to the highest.
// ratio=0.01 corresponds to a very small offset - the most skewed distribution for a given pair (N, exp)
// ratio=0.99 corresponds to a very large offset - the most nearly uniform distribution for a given (N, exp)
func getZipfOffset(N int64, exp, ratio float64) float64 {
z := math.Pow(ratio, 1/exp)
return z * float64(N-1) / (1 - z)
}
// Init sets up the benchmark based on the agent number and initializes the
// client.
func (b *Zipf) Init(hosts []string, agentNum int) error {
b.Name = "zipf"
b.Seed = b.Seed + int64(agentNum)
rnd := rand.New(rand.NewSource(b.Seed))
bitmapOffset := getZipfOffset(b.BitmapIDRange, b.BitmapExponent, b.BitmapRatio)
b.bitmapRng = rand.NewZipf(rnd, b.BitmapExponent, bitmapOffset, uint64(b.BitmapIDRange-1))
profileOffset := getZipfOffset(b.ProfileIDRange, b.ProfileExponent, b.ProfileRatio)
b.profileRng = rand.NewZipf(rnd, b.ProfileExponent, profileOffset, uint64(b.ProfileIDRange-1))
b.bitmapPerm = NewPermutationGenerator(b.BitmapIDRange, b.Seed)
b.profilePerm = NewPermutationGenerator(b.ProfileIDRange, b.Seed+1)
if b.Operation != "set" && b.Operation != "clear" {
return fmt.Errorf("Unsupported operation: \"%s\" (must be \"set\" or \"clear\")", b.Operation)
}
return b.HasClient.Init(hosts, agentNum)
}
// Run runs the Zipf benchmark
func (b *Zipf) Run(ctx context.Context) map[string]interface{} {
results := make(map[string]interface{})
if b.client == nil {
results["error"] = fmt.Errorf("No client set for Zipf")
return results
}
operation := "SetBit"
if b.Operation == "clear" {
operation = "ClearBit"
}
s := NewStats()
var start time.Time
for n := 0; n < b.Iterations; n++ {
// generate IDs from Zipf distribution
bitmapIDOriginal := b.bitmapRng.Uint64()
profIDOriginal := b.profileRng.Uint64()
// permute IDs randomly, but repeatably
bitmapID := b.bitmapPerm.Next(int64(bitmapIDOriginal))
profID := b.profilePerm.Next(int64(profIDOriginal))
query := fmt.Sprintf("%s(%d, 'frame.n', %d)", operation, b.BaseBitmapID+int64(bitmapID), b.BaseProfileID+int64(profID))
start = time.Now()
_, err := b.client.ExecuteQuery(ctx, b.DB, query, true)
if err != nil {
results["error"] = fmt.Sprintf("Error executing query in zipf: %v", err)
return results
}
s.Add(time.Now().Sub(start))
}
AddToResults(s, results)
return results
}

29
build/build.go
View file

@ -1,29 +0,0 @@
package build
import (
"fmt"
"io"
"io/ioutil"
"os"
"os/exec"
)
func Binary(pkg, goos, goarch string) (io.Reader, error) {
binFile, err := ioutil.TempFile("", "pilosactl")
if err != nil {
return nil, fmt.Errorf("build binary: %v", err)
}
com := exec.Command("go", "build", "-o", binFile.Name(), pkg)
com.Env = append([]string{"GOOS=" + goos, "GOARCH=" + goarch}, os.Environ()...)
err = com.Run()
if err != nil {
return nil, err
}
f, err := os.Open(binFile.Name())
if err != nil {
return nil, err
}
return f, nil
}

97
client.go
View file

@ -108,6 +108,51 @@ func (c *Client) Schema(ctx context.Context) ([]*DBInfo, error) {
return rsp.DBs, nil
}
// CreateDB creates a new database on the server.
func (c *Client) CreateDB(ctx context.Context, db string, opt DBOptions) error {
// Encode query request.
buf, err := json.Marshal(&postDBRequest{
DB: db,
Options: opt,
})
if err != nil {
return err
}
// Create URL & HTTP request.
u := url.URL{Scheme: "http", Host: c.host, Path: "/db"}
req, err := http.NewRequest("POST", u.String(), bytes.NewReader(buf))
if err != nil {
return err
}
req.Header.Set("Content-Length", strconv.Itoa(len(buf)))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
// Execute request against the host.
resp, err := c.HTTPClient.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
// Read body.
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
// Handle response based on status code.
switch resp.StatusCode {
case http.StatusOK:
return nil // ok
case http.StatusConflict:
return ErrDatabaseExists
default:
return errors.New(string(body))
}
}
// FragmentNodes returns a list of nodes that own a slice.
func (c *Client) FragmentNodes(ctx context.Context, db string, slice uint64) ([]*Node, error) {
// Execute request against the host.
@ -596,6 +641,56 @@ func (c *Client) restoreSliceFrom(ctx context.Context, buf []byte, db, frame str
return nil
}
// CreateFrame creates a new frame on the server.
func (c *Client) CreateFrame(ctx context.Context, db, frame string, opt FrameOptions) error {
if db == "" {
return ErrDatabaseRequired
}
// Encode query request.
buf, err := json.Marshal(&postFrameRequest{
DB: db,
Frame: frame,
Options: opt,
})
if err != nil {
return err
}
// Create URL & HTTP request.
u := url.URL{Scheme: "http", Host: c.host, Path: "/frame"}
req, err := http.NewRequest("POST", u.String(), bytes.NewReader(buf))
if err != nil {
return err
}
req.Header.Set("Content-Length", strconv.Itoa(len(buf)))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "application/json")
// Execute request against the host.
resp, err := c.HTTPClient.Do(req.WithContext(ctx))
if err != nil {
return err
}
defer resp.Body.Close()
// Read body.
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
// Handle response based on status code.
switch resp.StatusCode {
case http.StatusOK:
return nil // ok
case http.StatusConflict:
return ErrFrameExists
default:
return errors.New(string(body))
}
}
// RestoreFrame restores an entire frame from a host in another cluster.
func (c *Client) RestoreFrame(ctx context.Context, host, db, frame string) error {
u := url.URL{
@ -797,6 +892,8 @@ func (c *Client) BitmapAttrDiff(ctx context.Context, db, frame string, blks []At
// Return error if status is not OK.
switch resp.StatusCode {
case http.StatusOK: // ok
case http.StatusNotFound:
return nil, ErrFrameNotFound
default:
return nil, fmt.Errorf("unexpected status: code=%d", resp.StatusCode)
}

3
client_test.go
View file

@ -71,6 +71,9 @@ func TestClient_BackupRestore(t *testing.T) {
}
// Restore to a different frame.
if _, err := idx.MustCreateDBIfNotExists("x", pilosa.DBOptions{}).CreateFrameIfNotExists("y", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
if err := c.RestoreFrom(context.Background(), &buf, "x", "y"); err != nil {
t.Fatal(err)
}

42
cmd/backup.go Normal file
View file

@ -0,0 +1,42 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var backuper = ctl.NewBackupCommand(os.Stdin, os.Stdout, os.Stderr)
var backupCmd = &cobra.Command{
Use: "backup",
Short: "Backup data from pilosa.",
Long: `
Backs up the database and frame from across the cluster into a single file.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := backuper.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
backupCmd.Flags().StringVarP(&backuper.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
backupCmd.Flags().StringVarP(&backuper.Database, "database", "d", "", "Pilosa database to backup into.")
backupCmd.Flags().StringVarP(&backuper.Frame, "frame", "f", "", "Frame to backup into.")
backupCmd.Flags().StringVarP(&backuper.Path, "output-file", "o", "", "File to write backup to - default stdout")
err := viper.BindPFlags(backupCmd.Flags())
if err != nil {
log.Fatalf("Error binding backup flags: %v", err)
}
RootCmd.AddCommand(backupCmd)
}

43
cmd/bench.go Normal file
View file

@ -0,0 +1,43 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var bencher = ctl.NewBenchCommand(os.Stdin, os.Stdout, os.Stderr)
var benchCmd = &cobra.Command{
Use: "bench",
Short: "Benchmark operations.",
Long: `
Executes a benchmark for a given operation against the database.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := bencher.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
benchCmd.Flags().StringVarP(&bencher.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
benchCmd.Flags().StringVarP(&bencher.Database, "database", "d", "", "Pilosa database to benchmark.")
benchCmd.Flags().StringVarP(&bencher.Frame, "frame", "f", "", "Frame to benchmark.")
benchCmd.Flags().StringVarP(&bencher.Op, "operation", "o", "set-bit", "Operation to perform: choose from [set-bit]")
benchCmd.Flags().IntVarP(&bencher.N, "num", "n", 0, "Number of operations to perform.")
err := viper.BindPFlags(benchCmd.Flags())
if err != nil {
log.Fatalf("Error binding bench flags: %v", err)
}
RootCmd.AddCommand(benchCmd)
}

35
cmd/check.go Normal file
View file

@ -0,0 +1,35 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var checker = ctl.NewCheckCommand(os.Stdin, os.Stdout, os.Stderr)
var checkCmd = &cobra.Command{
Use: "check <path> [path2]...",
Short: "Do a consistency check on a pilosa data file.",
Long: `
Performs a consistency check on data files.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
}
checker.Paths = args
if err := checker.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(checkCmd)
}

29
cmd/config.go Normal file
View file

@ -0,0 +1,29 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var conf = ctl.NewConfigCommand(os.Stdin, os.Stdout, os.Stderr)
var confCmd = &cobra.Command{
Use: "config",
Short: "Print the default configuration.",
Long: `config prints the default configuration to stdout
`,
Run: func(cmd *cobra.Command, args []string) {
if err := conf.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(confCmd)
}

49
cmd/export.go Normal file
View file

@ -0,0 +1,49 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var exporter = ctl.NewExportCommand(os.Stdin, os.Stdout, os.Stderr)
var exportCmd = &cobra.Command{
Use: "export",
Short: "Export data from pilosa.",
Long: `
Bulk exports a fragment to a CSV file. If the OUTFILE is not specified then
the output is written to STDOUT.
The format of the CSV file is:
BITMAPID,PROFILEID
The file does not contain any headers.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := exporter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
exportCmd.Flags().StringVarP(&exporter.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
exportCmd.Flags().StringVarP(&exporter.Database, "database", "d", "", "Pilosa database to export into.")
exportCmd.Flags().StringVarP(&exporter.Frame, "frame", "f", "", "Frame to export into.")
exportCmd.Flags().StringVarP(&exporter.Path, "output-file", "o", "", "File to write export to - default stdout")
err := viper.BindPFlags(exportCmd.Flags())
if err != nil {
log.Fatalf("Error binding export flags: %v", err)
}
RootCmd.AddCommand(exportCmd)
}

50
cmd/import.go Normal file
View file

@ -0,0 +1,50 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var importer = ctl.NewImportCommand(os.Stdin, os.Stdout, os.Stderr)
var importCmd = &cobra.Command{
Use: "import",
Short: "Bulk load data into pilosa.",
Long: `Bulk imports one or more CSV files to a host's database and frame. The bits
of the CSV file are grouped by slice for the most efficient import.
The format of the CSV file is:
BITMAPID,PROFILEID,[TIME]
The file should contain no headers. The TIME column is optional and can be
omitted. If it is present then its format should be YYYY-MM-DDTHH:MM.
`,
Run: func(cmd *cobra.Command, args []string) {
importer.Paths = args
if err := importer.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
importCmd.Flags().StringVarP(&importer.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
importCmd.Flags().StringVarP(&importer.Database, "database", "d", "", "Pilosa database to import into.")
importCmd.Flags().StringVarP(&importer.Frame, "frame", "f", "", "Frame to import into.")
importCmd.Flags().IntVarP(&importer.BufferSize, "buffer-size", "s", 10000000, "Number of bits to buffer/sort before importing.")
err := viper.BindPFlags(importCmd.Flags())
if err != nil {
log.Fatalf("Error binding import flags: %v", err)
}
RootCmd.AddCommand(importCmd)
}

38
cmd/inspect.go Normal file
View file

@ -0,0 +1,38 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var inspecter = ctl.NewInspectCommand(os.Stdin, os.Stdout, os.Stderr)
var inspectCmd = &cobra.Command{
Use: "inspect",
Short: "Get stats on a pilosa data file.",
Long: `
Inspects a data file and provides stats.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
} else if len(args) > 1 {
fmt.Println("only one path allowed")
return
}
inspecter.Path = args[0]
if err := inspecter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(inspectCmd)
}

194
cmd/pilosa/main.go
View file

@ -1,203 +1,15 @@
package main
import (
"errors"
"flag"
"fmt"
"io"
"math/rand"
"net/http"
"os"
"os/signal"
"path/filepath"
"runtime/pprof"
"strings"
"time"
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
)
// Version and BuildTime hold the version/build time information passed in at compile time.
var (
Version string
BuildTime string
)
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
rand.Seed(time.Now().UTC().UnixNano())
}
const (
// DefaultDataDir is the default data directory.
DefaultDataDir = "~/.pilosa"
"github.com/pilosa/pilosa/cmd"
)
func main() {
// Limit the number of connections that a server can make to a single node
// in order to prevent a cluster storm.
// TODO: make this configurable
http.DefaultTransport.(*http.Transport).MaxIdleConnsPerHost = 64
m := NewMain()
m.Server.Handler.Version = Version
fmt.Fprintf(m.Stderr, "Pilosa %s, build time %s\n", Version, BuildTime)
// Parse command line arguments.
if err := m.ParseFlags(os.Args[1:]); err != nil {
fmt.Fprintln(m.Stderr, err)
os.Exit(2)
}
// Start CPU profiling.
if m.CPUProfile != "" {
f, err := os.Create(m.CPUProfile)
if err != nil {
fmt.Fprintf(m.Stderr, "create cpu profile: %v", err)
os.Exit(1)
}
defer f.Close()
fmt.Fprintln(m.Stderr, "Starting cpu profile")
pprof.StartCPUProfile(f)
time.AfterFunc(m.CPUTime, func() {
fmt.Fprintln(m.Stderr, "Stopping cpu profile")
pprof.StopCPUProfile()
f.Close()
})
}
// Execute the program.
if err := m.Run(); err != nil {
fmt.Fprintln(m.Stderr, err)
fmt.Fprintln(m.Stderr, "stopping profile")
os.Exit(1)
}
// First SIGKILL causes server to shut down gracefully.
c := make(chan os.Signal, 2)
signal.Notify(c, os.Interrupt)
sig := <-c
fmt.Fprintf(m.Stderr, "Received %s; gracefully shutting down...\n", sig.String())
// Second signal causes a hard shutdown.
go func() { <-c; os.Exit(1) }()
if err := m.Close(); err != nil {
fmt.Fprintln(m.Stderr, err)
if err := cmd.RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
// Main represents the main program execution.
type Main struct {
Server *pilosa.Server
// Configuration options.
ConfigPath string
Config *pilosa.Config
// Profiling options.
CPUProfile string
CPUTime time.Duration
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain returns a new instance of Main.
func NewMain() *Main {
return &Main{
Server: pilosa.NewServer(),
Config: pilosa.NewConfig(),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the main program execution.
func (m *Main) Run(args ...string) error {
// Notify user of config file.
if m.ConfigPath != "" {
fmt.Fprintf(m.Stdout, "Using config: %s\n", m.ConfigPath)
}
// Setup logging output.
m.Server.LogOutput = m.Stderr
// Configure index.
fmt.Fprintf(m.Stderr, "Using data from: %s\n", m.Config.DataDir)
m.Server.Index.Path = m.Config.DataDir
m.Server.Index.Stats = pilosa.NewExpvarStatsClient()
// Build cluster from config file.
m.Server.Host = m.Config.Host
m.Server.Cluster = m.Config.PilosaCluster()
// Set configuration options.
m.Server.AntiEntropyInterval = time.Duration(m.Config.AntiEntropy.Interval)
// Initialize server.
if err := m.Server.Open(); err != nil {
return err
}
fmt.Fprintf(m.Stderr, "Listening as http://%s\n", m.Server.Host)
return nil
}
// Close shuts down the server.
func (m *Main) Close() error {
return m.Server.Close()
}
// ParseFlags parses command line flags from args.
func (m *Main) ParseFlags(args []string) error {
fs := flag.NewFlagSet("pilosa", flag.ContinueOnError)
fs.StringVar(&m.CPUProfile, "cpuprofile", "", "cpu profile")
fs.DurationVar(&m.CPUTime, "cputime", 30*time.Second, "cpu profile duration")
fs.StringVar(&m.ConfigPath, "config", "", "config path")
fs.SetOutput(m.Stderr)
if err := fs.Parse(args); err != nil {
return err
}
// Load config, if specified.
if m.ConfigPath != "" {
if _, err := toml.DecodeFile(m.ConfigPath, &m.Config); err != nil {
return err
}
}
// Use default data directory if one is not specified.
if m.Config.DataDir == "" {
m.Config.DataDir = DefaultDataDir
}
// Expand home directory.
prefix := "~" + string(filepath.Separator)
if strings.HasPrefix(m.Config.DataDir, prefix) {
// u, err := user.Current()
HomeDir := os.Getenv("HOME")
/*if err != nil {
return err
} else*/if HomeDir == "" {
return errors.New("data directory not specified and no home dir available")
}
m.Config.DataDir = filepath.Join(HomeDir, strings.TrimPrefix(m.Config.DataDir, prefix))
}
return nil
}

9
cmd/pilosactl/import.json
View file

@ -1,9 +0,0 @@
{
"benchmarks": [
{
"num": 1,
"name": "import",
"args": ["import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width", "import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width", "-random-bitmap-order", "-db", "randoload"]
}
]
}

1720
cmd/pilosactl/main.go
View file

File diff suppressed because it is too large Load diff

1
cmd/pilosactl/main_test.go
View file

@ -1 +0,0 @@
package main_test

8
cmd/pilosactl/multidbspawn.json
View file

@ -1,8 +0,0 @@
{
"benchmarks": [
{
"num": 3,
"args": ["multi-db-set-bits", "-iterations", "30000", "-client-type", "round_robin"]
}
]
}

11
cmd/pilosactl/randSetAndQuery.json
View file

@ -1,11 +0,0 @@
{
"benchmarks": [
{
"num": 3,
"args": [
"random-set-bits", "-iterations", "30000", "-profile-id-range", "1000000", "-bitmap-id-range", "1000000", "-seed", "2345", "-client-type", "round_robin", "-db", "randsetandquery",
"random-query", "-iterations", "1000", "-bitmap-id-range", "1000000", "-seed", "1239", "-dbs", "randsetandquery"
]
}
]
}

8
cmd/pilosactl/randspawn.json
View file

@ -1,8 +0,0 @@
{
"benchmarks": [
{
"num": 3,
"args": ["random-set-bits", "-iterations", "30000", "-profile-id-range", "1000000", "-bitmap-id-range", "1000000", "-seed", "2345", "-client-type", "round_robin"]
}
]
}

8
cmd/pilosactl/slice-height.json
View file

@ -1,8 +0,0 @@
{
"benchmarks": [
{
"num": 1,
"args": ["-human", "slice-height", "-max-time", "1", "-max-bits-per-map", "100"]
}
]
}

14
cmd/pilosactl/spawn.json
View file

@ -1,14 +0,0 @@
{
"benchmarks": [
{
"num": 3,
"name": "set-diags",
"args": ["diagonal-set-bits", "-iterations", "30000", "-client-type", "round_robin"]
},
{
"num": 2,
"name": "rand-plus-zipf",
"args": ["random-set-bits", "-iterations", "20000", "zipf", "-iterations", "100"]
}
]
}

12
cmd/pilosactl/zipfspawn.json
View file

@ -1,12 +0,0 @@
{
"benchmarks": [
{
"num": 1,
"args": ["zipf", "-iterations", "10000", "-profile-id-range", "100", "-bitmap-id-range", "100", "-seed", "2345", "-client-type", "round_robin", "-bitmap-exponent", "1.001", "-bitmap-ratio", ".9", "-profile-exponent", "1.001", "-profile-ratio", ".3"]
},
{
"num": 1,
"args": ["zipf", "-iterations", "10000", "-profile-id-range", "100", "-bitmap-id-range", "100", "-seed", "2345", "-client-type", "round_robin", "-bitmap-exponent", "1.001", "-bitmap-ratio", ".9", "-profile-exponent", "1.001", "-profile-ratio", ".3", "-operation", "clear"]
}
]
}

42
cmd/restore.go Normal file
View file

@ -0,0 +1,42 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var restorer = ctl.NewRestoreCommand(os.Stdin, os.Stdout, os.Stderr)
var restoreCmd = &cobra.Command{
Use: "restore",
Short: "Restore data to pilosa from a backup file.",
Long: `
Restores a frame to the cluster from a backup file.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := restorer.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
restoreCmd.Flags().StringVarP(&restorer.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
restoreCmd.Flags().StringVarP(&restorer.Database, "database", "d", "", "Pilosa database to restore into.")
restoreCmd.Flags().StringVarP(&restorer.Frame, "frame", "f", "", "Frame to restore into.")
restoreCmd.Flags().StringVarP(&restorer.Path, "input-file", "i", "", "File to restore from.")
err := viper.BindPFlags(restoreCmd.Flags())
if err != nil {
log.Fatalf("Error binding restore flags: %v", err)
}
RootCmd.AddCommand(restoreCmd)
}

33
cmd/root.go Normal file
View file

@ -0,0 +1,33 @@
package cmd
import "github.com/spf13/cobra"
var (
Version string
BuildTime string
)
var RootCmd = &cobra.Command{
Use: "pilosa",
Short: "Pilosa - A Distributed In-memory Binary Bitmap Index.",
// TODO - is documentation actually there?
Long: `Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at http://pilosa.com/docs
`,
}
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
RootCmd.Long = RootCmd.Long + "Version: " + Version + "\nBuild Time: " + BuildTime + "\n"
}

89
cmd/server.go Normal file
View file

@ -0,0 +1,89 @@
package cmd
import (
"fmt"
"log"
"os"
"os/signal"
"runtime/pprof"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/server"
)
var serve = server.NewCommand()
var serveCmd = &cobra.Command{
Use: "server",
Short: "Run Pilosa.",
Long: `pilosa server runs Pilosa.
It will load existing data from the configured
directory, and start listening client connections
on the configured port.`,
Run: func(cmd *cobra.Command, args []string) {
serve.Server.Handler.Version = server.Version
fmt.Fprintf(serve.Stderr, "Pilosa %s, build time %s\n", server.Version, server.BuildTime)
// Parse command line arguments.
if err := serve.SetupConfig(args); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(2)
}
// Start CPU profiling.
if serve.CPUProfile != "" {
f, err := os.Create(serve.CPUProfile)
if err != nil {
fmt.Fprintf(serve.Stderr, "create cpu profile: %v", err)
os.Exit(1)
}
defer f.Close()
fmt.Fprintln(serve.Stderr, "Starting cpu profile")
pprof.StartCPUProfile(f)
time.AfterFunc(serve.CPUTime, func() {
fmt.Fprintln(serve.Stderr, "Stopping cpu profile")
pprof.StopCPUProfile()
f.Close()
})
}
// Execute the program.
if err := serve.Run(); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(1)
}
// First SIGKILL causes server to shut down gracefully.
c := make(chan os.Signal, 2)
signal.Notify(c, os.Interrupt)
sig := <-c
fmt.Fprintf(serve.Stderr, "Received %s; gracefully shutting down...\n", sig.String())
// Second signal causes a hard shutdown.
go func() { <-c; os.Exit(1) }()
if err := serve.Close(); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(1)
}
},
}
func init() {
serveCmd.Flags().StringVarP(&serve.ConfigPath, "config", "c", "", "Configuration file to read from")
serveCmd.Flags().StringVarP(&serve.CPUProfile, "cpuprofile", "", "", "Where to store CPU profile")
serveCmd.Flags().DurationVarP(&serve.CPUTime, "cputime", "", 30*time.Second, "CPU profile duration")
err := viper.BindPFlags(serveCmd.Flags())
if err != nil {
log.Fatalf("Error binding server flags: %v", err)
}
RootCmd.AddCommand(serveCmd)
}

44
cmd/sort.go Normal file
View file

@ -0,0 +1,44 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var sorter = ctl.NewSortCommand(os.Stdin, os.Stdout, os.Stderr)
var sortCmd = &cobra.Command{
Use: "sort <path>",
Short: "Sort import data for optimal import performance.",
Long: `
Sorts the import data at PATH into the optimal sort order for importing.
The format of the CSV file is:
BITMAPID,PROFILEID
The file should contain no headers.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
} else if len(args) > 1 {
fmt.Println("only one path supported")
return
}
sorter.Path = args[0]
if err := sorter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(sortCmd)
}

11
creator/creator.go
View file

@ -1,11 +0,0 @@
// creator contains code for standing up pilosa clusters
package creator
import "io"
type Cluster interface {
Start() error
Hosts() []string
Shutdown() error
Logs() []io.Reader
}

94
creator/local.go
View file

@ -1,94 +0,0 @@
package creator
import (
"fmt"
"io"
"io/ioutil"
"os"
"path/filepath"
"strconv"
"github.com/pilosa/pilosa"
)
type LocalCluster struct {
ReplicaN int
ServerN int
hosts []string
logs []io.Reader
servers []*pilosa.Server
cluster *pilosa.Cluster
path string
}
func (localCluster *LocalCluster) Start() error {
BasePort := 19327
localCluster.hosts = make([]string, localCluster.ServerN)
localCluster.servers = make([]*pilosa.Server, localCluster.ServerN)
localCluster.logs = make([]io.Reader, localCluster.ServerN)
path, err := ioutil.TempDir("", "pilosa-bench-")
if err != nil {
return err
}
localCluster.path = path
// Build cluster configuration.
cluster := pilosa.NewCluster()
cluster.ReplicaN = localCluster.ReplicaN
for i := 0; i < localCluster.ServerN; i++ {
cluster.Nodes = append(cluster.Nodes, &pilosa.Node{
Host: fmt.Sprintf("localhost:%d", BasePort+i),
})
}
localCluster.cluster = cluster
// Build servers.
for i := range localCluster.servers {
// Make server work directory.
if err := os.MkdirAll(filepath.Join(path, strconv.Itoa(i)), 0777); err != nil {
return err
}
// Build server.
s := pilosa.NewServer()
s.Host = fmt.Sprintf("localhost:%d", BasePort+i)
s.Cluster = cluster
s.Index.Path = filepath.Join(path, strconv.Itoa(i), "data")
// Create log stream
localCluster.logs[i], s.LogOutput = io.Pipe()
localCluster.servers[i] = s
}
// Open all servers.
for i, s := range localCluster.servers {
if err := s.Open(); err != nil {
return err
}
localCluster.hosts[i] = s.Host
}
return nil
}
func (c *LocalCluster) Hosts() []string { return c.hosts }
func (c *LocalCluster) Logs() []io.Reader { return c.logs }
func (c *LocalCluster) Shutdown() error {
errs := ""
for _, s := range c.servers {
if err := s.Close(); err != nil {
errs = errs + err.Error() + "; "
}
}
if err := os.RemoveAll(c.path); err != nil {
errs = errs + err.Error() + ";"
}
if errs != "" {
return fmt.Errorf(errs)
}
return nil
}

187
creator/remote.go
View file

@ -1,187 +0,0 @@
package creator
import (
"fmt"
"io"
"math/rand"
"net"
"strconv"
"sync"
"time"
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
"github.com/pilosa/pilosa/build"
pssh "github.com/pilosa/pilosa/ssh"
"golang.org/x/crypto/ssh"
)
type RemoteCluster struct {
ClusterHosts []string
ReplicaN int
SSHUser string
Keyfile string
Key []byte
GoMaxProcs int
CopyBinary bool
GOOS string
GOARCH string
Stderr io.Writer
wg *sync.WaitGroup
logs []io.Reader
sessions []*ssh.Session
pipeRs []*io.PipeReader
pipeWs []*io.PipeWriter
stdins []io.WriteCloser
}
// Start creates a configuration for each host in the cluster, copies it to the
// node, and starts the pilosa process on the remote host.
func (c *RemoteCluster) Start() error {
c.logs = make([]io.Reader, 0)
if len(c.ClusterHosts) == 0 {
return fmt.Errorf("no type or hosts specified - cannot continue")
}
fleet, err := pssh.NewFleet(c.ClusterHosts, c.SSHUser, c.Keyfile, c.Stderr)
if err != nil {
return fmt.Errorf("connecting to cluster hosts: %v", err)
}
cmdName := "pilosa"
if c.CopyBinary {
cmdName = "/tmp/pilosa" + strconv.Itoa(rand.Int())
fmt.Fprintf(c.Stderr, "create: building pilosa binary with GOOS=%v and GOARCH=%v to copy to hosts at %v", c.GOOS, c.GOARCH, cmdName)
pkg := "github.com/pilosa/pilosa/cmd/pilosa"
bin, err := build.Binary(pkg, c.GOOS, c.GOARCH)
if err != nil {
return fmt.Errorf("building binary: %v", err)
}
err = fleet.WriteFile(cmdName, "+x", bin)
if err != nil {
return fmt.Errorf("writing binary to fleet: %v", err)
}
}
// build config
conf := pilosa.NewConfigForHosts(c.ClusterHosts)
conf.Cluster.ReplicaN = c.ReplicaN
// copy config to remote hosts and start pilosa
c.wg = &sync.WaitGroup{}
for _, hostport := range c.ClusterHosts {
// Set up config for this host
host, port, err := net.SplitHostPort(hostport)
if err != nil {
return fmt.Errorf("splitting hostport: %v", err)
}
conf.Host = hostport
conf.DataDir = "~/.pilosa" + port
// Get client for host
client, err := fleet.Get(host)
if err != nil {
return fmt.Errorf("connecting to host: %v", err)
}
configname := "pilosa" + port + ".conf"
w, err := client.OpenFile(configname, "")
if err != nil {
return fmt.Errorf("opening remote config file: %v", err)
}
enc := toml.NewEncoder(w)
err = enc.Encode(conf)
if err != nil {
return fmt.Errorf("encoding config: %v", err)
}
err = w.Close()
if err != nil {
return fmt.Errorf("closing config writer: %v", err)
}
// Start pilosa on remote host
sess, err := client.NewSession()
if err != nil {
return err
}
// Have to request pty in order to be able to kill remote process
// reliably.
modes := ssh.TerminalModes{
ssh.ISIG: 1,
ssh.ECHO: 0,
}
err = sess.RequestPty("vt100", 40, 80, modes)
if err != nil {
return fmt.Errorf("request pty error: %v", err)
}
pipeR, pipeW := io.Pipe()
sess.Stdout = pipeW
sess.Stderr = pipeW
inpipe, err := sess.StdinPipe()
if err != nil {
return err
}
c.logs = append(c.logs, pipeR)
c.sessions = append(c.sessions, sess)
c.pipeRs = append(c.pipeRs, pipeR)
c.pipeWs = append(c.pipeWs, pipeW)
c.stdins = append(c.stdins, inpipe)
gomaxprocsString := ""
if c.GoMaxProcs != 0 {
gomaxprocsString = "GOMAXPROCS=" + strconv.Itoa(c.GoMaxProcs) + " "
}
err = sess.Start(gomaxprocsString + cmdName + " -config " + configname)
if err != nil {
return err
}
c.wg.Add(1)
go func() {
defer c.wg.Done()
err = sess.Wait()
if err != nil {
fmt.Fprintf(c.Stderr, "problem with remote pilosa process: %v\n", err)
}
}()
}
return nil
}
func (c *RemoteCluster) Hosts() []string { return c.ClusterHosts }
func (c *RemoteCluster) Logs() []io.Reader { return c.logs }
func (c *RemoteCluster) Shutdown() error {
for i, sess := range c.sessions {
var err error
_, err = c.stdins[i].Write([]byte{3}) // Send Control C
if err != nil {
fmt.Fprintf(c.Stderr, "Error write-signaling remote process: %v\n", err)
}
// signaling isn't supported by many ssh servers - hence the hack above
err = sess.Signal(ssh.SIGINT)
if err != nil {
fmt.Fprintf(c.Stderr, "Error signaling remote process: %v\n", err)
}
}
done := make(chan struct{}, 1)
go func() {
c.wg.Wait()
close(done)
}()
select {
case <-done:
return nil
case <-time.After(time.Second * 5):
for _, sess := range c.sessions {
err := sess.Close()
if err != nil {
fmt.Fprintf(c.Stderr, "Error closing remote session: %v\n", err)
}
}
return fmt.Errorf("timed out waiting for remote processes to exit")
}
}

72
ctl/backup.go Normal file
View file

@ -0,0 +1,72 @@
package ctl
import (
"context"
"errors"
"io"
"os"
"github.com/pilosa/pilosa"
)
// BackupCommand represents a command for backing up a frame.
type BackupCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to backup.
Database string
Frame string
// Output file to write to.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewBackupCommand returns a new instance of BackupCommand.
func NewBackupCommand(stdin io.Reader, stdout, stderr io.Writer) *BackupCommand {
return &BackupCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the backup.
func (cmd *BackupCommand) Run(ctx context.Context) error {
// Validate arguments.
if cmd.Path == "" {
return errors.New("output file required")
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Open output file.
f, err := os.Create(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Begin streaming backup.
if err := client.BackupTo(ctx, f, cmd.Database, cmd.Frame); err != nil {
return err
}
// Sync & close file to ensure durability.
if err := f.Sync(); err != nil {
return err
} else if err = f.Close(); err != nil {
return err
}
return nil
}

92
ctl/bench.go Normal file
View file

@ -0,0 +1,92 @@
package ctl
import (
"context"
"errors"
"fmt"
"io"
"math/rand"
"time"
"github.com/pilosa/pilosa"
)
// BenchCommand represents a command for benchmarking database operations.
type BenchCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to execute against.
Database string
Frame string
// Type of operation and number to execute.
Op string
N int
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewBenchCommand returns a new instance of BenchCommand.
func NewBenchCommand(stdin io.Reader, stdout, stderr io.Writer) *BenchCommand {
return &BenchCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the bench command.
func (cmd *BenchCommand) Run(ctx context.Context) error {
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
switch cmd.Op {
case "set-bit":
return cmd.runSetBit(ctx, client)
case "":
return errors.New("op required")
default:
return fmt.Errorf("unknown bench op: %q", cmd.Op)
}
}
// runSetBit executes a benchmark of random SetBit() operations.
func (cmd *BenchCommand) runSetBit(ctx context.Context, client *pilosa.Client) error {
if cmd.N == 0 {
return errors.New("operation count required")
} else if cmd.Database == "" {
return pilosa.ErrDatabaseRequired
} else if cmd.Frame == "" {
return pilosa.ErrFrameRequired
}
const maxBitmapID = 1000
const maxProfileID = 100000
startTime := time.Now()
// Execute operation continuously.
for i := 0; i < cmd.N; i++ {
bitmapID := rand.Intn(maxBitmapID)
profileID := rand.Intn(maxProfileID)
q := fmt.Sprintf(`SetBit(id=%d, frame="%s", profileID=%d)`, bitmapID, cmd.Frame, profileID)
if _, err := client.ExecuteQuery(ctx, cmd.Database, q, true); err != nil {
return err
}
}
// Print results.
elapsed := time.Since(startTime)
fmt.Fprintf(cmd.Stdout, "Executed %d operations in %s (%0.3f op/sec)\n", cmd.N, elapsed, float64(cmd.N)/elapsed.Seconds())
return nil
}

114
ctl/check.go Normal file
View file

@ -0,0 +1,114 @@
package ctl
import (
"context"
"fmt"
"io"
"os"
"path/filepath"
"syscall"
"github.com/pilosa/pilosa/roaring"
)
// CheckCommand represents a command for performing consistency checks on data files.
type CheckCommand struct {
// Data file paths.
Paths []string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewCheckCommand returns a new instance of CheckCommand.
func NewCheckCommand(stdin io.Reader, stdout, stderr io.Writer) *CheckCommand {
return &CheckCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the check command.
func (cmd *CheckCommand) Run(ctx context.Context) error {
for _, path := range cmd.Paths {
switch filepath.Ext(path) {
case "":
if err := cmd.checkBitmapFile(path); err != nil {
return err
}
case ".cache":
if err := cmd.checkCacheFile(path); err != nil {
return err
}
case ".snapshotting":
if err := cmd.checkSnapshotFile(path); err != nil {
return err
}
}
}
return nil
}
// checkBitmapFile performs a consistency check on path for a roaring bitmap file.
func (cmd *CheckCommand) checkBitmapFile(path string) error {
// Open file handle.
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
// Memory map the file.
data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
return err
}
defer syscall.Munmap(data)
// Attach the mmap file to the bitmap.
bm := roaring.NewBitmap()
if err := bm.UnmarshalBinary(data); err != nil {
return err
}
// Perform consistency check.
if err := bm.Check(); err != nil {
// Print returned errors.
switch err := err.(type) {
case roaring.ErrorList:
for i := range err {
fmt.Fprintf(cmd.Stdout, "%s: %s\n", path, err[i].Error())
}
default:
fmt.Fprintf(cmd.Stdout, "%s: %s\n", path, err.Error())
}
}
// Print success message if no errors were found.
fmt.Fprintf(cmd.Stdout, "%s: ok\n", path)
return nil
}
// checkCacheFile performs a consistency check on path for a cache file.
func (cmd *CheckCommand) checkCacheFile(path string) error {
fmt.Fprintf(cmd.Stderr, "%s: ignoring cache file\n", path)
return nil
}
// checkSnapshotFile performs a consistency check on path for a snapshot file.
func (cmd *CheckCommand) checkSnapshotFile(path string) error {
fmt.Fprintf(cmd.Stderr, "%s: ignoring snapshot file\n", path)
return nil
}

43
ctl/config.go Normal file
View file

@ -0,0 +1,43 @@
package ctl
import (
"context"
"fmt"
"io"
"strings"
)
// ConfigCommand represents a command for printing a default config.
type ConfigCommand struct {
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewConfigCommand returns a new instance of ConfigCommand.
func NewConfigCommand(stdin io.Reader, stdout, stderr io.Writer) *ConfigCommand {
return &ConfigCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run prints out the default config.
func (cmd *ConfigCommand) Run(ctx context.Context) error {
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(`
data-dir = "~/.pilosa"
host = "localhost:15000"
[cluster]
replicas = 1
[[cluster.node]]
host = "localhost:15000"
[plugins]
path = ""
`)+"\n")
return nil
}

91
ctl/export.go Normal file
View file

@ -0,0 +1,91 @@
package ctl
import (
"context"
"io"
"log"
"os"
"github.com/pilosa/pilosa"
)
// ExportCommand represents a command for bulk exporting data from a server.
type ExportCommand struct {
// Remote host and port.
Host string
// Name of the database & frame to export from.
Database string
Frame string
// Filename to export to.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewExportCommand returns a new instance of ExportCommand.
func NewExportCommand(stdin io.Reader, stdout, stderr io.Writer) *ExportCommand {
return &ExportCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the export.
func (cmd *ExportCommand) Run(ctx context.Context) error {
logger := log.New(cmd.Stderr, "", log.LstdFlags)
// Validate arguments.
if cmd.Database == "" {
return pilosa.ErrDatabaseRequired
} else if cmd.Frame == "" {
return pilosa.ErrFrameRequired
}
// Use output file, if specified.
// Otherwise use STDOUT.
var w io.Writer = cmd.Stdout
if cmd.Path != "" {
f, err := os.Create(cmd.Path)
if err != nil {
return err
}
defer f.Close()
w = f
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Determine slice count.
maxSlices, err := client.MaxSliceByDatabase(ctx)
if err != nil {
return err
}
// Export each slice.
for slice := uint64(0); slice <= maxSlices[cmd.Database]; slice++ {
logger.Printf("exporting slice: %d", slice)
if err := client.ExportCSV(ctx, cmd.Database, cmd.Frame, slice, w); err != nil {
return err
}
}
// Close writer, if applicable.
if w, ok := w.(io.Closer); ok {
if err := w.Close(); err != nil {
return err
}
}
return nil
}

41
pilosactl/import.go → ctl/import.go
View file

@ -1,17 +1,14 @@
package pilosactl
package ctl
import (
"context"
"encoding/csv"
"errors"
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strconv"
"strings"
"time"
"github.com/pilosa/pilosa"
@ -56,41 +53,6 @@ func (cmd *ImportCommand) String() string {
return fmt.Sprint(*cmd)
}
// ParseFlags parses command line flags from args.
func (cmd *ImportCommand) ParseFlags(args []string) error {
fs := flag.NewFlagSet("pilosactl", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.StringVar(&cmd.Host, "host", "localhost:15000", "host:port")
fs.StringVar(&cmd.Database, "d", "", "database")
fs.StringVar(&cmd.Frame, "f", "", "frame")
fs.IntVar(&cmd.BufferSize, "buffer-size", cmd.BufferSize, "buffer size")
if err := fs.Parse(args); err != nil {
return err
}
// Extract the import paths.
cmd.Paths = fs.Args()
return nil
}
// Usage returns the usage message to be printed.
func (cmd *ImportCommand) Usage() string {
return strings.TrimSpace(`
usage: pilosactl import -host HOST -d database -f frame paths
Bulk imports one or more CSV files to a host's database and frame. The bits
of the CSV file are grouped by slice for the most efficient import.
The format of the CSV file is:
BITMAPID,PROFILEID,[TIME]
The file should contain no headers. The TIME column is optional and can be
omitted. If it is present then its format should be YYYY-MM-DDTHH:MM.
`)
}
// Run executes the main program execution.
func (cmd *ImportCommand) Run(ctx context.Context) error {
logger := log.New(cmd.Stderr, "", log.LstdFlags)
@ -104,7 +66,6 @@ func (cmd *ImportCommand) Run(ctx context.Context) error {
} else if len(cmd.Paths) == 0 {
return errors.New("path required")
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {

94
ctl/inspect.go Normal file
View file

@ -0,0 +1,94 @@
package ctl
import (
"context"
"fmt"
"io"
"os"
"syscall"
"text/tabwriter"
"time"
"unsafe"
"github.com/pilosa/pilosa/roaring"
)
// InspectCommand represents a command for inspecting fragment data files.
type InspectCommand struct {
// Path to data file
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewInspectCommand returns a new instance of InspectCommand.
func NewInspectCommand(stdin io.Reader, stdout, stderr io.Writer) *InspectCommand {
return &InspectCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the inspect command.
func (cmd *InspectCommand) Run(ctx context.Context) error {
// Open file handle.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
// Memory map the file.
data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
return err
}
defer syscall.Munmap(data)
// Attach the mmap file to the bitmap.
t := time.Now()
fmt.Fprintf(cmd.Stderr, "unmarshaling bitmap...")
bm := roaring.NewBitmap()
if err := bm.UnmarshalBinary(data); err != nil {
return err
}
fmt.Fprintf(cmd.Stderr, " (%s)\n", time.Since(t))
// Retrieve stats.
t = time.Now()
fmt.Fprintf(cmd.Stderr, "calculating stats...")
info := bm.Info()
fmt.Fprintf(cmd.Stderr, " (%s)\n", time.Since(t))
// Print top-level info.
fmt.Fprintf(cmd.Stdout, "== Bitmap Info ==\n")
fmt.Fprintf(cmd.Stdout, "Containers: %d\n", len(info.Containers))
fmt.Fprintf(cmd.Stdout, "Operations: %d\n", info.OpN)
fmt.Fprintln(cmd.Stdout, "")
// Print info for each container.
fmt.Fprintln(cmd.Stdout, "== Containers ==")
tw := tabwriter.NewWriter(cmd.Stdout, 0, 8, 0, '\t', 0)
fmt.Fprintf(tw, "%s\t%s\t% 8s \t% 8s\t%s\n", "KEY", "TYPE", "N", "ALLOC", "OFFSET")
for _, ci := range info.Containers {
fmt.Fprintf(tw, "%d\t%s\t% 8d \t% 8d \t0x%08x\n",
ci.Key,
ci.Type,
ci.N,
ci.Alloc,
uintptr(ci.Pointer)-uintptr(unsafe.Pointer(&data[0])),
)
}
tw.Flush()
return nil
}

65
ctl/restore.go Normal file
View file

@ -0,0 +1,65 @@
package ctl
import (
"context"
"errors"
"io"
"os"
"github.com/pilosa/pilosa"
)
// RestoreCommand represents a command for restoring a frame from a backup.
type RestoreCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to backup.
Database string
Frame string
// Import file to read from.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewRestoreCommand returns a new instance of RestoreCommand.
func NewRestoreCommand(stdin io.Reader, stdout, stderr io.Writer) *RestoreCommand {
return &RestoreCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the restore command.
func (cmd *RestoreCommand) Run(ctx context.Context) error {
// Validate arguments.
if cmd.Path == "" {
return errors.New("backup file required")
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Open backup file.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Restore backup file to the cluster.
if err := client.RestoreFrom(ctx, f, cmd.Database, cmd.Frame); err != nil {
return err
}
return nil
}

138
ctl/sort.go Normal file
View file

@ -0,0 +1,138 @@
package ctl
import (
"bufio"
"context"
"encoding/csv"
"errors"
"fmt"
"io"
"os"
"sort"
"strconv"
"time"
"github.com/pilosa/pilosa"
)
// SortCommand represents a command for sorting import data.
type SortCommand struct {
// Filename to sort
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewSortCommand returns a new instance of SortCommand.
func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *SortCommand {
return &SortCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the sort command.
func (cmd *SortCommand) Run(ctx context.Context) error {
// Open file for reading.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Read rows as bits.
r := csv.NewReader(f)
r.FieldsPerRecord = -1
a := make([]pilosa.Bit, 0, 1000000)
for {
bitmapID, profileID, timestamp, err := readCSVRow(r)
if err == io.EOF {
break
} else if err == errBlank {
continue
} else if err != nil {
return err
}
a = append(a, pilosa.Bit{BitmapID: bitmapID, ProfileID: profileID, Timestamp: timestamp})
}
// Sort bits by position.
sort.Sort(pilosa.BitsByPos(a))
// Rewrite to STDOUT.
w := bufio.NewWriter(cmd.Stdout)
buf := make([]byte, 0, 1024)
for _, bit := range a {
// Write CSV to buffer.
buf = buf[:0]
buf = strconv.AppendUint(buf, bit.BitmapID, 10)
buf = append(buf, ',')
buf = strconv.AppendUint(buf, bit.ProfileID, 10)
if bit.Timestamp != 0 {
buf = append(buf, ',')
buf = append(buf, time.Unix(0, bit.Timestamp).UTC().Format(pilosa.TimeFormat)...)
}
buf = append(buf, '\n')
// Write to output.
if _, err := w.Write(buf); err != nil {
return err
}
}
// Ensure buffer is flushed before exiting.
if err := w.Flush(); err != nil {
return err
}
return nil
}
// readCSVRow reads a bitmap/profile pair from a CSV row.
func readCSVRow(r *csv.Reader) (bitmapID, profileID uint64, timestamp int64, err error) {
// Read CSV row.
record, err := r.Read()
if err != nil {
return 0, 0, 0, err
}
// Ignore blank rows.
if record[0] == "" {
return 0, 0, 0, errBlank
} else if len(record) < 2 {
return 0, 0, 0, fmt.Errorf("bad column count: %d", len(record))
}
// Parse bitmap id.
bitmapID, err = strconv.ParseUint(record[0], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid bitmap id: %q", record[0])
}
// Parse bitmap id.
profileID, err = strconv.ParseUint(record[1], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid profile id: %q", record[1])
}
// Parse timestamp, if available.
if len(record) > 2 && record[2] != "" {
t, err := time.Parse(pilosa.TimeFormat, record[2])
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid timestamp: %q", record[2])
}
timestamp = t.UnixNano()
}
return bitmapID, profileID, timestamp, nil
}
// errBlank indicates a blank row in a CSV file.
var errBlank = errors.New("blank row")

142
db.go
View file

@ -15,6 +15,11 @@ import (
"github.com/pilosa/pilosa/internal"
)
// Default database settings.
const (
DefaultColumnLabel = "profileID"
)
// DB represents a container for frames.
type DB struct {
mu sync.Mutex
@ -25,6 +30,9 @@ type DB struct {
// This can be overridden by individual frames.
timeQuantum TimeQuantum
// Label used for referring to columns in database.
columnLabel string
// Frames by name.
frames map[string]*Frame
@ -40,7 +48,12 @@ type DB struct {
}
// NewDB returns a new instance of DB.
func NewDB(path, name string) *DB {
func NewDB(path, name string) (*DB, error) {
err := ValidateName(name)
if err != nil {
return nil, err
}
return &DB{
path: path,
name: name,
@ -49,9 +62,11 @@ func NewDB(path, name string) *DB {
profileAttrStore: NewAttrStore(filepath.Join(path, "data")),
columnLabel: DefaultColumnLabel,
stats: NopStatsClient,
LogOutput: ioutil.Discard,
}
}, nil
}
// Name returns name of the database.
@ -63,6 +78,33 @@ func (db *DB) Path() string { return db.path }
// ProfileAttrStore returns the storage for profile attributes.
func (db *DB) ProfileAttrStore() *AttrStore { return db.profileAttrStore }
// SetColumnLabel sets the column label. Persists to meta file on update.
func (db *DB) SetColumnLabel(v string) error {
db.mu.Lock()
defer db.mu.Unlock()
// Ignore if no change occurred.
if v == "" || db.columnLabel == v {
return nil
}
// Persist meta data to disk on change.
db.columnLabel = v
if err := db.saveMeta(); err != nil {
return err
}
return nil
}
// ColumnLabel returns the column label.
func (db *DB) ColumnLabel() string {
db.mu.Lock()
v := db.columnLabel
db.mu.Unlock()
return v
}
// Open opens and initializes the database.
func (db *DB) Open() error {
// Ensure the path exists.
@ -104,7 +146,10 @@ func (db *DB) openFrames() error {
continue
}
fr := db.newFrame(db.FramePath(filepath.Base(fi.Name())), filepath.Base(fi.Name()))
fr, err := db.newFrame(db.FramePath(filepath.Base(fi.Name())), filepath.Base(fi.Name()))
if err != nil {
return ErrName
}
if err := fr.Open(); err != nil {
return fmt.Errorf("open frame: name=%s, err=%s", fr.Name(), err)
}
@ -123,6 +168,7 @@ func (db *DB) loadMeta() error {
buf, err := ioutil.ReadFile(filepath.Join(db.path, "meta"))
if os.IsNotExist(err) {
db.timeQuantum = ""
db.columnLabel = DefaultColumnLabel
return nil
} else if err != nil {
return err
@ -134,6 +180,7 @@ func (db *DB) loadMeta() error {
// Copy metadata fields.
db.timeQuantum = TimeQuantum(pb.TimeQuantum)
db.columnLabel = pb.ColumnLabel
return nil
}
@ -141,7 +188,10 @@ func (db *DB) loadMeta() error {
// saveMeta writes meta data for the database.
func (db *DB) saveMeta() error {
// Marshal metadata.
buf, err := proto.Marshal(&internal.DB{TimeQuantum: string(db.timeQuantum)})
buf, err := proto.Marshal(&internal.DB{
TimeQuantum: string(db.timeQuantum),
ColumnLabel: db.columnLabel,
})
if err != nil {
return err
}
@ -244,28 +294,52 @@ func (db *DB) Frames() []*Frame {
return a
}
// CreateFrameIfNotExists returns a frame in the database by name.
func (db *DB) CreateFrameIfNotExists(name string) (*Frame, error) {
// CreateFrame creates a frame.
func (db *DB) CreateFrame(name string, opt FrameOptions) (*Frame, error) {
db.mu.Lock()
defer db.mu.Unlock()
return db.createFrameIfNotExists(name)
// Ensure frame doesn't already exist.
if db.frames[name] != nil {
return nil, ErrFrameExists
}
return db.createFrame(name, opt)
}
func (db *DB) createFrameIfNotExists(name string) (*Frame, error) {
if name == "" {
return nil, errors.New("frame name required")
}
// CreateFrameIfNotExists creates a frame with the given options if it doesn't exist.
func (db *DB) CreateFrameIfNotExists(name string, opt FrameOptions) (*Frame, error) {
db.mu.Lock()
defer db.mu.Unlock()
// Find frame in cache first.
if f := db.frames[name]; f != nil {
return f, nil
}
// Initialize and open frame.
f := db.newFrame(db.FramePath(name), name)
return db.createFrame(name, opt)
}
func (db *DB) createFrame(name string, opt FrameOptions) (*Frame, error) {
if name == "" {
return nil, errors.New("frame name required")
}
// Initialize frame.
f, err := db.newFrame(db.FramePath(name), name)
if err != nil {
return nil, err
}
// Open frame.
if err := f.Open(); err != nil {
return nil, err
}
// Update options.
f.SetRowLabel(opt.RowLabel)
// Add to database's frame lookup.
db.frames[name] = f
db.stats.Count("frameN", 1)
@ -273,11 +347,14 @@ func (db *DB) createFrameIfNotExists(name string) (*Frame, error) {
return f, nil
}
func (db *DB) newFrame(path, name string) *Frame {
f := NewFrame(path, db.name, name)
func (db *DB) newFrame(path, name string) (*Frame, error) {
f, err := NewFrame(path, db.name, name)
if err != nil {
return nil, err
}
f.LogOutput = db.LogOutput
f.stats = db.stats.WithTags(fmt.Sprintf("frame:%s", name))
return f
return f, nil
}
// DeleteFrame removes a frame from the database.
@ -309,22 +386,13 @@ func (db *DB) DeleteFrame(name string) error {
return nil
}
// CreateFragmentIfNotExists returns a fragment in the database by name/slice.
func (db *DB) CreateFragmentIfNotExists(name string, slice uint64) (*Fragment, error) {
f, err := db.CreateFrameIfNotExists(name)
if err != nil {
return nil, err
}
return f.CreateFragmentIfNotExists(slice)
}
// SetBit sets a bit for a given profile & bitmap.
// If a timestamp is specified then set all bits for the different quantum units.
func (db *DB) SetBit(name string, bitmapID, profileID uint64, t *time.Time) (changed bool, err error) {
// Read frame.
f, err := db.CreateFrameIfNotExists(name)
if err != nil {
return changed, err
f := db.Frame(name)
if f == nil {
return changed, ErrFrameNotFound
}
// If this is a non-time bit then simply set the bit on the frame.
@ -342,8 +410,9 @@ func (db *DB) SetBit(name string, bitmapID, profileID uint64, t *time.Time) (cha
}
// If a timestamp is specified then set bits across all frames for the quantum.
opt := f.Options()
for _, subname := range FramesByTime(name, *t, q) {
f, err := db.CreateFrameIfNotExists(subname)
f, err := db.CreateFrameIfNotExists(subname, opt)
if err != nil {
return changed, err
}
@ -360,9 +429,9 @@ func (db *DB) SetBit(name string, bitmapID, profileID uint64, t *time.Time) (cha
// Import bulk imports data.
func (db *DB) Import(name string, bitmapIDs, profileIDs []uint64, timestamps []*time.Time) error {
// Read frame.
f, err := db.CreateFrameIfNotExists(name)
if err != nil {
return err
f := db.Frame(name)
if f == nil {
return ErrFrameNotFound
}
// Determine quantum if timestamps are set.
@ -405,12 +474,12 @@ func (db *DB) Import(name string, bitmapIDs, profileIDs []uint64, timestamps []*
// Import into each fragment.
for key, data := range dataByFragment {
f, err := db.CreateFragmentIfNotExists(key.Frame, key.Slice)
frag, err := f.CreateFragmentIfNotExists(key.Slice)
if err != nil {
return err
}
if err := f.Import(data.BitmapIDs, data.ProfileIDs); err != nil {
if err := frag.Import(data.BitmapIDs, data.ProfileIDs); err != nil {
return err
}
}
@ -472,6 +541,11 @@ func (db *DB) SetRemoteMaxSlice(newmax uint64) {
db.remoteMaxSlice = newmax
}
// DBOptions represents options to set when initializing a db.
type DBOptions struct {
ColumnLabel string `json:"columnLabel,omitempty"`
}
// hasTime returns true if a contains a non-nil time.
func hasTime(a []*time.Time) bool {
for _, t := range a {

30
db_test.go
View file

@ -15,7 +15,7 @@ func TestDB_CreateFrameIfNotExists(t *testing.T) {
defer db.Close()
// Create frame.
f, err := db.CreateFrameIfNotExists("f")
f, err := db.CreateFrameIfNotExists("f", pilosa.FrameOptions{})
if err != nil {
t.Fatal(err)
} else if f == nil {
@ -23,7 +23,7 @@ func TestDB_CreateFrameIfNotExists(t *testing.T) {
}
// Retrieve existing frame.
other, err := db.CreateFrameIfNotExists("f")
other, err := db.CreateFrameIfNotExists("f", pilosa.FrameOptions{})
if err != nil {
t.Fatal(err)
} else if f != other {
@ -41,7 +41,7 @@ func TestDB_DeleteFrame(t *testing.T) {
defer db.Close()
// Create frame.
if _, err := db.CreateFrameIfNotExists("f"); err != nil {
if _, err := db.CreateFrameIfNotExists("f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
@ -89,7 +89,11 @@ func NewDB() *DB {
if err != nil {
panic(err)
}
return &DB{DB: pilosa.NewDB(path, "d")}
db, err := pilosa.NewDB(path, "d")
if err != nil {
panic(err)
}
return &DB{DB: db}
}
// MustOpenDB returns a new, opened database at a temporary path. Panic on error.
@ -109,12 +113,16 @@ func (db *DB) Close() error {
// Reopen closes the database and reopens it.
func (db *DB) Reopen() error {
var err error
if err := db.DB.Close(); err != nil {
return err
}
path, name := db.Path(), db.Name()
db.DB = pilosa.NewDB(path, name)
db.DB, err = pilosa.NewDB(path, name)
if err != nil {
return err
}
if err := db.Open(); err != nil {
return err
@ -130,3 +138,15 @@ func (db *DB) MustSetBit(name string, bitmapID, profileID uint64, t *time.Time)
}
return changed
}
// Ensure database can delete a frame.
func TestDB_InvalidName(t *testing.T) {
path, err := ioutil.TempDir("", "pilosa-db-")
if err != nil {
panic(err)
}
db, err := pilosa.NewDB(path, "ABC")
if db != nil {
t.Fatalf("unexpected db name %s", db)
}
}

182
executor.go
View file

@ -129,12 +129,14 @@ func (e *Executor) executeBitmapCall(ctx context.Context, db string, c *pql.Call
// Attach bitmap attributes for Bitmap() calls.
bm, _ := other.(*Bitmap)
if c.Name == "Bitmap" {
id, _ := c.Args["id"].(uint64)
frame, _ := c.Args["frame"].(string)
fr := e.Index.Frame(db, frame)
if fr != nil {
attrs, err := fr.BitmapAttrStore().Attrs(id)
rowLabel := fr.RowLabel()
rowID, _ := c.Args[rowLabel].(uint64)
attrs, err := fr.BitmapAttrStore().Attrs(rowID)
if err != nil {
return nil, err
}
@ -272,6 +274,9 @@ func (e *Executor) executeTopNSlice(ctx context.Context, db string, c *pql.Call,
// executeDifferenceSlice executes a difference() call for a local slice.
func (e *Executor) executeDifferenceSlice(ctx context.Context, db string, c *pql.Call, slice uint64) (*Bitmap, error) {
var other *Bitmap
if len(c.Children) == 0 {
return nil, fmt.Errorf("empty Difference query is currently not supported")
}
for i, input := range c.Children {
bm, err := e.executeBitmapCallSlice(ctx, db, input, slice)
if err != nil {
@ -289,22 +294,35 @@ func (e *Executor) executeDifferenceSlice(ctx context.Context, db string, c *pql
}
func (e *Executor) executeBitmapSlice(ctx context.Context, db string, c *pql.Call, slice uint64) (*Bitmap, error) {
id, _ := c.Args["id"].(uint64)
frame, _ := c.Args["frame"].(string)
if frame == "" {
frame = DefaultFrame
}
f := e.Index.Fragment(db, frame, slice)
f := e.Index.Frame(db, frame)
if f == nil {
return nil, ErrFrameNotFound
}
rowLabel := f.RowLabel()
rowID, ok := c.Args[rowLabel].(uint64)
if !ok {
return nil, fmt.Errorf("Bitmap() field required: %s", rowLabel)
}
frag := e.Index.Fragment(db, frame, slice)
if frag == nil {
return NewBitmap(), nil
}
return f.Bitmap(id), nil
return frag.Bitmap(rowID), nil
}
// executeIntersectSlice executes a intersect() call for a local slice.
func (e *Executor) executeIntersectSlice(ctx context.Context, db string, c *pql.Call, slice uint64) (*Bitmap, error) {
var other *Bitmap
if len(c.Children) == 0 {
return nil, fmt.Errorf("empty Intersect query is currently not supported")
}
for i, input := range c.Children {
bm, err := e.executeBitmapCallSlice(ctx, db, input, slice)
if err != nil {
@ -323,7 +341,21 @@ func (e *Executor) executeIntersectSlice(ctx context.Context, db string, c *pql.
// executeRangeSlice executes a range() call for a local slice.
func (e *Executor) executeRangeSlice(ctx context.Context, db string, c *pql.Call, slice uint64) (*Bitmap, error) {
id, _ := c.Args["id"].(uint64)
// Parse frame, use default if unset.
frame, _ := c.Args["frame"].(string)
if frame == "" {
frame = DefaultFrame
}
// Retrieve base frame.
f := e.Index.Frame(db, frame)
if f == nil {
return nil, ErrFrameNotFound
}
rowLabel := f.RowLabel()
// Read row id.
rowID, _ := c.Args[rowLabel].(uint64)
// Parse start time.
startTimeStr, ok := c.Args["start"].(string)
@ -345,18 +377,6 @@ func (e *Executor) executeRangeSlice(ctx context.Context, db string, c *pql.Call
return nil, errors.New("cannot parse Range() end time")
}
// Parse frame, use default if unset.
frame, _ := c.Args["frame"].(string)
if frame == "" {
frame = DefaultFrame
}
// Retrieve base frame.
f := e.Index.Frame(db, frame)
if f == nil {
return &Bitmap{}, nil
}
// If no quantum exists then return an empty bitmap.
q := f.TimeQuantum()
if q == "" {
@ -370,14 +390,14 @@ func (e *Executor) executeRangeSlice(ctx context.Context, db string, c *pql.Call
if f == nil {
continue
}
bm = bm.Union(f.Bitmap(id))
bm = bm.Union(f.Bitmap(rowID))
}
return bm, nil
}
// executeUnionSlice executes a union() call for a local slice.
func (e *Executor) executeUnionSlice(ctx context.Context, db string, c *pql.Call, slice uint64) (*Bitmap, error) {
var other *Bitmap
other := NewBitmap()
for i, input := range c.Children {
bm, err := e.executeBitmapCallSlice(ctx, db, input, slice)
if err != nil {
@ -439,27 +459,42 @@ func (e *Executor) executeClearBit(ctx context.Context, db string, c *pql.Call,
return false, errors.New("ClearBit() frame required")
}
id, ok := c.Args["id"].(uint64)
// Lookup column label.
d := e.Index.DB(db)
if d == nil {
return false, nil
}
columnLabel := d.ColumnLabel()
// Lookup row label.
f := e.Index.Frame(db, frame)
if f == nil {
return false, nil
}
rowLabel := f.RowLabel()
// Read row & column ids.
rowID, ok := c.Args[rowLabel].(uint64)
if !ok {
return false, errors.New("ClearBit() id required")
return false, fmt.Errorf("ClearBit() field required: %s", rowLabel)
}
profileID, ok := c.Args["profileID"].(uint64)
colID, ok := c.Args[columnLabel].(uint64)
if !ok {
return false, errors.New("ClearBit() profileID required")
return false, fmt.Errorf("ClearBit() field required: %s", columnLabel)
}
slice := profileID / SliceWidth
slice := colID / SliceWidth
ret := false
for _, node := range e.Cluster.FragmentNodes(db, slice) {
// Update locally if host matches.
if node.Host == e.Host {
f := e.Index.Fragment(db, frame, slice)
if f == nil {
frag := e.Index.Fragment(db, frame, slice)
if frag == nil {
return false, nil
}
val, err := f.ClearBit(id, profileID)
val, err := frag.ClearBit(rowID, colID)
if err != nil {
return false, err
} else if val {
@ -486,40 +521,56 @@ func (e *Executor) executeClearBit(ctx context.Context, db string, c *pql.Call,
func (e *Executor) executeSetBit(ctx context.Context, db string, c *pql.Call, opt *ExecOptions) (bool, error) {
frame, ok := c.Args["frame"].(string)
if !ok {
return false, errors.New("SetBit() frame required")
return false, errors.New("SetBit() field required: frame")
}
id, ok := c.Args["id"].(uint64)
if !ok {
return false, errors.New("SetBit() id required")
// Retrieve frame.
d := e.Index.DB(db)
if d == nil {
return false, ErrFrameNotFound
}
f := d.Frame(frame)
if f == nil {
return false, ErrFrameNotFound
}
profileID, ok := c.Args["profileID"].(uint64)
// Retrieve labels.
columnLabel := d.ColumnLabel()
rowLabel := f.RowLabel()
// Read fields using labels.
rowID, ok := c.Args[rowLabel].(uint64)
if !ok {
return false, errors.New("SetBit() profileID required")
return false, fmt.Errorf("SetBit() field required: %s", rowLabel)
}
colID, ok := c.Args[columnLabel].(uint64)
if !ok {
return false, fmt.Errorf("SetBit() field required: %s", columnLabel)
}
var timestamp *time.Time
sTimestamp, ok := c.Args["timestamp"].(string)
if ok {
t, err := time.Parse("2006-01-02T15:04:05", sTimestamp)
t, err := time.Parse(TimeFormat, sTimestamp)
if err != nil {
return false, fmt.Errorf("invalid date: %s", sTimestamp)
}
timestamp = &t
}
slice := profileID / SliceWidth
slice := colID / SliceWidth
ret := false
for _, node := range e.Cluster.FragmentNodes(db, slice) {
// Update locally if host matches.
if node.Host == e.Host {
db, err := e.Index.CreateDBIfNotExists(db)
if err != nil {
return false, fmt.Errorf("db: %s", err)
d := e.Index.DB(db)
if d == nil {
return false, ErrDatabaseNotFound
}
val, err := db.SetBit(frame, id, profileID, timestamp)
val, err := d.SetBit(frame, rowID, colID, timestamp)
if err != nil {
return false, err
} else if val {
@ -550,24 +601,26 @@ func (e *Executor) executeSetBitmapAttrs(ctx context.Context, db string, c *pql.
return errors.New("SetBitmapAttrs() frame required")
}
id, ok := c.Args["id"].(uint64)
// Retrieve frame.
frame := e.Index.Frame(db, frameName)
if frame == nil {
return ErrFrameNotFound
}
rowLabel := frame.RowLabel()
// Parse labels.
rowID, ok := c.Args[rowLabel].(uint64)
if !ok {
return errors.New("SetBitmapAttrs() id required")
return fmt.Errorf("SetBitmapAttrs() field required: %s", rowLabel)
}
// Copy args and remove reserved fields.
attrs := pql.CopyArgs(c.Args)
delete(attrs, "frame")
delete(attrs, "id")
// Retrieve frame.
frame, err := e.Index.CreateFrameIfNotExists(db, frameName)
if err != nil {
return err
}
delete(attrs, rowLabel)
// Set attributes.
if err := frame.BitmapAttrStore().SetAttrs(id, attrs); err != nil {
if err := frame.BitmapAttrStore().SetAttrs(rowID, attrs); err != nil {
return err
}
@ -606,15 +659,22 @@ func (e *Executor) executeBulkSetBitmapAttrs(ctx context.Context, db string, cal
return nil, errors.New("SetBitmapAttrs() frame required")
}
id, ok := c.Args["id"].(uint64)
// Retrieve frame.
f := e.Index.Frame(db, frame)
if f == nil {
return nil, ErrFrameNotFound
}
rowLabel := f.RowLabel()
rowID, ok := c.Args[rowLabel].(uint64)
if !ok {
return nil, errors.New("SetBitmapAttrs() id required")
return nil, fmt.Errorf("SetBitmapAttrs() field required: %s", rowLabel)
}
// Copy args and remove reserved fields.
attrs := pql.CopyArgs(c.Args)
delete(attrs, "frame")
delete(attrs, "id")
delete(attrs, rowLabel)
// Create frame group, if not exists.
frameMap := m[frame]
@ -624,9 +684,9 @@ func (e *Executor) executeBulkSetBitmapAttrs(ctx context.Context, db string, cal
}
// Set or merge attributes.
attr := frameMap[id]
attr := frameMap[rowID]
if attr == nil {
frameMap[id] = cloneAttrs(attrs)
frameMap[rowID] = cloneAttrs(attrs)
} else {
for k, v := range attrs {
attr[k] = v
@ -637,9 +697,9 @@ func (e *Executor) executeBulkSetBitmapAttrs(ctx context.Context, db string, cal
// Bulk insert attributes by frame.
for name, frameMap := range m {
// Retrieve frame.
frame, err := e.Index.CreateFrameIfNotExists(db, name)
if err != nil {
return nil, err
frame := e.Index.Frame(db, name)
if frame == nil {
return nil, ErrFrameNotFound
}
// Set attributes.
@ -686,9 +746,9 @@ func (e *Executor) executeSetProfileAttrs(ctx context.Context, db string, c *pql
delete(attrs, "id")
// Retrieve database.
d, err := e.Index.CreateDBIfNotExists(db)
if err != nil {
return err
d := e.Index.DB(db)
if d == nil {
return ErrDatabaseNotFound
}
// Set attributes.

73
executor_test.go
View file

@ -50,6 +50,18 @@ func TestExecutor_Execute_Difference(t *testing.T) {
}
}
// Ensure an empty difference query behaves properly.
func TestExecutor_Execute_Empty_Difference(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
idx.MustCreateFragmentIfNotExists("d", "general", 0).MustSetBits(10, 1)
e := NewExecutor(idx.Index, NewCluster(1))
if res, err := e.Execute(context.Background(), "d", MustParse(`Difference()`), nil, nil); err == nil {
t.Fatalf("Empty Difference query should give error, but got %v", res)
}
}
// Ensure an intersect query can be executed.
func TestExecutor_Execute_Intersect(t *testing.T) {
idx := MustOpenIndex()
@ -70,6 +82,17 @@ func TestExecutor_Execute_Intersect(t *testing.T) {
}
}
// Ensure an empty intersect query behaves properly.
func TestExecutor_Execute_Empty_Intersect(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
e := NewExecutor(idx.Index, NewCluster(1))
if res, err := e.Execute(context.Background(), "d", MustParse(`Intersect()`), nil, nil); err == nil {
t.Fatalf("Empty Intersect query should give error, but got %v", res)
}
}
// Ensure a union query can be executed.
func TestExecutor_Execute_Union(t *testing.T) {
idx := MustOpenIndex()
@ -89,6 +112,20 @@ func TestExecutor_Execute_Union(t *testing.T) {
}
}
// Ensure an empty union query behaves properly.
func TestExecutor_Execute_Empty_Union(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
idx.MustCreateFragmentIfNotExists("d", "general", 0).MustSetBits(10, 0)
e := NewExecutor(idx.Index, NewCluster(1))
if res, err := e.Execute(context.Background(), "d", MustParse(`Union()`), nil, nil); err != nil {
t.Fatal(err)
} else if bits := res[0].(*pilosa.Bitmap).Bits(); !reflect.DeepEqual(bits, []uint64{}) {
t.Fatalf("unexpected bits: %+v", bits)
}
}
// Ensure a count query can be executed.
func TestExecutor_Execute_Count(t *testing.T) {
idx := MustOpenIndex()
@ -141,6 +178,14 @@ func TestExecutor_Execute_SetBitmapAttrs(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
// Create frames.
db := idx.MustCreateDBIfNotExists("d", pilosa.DBOptions{})
if _, err := db.CreateFrameIfNotExists("f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
} else if _, err := db.CreateFrameIfNotExists("xxx", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Set two fields on f/10.
// Also set fields on other bitmaps and frames to test isolation.
e := NewExecutor(idx.Index, NewCluster(1))
@ -150,7 +195,7 @@ func TestExecutor_Execute_SetBitmapAttrs(t *testing.T) {
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBitmapAttrs(id=200, frame=f, YYY=1)`), nil, nil); err != nil {
t.Fatal(err)
}
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBitmapAttrs(id=10, frame=XXX, YYY=1)`), nil, nil); err != nil {
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBitmapAttrs(id=10, frame=xxx, YYY=1)`), nil, nil); err != nil {
t.Fatal(err)
}
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBitmapAttrs(id=10, frame=f, baz=123, bat=true)`), nil, nil); err != nil {
@ -333,8 +378,16 @@ func TestExecutor_Execute_Range(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
db := idx.MustCreateDBIfNotExists("d")
// Create database.
db := idx.MustCreateDBIfNotExists("d", pilosa.DBOptions{})
db.SetTimeQuantum(pilosa.TimeQuantum("YMDH"))
// Create frame.
if _, err := db.CreateFrameIfNotExists("f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Set bits.
db.MustSetBit("f", 1, 2, MustParseTimePtr("1999-12-31 00:00"))
db.MustSetBit("f", 1, 3, MustParseTimePtr("2000-01-01 00:00"))
db.MustSetBit("f", 1, 4, MustParseTimePtr("2000-01-02 00:00"))
@ -450,6 +503,11 @@ func TestExecutor_Execute_Remote_SetBit(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
// Create frame.
if _, err := idx.MustCreateDBIfNotExists("d", pilosa.DBOptions{}).CreateFrame("f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
e := NewExecutor(idx.Index, c)
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBit(id=10, frame=f, profileID=2)`), nil, nil); err != nil {
t.Fatal(err)
@ -479,7 +537,7 @@ func TestExecutor_Execute_Remote_SetBit_With_Timestamp(t *testing.T) {
s.Handler.Executor.ExecuteFn = func(ctx context.Context, db string, query *pql.Query, slices []uint64, opt *pilosa.ExecOptions) ([]interface{}, error) {
if db != `d` {
t.Fatalf("unexpected db: %s", db)
} else if query.String() != `SetBit(frame="f", id=10, profileID=2, timestamp="2016-12-11T10:09:07")` {
} else if query.String() != `SetBit(frame="f", id=10, profileID=2, timestamp="2016-12-11T10:09")` {
t.Fatalf("unexpected query: %s", query.String())
}
remoteCalled = true
@ -489,7 +547,7 @@ func TestExecutor_Execute_Remote_SetBit_With_Timestamp(t *testing.T) {
// Create local executor data.
idx := MustOpenIndex()
defer idx.Close()
idx.CreateDBIfNotExists("d")
idx.CreateDBIfNotExists("d", pilosa.DBOptions{})
oldQuantum := idx.DB("d").TimeQuantum()
defer func() {
// restore db quantum
@ -498,8 +556,13 @@ func TestExecutor_Execute_Remote_SetBit_With_Timestamp(t *testing.T) {
// need to set the quantum otherwise SetBit fails silently
idx.DB("d").SetTimeQuantum("Y")
// Create frame.
if _, err := idx.MustCreateDBIfNotExists("d", pilosa.DBOptions{}).CreateFrame("f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
e := NewExecutor(idx.Index, c)
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBit(id=10, frame=f, profileID=2, timestamp="2016-12-11T10:09:07")`), nil, nil); err != nil {
if _, err := e.Execute(context.Background(), "d", MustParse(`SetBit(id=10, frame=f, profileID=2, timestamp="2016-12-11T10:09")`), nil, nil); err != nil {
t.Fatal(err)
}

68
frame.go
View file

@ -18,6 +18,11 @@ const (
FrameSuffixRank = ".n"
)
// Default frame settings.
const (
DefaultRowLabel = "id"
)
// Frame represents a container for fragments.
type Frame struct {
mu sync.Mutex
@ -34,11 +39,19 @@ type Frame struct {
stats StatsClient
// Label used for referring to a row.
rowLabel string
LogOutput io.Writer
}
// NewFrame returns a new instance of frame.
func NewFrame(path, db, name string) *Frame {
func NewFrame(path, db, name string) (*Frame, error) {
err := ValidateName(name)
if err != nil {
return nil, err
}
return &Frame{
path: path,
db: db,
@ -49,8 +62,10 @@ func NewFrame(path, db, name string) *Frame {
stats: NopStatsClient,
rowLabel: DefaultRowLabel,
LogOutput: ioutil.Discard,
}
}, nil
}
// Name returns the name the frame was initialized with.
@ -79,6 +94,43 @@ func (f *Frame) MaxSlice() uint64 {
return max
}
// SetRowLabel sets the row labels. Persists to meta file on update.
func (f *Frame) SetRowLabel(v string) error {
f.mu.Lock()
defer f.mu.Unlock()
// Ignore if no change occurred.
if v == "" || f.rowLabel == v {
return nil
}
// Persist meta data to disk on change.
f.rowLabel = v
if err := f.saveMeta(); err != nil {
return err
}
return nil
}
// RowLabel returns the row label.
func (f *Frame) RowLabel() string {
f.mu.Lock()
v := f.rowLabel
f.mu.Unlock()
return v
}
// Options returns all options for this frame.
func (f *Frame) Options() FrameOptions {
f.mu.Lock()
opt := FrameOptions{
RowLabel: f.rowLabel,
}
f.mu.Unlock()
return opt
}
// Open opens and initializes the frame.
func (f *Frame) Open() error {
if err := func() error {
@ -153,6 +205,7 @@ func (f *Frame) loadMeta() error {
buf, err := ioutil.ReadFile(filepath.Join(f.path, "meta"))
if os.IsNotExist(err) {
f.timeQuantum = ""
f.rowLabel = DefaultRowLabel
return nil
} else if err != nil {
return err
@ -164,6 +217,7 @@ func (f *Frame) loadMeta() error {
// Copy metadata fields.
f.timeQuantum = TimeQuantum(pb.TimeQuantum)
f.rowLabel = pb.RowLabel
return nil
}
@ -171,7 +225,10 @@ func (f *Frame) loadMeta() error {
// saveMeta writes meta data for the frame.
func (f *Frame) saveMeta() error {
// Marshal metadata.
buf, err := proto.Marshal(&internal.Frame{TimeQuantum: string(f.timeQuantum)})
buf, err := proto.Marshal(&internal.Frame{
TimeQuantum: string(f.timeQuantum),
RowLabel: f.rowLabel,
})
if err != nil {
return err
}
@ -318,3 +375,8 @@ type frameInfoSlice []*FrameInfo
func (p frameInfoSlice) Swap(i, j int) { p[i], p[j] = p[j], p[i] }
func (p frameInfoSlice) Len() int { return len(p) }
func (p frameInfoSlice) Less(i, j int) bool { return p[i].Name < p[j].Name }
// FrameOptions represents options to set when initializing a frame.
type FrameOptions struct {
RowLabel string `json:"rowLabel,omitempty"`
}

25
frame_test.go
View file

@ -65,8 +65,11 @@ func NewFrame() *Frame {
if err != nil {
panic(err)
}
return &Frame{Frame: pilosa.NewFrame(path, "d", "f")}
frame, err := pilosa.NewFrame(path, "d", "f")
if err != nil {
panic(err)
}
return &Frame{Frame: frame}
}
// MustOpenFrame returns a new, opened frame at a temporary path. Panic on error.
@ -86,15 +89,31 @@ func (f *Frame) Close() error {
// Reopen closes the database and reopens it.
func (f *Frame) Reopen() error {
var err error
if err := f.Frame.Close(); err != nil {
return err
}
path, db, name := f.Path(), f.DB(), f.Name()
f.Frame = pilosa.NewFrame(path, db, name)
f.Frame, err = pilosa.NewFrame(path, db, name)
if err != nil {
return err
}
if err := f.Open(); err != nil {
return err
}
return nil
}
// NewFrame returns a new instance of Frame d/0.
func TestFrame_NameRestriction(t *testing.T) {
path, err := ioutil.TempDir("", "pilosa-frame-")
if err != nil {
panic(err)
}
frame, err := pilosa.NewFrame(path, "d", "ABC")
if frame != nil {
t.Fatalf("unexpected frame name %s", err)
}
}

88
glide.lock generated
View file

@ -1,34 +1,6 @@
hash: d11cedeb5b4d17ad5bc6d13441cfbb8769c74fbb5fe33839dd0988ee4e73cd4b
updated: 2017-01-12T14:02:12.015478278-06:00
hash: 7de62dbaf3cc1dc4959f4f6d8213102cb182b4dd7a87b3ac29260ad6bc1b0cef
updated: 2017-03-03T12:25:48.088390296-06:00
imports:
- name: github.com/aws/aws-sdk-go
version: 63ce630574a5ec05ecd8e8de5cea16332a5a684d
subpackages:
- aws
- aws/awserr
- aws/awsutil
- aws/client
- aws/client/metadata
- aws/corehandlers
- aws/credentials
- aws/credentials/ec2rolecreds
- aws/credentials/endpointcreds
- aws/credentials/stscreds
- aws/defaults
- aws/ec2metadata
- aws/endpoints
- aws/request
- aws/session
- aws/signer/v4
- private/protocol
- private/protocol/query
- private/protocol/query/queryutil
- private/protocol/rest
- private/protocol/restxml
- private/protocol/xml/xmlutil
- private/waiter
- service/s3
- service/sts
- name: github.com/boltdb/bolt
version: 4b1ebc1869ad66568b313d0dc410e2be72670dda
- name: github.com/BurntSushi/toml
@ -41,8 +13,8 @@ imports:
version: 346938d642f2ec3594ed81d874461961cd0faa76
subpackages:
- spew
- name: github.com/go-ini/ini
version: 6f66b0e091edb3c7b380f7c4f0f884274d550b67
- name: github.com/fsnotify/fsnotify
version: 7d7316ed6e1ed2de075aab8dfc76de5d158d66e1
- name: github.com/gogo/protobuf
version: a9cd0c35b97daf74d0ebf3514c5254814b2703b4
subpackages:
@ -52,23 +24,55 @@ imports:
subpackages:
- lru
- name: github.com/golang/protobuf
version: 8ee79997227bf9b34611aee7946ae64735e6fd93
version: 888eb0692c857ec880338addf316bd662d5e630e
subpackages:
- proto
- name: github.com/jmespath/go-jmespath
version: bd40a432e4c76585ef6b72d3fd96fb9b6dc7b68d
- name: github.com/hashicorp/hcl
version: 630949a3c5fa3c613328e1b8256052cbc2327c9b
subpackages:
- hcl/ast
- hcl/parser
- hcl/scanner
- hcl/strconv
- hcl/token
- json/parser
- json/scanner
- json/token
- name: github.com/inconshreveable/mousetrap
version: 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75
- name: github.com/magiconair/properties
version: b3b15ef068fd0b17ddf408a23669f20811d194d2
- name: github.com/mitchellh/mapstructure
version: db1efb556f84b25a0a13a04aad883943538ad2e0
- name: github.com/pelletier/go-buffruneio
version: c37440a7cf42ac63b919c752ca73a85067e05992
- name: github.com/pelletier/go-toml
version: 13d49d4606eb801b8f01ae542b4afc4c6ee3d84a
- name: github.com/satori/go.uuid
version: 879c5887cd475cd7864858769793b2ceb0d44feb
- name: golang.org/x/crypto
version: c3b1d0d6d8690eaebe3064711b026770cc37efa3
- name: github.com/spf13/afero
version: 9be650865eab0c12963d8753212f4f9c66cdcf12
subpackages:
- curve25519
- ed25519
- ed25519/internal/edwards25519
- ssh
- ssh/agent
- mem
- name: github.com/spf13/cast
version: 4f1683a2242a92e62d6ff705a30e435cbf2b50a3
- name: github.com/spf13/cobra
version: fcd0c5a1df88f5d6784cb4feead962c3f3d0b66c
- name: github.com/spf13/jwalterweatherman
version: fa7ca7e836cf3a8bb4ebf799f472c12d7e903d66
- name: github.com/spf13/pflag
version: 9ff6c6923cfffbcd502984b8e0c80539a94968b7
- name: github.com/spf13/viper
version: 7538d73b4eb9511d85a9f1dfef202eeb8ac260f4
- name: golang.org/x/sys
version: c200b10b5d5e122be351b67af224adc6128af5bf
subpackages:
- unix
- name: golang.org/x/text
version: 5a42fa2464759cbb7ee0af9de00b54d69f09a29c
subpackages:
- transform
- unicode/norm
- name: gopkg.in/yaml.v2
version: a3f3340b5840cee44f372bddb5880fcbc419b46a
testImports: []

4
glide.yaml
View file

@ -27,5 +27,5 @@ import:
- package: github.com/golang/protobuf
- package: github.com/satori/go.uuid
version: ^1.1.0
- package: github.com/aws/aws-sdk-go
version: ^1.6.10
- package: github.com/spf13/cobra
- package: github.com/spf13/viper

133
handler.go
View file

@ -108,6 +108,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
case "/db":
switch r.Method {
case "POST":
h.handlePostDB(w, r)
case "DELETE":
h.handleDeleteDB(w, r)
default:
@ -129,6 +131,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
}
case "/frame":
switch r.Method {
case "POST":
h.handlePostFrame(w, r)
case "DELETE":
h.handleDeleteFrame(w, r)
default:
@ -302,6 +306,38 @@ type sliceMaxResponse struct {
MaxSlices map[string]uint64 `json:"MaxSlices"`
}
// handlePostDB handles POST /db request.
func (h *Handler) handlePostDB(w http.ResponseWriter, r *http.Request) {
// Decode request.
var req postDBRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Create database.
_, err := h.Index.CreateDB(req.DB, req.Options)
if err == ErrDatabaseExists {
http.Error(w, err.Error(), http.StatusConflict)
return
} else if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Encode response.
if err := json.NewEncoder(w).Encode(postDBResponse{}); err != nil {
h.logger().Printf("response encoding error: %s", err)
}
}
type postDBRequest struct {
DB string `json:"db"`
Options DBOptions `json:"options"`
}
type postDBResponse struct{}
// handleDeleteDB handles DELETE /db request.
func (h *Handler) handleDeleteDB(w http.ResponseWriter, r *http.Request) {
// Decode request.
@ -346,9 +382,9 @@ func (h *Handler) handlePatchDBTimeQuantum(w http.ResponseWriter, r *http.Reques
}
// Retrieve database by name.
db, err := h.Index.CreateDBIfNotExists(req.DB)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
db := h.Index.DB(req.DB)
if db == nil {
http.Error(w, ErrDatabaseNotFound.Error(), http.StatusNotFound)
return
}
@ -381,9 +417,9 @@ func (h *Handler) handlePostDBAttrDiff(w http.ResponseWriter, r *http.Request) {
}
// Retrieve database from index.
db, err := h.Index.CreateDBIfNotExists(req.DB)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
db := h.Index.DB(req.DB)
if db == nil {
http.Error(w, ErrDatabaseNotFound.Error(), http.StatusNotFound)
return
}
@ -427,6 +463,46 @@ type postDBAttrDiffResponse struct {
Attrs map[uint64]map[string]interface{} `json:"attrs"`
}
// handlePostFrame handles POST /frame request.
func (h *Handler) handlePostFrame(w http.ResponseWriter, r *http.Request) {
// Decode request.
var req postFrameRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Find database.
db := h.Index.DB(req.DB)
if db == nil {
http.Error(w, ErrDatabaseNotFound.Error(), http.StatusNotFound)
return
}
// Create frame.
_, err := db.CreateFrame(req.Frame, req.Options)
if err == ErrFrameExists {
http.Error(w, err.Error(), http.StatusConflict)
return
} else if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Encode response.
if err := json.NewEncoder(w).Encode(postFrameResponse{}); err != nil {
h.logger().Printf("response encoding error: %s", err)
}
}
type postFrameRequest struct {
DB string `json:"db"`
Frame string `json:"frame"`
Options FrameOptions `json:"options"`
}
type postFrameResponse struct{}
// handleDeleteFrame handles DELETE /frame request.
func (h *Handler) handleDeleteFrame(w http.ResponseWriter, r *http.Request) {
// Decode request.
@ -481,9 +557,9 @@ func (h *Handler) handlePatchFrameTimeQuantum(w http.ResponseWriter, r *http.Req
}
// Retrieve database by name.
f, err := h.Index.CreateFrameIfNotExists(req.DB, req.Frame)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
f := h.Index.Frame(req.DB, req.Frame)
if f == nil {
http.Error(w, ErrFrameNotFound.Error(), http.StatusNotFound)
return
}
@ -517,9 +593,9 @@ func (h *Handler) handlePostFrameAttrDiff(w http.ResponseWriter, r *http.Request
}
// Retrieve database from index.
f, err := h.Index.CreateFrameIfNotExists(req.DB, req.Frame)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
f := h.Index.Frame(req.DB, req.Frame)
if f == nil {
http.Error(w, ErrFrameNotFound.Error(), http.StatusNotFound)
return
}
@ -717,10 +793,10 @@ func (h *Handler) handlePostImport(w http.ResponseWriter, r *http.Request) {
// Find the correct fragment.
h.logger().Println("importing:", req.DB, req.Frame, req.Slice)
db, err := h.Index.CreateDBIfNotExists(req.DB)
if err != nil {
h.logger().Printf("fragment error: db=%s, frame=%s, slice=%d, err=%s", req.DB, req.Frame, req.Slice, err)
http.Error(w, "fragment error", http.StatusInternalServerError)
db := h.Index.DB(req.DB)
if db == nil {
h.logger().Printf("fragment error: db=%s, frame=%s, slice=%d, err=%s", req.DB, req.Frame, req.Slice, ErrDatabaseNotFound.Error())
http.Error(w, ErrDatabaseNotFound.Error(), http.StatusNotFound)
return
}
@ -728,6 +804,7 @@ func (h *Handler) handlePostImport(w http.ResponseWriter, r *http.Request) {
err = db.Import(req.Frame, req.BitmapIDs, req.ProfileIDs, timestamps)
if err != nil {
h.logger().Printf("import error: db=%s, frame=%s, slice=%d, bits=%d, err=%s", req.DB, req.Frame, req.Slice, len(req.ProfileIDs), err)
return
}
// Marshal response object.
@ -850,15 +927,22 @@ func (h *Handler) handlePostFragmentData(w http.ResponseWriter, r *http.Request)
return
}
// Retrieve fragment from index.
f, err := h.Index.CreateFragmentIfNotExists(q.Get("db"), q.Get("frame"), slice)
// Retrieve frame.
f := h.Index.Frame(q.Get("db"), q.Get("frame"))
if f == nil {
http.Error(w, ErrFrameNotFound.Error(), http.StatusNotFound)
return
}
// Retrieve fragment from frame.
frag, err := f.CreateFragmentIfNotExists(slice)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
// Read fragment in from request body.
if _, err := f.ReadFrom(r.Body); err != nil {
if _, err := frag.ReadFrom(r.Body); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
@ -974,8 +1058,15 @@ func (h *Handler) handlePostFrameRestore(w http.ResponseWriter, r *http.Request)
continue
}
// Retrieve frame.
f := h.Index.Frame(db, frame)
if f == nil {
http.Error(w, ErrFrameNotFound.Error(), http.StatusNotFound)
return
}
// Otherwise retrieve the local fragment.
f, err := h.Index.CreateFragmentIfNotExists(db, frame, slice)
frag, err := f.CreateFragmentIfNotExists(slice)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
@ -993,7 +1084,7 @@ func (h *Handler) handlePostFrameRestore(w http.ResponseWriter, r *http.Request)
// Restore to local frame and always close reader.
if err := func() error {
defer rd.Close()
if _, err := f.ReadFrom(rd); err != nil {
if _, err := frag.ReadFrom(rd); err != nil {
return err
}
return nil

35
handler_test.go
View file

@ -33,13 +33,17 @@ func TestHandler_NotFound(t *testing.T) {
func TestHandler_Schema(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
if _, err := idx.CreateFrameIfNotExists("d0", "f1"); err != nil {
d0 := idx.MustCreateDBIfNotExists("d0", pilosa.DBOptions{})
d1 := idx.MustCreateDBIfNotExists("d1", pilosa.DBOptions{})
if _, err := d0.CreateFrameIfNotExists("f1", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateFrameIfNotExists("d1", "f0"); err != nil {
if _, err := d1.CreateFrameIfNotExists("f0", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
if _, err := idx.CreateFrameIfNotExists("d0", "f0"); err != nil {
if _, err := d0.CreateFrameIfNotExists("f0", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
@ -210,7 +214,7 @@ func TestHandler_Query_Bitmap_Profiles_JSON(t *testing.T) {
defer idx.Close()
// Create database and set profile attributes.
db, err := idx.CreateDBIfNotExists("d")
db, err := idx.CreateDBIfNotExists("d", pilosa.DBOptions{})
if err != nil {
t.Fatal(err)
} else if err := db.ProfileAttrStore().SetAttrs(3, map[string]interface{}{"x": "y"}); err != nil {
@ -275,7 +279,7 @@ func TestHandler_Query_Bitmap_Profiles_Protobuf(t *testing.T) {
defer idx.Close()
// Create database and set profile attributes.
db, err := idx.CreateDBIfNotExists("d")
db, err := idx.CreateDBIfNotExists("d", pilosa.DBOptions{})
if err != nil {
t.Fatal(err)
} else if err := db.ProfileAttrStore().SetAttrs(1, map[string]interface{}{"x": "y"}); err != nil {
@ -451,7 +455,7 @@ func TestHandler_DB_Delete(t *testing.T) {
defer s.Close()
// Create database.
if _, err := idx.CreateDBIfNotExists("d"); err != nil {
if _, err := idx.CreateDBIfNotExists("d", pilosa.DBOptions{}); err != nil {
t.Fatal(err)
}
@ -481,7 +485,8 @@ func TestHandler_DB_Delete(t *testing.T) {
func TestHandler_DeleteFrame(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
if _, err := idx.CreateFrameIfNotExists("d0", "f1"); err != nil {
d0 := idx.MustCreateDBIfNotExists("d0", pilosa.DBOptions{})
if _, err := d0.CreateFrameIfNotExists("f1", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
@ -502,6 +507,7 @@ func TestHandler_DeleteFrame(t *testing.T) {
func TestHandler_SetDBTimeQuantum(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
idx.MustCreateDBIfNotExists("d0", pilosa.DBOptions{})
h := NewHandler()
h.Index = idx.Index
@ -521,6 +527,11 @@ func TestHandler_SetFrameTimeQuantum(t *testing.T) {
idx := MustOpenIndex()
defer idx.Close()
// Create frame.
if _, err := idx.MustCreateDBIfNotExists("d0", pilosa.DBOptions{}).CreateFrame("f1", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
h := NewHandler()
h.Index = idx.Index
w := httptest.NewRecorder()
@ -544,7 +555,7 @@ func TestHandler_DB_AttrStore_Diff(t *testing.T) {
defer s.Close()
// Set attributes on the database.
db, err := idx.CreateDBIfNotExists("d")
db, err := idx.CreateDBIfNotExists("d", pilosa.DBOptions{})
if err != nil {
t.Fatal(err)
}
@ -593,7 +604,8 @@ func TestHandler_Frame_AttrStore_Diff(t *testing.T) {
defer s.Close()
// Set attributes on the database.
f, err := idx.CreateFrameIfNotExists("d", "f")
d := idx.MustCreateDBIfNotExists("d", pilosa.DBOptions{})
f, err := d.CreateFrameIfNotExists("f", pilosa.FrameOptions{})
if err != nil {
t.Fatal(err)
}
@ -657,6 +669,11 @@ func TestHandler_Fragment_BackupRestore(t *testing.T) {
t.Fatalf("unexpected backup status code: %d", resp.StatusCode)
}
// Create frame.
if _, err := idx.MustCreateDBIfNotExists("x", pilosa.DBOptions{}).CreateFrame("y", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Restore backup to slice x/y/0.
if resp, err := http.Post(s.URL+"/fragment/data?db=x&frame=y&slice=0", "application/octet-stream", resp.Body); err != nil {
t.Fatal(err)

86
index.go
View file

@ -78,7 +78,10 @@ func (i *Index) Open() error {
i.logger().Printf("opening database: %s", filepath.Base(fi.Name()))
db := i.newDB(i.DBPath(filepath.Base(fi.Name())), filepath.Base(fi.Name()))
db, err := i.newDB(i.DBPath(filepath.Base(fi.Name())), filepath.Base(fi.Name()))
if err != nil {
return ErrName
}
if err := db.Open(); err != nil {
return fmt.Errorf("open db: name=%s, err=%s", db.Name(), err)
}
@ -156,15 +159,33 @@ func (i *Index) DBs() []*DB {
return a
}
// CreateDBIfNotExists returns a database by name.
// The database is created if it does not already exist.
func (i *Index) CreateDBIfNotExists(name string) (*DB, error) {
// CreateDB creates a database.
func (i *Index) CreateDB(name string, opt DBOptions) (*DB, error) {
i.mu.Lock()
defer i.mu.Unlock()
return i.createDBIfNotExists(name)
// Ensure db doesn't already exist.
if i.dbs[name] != nil {
return nil, ErrDatabaseExists
}
return i.createDB(name, opt)
}
func (i *Index) createDBIfNotExists(name string) (*DB, error) {
// CreateDBIfNotExists returns a database by name.
// The database is created if it does not already exist.
func (i *Index) CreateDBIfNotExists(name string, opt DBOptions) (*DB, error) {
i.mu.Lock()
defer i.mu.Unlock()
// Find frame in cache first.
if db := i.dbs[name]; db != nil {
return db, nil
}
return i.createDB(name, opt)
}
func (i *Index) createDB(name string, opt DBOptions) (*DB, error) {
if name == "" {
return nil, errors.New("database name required")
}
@ -175,10 +196,18 @@ func (i *Index) createDBIfNotExists(name string) (*DB, error) {
}
// Otherwise create a new database.
db := i.newDB(i.DBPath(name), name)
db, err := i.newDB(i.DBPath(name), name)
if err != nil {
return nil, err
}
if err := db.Open(); err != nil {
return nil, err
}
// Update options.
db.SetColumnLabel(opt.ColumnLabel)
i.dbs[db.Name()] = db
i.Stats.Count("dbN", 1)
@ -186,11 +215,14 @@ func (i *Index) createDBIfNotExists(name string) (*DB, error) {
return db, nil
}
func (i *Index) newDB(path, name string) *DB {
db := NewDB(path, name)
func (i *Index) newDB(path, name string) (*DB, error) {
db, err := NewDB(path, name)
if err != nil {
return nil, err
}
db.LogOutput = i.LogOutput
db.stats = i.Stats.WithTags(fmt.Sprintf("db:%s", db.Name()))
return db
return db, nil
}
// DeleteDB removes a database from the index.
@ -231,16 +263,6 @@ func (i *Index) Frame(db, name string) *Frame {
return d.Frame(name)
}
// CreateFrameIfNotExists returns the frame for a database & name.
// The frame is created if it doesn't already exist.
func (i *Index) CreateFrameIfNotExists(db, name string) (*Frame, error) {
d, err := i.CreateDBIfNotExists(db)
if err != nil {
return nil, err
}
return d.CreateFrameIfNotExists(name)
}
// Fragment returns the fragment for a database, frame & slice.
func (i *Index) Fragment(db, frame string, slice uint64) *Fragment {
f := i.Frame(db, frame)
@ -250,16 +272,6 @@ func (i *Index) Fragment(db, frame string, slice uint64) *Fragment {
return f.Fragment(slice)
}
// CreateFragmentIfNotExists returns the fragment for a database, frame & slice.
// The fragment is created if it doesn't already exist.
func (i *Index) CreateFragmentIfNotExists(db, frame string, slice uint64) (*Fragment, error) {
f, err := i.CreateFrameIfNotExists(db, frame)
if err != nil {
return nil, err
}
return f.CreateFragmentIfNotExists(slice)
}
// monitorCacheFlush periodically flushes all fragment caches sequentially.
// This is run in a goroutine.
func (i *Index) monitorCacheFlush() {
@ -434,7 +446,9 @@ func (s *IndexSyncer) syncFrame(db, name string) error {
// Retrieve attributes from differing blocks.
// Skip update and recomputation if no attributes have changed.
m, err := client.BitmapAttrDiff(context.Background(), db, name, blks)
if err != nil {
if err == ErrFrameNotFound {
continue // frame not created remotely yet, skip
} else if err != nil {
return err
} else if len(m) == 0 {
continue
@ -457,15 +471,21 @@ func (s *IndexSyncer) syncFrame(db, name string) error {
// syncFragment synchronizes a fragment with the rest of the cluster.
func (s *IndexSyncer) syncFragment(db, frame string, slice uint64) error {
// Retrieve local frame.
f := s.Index.Frame(db, frame)
if f == nil {
return ErrFrameNotFound
}
// Ensure fragment exists locally.
f, err := s.Index.CreateFragmentIfNotExists(db, frame, slice)
frag, err := f.CreateFragmentIfNotExists(slice)
if err != nil {
return err
}
// Sync fragments together.
fs := FragmentSyncer{
Fragment: f,
Fragment: frag,
Host: s.Host,
Cluster: s.Cluster,
Closing: s.Closing,

22
index_test.go
View file

@ -72,6 +72,13 @@ func TestIndexSyncer_SyncIndex(t *testing.T) {
cluster.Nodes[0].Host = "localhost:0"
cluster.Nodes[1].Host = MustParseURLHost(s.URL)
// Create frames on nodes.
for _, idx := range []*Index{idx0, idx1} {
idx.MustCreateFrameIfNotExists("d", "f")
idx.MustCreateFrameIfNotExists("d", "f0")
idx.MustCreateFrameIfNotExists("y", "z")
}
// Set data on the local index.
f := idx0.MustCreateFragmentIfNotExists("d", "f", 0)
if _, err := f.SetBit(0, 10); err != nil {
@ -191,8 +198,8 @@ func (i *Index) Close() error {
}
// MustCreateDBIfNotExists returns a given db. Panic on error.
func (i *Index) MustCreateDBIfNotExists(db string) *DB {
d, err := i.Index.CreateDBIfNotExists(db)
func (i *Index) MustCreateDBIfNotExists(db string, opt pilosa.DBOptions) *DB {
d, err := i.Index.CreateDBIfNotExists(db, opt)
if err != nil {
panic(err)
}
@ -201,7 +208,7 @@ func (i *Index) MustCreateDBIfNotExists(db string) *DB {
// MustCreateFrameIfNotExists returns a given frame. Panic on error.
func (i *Index) MustCreateFrameIfNotExists(db, frame string) *Frame {
f, err := i.Index.CreateFrameIfNotExists(db, frame)
f, err := i.MustCreateDBIfNotExists(db, pilosa.DBOptions{}).CreateFrameIfNotExists(frame, pilosa.FrameOptions{})
if err != nil {
panic(err)
}
@ -210,9 +217,14 @@ func (i *Index) MustCreateFrameIfNotExists(db, frame string) *Frame {
// MustCreateFragmentIfNotExists returns a given fragment. Panic on error.
func (i *Index) MustCreateFragmentIfNotExists(db, frame string, slice uint64) *Fragment {
f, err := i.Index.CreateFragmentIfNotExists(db, frame, slice)
d := i.MustCreateDBIfNotExists(db, pilosa.DBOptions{})
f, err := d.CreateFrameIfNotExists(frame, pilosa.FrameOptions{})
if err != nil {
panic(err)
}
return &Fragment{Fragment: f}
frag, err := f.CreateFragmentIfNotExists(slice)
if err != nil {
panic(err)
}
return &Fragment{Fragment: frag}
}

172
internal/internal.pb.go
View file

@ -48,6 +48,7 @@ const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package
type DB struct {
TimeQuantum string `protobuf:"bytes,1,opt,name=TimeQuantum,proto3" json:"TimeQuantum,omitempty"`
ColumnLabel string `protobuf:"bytes,2,opt,name=ColumnLabel,proto3" json:"ColumnLabel,omitempty"`
}
func (m *DB) Reset() { *m = DB{} }
@ -57,6 +58,7 @@ func (*DB) Descriptor() ([]byte, []int) { return fileDescriptorInternal, []int{0
type Frame struct {
TimeQuantum string `protobuf:"bytes,1,opt,name=TimeQuantum,proto3" json:"TimeQuantum,omitempty"`
RowLabel string `protobuf:"bytes,2,opt,name=RowLabel,proto3" json:"RowLabel,omitempty"`
}
func (m *Frame) Reset() { *m = Frame{} }
@ -323,6 +325,12 @@ func (m *DB) MarshalTo(dAtA []byte) (int, error) {
i = encodeVarintInternal(dAtA, i, uint64(len(m.TimeQuantum)))
i += copy(dAtA[i:], m.TimeQuantum)
}
if len(m.ColumnLabel) > 0 {
dAtA[i] = 0x12
i++
i = encodeVarintInternal(dAtA, i, uint64(len(m.ColumnLabel)))
i += copy(dAtA[i:], m.ColumnLabel)
}
return i, nil
}
@ -347,6 +355,12 @@ func (m *Frame) MarshalTo(dAtA []byte) (int, error) {
i = encodeVarintInternal(dAtA, i, uint64(len(m.TimeQuantum)))
i += copy(dAtA[i:], m.TimeQuantum)
}
if len(m.RowLabel) > 0 {
dAtA[i] = 0x12
i++
i = encodeVarintInternal(dAtA, i, uint64(len(m.RowLabel)))
i += copy(dAtA[i:], m.RowLabel)
}
return i, nil
}
@ -1055,6 +1069,10 @@ func (m *DB) Size() (n int) {
if l > 0 {
n += 1 + l + sovInternal(uint64(l))
}
l = len(m.ColumnLabel)
if l > 0 {
n += 1 + l + sovInternal(uint64(l))
}
return n
}
@ -1065,6 +1083,10 @@ func (m *Frame) Size() (n int) {
if l > 0 {
n += 1 + l + sovInternal(uint64(l))
}
l = len(m.RowLabel)
if l > 0 {
n += 1 + l + sovInternal(uint64(l))
}
return n
}
@ -1425,6 +1447,35 @@ func (m *DB) Unmarshal(dAtA []byte) error {
}
m.TimeQuantum = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field ColumnLabel", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowInternal
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthInternal
}
postIndex := iNdEx + intStringLen
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.ColumnLabel = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
default:
iNdEx = preIndex
skippy, err := skipInternal(dAtA[iNdEx:])
@ -1504,6 +1555,35 @@ func (m *Frame) Unmarshal(dAtA []byte) error {
}
m.TimeQuantum = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
case 2:
if wireType != 2 {
return fmt.Errorf("proto: wrong wireType = %d for field RowLabel", wireType)
}
var stringLen uint64
for shift := uint(0); ; shift += 7 {
if shift >= 64 {
return ErrIntOverflowInternal
}
if iNdEx >= l {
return io.ErrUnexpectedEOF
}
b := dAtA[iNdEx]
iNdEx++
stringLen |= (uint64(b) & 0x7F) << shift
if b < 0x80 {
break
}
}
intStringLen := int(stringLen)
if intStringLen < 0 {
return ErrInvalidLengthInternal
}
postIndex := iNdEx + intStringLen
if postIndex > l {
return io.ErrUnexpectedEOF
}
m.RowLabel = string(dAtA[iNdEx:postIndex])
iNdEx = postIndex
default:
iNdEx = preIndex
skippy, err := skipInternal(dAtA[iNdEx:])
@ -3831,49 +3911,51 @@ var (
func init() { proto.RegisterFile("internal.proto", fileDescriptorInternal) }
var fileDescriptorInternal = []byte{
// 696 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0xac, 0x55, 0xcb, 0x6a, 0x14, 0x41,
0x14, 0xb5, 0xa6, 0xbb, 0xe7, 0x71, 0x27, 0x19, 0x26, 0xc5, 0x28, 0x4d, 0x90, 0xa1, 0x29, 0x54,
0x5a, 0xc1, 0x04, 0xe2, 0x46, 0x44, 0x10, 0x3b, 0x13, 0x71, 0x90, 0x84, 0xa4, 0x12, 0xdd, 0xb9,
0x68, 0x93, 0x32, 0x69, 0xd2, 0x2f, 0xab, 0xab, 0xc5, 0x59, 0xba, 0x70, 0xe3, 0x17, 0x88, 0x7e,
0x81, 0x7f, 0xe2, 0xd2, 0x4f, 0x90, 0xf8, 0x23, 0x52, 0x8f, 0x7e, 0x44, 0x31, 0x66, 0xe1, 0xae,
0xef, 0xb9, 0x75, 0x6f, 0x9d, 0x73, 0x1f, 0xd5, 0x30, 0x8a, 0x52, 0xc1, 0x78, 0x1a, 0xc6, 0x6b,
0x39, 0xcf, 0x44, 0x86, 0xfb, 0x95, 0x4d, 0x6e, 0x41, 0x67, 0x16, 0x60, 0x0f, 0x86, 0x07, 0x51,
0xc2, 0xf6, 0xca, 0x30, 0x15, 0x65, 0xe2, 0x22, 0x0f, 0xf9, 0x03, 0xda, 0x86, 0xc8, 0x6d, 0x70,
0x9e, 0xf0, 0x30, 0x61, 0x97, 0x38, 0x1a, 0x40, 0x37, 0x88, 0x44, 0x12, 0xe6, 0x18, 0x83, 0x1d,
0x44, 0xa2, 0x70, 0x91, 0x67, 0xf9, 0x36, 0x55, 0xdf, 0xf8, 0x06, 0x38, 0x8f, 0x85, 0xe0, 0x85,
0xdb, 0xf1, 0x2c, 0x7f, 0xb8, 0x31, 0x5a, 0xab, 0xa9, 0x49, 0x98, 0x6a, 0x27, 0x59, 0x03, 0x7b,
0x37, 0x8c, 0x38, 0x1e, 0x83, 0xf5, 0x8c, 0x2d, 0xd4, 0x2d, 0x36, 0x95, 0x9f, 0x78, 0x02, 0xce,
0x66, 0x56, 0xa6, 0xc2, 0xed, 0x28, 0x4c, 0x1b, 0xe4, 0x25, 0x58, 0x41, 0x24, 0xf0, 0x2a, 0xf4,
0xf5, 0xd5, 0xf3, 0x99, 0x89, 0xa9, 0x6d, 0x7c, 0x1d, 0x06, 0xbb, 0x3c, 0x7b, 0x1d, 0xc5, 0x6c,
0x3e, 0x33, 0xc1, 0x0d, 0x20, 0xbd, 0x52, 0x43, 0x21, 0xc2, 0x24, 0x77, 0x2d, 0x0f, 0xf9, 0x16,
0x6d, 0x00, 0xf2, 0x08, 0x7a, 0xe6, 0x28, 0x1e, 0x41, 0xa7, 0x4e, 0xde, 0x99, 0xcf, 0x2e, 0xa9,
0xe7, 0x23, 0x02, 0x5b, 0x7e, 0xb5, 0x05, 0x0d, 0xb4, 0x20, 0x0c, 0xf6, 0xc1, 0x22, 0x67, 0x86,
0x92, 0xfa, 0x96, 0x45, 0xde, 0x17, 0x3c, 0x4a, 0x8f, 0x5f, 0x84, 0x71, 0xc9, 0x14, 0x9f, 0x01,
0x6d, 0x43, 0x92, 0xef, 0xf3, 0x28, 0x15, 0xda, 0x6f, 0x6b, 0x35, 0x35, 0x20, 0xbd, 0x41, 0x96,
0xc5, 0xda, 0xeb, 0x78, 0xc8, 0xef, 0xd3, 0x06, 0x20, 0xeb, 0xd0, 0x93, 0x5c, 0xb6, 0xc3, 0xbc,
0x61, 0x8f, 0x2e, 0x62, 0xff, 0x19, 0xc1, 0xd2, 0x5e, 0xc9, 0xf8, 0x82, 0xb2, 0x37, 0x25, 0x2b,
0x84, 0x2c, 0xc2, 0x2c, 0x30, 0x22, 0xe4, 0xfc, 0x4c, 0xc0, 0x51, 0x7e, 0x25, 0x62, 0x40, 0xb5,
0x81, 0xaf, 0x41, 0x77, 0x3f, 0x8e, 0x0e, 0x59, 0xe1, 0x5a, 0x6a, 0x00, 0x8c, 0x25, 0xbb, 0x64,
0xaa, 0x59, 0x28, 0xea, 0x7d, 0x5a, 0xdb, 0xd8, 0x85, 0x5e, 0x35, 0x5a, 0x8e, 0xca, 0x55, 0x99,
0x32, 0x1b, 0x65, 0x49, 0x26, 0x98, 0xdb, 0x55, 0x31, 0xc6, 0x22, 0xef, 0x11, 0x2c, 0x1b, 0x72,
0x45, 0x9e, 0xa5, 0x05, 0x93, 0x35, 0xde, 0xe2, 0xbc, 0xaa, 0xf1, 0x16, 0xe7, 0x78, 0x1d, 0x7a,
0x94, 0x15, 0x65, 0x2c, 0xaa, 0x36, 0x5d, 0x6d, 0x84, 0x56, 0xb1, 0x65, 0x2c, 0x68, 0x75, 0x0a,
0xdf, 0x6d, 0x51, 0xb4, 0x54, 0xc4, 0x4a, 0x13, 0x61, 0x3c, 0x0d, 0x6b, 0xf2, 0x01, 0xc1, 0xb0,
0x95, 0x07, 0xfb, 0xd5, 0x0a, 0x28, 0x12, 0xc3, 0x8d, 0x71, 0x13, 0xac, 0x71, 0x5a, 0xad, 0xc8,
0x12, 0xa0, 0x1d, 0xd3, 0x7a, 0xb4, 0x23, 0xdb, 0x21, 0xc7, 0xbe, 0xba, 0xb3, 0xd5, 0x0e, 0x09,
0x53, 0xed, 0x94, 0x35, 0xda, 0x3c, 0x09, 0xd3, 0x63, 0x76, 0x64, 0xca, 0x57, 0x99, 0xe4, 0x2b,
0x82, 0xe5, 0x79, 0x92, 0x67, 0x5c, 0x5c, 0xd0, 0x29, 0xb5, 0xc7, 0x55, 0xa7, 0xf4, 0x52, 0x4f,
0xc0, 0x51, 0xbd, 0x51, 0x93, 0x66, 0x53, 0x6d, 0xa8, 0x29, 0x32, 0xdb, 0x23, 0x1b, 0x25, 0x5b,
0xd8, 0x00, 0x78, 0x0a, 0x50, 0xaf, 0x4f, 0xe1, 0x3a, 0xca, 0xdd, 0x42, 0xa4, 0xbf, 0x5e, 0xa0,
0xc2, 0xed, 0x7a, 0x96, 0x6f, 0xd1, 0x16, 0x42, 0x08, 0x8c, 0x2a, 0xaa, 0x7f, 0xeb, 0x1b, 0x39,
0x82, 0x71, 0x10, 0x67, 0x87, 0xa7, 0xb3, 0x50, 0x84, 0xff, 0x43, 0xd1, 0x04, 0x1c, 0x95, 0xcf,
0x6c, 0x8c, 0x36, 0xc8, 0x1e, 0xac, 0xb4, 0x6e, 0x31, 0x64, 0xce, 0x89, 0x47, 0x17, 0x8b, 0xef,
0xfc, 0x2e, 0x9e, 0xdc, 0x04, 0x67, 0x33, 0x3c, 0x3c, 0xf9, 0x47, 0x1a, 0xf2, 0x05, 0xc1, 0xca,
0x76, 0xf8, 0x4e, 0xef, 0x45, 0x7d, 0xf5, 0x53, 0x18, 0xd4, 0xa0, 0x59, 0xcc, 0x3b, 0xcd, 0x24,
0xfc, 0x71, 0xbe, 0x41, 0xb6, 0x52, 0xc1, 0x17, 0xb4, 0x09, 0x5e, 0x7d, 0x08, 0xa3, 0xf3, 0x4e,
0x59, 0xe3, 0xd3, 0xe6, 0xfd, 0x39, 0xd5, 0x0f, 0xea, 0x5b, 0xf5, 0x4e, 0x98, 0x07, 0x55, 0x19,
0x0f, 0x3a, 0xf7, 0x51, 0x30, 0xfe, 0x76, 0x36, 0x45, 0xdf, 0xcf, 0xa6, 0xe8, 0xc7, 0xd9, 0x14,
0x7d, 0xfa, 0x39, 0xbd, 0xf2, 0xaa, 0xab, 0x7e, 0x1f, 0xf7, 0x7e, 0x05, 0x00, 0x00, 0xff, 0xff,
0x04, 0xa0, 0xe3, 0x0c, 0x50, 0x06, 0x00, 0x00,
// 721 bytes of a gzipped FileDescriptorProto
0x1f, 0x8b, 0x08, 0x00, 0x00, 0x09, 0x6e, 0x88, 0x02, 0xff, 0xac, 0x55, 0xcb, 0x6a, 0x14, 0x4d,
0x14, 0xfe, 0x6b, 0xba, 0x7b, 0x2e, 0x67, 0x92, 0x61, 0x52, 0xcc, 0xff, 0xd3, 0x84, 0x9f, 0x61,
0x28, 0x14, 0x06, 0xc1, 0x04, 0xe2, 0x46, 0x44, 0x10, 0x7b, 0x66, 0x24, 0x83, 0x26, 0x24, 0x95,
0xe8, 0xce, 0x45, 0x25, 0x29, 0x93, 0x26, 0x7d, 0x19, 0xab, 0xab, 0xd5, 0x59, 0xba, 0x70, 0xe3,
0x13, 0x88, 0x3e, 0x81, 0x6f, 0xe2, 0xd2, 0x47, 0x90, 0xf8, 0x22, 0x52, 0x97, 0xbe, 0x44, 0x31,
0x66, 0xe1, 0xae, 0xbf, 0xef, 0xd4, 0x39, 0xf5, 0x9d, 0x5b, 0x35, 0xf4, 0xc2, 0x44, 0x72, 0x91,
0xb0, 0x68, 0x63, 0x21, 0x52, 0x99, 0xe2, 0x76, 0x81, 0xc9, 0x36, 0x34, 0xa6, 0x01, 0x1e, 0x41,
0xf7, 0x30, 0x8c, 0xf9, 0x7e, 0xce, 0x12, 0x99, 0xc7, 0x3e, 0x1a, 0xa1, 0x71, 0x87, 0xd6, 0x29,
0x75, 0x62, 0x92, 0x46, 0x79, 0x9c, 0x3c, 0x61, 0x47, 0x3c, 0xf2, 0x1b, 0xe6, 0x44, 0x8d, 0x22,
0x33, 0xf0, 0x1e, 0x09, 0x16, 0xf3, 0x6b, 0x04, 0x5b, 0x87, 0x36, 0x4d, 0x5f, 0xd7, 0x23, 0x95,
0x98, 0x04, 0xd0, 0x0c, 0x42, 0x19, 0xb3, 0x05, 0xc6, 0xe0, 0x06, 0xa1, 0xcc, 0x7c, 0x34, 0x72,
0xc6, 0x2e, 0xd5, 0xdf, 0xf8, 0x06, 0x78, 0x0f, 0xa5, 0x14, 0x99, 0xdf, 0x18, 0x39, 0xe3, 0xee,
0x56, 0x6f, 0xa3, 0x4c, 0x4c, 0xd1, 0xd4, 0x18, 0xc9, 0x06, 0xb8, 0x7b, 0x2c, 0x14, 0xb8, 0x0f,
0xce, 0x63, 0xbe, 0xd4, 0x0a, 0x5c, 0xaa, 0x3e, 0xf1, 0x00, 0xbc, 0x49, 0x9a, 0x27, 0x52, 0x5f,
0xeb, 0x52, 0x03, 0xc8, 0x73, 0x70, 0x82, 0x50, 0x2a, 0x59, 0xe6, 0xea, 0xf9, 0xd4, 0xfa, 0x94,
0x18, 0xff, 0x0f, 0x9d, 0x3d, 0x91, 0xbe, 0x08, 0x23, 0x3e, 0x9f, 0x5a, 0xe7, 0x8a, 0x50, 0x56,
0x95, 0x5f, 0x26, 0x59, 0xbc, 0xf0, 0x9d, 0x11, 0x1a, 0x3b, 0xb4, 0x22, 0xc8, 0x03, 0x68, 0xd9,
0xa3, 0xb8, 0x07, 0x8d, 0x32, 0x78, 0x63, 0x3e, 0xbd, 0x66, 0x3e, 0xef, 0x11, 0xb8, 0xea, 0xab,
0x9e, 0x50, 0xc7, 0x24, 0x84, 0xc1, 0x3d, 0x5c, 0x2e, 0xb8, 0x95, 0xa4, 0xbf, 0x55, 0x03, 0x0e,
0xa4, 0x08, 0x93, 0xd3, 0x67, 0x2c, 0xca, 0xb9, 0xd6, 0xd3, 0xa1, 0x75, 0x4a, 0xe9, 0x7d, 0x1a,
0x26, 0xd2, 0xd8, 0x5d, 0x93, 0x4d, 0x49, 0x28, 0x6b, 0x90, 0xa6, 0x91, 0xb1, 0x7a, 0x23, 0x34,
0x6e, 0xd3, 0x8a, 0x20, 0x9b, 0xd0, 0x52, 0x5a, 0x76, 0xd8, 0xa2, 0x52, 0x8f, 0xae, 0x52, 0xff,
0x11, 0xc1, 0xca, 0x7e, 0xce, 0xc5, 0x92, 0xf2, 0x97, 0x39, 0xcf, 0xa4, 0x2a, 0xc2, 0x34, 0xb0,
0x49, 0xa8, 0xe9, 0x1b, 0x80, 0xa7, 0xed, 0x76, 0x16, 0x0c, 0xc0, 0xff, 0x41, 0xf3, 0x20, 0x0a,
0x8f, 0x79, 0xe6, 0x3b, 0x7a, 0x00, 0x2c, 0x52, 0x5d, 0xb2, 0xd5, 0xcc, 0xb4, 0xf4, 0x36, 0x2d,
0x31, 0xf6, 0xa1, 0x55, 0x8c, 0x9d, 0xa7, 0x63, 0x15, 0x50, 0x45, 0xa3, 0x3c, 0x4e, 0x25, 0xf7,
0x9b, 0xda, 0xc7, 0x22, 0xf2, 0x16, 0xc1, 0xaa, 0x15, 0x97, 0x2d, 0xd2, 0x24, 0xe3, 0xaa, 0xc6,
0x33, 0x21, 0x8a, 0x1a, 0xcf, 0x84, 0xc0, 0x9b, 0xd0, 0xa2, 0x3c, 0xcb, 0x23, 0x59, 0xb4, 0xe9,
0xdf, 0x2a, 0xd1, 0xc2, 0x37, 0x8f, 0x24, 0x2d, 0x4e, 0xe1, 0xdb, 0x35, 0x89, 0x8e, 0xf6, 0x58,
0xab, 0x3c, 0xac, 0xa5, 0x52, 0x4d, 0xde, 0x21, 0xe8, 0xd6, 0xe2, 0xe0, 0x71, 0xb1, 0x02, 0x5a,
0x44, 0x77, 0xab, 0x5f, 0x39, 0x1b, 0x9e, 0x16, 0x2b, 0xb2, 0x02, 0x68, 0xd7, 0xb6, 0x1e, 0xed,
0xaa, 0x76, 0xa8, 0xb1, 0x2f, 0xee, 0xac, 0xb5, 0x43, 0xd1, 0xd4, 0x18, 0x55, 0x8d, 0x26, 0x67,
0x2c, 0x39, 0xe5, 0x27, 0xb6, 0x7c, 0x05, 0x24, 0x9f, 0x11, 0xac, 0xce, 0xe3, 0x45, 0x2a, 0xe4,
0x15, 0x9d, 0xd2, 0x3b, 0x5e, 0x74, 0xca, 0x2c, 0xfc, 0x00, 0x3c, 0xdd, 0x1b, 0x3d, 0x69, 0x2e,
0x35, 0x40, 0x4f, 0x91, 0xdd, 0x1e, 0xd5, 0x28, 0xd5, 0xc2, 0x8a, 0xc0, 0x43, 0x80, 0x72, 0x7d,
0x32, 0xdf, 0xd3, 0xe6, 0x1a, 0xa3, 0xec, 0xe5, 0x02, 0x65, 0x7e, 0x73, 0xe4, 0x8c, 0x1d, 0x5a,
0x63, 0x08, 0x81, 0x5e, 0x21, 0xf5, 0x77, 0x7d, 0x23, 0x27, 0xd0, 0x0f, 0xa2, 0xf4, 0xf8, 0x7c,
0xca, 0x24, 0xfb, 0x1b, 0x19, 0x0d, 0xc0, 0xd3, 0xf1, 0xec, 0xc6, 0x18, 0x40, 0xf6, 0x61, 0xad,
0x76, 0x8b, 0x15, 0x73, 0x29, 0x79, 0x74, 0x75, 0xf2, 0x8d, 0x9f, 0x93, 0x27, 0x37, 0xc1, 0x9b,
0xb0, 0xe3, 0xb3, 0x3f, 0x84, 0x21, 0x9f, 0x10, 0xac, 0xed, 0xb0, 0x37, 0x66, 0x2f, 0xca, 0xab,
0xb7, 0xa1, 0x53, 0x92, 0x76, 0x31, 0x6f, 0x55, 0x93, 0xf0, 0xcb, 0xf9, 0x8a, 0x99, 0x25, 0x52,
0x2c, 0x69, 0xe5, 0xbc, 0x7e, 0x1f, 0x7a, 0x97, 0x8d, 0xaa, 0xc6, 0xe7, 0xd5, 0xfb, 0x73, 0x6e,
0x1e, 0xd4, 0x57, 0xfa, 0x9d, 0xb0, 0x0f, 0xaa, 0x06, 0xf7, 0x1a, 0x77, 0x51, 0xd0, 0xff, 0x72,
0x31, 0x44, 0x5f, 0x2f, 0x86, 0xe8, 0xdb, 0xc5, 0x10, 0x7d, 0xf8, 0x3e, 0xfc, 0xe7, 0xa8, 0xa9,
0x7f, 0x3e, 0x77, 0x7e, 0x04, 0x00, 0x00, 0xff, 0xff, 0xf3, 0x47, 0xed, 0xb3, 0x8e, 0x06, 0x00,
0x00,
}

2
internal/internal.proto
View file

@ -4,10 +4,12 @@ package internal;
message DB {
string TimeQuantum = 1;
string ColumnLabel = 2;
}
message Frame {
string TimeQuantum = 1;
string RowLabel = 2;
}
message Bitmap {

30
pilosa.go
View file

@ -4,25 +4,33 @@ import (
"errors"
"github.com/pilosa/pilosa/internal"
"regexp"
)
// System errors.
var (
// ErrHostRequired is returned when excuting a remote operation without a host.
ErrHostRequired = errors.New("host required")
// ErrDatabaseRequired is returned when no database is specified.
ErrDatabaseRequired = errors.New("database required")
ErrDatabaseExists = errors.New("database already exists")
ErrDatabaseNotFound = errors.New("database not found")
ErrFrameRequired = errors.New("frame required")
ErrFrameExists = errors.New("frame already exists")
ErrFrameNotFound = errors.New("frame not found")
// ErrFrameRequired is returned when no frame is specified.
ErrFrameRequired = errors.New("frame required")
ErrName = errors.New("name restricted to [a-z0-9_-]")
// ErrFragmentNotFound is returned when a fragment does not exist.
ErrFragmentNotFound = errors.New("fragment not found")
// ErrQueryRequired is returned when no query is specified.
ErrQueryRequired = errors.New("query required")
ErrQueryRequired = errors.New("query required")
)
// Regular expression to valuate db and frame's name
// Todo: remove . when frame doesn't require . for topN
var nameRegexp = regexp.MustCompile(`^([a-z0-9._-]{1,64}$)`)
// Profile represents vertical column in a database.
// A profile can have a set of attributes attached to it.
type Profile struct {
@ -75,3 +83,13 @@ func decodeProfile(pb *internal.Profile) *Profile {
// TimeFormat is the go-style time format used to parse string dates.
const TimeFormat = "2006-01-02T15:04"
// Restrict name using regex
func ValidateName(name string) error {
validName := nameRegexp.Match([]byte(name))
if validName == false{
return ErrName
}
return nil
}

3
pql/ast.go
View file

@ -100,8 +100,7 @@ func (c *Call) String() string {
case []uint64:
fmt.Fprintf(&buf, "%v=%s", key, joinUint64Slice(v))
case time.Time:
layout := "2006-01-02T15:04:05"
fmt.Fprintf(&buf, "%v=\"%s\"", key, v.Format(layout))
fmt.Fprintf(&buf, "%v=\"%s\"", key, v.Format(TimeFormat))
default:
fmt.Fprintf(&buf, "%v=%v", key, v)
}

6
server.go
View file

@ -211,9 +211,9 @@ func (s *Server) monitorMaxSlices() {
localdb.SetRemoteMaxSlice(newmax)
}
} else {
d, err := s.Index.CreateDBIfNotExists(db)
if err != nil {
s.logger().Printf("Failed to create DB locally: %s", db)
d := s.Index.DB(db)
if d == nil {
s.logger().Printf("Local DB not found: %s", db)
return
}
oldmaxslices[db] = newmax

131
server/server.go Normal file
View file

@ -0,0 +1,131 @@
package server
import (
"errors"
"fmt"
"io"
"math/rand"
"os"
"path/filepath"
"strings"
"time"
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
)
// Version and BuildTime hold the version/build time information passed in at compile time.
var (
Version string
BuildTime string
)
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
rand.Seed(time.Now().UTC().UnixNano())
}
const (
// DefaultDataDir is the default data directory.
DefaultDataDir = "~/.pilosa"
)
// Command represents the state of the pilosa server command.
type Command struct {
Server *pilosa.Server
// Configuration options.
ConfigPath string
Config *pilosa.Config
// Profiling options.
CPUProfile string
CPUTime time.Duration
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain returns a new instance of Main.
func NewCommand() *Command {
return &Command{
Server: pilosa.NewServer(),
Config: pilosa.NewConfig(),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the pilosa server.
func (m *Command) Run(args ...string) error {
// Notify user of config file.
if m.ConfigPath != "" {
fmt.Fprintf(m.Stdout, "Using config: %s\n", m.ConfigPath)
}
// Setup logging output.
m.Server.LogOutput = m.Stderr
// Configure index.
fmt.Fprintf(m.Stderr, "Using data from: %s\n", m.Config.DataDir)
m.Server.Index.Path = m.Config.DataDir
m.Server.Index.Stats = pilosa.NewExpvarStatsClient()
// Build cluster from config file.
m.Server.Host = m.Config.Host
m.Server.Cluster = m.Config.PilosaCluster()
// Set configuration options.
m.Server.AntiEntropyInterval = time.Duration(m.Config.AntiEntropy.Interval)
// Initialize server.
if err := m.Server.Open(); err != nil {
return err
}
fmt.Fprintf(m.Stderr, "Listening as http://%s\n", m.Server.Host)
return nil
}
// Close shuts down the server.
func (m *Command) Close() error {
return m.Server.Close()
}
// SetupConfig loads the config file if specified and sets state on the Command.
func (m *Command) SetupConfig(args []string) error {
// Load config, if specified.
if m.ConfigPath != "" {
if _, err := toml.DecodeFile(m.ConfigPath, &m.Config); err != nil {
return err
}
}
// Use default data directory if one is not specified.
if m.Config.DataDir == "" {
m.Config.DataDir = DefaultDataDir
}
// Expand home directory.
prefix := "~" + string(filepath.Separator)
if strings.HasPrefix(m.Config.DataDir, prefix) {
HomeDir := os.Getenv("HOME")
if HomeDir == "" {
return errors.New("data directory not specified and no home dir available")
}
m.Config.DataDir = filepath.Join(HomeDir, strings.TrimPrefix(m.Config.DataDir, prefix))
}
return nil
}

75
cmd/pilosa/main_test.go → server/server_test.go
View file

@ -1,4 +1,4 @@
package main_test
package server_test
import (
"bytes"
@ -18,7 +18,7 @@ import (
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
main "github.com/pilosa/pilosa/cmd/pilosa"
"github.com/pilosa/pilosa/server"
)
// Ensure program can process queries and maintain consistency.
@ -31,8 +31,20 @@ func TestMain_Set_Quick(t *testing.T) {
m := MustRunMain()
defer m.Close()
// Create client.
client, err := pilosa.NewClient(m.Server.Host)
if err != nil {
t.Fatal(err)
}
// Execute SetBit() commands.
for _, cmd := range cmds {
if err := client.CreateDB(context.Background(), "d", pilosa.DBOptions{}); err != nil && err != pilosa.ErrDatabaseExists {
t.Fatal(err)
}
if err := client.CreateFrame(context.Background(), "d", cmd.Frame, pilosa.FrameOptions{}); err != nil && err != pilosa.ErrFrameExists {
t.Fatal(err)
}
if _, err := m.Query("db=d", fmt.Sprintf(`SetBit(id=%d, frame=%q, profileID=%d)`, cmd.ID, cmd.Frame, cmd.ProfileID)); err != nil {
t.Fatal(err)
}
@ -95,6 +107,16 @@ func TestMain_SetBitmapAttrs(t *testing.T) {
m := MustRunMain()
defer m.Close()
// Create frames.
client := m.Client()
if err := client.CreateDB(context.Background(), "d", pilosa.DBOptions{}); err != nil && err != pilosa.ErrDatabaseExists {
t.Fatal(err)
} else if err := client.CreateFrame(context.Background(), "d", "x.n", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
} else if err := client.CreateFrame(context.Background(), "d", "z", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Set bits on different bitmaps in different frames.
if _, err := m.Query("db=d", `SetBit(id=1, frame="x.n", profileID=100)`); err != nil {
t.Fatal(err)
@ -144,6 +166,14 @@ func TestMain_SetProfileAttrs(t *testing.T) {
m := MustRunMain()
defer m.Close()
// Create frames.
client := m.Client()
if err := client.CreateDB(context.Background(), "d", pilosa.DBOptions{}); err != nil && err != pilosa.ErrDatabaseExists {
t.Fatal(err)
} else if err := client.CreateFrame(context.Background(), "d", "x.n", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Set bits on bitmap.
if _, err := m.Query("db=d", `SetBit(id=1, frame="x.n", profileID=100)`); err != nil {
t.Fatal(err)
@ -190,6 +220,14 @@ func TestMain_FrameRestore(t *testing.T) {
}
m1.Server.Cluster.Nodes = m0.Server.Cluster.Nodes
// Create frames.
client := m0.Client()
if err := client.CreateDB(context.Background(), "d", pilosa.DBOptions{}); err != nil && err != pilosa.ErrDatabaseExists {
t.Fatal(err)
} else if err := client.CreateFrame(context.Background(), "d", "f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
}
// Write data on first cluster.
if _, err := m0.Query("db=d", `
SetBit(id=1, frame="f", profileID=100)
@ -218,6 +256,10 @@ func TestMain_FrameRestore(t *testing.T) {
client, err := pilosa.NewClient(m2.Server.Host)
if err != nil {
t.Fatal(err)
} else if err := m2.Client().CreateDB(context.Background(), "d", pilosa.DBOptions{}); err != nil && err != pilosa.ErrDatabaseExists {
t.Fatal(err)
} else if err := m2.Client().CreateFrame(context.Background(), "d", "f", pilosa.FrameOptions{}); err != nil {
t.Fatal(err)
} else if err := client.RestoreFrame(context.Background(), m0.Server.Host, "d", "f"); err != nil {
t.Fatal(err)
}
@ -262,7 +304,7 @@ path = "/path/to/plugins"
// Main represents a test wrapper for main.Main.
type Main struct {
*main.Main
*server.Command
Stdin bytes.Buffer
Stdout bytes.Buffer
@ -276,16 +318,16 @@ func NewMain() *Main {
panic(err)
}
m := &Main{Main: main.NewMain()}
m := &Main{Command: server.NewCommand()}
m.Config.DataDir = path
m.Config.Host = "localhost:0"
m.Main.Stdin = &m.Stdin
m.Main.Stdout = &m.Stdout
m.Main.Stderr = &m.Stderr
m.Command.Stdin = &m.Stdin
m.Command.Stdout = &m.Stdout
m.Command.Stderr = &m.Stderr
if testing.Verbose() {
m.Main.Stdout = io.MultiWriter(os.Stdout, m.Main.Stdout)
m.Main.Stderr = io.MultiWriter(os.Stderr, m.Main.Stderr)
m.Command.Stdout = io.MultiWriter(os.Stdout, m.Command.Stdout)
m.Command.Stderr = io.MultiWriter(os.Stderr, m.Command.Stderr)
}
return m
@ -303,18 +345,18 @@ func MustRunMain() *Main {
// Close closes the program and removes the underlying data directory.
func (m *Main) Close() error {
defer os.RemoveAll(m.Config.DataDir)
return m.Main.Close()
return m.Command.Close()
}
// Reopen closes the program and reopens it.
func (m *Main) Reopen() error {
if err := m.Main.Close(); err != nil {
if err := m.Command.Close(); err != nil {
return err
}
// Create new main with the same config.
config := m.Config
m.Main = main.NewMain()
m.Command = server.NewCommand()
m.Config = config
// Run new program.
@ -327,6 +369,15 @@ func (m *Main) Reopen() error {
// URL returns the base URL string for accessing the running program.
func (m *Main) URL() string { return "http://" + m.Server.Addr().String() }
// Client returns a client to connect to the program.
func (m *Main) Client() *pilosa.Client {
client, err := pilosa.NewClient(m.Server.Host)
if err != nil {
panic(err)
}
return client
}
// Query executes a query against the program through the HTTP API.
func (m *Main) Query(rawQuery, query string) (string, error) {
resp := MustDo("POST", m.URL()+"/query?"+rawQuery, query)

225
ssh/ssh.go
View file

@ -1,225 +0,0 @@
package ssh
import (
"errors"
"fmt"
"io"
"net"
"os"
"os/user"
"strings"
"syscall"
"golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/agent"
"golang.org/x/crypto/ssh/terminal"
)
type Client struct {
client *ssh.Client
Stderr io.Writer
password string
}
// NewClient wraps up some of the complexity of using the crypto/ssh pacakge
// directly assuming you want to connect using public key auth and you can pass
// a keyfile or your key is accessible through ssh agent.
func NewClient(host, username, keyfile string, stderr io.Writer) (*Client, error) {
if username == "" {
user, err := user.Current()
if err != nil {
return nil, err
}
username = user.Username
}
var authkey ssh.AuthMethod
if keyfile == "" {
sshAgent, err := net.Dial("unix", os.Getenv("SSH_AUTH_SOCK"))
if err != nil {
return nil, err
}
authkey = ssh.PublicKeysCallback(agent.NewClient(sshAgent).Signers)
} else {
return nil, fmt.Errorf("using a keyfile is unimplemented")
}
authpw := ssh.PasswordCallback(func() (secret string, err error) {
fmt.Fprintf(stderr, "password for %v@%v: ", username, host)
pwbytes, err := terminal.ReadPassword(syscall.Stdin)
fmt.Fprintln(stderr)
return string(pwbytes), err
})
config := &ssh.ClientConfig{
User: username,
Auth: []ssh.AuthMethod{authkey, authpw},
}
if strings.Index(host, ":") == -1 {
host = host + ":22"
}
client, err := ssh.Dial("tcp", host, config)
if err != nil {
return nil, fmt.Errorf("NewSSH failed Dial - host: %v, config: %v, err: %v ", host, config, err)
}
return &Client{client: client, Stderr: stderr}, nil
}
type Fleet map[string]*Client
func NewFleet(hosts []string, username, keyfile string, stderr io.Writer) (Fleet, error) {
hosts = DedupHosts(hosts)
clients := make(map[string]*Client, len(hosts))
for _, host := range hosts {
client, err := NewClient(host, username, keyfile, stderr)
if err != nil {
return nil, err
}
clients[host] = client
}
return clients, nil
}
// DedupHosts takes a slice of hosts, strips off any specified ports, and
// returns a de-duplicated slice of hosts.
func DedupHosts(hosts []string) []string {
seenHosts := make(map[string]bool)
ret := []string{}
for _, h := range hosts {
colonIdx := strings.Index(h, ":")
if colonIdx != -1 {
h = h[:colonIdx]
}
if !seenHosts[h] {
ret = append(ret, h)
}
seenHosts[h] = true
}
return ret
}
func (s *Client) NewSession() (*ssh.Session, error) {
return s.client.NewSession()
}
type remoteFile struct {
w io.WriteCloser
sess *ssh.Session
}
func (r *remoteFile) Write(p []byte) (n int, err error) {
return r.w.Write(p)
}
func (r *remoteFile) Close() error {
errc := r.w.Close()
errw := r.sess.Wait()
if errc != nil || errw != nil {
return fmt.Errorf("error closing remote file - close: '%v', wait: '%v'", errc, errw)
}
return nil
}
// OpenFile creates or truncates an existing file of the given name on the
// remote host, and returns a WriteCloser which will write to that file. perm
// will be passed directly to chmod to set the file permissions. rm, touch,
// chmod, cat and support for semicolons, double ampersand, and output
// redirection (>>) must be available in the remote shell.
func (s *Client) OpenFile(name, perm string) (io.WriteCloser, error) {
sess, err := s.NewSession()
if err != nil {
return nil, err
}
w, err := sess.StdinPipe()
if err != nil {
return nil, err
}
if perm == "" {
perm = "0664"
}
err = sess.Start(fmt.Sprintf("rm %v; touch %v && chmod %v %v && cat >> %v", name, name, perm, name, name))
if err != nil {
return nil, err
}
return &remoteFile{w: w, sess: sess}, nil
}
type multiWriteCloser struct {
wcs []io.WriteCloser
ws []io.Writer
}
func newMultiWriteCloser() *multiWriteCloser {
return &multiWriteCloser{
wcs: make([]io.WriteCloser, 0),
ws: make([]io.Writer, 0),
}
}
func (mwc *multiWriteCloser) add(wc io.WriteCloser) {
mwc.wcs = append(mwc.wcs, wc)
mwc.ws = append(mwc.ws, wc)
}
func (mwc *multiWriteCloser) Write(p []byte) (n int, err error) {
mw := io.MultiWriter(mwc.ws...)
return mw.Write(p)
}
func (mwc *multiWriteCloser) Close() error {
errStr := ""
for _, wc := range mwc.wcs {
err := wc.Close()
if err != nil {
errStr = errStr + "; " + err.Error()
}
}
if errStr != "" {
return errors.New(errStr)
}
return nil
}
func (sf Fleet) OpenFile(name, perm string) (io.WriteCloser, error) {
writers := newMultiWriteCloser()
for _, cli := range sf {
wc, err := cli.OpenFile(name, "+x")
if err != nil {
return nil, err
}
writers.add(wc)
}
return writers, nil
}
// WriteFile writes all of data (until EOF) into a file with the given name on
// each of the hosts in the fleet. It sets the permissions on the file to <perm>
// which is any valid input to chmod. If <perm> is the empty string, it defaults
// to 0664
func (sf Fleet) WriteFile(name, perm string, data io.Reader) error {
wc, err := sf.OpenFile(name, perm)
if err != nil {
return fmt.Errorf("opening: %v", err)
}
_, err = io.Copy(wc, data)
if err != nil {
return fmt.Errorf("copying: %v", err)
}
return wc.Close()
}
func (sf Fleet) Get(host string) (*Client, error) {
colonIdx := strings.Index(host, ":")
if colonIdx != -1 {
host = host[:colonIdx]
}
if client, ok := sf[host]; ok {
return client, nil
}
return nil, fmt.Errorf("No client found in fleet: %v for host: %v", sf, host)
}