scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-07 00:55:55 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch '273-use-subcommands'

Browse source
This commit is contained in:
Matt Jaffee 2017-03-07 13:19:18 -06:00
commit 59bbd583d7
28 changed files with 1408 additions and 1351 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

92
README.md
View file

@ -23,18 +23,16 @@ $ go install github.com/pilosa/pilosa/cmd/...
Now run a single pilosa node with the default configuration:
```sh
pilosa
pilosa server
```
If you would like to quickly create a multi-node pilosa cluster, see the `pilosactl create` documentation.
## Configuration
You can specify a configuration by setting the `-config` flag when running `pilosa`.
```sh
pilosa -config custom-config-file.cfg
pilosa server --config custom-config-file.cfg
```
The config file uses the [TOML](https://github.com/toml-lang/toml) configuration file format,
@ -54,6 +52,12 @@ host = "127.0.0.1:15000"
host = "127.0.0.1:15001"
```
You can generate a template config file with default values with:
```sh
pilosa config
```
The first two configuration options will be unique to each node in the cluster:
`data-dir`: directory in which data is stored to disk
@ -256,83 +260,3 @@ $ go install --ldflags="-X main.Version=1.0.0"
```
[Glide]: http://glide.sh/
## Pilosactl
Pilosactl contains a suite of tools for interacting with pilosa. Run `pilosactl` for an overview of commands, and `pilosactl <command> -h` for specific information on that command.
### Create
`pilosactl create` is used to create pilosa clusters. It has a number of options for controlling how the cluster is configured, what hosts it is on, and even the ability to build the pilosa binary locally and copy it to each cluster node automatically. To start pilosa on remote hosts, you only need `ssh` access to those hosts. See `pilosactl create -h` for a full list of options.
Examples:
Create a 5 node cluster locally (using 5 different ports), with a replication factor of 2.
```
pilosactl create \
-serverN 5 \
-replicaN 2
```
Create a cluster on 3 remote hosts - all logs will come to local stderr, pilosa binary must be available on remote hosts. The ssh user on the remote hosts needs to be the same as your local user. Otherwise use the `ssh-user` option.
```
pilosactl create \
-hosts="node1.example.com:15000,node2.example.com:15000,node3.example.com:15000"
```
Create a cluster on 3 remote hosts running OSX, but build the binary locally and copy it up. Stream the stderr of each node to a separate local log file.
```
pilosactl create \
-hosts="mac1.example.com:15000,mac2.example.com:15000,mac3.example.com:15000" \
-copy-binary \
-goos=darwin \
-goarch=amd64 \
-log-file-prefix=clusterlogs
```
### Bagent
`pilosactl bagent` is what you want if you just want to run a simple benchmark against an existing cluster. Running it with no arguments will print some help, including the set of subcommands that it may be passed. Calling a subcommand with `-h'` will print the options for that subcommand. The `agent-num` flag can be passed an integer which can change the behavior the benchmarks that are run. This is useful when multiple invocations of the same benchmark are made by the `bspawn` command - they can each (for example) set different bits even though they all have the same arguments.
E.G.
```
pilosactl bagent \
-hosts="localhost:15000,localhost:15001" \
import -h
```
Multiple subcommands and their arguments may be concatenated at the command line and they will be run serially. This is useful (i.e.) for importing a bunch of data, and then executing queries against it.
This will generate and import a bunch of data, and then execute random queries against it.
```
pilosactl bagent \
-hosts="localhost:15000,localhost:15001" \
import -max-bits-per-map=10000 \
random-query -iterations 100
```
### Bspawn
`pilosactl bspawn` allows you to automate the creation of clusters and the running of complex benchmarks which span multiple benchmark agents against them. It has a number of options which are described by `pilosactl bspawn` with no arguments, and also takes a config file which describes the Benchmark itself - this file is described below.
#### Configuration Format
The configuration file is a json object with the top level key `benchmarks`. This contains a list of objects each of which represents a `bagent` command (the `args` key) that will be run some number of times concurrently (the `num` key), and a `name` which should describe the overall effect that command. An example is below.
```json
{
"benchmarks": [
{
"num": 3,
"name": "set-diags",
"args": ["diagonal-set-bits", "-iterations", "30000", "-client-type", "round_robin"]
},
{
"num": 2,
"name": "rand-plus-zipf",
"args": ["random-set-bits", "-iterations", "20000", "zipf", "-iterations", "100"]
}
]
}
```
All of the benchmarks, and agents are run concurrently. Each agent will be passed an `agent-num` which can modify the behavior in a way that is benchmark specific. See the documentation for each benchmark to see how `agent-num` changes its behavior.

42
cmd/backup.go Normal file
View file

@ -0,0 +1,42 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var backuper = ctl.NewBackupCommand(os.Stdin, os.Stdout, os.Stderr)
var backupCmd = &cobra.Command{
Use: "backup",
Short: "Backup data from pilosa.",
Long: `
Backs up the database and frame from across the cluster into a single file.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := backuper.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
backupCmd.Flags().StringVarP(&backuper.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
backupCmd.Flags().StringVarP(&backuper.Database, "database", "d", "", "Pilosa database to backup into.")
backupCmd.Flags().StringVarP(&backuper.Frame, "frame", "f", "", "Frame to backup into.")
backupCmd.Flags().StringVarP(&backuper.Path, "output-file", "o", "", "File to write backup to - default stdout")
err := viper.BindPFlags(backupCmd.Flags())
if err != nil {
log.Fatalf("Error binding backup flags: %v", err)
}
RootCmd.AddCommand(backupCmd)
}

43
cmd/bench.go Normal file
View file

@ -0,0 +1,43 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var bencher = ctl.NewBenchCommand(os.Stdin, os.Stdout, os.Stderr)
var benchCmd = &cobra.Command{
Use: "bench",
Short: "Benchmark operations.",
Long: `
Executes a benchmark for a given operation against the database.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := bencher.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
benchCmd.Flags().StringVarP(&bencher.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
benchCmd.Flags().StringVarP(&bencher.Database, "database", "d", "", "Pilosa database to benchmark.")
benchCmd.Flags().StringVarP(&bencher.Frame, "frame", "f", "", "Frame to benchmark.")
benchCmd.Flags().StringVarP(&bencher.Op, "operation", "o", "set-bit", "Operation to perform: choose from [set-bit]")
benchCmd.Flags().IntVarP(&bencher.N, "num", "n", 0, "Number of operations to perform.")
err := viper.BindPFlags(benchCmd.Flags())
if err != nil {
log.Fatalf("Error binding bench flags: %v", err)
}
RootCmd.AddCommand(benchCmd)
}

35
cmd/check.go Normal file
View file

@ -0,0 +1,35 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var checker = ctl.NewCheckCommand(os.Stdin, os.Stdout, os.Stderr)
var checkCmd = &cobra.Command{
Use: "check <path> [path2]...",
Short: "Do a consistency check on a pilosa data file.",
Long: `
Performs a consistency check on data files.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
}
checker.Paths = args
if err := checker.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(checkCmd)
}

29
cmd/config.go Normal file
View file

@ -0,0 +1,29 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var conf = ctl.NewConfigCommand(os.Stdin, os.Stdout, os.Stderr)
var confCmd = &cobra.Command{
Use: "config",
Short: "Print the default configuration.",
Long: `config prints the default configuration to stdout
`,
Run: func(cmd *cobra.Command, args []string) {
if err := conf.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(confCmd)
}

49
cmd/export.go Normal file
View file

@ -0,0 +1,49 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var exporter = ctl.NewExportCommand(os.Stdin, os.Stdout, os.Stderr)
var exportCmd = &cobra.Command{
Use: "export",
Short: "Export data from pilosa.",
Long: `
Bulk exports a fragment to a CSV file. If the OUTFILE is not specified then
the output is written to STDOUT.
The format of the CSV file is:
BITMAPID,PROFILEID
The file does not contain any headers.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := exporter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
exportCmd.Flags().StringVarP(&exporter.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
exportCmd.Flags().StringVarP(&exporter.Database, "database", "d", "", "Pilosa database to export into.")
exportCmd.Flags().StringVarP(&exporter.Frame, "frame", "f", "", "Frame to export into.")
exportCmd.Flags().StringVarP(&exporter.Path, "output-file", "o", "", "File to write export to - default stdout")
err := viper.BindPFlags(exportCmd.Flags())
if err != nil {
log.Fatalf("Error binding export flags: %v", err)
}
RootCmd.AddCommand(exportCmd)
}

50
cmd/import.go Normal file
View file

@ -0,0 +1,50 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var importer = ctl.NewImportCommand(os.Stdin, os.Stdout, os.Stderr)
var importCmd = &cobra.Command{
Use: "import",
Short: "Bulk load data into pilosa.",
Long: `Bulk imports one or more CSV files to a host's database and frame. The bits
of the CSV file are grouped by slice for the most efficient import.
The format of the CSV file is:
BITMAPID,PROFILEID,[TIME]
The file should contain no headers. The TIME column is optional and can be
omitted. If it is present then its format should be YYYY-MM-DDTHH:MM.
`,
Run: func(cmd *cobra.Command, args []string) {
importer.Paths = args
if err := importer.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
importCmd.Flags().StringVarP(&importer.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
importCmd.Flags().StringVarP(&importer.Database, "database", "d", "", "Pilosa database to import into.")
importCmd.Flags().StringVarP(&importer.Frame, "frame", "f", "", "Frame to import into.")
importCmd.Flags().IntVarP(&importer.BufferSize, "buffer-size", "s", 10000000, "Number of bits to buffer/sort before importing.")
err := viper.BindPFlags(importCmd.Flags())
if err != nil {
log.Fatalf("Error binding import flags: %v", err)
}
RootCmd.AddCommand(importCmd)
}

38
cmd/inspect.go Normal file
View file

@ -0,0 +1,38 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var inspecter = ctl.NewInspectCommand(os.Stdin, os.Stdout, os.Stderr)
var inspectCmd = &cobra.Command{
Use: "inspect",
Short: "Get stats on a pilosa data file.",
Long: `
Inspects a data file and provides stats.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
} else if len(args) > 1 {
fmt.Println("only one path allowed")
return
}
inspecter.Path = args[0]
if err := inspecter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(inspectCmd)
}

188
cmd/pilosa/main.go
View file

@ -1,197 +1,15 @@
package main
import (
"errors"
"flag"
"fmt"
"io"
"math/rand"
"os"
"os/signal"
"path/filepath"
"runtime/pprof"
"strings"
"time"
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
)
// Version and BuildTime hold the version/build time information passed in at compile time.
var (
Version string
BuildTime string
)
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
rand.Seed(time.Now().UTC().UnixNano())
}
const (
// DefaultDataDir is the default data directory.
DefaultDataDir = "~/.pilosa"
"github.com/pilosa/pilosa/cmd"
)
func main() {
m := NewMain()
m.Server.Handler.Version = Version
fmt.Fprintf(m.Stderr, "Pilosa %s, build time %s\n", Version, BuildTime)
// Parse command line arguments.
if err := m.ParseFlags(os.Args[1:]); err != nil {
fmt.Fprintln(m.Stderr, err)
os.Exit(2)
}
// Start CPU profiling.
if m.CPUProfile != "" {
f, err := os.Create(m.CPUProfile)
if err != nil {
fmt.Fprintf(m.Stderr, "create cpu profile: %v", err)
os.Exit(1)
}
defer f.Close()
fmt.Fprintln(m.Stderr, "Starting cpu profile")
pprof.StartCPUProfile(f)
time.AfterFunc(m.CPUTime, func() {
fmt.Fprintln(m.Stderr, "Stopping cpu profile")
pprof.StopCPUProfile()
f.Close()
})
}
// Execute the program.
if err := m.Run(); err != nil {
fmt.Fprintln(m.Stderr, err)
fmt.Fprintln(m.Stderr, "stopping profile")
os.Exit(1)
}
// First SIGKILL causes server to shut down gracefully.
c := make(chan os.Signal, 2)
signal.Notify(c, os.Interrupt)
sig := <-c
fmt.Fprintf(m.Stderr, "Received %s; gracefully shutting down...\n", sig.String())
// Second signal causes a hard shutdown.
go func() { <-c; os.Exit(1) }()
if err := m.Close(); err != nil {
fmt.Fprintln(m.Stderr, err)
if err := cmd.RootCmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(1)
}
}
// Main represents the main program execution.
type Main struct {
Server *pilosa.Server
// Configuration options.
ConfigPath string
Config *pilosa.Config
// Profiling options.
CPUProfile string
CPUTime time.Duration
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain returns a new instance of Main.
func NewMain() *Main {
return &Main{
Server: pilosa.NewServer(),
Config: pilosa.NewConfig(),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the main program execution.
func (m *Main) Run(args ...string) error {
// Notify user of config file.
if m.ConfigPath != "" {
fmt.Fprintf(m.Stdout, "Using config: %s\n", m.ConfigPath)
}
// Setup logging output.
m.Server.LogOutput = m.Stderr
// Configure index.
fmt.Fprintf(m.Stderr, "Using data from: %s\n", m.Config.DataDir)
m.Server.Index.Path = m.Config.DataDir
m.Server.Index.Stats = pilosa.NewExpvarStatsClient()
// Build cluster from config file.
m.Server.Host = m.Config.Host
m.Server.Cluster = m.Config.PilosaCluster()
// Set configuration options.
m.Server.AntiEntropyInterval = time.Duration(m.Config.AntiEntropy.Interval)
// Initialize server.
if err := m.Server.Open(); err != nil {
return err
}
fmt.Fprintf(m.Stderr, "Listening as http://%s\n", m.Server.Host)
return nil
}
// Close shuts down the server.
func (m *Main) Close() error {
return m.Server.Close()
}
// ParseFlags parses command line flags from args.
func (m *Main) ParseFlags(args []string) error {
fs := flag.NewFlagSet("pilosa", flag.ContinueOnError)
fs.StringVar(&m.CPUProfile, "cpuprofile", "", "cpu profile")
fs.DurationVar(&m.CPUTime, "cputime", 30*time.Second, "cpu profile duration")
fs.StringVar(&m.ConfigPath, "config", "", "config path")
fs.SetOutput(m.Stderr)
if err := fs.Parse(args); err != nil {
return err
}
// Load config, if specified.
if m.ConfigPath != "" {
if _, err := toml.DecodeFile(m.ConfigPath, &m.Config); err != nil {
return err
}
}
// Use default data directory if one is not specified.
if m.Config.DataDir == "" {
m.Config.DataDir = DefaultDataDir
}
// Expand home directory.
prefix := "~" + string(filepath.Separator)
if strings.HasPrefix(m.Config.DataDir, prefix) {
// u, err := user.Current()
HomeDir := os.Getenv("HOME")
/*if err != nil {
return err
} else*/if HomeDir == "" {
return errors.New("data directory not specified and no home dir available")
}
m.Config.DataDir = filepath.Join(HomeDir, strings.TrimPrefix(m.Config.DataDir, prefix))
}
return nil
}

1028
cmd/pilosactl/main.go
View file

File diff suppressed because it is too large Load diff

1
cmd/pilosactl/main_test.go
View file

@ -1 +0,0 @@
package main_test

42
cmd/restore.go Normal file
View file

@ -0,0 +1,42 @@
package cmd
import (
"context"
"fmt"
"log"
"os"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/ctl"
)
var restorer = ctl.NewRestoreCommand(os.Stdin, os.Stdout, os.Stderr)
var restoreCmd = &cobra.Command{
Use: "restore",
Short: "Restore data to pilosa from a backup file.",
Long: `
Restores a frame to the cluster from a backup file.
`,
Run: func(cmd *cobra.Command, args []string) {
if err := restorer.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
restoreCmd.Flags().StringVarP(&restorer.Host, "host", "", "localhost:15000", "host:port of Pilosa.")
restoreCmd.Flags().StringVarP(&restorer.Database, "database", "d", "", "Pilosa database to restore into.")
restoreCmd.Flags().StringVarP(&restorer.Frame, "frame", "f", "", "Frame to restore into.")
restoreCmd.Flags().StringVarP(&restorer.Path, "input-file", "i", "", "File to restore from.")
err := viper.BindPFlags(restoreCmd.Flags())
if err != nil {
log.Fatalf("Error binding restore flags: %v", err)
}
RootCmd.AddCommand(restoreCmd)
}

33
cmd/root.go Normal file
View file

@ -0,0 +1,33 @@
package cmd
import "github.com/spf13/cobra"
var (
Version string
BuildTime string
)
var RootCmd = &cobra.Command{
Use: "pilosa",
Short: "Pilosa - A Distributed In-memory Binary Bitmap Index.",
// TODO - is documentation actually there?
Long: `Pilosa is a fast index to turbocharge your database.
This binary contains Pilosa itself, as well as common
tools for administering pilosa, importing/exporting data,
backing up, and more. Complete documentation is available
at http://pilosa.com/docs
`,
}
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
RootCmd.Long = RootCmd.Long + "Version: " + Version + "\nBuild Time: " + BuildTime + "\n"
}

89
cmd/server.go Normal file
View file

@ -0,0 +1,89 @@
package cmd
import (
"fmt"
"log"
"os"
"os/signal"
"runtime/pprof"
"time"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/pilosa/pilosa/server"
)
var serve = server.NewCommand()
var serveCmd = &cobra.Command{
Use: "server",
Short: "Run Pilosa.",
Long: `pilosa server runs Pilosa.
It will load existing data from the configured
directory, and start listening client connections
on the configured port.`,
Run: func(cmd *cobra.Command, args []string) {
serve.Server.Handler.Version = server.Version
fmt.Fprintf(serve.Stderr, "Pilosa %s, build time %s\n", server.Version, server.BuildTime)
// Parse command line arguments.
if err := serve.SetupConfig(args); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(2)
}
// Start CPU profiling.
if serve.CPUProfile != "" {
f, err := os.Create(serve.CPUProfile)
if err != nil {
fmt.Fprintf(serve.Stderr, "create cpu profile: %v", err)
os.Exit(1)
}
defer f.Close()
fmt.Fprintln(serve.Stderr, "Starting cpu profile")
pprof.StartCPUProfile(f)
time.AfterFunc(serve.CPUTime, func() {
fmt.Fprintln(serve.Stderr, "Stopping cpu profile")
pprof.StopCPUProfile()
f.Close()
})
}
// Execute the program.
if err := serve.Run(); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(1)
}
// First SIGKILL causes server to shut down gracefully.
c := make(chan os.Signal, 2)
signal.Notify(c, os.Interrupt)
sig := <-c
fmt.Fprintf(serve.Stderr, "Received %s; gracefully shutting down...\n", sig.String())
// Second signal causes a hard shutdown.
go func() { <-c; os.Exit(1) }()
if err := serve.Close(); err != nil {
fmt.Fprintln(serve.Stderr, err)
os.Exit(1)
}
},
}
func init() {
serveCmd.Flags().StringVarP(&serve.ConfigPath, "config", "c", "", "Configuration file to read from")
serveCmd.Flags().StringVarP(&serve.CPUProfile, "cpuprofile", "", "", "Where to store CPU profile")
serveCmd.Flags().DurationVarP(&serve.CPUTime, "cputime", "", 30*time.Second, "CPU profile duration")
err := viper.BindPFlags(serveCmd.Flags())
if err != nil {
log.Fatalf("Error binding server flags: %v", err)
}
RootCmd.AddCommand(serveCmd)
}

44
cmd/sort.go Normal file
View file

@ -0,0 +1,44 @@
package cmd
import (
"context"
"fmt"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var sorter = ctl.NewSortCommand(os.Stdin, os.Stdout, os.Stderr)
var sortCmd = &cobra.Command{
Use: "sort <path>",
Short: "Sort import data for optimal import performance.",
Long: `
Sorts the import data at PATH into the optimal sort order for importing.
The format of the CSV file is:
BITMAPID,PROFILEID
The file should contain no headers.
`,
Run: func(cmd *cobra.Command, args []string) {
if len(args) == 0 {
fmt.Println("path required")
return
} else if len(args) > 1 {
fmt.Println("only one path supported")
return
}
sorter.Path = args[0]
if err := sorter.Run(context.Background()); err != nil {
fmt.Println(err)
}
},
}
func init() {
RootCmd.AddCommand(sortCmd)
}

72
ctl/backup.go Normal file
View file

@ -0,0 +1,72 @@
package ctl
import (
"context"
"errors"
"io"
"os"
"github.com/pilosa/pilosa"
)
// BackupCommand represents a command for backing up a frame.
type BackupCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to backup.
Database string
Frame string
// Output file to write to.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewBackupCommand returns a new instance of BackupCommand.
func NewBackupCommand(stdin io.Reader, stdout, stderr io.Writer) *BackupCommand {
return &BackupCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the backup.
func (cmd *BackupCommand) Run(ctx context.Context) error {
// Validate arguments.
if cmd.Path == "" {
return errors.New("output file required")
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Open output file.
f, err := os.Create(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Begin streaming backup.
if err := client.BackupTo(ctx, f, cmd.Database, cmd.Frame); err != nil {
return err
}
// Sync & close file to ensure durability.
if err := f.Sync(); err != nil {
return err
} else if err = f.Close(); err != nil {
return err
}
return nil
}

92
ctl/bench.go Normal file
View file

@ -0,0 +1,92 @@
package ctl
import (
"context"
"errors"
"fmt"
"io"
"math/rand"
"time"
"github.com/pilosa/pilosa"
)
// BenchCommand represents a command for benchmarking database operations.
type BenchCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to execute against.
Database string
Frame string
// Type of operation and number to execute.
Op string
N int
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewBenchCommand returns a new instance of BenchCommand.
func NewBenchCommand(stdin io.Reader, stdout, stderr io.Writer) *BenchCommand {
return &BenchCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the bench command.
func (cmd *BenchCommand) Run(ctx context.Context) error {
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
switch cmd.Op {
case "set-bit":
return cmd.runSetBit(ctx, client)
case "":
return errors.New("op required")
default:
return fmt.Errorf("unknown bench op: %q", cmd.Op)
}
}
// runSetBit executes a benchmark of random SetBit() operations.
func (cmd *BenchCommand) runSetBit(ctx context.Context, client *pilosa.Client) error {
if cmd.N == 0 {
return errors.New("operation count required")
} else if cmd.Database == "" {
return pilosa.ErrDatabaseRequired
} else if cmd.Frame == "" {
return pilosa.ErrFrameRequired
}
const maxBitmapID = 1000
const maxProfileID = 100000
startTime := time.Now()
// Execute operation continuously.
for i := 0; i < cmd.N; i++ {
bitmapID := rand.Intn(maxBitmapID)
profileID := rand.Intn(maxProfileID)
q := fmt.Sprintf(`SetBit(id=%d, frame="%s", profileID=%d)`, bitmapID, cmd.Frame, profileID)
if _, err := client.ExecuteQuery(ctx, cmd.Database, q, true); err != nil {
return err
}
}
// Print results.
elapsed := time.Since(startTime)
fmt.Fprintf(cmd.Stdout, "Executed %d operations in %s (%0.3f op/sec)\n", cmd.N, elapsed, float64(cmd.N)/elapsed.Seconds())
return nil
}

114
ctl/check.go Normal file
View file

@ -0,0 +1,114 @@
package ctl
import (
"context"
"fmt"
"io"
"os"
"path/filepath"
"syscall"
"github.com/pilosa/pilosa/roaring"
)
// CheckCommand represents a command for performing consistency checks on data files.
type CheckCommand struct {
// Data file paths.
Paths []string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewCheckCommand returns a new instance of CheckCommand.
func NewCheckCommand(stdin io.Reader, stdout, stderr io.Writer) *CheckCommand {
return &CheckCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the check command.
func (cmd *CheckCommand) Run(ctx context.Context) error {
for _, path := range cmd.Paths {
switch filepath.Ext(path) {
case "":
if err := cmd.checkBitmapFile(path); err != nil {
return err
}
case ".cache":
if err := cmd.checkCacheFile(path); err != nil {
return err
}
case ".snapshotting":
if err := cmd.checkSnapshotFile(path); err != nil {
return err
}
}
}
return nil
}
// checkBitmapFile performs a consistency check on path for a roaring bitmap file.
func (cmd *CheckCommand) checkBitmapFile(path string) error {
// Open file handle.
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
// Memory map the file.
data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
return err
}
defer syscall.Munmap(data)
// Attach the mmap file to the bitmap.
bm := roaring.NewBitmap()
if err := bm.UnmarshalBinary(data); err != nil {
return err
}
// Perform consistency check.
if err := bm.Check(); err != nil {
// Print returned errors.
switch err := err.(type) {
case roaring.ErrorList:
for i := range err {
fmt.Fprintf(cmd.Stdout, "%s: %s\n", path, err[i].Error())
}
default:
fmt.Fprintf(cmd.Stdout, "%s: %s\n", path, err.Error())
}
}
// Print success message if no errors were found.
fmt.Fprintf(cmd.Stdout, "%s: ok\n", path)
return nil
}
// checkCacheFile performs a consistency check on path for a cache file.
func (cmd *CheckCommand) checkCacheFile(path string) error {
fmt.Fprintf(cmd.Stderr, "%s: ignoring cache file\n", path)
return nil
}
// checkSnapshotFile performs a consistency check on path for a snapshot file.
func (cmd *CheckCommand) checkSnapshotFile(path string) error {
fmt.Fprintf(cmd.Stderr, "%s: ignoring snapshot file\n", path)
return nil
}

43
ctl/config.go Normal file
View file

@ -0,0 +1,43 @@
package ctl
import (
"context"
"fmt"
"io"
"strings"
)
// ConfigCommand represents a command for printing a default config.
type ConfigCommand struct {
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewConfigCommand returns a new instance of ConfigCommand.
func NewConfigCommand(stdin io.Reader, stdout, stderr io.Writer) *ConfigCommand {
return &ConfigCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run prints out the default config.
func (cmd *ConfigCommand) Run(ctx context.Context) error {
fmt.Fprintln(cmd.Stdout, strings.TrimSpace(`
data-dir = "~/.pilosa"
host = "localhost:15000"
[cluster]
replicas = 1
[[cluster.node]]
host = "localhost:15000"
[plugins]
path = ""
`)+"\n")
return nil
}

91
ctl/export.go Normal file
View file

@ -0,0 +1,91 @@
package ctl
import (
"context"
"io"
"log"
"os"
"github.com/pilosa/pilosa"
)
// ExportCommand represents a command for bulk exporting data from a server.
type ExportCommand struct {
// Remote host and port.
Host string
// Name of the database & frame to export from.
Database string
Frame string
// Filename to export to.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewExportCommand returns a new instance of ExportCommand.
func NewExportCommand(stdin io.Reader, stdout, stderr io.Writer) *ExportCommand {
return &ExportCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the export.
func (cmd *ExportCommand) Run(ctx context.Context) error {
logger := log.New(cmd.Stderr, "", log.LstdFlags)
// Validate arguments.
if cmd.Database == "" {
return pilosa.ErrDatabaseRequired
} else if cmd.Frame == "" {
return pilosa.ErrFrameRequired
}
// Use output file, if specified.
// Otherwise use STDOUT.
var w io.Writer = cmd.Stdout
if cmd.Path != "" {
f, err := os.Create(cmd.Path)
if err != nil {
return err
}
defer f.Close()
w = f
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Determine slice count.
maxSlices, err := client.MaxSliceByDatabase(ctx)
if err != nil {
return err
}
// Export each slice.
for slice := uint64(0); slice <= maxSlices[cmd.Database]; slice++ {
logger.Printf("exporting slice: %d", slice)
if err := client.ExportCSV(ctx, cmd.Database, cmd.Frame, slice, w); err != nil {
return err
}
}
// Close writer, if applicable.
if w, ok := w.(io.Closer); ok {
if err := w.Close(); err != nil {
return err
}
}
return nil
}

40
pilosactl/import.go → ctl/import.go
View file

@ -1,17 +1,14 @@
package pilosactl
package ctl
import (
"context"
"encoding/csv"
"errors"
"flag"
"fmt"
"io"
"io/ioutil"
"log"
"os"
"strconv"
"strings"
"time"
"github.com/pilosa/pilosa"
@ -56,41 +53,6 @@ func (cmd *ImportCommand) String() string {
return fmt.Sprint(*cmd)
}
// ParseFlags parses command line flags from args.
func (cmd *ImportCommand) ParseFlags(args []string) error {
fs := flag.NewFlagSet("pilosactl", flag.ContinueOnError)
fs.SetOutput(ioutil.Discard)
fs.StringVar(&cmd.Host, "host", "localhost:15000", "host:port")
fs.StringVar(&cmd.Database, "d", "", "database")
fs.StringVar(&cmd.Frame, "f", "", "frame")
fs.IntVar(&cmd.BufferSize, "buffer-size", cmd.BufferSize, "buffer size")
if err := fs.Parse(args); err != nil {
return err
}
// Extract the import paths.
cmd.Paths = fs.Args()
return nil
}
// Usage returns the usage message to be printed.
func (cmd *ImportCommand) Usage() string {
return strings.TrimSpace(`
usage: pilosactl import -host HOST -d database -f frame paths
Bulk imports one or more CSV files to a host's database and frame. The bits
of the CSV file are grouped by slice for the most efficient import.
The format of the CSV file is:
BITMAPID,PROFILEID,[TIME]
The file should contain no headers. The TIME column is optional and can be
omitted. If it is present then its format should be YYYY-MM-DDTHH:MM.
`)
}
// Run executes the main program execution.
func (cmd *ImportCommand) Run(ctx context.Context) error {
logger := log.New(cmd.Stderr, "", log.LstdFlags)

94
ctl/inspect.go Normal file
View file

@ -0,0 +1,94 @@
package ctl
import (
"context"
"fmt"
"io"
"os"
"syscall"
"text/tabwriter"
"time"
"unsafe"
"github.com/pilosa/pilosa/roaring"
)
// InspectCommand represents a command for inspecting fragment data files.
type InspectCommand struct {
// Path to data file
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewInspectCommand returns a new instance of InspectCommand.
func NewInspectCommand(stdin io.Reader, stdout, stderr io.Writer) *InspectCommand {
return &InspectCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the inspect command.
func (cmd *InspectCommand) Run(ctx context.Context) error {
// Open file handle.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return err
}
// Memory map the file.
data, err := syscall.Mmap(int(f.Fd()), 0, int(fi.Size()), syscall.PROT_READ, syscall.MAP_SHARED)
if err != nil {
return err
}
defer syscall.Munmap(data)
// Attach the mmap file to the bitmap.
t := time.Now()
fmt.Fprintf(cmd.Stderr, "unmarshaling bitmap...")
bm := roaring.NewBitmap()
if err := bm.UnmarshalBinary(data); err != nil {
return err
}
fmt.Fprintf(cmd.Stderr, " (%s)\n", time.Since(t))
// Retrieve stats.
t = time.Now()
fmt.Fprintf(cmd.Stderr, "calculating stats...")
info := bm.Info()
fmt.Fprintf(cmd.Stderr, " (%s)\n", time.Since(t))
// Print top-level info.
fmt.Fprintf(cmd.Stdout, "== Bitmap Info ==\n")
fmt.Fprintf(cmd.Stdout, "Containers: %d\n", len(info.Containers))
fmt.Fprintf(cmd.Stdout, "Operations: %d\n", info.OpN)
fmt.Fprintln(cmd.Stdout, "")
// Print info for each container.
fmt.Fprintln(cmd.Stdout, "== Containers ==")
tw := tabwriter.NewWriter(cmd.Stdout, 0, 8, 0, '\t', 0)
fmt.Fprintf(tw, "%s\t%s\t% 8s \t% 8s\t%s\n", "KEY", "TYPE", "N", "ALLOC", "OFFSET")
for _, ci := range info.Containers {
fmt.Fprintf(tw, "%d\t%s\t% 8d \t% 8d \t0x%08x\n",
ci.Key,
ci.Type,
ci.N,
ci.Alloc,
uintptr(ci.Pointer)-uintptr(unsafe.Pointer(&data[0])),
)
}
tw.Flush()
return nil
}

65
ctl/restore.go Normal file
View file

@ -0,0 +1,65 @@
package ctl
import (
"context"
"errors"
"io"
"os"
"github.com/pilosa/pilosa"
)
// RestoreCommand represents a command for restoring a frame from a backup.
type RestoreCommand struct {
// Destination host and port.
Host string
// Name of the database & frame to backup.
Database string
Frame string
// Import file to read from.
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewRestoreCommand returns a new instance of RestoreCommand.
func NewRestoreCommand(stdin io.Reader, stdout, stderr io.Writer) *RestoreCommand {
return &RestoreCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the restore command.
func (cmd *RestoreCommand) Run(ctx context.Context) error {
// Validate arguments.
if cmd.Path == "" {
return errors.New("backup file required")
}
// Create a client to the server.
client, err := pilosa.NewClient(cmd.Host)
if err != nil {
return err
}
// Open backup file.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Restore backup file to the cluster.
if err := client.RestoreFrom(ctx, f, cmd.Database, cmd.Frame); err != nil {
return err
}
return nil
}

138
ctl/sort.go Normal file
View file

@ -0,0 +1,138 @@
package ctl
import (
"bufio"
"context"
"encoding/csv"
"errors"
"fmt"
"io"
"os"
"sort"
"strconv"
"time"
"github.com/pilosa/pilosa"
)
// SortCommand represents a command for sorting import data.
type SortCommand struct {
// Filename to sort
Path string
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewSortCommand returns a new instance of SortCommand.
func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *SortCommand {
return &SortCommand{
Stdin: stdin,
Stdout: stdout,
Stderr: stderr,
}
}
// Run executes the sort command.
func (cmd *SortCommand) Run(ctx context.Context) error {
// Open file for reading.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Read rows as bits.
r := csv.NewReader(f)
r.FieldsPerRecord = -1
a := make([]pilosa.Bit, 0, 1000000)
for {
bitmapID, profileID, timestamp, err := readCSVRow(r)
if err == io.EOF {
break
} else if err == errBlank {
continue
} else if err != nil {
return err
}
a = append(a, pilosa.Bit{BitmapID: bitmapID, ProfileID: profileID, Timestamp: timestamp})
}
// Sort bits by position.
sort.Sort(pilosa.BitsByPos(a))
// Rewrite to STDOUT.
w := bufio.NewWriter(cmd.Stdout)
buf := make([]byte, 0, 1024)
for _, bit := range a {
// Write CSV to buffer.
buf = buf[:0]
buf = strconv.AppendUint(buf, bit.BitmapID, 10)
buf = append(buf, ',')
buf = strconv.AppendUint(buf, bit.ProfileID, 10)
if bit.Timestamp != 0 {
buf = append(buf, ',')
buf = append(buf, time.Unix(0, bit.Timestamp).UTC().Format(pilosa.TimeFormat)...)
}
buf = append(buf, '\n')
// Write to output.
if _, err := w.Write(buf); err != nil {
return err
}
}
// Ensure buffer is flushed before exiting.
if err := w.Flush(); err != nil {
return err
}
return nil
}
// readCSVRow reads a bitmap/profile pair from a CSV row.
func readCSVRow(r *csv.Reader) (bitmapID, profileID uint64, timestamp int64, err error) {
// Read CSV row.
record, err := r.Read()
if err != nil {
return 0, 0, 0, err
}
// Ignore blank rows.
if record[0] == "" {
return 0, 0, 0, errBlank
} else if len(record) < 2 {
return 0, 0, 0, fmt.Errorf("bad column count: %d", len(record))
}
// Parse bitmap id.
bitmapID, err = strconv.ParseUint(record[0], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid bitmap id: %q", record[0])
}
// Parse bitmap id.
profileID, err = strconv.ParseUint(record[1], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid profile id: %q", record[1])
}
// Parse timestamp, if available.
if len(record) > 2 && record[2] != "" {
t, err := time.Parse(pilosa.TimeFormat, record[2])
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid timestamp: %q", record[2])
}
timestamp = t.UnixNano()
}
return bitmapID, profileID, timestamp, nil
}
// errBlank indicates a blank row in a CSV file.
var errBlank = errors.New("blank row")

50
glide.lock generated
View file

@ -1,5 +1,5 @@
hash: 469de49a1736f34a11e9b0e490f7c1da1d8cb0219fed4bf3ad9e71344ca7f58a
updated: 2017-02-09T17:03:01.816613507-06:00
hash: 7de62dbaf3cc1dc4959f4f6d8213102cb182b4dd7a87b3ac29260ad6bc1b0cef
updated: 2017-03-03T12:25:48.088390296-06:00
imports:
- name: github.com/boltdb/bolt
version: 4b1ebc1869ad66568b313d0dc410e2be72670dda
@ -13,6 +13,8 @@ imports:
version: 346938d642f2ec3594ed81d874461961cd0faa76
subpackages:
- spew
- name: github.com/fsnotify/fsnotify
version: 7d7316ed6e1ed2de075aab8dfc76de5d158d66e1
- name: github.com/gogo/protobuf
version: a9cd0c35b97daf74d0ebf3514c5254814b2703b4
subpackages:
@ -23,10 +25,54 @@ imports:
- lru
- name: github.com/golang/protobuf
version: 888eb0692c857ec880338addf316bd662d5e630e
subpackages:
- proto
- name: github.com/hashicorp/hcl
version: 630949a3c5fa3c613328e1b8256052cbc2327c9b
subpackages:
- hcl/ast
- hcl/parser
- hcl/scanner
- hcl/strconv
- hcl/token
- json/parser
- json/scanner
- json/token
- name: github.com/inconshreveable/mousetrap
version: 76626ae9c91c4f2a10f34cad8ce83ea42c93bb75
- name: github.com/magiconair/properties
version: b3b15ef068fd0b17ddf408a23669f20811d194d2
- name: github.com/mitchellh/mapstructure
version: db1efb556f84b25a0a13a04aad883943538ad2e0
- name: github.com/pelletier/go-buffruneio
version: c37440a7cf42ac63b919c752ca73a85067e05992
- name: github.com/pelletier/go-toml
version: 13d49d4606eb801b8f01ae542b4afc4c6ee3d84a
- name: github.com/satori/go.uuid
version: 879c5887cd475cd7864858769793b2ceb0d44feb
- name: github.com/spf13/afero
version: 9be650865eab0c12963d8753212f4f9c66cdcf12
subpackages:
- mem
- name: github.com/spf13/cast
version: 4f1683a2242a92e62d6ff705a30e435cbf2b50a3
- name: github.com/spf13/cobra
version: fcd0c5a1df88f5d6784cb4feead962c3f3d0b66c
- name: github.com/spf13/jwalterweatherman
version: fa7ca7e836cf3a8bb4ebf799f472c12d7e903d66
- name: github.com/spf13/pflag
version: 9ff6c6923cfffbcd502984b8e0c80539a94968b7
- name: github.com/spf13/viper
version: 7538d73b4eb9511d85a9f1dfef202eeb8ac260f4
- name: golang.org/x/sys
version: c200b10b5d5e122be351b67af224adc6128af5bf
subpackages:
- unix
- name: golang.org/x/text
version: 5a42fa2464759cbb7ee0af9de00b54d69f09a29c
subpackages:
- transform
- unicode/norm
- name: gopkg.in/yaml.v2
version: a3f3340b5840cee44f372bddb5880fcbc419b46a
testImports: []

2
glide.yaml
View file

@ -27,3 +27,5 @@ import:
- package: github.com/golang/protobuf
- package: github.com/satori/go.uuid
version: ^1.1.0
- package: github.com/spf13/cobra
- package: github.com/spf13/viper

131
server/server.go Normal file
View file

@ -0,0 +1,131 @@
package server
import (
"errors"
"fmt"
"io"
"math/rand"
"os"
"path/filepath"
"strings"
"time"
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
)
// Version and BuildTime hold the version/build time information passed in at compile time.
var (
Version string
BuildTime string
)
func init() {
if Version == "" {
Version = "v0.0.0"
}
if BuildTime == "" {
BuildTime = "not recorded"
}
rand.Seed(time.Now().UTC().UnixNano())
}
const (
// DefaultDataDir is the default data directory.
DefaultDataDir = "~/.pilosa"
)
// Command represents the state of the pilosa server command.
type Command struct {
Server *pilosa.Server
// Configuration options.
ConfigPath string
Config *pilosa.Config
// Profiling options.
CPUProfile string
CPUTime time.Duration
// Standard input/output
Stdin io.Reader
Stdout io.Writer
Stderr io.Writer
}
// NewMain returns a new instance of Main.
func NewCommand() *Command {
return &Command{
Server: pilosa.NewServer(),
Config: pilosa.NewConfig(),
Stdin: os.Stdin,
Stdout: os.Stdout,
Stderr: os.Stderr,
}
}
// Run executes the pilosa server.
func (m *Command) Run(args ...string) error {
// Notify user of config file.
if m.ConfigPath != "" {
fmt.Fprintf(m.Stdout, "Using config: %s\n", m.ConfigPath)
}
// Setup logging output.
m.Server.LogOutput = m.Stderr
// Configure index.
fmt.Fprintf(m.Stderr, "Using data from: %s\n", m.Config.DataDir)
m.Server.Index.Path = m.Config.DataDir
m.Server.Index.Stats = pilosa.NewExpvarStatsClient()
// Build cluster from config file.
m.Server.Host = m.Config.Host
m.Server.Cluster = m.Config.PilosaCluster()
// Set configuration options.
m.Server.AntiEntropyInterval = time.Duration(m.Config.AntiEntropy.Interval)
// Initialize server.
if err := m.Server.Open(); err != nil {
return err
}
fmt.Fprintf(m.Stderr, "Listening as http://%s\n", m.Server.Host)
return nil
}
// Close shuts down the server.
func (m *Command) Close() error {
return m.Server.Close()
}
// SetupConfig loads the config file if specified and sets state on the Command.
func (m *Command) SetupConfig(args []string) error {
// Load config, if specified.
if m.ConfigPath != "" {
if _, err := toml.DecodeFile(m.ConfigPath, &m.Config); err != nil {
return err
}
}
// Use default data directory if one is not specified.
if m.Config.DataDir == "" {
m.Config.DataDir = DefaultDataDir
}
// Expand home directory.
prefix := "~" + string(filepath.Separator)
if strings.HasPrefix(m.Config.DataDir, prefix) {
HomeDir := os.Getenv("HOME")
if HomeDir == "" {
return errors.New("data directory not specified and no home dir available")
}
m.Config.DataDir = filepath.Join(HomeDir, strings.TrimPrefix(m.Config.DataDir, prefix))
}
return nil
}

24
cmd/pilosa/main_test.go → server/server_test.go
View file

@ -1,4 +1,4 @@
package main_test
package server_test
import (
"bytes"
@ -18,7 +18,7 @@ import (
"github.com/BurntSushi/toml"
"github.com/pilosa/pilosa"
main "github.com/pilosa/pilosa/cmd/pilosa"
"github.com/pilosa/pilosa/server"
)
// Ensure program can process queries and maintain consistency.
@ -304,7 +304,7 @@ path = "/path/to/plugins"
// Main represents a test wrapper for main.Main.
type Main struct {
*main.Main
*server.Command
Stdin bytes.Buffer
Stdout bytes.Buffer
@ -318,16 +318,16 @@ func NewMain() *Main {
panic(err)
}
m := &Main{Main: main.NewMain()}
m := &Main{Command: server.NewCommand()}
m.Config.DataDir = path
m.Config.Host = "localhost:0"
m.Main.Stdin = &m.Stdin
m.Main.Stdout = &m.Stdout
m.Main.Stderr = &m.Stderr
m.Command.Stdin = &m.Stdin
m.Command.Stdout = &m.Stdout
m.Command.Stderr = &m.Stderr
if testing.Verbose() {
m.Main.Stdout = io.MultiWriter(os.Stdout, m.Main.Stdout)
m.Main.Stderr = io.MultiWriter(os.Stderr, m.Main.Stderr)
m.Command.Stdout = io.MultiWriter(os.Stdout, m.Command.Stdout)
m.Command.Stderr = io.MultiWriter(os.Stderr, m.Command.Stderr)
}
return m
@ -345,18 +345,18 @@ func MustRunMain() *Main {
// Close closes the program and removes the underlying data directory.
func (m *Main) Close() error {
defer os.RemoveAll(m.Config.DataDir)
return m.Main.Close()
return m.Command.Close()
}
// Reopen closes the program and reopens it.
func (m *Main) Reopen() error {
if err := m.Main.Close(); err != nil {
if err := m.Command.Close(); err != nil {
return err
}
// Create new main with the same config.
config := m.Config
m.Main = main.NewMain()
m.Command = server.NewCommand()
m.Config = config
// Run new program.