diff --git a/README.md b/README.md index 3617c8678..129e2939a 100644 --- a/README.md +++ b/README.md @@ -19,12 +19,14 @@ Now you can install the `pilosa` binary: $ go install github.com/pilosa/pilosa/cmd/... ``` -Now run `pilosa` with the default configuration: +Now run a single pilosa node with the default configuration: ```sh pilosa ``` +If you would like to quickly create a multi-node pilosa cluster, see the `pilosactl create` documentation. + ## Configuration You can specify a configuration by setting the `-config` flag when running `pilosa`. @@ -221,49 +223,75 @@ $ go install --ldflags="-X main.Version=1.0.0" [Glide]: http://glide.sh/ -## Benchmarks +## Pilosactl -The usual interface for running benchmarks is: +### Create + +`pilosactl create` is used to create pilosa clusters. It has a number of options for controlling how the cluster is configured, what hosts it is on, and even the ability to build the pilosa binary locally and copy it to each cluster node automatically. To start pilosa on remote hosts, you only need `ssh` access to those hosts. See `pilosactl -h` for a full list of options. + +Examples: + +Create a 5 node cluster locally (using 5 different ports), with a replication factor of 2. ``` -pilosactl bspawn benchmark-file.json +pilosactl create + -serverN 5 + -replicaN 2 ``` -There are several example json config files in `cmd/pilosactl` -The `bspawn` command calls other `pilosactl` subcommands such as `create` and `bagent` to perform the benchmarks. These commands can also be used directly if one wishes e.g. to just create a cluster, or locally run a benchmarks against an existing cluster. Pass the `-help` flag to either to get more information about its usage. +Create a cluster on 3 remote hosts - all logs will come to local stderr, pilosa binary must be available on remote hosts. The ssh user on the remote hosts needs to be the same as your local user. Otherwise use the `ssh-user` option. +``` +pilosactl create + -hosts="node1.example.com:15000,node2.example.com:15000,node3.example.com:15000" +``` -### Configuration Format +Create a cluster on 3 remote hosts running OSX, but build the binary locally and copy it up. Stream the stderr of each node to a separate local log file. +``` +pilosactl create + -hosts="mac1.example.com:15000,mac2.example.com:15000,mac3.example.com:15000" + -copy-binary + -goos=darwin + -goarch=amd64 + -log-file-prefix=clusterlogs +``` -bspawn uses a json config format that has 5 top level items - an example is below. +### Bagent +`pilosactl bagent` is what you want if you just want to run a simple benchmark against an existing cluster. Running it with no arguments will print some help, including the set of subcommands that it may be passed. Calling a subcommand with `-h'` will print the options for that subcommand. The `agent-num` flag can be passed an integer which can change the behavior the benchmarks that are run. This is useful when multiple invocations of the same benchmark are made by the `bspawn` command - they can each (for example) set different bits even though they all have the same arguments. + +E.G. +``` +pilosactl bagent import -h +``` + +Multiple subcommands and their arguments may be concatenated at the command line and they will be run serially. This is useful (i.e.) for importing a bunch of data, and then executing queries against it. + +This will generate and import a bunch of data, and then execute random queries against it. + +``` +pilosactl bagent import -max-bits-per-map=10000 random-query -iterations 100 +``` + +### Bspawn +`pilosactl bspawn` allows you to automate the creation of clusters and the running of complex benchmarks which span multiple benchmark agents against them. It has a number of options which are described by `pilosactl bspawn` with no arguments, and also takes a config file which describes the Benchmark itself - this file is described below. + +#### Configuration Format + +The configuration file is a json object with the top level key `benchmarks`. This contains a list of objects each of which represents a `bagent` command (the `args` key) that will be run some number of times concurrently (the `num` key), and a `name` which should describe the overall effect that command. An example is below. ```json { - "CreatorArgs": ["-type", "local", "-serverN", "1", "-replicaN", "1"], - "PilosaHosts": ["localhost:19327"], - "AgentHosts": ["agent.example.com"], - "Benchmarks": [ + "benchmarks": [ { - "Num": 1, - "Args": ["import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width"] + "num": 3, + "name": "set-diags", + "args": ["diagonal-set-bits", "-iterations", "30000", "-client-type", "round_robin"] }, { - "Num": 1, - "Args": ["import", "-max-bitmap-id", "100000", "-max-profile-id", "10000", "-max-bits-per-map", "100", "-seed", "0", "-agent-controls", "width", "-random-bitmap-order", "-db", "randoload"] + "num": 2, + "name": "rand-plus-zipf", + "args": ["random-set-bits", "-iterations", "20000", "zipf-set-bits", "-iterations", "100"] } ] } - ``` -#### CreatorArgs -Specifies the pilosa cluster that should be created to run benchmarks against. For more information about the configuration for this option, see the `pilosactl create -help` - -#### PilosaHosts -If PilosaHosts is set, CreatorArgs will be ignored, and an existing pilosa cluster specified by the list of hosts will be used. - -#### AgentHosts -If AgentHosts is not empty, the agents specified here are used; if it is empty, agents will be run locally. - -#### Benchmarks -Benchmarks is where the actual benchmarks to run are specified - each contains a `Num` which is the number of agents that should run that benchmark, and Args which specifies the benchmark. The benchmarks in the `Benchmarks` list will be run concurrently. For more information about Args, see the `pilosactl bagent -help`. - -For documentation on a specific `bagent` subcommand do `pilosactl bagent -help` +All of the benchmarks, and agents are run concurrently. Each agent will be passed an `agent-num` which can modify the behavior in a way that is benchmark specific. See the documentation for each benchmark to see how `agent-num` changes its behavior.