scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-10-08 03:47:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge branch 'master' of github.com:pilosa/pilosa into input-definition

Browse source
This commit is contained in:
Linh Vo 2017-06-08 14:23:53 -05:00
commit 3013262e01
11 changed files with 85 additions and 347 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

59
CHANGELOG.md Normal file
View file

@ -0,0 +1,59 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/)
and this project adheres to [Semantic Versioning](http://semver.org/).
## [0.4.0] - 2017-06-08
This version contains 53 contributions from 13 contributors (including 4 volunteer contributors). There are 96 files changed, 6373 insertions, and 770 deletions. Test coverage is currently 73%.
*Note that data files created in Pilosa < 0.4.0 are not compatible with Pilosa 0.4.0 as a result of [#520](https://github.com/pilosa/pilosa/pull/520).*
### Added
- Support metric reporting through StatsD protocol ([#468](https://github.com/pilosa/pilosa/pull/468), [#568](https://github.com/pilosa/pilosa/pull/568), [#580](https://github.com/pilosa/pilosa/pull/580))
- Improve test coverage for ctl package ([#586](https://github.com/pilosa/pilosa/pull/586))
- Add support for bit flip (negate) in roaring ([#592](https://github.com/pilosa/pilosa/pull/592))
- Add xor support to roaring ([#571](https://github.com/pilosa/pilosa/pull/571))
- Improve WebUI autocomplete ([#560](https://github.com/pilosa/pilosa/pull/560))
- Add syntax hints tooltip to WebUI ([#537](https://github.com/pilosa/pilosa/pull/537))
- Implement 'config' CLI command. ([#541](https://github.com/pilosa/pilosa/pull/541))
- Move docs into repo ([#563](https://github.com/pilosa/pilosa/pull/563))
- Add inverse TopN() support ([#551](https://github.com/pilosa/pilosa/pull/551))
- Add various Makefile updates ([#540](https://github.com/pilosa/pilosa/pull/540))
- Provide details on Glide checksum mismatch ([#546](https://github.com/pilosa/pilosa/pull/546))
- Add Docker multi-stage build ([#535](https://github.com/pilosa/pilosa/pull/535))
- Support inverse Range() queries ([#533](https://github.com/pilosa/pilosa/pull/533))
- Support colon commands in WebUI ([#529](https://github.com/pilosa/pilosa/pull/529), [#510](https://github.com/pilosa/pilosa/pull/510))
### Changed
- Increase default partition count from 16 to 256 (BREAKING CHANGE) ([#520](https://github.com/pilosa/pilosa/pull/520))
- Validate unknown query params ([#578](https://github.com/pilosa/pilosa/pull/578))
- Validate configuration file ([#573](https://github.com/pilosa/pilosa/pull/573))
- Change default cache type to ranked ([#524](https://github.com/pilosa/pilosa/pull/524))
- Add max-writes-per-requests limit ([#525](https://github.com/pilosa/pilosa/pull/525))
### Fixed
- Add "make test" to PHONY section of Makefile ([#605](https://github.com/pilosa/pilosa/pull/605))
- Fix failing tests when IPv6 is disabled ([#594](https://github.com/pilosa/pilosa/pull/594))
- Add minor docs fix, indent in JSON ([#599](https://github.com/pilosa/pilosa/pull/599))
- Fix BroadcastHandler handle missing index error ([#597](https://github.com/pilosa/pilosa/pull/597))
- Add WebUI fixes ([#589](https://github.com/pilosa/pilosa/pull/589))
- Fix support for 32-bit Linux ([#549](https://github.com/pilosa/pilosa/pull/549), [#565](https://github.com/pilosa/pilosa/pull/565))
- Fix 3 separate bugs in bitmapCountRange ([#559](https://github.com/pilosa/pilosa/pull/559))
- Add client support for MaxInverseSliceByIndex ([#555](https://github.com/pilosa/pilosa/pull/555))
- Fix bug in `handleGetSliceMax` ([#554](https://github.com/pilosa/pilosa/pull/554))
- Default to `standard` view in export command ([#548](https://github.com/pilosa/pilosa/pull/548))
- Fix vet issues with the assembly code in Roaring ([#528](https://github.com/pilosa/pilosa/pull/528))
- Prevent row labels that match the column label ([#503](https://github.com/pilosa/pilosa/pull/503))
- Fix roaring test: TestBitmap_Quick_Array1 ([#507](https://github.com/pilosa/pilosa/pull/507))
- Don't try to create inverse views on Import() when inverseEnabled is false ([#462](https://github.com/pilosa/pilosa/pull/462))
### Performance
- Set n based on array length instead of incrementing repeatedly ([#590](https://github.com/pilosa/pilosa/pull/590))
- Rewrite intersectCountArrayBitmap for perf test ([#577](https://github.com/pilosa/pilosa/pull/577))
- Check for duplicate attributes under read lock on insert ([#562](https://github.com/pilosa/pilosa/pull/562))
[Unreleased]: https://github.com/pilosa/pilosa/compare/v0.4.0...HEAD
[0.4.0]: https://github.com/pilosa/pilosa/compare/v0.3.1...v0.4.0

2
Makefile
View file

@ -1,4 +1,4 @@
.PHONY: glide vendor-update docker pilosa crossbuild install generate statik release
.PHONY: glide vendor-update docker pilosa crossbuild install generate statik release test
GLIDE := $(shell command -v glide 2>/dev/null)
STATIK := $(shell command -v statik 2>/dev/null)

1
cmd/import.go
View file

@ -54,6 +54,7 @@ omitted. If it is present then its format should be YYYY-MM-DDTHH:MM.
flags.StringVarP(&Importer.Index, "index", "i", "", "Pilosa index to import into.")
flags.StringVarP(&Importer.Frame, "frame", "f", "", "Frame to import into.")
flags.IntVarP(&Importer.BufferSize, "buffer-size", "s", 10000000, "Number of bits to buffer/sort before importing.")
flags.BoolVarP(&Importer.Sort, "sort", "", false, "Enables sorting before import.")
return importCmd
}

64
cmd/sort.go
View file

@ -1,64 +0,0 @@
// Copyright 2017 Pilosa Corp.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"context"
"fmt"
"io"
"os"
"github.com/spf13/cobra"
"github.com/pilosa/pilosa/ctl"
)
var Sorter *ctl.SortCommand
func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *cobra.Command {
Sorter = ctl.NewSortCommand(os.Stdin, os.Stdout, os.Stderr)
sortCmd := &cobra.Command{
Use: "sort <path>",
Short: "Sort import data for optimal import performance.",
Long: `
Sorts the import data at PATH into the optimal sort order for importing.
The format of the CSV file is:
ROWID,COLUMNID
The file should contain no headers.
`,
RunE: func(cmd *cobra.Command, args []string) error {
if len(args) == 0 {
return fmt.Errorf("path required")
} else if len(args) > 1 {
return fmt.Errorf("only one path supported")
}
Sorter.Path = args[0]
if err := Sorter.Run(context.Background()); err != nil {
return err
}
return nil
},
}
return sortCmd
}
func init() {
subcommandFns["sort"] = NewSortCommand
}

43
cmd/sort_test.go
View file

@ -1,43 +0,0 @@
// Copyright 2017 Pilosa Corp.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd_test
import (
"strings"
"testing"
)
func TestSortHelp(t *testing.T) {
output, err := ExecNewRootCommand(t, "sort", "--help")
if !strings.Contains(output, "Usage:") ||
!strings.Contains(output, "Flags:") ||
!strings.Contains(output, "pilosa sort") || err != nil {
t.Fatalf("Command 'sort --help' not working, err: '%v', output: '%s'", err, output)
}
}
func TestSortNoPath(t *testing.T) {
output, err := ExecNewRootCommand(t, "sort")
if !strings.Contains(err.Error(), "path required") {
t.Fatalf("Command 'sort' without args should error but: err: '%v', output: '%v'", err, output)
}
}
func TestSortMultiPath(t *testing.T) {
output, err := ExecNewRootCommand(t, "sort", "one", "two")
if !strings.Contains(err.Error(), "only one path") {
t.Fatalf("Command 'sort' without args should error but: err: '%v', output: '%v'", err, output)
}
}

8
ctl/import.go
View file

@ -22,6 +22,7 @@ import (
"io"
"log"
"os"
"sort"
"strconv"
"time"
@ -43,6 +44,9 @@ type ImportCommand struct {
// Size of buffer used to chunk import.
BufferSize int `json:"bufferSize"`
// Enables sorting of data file before import.
Sort bool `json:"sort"`
// Reusable client.
Client *pilosa.Client `json:"-"`
@ -185,6 +189,10 @@ func (cmd *ImportCommand) importBits(ctx context.Context, bits []pilosa.Bit) err
// Parse path into bits.
for slice, bits := range bitsBySlice {
if cmd.Sort {
sort.Sort(pilosa.BitsByPos(bits))
}
logger.Printf("importing slice: %d, n=%d", slice, len(bits))
if err := cmd.Client.Import(ctx, cmd.Index, cmd.Frame, slice, bits); err != nil {
return err

148
ctl/sort.go
View file

@ -1,148 +0,0 @@
// Copyright 2017 Pilosa Corp.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package ctl
import (
"bufio"
"context"
"encoding/csv"
"errors"
"fmt"
"io"
"os"
"sort"
"strconv"
"time"
"github.com/pilosa/pilosa"
)
// SortCommand represents a command for sorting import data.
type SortCommand struct {
// Filename to sort
Path string
// Standard input/output
*pilosa.CmdIO
}
// NewSortCommand returns a new instance of SortCommand.
func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *SortCommand {
return &SortCommand{
CmdIO: pilosa.NewCmdIO(stdin, stdout, stderr),
}
}
// Run executes the sort command.
func (cmd *SortCommand) Run(ctx context.Context) error {
// Open file for reading.
f, err := os.Open(cmd.Path)
if err != nil {
return err
}
defer f.Close()
// Read rows as bits.
r := csv.NewReader(f)
r.FieldsPerRecord = -1
a := make([]pilosa.Bit, 0, 1000000)
for {
rowID, columnID, timestamp, err := readCSVRow(r)
if err == io.EOF {
break
} else if err == errBlank {
continue
} else if err != nil {
return err
}
a = append(a, pilosa.Bit{RowID: rowID, ColumnID: columnID, Timestamp: timestamp})
}
// Sort bits by position.
sort.Sort(pilosa.BitsByPos(a))
// Rewrite to STDOUT.
w := bufio.NewWriter(cmd.Stdout)
buf := make([]byte, 0, 1024)
for _, bit := range a {
// Write CSV to buffer.
buf = buf[:0]
buf = strconv.AppendUint(buf, bit.RowID, 10)
buf = append(buf, ',')
buf = strconv.AppendUint(buf, bit.ColumnID, 10)
if bit.Timestamp != 0 {
buf = append(buf, ',')
buf = append(buf, time.Unix(0, bit.Timestamp).UTC().Format(pilosa.TimeFormat)...)
}
buf = append(buf, '\n')
// Write to output.
if _, err := w.Write(buf); err != nil {
return err
}
}
// Ensure buffer is flushed before exiting.
if err := w.Flush(); err != nil {
return err
}
return nil
}
// readCSVRow reads a row/column pair from a CSV row.
func readCSVRow(r *csv.Reader) (rowID, columnID uint64, timestamp int64, err error) {
// Read CSV row.
record, err := r.Read()
if err != nil {
return 0, 0, 0, err
}
// Ignore blank rows.
if record[0] == "" {
return 0, 0, 0, errBlank
} else if len(record) < 2 {
return 0, 0, 0, fmt.Errorf("bad column count: %d", len(record))
}
// Parse row id.
rowID, err = strconv.ParseUint(record[0], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid row id: %q", record[0])
}
// Parse column id.
columnID, err = strconv.ParseUint(record[1], 10, 64)
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid column id: %q", record[1])
}
// Parse timestamp, if available.
if len(record) > 2 && record[2] != "" {
t, err := time.Parse(pilosa.TimeFormat, record[2])
if err != nil {
return 0, 0, 0, fmt.Errorf("invalid timestamp: %q", record[2])
}
timestamp = t.UnixNano()
}
return rowID, columnID, timestamp, nil
}
// errBlank indicates a blank row in a CSV file.
var errBlank = errors.New("blank row")

83
ctl/sort_test.go
View file

@ -1,83 +0,0 @@
// Copyright 2017 Pilosa Corp.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package ctl
import (
"bytes"
"golang.org/x/net/context"
"io"
"io/ioutil"
"os"
"strings"
"testing"
)
func TestSortCommand_Run(t *testing.T) {
file, _ := ioutil.TempFile("", "file.csv")
content := "3,3\n1,2\n2,4"
file.Write([]byte(content))
file.Close()
rder := []byte{}
stdin := bytes.NewReader(rder)
r, w, _ := os.Pipe()
cm := NewSortCommand(stdin, w, w)
cm.Path = file.Name()
err := cm.Run(context.Background())
w.Close()
var buf bytes.Buffer
io.Copy(&buf, r)
if err != nil {
t.Fatal(err)
} else if !strings.Contains(buf.String(), "1,2\n2,4\n3,3") {
t.Fatalf("File is not sorted, actual result: %s", buf.String())
}
}
func TestSortCommand_InvalidFile(t *testing.T) {
buf := bytes.Buffer{}
stdin, stdout, stderr := GetIO(buf)
file, _ := ioutil.TempFile("", "file.csv")
file.Write([]byte("3,3\na,8\n2,4"))
file.Close()
cm := NewSortCommand(stdin, stdout, stderr)
cm.Path = file.Name()
err := cm.Run(context.Background())
if !strings.Contains(err.Error(), "invalid row id") {
t.Fatalf("expect err: invalid row id, actual: %s", err)
}
file, _ = ioutil.TempFile("", "file.csv")
file.Write([]byte("3,3\n1,a\n2,4"))
file.Close()
cm.Path = file.Name()
err = cm.Run(context.Background())
if !strings.Contains(err.Error(), "invalid column id") {
t.Fatalf("expect err: invalid column id, actual: %s", err)
}
file, _ = ioutil.TempFile("", "file.csv")
file.Write([]byte("3,3,1234\n1,2,34345\n2,4"))
file.Close()
cm.Path = file.Name()
err = cm.Run(context.Background())
if !strings.Contains(err.Error(), "invalid timestamp") {
t.Fatalf("expect err: invalid timestamp, actual: %s", err)
}
}

5
docs/administration.md
View file

@ -36,9 +36,10 @@ While Pilosa does have some high system requirements it is not a best practice t
The import API expects a csv of RowID,ColumnID's.
When importing large datasets remember it is much faster to pre sort the data by RowID and then by ColumnID in ascending order. You can use `pilosa sort CSV_FILE` to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results.
When importing large datasets remember it is much faster to pre sort the data by RowID and then by ColumnID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results.
```
pilosa import -d project -f stargazer project-stargazer.csv
pilosa import --sort -d project -f stargazer project-stargazer.csv
```
#### Exporting

4
docs/installation.md
View file

@ -61,7 +61,6 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende
inspect Get stats on a pilosa data file.
restore Restore data to pilosa from a backup file.
server Run Pilosa.
sort Sort import data for optimal import performance.
Flags:
-c, --config string Configuration file to read from.
@ -121,7 +120,6 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende
inspect Get stats on a pilosa data file.
restore Restore data to pilosa from a backup file.
server Run Pilosa.
sort Sort import data for optimal import performance.
Flags:
-c, --config string Configuration file to read from.
@ -211,7 +209,6 @@ There are three ways to install Pilosa on Linux: download the binary (recommende
inspect Get stats on a pilosa data file.
restore Restore data to pilosa from a backup file.
server Run Pilosa.
sort Sort import data for optimal import performance.
Flags:
-c, --config string Configuration file to read from.
@ -271,7 +268,6 @@ There are three ways to install Pilosa on Linux: download the binary (recommende
inspect Get stats on a pilosa data file.
restore Restore data to pilosa from a backup file.
server Run Pilosa.
sort Sort import data for optimal import performance.
Flags:
-c, --config string Configuration file to read from.

15
docs/query-language.md
View file

@ -326,12 +326,13 @@ Return `2`
```
TopN([BITMAP_CALL], <frame=STRING>, [n=UINT],
[<field=ATTR_NAME>, <filters=[]ATTR_VALUE>])
[inverse=true], [<field=ATTR_NAME>, <filters=[]ATTR_VALUE>])
```
**Description:**
Return the id and count of the top `n` bitmaps (by count of bits) in the frame.
`inverse=true` specifies that the call should operate on the [inverse view ](../data-model/#inverse).
The `field` and `filters` arguments work together to only return Bitmaps which
have the attribute specified by `field` with one of the values specified in
`filters`.
@ -346,10 +347,20 @@ TopN(frame="stargazer")
Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}, {"key": 3, "count": 1}]`
* key is a user
* key is a user ID
* count is amount of repositories
* Results are the number of repositories that each user starred in descending order for all users in the stargazer frame, for example user 1 starred two repositories, user 2 starred two repositories, user 3 starred one repository.
```
TopN(frame="stargazer", inverse=true)
```
Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}, {"key": 3, "count": 1}]`
* key is a repository ID
* count is amount of users
* Results are the number of users that starred each repository in descending order for all respositories in the stargazer frame.
```
TopN(frame="stargazer", n=2)
```