From 87475220bc19f6b0355d7987caa0e96e7591a95b Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Tue, 30 May 2017 11:35:01 -0500 Subject: [PATCH 1/7] Document inverse=true for TopN --- docs/query-language.md | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/docs/query-language.md b/docs/query-language.md index f35b88ad3..24c9b2567 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -326,12 +326,13 @@ Return `2` ``` TopN([BITMAP_CALL], , [n=UINT], - [, ]) + [inverse=true], [, ]) ``` **Description:** Return the id and count of the top `n` bitmaps (by count of bits) in the frame. +`inverse=true` specifies that the call should operate on the [inverse view ](../data-model/#inverse). The `field` and `filters` arguments work together to only return Bitmaps which have the attribute specified by `field` with one of the values specified in `filters`. @@ -346,10 +347,20 @@ TopN(frame="stargazer") Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}, {"key": 3, "count": 1}]` -* key is a user +* key is a user ID * count is amount of repositories * Results are the number of repositories that each user starred in descending order for all users in the stargazer frame, for example user 1 starred two repositories, user 2 starred two repositories, user 3 starred one repository. +``` +TopN(frame="stargazer", inverse=true) +``` + +Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}, {"key": 3, "count": 1}]` + +* key is a repository ID +* count is amount of users +* Results are the number of user that starred each repository in descending order for all respositories in the stargazer frame. + ``` TopN(frame="stargazer", n=2) ``` From 7aeca13085ab917a15e5e164578c2e033d80a1bf Mon Sep 17 00:00:00 2001 From: Alan Bernstein Date: Thu, 1 Jun 2017 11:10:33 -0500 Subject: [PATCH 2/7] Fix typo --- docs/query-language.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/query-language.md b/docs/query-language.md index 24c9b2567..1d0e01ac4 100644 --- a/docs/query-language.md +++ b/docs/query-language.md @@ -359,7 +359,7 @@ Returns `[{"key": 1, "count": 2}, {"key": 2, "count": 2}, {"key": 3, "count": 1} * key is a repository ID * count is amount of users -* Results are the number of user that starred each repository in descending order for all respositories in the stargazer frame. +* Results are the number of users that starred each repository in descending order for all respositories in the stargazer frame. ``` TopN(frame="stargazer", n=2) From fd2592e26142f540a1928583aa8cd10c541058c6 Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Mon, 5 Jun 2017 15:45:36 -0500 Subject: [PATCH 3/7] Add "test" to PHONY list in Makefile --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 695946262..563059137 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: glide vendor-update docker pilosa crossbuild install generate statik release +.PHONY: glide vendor-update docker pilosa crossbuild install generate statik release test GLIDE := $(shell command -v glide 2>/dev/null) STATIK := $(shell command -v statik 2>/dev/null) From a171f571593ac6f8087b4c6df3c18c170c67b56a Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Wed, 7 Jun 2017 23:53:54 -0500 Subject: [PATCH 4/7] Add CHANGELOG.md with 0.4.0 release notes. --- CHANGELOG.md | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000..a282643bf --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,55 @@ +# Changelog + +All notable changes to this project will be documented in this file. + +The format is based on [Keep a Changelog](http://keepachangelog.com/) +and this project adheres to [Semantic Versioning](http://semver.org/). + +## [0.4.0] - 2017-06-08 + +### Added +- Support metric reporting through StatsD protocol ([#468](https://github.com/pilosa/pilosa/pull/468), [#568](https://github.com/pilosa/pilosa/pull/568), [#580](https://github.com/pilosa/pilosa/pull/580)) +- Improve test coverage for ctl package ([#586](https://github.com/pilosa/pilosa/pull/586)) +- Add support for bit flip (negate) in roaring ([#592](https://github.com/pilosa/pilosa/pull/592)) +- Add xor support to roaring ([#571](https://github.com/pilosa/pilosa/pull/571)) +- WebUI autocomplete improvements ([#560](https://github.com/pilosa/pilosa/pull/560)) +- Add syntax hints tooltip to WebUI ([#537](https://github.com/pilosa/pilosa/pull/537)) +- Implement 'config' CLI command. ([#541](https://github.com/pilosa/pilosa/pull/541)) +- Move docs into repo ([#563](https://github.com/pilosa/pilosa/pull/563)) +- Add inverse TopN() support ([#551](https://github.com/pilosa/pilosa/pull/551)) +- Various Makefile updates ([#540](https://github.com/pilosa/pilosa/pull/540)) +- Provide details on Glide checksum mismatch ([#546](https://github.com/pilosa/pilosa/pull/546)) +- Add Docker multi-stage build ([#535](https://github.com/pilosa/pilosa/pull/535)) +- Support inverse Range() queries ([#533](https://github.com/pilosa/pilosa/pull/533)) +- Support colon commands in WebUI ([#529](https://github.com/pilosa/pilosa/pull/529), [#510](https://github.com/pilosa/pilosa/pull/510)) + +### Changed +- Increase default partition count from 16 to 256 (BREAKING CHANGE) ([#520](https://github.com/pilosa/pilosa/pull/520)) +- Validate unknown query params ([#578](https://github.com/pilosa/pilosa/pull/578)) +- Validate configuration file ([#573](https://github.com/pilosa/pilosa/pull/573)) +- Change default cache type to ranked ([#524](https://github.com/pilosa/pilosa/pull/524)) +- Add max-writes-per-requests limit ([#525](https://github.com/pilosa/pilosa/pull/525)) + +### Fixed +- Add "make test" to PHONY section of Makefile ([#605](https://github.com/pilosa/pilosa/pull/605)) +- Fix failing tests when IPv6 is disabled ([#594](https://github.com/pilosa/pilosa/pull/594)) +- Minor docs fix, indent in JSON ([#599](https://github.com/pilosa/pilosa/pull/599)) +- BroadcastHandler handle missing index error ([#597](https://github.com/pilosa/pilosa/pull/597)) +- WebUI fixes ([#589](https://github.com/pilosa/pilosa/pull/589)) +- Support for 32-bit Linux ([#549](https://github.com/pilosa/pilosa/pull/549), [#565](https://github.com/pilosa/pilosa/pull/565)) +- Fix 3 separate bugs in bitmapCountRange ([#559](https://github.com/pilosa/pilosa/pull/559)) +- Add client support for MaxInverseSliceByIndex ([#555](https://github.com/pilosa/pilosa/pull/555)) +- Fix bug in `handleGetSliceMax` ([#554](https://github.com/pilosa/pilosa/pull/554)) +- Default to `standard` view in export command ([#548](https://github.com/pilosa/pilosa/pull/548)) +- Roaring: fix vet issues with the assembly code ([#528](https://github.com/pilosa/pilosa/pull/528)) +- Prevent row labels that match the column label ([#503](https://github.com/pilosa/pilosa/pull/503)) +- Fix roaring test: TestBitmap_Quick_Array1 ([#507](https://github.com/pilosa/pilosa/pull/507)) +- Don't try to create inverse views on Import() when inverseEnabled is false ([#462](https://github.com/pilosa/pilosa/pull/462)) + +### Performance +- Set n based on array length instead of incrementing repeatedly ([#590](https://github.com/pilosa/pilosa/pull/590)) +- Rewrite intersectCountArrayBitmap for perf test ([#577](https://github.com/pilosa/pilosa/pull/577)) +- Check for duplicate attributes under read lock on insert ([#562](https://github.com/pilosa/pilosa/pull/562)) + +[Unreleased]: https://github.com/pilosa/pilosa/compare/v0.4.0...HEAD +[0.4.0]: https://github.com/pilosa/pilosa/compare/v0.3.1...v0.4.0 From 14d8f681ab9f8286e8d0490901846c3687b317af Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Thu, 8 Jun 2017 09:19:30 -0500 Subject: [PATCH 5/7] Add notes about release. --- CHANGELOG.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index a282643bf..36786fa3f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## [0.4.0] - 2017-06-08 +This version contains 53 contributions from 13 contributors (including 4 volunteer contributors). There are 96 files changed, 6373 insertions, and 770 deletions. Test coverage is currently 73%. + +*Note that data files created in Pilosa < 0.4.0 are not compatible with Pilosa 0.4.0 as a result of [#520](https://github.com/pilosa/pilosa/pull/520).* + ### Added - Support metric reporting through StatsD protocol ([#468](https://github.com/pilosa/pilosa/pull/468), [#568](https://github.com/pilosa/pilosa/pull/568), [#580](https://github.com/pilosa/pilosa/pull/580)) - Improve test coverage for ctl package ([#586](https://github.com/pilosa/pilosa/pull/586)) From 5e5abfed7ced38471a07cf3be136a319c58a35db Mon Sep 17 00:00:00 2001 From: Cody Soyland Date: Thu, 8 Jun 2017 09:34:15 -0500 Subject: [PATCH 6/7] Use consistent verb format in release notes --- CHANGELOG.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 36786fa3f..be9c99bd1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,12 +16,12 @@ This version contains 53 contributions from 13 contributors (including 4 volunte - Improve test coverage for ctl package ([#586](https://github.com/pilosa/pilosa/pull/586)) - Add support for bit flip (negate) in roaring ([#592](https://github.com/pilosa/pilosa/pull/592)) - Add xor support to roaring ([#571](https://github.com/pilosa/pilosa/pull/571)) -- WebUI autocomplete improvements ([#560](https://github.com/pilosa/pilosa/pull/560)) +- Improve WebUI autocomplete ([#560](https://github.com/pilosa/pilosa/pull/560)) - Add syntax hints tooltip to WebUI ([#537](https://github.com/pilosa/pilosa/pull/537)) - Implement 'config' CLI command. ([#541](https://github.com/pilosa/pilosa/pull/541)) - Move docs into repo ([#563](https://github.com/pilosa/pilosa/pull/563)) - Add inverse TopN() support ([#551](https://github.com/pilosa/pilosa/pull/551)) -- Various Makefile updates ([#540](https://github.com/pilosa/pilosa/pull/540)) +- Add various Makefile updates ([#540](https://github.com/pilosa/pilosa/pull/540)) - Provide details on Glide checksum mismatch ([#546](https://github.com/pilosa/pilosa/pull/546)) - Add Docker multi-stage build ([#535](https://github.com/pilosa/pilosa/pull/535)) - Support inverse Range() queries ([#533](https://github.com/pilosa/pilosa/pull/533)) @@ -37,15 +37,15 @@ This version contains 53 contributions from 13 contributors (including 4 volunte ### Fixed - Add "make test" to PHONY section of Makefile ([#605](https://github.com/pilosa/pilosa/pull/605)) - Fix failing tests when IPv6 is disabled ([#594](https://github.com/pilosa/pilosa/pull/594)) -- Minor docs fix, indent in JSON ([#599](https://github.com/pilosa/pilosa/pull/599)) -- BroadcastHandler handle missing index error ([#597](https://github.com/pilosa/pilosa/pull/597)) -- WebUI fixes ([#589](https://github.com/pilosa/pilosa/pull/589)) -- Support for 32-bit Linux ([#549](https://github.com/pilosa/pilosa/pull/549), [#565](https://github.com/pilosa/pilosa/pull/565)) +- Add minor docs fix, indent in JSON ([#599](https://github.com/pilosa/pilosa/pull/599)) +- Fix BroadcastHandler handle missing index error ([#597](https://github.com/pilosa/pilosa/pull/597)) +- Add WebUI fixes ([#589](https://github.com/pilosa/pilosa/pull/589)) +- Fix support for 32-bit Linux ([#549](https://github.com/pilosa/pilosa/pull/549), [#565](https://github.com/pilosa/pilosa/pull/565)) - Fix 3 separate bugs in bitmapCountRange ([#559](https://github.com/pilosa/pilosa/pull/559)) - Add client support for MaxInverseSliceByIndex ([#555](https://github.com/pilosa/pilosa/pull/555)) - Fix bug in `handleGetSliceMax` ([#554](https://github.com/pilosa/pilosa/pull/554)) - Default to `standard` view in export command ([#548](https://github.com/pilosa/pilosa/pull/548)) -- Roaring: fix vet issues with the assembly code ([#528](https://github.com/pilosa/pilosa/pull/528)) +- Fix vet issues with the assembly code in Roaring ([#528](https://github.com/pilosa/pilosa/pull/528)) - Prevent row labels that match the column label ([#503](https://github.com/pilosa/pilosa/pull/503)) - Fix roaring test: TestBitmap_Quick_Array1 ([#507](https://github.com/pilosa/pilosa/pull/507)) - Don't try to create inverse views on Import() when inverseEnabled is false ([#462](https://github.com/pilosa/pilosa/pull/462)) From 1c34ea7f8f2808364a307fded4a0a00eb40ab5f7 Mon Sep 17 00:00:00 2001 From: Ben Johnson Date: Mon, 5 Jun 2017 20:24:28 -0600 Subject: [PATCH 7/7] Add sorting flag to import command. --- cmd/import.go | 1 + cmd/sort.go | 64 ------------------ cmd/sort_test.go | 43 ------------ ctl/import.go | 8 +++ ctl/sort.go | 148 ----------------------------------------- ctl/sort_test.go | 83 ----------------------- docs/administration.md | 5 +- docs/installation.md | 4 -- 8 files changed, 12 insertions(+), 344 deletions(-) delete mode 100644 cmd/sort.go delete mode 100644 cmd/sort_test.go delete mode 100644 ctl/sort.go delete mode 100644 ctl/sort_test.go diff --git a/cmd/import.go b/cmd/import.go index bd4be6a9d..944b8b9e6 100644 --- a/cmd/import.go +++ b/cmd/import.go @@ -54,6 +54,7 @@ omitted. If it is present then its format should be YYYY-MM-DDTHH:MM. flags.StringVarP(&Importer.Index, "index", "i", "", "Pilosa index to import into.") flags.StringVarP(&Importer.Frame, "frame", "f", "", "Frame to import into.") flags.IntVarP(&Importer.BufferSize, "buffer-size", "s", 10000000, "Number of bits to buffer/sort before importing.") + flags.BoolVarP(&Importer.Sort, "sort", "", false, "Enables sorting before import.") return importCmd } diff --git a/cmd/sort.go b/cmd/sort.go deleted file mode 100644 index e74f6cc76..000000000 --- a/cmd/sort.go +++ /dev/null @@ -1,64 +0,0 @@ -// Copyright 2017 Pilosa Corp. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package cmd - -import ( - "context" - "fmt" - "io" - "os" - - "github.com/spf13/cobra" - - "github.com/pilosa/pilosa/ctl" -) - -var Sorter *ctl.SortCommand - -func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *cobra.Command { - - Sorter = ctl.NewSortCommand(os.Stdin, os.Stdout, os.Stderr) - - sortCmd := &cobra.Command{ - Use: "sort ", - Short: "Sort import data for optimal import performance.", - Long: ` -Sorts the import data at PATH into the optimal sort order for importing. - -The format of the CSV file is: - - ROWID,COLUMNID - -The file should contain no headers. -`, - RunE: func(cmd *cobra.Command, args []string) error { - if len(args) == 0 { - return fmt.Errorf("path required") - } else if len(args) > 1 { - return fmt.Errorf("only one path supported") - } - Sorter.Path = args[0] - if err := Sorter.Run(context.Background()); err != nil { - return err - } - return nil - }, - } - return sortCmd -} - -func init() { - subcommandFns["sort"] = NewSortCommand -} diff --git a/cmd/sort_test.go b/cmd/sort_test.go deleted file mode 100644 index f95332886..000000000 --- a/cmd/sort_test.go +++ /dev/null @@ -1,43 +0,0 @@ -// Copyright 2017 Pilosa Corp. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package cmd_test - -import ( - "strings" - "testing" -) - -func TestSortHelp(t *testing.T) { - output, err := ExecNewRootCommand(t, "sort", "--help") - if !strings.Contains(output, "Usage:") || - !strings.Contains(output, "Flags:") || - !strings.Contains(output, "pilosa sort") || err != nil { - t.Fatalf("Command 'sort --help' not working, err: '%v', output: '%s'", err, output) - } -} - -func TestSortNoPath(t *testing.T) { - output, err := ExecNewRootCommand(t, "sort") - if !strings.Contains(err.Error(), "path required") { - t.Fatalf("Command 'sort' without args should error but: err: '%v', output: '%v'", err, output) - } -} - -func TestSortMultiPath(t *testing.T) { - output, err := ExecNewRootCommand(t, "sort", "one", "two") - if !strings.Contains(err.Error(), "only one path") { - t.Fatalf("Command 'sort' without args should error but: err: '%v', output: '%v'", err, output) - } -} diff --git a/ctl/import.go b/ctl/import.go index 3b4a74d9d..a12cb0d06 100644 --- a/ctl/import.go +++ b/ctl/import.go @@ -22,6 +22,7 @@ import ( "io" "log" "os" + "sort" "strconv" "time" @@ -43,6 +44,9 @@ type ImportCommand struct { // Size of buffer used to chunk import. BufferSize int `json:"bufferSize"` + // Enables sorting of data file before import. + Sort bool `json:"sort"` + // Reusable client. Client *pilosa.Client `json:"-"` @@ -185,6 +189,10 @@ func (cmd *ImportCommand) importBits(ctx context.Context, bits []pilosa.Bit) err // Parse path into bits. for slice, bits := range bitsBySlice { + if cmd.Sort { + sort.Sort(pilosa.BitsByPos(bits)) + } + logger.Printf("importing slice: %d, n=%d", slice, len(bits)) if err := cmd.Client.Import(ctx, cmd.Index, cmd.Frame, slice, bits); err != nil { return err diff --git a/ctl/sort.go b/ctl/sort.go deleted file mode 100644 index 2d28e02c4..000000000 --- a/ctl/sort.go +++ /dev/null @@ -1,148 +0,0 @@ -// Copyright 2017 Pilosa Corp. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package ctl - -import ( - "bufio" - "context" - "encoding/csv" - "errors" - "fmt" - "io" - "os" - "sort" - "strconv" - "time" - - "github.com/pilosa/pilosa" -) - -// SortCommand represents a command for sorting import data. -type SortCommand struct { - // Filename to sort - Path string - - // Standard input/output - *pilosa.CmdIO -} - -// NewSortCommand returns a new instance of SortCommand. -func NewSortCommand(stdin io.Reader, stdout, stderr io.Writer) *SortCommand { - return &SortCommand{ - CmdIO: pilosa.NewCmdIO(stdin, stdout, stderr), - } -} - -// Run executes the sort command. -func (cmd *SortCommand) Run(ctx context.Context) error { - // Open file for reading. - f, err := os.Open(cmd.Path) - if err != nil { - return err - } - defer f.Close() - - // Read rows as bits. - r := csv.NewReader(f) - r.FieldsPerRecord = -1 - a := make([]pilosa.Bit, 0, 1000000) - for { - rowID, columnID, timestamp, err := readCSVRow(r) - if err == io.EOF { - break - } else if err == errBlank { - continue - } else if err != nil { - return err - } - a = append(a, pilosa.Bit{RowID: rowID, ColumnID: columnID, Timestamp: timestamp}) - } - - // Sort bits by position. - sort.Sort(pilosa.BitsByPos(a)) - - // Rewrite to STDOUT. - w := bufio.NewWriter(cmd.Stdout) - buf := make([]byte, 0, 1024) - for _, bit := range a { - // Write CSV to buffer. - buf = buf[:0] - buf = strconv.AppendUint(buf, bit.RowID, 10) - - buf = append(buf, ',') - buf = strconv.AppendUint(buf, bit.ColumnID, 10) - - if bit.Timestamp != 0 { - buf = append(buf, ',') - buf = append(buf, time.Unix(0, bit.Timestamp).UTC().Format(pilosa.TimeFormat)...) - } - - buf = append(buf, '\n') - - // Write to output. - if _, err := w.Write(buf); err != nil { - return err - } - } - - // Ensure buffer is flushed before exiting. - if err := w.Flush(); err != nil { - return err - } - - return nil -} - -// readCSVRow reads a row/column pair from a CSV row. -func readCSVRow(r *csv.Reader) (rowID, columnID uint64, timestamp int64, err error) { - // Read CSV row. - record, err := r.Read() - if err != nil { - return 0, 0, 0, err - } - - // Ignore blank rows. - if record[0] == "" { - return 0, 0, 0, errBlank - } else if len(record) < 2 { - return 0, 0, 0, fmt.Errorf("bad column count: %d", len(record)) - } - - // Parse row id. - rowID, err = strconv.ParseUint(record[0], 10, 64) - if err != nil { - return 0, 0, 0, fmt.Errorf("invalid row id: %q", record[0]) - } - - // Parse column id. - columnID, err = strconv.ParseUint(record[1], 10, 64) - if err != nil { - return 0, 0, 0, fmt.Errorf("invalid column id: %q", record[1]) - } - - // Parse timestamp, if available. - if len(record) > 2 && record[2] != "" { - t, err := time.Parse(pilosa.TimeFormat, record[2]) - if err != nil { - return 0, 0, 0, fmt.Errorf("invalid timestamp: %q", record[2]) - } - timestamp = t.UnixNano() - } - - return rowID, columnID, timestamp, nil -} - -// errBlank indicates a blank row in a CSV file. -var errBlank = errors.New("blank row") diff --git a/ctl/sort_test.go b/ctl/sort_test.go deleted file mode 100644 index 1f12a5220..000000000 --- a/ctl/sort_test.go +++ /dev/null @@ -1,83 +0,0 @@ -// Copyright 2017 Pilosa Corp. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package ctl - -import ( - "bytes" - "golang.org/x/net/context" - "io" - "io/ioutil" - "os" - "strings" - "testing" -) - -func TestSortCommand_Run(t *testing.T) { - file, _ := ioutil.TempFile("", "file.csv") - content := "3,3\n1,2\n2,4" - file.Write([]byte(content)) - file.Close() - - rder := []byte{} - stdin := bytes.NewReader(rder) - r, w, _ := os.Pipe() - - cm := NewSortCommand(stdin, w, w) - cm.Path = file.Name() - err := cm.Run(context.Background()) - w.Close() - var buf bytes.Buffer - io.Copy(&buf, r) - if err != nil { - t.Fatal(err) - } else if !strings.Contains(buf.String(), "1,2\n2,4\n3,3") { - t.Fatalf("File is not sorted, actual result: %s", buf.String()) - } -} - -func TestSortCommand_InvalidFile(t *testing.T) { - buf := bytes.Buffer{} - stdin, stdout, stderr := GetIO(buf) - file, _ := ioutil.TempFile("", "file.csv") - file.Write([]byte("3,3\na,8\n2,4")) - file.Close() - - cm := NewSortCommand(stdin, stdout, stderr) - cm.Path = file.Name() - err := cm.Run(context.Background()) - if !strings.Contains(err.Error(), "invalid row id") { - t.Fatalf("expect err: invalid row id, actual: %s", err) - - } - - file, _ = ioutil.TempFile("", "file.csv") - file.Write([]byte("3,3\n1,a\n2,4")) - file.Close() - cm.Path = file.Name() - err = cm.Run(context.Background()) - if !strings.Contains(err.Error(), "invalid column id") { - t.Fatalf("expect err: invalid column id, actual: %s", err) - - } - - file, _ = ioutil.TempFile("", "file.csv") - file.Write([]byte("3,3,1234\n1,2,34345\n2,4")) - file.Close() - cm.Path = file.Name() - err = cm.Run(context.Background()) - if !strings.Contains(err.Error(), "invalid timestamp") { - t.Fatalf("expect err: invalid timestamp, actual: %s", err) - } -} diff --git a/docs/administration.md b/docs/administration.md index fca8db4dc..322b68e12 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -36,9 +36,10 @@ While Pilosa does have some high system requirements it is not a best practice t The import API expects a csv of RowID,ColumnID's. -When importing large datasets remember it is much faster to pre sort the data by RowID and then by ColumnID in ascending order. You can use `pilosa sort CSV_FILE` to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results. +When importing large datasets remember it is much faster to pre sort the data by RowID and then by ColumnID in ascending order. You can use the `--sort` flag to do that. Also, avoid querying Pilosa until the import is complete, otherwise you will experience inconsistent results. + ``` -pilosa import -d project -f stargazer project-stargazer.csv +pilosa import --sort -d project -f stargazer project-stargazer.csv ``` #### Exporting diff --git a/docs/installation.md b/docs/installation.md index c97ed4303..af04cc240 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -61,7 +61,6 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende inspect Get stats on a pilosa data file. restore Restore data to pilosa from a backup file. server Run Pilosa. - sort Sort import data for optimal import performance. Flags: -c, --config string Configuration file to read from. @@ -121,7 +120,6 @@ There are three ways to install Pilosa on MacOS: download the binary (recommende inspect Get stats on a pilosa data file. restore Restore data to pilosa from a backup file. server Run Pilosa. - sort Sort import data for optimal import performance. Flags: -c, --config string Configuration file to read from. @@ -211,7 +209,6 @@ There are three ways to install Pilosa on Linux: download the binary (recommende inspect Get stats on a pilosa data file. restore Restore data to pilosa from a backup file. server Run Pilosa. - sort Sort import data for optimal import performance. Flags: -c, --config string Configuration file to read from. @@ -271,7 +268,6 @@ There are three ways to install Pilosa on Linux: download the binary (recommende inspect Get stats on a pilosa data file. restore Restore data to pilosa from a backup file. server Run Pilosa. - sort Sort import data for optimal import performance. Flags: -c, --config string Configuration file to read from.