Commit graph

38 commits

Author SHA1 Message Date
Michael Baird
59f26ce585 removed no-op stats tracking, fixed typos 2017-05-30 11:13:39 -05:00
Michael Baird
d4e9e5cdbc cache StatsD recalculate, invalidate, and threshold counts 2017-05-16 15:21:26 -05:00
Michael Baird
4a051900c0 Add sampling rate to StatsD functions, and fix data dog logging 2017-05-16 10:35:49 -05:00
Michael Baird
ee159b455d Merge remote-tracking branch 'origin/master' into metrics
Conflicts:
	.gitignore
	cluster.go
	frame.go
	glide.lock
	index.go
2017-05-01 10:41:04 -05:00
Cody Soyland
3996b56b88 Apply Apache License 2.0 2017-04-28 14:22:03 -05:00
Michael Baird
d2a7869142 comment cleanup 2017-04-25 16:40:39 -05:00
Michael Baird
3972d0cfaa added missing comments for godocs 2017-04-25 13:31:12 -05:00
Michael Baird
5a8c165c53 Merge remote-tracking branch 'origin/master' into metrics
Conflicts:
	cache.go
	cluster.go
	db.go
	executor.go
	fragment.go
	frame.go
	gossip/gossip.go
	handler.go
	index.go
	server.go
	server/server.go
2017-04-24 16:35:25 -05:00
Ben Johnson
d34687121f
Rename bitmap/profile to row/column. 2017-04-21 21:23:24 -06:00
Michael Baird
40cb2adef6 add cache stats 2017-04-19 16:27:39 -05:00
Travis
b0f1fc7523 Makes Messenger a first-class object under Server (with pointers in Handler and Index).
Primary message interface is the MessageBroker which is an attribute of the Messenger.
MessageBroker implementations:
- Gossip (memberlist)
- Broadcast (uses HTTP, received by existing Handler)
- Static (no-ops)

Changes CacheSize from `int` to `uint32` for consistency with protobuf.
Removes unnecessary dependencies in glide:
- `github.com/aws/aws-sdk-go`
- `golang.org/x/net` (although this gets included by memberlist)

TODO:
- [ ] Add tests around the Messenger and MessageBroker objects.
- [ ] Refactor CreateSliceMessage to work with views.
- [ ] Support propogation of meta data on PATCH calls.

fixing some issues from last rebase
2017-04-19 10:33:26 -05:00
Michael Baird
8e2e9254f7 fixed comment 2017-04-19 10:26:47 -05:00
Michael Baird
e864a45d57 cleaned up the comments and code for the cache threshold. Also made the max size the set value of the cache rather than the calculated index. This ensures the user will not receive less cached values than they expect 2017-04-19 10:26:47 -05:00
Michael Baird
9321637d18 trim the ranked cache set on the 1st recalculate call 2017-04-19 10:06:11 -05:00
Matt Jaffee
e6309d45e1 make sure exported functions are godoc'd 2017-04-18 15:10:59 -05:00
Linh Vo
7979b4d69b set profile attr for columnLabel 2017-03-24 12:18:59 -05:00
Linh Vo
ac8018c4f3 change Pair struct Key to ID 2017-03-23 12:38:09 -05:00
Linh Vo
67a691dfe0 use id instead of key in topN results 2017-03-23 12:02:57 -05:00
Travis
2b94be965e Make sure the TopN test is actually using a RankCache supported frame.
Implement cache.Recalculate to bypass the 10-second delay built into cache.Invalidate().
2017-03-10 11:07:18 -06:00
Travis
c013661880 bug fix in SimpleCache.Fetch (really just reverting my change) 2017-03-02 16:00:34 -06:00
Travis
cbac9ec88b move BitmapCache into cache.go. adjust some variable names for clarity 2017-03-02 15:17:21 -06:00
Todd Gruben
9237e7b699 rank cache update after count
heap sort order backwards

WIP TopN accuracy

adjusted first phase topn to collect all slices id's

incorrect handling of large topns

cache performance enhancement

fix for failed test TestMain_FrameRestore

remove unused code and fix some variable names
2017-03-02 12:32:53 -06:00
Travis
c368288638 fix BulkAdd comment 2017-02-27 11:54:59 -06:00
Travis
b4d6181285 no longer need to implement Refresh() since that was removed` 2017-02-24 12:57:41 -06:00
Todd Gruben
ff52ffa047 WIP TopN optimization 2017-02-23 15:08:18 -06:00
Todd Gruben
43ede4d9c8 WIP increase caching and topn adjustments 2017-02-21 09:28:01 -06:00
Todd Gruben
028539f10d top bug in cache refresh 2017-02-17 15:18:38 -05:00
Ben Johnson
c6e2b1294b Add time-based frames and Range() support. 2016-12-08 12:29:49 -07:00
Cody Soyland
5206461d51 Migrate from Umbel to Pilosa organization on Github 2016-11-28 15:21:11 -06:00
Ben Johnson
2322201bf4 Add slice to BitmapSegment operations.
Previously the `slice` was not copied to each new segment when
`Intersect()`, `Union()`, and `Difference()` operations were performed.
This causes issues when those bitmaps were then operated on later such
as performing a `TopN()` on a source bitmap.
2016-10-13 10:54:40 -06:00
Ben Johnson
6ad7d0bfab fix count() cache retrieval 2016-08-12 11:02:52 -06:00
Ben Johnson
722d697228 refactor in-memory bitmap storage
This commit changes the underlying storage for `Bitmap` from a
red-black tree to a roaring bitmap. It also removes bitmaps from
the cache and only stores the bitmap count.
2016-05-24 15:01:49 -06:00
Ben Johnson
569bafdc7d add bulk call support to the query endpoint
This commit changes the `pql.Query` so that it can accept one or more
top-level calls instead of only one.

The query request format change because a query with a single call is
still valid. However, the result format now returns a `results` field
that has one result for each top-level call. The `profiles` field is
still the same, however, it combines all profiles from all bitmap
responses into one return so that there's not duplicate attributes.

Fixes #59
2016-03-08 14:34:37 -07:00
Ben Johnson
5a63a14742 refetch full counts for TopN()
This commits adds refetching to get the actual TopN() count
after the top bitmaps are determined.

Fixes #57
2016-03-03 13:03:20 -07:00
Ben Johnson
b28e961f2e add cache persistence
This commit adds the ability of the Fragment to flush the cache
bitmap IDs to disk periodically. They can then be reloaded when
the fragment is reopened.
2016-02-13 10:02:53 -07:00
Ben Johnson
d3500c64f7 TopN()
This commit adds TopN() functionality to the fragment, executor
and handler.
2016-02-09 14:43:49 -07:00
Ben Johnson
784beb074e roaring storage
This commit attaches the mmap-backed roaring bitmaps as storage
to the fragments. Currently snapshotting is not supported from
Pilosa so the WAL will grow indefinitely, however, it is not a
difficult change to make.
2015-12-17 20:35:33 -07:00
Ben Johnson
a632d08629 refactor
This commit refactors the pilosa codebase. It makes several major
changes:

* Removes bitmap handles
* Removes dispatch/hold/transport
* Removes etcd dependency
* Adds consistent hash ring for slice placement
* Refactors parser/lexer
* Adds strong typing to PQL AST
* Flattens package hierarchy
2015-12-02 15:34:49 -07:00