mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-10-08 03:47:51 +00:00
No description
c658e771b0
So there's a lot going on here.
Percentile just did not work, even a little, with decimals.
In theory we try to make the int val part of ValCount work, in
ValCountize, but you can't actually use that for everything because
it unconditionally adds bsig.Base even when it shouldn't. But it
doesn't matter that we were returning those values from, say,
(Field).MinForShard, because ValCount.Smaller was not preserving them
when identifying the smaller of two Decimal ValCounts anyway.
And even if it did, the logic in Percentile wouldn't have worked
with passing the raw unscaled integer in as a value to compare
against.
But that's fine because the logic was also more generally wrong.
According to the existing logic, a value is the median value if
exactly as many values are less than it as are greater than it.
This is... not actually very accurate to what we usually mean by
"median". Because some values are *equal* to a given value. So
for instance, say you have the values {1, 1, 1, [a million 2s], 3}.
Our logic would regard 2 as being too high to be the median, because
3 times as many values are lower as are higher.
New interpretation: Imagine a sorted list of all your values, with
N entries. You want the Nth percentile, which is to say, you want N%
of values to be less than the vale you pick, and (100-N)% to be greater.
You can round both of these down. So for instance, if you have 6 values,
and want the median, you want 3 values greater, and 3 values less. To
be picky, we could demand the average of those middle two values, but
we're not in a good position to do that in this implementation.
If the number of desired things less than, or greater than, a target
is 0, we can short-circuit to the minimum or maximum value. This can
happen when nth is close to an end and the number of things is small,
not just at nth=0/nth=100.
So we rework this, and we rework the tests for this behavior to reflect
that logic.
We change executePercentile to be able to return a nil rather than
a weird ValCount in cases where there's no result, such as when
there's no values to compute a percentile of.
We also change the SQL tests to match the new behavior, since some
of them were expecting everything done on a decimal field with values
10-13 to come back as 10.00 as a decimal because that is what the
code returned.
We also propagate these changes to DAX, and along the way, fix up a
TODO item in the DAX copy, and stop skipping the test that was
failing because of that TODO item.
|
||
|---|---|---|
| .github/workflows | ||
| .gitlab | ||
| api/client | ||
| authn | ||
| authz | ||
| batch | ||
| buffer | ||
| bufferpool | ||
| cli | ||
| client | ||
| cmd | ||
| context | ||
| ctl | ||
| dax | ||
| debugstats | ||
| disco | ||
| encoding/proto | ||
| errors | ||
| etcd | ||
| extendiblehash | ||
| gcnotify | ||
| generator | ||
| gopsutil | ||
| hash | ||
| idk | ||
| install | ||
| internal | ||
| lattice | ||
| logger | ||
| lru | ||
| mock | ||
| monitor | ||
| net | ||
| pb | ||
| pql | ||
| prometheus | ||
| proto | ||
| qa | ||
| querycontext | ||
| rbf | ||
| roaring | ||
| runners | ||
| scripts | ||
| server | ||
| shardwidth | ||
| short_txkey | ||
| sql | ||
| sql3 | ||
| statik | ||
| stats | ||
| storage | ||
| systemlayer | ||
| syswrap | ||
| task | ||
| test | ||
| testdata | ||
| testhook | ||
| toml | ||
| tracing | ||
| txkey | ||
| vprint | ||
| wireprotocol | ||
| .gitignore | ||
| .golangci.yml | ||
| api.go | ||
| api_directive.go | ||
| api_directive_internal_test.go | ||
| api_directive_test.go | ||
| api_test.go | ||
| apimethod_string.go | ||
| apply.go | ||
| arrow.go | ||
| arrow_test.go | ||
| audit.go | ||
| audit_internal_test.go | ||
| audit_test.go | ||
| broadcast.go | ||
| bsi.go | ||
| bsi_test.go | ||
| cache.go | ||
| cache_test.go | ||
| catcher.go | ||
| cluster.go | ||
| cluster_internal_test.go | ||
| CODE_OF_CONDUCT.md | ||
| const_amd64.go | ||
| const_other.go | ||
| dataframe_test.go | ||
| dbshard.go | ||
| dbshard_internal_test.go | ||
| dbshard_test.go | ||
| delete_test.go | ||
| diagnostics.go | ||
| diagnostics_internal_test.go | ||
| doc.go | ||
| Dockerfile | ||
| Dockerfile-clustertests | ||
| Dockerfile-clustertests-client | ||
| Dockerfile-datagen | ||
| Dockerfile-dax | ||
| Dockerfile-dax-quick | ||
| Dockerfile-fbsql | ||
| et_test.go | ||
| event.go | ||
| executor.go | ||
| executor_internal_test.go | ||
| executor_test.go | ||
| field.go | ||
| field_internal_test.go | ||
| field_test.go | ||
| filesystem.go | ||
| fragment.go | ||
| fragment_internal_test.go | ||
| gc.go | ||
| gid.go | ||
| go.mod | ||
| go.sum | ||
| hack.go | ||
| handler.go | ||
| handler_test.go | ||
| holder.go | ||
| holder_internal_test.go | ||
| holder_test.go | ||
| http_handler.go | ||
| http_handler_internal_test.go | ||
| http_handler_test.go | ||
| http_translator.go | ||
| http_translator_test.go | ||
| idalloc.go | ||
| idalloc_test.go | ||
| importer.go | ||
| index.go | ||
| index_internal_test.go | ||
| index_test.go | ||
| internal_client.go | ||
| internal_client_test.go | ||
| iterator.go | ||
| iterator_internal_test.go | ||
| LICENSE | ||
| LICENSE-2.0.txt | ||
| license.exceptions | ||
| like.go | ||
| like_test.go | ||
| main_test.go | ||
| Makefile | ||
| metrics.go | ||
| nfpm.yaml | ||
| NOTICE | ||
| null_test.go | ||
| performancecounters.go | ||
| pilosa.go | ||
| pilosa_internal_test.go | ||
| pilosa_test.go | ||
| pprof.go | ||
| rbf.go | ||
| README.md | ||
| row.go | ||
| row_test.go | ||
| schema.go | ||
| serializer.go | ||
| server.go | ||
| server_internal_test.go | ||
| server_test.go | ||
| sql_test.go | ||
| stattx.go | ||
| systemlayer.go | ||
| time.go | ||
| time_internal_test.go | ||
| tracker.go | ||
| tracker_test.go | ||
| transaction.go | ||
| transaction_test.go | ||
| translate.go | ||
| translate_boltdb.go | ||
| translate_boltdb_internal_test.go | ||
| translate_boltdb_test.go | ||
| translator_test.go | ||
| tx.go | ||
| tx_internal_test.go | ||
| tx_test.go | ||
| txfactory.go | ||
| txfactory_internal_test.go | ||
| util.go | ||
| util_test.go | ||
| utils_internal_test.go | ||
| verchk.go | ||
| version.go | ||
| view.go | ||
| view_internal_test.go | ||
| wire_response.go | ||
FeatureBase
Pilosa is now FeatureBase
As of September 7, 2022, the Pilosa project is now FeatureBase. The core of the project remains the same: FeatureBase is the first real-time distributed database built entirely on bitmaps. (More information about updated capabilities and improvements below.)
FeatureBase delivers low-latency query results, regardless of throughput or query volumes, on fresh data with extreme efficiency. It works because bitmaps are faster, simpler, and far more I/O efficient than traditional column-oriented data formats. With FeatureBase, you can ingest data from batch data sources (e.g. S3, CSV, Snowflake, BigQuery, etc.) and/or streaming data sources (e.g. Kafka/Confluent, Kinesis, Pulsar).
For more information about FeatureBase, please visit www.featurebase.com.
Getting Started
Build FeatureBase Server from source
- Install go. Ensure that your shell's search path includes the go/bin directory.
- Clone the FeatureBase repository (or download as zip).
- In the featurebase directory, run
make installto compile the FeatureBase server binary. By default, it will be installed in the go/bin directory. - In the idk directory, run
make installto compile the ingester binaries. By default, they will be installed in the go/bin directory. - Run
featurebase server --handler.allowed-origins=http://localhost:3000to run FeatureBase server with default settings (learn more about configuring FeatureBase at the link below). The--handler.allowed-originsparameter allows the standalone web UI to talk to the server; this can be omitted if the web UI is not needed. - Run
curl localhost:10101/statusto verify the server is running and accessible.
Ingest Data and Query
- Run
molecula-consumer-csv \
--index repository \
--header "language__ID_F,project_id__ID_F" \
--id-field project_id \
--batch-size 1000 \
--files example.csv
This will ingest the example.csv file into a FeatureBase table called repository. If the table does not exist, it will be automatically created. Learn more about ingesting data into FeatureBase
- Query your data.
curl localhost:10101/index/repository/query \
-X POST \
-d 'Row(example=5)'
Learn about supported SQL, native Pilosa Query Language (PQL).
Data Model
Because FeatureBase is built on bitmaps, there is bit of a learning curve to grasp how your data is represented. Learn about Data Modeling.
More Information
Community
You can email us at community@featurebase.com or learn more about contributing at https://www.featurebase.com/community.
Chat with us: https://discord.gg/FBn2vEp7Na
What's Changed Since the Pilosa Days?
A lot has changed since the days of Pilosa. This list highlights some new capabilites included in FeatureBase. We have also made signficant improvements to the performance, scalability, and stability of the FeatureBase product.
- Query Languages: FeatureBase supports Pilosa Query Language (PQL), as well as SQL
- Stream and Batch Ingest: Combine real-time data streams with batch historical data and act on it within milliseconds.
- Mutable: Perform inserts, updates, and deletes at scale, in real time and on-the-fly. This is key for meeting data compliance requirements, and for reflecting the constantly-changing nature of high-volume data.
- Multi-Valued Set Fields: Store multiple comma-delimited values within a single field while increasing query performance of counts, TopKs, etc.
- Time Quantums: Setting a time quantum on a field creates extra views which allow ranged Row queries down to the time interval specified. For example, if the time quantum is set to YMD, ranged Row queries down to the granularity of a day are supported.
- RBF storage backend: this is a new compressed bitmap format which improves performance in a number of ways: ACID support on a per shard basis, prevents issues with the number of open files, reduces memory allocation and lock contention for reads, provides more consistent garbage collection, and allows backups to run concurrently with writes. However, because of this change, Pilosa backup files cannot be restored into FeatureBase.
License
FeatureBase is licensed under the Apache License, Version 2.0