mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 02:44:59 +00:00
No description
b7e9879526
This forward-ports a number of tests from the previous SQL implementation. The porting is approximate in a number of ways, and not all tests are implemented/tested yet. In particular, several tests are currently disabled because we don't support `limit n` constructs. The tests that were primarily tests of the parser have been brought forward as parser tests. One of them has been altered to add parentheses, because our parser interprets fld1 between 1 and 3 and fld2 = 2 as: fld1 between (1 and 3) and (fld2 = 2) which is invalid, while the old parser apparently interpreted it as: (fld1 between 1 and 3) and (fld2 = 2) We have not yet verified the SQL spec's requirements here, but sqlite agrees with our old parser, not our new parser, so this may be a regression. The old tests expected an INNER JOIN to suppress duplicate values. Our new code does not, which is consistent with other SQL implementations. This is a change, but the old behavior appears to have been wrong. (You can still suppress duplicate values by specifying DISTINCT.) In the previous implementations, a value like `count(*)` had `count(*)` as its column name. In the new implementation, it has an empty string as its column name. Related to this, the prior implementation allowed you to write select age, count(*) from grouper group by age having count > 1 but the new implementationt requires that to be spelled as having count(*) > 1 This is consistent with other SQL implementations, so I think the new behavior is correct. The behavior of SHOW COLUMNS and SHOW TABLES has changed, in that the specific results returned are significantly different. Perhaps more significantly, the old system spelled the former query as SHOW FIELDS, rather than SHOW COLUMNS. This may be considered a regression, in that `SHOW FIELDS` no longer works, and we should consider whether any hypothetical users might have been relying on the output of either of these. (I hope not, the new output is much better.) Some of the old tests (the ones in handler_test) were accommodated by adding a couple of specific test cases to existing tests, specifically: * handling timestamp values with `Z` rather than `+00:00` * a join with a WHERE clause referring to fields in both source tables We introduce a new "partial" comparison type, because there's no way for a test of `SHOW TABLES` to contain a correct table row, because `SHOW TABLES` includes timestamps from when tables were created. I'm not sure this is the right way to do this. We add corresponding changes to dax_test, because the DAX tree tests against the SQL tests. We change the returned types of field names and field types to plain strings, ironically because DAX needs this -- the test code in the DAX tree is getting them back as plain strings, rather than as dax.FieldName and dax.BaseType. The tests using `having` are commented out because they don't seem to be working, a ticket has been filed for this. Two of the tests that should return strings are instead returning untranslated integer IDs, but only for DAX, not for the regular SQL tests, and the `delete` test has been commented out for DAX-specific errors. If we merge this, the next step is to ticket those and address them separately. |
||
|---|---|---|
| .github/workflows | ||
| .gitlab | ||
| api/client | ||
| authn | ||
| authz | ||
| batch | ||
| buffer | ||
| bufferpool | ||
| cli | ||
| client | ||
| cmd | ||
| context | ||
| ctl | ||
| dax | ||
| debugstats | ||
| disco | ||
| encoding/proto | ||
| errors | ||
| etcd | ||
| extendiblehash | ||
| gcnotify | ||
| generator | ||
| gopsutil | ||
| hash | ||
| idk | ||
| install | ||
| internal | ||
| lattice | ||
| logger | ||
| lru | ||
| mock | ||
| monitor | ||
| net | ||
| pb | ||
| pql | ||
| prometheus | ||
| proto | ||
| qa | ||
| querycontext | ||
| rbf | ||
| roaring | ||
| runners | ||
| scripts | ||
| server | ||
| shardwidth | ||
| short_txkey | ||
| sql | ||
| sql3 | ||
| statik | ||
| stats | ||
| storage | ||
| systemlayer | ||
| syswrap | ||
| task | ||
| test | ||
| testdata | ||
| testhook | ||
| toml | ||
| tracing | ||
| txkey | ||
| vprint | ||
| wireprotocol | ||
| .gitignore | ||
| .golangci.yml | ||
| api.go | ||
| api_directive.go | ||
| api_directive_internal_test.go | ||
| api_directive_test.go | ||
| api_test.go | ||
| apimethod_string.go | ||
| apply.go | ||
| arrow.go | ||
| arrow_test.go | ||
| audit.go | ||
| audit_internal_test.go | ||
| audit_test.go | ||
| broadcast.go | ||
| bsi.go | ||
| bsi_test.go | ||
| cache.go | ||
| cache_test.go | ||
| catcher.go | ||
| cluster.go | ||
| cluster_internal_test.go | ||
| CODE_OF_CONDUCT.md | ||
| const_amd64.go | ||
| const_other.go | ||
| dataframe_test.go | ||
| dbshard.go | ||
| dbshard_internal_test.go | ||
| dbshard_test.go | ||
| delete_test.go | ||
| diagnostics.go | ||
| diagnostics_internal_test.go | ||
| doc.go | ||
| Dockerfile | ||
| Dockerfile-clustertests | ||
| Dockerfile-clustertests-client | ||
| Dockerfile-datagen | ||
| Dockerfile-dax | ||
| Dockerfile-dax-quick | ||
| et_test.go | ||
| event.go | ||
| executor.go | ||
| executor_internal_test.go | ||
| executor_test.go | ||
| field.go | ||
| field_internal_test.go | ||
| field_test.go | ||
| filesystem.go | ||
| fragment.go | ||
| fragment_internal_test.go | ||
| gc.go | ||
| gid.go | ||
| go.mod | ||
| go.sum | ||
| hack.go | ||
| handler.go | ||
| handler_test.go | ||
| holder.go | ||
| holder_internal_test.go | ||
| holder_test.go | ||
| http_handler.go | ||
| http_handler_internal_test.go | ||
| http_handler_test.go | ||
| http_translator.go | ||
| http_translator_test.go | ||
| idalloc.go | ||
| idalloc_test.go | ||
| importer.go | ||
| index.go | ||
| index_internal_test.go | ||
| index_test.go | ||
| internal_client.go | ||
| internal_client_test.go | ||
| iterator.go | ||
| iterator_internal_test.go | ||
| LICENSE | ||
| LICENSE-2.0.txt | ||
| license.exceptions | ||
| like.go | ||
| like_test.go | ||
| main_test.go | ||
| Makefile | ||
| metrics.go | ||
| nfpm.yaml | ||
| NOTICE | ||
| performancecounters.go | ||
| pilosa.go | ||
| pilosa_internal_test.go | ||
| pilosa_test.go | ||
| pprof.go | ||
| rbf.go | ||
| README.md | ||
| row.go | ||
| row_test.go | ||
| schema.go | ||
| serializer.go | ||
| server.go | ||
| server_internal_test.go | ||
| server_test.go | ||
| sql_test.go | ||
| stattx.go | ||
| systemlayer.go | ||
| time.go | ||
| time_internal_test.go | ||
| tracker.go | ||
| tracker_test.go | ||
| transaction.go | ||
| transaction_test.go | ||
| translate.go | ||
| translate_boltdb.go | ||
| translate_boltdb_internal_test.go | ||
| translate_boltdb_test.go | ||
| translator_test.go | ||
| tx.go | ||
| tx_internal_test.go | ||
| tx_test.go | ||
| txfactory.go | ||
| txfactory_internal_test.go | ||
| util.go | ||
| util_test.go | ||
| utils_internal_test.go | ||
| verchk.go | ||
| version.go | ||
| view.go | ||
| view_internal_test.go | ||
| wire_response.go | ||
FeatureBase
Pilosa is now FeatureBase
As of September 7, 2022, the Pilosa project is now FeatureBase. The core of the project remains the same: FeatureBase is the first real-time distributed database built entirely on bitmaps. (More information about updated capabilities and improvements below.)
FeatureBase delivers low-latency query results, regardless of throughput or query volumes, on fresh data with extreme efficiency. It works because bitmaps are faster, simpler, and far more I/O efficient than traditional column-oriented data formats. With FeatureBase, you can ingest data from batch data sources (e.g. S3, CSV, Snowflake, BigQuery, etc.) and/or streaming data sources (e.g. Kafka/Confluent, Kinesis, Pulsar).
For more information about FeatureBase, please visit www.featurebase.com.
Getting Started
Build FeatureBase Server from source
- Install go. Ensure that your shell's search path includes the go/bin directory.
- Clone the FeatureBase repository (or download as zip).
- In the featurebase directory, run
make installto compile the FeatureBase server binary. By default, it will be installed in the go/bin directory. - In the idk directory, run
make installto compile the ingester binaries. By default, they will be installed in the go/bin directory. - Run
featurebase server --handler.allowed-origins=http://localhost:3000to run FeatureBase server with default settings (learn more about configuring FeatureBase at the link below). The--handler.allowed-originsparameter allows the standalone web UI to talk to the server; this can be omitted if the web UI is not needed. - Run
curl localhost:10101/statusto verify the server is running and accessible.
Ingest Data and Query
- Run
molecula-consumer-csv \
--index repository \
--header "language__ID_F,project_id__ID_F" \
--id-field project_id \
--batch-size 1000 \
--files example.csv
This will ingest the example.csv file into a FeatureBase table called repository. If the table does not exist, it will be automatically created. Learn more about ingesting data into FeatureBase
- Query your data.
curl localhost:10101/index/repository/query \
-X POST \
-d 'Row(example=5)'
Learn about supported SQL, native Pilosa Query Language (PQL).
Data Model
Because FeatureBase is built on bitmaps, there is bit of a learning curve to grasp how your data is represented. Learn about Data Modeling.
More Information
Community
You can email us at community@featurebase.com or learn more about contributing at https://www.featurebase.com/community.
Chat with us: https://discord.gg/FBn2vEp7Na
What's Changed Since the Pilosa Days?
A lot has changed since the days of Pilosa. This list highlights some new capabilites included in FeatureBase. We have also made signficant improvements to the performance, scalability, and stability of the FeatureBase product.
- Query Languages: FeatureBase supports Pilosa Query Language (PQL), as well as SQL
- Stream and Batch Ingest: Combine real-time data streams with batch historical data and act on it within milliseconds.
- Mutable: Perform inserts, updates, and deletes at scale, in real time and on-the-fly. This is key for meeting data compliance requirements, and for reflecting the constantly-changing nature of high-volume data.
- Multi-Valued Set Fields: Store multiple comma-delimited values within a single field while increasing query performance of counts, TopKs, etc.
- Time Quantums: Setting a time quantum on a field creates extra views which allow ranged Row queries down to the time interval specified. For example, if the time quantum is set to YMD, ranged Row queries down to the granularity of a day are supported.
- RBF storage backend: this is a new compressed bitmap format which improves performance in a number of ways: ACID support on a per shard basis, prevents issues with the number of open files, reduces memory allocation and lock contention for reads, provides more consistent garbage collection, and allows backups to run concurrently with writes. However, because of this change, Pilosa backup files cannot be restored into FeatureBase.
License
FeatureBase is licensed under the Apache License, Version 2.0