mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
No description
244d80753e
Buckle in, this one's a ride. This is attached to the same PR as a fix for exiting abruptly during some tests because I ran into that issue, and comprehended it, while trying to track down weird and sporadic test failures that were actually this issue. The actual, underlying, problem: `make test`, by running all the tests at once, was hitting a bug that was mostly effectively triggered by running the `dax/test/dax` tests, and the top-level `featurebase/v3` tests, at the same time. However, the interaction was nothing as obvious as temporary files, etcd configuration, or whatever. We were running out of port numbers. The tests were using a bit over 30k simultaneous established TCP connections, each to different ports, because we were creating new clients for basically every single operation. For instance, in a single SQL test that did an import and then a read, we were creating a new client for each field written to, and then also creating a new client for each field in results that needed key translation. And none of these clients were closed or timed out in any way. In fact, Go doesn't really *do* "closing" of clients; the closest is that an http.Client can be told to close idle connections that it has been keeping open. The worst offenders were both named `fbClient`, and were nigh-identical, except one of them was implemented as a method on `importer` in the IDK tree, and one was a standalone function. It may seem surprising that the method on `importer` is using a shared client pool for all importers, rather than a new pool for each importer. This is because we potentially make quite a few importers during tests. Before this, running either of the dax tests or the top-level tests would show well over ten thousand simultaneous ESTABLISHED connections. After this, the dax tests used nearly twenty. The problem with port consumption like this, while more noticeable on MacOS, is also something we could hit on the CI runners, especially if a single runner ended up with more than one test suite running at the same time. This probably manifests as sporadic very strange failures of CI, with messages about "cannot assign requested address". (Note that an outgoing connection to a successfully-created port requires *another* port to be assigned for the outbound socket.) This was complicated dramatically by the fact that, for some utterly cursed reason, it was *especially* common for the point at which we hit this, in the top-level featurebase tests, to be running one of the backup tests in TestVariousQueries, and specifically, to be hitting it on the dataframe part of the backup... Which is to say, on the *one* path in the backup function that called log.Fatal, and thus terminated the featurebase process abruptly without further commentary. |
||
|---|---|---|
| .github/workflows | ||
| .gitlab | ||
| api/client | ||
| authn | ||
| authz | ||
| batch | ||
| buffer | ||
| bufferpool | ||
| cli | ||
| client | ||
| cmd | ||
| context | ||
| ctl | ||
| dax | ||
| debugstats | ||
| disco | ||
| encoding/proto | ||
| errors | ||
| etcd | ||
| extendiblehash | ||
| gcnotify | ||
| generator | ||
| gopsutil | ||
| hash | ||
| idk | ||
| install | ||
| internal | ||
| lattice | ||
| logger | ||
| lru | ||
| mock | ||
| monitor | ||
| net | ||
| pb | ||
| pql | ||
| prometheus | ||
| proto | ||
| qa | ||
| querycontext | ||
| rbf | ||
| roaring | ||
| runners | ||
| scripts | ||
| server | ||
| shardwidth | ||
| short_txkey | ||
| sql | ||
| sql3 | ||
| statik | ||
| stats | ||
| storage | ||
| systemlayer | ||
| syswrap | ||
| task | ||
| test | ||
| testdata | ||
| testhook | ||
| toml | ||
| tracing | ||
| txkey | ||
| vprint | ||
| wireprotocol | ||
| .gitignore | ||
| .golangci.yml | ||
| api.go | ||
| api_directive.go | ||
| api_directive_internal_test.go | ||
| api_directive_test.go | ||
| api_test.go | ||
| apimethod_string.go | ||
| apply.go | ||
| arrow.go | ||
| arrow_test.go | ||
| audit.go | ||
| audit_internal_test.go | ||
| audit_test.go | ||
| broadcast.go | ||
| bsi.go | ||
| bsi_test.go | ||
| cache.go | ||
| cache_test.go | ||
| catcher.go | ||
| cluster.go | ||
| cluster_internal_test.go | ||
| CODE_OF_CONDUCT.md | ||
| const_amd64.go | ||
| const_other.go | ||
| dataframe_test.go | ||
| dbshard.go | ||
| dbshard_internal_test.go | ||
| dbshard_test.go | ||
| delete_test.go | ||
| diagnostics.go | ||
| diagnostics_internal_test.go | ||
| doc.go | ||
| Dockerfile | ||
| Dockerfile-clustertests | ||
| Dockerfile-clustertests-client | ||
| Dockerfile-datagen | ||
| Dockerfile-dax | ||
| Dockerfile-dax-quick | ||
| et_test.go | ||
| event.go | ||
| executor.go | ||
| executor_internal_test.go | ||
| executor_test.go | ||
| field.go | ||
| field_internal_test.go | ||
| field_test.go | ||
| filesystem.go | ||
| fragment.go | ||
| fragment_internal_test.go | ||
| gc.go | ||
| gid.go | ||
| go.mod | ||
| go.sum | ||
| hack.go | ||
| handler.go | ||
| handler_test.go | ||
| holder.go | ||
| holder_internal_test.go | ||
| holder_test.go | ||
| http_handler.go | ||
| http_handler_internal_test.go | ||
| http_handler_test.go | ||
| http_translator.go | ||
| http_translator_test.go | ||
| idalloc.go | ||
| idalloc_test.go | ||
| importer.go | ||
| index.go | ||
| index_internal_test.go | ||
| index_test.go | ||
| internal_client.go | ||
| internal_client_test.go | ||
| iterator.go | ||
| iterator_internal_test.go | ||
| LICENSE | ||
| LICENSE-2.0.txt | ||
| license.exceptions | ||
| like.go | ||
| like_test.go | ||
| main_test.go | ||
| Makefile | ||
| metrics.go | ||
| nfpm.yaml | ||
| NOTICE | ||
| performancecounters.go | ||
| pilosa.go | ||
| pilosa_internal_test.go | ||
| pilosa_test.go | ||
| pprof.go | ||
| rbf.go | ||
| README.md | ||
| row.go | ||
| row_test.go | ||
| schema.go | ||
| serializer.go | ||
| server.go | ||
| server_internal_test.go | ||
| server_test.go | ||
| sql_test.go | ||
| stattx.go | ||
| systemlayer.go | ||
| time.go | ||
| time_internal_test.go | ||
| tracker.go | ||
| tracker_test.go | ||
| transaction.go | ||
| transaction_test.go | ||
| translate.go | ||
| translate_boltdb.go | ||
| translate_boltdb_internal_test.go | ||
| translate_boltdb_test.go | ||
| translator_test.go | ||
| tx.go | ||
| tx_internal_test.go | ||
| tx_test.go | ||
| txfactory.go | ||
| txfactory_internal_test.go | ||
| util.go | ||
| util_test.go | ||
| utils_internal_test.go | ||
| verchk.go | ||
| version.go | ||
| view.go | ||
| view_internal_test.go | ||
| wire_response.go | ||
FeatureBase
Pilosa is now FeatureBase
As of September 7, 2022, the Pilosa project is now FeatureBase. The core of the project remains the same: FeatureBase is the first real-time distributed database built entirely on bitmaps. (More information about updated capabilities and improvements below.)
FeatureBase delivers low-latency query results, regardless of throughput or query volumes, on fresh data with extreme efficiency. It works because bitmaps are faster, simpler, and far more I/O efficient than traditional column-oriented data formats. With FeatureBase, you can ingest data from batch data sources (e.g. S3, CSV, Snowflake, BigQuery, etc.) and/or streaming data sources (e.g. Kafka/Confluent, Kinesis, Pulsar).
For more information about FeatureBase, please visit www.featurebase.com.
Getting Started
Build FeatureBase Server from source
- Install go. Ensure that your shell's search path includes the go/bin directory.
- Clone the FeatureBase repository (or download as zip).
- In the featurebase directory, run
make installto compile the FeatureBase server binary. By default, it will be installed in the go/bin directory. - In the idk directory, run
make installto compile the ingester binaries. By default, they will be installed in the go/bin directory. - Run
featurebase server --handler.allowed-origins=http://localhost:3000to run FeatureBase server with default settings (learn more about configuring FeatureBase at the link below). The--handler.allowed-originsparameter allows the standalone web UI to talk to the server; this can be omitted if the web UI is not needed. - Run
curl localhost:10101/statusto verify the server is running and accessible.
Ingest Data and Query
- Run
molecula-consumer-csv \
--index repository \
--header "language__ID_F,project_id__ID_F" \
--id-field project_id \
--batch-size 1000 \
--files example.csv
This will ingest the example.csv file into a FeatureBase table called repository. If the table does not exist, it will be automatically created. Learn more about ingesting data into FeatureBase
- Query your data.
curl localhost:10101/index/repository/query \
-X POST \
-d 'Row(example=5)'
Learn about supported SQL, native Pilosa Query Language (PQL).
Data Model
Because FeatureBase is built on bitmaps, there is bit of a learning curve to grasp how your data is represented. Learn about Data Modeling.
More Information
Community
You can email us at community@featurebase.com or learn more about contributing at https://www.featurebase.com/community.
Chat with us: https://discord.gg/FBn2vEp7Na
What's Changed Since the Pilosa Days?
A lot has changed since the days of Pilosa. This list highlights some new capabilites included in FeatureBase. We have also made signficant improvements to the performance, scalability, and stability of the FeatureBase product.
- Query Languages: FeatureBase supports Pilosa Query Language (PQL), as well as SQL
- Stream and Batch Ingest: Combine real-time data streams with batch historical data and act on it within milliseconds.
- Mutable: Perform inserts, updates, and deletes at scale, in real time and on-the-fly. This is key for meeting data compliance requirements, and for reflecting the constantly-changing nature of high-volume data.
- Multi-Valued Set Fields: Store multiple comma-delimited values within a single field while increasing query performance of counts, TopKs, etc.
- Time Quantums: Setting a time quantum on a field creates extra views which allow ranged Row queries down to the time interval specified. For example, if the time quantum is set to YMD, ranged Row queries down to the granularity of a day are supported.
- RBF storage backend: this is a new compressed bitmap format which improves performance in a number of ways: ACID support on a per shard basis, prevents issues with the number of open files, reduces memory allocation and lock contention for reads, provides more consistent garbage collection, and allows backups to run concurrently with writes. However, because of this change, Pilosa backup files cannot be restored into FeatureBase.
License
FeatureBase is licensed under the Apache License, Version 2.0