mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
aa17b8d725
* Enable linter: stylecheck This enabled the stylecheck linter, but excludes some staticchecks for now. The following are ignored because they will take a bit of time to address, but the intention is to address them and remove them from the exclusion list. ST1000: at least one file in a package should have a package comment ST1003: golang naming standards ST1008: error should be returned as the last argument ST1016: methods on the same type should have the same receiver name ST1020: comment on exported function * Address ST1015 For some reason this failed in CI but not locally. I can't figure out why that check isn't happening locally. This just moves the switch statements around so that the `default` is the first (or last) item. * Adjust error string in test to match case-adjusted error * Remove TestCloseTimeout |
||
|---|---|---|
| .. | ||
| gen | ||
| testdata | ||
| all-field-types.go | ||
| bank.go | ||
| claim.go | ||
| cmd.go | ||
| common.go | ||
| custom.go | ||
| custom_test.go | ||
| customer.go | ||
| customer_segmentation.go | ||
| customer_segmentation_test.go | ||
| equipment.data.go | ||
| equipment.go | ||
| example.go | ||
| hobbies.data.go | ||
| item.go | ||
| kitchen-sink-keyed.go | ||
| kitchen-sink.go | ||
| locations.data.go | ||
| network.go | ||
| README.md | ||
| shared.go | ||
| sites.data.go | ||
| sites.go | ||
| sizing.go | ||
| skills.data.go | ||
| stringpk.go | ||
| timeseries.go | ||
| titles.data.go | ||
| transactions.go | ||
| transactions_scenario_1.go | ||
| uscities.data.go | ||
| warranty.go | ||
| zip_codes.data.go | ||
Datagen Tool
Help Usage
Usage of datagen:
-c, --concurrency int Number of concurrent sources and indexing routines to launch. (default 1)
--dry-run Dry run - just flag parsing.
-e, --end-at uint ID at which to stop generating records.
--kafka.batch-size int Number of records to generate before sending them to Kafka all at once. Generally, larger means better throughput and more memory usage. (default 1000)
--kafka.hosts strings Comma separated list of host:port pairs for Kafka. (default [])
--kafka.registry-url string Location of Confluent Schema Registry. Must start with 'https://' if you want to use TLS.
--kafka.subject string Kafka schema subject.
--kafka.topic string Kafka topic to post to.
--pilosa.batch-size int Number of records to read before indexing all of them at once. Generally, larger means better throughput and more memory usage. 1,048,576 might be a good number.
--pilosa.cache-length uint Number of batches of ID mappings to cache. (default 64)
--pilosa.hosts strings Comma separated list of host:port pairs for Pilosa. (default [])
--pilosa.index string Name of Pilosa index.
--seed int Seed to use for any random number generation.
-s, --source string Source generator type. Running datagen with no arguments will list the available source types.
-b, --start-from uint ID at which to start generating records.
-t, --target string Destination for the generated data: [kafka, pilosa]. (default "pilosa")
--track-progress Periodically print status updates on how many records have been sourced.
Example Usage
The following command will create 100 records in Pilosa index (starting at ID 0 and ending at ID 99)
in the equipment index using the equipment data generator.
datagen --source=equipment --pilosa.index=equipment --end-at=99
Adding New Sources
TODO: redo README (or delete?)
If you're looking to add a new Source to datagen, the best thing to do is use the special "custom" datagen source (datagen --source=custom --custom-config=somefile.yaml) and write a somefile.yaml which describes the data you want to generate. An example can be found in datagen/testdata/custom.yaml, and there are some more in the molecula/technical-validation repo.