scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-09-12 07:41:02 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Merge pull request #1213 from jaffee/1204-pdk-docs

Browse source 
remove outdated pdk docs, and write some new ones
This commit is contained in:
Matthew Jaffee 2018-04-24 09:24:35 -05:00 • committed by GitHub
commit abda2a1267
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 58 additions and 52 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

110
docs/pdk.md
View file

@ -2,66 +2,72 @@
title = "PDK"
weight = 11
nav = [
"Examples and Executables",
"Library",
"Examples",
]
+++
## PDK
The [Pilosa Dev Kit](https://github.com/pilosa/pdk) contains Go libraries to help you use Pilosa effectively. From importing data quickly, to managing the mappings from contiguous integer ids to values of other types, the PDK should help you get off the ground quickly.
The [Pilosa Dev Kit](https://github.com/pilosa/pdk) contains executables, examples, and Go libraries to help you use Pilosa effectively.
### Examples and Executables
Running `pdk -h` will give the most up to date list of all the tools and examples that PDK provides. We'll cover a few of the more important ones here.
#### Kafka
`pdk kafka` reads either JSON or Avro encoded records from Kafka (using the
Confluent Schema Registry in the case of Avro), and indexes them in Pilosa. Each
record from Kafka is assigned a Pilosa column, and each value in a record is
assigned a row or field. Frame and field names are built from the "path" through
the record to arrive at that field. For example:
```json
{
"name": "jill",
"favorite_foods": ["corn chips", "chipotle dip"],
"location": {
"city": "Austin",
"state": "Texas",
"latitude": 3754,
"longitude": 4526
},
"active": true,
"age": 27
}
```
This JSON object would result in the following Pilosa schema:
| Name | Field | Type | Size/Min | Max |
|----------------|-----------|--------|----------|------------|
| name | | ranked | 100000 | |
| favorite_foods | | ranked | 100000 | |
| default | | Ranked | 100000 | |
| | age | int | 0 | 2147483647 |
| location | | ranked | 1000 | |
| | latitude | int | 0 | 2147483647 |
| | longitude | int | 0 | 2147483647 |
| location-city | | ranked | 100000 | |
| location-state | | ranked | 100000 | |
All frames are created as ranked frames by default, and fields are created with
a minmum size of zero and a fixed maximum of 2147483647. Fields at the top level
are created in the default frame. Frames are a dash-separated concatenation of
all key values in the path - you can see this with frames like location-city.
Most the options to `pdk kafka` are self-explanatory (kafka hosts, pilosa hosts,
kafka topics, kafka group, etc.), but there are a few options that give some
control over the way data is indexed, and ingestion performance.
* `--batch-size`: The batch size control how many set bits or values are batched up to be imported *per frame*. So for fields that have one value per record, you have to wait for `batch-size` records to come through before you'll see the data indexed in Pilosa. Fields like `favorite_foods` which can have multiple values could be indexed sooner.
* `--framer.collapse`: This is a list of strings which will be removed from the frame names created by dash-concatentating all names in the JSON path to a value. E.G. if "location" were listed in `framer.collapse`, then there would be frames named "city" and "state" rather than "location-city" and "location-state".
* `--framer.ignore`: This allows you to skip indexing on any path containing these strings. If you have a field like email address or some other unique ID, you might not want to index it.
* `--subject-path`: If nothing is passed for this option, then each record will be assigned a unique sequential column ID. If `subject-path` is specified, then the value at this path in the record will be mapped to a column ID. If the same value appears in another record, the same column ID will be used.
* `--proxy`: The PDK ingests data, but also keeps a mapping for string values to row IDs, and from subjects to column ids. Because of this, querying Pilosa directly may not be useful, since it only returns integer row and column ids. The PDK will start a proxy server which intercepts requests to Pilosa using strings for row and column ids, and translates them to the integers that Pilosa understands. It will also translate responses so that (e.g.) a TopN query will return `{"results":[[{"Key":"chipotle dip","Count":1},{"Key":"corn chips","Count":1}]]}`. By default, the mapping is stored in an embedded leveldb.
The PDK also contains some fully worked examples which make use of its tools. These are available in the `usecase` subdirectory and can be run as subcommands of the `pdk` binary.
### Library
#### Mapping
For now, the [Godocs](https://godoc.org/github.com/pilosa/pdk) have the most up to date library documentation.
Importing data into Pilosa is dependent on mapping it to integer IDs. PDK provides some predefined functions for inline mapping to simplify this process, supported by a framework for linking these mappings with the associated fields in a source CSV file. If no custom mapping is necessary, the entire import process can be described by an import definition file. The file is composed of four main parts:
* an enumeration of field names
* a list of parsers that are used to parse strings in the CSV to values
* a list of commonly used, named, mapper functions
* a list of ParserMappers - objects that encapsulate all of the work related to a single frame.
This definition file can quickly get long, and defining it manually would be quite tedious. That's why we have a tool to generate a definition file by looking at a data set. This will handle most of the legwork, but since it can only guess at the application, it uses the simplest mappings - each column gets mapped to one frame in an appropriate way. This is intended as a starting point, to be updated to suit your use of the PDK.
With this definition available, the PDK tool can run the import, which consists of these steps:
- create the index
- create all frames
- for each CSV file, read all rows
- for each CSV record:
- generate a columnID
- apply all ParserMappers, generating a list of (frame, ID) pairs
- set the appropriate bit. schematically: SetBit(row=rowID, frame=frame, col=columnID)
The process is summarized in this flowchart:
![Bitmapper flowchart](/img/docs/pdk-bitmapper-flowchart.svg)
Some of the simple mapper functions available with PDK include:
* YearMapper: Maps a `time.Time` value to an integer equal to the `Time`'s year.
* MonthMapper: Maps a `time.Time` value to an integer equal to the `Time`'s month, in [0, 11].
* DayOfWeekMapper: Maps a `time.Time` value to an integer equal to the `Time`'s day of the week, in [0, 6].
* HourMapper: Maps a `time.Time` value to an integer equal to the `Time`'s hour, in [0, 23].
* TimeOfDayMapper: Maps a `time.Time` value to the range [0, N-1], where N is specified by `Res`. This is useful if the resolution used by HourMapper is too small (or large). For example, TimeOfDayMapper with `Res`=48 maps to 48 half-hour bins.
* BoolMapper: Maps a boolean value to the range [0, 1].
* IntMapper: Maps an integer value to the range [Min, Max]. This is suitable for a field with a small- to moderate-sized domain.
* SparseIntMapper: Maps integer values through an arbitrary table, foreign keys for example. This is suitable if the table size is small.
* LinearFloatMapper: Maps floating point values through a linear function. Inputs in the range [`Min`, `Max`] are mapped to row IDs in the range [0, `Res - 1`], where each ID represents one of `Res` evenly spaced buckets.
* FloatMapper: Maps floating point values using arbitrary buckets, in case even spacing is not suitable. These buckets are specified with an array of floats representing the left end of each bucket.
* GridMapper: Maps a pair of floats to a single integer, identifying a cell in a rectangular grid. This can be used, for example, to represent (latitude, longitude) location coarsely, as in the taxi data example.
* CustomMapper: When none of the predefined mappers will work, or when multiple fields determine a row ID value, an arbitrary mapping function can be used. Define a function in Go, with the necessary behavior, and wrap it in a CustomMapper.
### Examples
Run `make install` to build and install the `pdk` binary which contains all the examples. Just running `pdk` will bring up a list of all the examples, with a brief description of each. `pdk help <example>` will bring up a more detailed description of that example along with all arguments that it accepts to configure its functionality.
<!--
#### Net
A detailed discussion of using Pilosa to index network traffic data is available [here - TODO]link blog post). This will discuss the implementation of `pdk net` as it relates to the use of the PDK library tools.
-->