mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-08-28 10:54:59 +00:00
016765d8a2
This partially-implemented prototype of the ingest API is based on our programmatic ingest API reference. It has noticable limitations, most crucially that it doesn't handle multi-node clusters right now. However, it basically implements the expected semantics. There's some noticeable performance issues to do with the high overhead of sorting bits in order to import them efficiently, but this is fixable. We also add the hooks to the internal client, and make the finisher logic a bit smarter. Much of this code was originally by Nia Weiss, but it's been merged and restructured a bit to get things broken into logical commits.
30 lines
1.4 KiB
Go
30 lines
1.4 KiB
Go
// Copyright 2021 Molecula Corp.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
// Package ingest provides tooling for accepting record-oriented data updates
|
|
// and converting them to data that can be efficiently merged into stored
|
|
// data. Nia's original description:
|
|
//
|
|
// but the overall pipeline is:
|
|
// 1. fetch the schema and use it to configure the codec
|
|
// 2. parse the data with the codec into vectors, while stuffing temp record key mappings into a string table
|
|
// 3. call *CreateKeys on the cluster for all of the things
|
|
// 4. generate an ID remapping table for record keys and apply it to all of the vectors
|
|
// 5. remap the string keys
|
|
// 6. group each vector by shard
|
|
// 7. convert the shard vectors into matrix updates
|
|
// 8. combine those matrix updates into a shard update
|
|
// 9. send the shard updates out over the internal client
|
|
// 10. the nodes apply them to RBF
|
|
package ingest
|