mirror of
https://github.com/featurebasedb/featurebase.git
synced 2026-10-16 15:53:35 +00:00
4c53f86e82
i used this script, a little clunky but it got the job done
```bash
for file in `find . -type f -print | grep '\.go'`; do
sed '1,/^\/\/ limitations under the License.$/d' $file > $file.tmp;
result=`cat $file.tmp`
if [[ result != "" ]]; then
gofmt $file.tmp &> /dev/null;
if [[ $? == 0 ]]; then
mv $file.tmp $file && gofmt -w $file;
else
rm $file.tmp;
fi
else
rm $file.tmp;
fi
done
```
16 lines
815 B
Go
16 lines
815 B
Go
// Package ingest provides tooling for accepting record-oriented data updates
|
|
// and converting them to data that can be efficiently merged into stored
|
|
// data. Nia's original description:
|
|
//
|
|
// but the overall pipeline is:
|
|
// 1. fetch the schema and use it to configure the codec
|
|
// 2. parse the data with the codec into vectors, while stuffing temp record key mappings into a string table
|
|
// 3. call *CreateKeys on the cluster for all of the things
|
|
// 4. generate an ID remapping table for record keys and apply it to all of the vectors
|
|
// 5. remap the string keys
|
|
// 6. group each vector by shard
|
|
// 7. convert the shard vectors into matrix updates
|
|
// 8. combine those matrix updates into a shard update
|
|
// 9. send the shard updates out over the internal client
|
|
// 10. the nodes apply them to RBF
|
|
package ingest
|