Commit graph

5381 commits

Author SHA1 Message Date
pokeeffe-molecula
a1fc6d04a1 introduce performance counters and system table fanout, plus refactor metrics (#2363)
* performance counters

* first cut of perf counters and system table fanout and a wire protocol
* significantly refactored prometheus support; removed statsd and exprvar

* removed node_id

* put dax subquery test back

* Change Translator.TranslateFieldIDs method to take a dax.TableKeyer

There are a bunch of other calls to the Translator interface methods
with currently take an `index string`, and those need to be converted to
dax.TableKeyer as well. But I need to review each call, because in at
least one place I noticed one being called with `result.Index` instead
of with the qtbl available. And I don't yet know how those could be
different.

Co-authored-by: Travis Turner <travis@molecula.com>
(cherry picked from commit 7f6ea0e6e5)
2023-01-19 21:35:02 +00:00
Коrd Campbell
b6d290487d
update license to full license 2023-01-12 18:23:09 -06:00
Joseph Friedrich
61f9698082
Merge pull request #2201 from FeatureBaseDB/readme-doc-link-fixes
updated links in readme to point to latest pages in docs
2023-01-12 17:42:04 -06:00
Joseph Friedrich
5274caece8
Merge pull request #2204 from FeatureBaseDB/1-10-2023-cherry-pick
v3.27.0 cherry pick from private
2023-01-12 16:05:13 -06:00
Joe Friedrich
475bf58465 resolve go vet errors related to redeclares 2023-01-12 20:32:49 +00:00
Kord Campbell
3a7d6efcda add LICENSE 2023-01-12 09:18:35 -06:00
Joe Friedrich
48178064fe sync cherry-picks 4750c2215..17cb2cbb7 2023-01-12 00:59:13 +00:00
Joe Friedrich
fee6d8553a sync cherry-picks 4750c2215..17cb2cbb7 2023-01-12 00:58:33 +00:00
Коrd Campbell
e2c3b4008c fixes FB-1873 (#2400)
Co-authored-by: Kord Campbell <kord@Bob.local>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
(cherry picked from commit 17cb2cbb77)
2023-01-12 00:49:58 +00:00
pokeeffe-molecula
0ddf69a322 fix sum aggregate (fb-1874) (#2404)
* handle sum aggregates with ints; handle escaped quotes in blob literals

* added test ceoverage

* skip subquery test for dax

(cherry picked from commit af475a27f2)
2023-01-12 00:49:58 +00:00
rachithrr
1cdcaf472f FB-1814: Implement ASCII() (#2378)
* FB-1814: Implement ASCII()

(cherry picked from commit f590227471)
2023-01-12 00:49:58 +00:00
Julio Martinez
9030a054b7 Create main log with 640 permissions, leave all other logs the same. (#2402)
Co-authored-by: Julio <julio.martinez@featurebase.com>
(cherry picked from commit 3f91cc3dd1)
2023-01-12 00:49:58 +00:00
Julio Martinez
c23608557b Add dd tracer and associated config options (#2403)
Co-authored-by: Julio <julio.martinez@featurebase.com>
(cherry picked from commit 581a4d19a4)
2023-01-12 00:49:58 +00:00
Joe Friedrich
7acf3265ca fix import paths and import cycles 2023-01-12 00:31:14 +00:00
Joe Friedrich
77305b55ed fix indent 2023-01-11 19:29:01 +00:00
Joe Friedrich
d22bd9430f fix import paths 2023-01-11 18:59:24 +00:00
pokeeffe-molecula
3b1707c568 include space_used column in show table output (#2401)
(cherry picked from commit 4750c2215f)
2023-01-10 23:28:09 +00:00
Lory Cloutier
2b9c13497e Implement SPACE() function for SQL3 (#2399)
FB-1861

(cherry picked from commit da74f0a312)
2023-01-10 23:28:09 +00:00
pokeeffe-molecula
5e151eed0e handle insert into timequantum fields with default 'now' time (fb-1868) (#2398)
* handle insert into timequantum fields with default 'now' time

* allocate on the stack

(cherry picked from commit 18fe6a35f6)
2023-01-10 23:28:09 +00:00
Jacob Brinlee
3b233d271d adding more error handling for rbf (#2395)
* adding more error handling for rbf

Co-authored-by: Jacob Brinlee <jacobbrinlee@Jacobs-MBP.attlocal.net>
(cherry picked from commit c9ce26ce96)
2023-01-10 23:28:09 +00:00
pokeeffe-molecula
a6317c58e0 added space_used columns (#2397)
added space_used columns to show tables and fb_cluster_nodes

(cherry picked from commit 164aac509e)
2023-01-10 23:28:09 +00:00
pokeeffe-molecula
4e43b767df Bug fix round up (fb-1841, fb-1819, fb-1867) (#2396)
* check root operator after optimize

* round of bug fixes

(cherry picked from commit 9f042216a7)
2023-01-10 23:28:09 +00:00
Travis Turner
01cae92d96 Change JSON tag name on WireQueryResponse from execution-time to exec_time (#2394)
* Change JSON response name from exec_time to execution-time

Execution time stopped working in the CLI because it uses the latest
json tag.

* Wait, don't break the interface.

* Add a test for the sql response json tags.

This is to make sure that if someone like Travis just goes and changes a
tag name to be more consistent, that we perhaps catch that before it
gets to the end user.

* Change exec_time to execution-time after all

(cherry picked from commit b5dd3ea02e)
2023-01-10 23:28:09 +00:00
tgruben
5ec31d4159 [FB-1831] distribute bulk insert to owning node (#2391)
* distribute bulk insert to owning node

(cherry picked from commit e8505d8a53)
2023-01-10 23:28:09 +00:00
pokeeffe-molecula
74ee3ebf0e implemented DISTINCT (fb-1562) (#2388)
* implemented distinct

* implemented distinct
* uses first cut of a buffer pool, and extendible hashing with thresholded spill to disk
* tests
* cleaned up some stuff around query plan output to make developing tooling easier
* added optimization to call PQL Distinct()

* fixed test

* fix for passing wrong index name in orchestrator

* back out change to DistinctTimestamp

* fix other instance of wrong table name being passed

* use full index name instead of abbreviated one for translation. sigh.

* removed some unused code

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
(cherry picked from commit f030d58d95)
2023-01-10 23:28:00 +00:00
Matthew Jaffee
36f2dcce9e clean up TODOs. adds a control channel for on-demand snapshotting
(cherry picked from commit 4cc1667399)
2023-01-10 23:27:48 +00:00
Matthew Jaffee
e09a9dca47 first cut at removing all the shard/field/partition versioning
some cleanup needed

(cherry picked from commit bc9057f492)
2023-01-10 23:27:40 +00:00
Travis Turner
69a174412d Small adjustments to support the Serverless cloud merge (#2385)
This just changes a make target and the CLI setup. Nothing in
featurebase is actually affected.

(cherry picked from commit 33bc69ccc6)
2023-01-10 23:27:10 +00:00
Matthew Jaffee
0f67a0c432 first cut at automatic snapshotting
- had to make sure we don't snapshot until directive is fully applied
on a computer... otherwise there's races between loading the files and
truncating the write log.

- added a dirty bit to resources and a bool return to incrementing the
write log... don't snapshot if it returns false because that means
there's been no writes. (but make sure you close the storage transaction!)

- added the actually snapshotting routine which just fires every
<timeout> and serially snapshots everything.

- tweaked some logging

- added ability to get all tables in an org/db or literally all. I
think I just needed the "literally all", but it was natural to allow
it to be scoped to org or DB as well.

(cherry picked from commit b8b08bc9eb)
2023-01-10 23:26:29 +00:00
pokeeffe-molecula
f8120d4833 Consistency in error handling (fb-1799) (#2383)
* return 200 once plan compilation starts; if error, return error in response.

* removed some commented out code that is definitely not needed.

(cherry picked from commit 9dda3ff215)
2023-01-10 23:26:29 +00:00
Matthew Jaffee
dfd4bb1191 Expose translation mvcc (#2381)
* expose Transaction on TranslateStore for DAX Snapshotting

* try to fix ramdisk nonsense

apparently, we were running in either a shell env or docker env
randomly, so this could sometimes pass and sometimes fail since the
shell env had the ramdisk set up and docker didn't.

Now we force to run in docker always and set up ramdisk explicitly.

* debug ramdisk issue?

* fix tests... and a buncha other stuff

The executor test I modified failed when I changed DefaultPartitionN
to 8, but just because stuff was out of order so I made it more
robust.

I edited some data gen stuff to make shorter lines because it was
making grep results unusable.

the actual fix is in translate_boltdb_test.go

* clean up, fix code review feedback

(cherry picked from commit 3693b9950a)
2023-01-10 23:25:58 +00:00
pokeeffe-molecula
a6c165dd57 Implement DELETE (fb 1557) (#2382)
* delete implementation with test coverage

* optimize IN expressions; stop linter complaining

* fixed some uncovered query cases

* skip test in DAX for now

(cherry picked from commit 021219935f)
2023-01-10 23:25:08 +00:00
Bruce Baranowski
36c020f076 Fb 1818 Implement PREFIX() and SUFFIX() (#2371)
* Implement Prefix and Suffix
* Update substring out-of-index handling

(cherry picked from commit d19f3e81da)
2023-01-10 23:25:08 +00:00
tgruben
427f7a4494 Force make to use bash (#2380)
(cherry picked from commit 72d03602e0)
2023-01-10 23:25:08 +00:00
Matthew Jaffee
a9b248bb29 "fix" backup tar test by just comparing lengths not byte for byte
(cherry picked from commit 1e2e698225)
2023-01-10 23:25:08 +00:00
Matthew Jaffee
71c0624abb remove in mem translate store
(cherry picked from commit 7ab117f5d1)
2023-01-10 23:23:18 +00:00
Matthew Jaffee
da4a42dfe9 use separate qcx for write/read in test
now that min/max queries don't use a write Tx it seems we need to
separate read and write in the tests. Not sure I 100% understand this.

(cherry picked from commit a58299def4)
2023-01-10 23:22:58 +00:00
Matthew Jaffee
da85614c24 set transactions to writable: false for Max/Min/Value calls. these were mistakenly set to true during cleanup
(cherry picked from commit 53a8f58e91)
2023-01-10 23:22:58 +00:00
tgruben
24372a9405 [FB-1822] Change dataframe disk format to Arrow from parquet (#2376)
* change default backend to arrow file format instead of parquet

(cherry picked from commit 6d96a1474e)
2023-01-10 23:22:58 +00:00
Fletcher Haynes
2993ab5ca1 This fixes a bug with displaying errors returned from the SQL3 endpoint (#2374)
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
(cherry picked from commit f65367ab46)
2023-01-10 23:22:58 +00:00
Lory Cloutier
5f15d3ad10 Fix bulk ingest queries on multi-node databases (#2375)
CLOUD-1252
Implemented Jaffee's fix of checking for b.useShardTransactionalEndpoint
and only running the start/finish transaction block if it's false. Moved
stats timing to a separate defer so it could stay out of the if.

(cherry picked from commit b1f5264a4b)
2023-01-10 23:22:58 +00:00
rachithrr
4aa34d3e1c FB-1815: Implement CHAR() (#2369)
(cherry picked from commit 27963441ab)
2023-01-10 23:22:58 +00:00
Travis Turner
99e3fd14d7 Fix PQL distinct in dax (#2360)
* Fix PQL distinct in dax

When issuing a PQL Distinct() call (or any other call with a "index=" arg),
this commit will attempt to convert the value in the index arg with a
TableKeyer.

* Apply change to call.Children as well

* Add some PQL Distinct (join) test coverage

(cherry picked from commit 4e8fe488de)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
211c3b759d add allow_missing_values option to bulk insert (fb-1823) (#2372)
* add allow_missing_values option to bulk insert

* test coverage

* review feedback

(cherry picked from commit 15d2ee8b07)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
bb2c805cac handling missing epoch constraint correctly (#2366)
(cherry picked from commit 843312dfc9)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
f42a33640a enable handling string representations of integers (#2367)
(cherry picked from commit 3528ec8fc0)
2023-01-10 23:22:58 +00:00
pokeeffe-molecula
4ef70c19da SHOW CREATE TABLE issues (fb-1810) (#2365)
* fixed ddl issues with cache type/size; removed shardwidth option; improved error message

(cherry picked from commit c88d60c9ab)
2023-01-10 23:22:58 +00:00
Matthew Jaffee
f6c0cf1112 rename stupid manager names
ManagerManager -> ResourceManager
Manager -> Resource

(cherry picked from commit 033be81799)
2023-01-10 23:22:58 +00:00
Matthew Jaffee
0e8773707a code review tweaks
(cherry picked from commit cf1c9dae9a)
2023-01-10 23:22:58 +00:00
Matthew Jaffee
9a441ef66b remove version/directive stuff from other snapshot endpoints
(cherry picked from commit 4366ad41fb)
2023-01-10 23:22:58 +00:00