Commit graph

9572 commits

Author SHA1 Message Date
Коrd Campbell
17cb2cbb77
fixes FB-1873 (#2400)
Co-authored-by: Kord Campbell <kord@Bob.local>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
2023-01-11 16:48:03 -08:00
pokeeffe-molecula
af475a27f2
fix sum aggregate (fb-1874) (#2404)
* handle sum aggregates with ints; handle escaped quotes in blob literals

* added test ceoverage

* skip subquery test for dax
2023-01-11 15:46:41 -06:00
rachithrr
f590227471
FB-1814: Implement ASCII() (#2378)
* FB-1814: Implement ASCII()
2023-01-11 12:32:27 -05:00
Julio Martinez
3f91cc3dd1
Create main log with 640 permissions, leave all other logs the same. (#2402)
Co-authored-by: Julio <julio.martinez@featurebase.com>
2023-01-10 14:52:33 -08:00
Julio Martinez
581a4d19a4
Add dd tracer and associated config options (#2403)
Co-authored-by: Julio <julio.martinez@featurebase.com>
2023-01-10 14:06:09 -08:00
pokeeffe-molecula
4750c2215f
include space_used column in show table output (#2401) 2023-01-10 11:03:57 -06:00
Lory Cloutier
da74f0a312
Implement SPACE() function for SQL3 (#2399)
FB-1861
2023-01-09 12:20:04 -06:00
pokeeffe-molecula
18fe6a35f6
handle insert into timequantum fields with default 'now' time (fb-1868) (#2398)
* handle insert into timequantum fields with default 'now' time

* allocate on the stack
2023-01-06 16:58:37 -06:00
Jacob Brinlee
c9ce26ce96
adding more error handling for rbf (#2395)
* adding more error handling for rbf

Co-authored-by: Jacob Brinlee <jacobbrinlee@Jacobs-MBP.attlocal.net>
2023-01-06 14:04:17 -06:00
pokeeffe-molecula
164aac509e
added space_used columns (#2397)
added space_used columns to show tables and fb_cluster_nodes
2023-01-06 11:43:17 -06:00
pokeeffe-molecula
9f042216a7
Bug fix round up (fb-1841, fb-1819, fb-1867) (#2396)
* check root operator after optimize

* round of bug fixes
2023-01-05 20:13:26 -06:00
Travis Turner
b5dd3ea02e
Change JSON tag name on WireQueryResponse from execution-time to exec_time (#2394)
* Change JSON response name from exec_time to execution-time

Execution time stopped working in the CLI because it uses the latest
json tag.

* Wait, don't break the interface.

* Add a test for the sql response json tags.

This is to make sure that if someone like Travis just goes and changes a
tag name to be more consistent, that we perhaps catch that before it
gets to the end user.

* Change exec_time to execution-time after all
2023-01-05 18:34:01 -06:00
tgruben
e8505d8a53
[FB-1831] distribute bulk insert to owning node (#2391)
* distribute bulk insert to owning node
2023-01-04 14:26:08 -06:00
pokeeffe-molecula
f030d58d95
implemented DISTINCT (fb-1562) (#2388)
* implemented distinct

* implemented distinct
* uses first cut of a buffer pool, and extendible hashing with thresholded spill to disk
* tests
* cleaned up some stuff around query plan output to make developing tooling easier
* added optimization to call PQL Distinct()

* fixed test

* fix for passing wrong index name in orchestrator

* back out change to DistinctTimestamp

* fix other instance of wrong table name being passed

* use full index name instead of abbreviated one for translation. sigh.

* removed some unused code

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
2023-01-03 22:13:31 -06:00
Matthew Jaffee
4cc1667399 clean up TODOs. adds a control channel for on-demand snapshotting 2023-01-03 15:31:32 -06:00
Matthew Jaffee
bc9057f492 first cut at removing all the shard/field/partition versioning
some cleanup needed
2023-01-03 15:31:32 -06:00
Travis Turner
33bc69ccc6
Small adjustments to support the Serverless cloud merge (#2385)
This just changes a make target and the CLI setup. Nothing in
featurebase is actually affected.
2022-12-22 12:22:09 -06:00
Matthew Jaffee
c8242684f1 dax/test/dax need an image now that they're running under Docker 2022-12-21 07:42:04 -06:00
Matthew Jaffee
b8b08bc9eb first cut at automatic snapshotting
- had to make sure we don't snapshot until directive is fully applied
on a computer... otherwise there's races between loading the files and
truncating the write log.

- added a dirty bit to resources and a bool return to incrementing the
write log... don't snapshot if it returns false because that means
there's been no writes. (but make sure you close the storage transaction!)

- added the actually snapshotting routine which just fires every
<timeout> and serially snapshots everything.

- tweaked some logging

- added ability to get all tables in an org/db or literally all. I
think I just needed the "literally all", but it was natural to allow
it to be scoped to org or DB as well.
2022-12-21 07:42:04 -06:00
pokeeffe-molecula
9dda3ff215
Consistency in error handling (fb-1799) (#2383)
* return 200 once plan compilation starts; if error, return error in response.

* removed some commented out code that is definitely not needed.
2022-12-20 09:32:19 -06:00
Matthew Jaffee
3693b9950a
Expose translation mvcc (#2381)
* expose Transaction on TranslateStore for DAX Snapshotting

* try to fix ramdisk nonsense

apparently, we were running in either a shell env or docker env
randomly, so this could sometimes pass and sometimes fail since the
shell env had the ramdisk set up and docker didn't.

Now we force to run in docker always and set up ramdisk explicitly.

* ramdisk mount should be defined on gitlab runner config now

* debug ramdisk issue?

* fix tests... and a buncha other stuff

Took retry out of CI config because I think it's doing more harm than
good at this point.

The executor test I modified failed when I changed DefaultPartitionN
to 8, but just because stuff was out of order so I made it more
robust.

I edited some data gen stuff to make shorter lines because it was
making grep results unusable.

the actual fix is in translate_boltdb_test.go

* clean up, fix code review feedback
2022-12-20 07:45:32 -06:00
pokeeffe-molecula
021219935f
Implement DELETE (fb 1557) (#2382)
* delete implementation with test coverage

* optimize IN expressions; stop linter complaining

* fixed some uncovered query cases

* skip test in DAX for now
2022-12-19 16:34:28 -06:00
Bruce Baranowski
d19f3e81da
Fb 1818 Implement PREFIX() and SUFFIX() (#2371)
* Implement Prefix and Suffix
* Update substring out-of-index handling
2022-12-19 16:31:27 -05:00
tgruben
72d03602e0
Force make to use bash (#2380) 2022-12-19 13:41:24 -06:00
Matthew Jaffee
1e2e698225 "fix" backup tar test by just comparing lengths not byte for byte 2022-12-17 13:10:46 -06:00
Matthew Jaffee
aeb2ddbf61 update ramdisk mount 2022-12-17 13:10:46 -06:00
Matthew Jaffee
7ab117f5d1 remove in mem translate store
going to try making a ramdisk in CI to run tests
2022-12-17 13:10:46 -06:00
Matthew Jaffee
a58299def4 use separate qcx for write/read in test
now that min/max queries don't use a write Tx it seems we need to
separate read and write in the tests. Not sure I 100% understand this.
2022-12-16 17:05:38 -06:00
Matthew Jaffee
53a8f58e91 set transactions to writable: false for Max/Min/Value calls. these were mistakenly set to true during cleanup 2022-12-16 17:05:38 -06:00
tgruben
6d96a1474e
[FB-1822] Change dataframe disk format to Arrow from parquet (#2376)
* change default backend to arrow file format instead of parquet
2022-12-16 14:42:39 -06:00
Fletcher Haynes
f65367ab46
This fixes a bug with displaying errors returned from the SQL3 endpoint (#2374)
Co-authored-by: Fletcher Haynes <fletcher.haynes@generalassemb.ly>
2022-12-16 08:22:27 -08:00
Lory Cloutier
b1f5264a4b
Fix bulk ingest queries on multi-node databases (#2375)
CLOUD-1252
Implemented Jaffee's fix of checking for b.useShardTransactionalEndpoint
and only running the start/finish transaction block if it's false. Moved
stats timing to a separate defer so it could stay out of the if.
2022-12-15 19:36:53 -06:00
rachithrr
27963441ab
FB-1815: Implement CHAR() (#2369) 2022-12-15 22:04:51 +05:30
Travis Turner
4e8fe488de
Fix PQL distinct in dax (#2360)
* Fix PQL distinct in dax

When issuing a PQL Distinct() call (or any other call with a "index=" arg),
this commit will attempt to convert the value in the index arg with a
TableKeyer.

* Apply change to call.Children as well

* Add some PQL Distinct (join) test coverage
2022-12-15 07:59:20 -06:00
pokeeffe-molecula
15d2ee8b07
add allow_missing_values option to bulk insert (fb-1823) (#2372)
* add allow_missing_values option to bulk insert

* test coverage

* review feedback
2022-12-14 18:32:27 -06:00
pokeeffe-molecula
843312dfc9
handling missing epoch constraint correctly (#2366) 2022-12-14 17:53:17 -06:00
pokeeffe-molecula
3528ec8fc0
enable handling string representations of integers (#2367) 2022-12-14 16:58:18 -06:00
pokeeffe-molecula
c88d60c9ab
SHOW CREATE TABLE issues (fb-1810) (#2365)
* fixed ddl issues with cache type/size; removed shardwidth option; improved error message
2022-12-14 16:27:55 -06:00
Matthew Jaffee
033be81799 rename stupid manager names
ManagerManager -> ResourceManager
Manager -> Resource
2022-12-14 14:23:12 -06:00
Matthew Jaffee
cf1c9dae9a code review tweaks 2022-12-14 14:23:12 -06:00
Matthew Jaffee
4366ad41fb remove version/directive stuff from other snapshot endpoints 2022-12-14 14:23:12 -06:00
Matthew Jaffee
3ddf79160f clean up unused code/comments 2022-12-14 14:23:12 -06:00
Matthew Jaffee
bee666c07c fix dumb issue on storage manager test
changed empty snapshots/writelogs to return nil which was causing NPE
2022-12-14 14:23:12 -06:00
Matthew Jaffee
797b8bc31f fix lint 2022-12-14 14:23:12 -06:00
Matthew Jaffee
87d1c31607 several fixes and debug logging
- check that serverlessStorage is not nil before closing it
- check that we don't already hold a lock on a serverless storage
  Manager before trying to load it. This fixed at least one test failure.
2022-12-14 14:23:12 -06:00
Matthew Jaffee
dbb6d53f9d implement closing on dax, remove all locks when shutting down 2022-12-14 14:23:12 -06:00
Matthew Jaffee
bbaa7dd0f1 more WIP 2022-12-14 14:23:12 -06:00
Matthew Jaffee
35c472a54f extremely WIP 2022-12-14 14:23:12 -06:00
Matthew Jaffee
690a9370e9 remove alpha director (unused) 2022-12-14 14:23:12 -06:00
rachithrr
a4f18fb25f
FB-1812: implement stringsplit() (#2362) 2022-12-15 01:11:29 +05:30