* Enable linter: stylecheck
This enabled the stylecheck linter, but excludes some staticchecks for
now. The following are ignored because they will take a bit of time to
address, but the intention is to address them and remove them from the
exclusion list.
ST1000: at least one file in a package should have a package comment
ST1003: golang naming standards
ST1008: error should be returned as the last argument
ST1016: methods on the same type should have the same receiver name
ST1020: comment on exported function
* Address ST1015
For some reason this failed in CI but not locally. I can't figure out
why that check isn't happening locally. This just moves the switch
statements around so that the `default` is the first (or last) item.
* Adjust error string in test to match case-adjusted error
* Remove TestCloseTimeout
* Add (commented out) linters that we should introduce
I went through the available linters and added (commented out) the ones
I think we should work on in the near term. In other words, fix them,
then uncomment them so they are enabled in CI.
* linter: errchkjson
* linter: ineffassign
* linter: gosimple
* linter: errname
* Add kafka support to CLI (fbsql)
This commit adds the ability to provide a `--kafka-config` command line
argument referncing a toml file to configure kafka.
* Move "Molecula Consumer" message to the logger; hide it in basic mode
* Fold decimal(scale) into kafka.source-type
* Build fbsql with cgo in docker for CI
* Re-organize the fbsql kafka config and setup.
Allow field config to use the table schema if no fields provided.
* Display timestamp fields with format RFC3339Nano
* remove kafkaRunner (no longer used)
* Fix cli/batch test (and make sure it's not excluded from CI)
The logic in our Makefile was exluding from tests any package with
`/batch` in the package name. This excluded `/cli/batch`, which is not
good.
This commit changes the exclusion logic to include the `/v3` portion of
the package name, so `/v3/batch`.
* Rename Basic() to SetBasic()
* tweak a bunch of logging and config
make overall logs less verbose and chatty
1 minute computer check-in interval
3 minute snapshot interval
remove CaptureLogger as it has same functionality as buffer logger
add a WithPrefix to the Logger interface so sub-services can have
different prefixes
* fix some lint
* fix lint... confused why this is coming up now
* first cut at pre-sort command that works on ndjson
* finish pre_sort command for CSV and JSON and add test
* try fixing golangci-lint
* remove some dumb lint checks
* more linter disabling
* take .golangci.yml from previous repo
* go fmt (facepalm)
* remove ioutil to fix lint
* adding kafka consumer config options (--kafka-max-poll-interval, --kafka-session-timeout, --kafka-group-instance-id, --kafka-socket-keepalive-enable, and --consumer-close-timeout)
* wrapping consumer.Close() in timeout. Will wait consumer-close-timeout seconds before forcing consumer to exit
* clean up logs
(cherry picked from commit 17cdc58d80)
* use t.Fatal(f) to abort tests, not panic
* make perf_able run at all, make it debug a bit better
switch perf-able to using same node type we use for other spot instances,
because otherwise it never finds any available capacity.
we switch the perf-able script to use the standard get_value function
instead of direct jq calls.
we try to grab server logs if the restore fails in the hopes of finding
out why the restore very occasionally fails.
* Fix some issues with running IDK tests in docker. (#2248)
*Stop running TestKafkaSourceIntegration with t.Parallel()
This test can't be run in parallel as it's currently written. Doing so
allows for interleaving of messages to the same kafka topic between
tests.
I didn't attempt to modify the test so it could be run in parallel. That
could be done, but left for someone more ambitious.
* Remove idk/testenv/certs which got accidentally committed.
also update .gitignore to include those.
* changes to add bool support in idk (#2240)
* initial changes to add bool support in idk
* modifying some default parameters for testing, will revert them later
* adding support for bool in making fragments function
* boolean values implementation without supporting empty or null values at this point
* Implement bool support in batch using a map (and a slice for nulls) (#2247)
* Implement bool support in batch using a map (and a slice for nulls)
* Keep the PackBools default for now
But set it explicity in the ingest tests which rely on it.
* Modify batch to construct bool update like mutex
The code in API.ImportRoaringShard has a switch statement which causes
bool fields to be handled like mutex fields. This means, that the
viewUpdate.Clear value should only contain data in the first "row" of
the fragment, which it will treat as records to clear for *all* rows.
This makes more sense for mutex fields; for bool fields, there's only
one other row to clear. But since the code is currently handling them
the same, we need to construct viewUpdate.Clear such that it conforms to
that pattern.
This commit also adds a test which covers this logic.
* Remove commented code; revert config for testing
This commit also removes the DELETE_SENTINEL case for non-packed bools,
since that isn't supported anyway.
* Revert default setting
* remove inconsistent type scope
* correcting the logic of string converstion to bool
* resolving an error in a test
* adding tests to cover code related to bool support in batch.go file and interface.go files
* modifying interfaces test
* added one more test case
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
* resolving bool null field ingestion error (#2254)
* resolving bool null field ingestion error
* testing issues
* adding null support for bools
* updating the null bool field ingestion
* trying to resolve issue when ingesting null value for bool type
* adding a clearing support for bool type
* resolving issues with bool null value ingestion
* updating the jwt go package version and removing changes made in docker compose file
* reverting jwt go version
* removing v4 of jwt
* adding a comment in test file to see if sonar cloud accepts this file
* don't obtain stack traces on rbf.Tx creation
We thought stack traces were mildly expensive. We were very wrong.
Due to a complicated issue in the Go runtime, simultaneous requests
for stack traces end up contending on a lock even when they're not
actually contending on any resources. I've filed a ticket in the
Go issue tracker for this:
https://github.com/golang/go/issues/56400
In the mean time: Under some workloads, we were seeing 85% of all
CPU time go into the stack backtraces, of which 81% went into the
contention on those locks. But even if you take away the contention,
that leaves us with 4/19 of all CPU time in our code going into
building those stack backtraces. That's a lot of overhead for a
feature we virtually never use.
We might consider adding a backtrace functionality here, possibly
using `runtime.Callers` which is much lower overhead, and allows us
to generate a backtrace on demand (no argument values available,
but then, we never read those because they're unformatted hex
values), but I don't think it's actually very informative to know
what the stack traces were of the Tx; they don't necessarily reflect
the current state of any ongoing use of the Tx, so we can't necessarily
correlate them to goroutine stack dumps, and so on.
* fb-1729 Enriched Table Metadata (#2255)
enriched metadata for tables
added support for the concept of a table and field owners in metadata; mechanism to derive owner from http request metadata; metadata for table description
* tightened up is/is not null filter expressions (FB-1741) (#2260)
Covers tightening up handling filter expressions that contain is/is not null ops. These filters may have to be translated into PQL calls to be passed to the executor and even though sql3 language supports nullability for any data type, currently only BSI fields are nullable at the storage engine level (there is a ticket to add support for non-BSI field here FB-1689: IS SQL Argument returns incorrect error) so when these fields are used in filter conditions we need to handle BSI and non-BSI fields differently.
* added a test to cover the keyword replace as being synonymous with insert (#2261)
* update molecula references to featurebase (#2262)
Co-authored-by: Seebs <seebs@molecula.com>
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Pranitha-malae <56414132+Pranitha-malae@users.noreply.github.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: pokeeffe-molecula <85502298+pokeeffe-molecula@users.noreply.github.com>
Co-authored-by: Stephanie Yang <stephanie@pilosa.com>
* removes unused filesize function
* removes ioutil usage
* updates ioutil.ReadAll to io.ReadAll
* updates ioutil.TempFile to os.CreateTemp
* updates ioutil.TempDir to os.MkdirTemp
* updates ioutil.ReadAll to os.ReadAll
* update ioutil.WriteFile to os.WriteFile
* updates ioutil.Discard to io.Discard
* updates ioutil.ReadDir to os.ReadDir where applicable
* removes unused code in idk
* creates type to use for context value keys
* replaces assert.Nil with assert.NoError for error checks
