scriptoriumadmin/featurebase
1
0
Fork 0
mirror of https://github.com/featurebasedb/featurebase.git synced 2026-08-28 10:54:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
featurebase/Makefile
pokeeffe-molecula 204558b46f added /sql endpoint; implemented SHOW TABLES (#1935) 
* squashed 45 commits into one :)

* tlt/sql experiment (#2035)

* Move PlanOperator to sql3/planner/types package

includes:
type PlanOperatorColumn struct
type PlanOperator interface

* Remove planner dependencies from pilosa package

The goal after this is to prevent the planner package (which doesn't exist yet)
from being imported by the pilosa package; we just want it injected into the server
in server/server.go. This is because the planner package uses pilosa types, so we need
to avoid circular dependencies.

Added ExecutionPlannerFn
Make public: pilosa.ExecOptions
Added a pilosa.Executor interface
Added a planner.types.CompilePlanner interface
Isolated the planner calls to:
- Executor.Execute()
- *API.[method]()

* Move executionplanner files into the sql3/planner package.

This required a bit of gymnastics, and there are some things around
FieldOptions which need to be addressed soon.

* Remove the hacky FieldOptions stuff I added earlier

This implementation just uses the pilosa.FieldOption functional options
provided by the API (as opposed to trying to build a FieldOptions
object.

It also changes field types to constants. These are private for now, but
if we need to make them public, we should put them in the planner/types
package.

* Implement the "scale" value from Decimal(scale)

Also, precision and scale were currently reversed in the parser. This
fixes that.

* Modify the parser to handle CACHETYPE <type> SIZE <size>

It's a little odd to me that the cache type values are Tokens, but I
guess it's ok. One thing to keep in mind is that FeatureBase expects
lowercase values, so this commit changes the parser to set the value to
the lowercase version of the type.

* Fix the /sql2 tests

This entailed a combination of commenting out or t.Skip()-ing tests
which covered code in the parser that has been commented out or removed
as not currently supported in sql3.

It also adds some coverage for the sql.Contraint stringers.

* Prevent JSON sql results from containing closing commas

This commit just re-works the existing output code to avoid inserting
closing commas (which results in invalid JSON).

* Enhance the CREATE TABLE test coverage.

In particular, ensure that the fields which get created in FeatureBase
are what we expect based on the fields defined in the CREATE TABLE
statement.

This also ensures that the TIMEQUANTUM and CACHETYPE contraints are not
provided for the same field (since those constraints are not supported
together).

* Adjust the EBNF file to indicate SIZE contraint is optional

A CACHETYPE can be provided without a SIZE. This change indicates that
SIZE is optional.

* Remove `executionplanner_` from file names (#2040)

* implementation of ALTER TABLE (sans column RENAME)

* refactored expression analysis; added more robust type checking; all unary and bin ops function on ints

* added type support for expressions; full bin/unary op support; added cast; more literal support

* cast int to all other types

* all literals (except idset, stringset & timestamp) make it thru; cast to all types with int as source now works

* implemented LIKE/NOT LIKE

* Implemented IS [NOT] NULL

* Move sql2 files into sql3/parser package (#2045)

* Move sql2 files into sql3/parser package

This also removes the sql2 package.

* Fix tests which were typing _id fields as INT intead of ID

* implemented BETWEEN, NOT BETWEEN

* Add featurebase/error package (#2046)

* Add featurebase/error package

I copied the `dax/errors` package which I am starting to use in the DAX
prototype into `featurebase/errors` in order to start using it with the
sql3 package. It's basically a wrapper around `github.com/pkg/errors`,
but it uses a customer coded error.

The sql package can define its own errors based on the
`featurebase/errors` types. Then do things like `Wrap()` and `Is()`.

* Address the linter complaints: shadowed variables, unreachable code

* implemented IN & NOT IN with expression lists

* first cut of CASE

* Fixed some errors from rebase

* updated bnf; removed unused code; tightened up error handling

* first crack at basic CLI for SQL3

Use: `featurebase cli`

Still lots to do here, but for example:

> select count(*) from tremor
+--------------+
|        COUNT |
+--------------+
| 1.158321e+06 |
+--------------+

* Iterate on the CLI (#2057)

Handle the errors.
Add an "exit" command.
Add some general formatting and white space.

Add termination character: ";" (semicolon)

This commit allows a user to provide multiple or partial SQL statements.

Example of multiple statements:
```
show tables; select * from foo;
```

Example of partial (multi-line) statements:
```
select *
from foo;
```

Don't uppercase the header values

* error refactoring; first cut of TOP; remove unused code; use log.Printf instead of fmt.Printf

* fixed a bug with QualifiedRef from refactoring; added bones of INSERT; removal of unused code; tightened up errors more; fixed failing tests

* single value list for INSERT

* Update bnf per discussion with Travis; INSERT now doing the requisite stuff

* Pat's eyes went square - nothing wrong with TOP, Pat needed to learn arrays again.

* improved some errors; fixed tests to suit

* send warnings back in the api; update CLI to display warnings

* start warning on stuff not implemented so we don't get bugged about it

* Tlt/sql experiment (#2063)

* Expresssion -> Expression

* Add SQL planner test

- adds a test to which it is easier to add tables and SQL statments
- un-exports all of the expression types
- removes the planner pointer from the expression types (it can be added
  back later if need be)

* Fix where clause on a string field

Prior to this commit, the binary expression for a where clause on a
string field was building the call by providing a range operator which
is typically used for BSI fields. This changes it to use the call.Args
for string values.

* Update planner tests to handle multiple sql for the same results

* Reorganize SQL tests

Introduce a test/helpers package and move shared MustQueryRows into that
package.

* Add a compatibility map for field types. (#2064)

This is primarily to address the fact that ID fields were previously
incompatible with INT literals.

We should probably consider introducing a custom type for FieldType
which can be used to define compatibilities.

* significantly refactored type checking

* Handle nil (NULL) values in the sql CLI. (#2067)

go-pretty panics if the interface{} field value is nil. This replaces
nil values with a "NULL" string.

* Squash some commits

fixed a still failing test

added line, col to all error messages

refactored source handling to enable table aliases

fixed some copypasta per review

warnings for order by & topn; implemented select as a source

starting to handle in (select...); added stub for optimizer

JSON-encode the sql error and warning strings (#2069)

Error strings with unencoded characters (like double quotes) were
resulting in invalid json.

got insert working; added symbol table; added concrete optimizer; added nascent NestedLoopsOperator; rewrite "where foo in (select..." as inner join

* all about the sets (#2085)

* implemented setcontains()

* implemented set literal; insert set column values; setcontains/all/any both in expr eval and pql filters

* Convert test to use latest framework. (#2086)

* fixed some comments

* removed refactored tests

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add support for Decimal fields to the sql test. (#2090)

* dates (#2094)

* return dates as strings in output; tightened up decimal type checking

* return dates as strings in output; tightened up decimal type checking

* fixed failing tests after decimal changes

* can now insert decimal values

* implemented insert for timestamp data type; implemented current_date, current_timestamp constants

* fixed some failing tests

* handle date literals from strings in insert statements

* changes from feedback

* Fix pointer method error

* sql3 API interface (#2110)

* Introduce API-related interfaces: SchemaAPI, ComputeAPI

The sql3 code was relying on the pointer: *pilosa.API in order to call
API methods directly on the local node. If we want to import and use the
sql3 package in another service (the DAX queryer, for example), we need
to be able to use an implementation of an interface for those API method
calls.

This commit introduces two interfaces, both automatically implemented by
pilosa.API:
- SchemaAPI
- ComputeAPI

* Convert sql3 code to use IndexInfo instead of Index

The sql3 code was relying on a *pilosa.Index and its methods to get
general information like index and field name, type, etc. This commit
converts everything to use a *pilosa.IndexInfo instead.

This allows us to modify the SchemaAPI interface to also return
IndexInfo instead of Index, which will be a lot easier to implement in a
non-pilosa package (like DAX); creating a *pilosa.Index requires
providing things like data directory paths and holders, which are not
necessary for these use cases.

* Unary and Binary Ops R US plus CAST (#2111)

* implemented string literal for timestamp epoch

* fixed failing test

* fixed the failing test again

* refactored tests; implemented unary op tests for all datatypes; implemented binop tests for int/int, int/id, int/decimal & ID/int

* implemented all binary ops for INT & all other types, ID & all other types

* implemented binary ops for DECIMAL types & all other types

* added STRING & BOOL to various tests; implemented all remaining binOp tests

* fix up some stuff after rebasing

* refactored test defs into multiple files; implemented CAST for every datatype

* added tests for like/not like

* addressed review feedback

* addressed type review feedback

* tightened up IS [NOT] NULL behavior plus tests (#2118)

* tightened up IS [NOT] NULL behavior plus tests

* BETWEEN/NOT BETWEEN with all data types

* addressed review feedback

* Handle negative integers in column min/max constraints (#2120)

This commit parses the min/max contraint as an expression, as opposed to
an int literal, so that negative values are treated as Unary
expressions.

There currently isn't support for min/max constraints on `decimal`
fiels, so for now this change only expects +/- integer values.

* Implement the CREATE TABLE keypartitions logic (#2123)

* Execution time, IN/NOT IN & multiple aggregates (#2124)

* added display of execution time

* IN/NOT IN tests for all data types

* fixed date parsing

* removed duplicative tests

* refactoring aggregates

* suport multiple aggregates

* Address review feedback

* final round of feedback

* Add method SchemaAPI.CreateIndexAndFields() (#2127)

In order to support a CREATE TABLE statement as a single command, this
commit alters the SchemaAPI interface to contain a single method which
handles both the index and its fields. It also updates the sql3 code to
use this interface instead of CreateIndex() and CreateField()
indepedently.

* Symbol Handling (Again) (#2129)

* Refactored symbol handling in the planner; re-instated the select as source tests

* removed commented out code

* addressing review feedback

* Move hard-coded _id field out of planner and into interface implementation (#2130)

This commit moves the hard-coded addition of the `_id` field from the
planner to the SchemaAPI.IndexInfo() implementation method.

NOTE: If anything was expecting SchemaAPI.Schema() to also return the
`_id` field as part of its field list in each table, then it would not
be there because the `_id` field is only added in the IndexInfo() method
for now. Currently that's not a problem because nothing is expecting the
`_id` field for `Schema()`.

* Multiple aggregates, all aggregates stand alone and in GROUP BY (#2132)

* handle multiple aggregates in group by queries

* added handling for avg() aggregate both stand alone and in group by

* tightened up sum & avg outside of group by

* added min, max & percentile

* added warnings

* Make MaterializedRowSet implement the PlanOperator interface. (#2133)

This commit refactors the PQLMultiGroupByOperator to have a PlanOperator
as its output. Then, when it initializes, it sets up a
MaterializedRowSet and populates that with the values from the multiple
group by operations.

* added explicit min/max pql operators

* saved a file I forgot to save

* per review

* Un-indent some if/else nesting (#2136)

Co-authored-by: Travis Turner <travis@pilosa.com>

* Add optional `name` argument to test structs.

This commit adds the `name` argument to `tableTest` and `sqlTest` so
that a test can be optionally named. This allows a developer to more
easily run/identify a particular test by name.

* Inbuilt functions (redux) (#2141)

* set functions type parameter type checking

* implemented datepart

* include SQL3 type in SHOW COLUMNS output

* fixed select as source; failing SHOW COLUMNS test

* select in select list

* dump output columns; handle optimization for select list subqueries

* make it an error to return multiple rows for a select list subquery

* added description

* contants and test coverage for datepart function

* SQL3 Refactor-palooza (#2182)

* removed unneeded IsAggregate()

* first cut of working nested loops operator aka INNER JOIN

* remove selectListItemPlanExpression

* added some warnings

* all the tests are passing again!

* addressed some linter complaints

* added basic order by

* bug fixes; added 'or replace'/'replace' to insert

* for insert references should return appropriately

* added back ability to use subquery singleton expressions

* removed dead code; fixed test

* json-able plan, Schema() plus refactoring

* fixed dumb code

* add some tests for time quantum behavior

* Code cleanup during review. Also fixed INSERT to keyed table bug.

This commit contains a lot of minor adjustments made during code review.

It also contains a bug fix that was preventing INSERT into a keyed table
(i.e. _id type STRING) from working.

Co-authored-by: Travis Turner <travis@molecula.com>

* Fix expected min/max on timestamp column test (decimal field)

I don't know why this changed, but presumably something to do with
decimal related work that happened on master.

* Fix compile problem after rebase

* review feedback

Co-authored-by: Matthew Jaffee <jaffee@pilosa.com>
Co-authored-by: Travis Turner <travis@pilosa.com>
Co-authored-by: Travis Turner <travis@molecula.com>
Co-authored-by: Fletcher Haynes <fletcher@capitalprawn.com>
2022-09-30 11:10:24 -07:00

384 lines
16 KiB
Makefile
Raw Blame History

.PHONY: build check-clean clean build-lattice cover cover-viz default docker docker-build docker-test docker-tag-push generate generate-protoc generate-pql generate-statik gometalinter install install-build-deps install-golangci-lint install-gometalinter install-protoc install-protoc-gen-gofast install-peg install-statik release release-build test testv testv-race testvsub testvsub-race test-txstore-rbf
CLONE_URL=github.com/pilosa/pilosa
VERSION := $(shell git describe --tags 2> /dev/null || echo unknown)
VARIANT = Molecula
GO=go
GOOS=$(shell $(GO) env GOOS)
GOARCH=$(shell $(GO) env GOARCH)
VERSION_ID=$(if $(TRIAL_DEADLINE),trial-$(TRIAL_DEADLINE)-,)$(VERSION)-$(GOOS)-$(GOARCH)
BRANCH := $(if $(CIRCLE_BRANCH),$(CIRCLE_BRANCH),$(shell git rev-parse --abbrev-ref HEAD))
BRANCH_ID := $(BRANCH)-$(GOOS)-$(GOARCH)
BUILD_TIME := $(shell date -u +%FT%T%z)
SHARD_WIDTH = 20
COMMIT := $(shell git describe --exact-match >/dev/null 2>&1 || git rev-parse --short HEAD)
LDFLAGS="-X github.com/featurebasedb/featurebase/v3.Version=$(VERSION) -X github.com/featurebasedb/featurebase/v3.BuildTime=$(BUILD_TIME) -X github.com/featurebasedb/featurebase/v3.Variant=$(VARIANT) -X github.com/featurebasedb/featurebase/v3.Commit=$(COMMIT) -X github.com/featurebasedb/featurebase/v3.TrialDeadline=$(TRIAL_DEADLINE)"
GO_VERSION=1.19
DOCKER_BUILD= # set to 1 to use `docker-build` instead of `build` when creating a release
BUILD_TAGS += shardwidth$(SHARD_WIDTH)
TEST_TAGS = roaringparanoia
UNAME := $(shell uname -s)
TEST_TIMEOUT=30m
RACE_TEST_TIMEOUT=90m
ifeq ($(UNAME), Darwin)
IS_MACOS:=1
else
IS_MACOS:=0
endif
export GO111MODULE=on
export GOPRIVATE=github.com/molecula
export CGO_ENABLED=0
# Run tests and compile Pilosa
default: test build
# Remove build directories
clean:
rm -rf vendor build
rm -f *.rpm *.deb
# Set up vendor directory using `go mod vendor`
vendor: go.mod
$(GO) mod vendor
version:
@echo $(VERSION)
# We build a list of packages that omits the IDK packages because the IDK
# packages require fancy environment setup.
GOPACKAGES := $(shell $(GO) list ./... | grep -v "/idk")
# Run test suite
test:
$(GO) test $(GOPACKAGES) -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -v -timeout $(TEST_TIMEOUT)
# Run test suite with race flag
test-race:
CGO_ENABLED=1 $(GO) test $(GOPACKAGES) -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -race -timeout $(RACE_TEST_TIMEOUT) -v
testv: testvsub
testv-race: testvsub-race
# testvsub: run go test -v in sub-directories in "local mode" with incremental output,
# avoiding go -test ./... "package list mode" which doesn't give output
# until the test run finishes. Package list mode makes it hard to
# find which test is hung/deadlocked.
#
testvsub:
@set -e; for pkg in $(GOPACKAGES); do \
if [ $${pkg:0:38} == "github.com/featurebasedb/featurebase/v3/idk" ]; then \
echo; echo "___ skipping subpkg $$pkg"; \
continue; \
fi; \
echo; echo "___ testing subpkg $$pkg"; \
$(GO) test -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -v -timeout $(RACE_TEST_TIMEOUT) $$pkg || break; \
echo; echo "999 done testing subpkg $$pkg"; \
done
testvsub-race:
@set -e; for pkg in $(GOPACKAGES); do \
echo; echo "___ testing subpkg $$pkg"; \
CGO_ENABLED=1 $(GO) test -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -v -race -timeout $(RACE_TEST_TIMEOUT) $$pkg || break; \
echo; echo "999 done testing subpkg $$pkg"; \
done
bench:
$(GO) test $(GOPACKAGES) -bench=. -run=NoneZ -timeout=127m $(TESTFLAGS)
# Run test suite with coverage enabled
cover:
mkdir -p build
$(MAKE) test TESTFLAGS="-coverprofile=build/coverage.out"
# Run test suite with coverage enabled and view coverage results in browser
cover-viz: cover
$(GO) tool cover -html=build/coverage.out
# Compile Pilosa
build:
$(GO) build -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/featurebase
# Create a single release build under the build directory
release-build:
$(MAKE) $(if $(DOCKER_BUILD),docker-)build FLAGS="-o build/featurebase-$(VERSION_ID)/featurebase"
cp NOTICE install/featurebase.conf install/featurebase*.service build/featurebase-$(VERSION_ID)
tar -cvz -C build -f build/featurebase-$(VERSION_ID).tar.gz featurebase-$(VERSION_ID)/
@echo Created release build: build/featurebase-$(VERSION_ID).tar.gz
test-release-build: docker-build
mv build/featurebase-$(VERSION_ID).tar.gz install/
cd install && docker build -t featurebase:test_installation \
-f test_installation.Dockerfile \
--build-arg release_tarball=featurebase-$(VERSION_ID).tar.gz .
mv install/featurebase-$(VERSION_ID).tar.gz build/
docker run -it -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
featurebase:test_installation
# Error out if there are untracked changes in Git
check-clean:
ifndef SKIP_CHECK_CLEAN
$(if $(shell git status --porcelain),$(error Git status is not clean! Please commit or checkout/reset changes.))
endif
# Create release build tarballs for all supported platforms. DEPRECATED: Use `docker-release`
release: check-clean generate-statik-docker
$(MAKE) release-build GOOS=darwin GOARCH=amd64
$(MAKE) release-build GOOS=darwin GOARCH=arm64
$(MAKE) release-build GOOS=linux GOARCH=amd64
$(MAKE) release-build GOOS=linux GOARCH=arm64
# Create release build tarballs for all supported platforms. Same as `release`, but without embedded Lattice UI.
release-sans-ui: check-clean
rm -f statik/statik.go
$(MAKE) release-build GOOS=darwin GOARCH=amd64
$(MAKE) release-build GOOS=darwin GOARCH=arm64
$(MAKE) release-build GOOS=linux GOARCH=amd64
$(MAKE) release-build GOOS=linux GOARCH=arm64
package:
go build -o featurebase ./cmd/featurebase
GOARCH=$(GOARCH) VERSION=$(VERSION) nfpm package --packager deb --target featurebase.$(VERSION).$(GOARCH).deb
GOARCH=$(GOARCH) VERSION=$(VERSION) nfpm package --packager rpm --target featurebase.$(VERSION).$(GOARCH).rpm
# We allow setting a custom docker-compose "project". Multiple of the
# same docker-compose environment can exist simultaneously as long as
# they use different projects (the project name is prepended to
# container names and such). This is useful in a CI environment where
# we might be running multiple instances of the tests concurrently.
PROJECT ?= clustertests
DOCKER_COMPOSE = docker-compose -p $(PROJECT)
# Run cluster integration tests using docker. Requires docker daemon to be
# running and docker-compose to be installed.
clustertests: vendor
$(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml down
$(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml build
$(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml up -d pilosa1 pilosa2 pilosa3
PROJECT=$(PROJECT) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml run client1
$(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml down
# Run the cluster tests with authentication enabled
AUTH_ARGS="-c /go/src/github.com/featurebasedb/featurebase/internal/clustertests/testdata/featurebase.conf"
authclustertests: vendor
CLUSTERTESTS_FB_ARGS=$(AUTH_ARGS) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml down
CLUSTERTESTS_FB_ARGS=$(AUTH_ARGS) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml build
CLUSTERTESTS_FB_ARGS=$(AUTH_ARGS) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml up -d pilosa1 pilosa2 pilosa3
PROJECT=$(PROJECT) ENABLE_AUTH=1 $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml run client1
CLUSTERTESTS_FB_ARGS=$(AUTH_ARGS) $(DOCKER_COMPOSE) -f internal/clustertests/docker-compose.yml down
# Install Pilosa
install:
$(GO) install -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/featurebase
# Install the single-node PLG version of FeatureBase
plg:
$(GO) build -tags='plg $(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/featurebase
install-bench:
$(GO) install -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/pilosa-bench
# Build the lattice assets
build-lattice:
docker build -t lattice:build ./lattice
export LATTICE=`docker create lattice:build`; docker cp $$LATTICE:/lattice/. ./lattice/build && docker rm $$LATTICE
# `go generate` protocol buffers
generate-protoc: require-protoc require-protoc-gen-gofast
$(GO) generate github.com/featurebasedb/featurebase/v3/pb
# `go generate` statik assets (lattice UI)
generate-statik: build-lattice require-statik
$(GO) generate github.com/featurebasedb/featurebase/v3/statik
# `go generate` statik assets (lattice UI) in Docker
generate-statik-docker: build-lattice
docker run --rm -t -v $(PWD):/pilosa golang:1.15.8 sh -c "go get github.com/rakyll/statik && /go/bin/statik -src=/pilosa/lattice/build -dest=/pilosa -f"
# `go generate` stringers
generate-stringer:
$(GO) generate github.com/featurebasedb/featurebase/v3
generate-pql: require-peg
cd pql && peg -inline pql.peg && cd ..
generate-proto-grpc: require-protoc require-protoc-gen-go
protoc -I proto proto/pilosa.proto --go_out=plugins=grpc:proto
protoc -I proto proto/vdsm/vdsm.proto --go_out=plugins=grpc:proto
# TODO: Modify above commands and remove the below mv if possible.
# See https://go-review.googlesource.com/c/protobuf/+/219298/ for info on --go-opt
# I couldn't get it to work during development - Cody
cp -r proto/github.com/featurebasedb/featurebase/v3/proto/ proto/
rm -rf proto/github.com
# `go generate` all needed packages
generate: generate-protoc generate-statik generate-stringer generate-pql
# Create release using Docker
docker-release:
$(MAKE) docker-build GOOS=linux GOARCH=amd64
$(MAKE) docker-build GOOS=linux GOARCH=arm64
$(MAKE) docker-build GOOS=darwin GOARCH=amd64
$(MAKE) docker-build GOOS=darwin GOARCH=arm64
# Build a release in Docker
docker-build: vendor
docker build \
--build-arg GO_VERSION=$(GO_VERSION) \
--build-arg MAKE_FLAGS="TRIAL_DEADLINE=$(TRIAL_DEADLINE) GOOS=$(GOOS) GOARCH=$(GOARCH)" \
--target pilosa-builder \
--tag featurebase:build .
docker create --name featurebase-build featurebase:build
mkdir -p build/featurebase-$(VERSION_ID)
docker cp featurebase-build:/pilosa/build/. ./build/featurebase-$(VERSION_ID)
cp NOTICE install/featurebase.conf install/featurebase*.service ./build/featurebase-$(VERSION_ID)
docker rm featurebase-build
tar -cvz -C build -f build/featurebase-$(VERSION_ID).tar.gz featurebase-$(VERSION_ID)/
# Create Docker image from Dockerfile
docker-image: vendor
docker build \
--build-arg GO_VERSION=$(GO_VERSION) \
--build-arg MAKE_FLAGS="TRIAL_DEADLINE=$(TRIAL_DEADLINE)" \
--tag featurebase:$(VERSION) .
@echo Created docker image: featurebase:$(VERSION)
# Create docker image (alias)
docker: docker-image # alias
# Tag and push a Docker image
docker-tag-push: vendor
docker tag "featurebase:$(VERSION)" $(DOCKER_TARGET)
docker push $(DOCKER_TARGET)
@echo Pushed docker image: $(DOCKER_TARGET)
# These commands (docker-idk and docker-idk-tag-push)
# are designed to be used in CI.
# docker-idk builds idk docker images and tags them - intended for use in CI.
docker-idk: vendor
docker build \
-f idk/Dockerfile \
--build-arg GO_VERSION=$(GO_VERSION) \
--build-arg MAKE_FLAGS="GOOS=$(GOOS) GOARCH=$(GOARCH) BUILD_CGO=$(BUILD_CGO)" \
--tag registry.gitlab.com/molecula/featurebase/idk:$(VERSION_ID) .
@echo Created docker image: registry.gitlab.com/molecula/featurebase/idk:$(VERSION_ID)
# docker-idk-tag-push pushes tagged docker images to the GitLab container
# registry - intended for use in CI.
docker-idk-tag-push:
docker push registry.gitlab.com/molecula/featurebase/idk:$(VERSION_ID)
@echo Pushed docker image: registry.gitlab.com/molecula/featurebase/idk:$(VERSION_ID)
# Install diagnostic pilosa-keydump tool. Allows viewing the keys in a transaction-engine directory.
pilosa-keydump:
$(GO) install -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/pilosa-keydump
# Install diagnostic pilosa-chk tool for string translations and fragment checksums.
pilosa-chk:
$(GO) install -tags='$(BUILD_TAGS)' -ldflags $(LDFLAGS) $(FLAGS) ./cmd/pilosa-chk
pilosa-fsck:
cd ./cmd/pilosa-fsck && make install && make release
# Run Pilosa tests inside Docker container
docker-test:
docker run --rm -v $(PWD):/go/src/$(CLONE_URL) -w /go/src/$(CLONE_URL) golang:$(GO_VERSION) go test -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -timeout $(TEST_TIMEOUT) $(GOPACKAGES)
# Must use bash in order to -o pipefail; otherwise the tee will hide red tests.
# run top tests, not subdirs. print summary red/green after.
# The \-\-\- FAIL avoids counting the extra two FAIL strings at then bottom of log.topt.
topt:
mv log.topt.roar log.topt.roar.prev || true
$(eval SHELL:=/bin/bash) set -o pipefail; $(GO) test -v -timeout $(RACE_TEST_TIMEOUT) -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) 2>&1 | tee log.topt.roar
@echo " log.topt.roar green: \c"; cat log.topt.roar | grep PASS |wc -l
@echo " log.topt.roar red: \c"; cat log.topt.roar | grep '\-\-\- FAIL' | wc -l
topt-race:
mv log.topt.race log.topt.race.prev || true
$(eval SHELL:=/bin/bash) set -o pipefail; CGO_ENABLED=1 $(GO) test -race -timeout $(RACE_TEST_TIMEOUT) -v -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) 2>&1 | tee log.topt.race
@echo " log.topt.race green: \c"; cat log.topt.race | grep PASS |wc -l
@echo " log.topt.race red: \c"; cat log.topt.race | grep '\-\-\- FAIL' | wc -l
# Run golangci-lint
golangci-lint: require-golangci-lint
golangci-lint run --timeout 3m --skip-files '.*\.peg\.go'
# Alias
linter: golangci-lint
# Better alias
ocd: golangci-lint
# Run gometalinter with custom flags
# Note the "./..." in gometalinter is still allowed, because we do want
# linting to reach IDK pagkages.
gometalinter: require-gometalinter vendor
GO111MODULE=off gometalinter --vendor --disable-all \
--deadline=300s \
--enable=deadcode \
--enable=gochecknoinits \
--enable=gofmt \
--enable=goimports \
--enable=gotype \
--enable=gotypex \
--enable=ineffassign \
--enable=interfacer \
--enable=maligned \
--enable=misspell \
--enable=nakedret \
--enable=staticcheck \
--enable=unconvert \
--enable=unparam \
--enable=vet \
--exclude "^internal/.*\.pb\.go" \
--exclude "^pql/pql.peg.go" \
./...
######################
# Build dependencies #
######################
# Verifies that needed build dependency is installed. Errors out if not installed.
require-%:
$(if $(shell command -v $* 2>/dev/null),\
$(info Verified build dependency "$*" is installed.),\
$(error Build dependency "$*" not installed. To install, try `make install-$*`))
install-build-deps: install-protoc-gen-gofast install-protoc install-statik install-stringer install-peg
install-statik:
go install github.com/rakyll/statik@latest
install-stringer:
GO111MODULE=off $(GO) get -u golang.org/x/tools/cmd/stringer
install-protoc-gen-gofast:
GO111MODULE=off $(GO) get -u github.com/gogo/protobuf/protoc-gen-gofast
install-protoc-gen-go:
GO111MODULE=off $(GO) get -u github.com/golang/protobuf/protoc-gen-go
install-protoc:
@echo This tool cannot automatically install protoc. Please download and install protoc from https://google.github.io/proto-lens/installing-protoc.html
@echo On mac, brew install protobuf seems to work.
@echo As of the commit that added this line, protoc-gen-gofast was at 226206f39bd7, and the protoc version in use was:
@echo $$ protoc --version
@echo libprotoc 3.19.4
install-peg:
GO111MODULE=off $(GO) get github.com/pointlander/peg
install-golangci-lint:
GO111MODULE=off $(GO) get github.com/golangci/golangci-lint/cmd/golangci-lint
install-gometalinter:
GO111MODULE=off $(GO) get -u github.com/alecthomas/gometalinter
GO111MODULE=off gometalinter --install
GO111MODULE=off $(GO) get github.com/remyoudompheng/go-misc/deadcode
test-external-lookup:
$(GO) test . -tags='$(BUILD_TAGS) $(TEST_TAGS)' $(TESTFLAGS) -run ^TestExternalLookup$$ -externalLookupDSN $(EXTERNAL_LOOKUP_DSN)
bnf:
ebnf2railroad --no-overview-diagram --no-optimizations ./sql3/sql3.ebnf
Reference in a new issue View git blame Copy permalink