Commit graph

268 commits

Author SHA1 Message Date
nagamocha3000
575854df8a Make index-key replication more resilient to network failures 2021-09-14 03:12:52 +03:00
Mahesh Arumugam
858f889745 FeatureBase Renaming: changing go.mod module name for featurebase 2021-07-19 09:20:30 -07:00
Kuba Podgórski
46fbde4f0b Update holder.go 2021-06-07 18:41:13 +02:00
Kuba Podgórski
1ffafd70eb Add return to handleCommitIDs 2021-05-26 15:43:21 +02:00
Nia Weiss
f4ba34247f
remove attributes
Attributes are unmaintained and unused.
They have become more of a liability than a benefit.
This change eliminates them from the codebase.
The only user-visible change (assuming that attrs are not used) is that the attrs field will no longer appear in row JSON.
2021-05-14 10:28:08 -04:00
Kuba Podgórski
a79a36232f Write remote available shards to etcd, instead of local file. 2021-05-07 15:33:18 +02:00
Seebs
1a5696fe23 centralize attempts to set/check limits
We check mmap limits, and try to set/increase our open file limits,
and we check the mmap limit when we start the server, and try to set
the open file limit every time we open a holder.

It's useless to do these things more than once, though. We migrate
these things to be run through a sync.Once, which runs all of them
the first time a server starts up, and then thereafter just returns
the error code from that first run. This should make test startup
ever so slightly cheaper, saving us potentially several microseconds,
but also reducing the spamminess of the message.

I've taken out the `sudo ulimit` advice since it's wrong, and the
documentation link is updated to point to our (now private!)
customer documentation.
2021-04-21 13:56:42 -05:00
Kuba Podgórski
ca1cbadb45 Make validateName function public, so other packages and projects (like IDK, Ingester) can re-use it 2021-04-14 13:33:26 +02:00
Alan Bernstein
285d0a0af8 Add log prefix levels 2021-04-12 20:33:39 -05:00
Nia
d5da6e229c
Merge branch 'master' into external-lookup 2021-03-31 08:12:18 -04:00
Nia Weiss
9bc1b23b7e
change "External" DB to "Lookup" DB 2021-03-29 12:54:29 -04:00
Kuba Podgórski
9a02004a4f Move vprint to separate pakage 2021-03-29 14:29:19 +02:00
Nia Weiss
c4aad290ed
address ExternalLookup review comments 2021-03-29 08:22:41 -04:00
Nia Weiss
e9b92e1cd4
add ExternalLookup query 2021-03-25 13:21:28 -04:00
Travis
9e0ea9709a
Create the data directory during "log startup" if it doesn't already exist.
Prior to using etcd for node membership, the data director was created
during the cluster topology setup. Since that no longer exists, we
weren't actually creating the data directory before getting to
logStartup(). So this change ensure that the data directory exists.
2021-03-11 17:31:46 -06:00
Alan Bernstein
8e123b821d Replace null with [] in schema response 2021-03-11 11:23:20 -06:00
Seebs
810f840839 combine field and index translation readers
There's no need to have two different translation readers, a single
reader can handle both partitions and fields at the same time, so we
can combine them. This may not actually change things much but was
a useful step in diagnosing a different problem with translate readers,
and I think it is a minor improvement so I'm preserving the patch
just in case.
2021-03-09 13:37:51 -06:00
Seebs
51744dc77a check for nil translate store while reading translate entries
If we are using replication, we can be a replica translate store for a
partition, which means we start a translate store reader to replicate
data for it. The translation logic does not admit *stopping* the
translate reader, only "resetting" it (stopping and immediately
restarting), so the translate reader just runs until it hits an error
and terminates, which it does even if perhaps it shouldn't. Oops.

Anyway, one potential failure mode is that if you hit timing just
right, you can end up trying to process translation *while* the
index is being closed, and the index can close its translation stores,
and make them all nil, right before we request a store and try to use
it. Another is a similar error, but during the initial startup of the
translate store readers. Either way, we want to error out of the
process cleanly if this happens.

This could also happen during initial creation, perhaps.

We're aborting translation sync on these errors, because otherwise
we'd continue accepting new keys, and then end up with our highest
known key being higher than some keys we missed; this way the next
restart will restart from the last key we have.
2021-03-08 17:04:52 -06:00
Travis
cad78f1d34
remove "Default*" from const names 2021-03-05 16:56:44 -06:00
Travis
5191d0c3f4
remove unnecessary skip check 2021-03-05 16:56:44 -06:00
Travis
4661f3ac08
reorganize the storage backends directory 2021-03-05 16:56:44 -06:00
Travis
4738a818d2
change attributes file ".data" to "column/row-attributes" 2021-03-05 16:56:44 -06:00
Travis
345d076fdf
introduce "fields" directory between index and field 2021-03-05 16:56:44 -06:00
Travis
4e7ec34c6d
change directory ".disco" to "disco" 2021-03-05 16:56:44 -06:00
Travis
7789e24965
introduce "indexes" directory between datadir and index 2021-03-05 16:56:43 -06:00
Nia
d205ea5441
Merge pull request #1442 from niaow/fix-foreignindex-race
Fix a race condition when deferring foreign-index initialization
2021-02-26 14:30:18 -05:00
Nia Weiss
3fbf6993d0
defer cluster messages until startup 2021-02-25 16:01:49 -05:00
Nia Weiss
ca918c187d
fix a race condition when deferring foreign-index initialization 2021-02-25 16:00:56 -05:00
Travis
2bbe1fdde0
remove remaining references to "coordinator" 2021-02-23 17:23:09 -06:00
Travis
8b0f18721e
remove Field.loadMeta() 2021-02-23 10:09:55 -06:00
Travis
d639e228ae
remove Index.saveMeta(). remove support for deleing existence field. 2021-02-23 10:09:55 -06:00
Travis
38d5459e25
convert holder decode* methods to functions 2021-02-23 10:09:55 -06:00
Travis
24c5b654c6
change IndexOptions to reference by value 2021-02-23 10:09:54 -06:00
Travis
e54f7d9a0b
ensure field.CreatedAt is set on loadField 2021-02-15 21:28:20 -06:00
Travis
b5d6c632bf
stop storing a value for views in etcd 2021-02-15 14:39:52 -06:00
Seebs
e2b6912d1c uninvert test for coordinator node / primary field translation node
If we're the primary field translation node, we don't need to set up
translation replication; we only need that if we're *not*. So it
makes sense to test if !IsPrimaryFieldTranslationNode... except that
the test is to determine whether to return early. So it should not
be inverted.
2021-02-15 10:19:03 -06:00
Travis
8f0270acda
adjust openExistenceField() to check on disk first 2021-02-12 20:35:36 -06:00
Kuba Podgórski
2f35b51db8
Fix endpoint tests + change BitDepth type to uint64 2021-02-12 20:35:36 -06:00
Travis
a6297dc48e
WIP: load schema from etcd on holder open; validate indexes, fields, views 2021-02-12 20:32:31 -06:00
Travis
c37424f24f
remove commented code 2021-02-10 11:57:23 -06:00
Travis
1c3b0f364d
implement ApplySchema, LoadSchema, and LoadSchemaMessage 2021-02-10 11:51:52 -06:00
Travis
bcb71b023a
fix misplaced _exists check 2021-02-10 11:47:26 -06:00
Travis
8fe1b9a37c
store views in etcd via Schemator 2021-02-10 11:47:26 -06:00
Kuba Podgórski
9faabd8535 Fix Test_TxFactory_UpdateBlueFromGreen_OnStartup 2021-02-10 14:26:52 +01:00
Kuba Podgórski
40063a6aa9 Fix TestHandler_PostSchemaCluster 2021-02-09 20:13:38 +01:00
Kuba Podgórski
0f97066914 Fix TestAPI_Import. Delete views when deleting a field. 2021-02-09 13:36:53 +01:00
Kuba Podgórski
37b309be16 Move schemator from API to index. Fix TestExecutor_Execute_SetRow 2021-02-08 21:54:39 +01:00
Antonio Navarro Perez
17a5bc51ca Add NopSchemator and NopSerializer
Signed-off-by: Antonio Navarro Perez <antnavper@gmail.com>
2021-02-08 18:36:53 +01:00
Travis
3542134100
WIP: implement Schemator 2021-02-08 10:42:55 -06:00
Travis
114f6a8751
add withViews argument to api.Schema() method 2021-02-08 10:42:55 -06:00