featurebase/idk/cmd
jonathan.cheng 43f63b93e3
Cloud 940, skip bad row, enable ingest subsequent rows (#2201)
* first try to skip decodeMessage() error

* force idk to skip a row if there are errors in recordizing

* add comment for removing returning errors from decodemessage()

* added ingest flag SkipBadRows and implementation for skipping Bad Rows (errors that come from recordizer)

* added some comments

* removed comment

* made changes as per discussion with jaffee and walter. i hope this works...

* adding unit tests to check functionality implemented for CLOUD-940

* addressing review comments

* changed a variable name in test file

* removed a variable from ingest test file

* testing sonarcloud failure

* drop spurious second sonar-scanner call

We call sonar-scanner on the IDK data, and then we change
into the IDK directory and try to run it again on the same files,
which don't exist.

* abandon idk change detection for now

the "changes" rule appears not to be good at detecting changes
in some cases. specifically, it appears that you have to be in
an "only:" clause, not a "rules" clause, to trigger the
merge-specific behavior which checks the entire merge branch
instead of the top commit, but that means that if your last
commit doesn't touch IDK, we don't run IDK tests, and I haven't
been able to fix this yet.

So for now, revert the IDK-specific change detection behavior,
which slows CI down but gets us test coverage.

* fix path references

we had three tests all creating idk_coverage.out, then we tried
to grab all files named coverage.out from the testdata directory.

* refactoring tests to avoid duplication

* reverting changes made for local testing

Co-authored-by: CHIN JUNG CHENG <chengcj@CHINs-MacBook-Pro.local>
Co-authored-by: Pranitha-malae <56414132+Pranitha-malae@users.noreply.github.com>
Co-authored-by: Pranitha-malae <pranitha453@gmail.com>
Co-authored-by: Seebs <seebs@molecula.com>
2022-10-06 16:20:35 -05:00
..
bankgen FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
datagen FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
ingester FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
kafkagen FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
kafkaput FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-csv FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-github FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-kafka FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-kafka-delete FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-kafka-sasl FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-kafka-static FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00
molecula-consumer-kinesis Cloud 940, skip bad row, enable ingest subsequent rows (#2201) 2022-10-06 16:20:35 -05:00
molecula-consumer-sql FB-1597: unifying idk and featurebase (#2160) 2022-07-28 17:23:16 -05:00