Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some questions around E2E Tests #496

Open
npow opened this issue Dec 28, 2022 · 0 comments
Open

Some questions around E2E Tests #496

npow opened this issue Dec 28, 2022 · 0 comments

Comments

@npow
Copy link

npow commented Dec 28, 2022

Hi, I am trying to write some E2E tests for my PR which adds DynamoDB as a reader/writer. I'm getting a FileNotFoundException, which from my experience means that the files do not exist on HDFS. Looking at the e2e/elasticsearch example, I do not see how the example files are made accessible to the docker containers, or how the tests use the updated JAR with the latest code. So I have the following questions:

  1. How do I make my newly added example files available to the spark-submit image?
  2. How do I run the tests with the updated metorikku.jar file?
- SUBMIT_COMMAND=spark-submit --packages com.audienceproject:spark-dynamodb_2.12:1.1.2 --class com.yotpo.metorikku.Metorikku metorikku.jar -c examples/dynamodb/movies.yaml
  1. I am trying to use dynamodb-local for the tests, but that requires additional configuration options to change the endpoint URL to be local. What is the recommended way to add test specific configs to the examples?

Error snippet:

dynamodb-spark-submit-1  | Exception in thread "main" java.io.FileNotFoundException: File examples/dynamodb/movies.yaml does not exist
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant