Development

Tools

• Just install the latest Docker Engine

docker version
docker-compose version

Then go through the sections below:

• Setup: How to setup a Git server just like a Github server would run
• Tests: Runs the tests with configuration repo

Setup

• Create a Git server using Docker: ./setup-github-simulator.sh
  • Will build a Docker Image
• Set the validator as hook/pre-receive

Create a Git server using Docker

• Run ./setup-github-simulator.sh to:
  • Create the official Docker image using the version from .env file

$ ./setup-github-simulator.sh

##### INTUIT SPRING CLOUD CONFIG VALIDATOR 1.1.1 #####

Fri Apr 26 03:35:07 PDT 2019

* Setting up a Github Server simulator

==========--------- Building the new docker image -----------==========

* docker-compose build

Building intuit-spring-cloud-config-validator
Step 1/12 : FROM marcellodesales/github-enterprise-prereceive-hook-base as tests
 ---> ef045e4c014b
Step 2/12 : RUN apk add --no-cache py-pip &&     pip2 install coverage
 ---> Using cache
 ---> 2d0cd9185f33
Step 3/12 : COPY requirements.txt /build/requirements.txt
 ---> Using cache
 ---> a87517ad81ef
Step 4/12 : RUN pip2 install -r /build/requirements.txt
 ---> Using cache
 ---> d355c86466b5
Step 5/12 : COPY ./tests /build/tests
 ---> Using cache
 ---> 9b98cc9143e9
Step 6/12 : COPY ./validate_config_files.py /build
 ---> 5420d38892f7
Step 7/12 : RUN coverage run -m unittest discover -v /build/tests
 ---> Running in a62631e7c4f8
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
test_some_yaml_yml_files_are_invalid (test_invalid_yaml_yml_duplicate_keys.InvalidYamlDuplicateKeysTests) ... ok
test_that_validation_index_is_dictionary (test_invalid_yaml_yml_duplicate_keys.InvalidYamlDuplicateKeysTests) ... ok
test_all_matrix_json_files_are_invalid (test_invalid_matrix_json_validation.InvalidMatrixFileTests) ... ok
test_that_validation_index_is_dictionary (test_invalid_matrix_json_validation.InvalidMatrixFileTests) ... ok
test_some_yaml_yml_files_are_invalid (test_invalid_yaml_yml_multi_document_validation.InvalidYamlMultiDocumentFileTests) ... Some Yaml Single documents are invalid
✘ is tests/fixtures/invalid-yaml-configs-duplicate-keys/circle.yml valid? False ERROR: [3:1: duplication of key "machine" in mapping (key-duplicates), 11:1: duplication of key "machine" in mapping (key-duplicates)]
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
The android matrix file is invalid
✔ is tests/fixtures/invalid-matrix-json-column/ttu-android.yml valid? True
✔ is tests/fixtures/invalid-matrix-json-column/application.properties valid? True
✔ is tests/fixtures/invalid-matrix-json-column/ttu-ios.yml valid? True
✔ is tests/fixtures/invalid-matrix-json-column/ttu.yaml valid? True
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Filtering Spring Cloud Config Server's files:  ['**/ok
test_that_validation_index_is_dictionary (test_invalid_yaml_yml_multi_document_validation.InvalidYamlMultiDocumentFileTests) ... ok
test_some_yaml_yml_files_are_invalid (test_invalid_properties_validation.InvalidPropertiesFileTests) ... *.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Some Yaml Multi documents are invalid
✘ is tests/fixtures/invalid-yaml-configs-multiple-documents-per-config/sp_boot_sample-e2e.yml valid? False ERROR: [4:3: syntax error: expected '<document start>', but found '<block mapping start>']
✘ is tests/fixtures/invalid-yaml-configs-multiple-documents-per-config/sp_boot_sample-dev.yml valid? False ERROR: [4:1: syntax error: could not find expected ':']
✔ is tests/fixtures/invalid-yaml-configs-multiple-documents-per-config/application.yml valid? True
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Properties files are invalid without associated values
✔ is tests/fixtures/invalid-properties-files/publisher-onboard_prod.yml valid? True
✔ is tests/fixtures/invalid-properties-files/publisher-onboard_preprod.yml valid? Trueok
test_that_validation_index_is_dictionary (test_invalid_properties_validation.InvalidPropertiesFileTests) ... ok
test_some_yaml_yml_files_are_invalid (test_invalid_yaml_yml_single_document_validation.InvalidYamlSingleDocumentFileTests) ... ok
test_that_validation_index_is_dictionary (test_invalid_yaml_yml_single_document_validation.InvalidYamlSingleDocumentFileTests) ... ok
test_all_properties_are_valid (test_all_valid_config_validation.AllSuccessfulTests) ...
✘ is tests/fixtures/invalid-properties-files/publisher.properties valid? False ERROR: local variable 'wspacere' referenced before assignment
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Some Yaml Single documents are invalid
✔ is tests/fixtures/invalid-yaml-configs-single-documents/publisher-prd.yml valid? True
✘ is tests/fixtures/invalid-yaml-configs-single-documents/publisher-onboard_preprod.yml valid? False ERROR: [5:4: syntax error: mapping values are not allowed here]
✔ is tests/fixtures/invalid-yaml-configs-single-documents/publisher-qal.yml valid? True
✔ is tests/fixtures/invalid-yaml-configs-single-documents/publisher.properties valid? True
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Filtering Spring Cloud Config Server's files: ok
test_that_validation_index_is_dictionary (test_all_valid_config_validation.AllSuccessfulTests) ... ok

----------------------------------------------------------------------
Ran 12 tests in 8.079s

OK
 ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
All config files are valid
✔ is tests/fixtures/all-valid-config/publisher.properties valid? True
✔ is tests/fixtures/all-valid-config/application-multi-documents.yml valid? True
✔ is tests/fixtures/all-valid-config/publisher-onboard_preprod.yml valid? True
Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
Removing intermediate container a62631e7c4f8
 ---> 20fb0583c64e
Step 8/12 : FROM marcellodesales/github-enterprise-prereceive-hook-base as runtime
 ---> ef045e4c014b
Step 9/12 : RUN apk add --no-cache py-pip
 ---> Using cache
 ---> 8a9b276198d6
Step 10/12 : COPY --from=tests /build/requirements.txt requirements.txt
 ---> Using cache
 ---> e0f7d7512daf
Step 11/12 : RUN pip2 install -r requirements.txt
 ---> Using cache
 ---> fd3bde0287ee
Step 12/12 : COPY --from=tests /build/validate_config_files.py /home/git/test.git/hooks/pre-receive
 ---> a19b36a5b483
Successfully built a19b36a5b483
Successfully tagged intuit/intuit-spring-cloud-config-validator:1.1.1

==========--------- Start the data container -----------==========

* docker run --name data intuit/intuit-spring-cloud-config-validator:1.1.1 /bin/true


==========--------- Start a new Github Server simulator -----------==========

* docker run --name gitrepo -e GITHUB_PULL_REQUEST_HEAD=00000000 -e GITHUB_PULL_REQUEST_BASE=f394274 -d -p 52311:22 -e GIT_DIR=/home/git/test.git intuit/intuit-spring-cloud-config-validator:1.1.1

gitrepo
gitrepo
c37899566502c56a5524bec6760d9442a649ba372d2a50e880b84aa5ff91cdb0

==========--------- Copying credentials from Github server Container -----------==========

* docker cp gitrepo:/home/git/.ssh/id_rsa /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server


Fri Apr 26 03:35:23 PDT 2019

* Finished setting up the server...
* Now you can run tests with test.sh copying the command below:

GITHUB_SIMULATOR_SSH_PATH=/Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server ./test.sh

Testing

• Now that you have the setup done, use the command-line provided to run the tests

##### INTUIT SPRING CLOUD CONFIG VALIDATOR #####

Fri Apr 26 03:35:31 PDT 2019

* Executing the validations in local Github repo

==========--------- Github Config Repo -----------==========

* Current Dir: /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/repo-test-msaas-2

total 72
drwxr-xr-x  13 mdesales  CORP\Domain Users   416 Apr 26 02:51 .
drwxr-xr-x  18 mdesales  CORP\Domain Users   576 Apr 26 03:35 ..
drwxr-xr-x  12 mdesales  CORP\Domain Users   384 Apr 26 03:21 .git
-rw-r--r--   1 mdesales  CORP\Domain Users    33 Apr 26 02:50 README.md
-rw-r--r--   1 mdesales  CORP\Domain Users   243 Apr 26 02:50 application.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   494 Apr 26 02:50 config_msaas_test_01-dev.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   494 Apr 26 02:50 config_msaas_test_01-e2e.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   494 Apr 26 02:50 config_msaas_test_01-prd.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   494 Apr 26 02:50 config_msaas_test_01-prf.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   494 Apr 26 02:50 config_msaas_test_01-qal.yml
-rw-r--r--   1 mdesales  CORP\Domain Users   358 Apr 26 02:50 config_msaas_test_01.yml
drwxr-xr-x   5 mdesales  CORP\Domain Users   160 Apr 26 02:50 idps
-rwxr-xr-x   1 mdesales  CORP\Domain Users  2317 Apr 26 02:51 test.sh

==========--------- Setting Test Remote Origin -----------==========

* Adding the git origin to current directory with server at 192.168.86.249

==========--------- Testing Remote Origin -----------==========

* Conectivity with the repo...
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git remote show test

Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
* remote test
  Fetch URL: [email protected]:test.git
  Push  URL: [email protected]:test.git
  HEAD branch: (unknown)

==========--------- Testing Remote Origin -----------==========

* Executing 'git push' to test github simulator
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git push -u test master
Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (18/18), 11.19 KiB | 11.19 MiB/s, done.
Total 18 (delta 0), reused 18 (delta 0)
remote: #####################################################
remote: fatal: Not a valid object name HEAD
remote: ### Intuit Spring Cloud Config Validator 1.1.1 ####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=5f2acca1293550dc4ecc7aab26e77c52d74f15aa ref=refs/heads/master
remote: BEFORE HEAD...
remote: => Validating SHA 5f2acca1293550dc4ecc7aab26e77c52d74f15aa
remote: Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
To 192.168.86.249:test.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'test'.


* Testing additional branch new-branch
GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 52311 -i /Users/mdesales/dev/github/public/intuit/intuit-spring-cloud-config-validator/.id_rsa_from_github_simulator_server" git push test new-branch -f
Warning: Permanently added '[192.168.86.249]:52311' (ECDSA) to the list of known hosts.
Total 0 (delta 0), reused 0 (delta 0)
remote: #####################################################
remote: ### Intuit Spring Cloud Config Validator 1.1.1 ####
remote: #####################################################
remote: Validating new branch...
remote: Processing commit=5f2acca1293550dc4ecc7aab26e77c52d74f15aa ref=refs/heads/new-branch
remote: BEFORE HEAD...
remote: => Validating SHA 5f2acca1293550dc4ecc7aab26e77c52d74f15aa
remote: Filtering Spring Cloud Config Server's files:  ['**/*.json', '**/*.yaml', '**/*.yml', '**/*.properties']
remote: (v) File config_msaas_test_01-dev.yml is valid!
remote: (v) File config_msaas_test_01-qal.yml is valid!
remote: (v) File idps/idps.yaml is valid!
remote: (v) File application.yml is valid!
remote: (v) File config_msaas_test_01.yml is valid!
remote: (v) File config_msaas_test_01-e2e.yml is valid!
remote: (v) File config_msaas_test_01-prd.yml is valid!
remote: (v) File config_msaas_test_01-prf.yml is valid!
To 192.168.86.249:test.git
 * [new branch]      new-branch -> new-branch