consume .netrc from get step to tasks step

[gowrisankar22]

Hello All,
I have a git resource something like this. I am using two different git servers as you can see in the git resource below.

resources:
- name: landscape
  type: git
  check_every: 10m
  icon: github-circle
  source:
    uri: https://github.server1.com/org/repo.git
    username: ((landscape_git_user))
    password: ((landscape_git_password))
    skip_ssl_verification: true
    branch: ((landscape_git_branch))
    private_key: ((landscape_git_pkey))
    submodule_credentials:
    - host: https://github.server2.com/org/repo.git
      username: ((submdoule_git_username))
      password: ((submdoule_git_password))

When add above git-resource to my pipeline job to a get step it creates a .netrc file inside the get step container.

netrc file:

machine github.server2.com login test1user password xxxxxxxx
default login test2user password xxxx

Actually I want to copy this file to the task step or else want to consume this via input_vars to define a variable(git URI, password, username, etc..) that will be available to the container so that one my other script can use this variable to do a git clone kind of an operation without prompting for a password.

@evanchaoli
Any idea how can I achieve this with help of input_vars?
concourse/rfcs#42

BRs, Gowrisankar

[jamieklassen]

definitely sounds like the examples in https://concourse-ci.org/jobs.html#load-var-step would help you

[gowrisankar22]

@pivotal-jamie-klassen @evanchaoli
get step for git-resource basically stores the credentials inside this path by default ~/.netrc or /root/netrc

bash-5.0# cat /root/.
./          ../         .gitconfig  .netrc

Can this be used to a job task example like this ??

- load_var: local-var
    file: ~/.netrc
    format: raw
      or 
- load_var: local-var
    file: /root/netrc
    format: raw

[gowrisankar22]

@pivotal-jamie-klassen @evanchaoli @vito
Also for the git resource, I have added create_files parameter but this is not working. I get
file 'data.json' does not specify where the file lives

resources:
- name: landscape
  type: git
  check_every: 10m
  icon: github-circle
  source:
    uri: ((landscape_git_repository))
    username: ((landscape_git_user))
    password: ((landscape_git_password))
    skip_ssl_verification: true
    branch: ((landscape_git_branch))
    private_key: ((landscape_git_pkey))
    ignore_paths: [ state/* ]
    submodule_credentials:
    - host: ((submodule_git_host))
      username: ((submdoule_git_username))
      password: ((submdoule_git_password))
    create_files:
      data.json: |
        {"repo": "((landscape_git_repository))"}
jobs:
- name: use-var
  plan:
  - get: landscape
  - load_var: data
    file: landscape/data.json
    format: json
  - task: use-vars
    params:
      URL: ((.:data.repo))
    config:
      platform: linux

      image_resource:
        type: mock
        source: {mirror_self: true}

      inputs:
      - name: landscape

      run:
        path: sh
        args:
        - -exc
        - |
          echo "$URL"

looks like git resoure type is not supported. I have to go with mock resource type only ?

[jamieklassen]

yeah, create_files is only on the mock-resource https://github.com/concourse/mock-resource/blob/master/types.go#L21-L25. the git resource doesn't put anything from its home directory in the output folder.

To get the git resource's out script to put its .netrc file in the output directory, a change would need to be made to the resource type itself. The current behaviour is that this is local to the get container, and I think this is sensible - it's full of private credentials. you could fork the git resource and add a param, say, that caused it to copy its .netrc to the output folder. Or you could write a reusable task that does something like this.

Why are you attached to using load_var? Why can't you have a bunch of params on your task step that read the values you need from a credential manager same as your git resource, like

- task: use-vars
  params:
    REPO_USERNAME: ((landscape_git_user))
    REPO_PASSWORD: ((landscape_git_password))
    SUBMODULE_USERNAME: ((submdoule_git_username))
    SUBMODULE_PASSWORD: ((submdoule_git_password))

You probably just want a convenient .netrc that you can stick in your home directory and have your git operations "just work". If I were you, I would extract a reusable task that creates that file, like this:

params:
  USERNAME:
  PASSWORD:
  SUBMODULE_HOST:
  SUBMODULE_USERNAME:
  SUBMODULE_PASSWORD:
outputs:
- name: output
run:
  path: sh
  args:
  - -c
  - |
    echo "machine $SUBMODULE_HOST login $SUBMODULE_USERNAME password $SUBMODULE_PASSWORD" >> "output/.netrc"
    echo "default login $USERNAME password $PASSWORD" >> "output/.netrc"

[gowrisankar22]

@pivotal-jamie-klassen
Consider we are setting the pipeline for the first time locally using fly-set command by passing all the required credentials and after that, we are updating the pipeline automatically via concourse itself. In this situation, we have to store the credentials somehow in the plain text format and have. to pass it via --load-vars-from . This was not the case with git-resource only for the first we require to pass it via --load-vars-from . and subsequent builds not required. so that we dont want to store the credentails any where as plain text. any idea this use case can be achieved ?

[jamieklassen]

If you're concerned about the safety of your credentials, it sounds like maybe you should be using a credential manager -- is that an option?

[jamieklassen]

I get that you're setting a pipeline with some creds initially. There's nothing stopping you from statically providing those creds to the task I've described as well, so that your pipeline could look something like:

resources:
- name: landscape
  type: git
  source:
    uri: ((landscape_git_repository))
    username: ((landscape_git_user))
    password: ((landscape_git_password))
    submodule_credentials:
    - host: ((submodule_git_host))
      username: ((submdoule_git_username))
      password: ((submdoule_git_password))
jobs:
- name: use-var
  plan:
  - get: landscape
  - task: extract-creds
    file: place/you/checked/in/my/task/yml/above # or you can inline the task config
    params:
      USERNAME: ((landscape_git_user))
      PASSWORD: ((landscape_git_password))
      SUBMODULE_HOST: ((submodule_git_host))
      SUBMODULE_USERNAME: ((submdoule_git_username))
      SUBMODULE_PASSWORD: ((submdoule_git_password))

And I've gotta say your statement that

we dont want to store the credentails any where as plain text

doesn't really apply here - if you're using --load-vars-from than anybody who can hit your GetPipeline endpoint can read your creds in plain text, and the credentials are already in plain text inside the resource get container. Maybe I'm misunderstanding what you mean here - what exactly do you mean by

after that, we are updating the pipeline automatically via concourse itself

If I'm understanding correctly, though, I think using a credential manager is the only reasonable solution to this problem.

[gowrisankar22]

@pivotal-jamie-klassen @evanchaoli yup vault can do but currently, I am trying to achieve with available concourse feature.. If am able to read the .netrc file from get step then my issue is more or less over.

I have changed assets/in script and I was able to achieve the thing very cool. Now I would like to submit a PR. Can you have look concourse/git-resource#322. ?

@@ -186,6 +186,11 @@ git --no-pager log -1 --pretty=format:"%ae" > .git/committer
 # pulled ref in following tasks and resources.
 echo "${return_ref}" > .git/ref

+if [ -f ~/.netrc ]; then
+   echo "copying netrc file to out dir"
+   cp ~/.netrc .
+fi
+

[jamieklassen]

Not really sure what

I am trying to achieve with available concourse feature

means. Concourse integrates with vault: https://concourse-ci.org/vault-credential-manager.html. Do you mean that you don't have access to a vault server?

[gowrisankar22]

@pivotal-jamie-klassen Actually we are yet to integrate with vault meanwhile I was exploring other options as well.

Also, is there any way to get multi branches cloned via git-resource.
Say for example : I have master branch and one remote branch. My git credentials are stored in remote branch and my pipeline trigger is via master branch. Currently I have added two different resources one for master and another one for remote to achieve my usecase. In order to eliminate that only I explored load_var feature to copy netrc to out dir from get step to git clone remote from task step..

Is there any option via git-resource for this use case ??

[jamieklassen]

And just in case it ends up being useful, https://github.com/cloudfoundry-community/git-multibranch-resource/ exists... but I'm really not confident that it is designed for what you are trying to do.

[gowrisankar22]

multi-branch-resource I have already it checked and this will not fit for us.

[gowrisankar22]

basically we want to have a multi-branch thing because our repo has a master branch and another remote branch called state branch which stores all the terraform states. We want to have both in place during the execution and respective contents are pushed back to the respective remote branches accordingly.
adding via git-submodule also will not work for us because of our implementation drawback.

[jamieklassen]

why don't you want to have two different resources?

[gowrisankar22]

@pivotal-jamie-klassen Not any specific reasons.. we want to make changes in many places. #concourse/git-resource#322