Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Action all the sudden reporting job status using "(merge)" suffix #805

Open
mellis481 opened this issue Feb 20, 2023 · 25 comments
Open

Action all the sudden reporting job status using "(merge)" suffix #805

mellis481 opened this issue Feb 20, 2023 · 25 comments
Labels
existing workaround triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. unexpected behavior User expected result, but got another

Comments

@mellis481
Copy link

I have a number of workflow jobs that use cypress-io/github-action which I have tied to branch protection status checks. Here is an example of a job:

- name: Cypress end-to-end tests
  uses: cypress-io/github-action@v4
  with:
    wait-on: 'https://localhost:9000/index.js'
    start: npm run start:local
    config-file: cypress/config/e2e.ts
    record: true
    install: false
    group: 'E2E'
    ci-build-id: ${{ needs.prepare.outputs.uuid }}
    tag: 'pull-request'

The name of the status check that I previously set for this job is "cypress: E2E" which uses the group property of the action. This has been working fine. All of the sudden, however, the previous status checks are not reporting. Instead, I'm seeing status checks being reported that have my expected checks, but with a "(merge)" suffix. eg:

image

Why is this happening?

@mellis481
Copy link
Author

mellis481 commented Feb 20, 2023

I manually re-ran the workflow and this time it reported the status checks without the "(merge)" suffix. I still very much want the answer to why this is happening seemingly all of the sudden. And it's, of course, necessary to be able to use a single status check regardless if a workflow was manually re-run or not.

@MikeMcC399
Copy link
Collaborator

MikeMcC399 commented Feb 21, 2023

@mellis481

Was this just a one-time occurrence or are you having repeatable issues? You would need to examine the runner workflow logs for the run which hung in order to see what might have gone wrong. If you find a need to re-run your workflow with debug logging enabled, then you can add:

env:
  DEBUG: '@cypress/github-action, cypress:*'

Your post shows that you are using cypress-io/github-action@v4 which is effectively v4.2.2 published on Nov 28, 2022 and unchanged since then.

To take advantage of on-going improvements and bug-fixes you should consider migrating to cypress-io/github-action@v5 which is the latest version available.

Which version of Cypress are you using?

@mellis481
Copy link
Author

v4.

This happened multiple times. I pushed an empty commit to the PR after seeing the problem initially and the corresponding workflow run again had the "(merge)" suffix. It wasn't until I manually re-ran the workflow where the statuses were reported without the "(merge)" suffix.

@MikeMcC399
Copy link
Collaborator

@mellis481

Is your screenshot from the pull request status on GitHub? Are you using a GitHub hosted runner or self-hosted? What do you have in your workflow event trigger section on:? I suspect that merge is a GitHub PR status. In open PRs in this repository I see (pull_request) where you see (merge):

image

v4.

I was actually asking about the version of Cypress in use, like 9.7.0 or 12.6.0 for instance. That is just for background information.

@mellis481
Copy link
Author

Is your screenshot from the pull request status on GitHub?

Yes.

Are you using a GitHub hosted runner or self-hosted?

Self-hosted.

What do you have in your workflow event trigger section on:? I suspect that merge is a GitHub PR status. In open PRs in this repository I see (pull_request) where you see (merge):

on:
  pull_request:
    branches:
      - master

I was actually asking about the version of Cypress in use, like 9.7.0 or 12.6.0 for instance. That is just for background information.

10.4.0.

@MikeMcC399
Copy link
Collaborator

@mellis481

Thanks for the additional information! It doesn't look like this is an issue caused by cypress-io/github-action. Again, I think you would need to dig into your log files to gain more insight.

@MikeMcC399
Copy link
Collaborator

@mellis481

One more thought: when I hover over the GitHub icon in my screenshot it says "@github-actions generated this status".

image

In your screenshot I can see only Cypress icons, so there is a difference.

Your issue has now been assigned to a Cypress engineer, so hopefully he will be able to chip in with more expertise than I have!

@mellis481
Copy link
Author

@MikeMcC399 Thanks! Yes, when I hover over the Cypress icons the popover says "@cypress generated status".

I'm not sure if it was clear from my workflow code, but our workflow job is parallelized and integrated with Cypress Dashboard.

@flotwig
Copy link
Contributor

flotwig commented Feb 28, 2023

@mellis481 I think I concur with @MikeMcC399 that this isn't related to the GitHub Action, since these status checks come directly from Cypress Cloud itself.

I spoke with @davidr-cy and he showed me that the relevant code paths that generate these status checks haven't changed in almost 2 years. (merge) is added if it's detected that the commit message is generated from a merge commit, and our logic around that hasn't changed.

So now we are wondering if GitHub is sending us different metadata all of a sudden, causing the change in behavior you're seeing. Could you please share your Cloud organization ID so we can debug further? You can find it under "Organization ID" inside of the Cypress Cloud.

@mellis481
Copy link
Author

mellis481 commented Feb 28, 2023

@flotwig c5365dbf-9c5b-4769-a95d-130d46f70241

@flotwig flotwig assigned mjhenkes and unassigned flotwig Feb 28, 2023
@flotwig
Copy link
Contributor

flotwig commented Feb 28, 2023

@mellis481 can you try switching to on: push instead of on: pull_request and seeing if it resolves the issue? It's what the docs recommend, for reasons explained there: https://github.com/cypress-io/github-action#record-test-results-on-cypress-cloud

@mellis481
Copy link
Author

@flotwig I'm confused with the recommendation in the article you shared. Those triggers are used in different use cases. on: pull_request should be used for PR gate workflows. on: push should be used for workflows that should be run when a PR is merged.

@mjhenkes
Copy link
Member

on push, happens whenever you push changes to a branch, be it the feature branch, master or main

on pull_request happens when 'the workflow runs when a pull request is opened or reopened or when the head branch of the pull request is updated'.

So after you've created a pull request, on push and on pull_request will both happen whenever you update the head branch.

According to the article the information provided by on push is more accurate than on pull_request but I haven't double checked that myself.

It's worth a shot to try though right? 🤷‍♂

@mellis481
Copy link
Author

mellis481 commented Mar 2, 2023

@mjhenkes The example workflow in the article you shared doesn't specify a branch so it will run the Cypress action on any pushed commit to any branch. That's basically the only scenario I can think of where the push and pull_request triggers could be used interchangeably.

In my situation (which is exceedingly common I can only assume), I only want to run Cypress tests for 1. PRs 2. against certain branches (in my case, just one: master). This means I have to use the pull_request trigger which will run when there is "activity on a pull request in the workflow's repository occurs" (on the master branch). If I were to use the push trigger like you recommend, the only way it would run would be is if I changed the branch targeted by the trigger from master to the branch the PR is for. And then for the next PR, the workflow would need to target that PR branch name. That, of course, isn't how PR gate workflows would ever work.

@mjhenkes
Copy link
Member

mjhenkes commented Mar 3, 2023

@mellis481, you do have a common use case, I agree, but as an experiment could you try using the push request and let us know if the jobs still have the 'merge suffix'? That will either narrow down the problem or eliminate the event as the cause of the issue.

@MikeMcC399
Copy link
Collaborator

@mellis481
Copy link
Author

mellis481 commented Mar 9, 2023

@MikeMcC399 I didn't include it in my original post listing code, but I do include COMMIT_MESSAGE which appears to be what solved the issue in that thread.

Here is my whole Cypress step FWIW:

- id: cypress-e2e-tests
  name: Cypress end-to-end tests
  uses: cypress-io/github-action@v4
  with:
    wait-on: 'https://localhost:9000/index.js'
    start: npm run start:local
    config-file: cypress/config/e2e.ts
    record: true
    install: false
    group: 'E2E'
    ci-build-id: ${{ needs.prepare.outputs.uuid }}
    tag: 'pull-request'
  env:
    NODE_TLS_REJECT_UNAUTHORIZED: 0
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}
    COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }}

@mjhenkes mjhenkes assigned AtofStryker and unassigned mjhenkes Mar 14, 2023
@nagash77 nagash77 added triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. and removed routed-to-e2e labels Apr 19, 2023
@MikeMcC399
Copy link
Collaborator

@mellis481

This issue seems to have gone stale and I'm unsure what the next steps might be.

A new version of the action has been published which automatically pulls in any available PR information when recording. See https://github.com/cypress-io/github-action#automatic-pr-number--url-detection

You would access this functionality using
cypress-io/github-action@v6 or
cypress-io/[email protected]

Again, I'm unsure whether that would change your experience, but I thought I would let you know anyway!

@mellis481
Copy link
Author

@MikeMcC399 I'm still seeing this issue using cypress-io/github-action@v6.

@MikeMcC399
Copy link
Collaborator

@mellis481

Many thanks for your update to say that you are still experiencing this issue.

It looks like it is not directly an issue with cypress-io/github-action but more with the Cypress Cloud GitHub Integration as described on Cypress Cloud > Integrations > Source Control > GitHub.

We'd need some assistance from the Cypress Cloud team on this, since this part of the solution is not open source. We can't see what it is doing or fix it in this repo.

@emilyrohrbough
This issue is assigned to you. Are you still involved?

@mellis481
Copy link
Author

@MikeMcC399 Interestingly, when you just said you think this is a Cypress Cloud thing, I thought I'd compare the project settings for the repo I'm having a problem with with another that I am not. The only thing that I found that was different was that the problem repo did not have a "Default branch" set in Cypress Cloud. I created a PR after setting the Default branch in Cypress Cloud (to master) and it appears to have resolved this issue.

I'm finding the symptoms of a missing "Default branch" in Cypress Cloud strange. As I described in the thread, it appended "(merge)" to the status checks, but what I also found is that, if a GitHub user other than the person who created the PR re-ran the workflow, the status checks reported fine.

In any event, I'll close this issue.

@MikeMcC399
Copy link
Collaborator

@mellis481

Congratulations on solving the issue! Thanks also for sharing what you did.

@mellis481
Copy link
Author

@MikeMcC399 Nevermind the issue is back again. :(

@mellis481 mellis481 reopened this Oct 19, 2023
@MikeMcC399
Copy link
Collaborator

@mellis481

Nevermind the issue is back again.

That is rather disappointing 🙁 !

My previous comments still hold though, that this is unlikely to be an issue with github-action and so the Cypress Cloud team would need to assist.

If you don't get any reaction here and you are on a paid Cypress Cloud plan (Team, Business, or Enterprise), then presumably you could also open a Cypress Cloud support ticket.

@mellis481
Copy link
Author

mellis481 commented Oct 23, 2023

(merge) is added if it's detected that the commit message is generated from a merge commit, and our logic around that hasn't changed.

@MikeMcC399 Thanks. I opened a Cypress Cloud support ticket. As you can see, I'm seeing COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}. Does anything about that seem problematic to you?

@flotwig Can you describe the logic that is used to append (merge) to status checks?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
existing workaround triaged Issue has been routed to backlog. This is not a commitment to have it prioritized by the team. unexpected behavior User expected result, but got another
Projects
None yet
Development

No branches or pull requests

7 participants