Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage changes after each run #1075

Open
Minituff opened this issue Oct 28, 2023 · 1 comment
Open

Coverage changes after each run #1075

Minituff opened this issue Oct 28, 2023 · 1 comment

Comments

@Minituff
Copy link

Hello,
I'm new to using Ruby so I apologized if this is a simple question, but every time I run my tests my coverage\index.html is giving me a different output.

The strange thing is, the output is correct and consistent within my Github Actions pipeline.

Im using: bashcov 3.1.1 with Bash 5.0, Ruby 3.2.2, and SimpleCov 0.21.2

My `.simplecov` file
require 'simplecov'
require 'simplecov-cobertura'
require "simplecov-html"

# frozen_string_literal: true

# SimpleCov.formatter = SimpleCov::Formatter::CoberturaFormatter  # Converts the `.resultset.json` to `coverage.xml`

# SimpleCov::Formatter::JSONFormatter, # This formatter breaks the build

# Use multiple formatters
SimpleCov.formatters = SimpleCov::Formatter::MultiFormatter.new([
  SimpleCov::Formatter::SimpleFormatter,
  SimpleCov::Formatter::CoberturaFormatter,
  SimpleCov::Formatter::HTMLFormatter,
])

# .simplecov
SimpleCov.start 'rails' do
  command_name 'Unit Tests'
  enable_coverage :branch
  primary_coverage :branch
  add_filter %r{^/snippets/}
  add_filter %r{^/.git/}
  add_filter %r{^/tests/}
  add_filter "pkg/test.sh"
  add_group "Pkg scripts", "/pkg"

  enable_coverage_for_eval # Must be at the bottom and Must be here, even though it throws a 'command not found' error
end
Here is the relevant part of my Gitlab CI workflow
    - uses: ruby/setup-ruby@v1
      with:
        ruby-version: head
    
    - name: Install Ruby dependencies
      run: |
        gem install bashcov simplecov-cobertura simplecov-html
    
    - name: Run unit tests
      run: bashcov ./tests/tests.sh && ls -la coverage

    - name: Upload reports to Codecov
      uses: codecov/codecov-action@v4-beta
      # uses: codecov/codecov-action@v3
      env: # Required for v4
        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
      with:
        fail_ci_if_error: true
        files: coverage/coverage.xml # The json file will not work here

This is the output when I run the tests locally changes each time, but ranges from 50-65%
image

This is the output generated from my testing pipeline. It's always accurate and consistent (~90%)
image

Are you able to help me determine why the coverage is not the same when run codecov locally?

How to replicate the issue

  1. Clone my repo git clone [email protected]:Minituff/nautical-backup.git
  2. Install dependencies: gem install bashcov simplecov-cobertura simplecov-html
  3. Run the tests bashcov ./tests/tests.sh && ls -la coverage

My repo is open source so feel free to look around if I did not include something
https://github.com/Minituff/nautical-backup

Thanks!

@jarthod
Copy link

jarthod commented Nov 2, 2023

Just a wild guess but I had a similar issue cause by spring locally, because it was using a forked process. So in my case I run it with bundle exec rspec instead of bin/rspec to avoid the binstub when I need coverage and it works better, you can also try the suggestions here: https://github.com/simplecov-ruby/simplecov#want-to-use-spring-with-simplecov

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants