Skip to content

Coverage lines on pull requests are incorrect when not branched off latest commit of base branch #961

Description

@MetRonnie

Describe the bug

When a GitHub Actions workflow runs on the pull_request event, actions/checkout by default does not check out the pull request branch itself, but merges it into the base branch and checks out this merge commit.

This means that the reported coverage is for this merge commit, but the Codecov website does not account for this. When you view the coverage for a PR on the Codecov website, it shows the files as they are in the PR branch, meaning the coverage lines are incorrect (when there have been other changes to the same files committed to the base branch).

Environment (please complete the following information):

  • Browser N/A
  • System N/A

To Reproduce

This is what I did in a simple test repo: https://github.com/MetRonnie/codecov-bug-demo

  1. Create a pull request off a commit that is not the latest commit on the base branch

Expected behavior
I guess the only solution is for the Codecov website to display the same internal merge commit that actions/checkout checked out, rather than the head commit of the PR

Screenshots

Executable lines are shown as unexecutable and vice versa!

Image

Here is the commit graph.

Image

The coverage hits/misses match the lines of the latest commit on master (while the displayed line contents do not match)

Image

Additional context

Originally reported 2 years ago in #556 but has not been addressed. A colleague ran into this today and was thoroughly confused.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions