Access control / Privilege escalation in CI automation via GitHub App tokens

PoC proof-of-concept (demonstrates how a leaked broad token could be abused prior to this fix): Assumptions: - An attacker has access to a leaking token with write permissions in Grafana's CI (e.g., grafana-delivery-bot or a GITHUB_TOKEN with write to contents). - The attacker can call GitHub REST APIs (or gh CLI) against grafana/grafana. Goal: - Demonstrate how leaking a broad CI token could be used to create a release tag that might trigger CI/publishing steps, enabling privilege abuse or supply-chain manipulation. Steps (to simulate pre-fix behavior): 1) Create a new annotated tag for a chosen commit (tag that may be picked up by release workflows): curl -X POST \ -H "Authorization: token <BROADER_CI_TOKEN>" \ -H "Accept: application/vnd.github+json" \ -d '{"tag":"exploit-2024.1","message":"Malicious tag by attacker","object":"<COMMIT_SHA>","type":"commit"}' \ https://api.github.com/repos/grafana/grafana/git/tags 2) Create a ref to the tag so it becomes a usable Git tag in the repo: curl -X POST \ -H "Authorization: token <BROADER_CI_TOKEN>" \ -H "Accept: application/vnd.github+json" \ -d '{"ref":"refs/tags/exploit-2024.1","sha":"<TAG_OBJECT_SHA>"}' \ https://api.github.com/repos/grafana/grafana/git/refs 3) Optional: Create a release tied to the tag (if the workflow triggers on releases): curl -X POST \ -H "Authorization: token <BROADER_CI_TOKEN>" \ -H "Accept: application/vnd.github+json" \ -d '{"tag_name":"exploit-2024.1","name":"Exploit 2024.1","body":"Malicious release created for testing","draft":false,"prerelease":false}' \ https://api.github.com/repos/grafana/grafana/releases Expected impact prior to the fix: - A leaked broad token could create a new tag and potentially trigger workflows that publish artifacts, release assets, or push to Docker Hub, enabling an attacker to insert malicious content into a release or alter the artifact delivery pipeline. How the fix mitigates this: - By using a GitHub App (grafana-pr-automation) with restricted permission_sets and per-context scopes, the ability to perform privileged actions is constrained to only what is necessary for the operation (e.g., tagging or PR migrations) and isolated per repository. This reduces the impact if a token is compromised. - The security-mirror writing capability is further restricted to a dedicated app (security-mirror-writer) for the security-mirror repo, preventing unrelated repos from misuse. - Replacing default tokens for sensitive steps with app tokens minimizes the risk of privilege escalation via CI automation.

diff --git a/.github/workflows/create-next-release-branch.yml b/.github/workflows/create-next-release-branch.yml index ac7415c68f36c..377adf21125ac 100644 --- a/.github/workflows/create-next-release-branch.yml +++ b/.github/workflows/create-next-release-branch.yml @@ -38,7 +38,8 @@ jobs: id: get-github-app-token uses: grafana/shared-workflows/actions/create-github-app-token@259ba21cb3ff07724f331e26d926d655d24b317b # create-github-app-token/v0.2.3 with: - github_app: grafana-delivery-bot + github_app: grafana-pr-automation + permission_set: ${{ startsWith(github.ref_name, 'release-') && 'writer-release' || 'writer-main' }} - name: Create release branch id: branch uses: grafana/grafana-github-actions-go/bump-release@main # zizmor: ignore[unpinned-uses] diff --git a/.github/workflows/create-release-tag.yml b/.github/workflows/create-release-tag.yml index 029247253242b..4c6538e9ee42d 100644 --- a/.github/workflows/create-release-tag.yml +++ b/.github/workflows/create-release-tag.yml @@ -22,18 +22,27 @@ jobs: create-tag: runs-on: ubuntu-arm64-small permissions: - contents: write + id-token: write + contents: read concurrency: group: create-release-tag-${{ inputs.tag }} cancel-in-progress: false steps: + # The default GITHUB_TOKEN can't create tag refs that would trigger other + # workflows (e.g. publish-technical-documentation-release.yml on push:tags), + # so use an app token that isn't subject to that restriction. + - name: Get GitHub App token + id: get-github-app-token + uses: grafana/shared-workflows/actions/create-github-app-token@259ba21cb3ff07724f331e26d926d655d24b317b # create-github-app-token/v0.2.3 + with: + github_app: grafana-pr-automation - name: Create annotated tag env: TAG: ${{ inputs.tag }} TAG_MESSAGE: ${{ inputs.message }} COMMIT_SHA: ${{ inputs.commit }} REPO: ${{ github.repository }} - GH_TOKEN: ${{ github.token }} + GH_TOKEN: ${{ steps.get-github-app-token.outputs.token }} run: | tag_sha=$(gh api "repos/${REPO}/git/tags" \ --method POST \ diff --git a/.github/workflows/create-security-branch.yml b/.github/workflows/create-security-branch.yml index 71f597d19a71a..506c5aecc6de9 100644 --- a/.github/workflows/create-security-branch.yml +++ b/.github/workflows/create-security-branch.yml @@ -51,7 +51,8 @@ jobs: id: get-github-app-token uses: grafana/shared-workflows/actions/create-github-app-token@259ba21cb3ff07724f331e26d926d655d24b317b # create-github-app-token/v0.2.3 with: - github_app: grafana-delivery-bot + github_app: ${{ inputs.repository == 'grafana/grafana-security-mirror' && 'grafana-security-mirror-writer' || 'grafana-pr-automation' }} + permission_set: ${{ startsWith(github.ref_name, 'release-') && 'writer-release' || 'writer-main' }} - name: Checkout repository uses: actions/checkout@v5 diff --git a/.github/workflows/migrate-prs.yml b/.github/workflows/migrate-prs.yml index 61853580b19c4..5b01cd1b99c4e 100644 --- a/.github/workflows/migrate-prs.yml +++ b/.github/workflows/migrate-prs.yml @@ -43,7 +43,8 @@ jobs: id: get-github-app-token uses: grafana/shared-workflows/actions/create-github-app-token@259ba21cb3ff07724f331e26d926d655d24b317b # create-github-app-token/v0.2.3 with: - github_app: grafana-delivery-bot + github_app: grafana-pr-automation + permission_set: ${{ startsWith(github.ref_name, 'release-') && 'pr-writer-release' || 'pr-writer-main' }} - name: Migrate PRs uses: grafana/grafana-github-actions-go/migrate-open-prs@main with: diff --git a/.github/workflows/release-build.yml b/.github/workflows/release-build.yml index d85d2de2f58b3..62e240bdb5d1b 100644 --- a/.github/workflows/release-build.yml +++ b/.github/workflows/release-build.yml @@ -832,145 +832,6 @@ jobs: environment: prod runs-on: ubuntu-x64-small - publish-dockerhub: - if: github.repository == 'grafana/grafana' && github.ref_name == 'main' - permissions: - contents: read - id-token: write - runs-on: ubuntu-x64-small - needs: - - setup - - verify-targz - - build-docker-linux-amd64 - - build-docker-linux-arm64 - - build-docker-linux-armv7 - - verify-packages - strategy: - fail-fast: false - matrix: - include: - - variant: alpine - full-artifact: artifacts-docker - slim-artifact: artifacts-docker-alpine-slim - full-tag-suffix: "" - slim-tag-suffix: "-slim" - - variant: ubuntu - full-artifact: artifacts-docker-ubuntu - slim-artifact: artifacts-docker-ubuntu-slim - full-tag-suffix: "-ubuntu" - slim-tag-suffix: "-ubuntu-slim" - - variant: distroless - full-artifact: artifacts-docker-distroless - slim-artifact: artifacts-docker-distroless-slim - full-tag-suffix: "-distroless" - slim-tag-suffix: "-distroless-slim" - steps: - - uses: grafana/shared-workflows/actions/dockerhub-login@dockerhub-login/v1.0.2 - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.full-artifact }}-linux-amd64 - path: dist - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.full-artifact }}-linux-arm64 - path: dist - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.full-artifact }}-linux-armv7 - path: dist - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.slim-artifact }}-linux-amd64 - path: dist - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.slim-artifact }}-linux-arm64 - path: dist - - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c - with: - name: ${{ matrix.slim-artifact }}-linux-armv7 - path: dist - - name: Push ${{ matrix.variant }} + ${{ matrix.variant }}-slim to Docker Hub - env: - VERSION: ${{ needs.setup.outputs.version }} - FULL_TAG_SUFFIX: ${{ matrix.full-tag-suffix }} - SLIM_TAG_SUFFIX: ${{ matrix.slim-tag-suffix }} - run: | - set -euo pipefail - TAG_VERSION="${VERSION//+/-}" - for f in dist/*.docker.tar.gz; do docker load -i "$f"; done \ - | grep "Loaded image:" | sed 's/Loaded image: //' | sort -u | tee docker_images - while read -r line; do docker push "$line"; done < docker_images - docker manifest create "grafana/grafana:main${FULL_TAG_SUFFIX}" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-amd64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-arm64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-armv7" - docker manifest create "grafana/grafana:main${SLIM_TAG_SUFFIX}" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-amd64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-arm64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-armv7" - docker manifest create "grafana/grafana-dev:${TAG_VERSION}${FULL_TAG_SUFFIX}" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-amd64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-arm64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${FULL_TAG_SUFFIX}-armv7" - docker manifest create "grafana/grafana-dev:${TAG_VERSION}${SLIM_TAG_SUFFIX}" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-amd64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-arm64" \ - "grafana/grafana-image-tags:${TAG_VERSION}${SLIM_TAG_SUFFIX}-armv7" - docker manifest push "grafana/grafana:main${FULL_TAG_SUFFIX}" - docker manifest push "grafana/grafana:main${SLIM_TAG_SUFFIX}" - docker manifest push "grafana/grafana-dev:${TAG_VERSION}${FULL_TAG_SUFFIX}" - docker manifest push "grafana/grafana-dev:${TAG_VERSION}${SLIM_TAG_SUFFIX}" - - run-meticulous-tests: - if: github.repository == 'grafana/grafana' - needs: - - setup - - publish-dockerhub - name: Run Meticulous tests - runs-on: ubuntu-x64-small - continue-on-error: true - permissions: - actions: write - contents: read - id-token: write - issues: write - pull-requests: write - statuses: read - - steps: - - uses: actions/checkout@v6 - with: - persist-credentials: false - - - name: Compute image tag - id: tag - env: - VERSION: ${{ needs.setup.outputs.version }} - run: echo "value=grafana/grafana-dev:${VERSION//+/-}" >> "$GITHUB_OUTPUT" - - - name: Pull image from Dockerhub - env: - IMAGE_TAG: ${{ steps.tag.outputs.value }} - run: docker pull "$IMAGE_TAG" - - - name: Get vault secrets - id: vault-secrets - uses: grafana/shared-workflows/actions/get-vault-secrets@main - with: - repo_secrets: | - METICULOUS_API_TOKEN=meticulous-ui:api-token - - - name: Run Meticulous tests - uses: alwaysmeticulous/report-diffs-action/upload-container@89c6081b30d17cfcc410b5c19c269cd7a4d67cdf # v1 - with: - api-token: ${{ env.METICULOUS_API_TOKEN }} - image-tag: ${{ steps.tag.outputs.value }} - container-port: 3000 - container-env: | - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin - dispatch-npm-canaries: if: github.repository == 'grafana/grafana' && github.ref_name == 'main' name: Dispatch publish NPM canaries