WIP: Add TMT gating test and workflow for bootupd #967
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @aaradhak, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request establishes a new, robust testing infrastructure for the bootupd project by integrating the Test Management Tool (TMT). It introduces a fundamental smoke test to confirm bootupd package installation and sets up a flexible test environment capable of handling bootupd binaries from either source builds or installed packages. This initiative significantly enhances the project's automated testing capabilities.
Highlights
- TMT Test Infrastructure: Introduces the Test Management Tool (TMT) framework to the project, laying the foundation for automated testing.
- Core Smoke Test: Adds a basic smoke test (
tests/core/core.fmf) to verify the successful installation of thebootupdpackage. - Flexible Test Environment Setup: Implements conditional
preparesteps intests/tmt/plans/main.fmfto support testingbootupdwhether it's built from source or installed as a distribution package. - TMT Smoke Plan: Defines a TMT plan (
tests/tmt/plans/smoke.fmf) to discover and execute all tests tagged with 'smoke'.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces TMT testing infrastructure for bootupd, including a core smoke test and a GitHub workflow. The changes are well-structured. I've found a couple of issues in the test preparation scripts: one related to the robustness of finding the bootupd binary when built from source, and a critical issue with an incorrect hardcoded path for the package-installed binary. Addressing these will improve the reliability of the new test suite.
Add TMT testing infrastructure including core smoke test with GitHub workflow integration.
aaradhak
force-pushed
the
gatingtestbootupd
branch
from
130b694 to
3d1b4c5
Compare
There was a problem hiding this comment.
I'm not opposed to this but it's also important to acknowledge that this is testing almost nothing.
As far as I understand without additional work, it will not be testing the source changes to the git repository here, only installing the existing package, which seems misleading at best.
I think what actually is solving that problem is packit which acts as a glue later on top of testing-farm/tmt to build rpms in a copr and then inject them by default.
But that gets to the next problem that doing useful bootupd testing requires being in a bootc/coreos system so we'd need to use e.g. https://tmt.readthedocs.io/en/1.42/plugins/provision.html#bootc
aaradhak
changed the title
Thank you for this information about bootc. Right now we are in the process of creating a gating test infrastructure for all the RPMs under CoreOS. It is a work in progress where I am trying to add a basic test in bootupd. Now I am planning to add a test from the CI to see if that works - https://github.com/coreos/bootupd/blob/main/.github/workflows/ci.yml |
There was a problem hiding this comment.
Bigger picture @lsm5 from the container runtimes team is also working on tmt related things (ref containers/skopeo#2640 ) and I think it might make sense for you (or others) working on TMT related things for coreos to at least have a quick sync.
Sure will do @mike-nguyen @HuijingHei @tlbueno : I would like to tag everyone to inform about this |
Neat! This will be super useful for testing some components. As Ash mentioned, the goal is to to have the framework in place and share it with the downstream gating and testing. We want to make it easy to add tests so we can increase test coverage. |
Of course, it makes total sense. However, tmt was basically IMO not really at all designed to be used "upstream" in this way by default. It was designed to try to replace/modernize a variety of testing systems that were more "downstream" focused. In particular, the thing I find just extremely hard to understand is the key use case of "build the software from source locally and then run the tests against it" with tmt. See also teemtee/tmt#1075 To summarize again my understanding of the status quo is that basically packit is trying to be the bridge between "build project from source, generate rpm" and then hand that off to testing/farm/tmt. But then packit is more of a service, and there's this I guess assumption that for local things people just know how to build rpms or something? And again this is especially extra relevant for projects like bootupd where testing them really wants an image-mode/coreos style environment. We have a horrendous hack for this in bootc (build a container image inside the target environment in each test locally and then This is all really quite important to understand and get right because if you look at what it's in this PR, it's not building bootupd from the upstream git...just installing the package inside a container from the existing repos. (i.e. (Also on a different topic I was actually kind of surprised that So I guess bottom line my recommendation is to wire up packit and testing-farm. That said, I definitely like the idea of using GHA as runner infrastructure (bypassing packit, COPR and testing farm infra) but if we do that it should at least build from source. |
Maybe I did not clarify this to Ash, bootupctl runs in bootc image, and we build container with https://github.com/coreos/bootupd/blob/main/.github/workflows/ci.yml#L47 that is using https://github.com/coreos/bootupd/blob/main/Dockerfile |
|
Hello, I see rust-bootupd is shipped on Fedora and CentOS Stream, so I'd highly recommend using a Packit workflow to manage Fedora releases as well as trigger TMT tests (on multiple Fedora and CentOS Stream environments). The same set of TMT tests can be run upstream as well as on Fedora and CentOS Stream (dist-git PRs, bodhi updates) with a single source of truth. We're already doing this on podman and many others in the containers org. And I see Btw, CentOS Stream official downstream updates can only be triggered via packit CLI for now because of access restrictions on lookaside cache. |
I filed bootc-dev/bootc#1473 on this and it's very relevant here too |
|
The original intent was to have the tests live near the source and only run it downstream in fedora, centos, and RHEL. In a separate repo, we were asked to run the same set gating tests upstream so they could catch gating failures earlier instead of getting blocked downstream during a critical release time. That made sense to us so we incorporated the feedback. Packit also makes sense and has better alignment with tmt so we will look into that option. |
@lsm5 who on your team is working on it? Can we sync up and see what the team is working on? |
|
I will start with enable packit firstly. See #1003 |
Add TMT testing infrastructure including core smoke test with GitHub workflow integration.