Saturday #fedora infra week in review! This week: Some fun with a vHMC, mass update/reboot cycle completed, scrapers/DDoSers update, rdu2-cc to rdu3 datacenter move updates, and I will be out next week. ;)
Kevin
infra weeksly recap: Late November 2025
Another busy week in fedora infrastructure. Here
Today in #fedora infra land: * tail end of the fedora 43 release party * downgraded docker-distribution on registries, 3.0.0 is really really slow on catalog request (under 1 s vs sometimes 30-40s) * Updated 3 anubis tickets with client changes suggestions * fixed pagure-stg02 and it all seems working! * created pagure01 to migrate pagure to on december 3rd * started syncing ~750GB of data from pagure.io to new pagure01
Today in my #fedora corner: * Updated and rebooted a zillion things. * Upgraded all the fedora builders to f43 * reinstalled all the x86 buildvm's (because we have to do this for 32bit builds to avoid >4G inodes) * Adjusted bodhi internal dns view to bypass anubis * Tweaked some anubis rules. It's really doing nicely now with the latest version. * Upgraded our prod openshift cluster to 4.20. * Our vHMC is behaving again after it's vmhost reboot. Hurray!
Today in my #fedora corner: * Some anubis adjustments for iot * FRCL meeting (Fedora, Red Hat, Centos) * Fixed a git divergence in our dns repo * Poked the vHMC with a stick more. Out of ideas, filed
Making sure you're not a bot!
 to track it * synced data from stg.pagure.io to a new dc one. Asked for ports opening to test. * Upgraded prod and stg openshift clusters to latest 4.19, then stg to 4.20 * Updated/rebooted some machines ahead of outage tomorrow (
Making sure you're not a bot!
 )
Today in my #fedora corner: Spent almost all my day on two things: * Our vHMC (the virtual hardware management console) for our power10 systems decided to start just dropping packets all the time talking to the systems. Still haven't figured out why. ;( Tried to update it, but thats been fraught with peril. * Adjusted policy and upgraded anubis again in our prod proxies. Hopefully this time it doesn't bother ostree stuff (should be allowed). Cranked up challenge level on src.fp.o forks.
Monday monday monday! * Made some more riscv koji keytabs * Some meetings * moved 2 older machines I had marked for openqa to become new vmhost-x86-iso01 and bvmhost-x86-riscv02. * Installed them and configured them * The iso one will be for migrating pagure.io and other things at that dc to the new one. * The riscv02 one will be to run a emulated riscv builder or two (for composes). * Created a pagure-stg02 on the iso host. More on it tomorrow. * Updated vHMC and now it's not talking right.:(
Some #homeassistant fun today. Our "smart" shower never rejoined the wireless network after the last power outage. It's a "u by moen" one. Reset it by: * hold down down temp button * select reset * select wifi * join it's wifi on your phone * run app on phone and select reset wifi * power cycle it and curse until it takes * press and hold up temp for the homekit code * delete in homeassistant, add the new one it detects (it seems to change HK code on reset). Fun times.
Finally another weekly recap (but it's been a few weeks) from my #fedora infra corner. This time: tcp timeouts fixed! 🎊 a bunch of outages lately. 😢 power10 saga finally over. :blobcatcheer: rdc-cc to rdu3 datacenter move news.
Kevin
infra weeksly recap: Early November 2025
Well, it
Today in infra #fedora land: * kojipkgs servers hit a super high load (~900-1000). Tracked the issue to bunch of requests for ostree directory listings that have each 65k ish files in them, which apache has to stat each one. Moved these directories to be cached by varnish. Hopefully that will prevent the issue. * Reassigned a bunch of packages with non human poc (
Making sure you're not a bot!
 ) * Fixed the f43 nightly cloud build tagging (
Making sure you're not a bot!
 ) <1/N>