(Newbie question) Did i handle my system crashing correctly?

MangoPenguin, 24 days ago

You’re much more patient than I am, my method is hold the power button down until it shuts off lol

rotopenguin, 25 days ago

Do the sysreq sometime when your system isn’t hung. If it isn’t enabled, welp you have to enable it harder.

Having ssh set up would be a way in when the whole graphics stack falls over (but the kernel is still alive in there). On intel there are /sys entries to dump GPU state, ATI probably has something similar. You have a reproducible bug, if you can get in and grab data while the gpu is in la-la-land, you might be able to submit a valuable bug report.

Para_lyzed, 25 days ago (edited 25 days ago)

This is not very common knowledge, but it is no longer recommended to press S or U before B for SysRq. The official documentation of sysrq has stopped recommending this practice, as it may be harmful to modern filesystems. Writing to a storage device while the kernel is in a bad state has the potential to cause corruption, and modern journaling filesystems like EXT4 and BTRFS are designed to survive crashes like this with minimal (or no) corruption. Instead, you’ll likely want to use Alt+SysRq+REIB (and make sure you are waiting multiple seconds between each keypress, as they do not complete instantly!).

You may instead try to kill the most memory intensive non-vital process with Alt+SysRq+RF, which may stop you from crashing to begin with (this works especially well for memory leaks). SysRq+F will invoke the oom (out of memory) killer, which will kill the most memory intensive non-vital process without causing a kernel panic.

If you need to restart, the most ideal situation is to enter a TTY and cleanly reboot, in which case you can do Alt+SysRq+R to grab control from the display manager, then Ctl+Alt+F3 or Ctl+Alt+F4 (I believe most distros have the first login session run on the TTY accessible from Ctl+Alt+F2) to switch to another TTY. You can then log in and do sudo systemctl reboot if your computer is still responsive. You may need to kill some processes before your system becomes responsive enough to log in on a TTY, which is where Alt+SysRq+F is useful, but in extreme situations it may require Alt+SysRq+EIB.

So a basic order of steps to try may look like:

1. Try Alt+SysRq+RF and wait a few seconds to see if your system starts responding. If not, you can either try it another time or two, or move on to 2.
2. See if you can switch to a TTY with Ctl+Alt+F3. If so, try to log in and sudo systemctl reboot. Else move onto 3.
3. If you are in a TTY, switch back to the main login with Ctl+Alt+F2. Then do Alt+SysRq+REIB.

In the spirit of other users giving mnemonic devices, you could remember REIB with Reboot Even If Broken, or the oom killer RF with Resolve Freeze (someone else can probably think of something better for RF; I’m not great at making mnemonic devices).

TL;DR: There are SysRq combinations that are less prone to damage/corruption than Alt+SysRq+REISUB, so use the above flowchart, or just remove the S and U for Alt+SysRq+REIB (if you don’t want to troubleshoot first) for less chance of filesystem corruption from a bad kernel. You can often recover the system without having to hard reset (Alt+SysRq+B). And ALWAYS wait between SysRq keys, as they do not finish instantly.

Ti_Ka, 25 days ago

Thanks for the detailed explanation on the sysreq keys & when & how to use them for unlocking a frozen system :D. Also for the systemctl bit because i wasnt even sure what to do if i had gotten to a console lol.

AnUnusualRelic, 25 days ago

There’s also the option of ssh-ing in to remove the offending process, and possibly restart the display manager.

You’ll need to hav another device available of course.

SteveTech, 25 days ago

This might be just me, but I prefer remembering what the keys actually do:

• R - Raw keyboard
• F - Free mem
• E - End everything
• I - kIll everything
• B - reBoot

Also good to know:

• S - Sync drives
• U - Unmount drives
• O - power Off
• C - Crash

governorkeagan, 25 days ago

I don’t remember that scene from RDR2, I guess I need to go back and play it from the beginning again!

Regarding the crash, here are my two cents. I’ve played RDR2 on Pop!_OS and EndeavourOS and have not really had any major issues (I’ll get to that in a second). My specs are similar but not quite the same as yours:

• Ryzen 5 3600
• Nvidia GTX 1660TI
• 32 GB RAM

The issues I’ve had have not been as major as the game/system crashing the same way yours did. However, I have had serious screen tearing and worsened performance (although not very often at all). Every time, I have updated the graphics driver or kernel (or both) and upon rebooting and starting the game again the issue was fixed. This may very well be the case for yourself as others have already mentioned.

qjkxbmwvz, 25 days ago

Only additional thing I would do would be to try to ssh into it to. Sounds like that wouldn’t have worked anyway. But if you can ssh into it while it’s in a degraded-but-not-completely-borked state you can poke around, troubleshoot, and of course cleanly reboot.

MrPenguinSky, 25 days ago

That’s about what I would have done. Try to switch to a console first like you did, but if that doesn’t work, force reboot and pray.

The worst that could happen is that you might have some data loss or mess up the OS (which has happened to me before). That’s why you should always have an up-to-date backup! It’ll save you one of these days.

Either way, try switching to a different proton version, or do some other fixes and test it out until it stops crashing. This honestly sounds like a freak (but common) problem. I wouldn’t worry about data loss or corruption that much if there’s no other signs of it (e.g., SMART reporting a dying drive), but a backup is generally helpful for any scenario.

kbal, 25 days ago (edited 25 days ago)

Ah, Fedora 40 is on kernel version 6.8.9 it seems, the bad one for amdgpu, and it doesn't look like their current build is patched to fix it, so it could be this bug. It's fixed in 6.8.10 and 6.9 if you have the ability to upgrade to those. Otherwise you might want to try reverting to the previous kernel version if that's easy in fedora.

(Edit to add that I didn't see the "im a complete newbie" bit... I'm just very aware of this recent bug because it gave me some trouble. Sorry if you did happen to start with a version that has this problem. It's really bad luck if so. But I don't really know.)

Ti_Ka, 25 days ago

It’s fixed in 6.8.10 and 6.9 if you have the ability to upgrade to those.

Honestly idk how id even begin to do that lol, and id also maybe rather not start my first week of linux use by immediately trying to change the kernel version on my own XD (either down or up). I did hear about an issue with rdr2 and kernel 6.8.9 from a reddit post which i found through someone writing about problems with the game on its protondb page. But i thought i was fine as my game worked normally until i encountered the crash & because the reddit and protondb post say its solved by enabling rebar which (iirc) i already have.

However idk if that reddit posts issue is the same/related to the one you linked. Since the rest of the game and my system seem to be mostly fine i think ill either just not play the game or specifically avoid the cutscene when i do (its in an optional quest luckily). And then ill maybe return to it after the updated kernel arrives on fedora to see if it solves the crash or not.

owatnext, 26 days ago

You did fine, as a new user I am surprised that you even attempted to get to the console and so on.

With forcefully powering off, the biggest (or I guess most likely) thing that could happen is losing data. As long as your install wasn’t working on updates or major system configuration changes, you’ll probably be fine in that regard; it would boot back up just fine.

GG.

rtxn, 26 days ago (edited 26 days ago)

Sounds to me like the kernel or the video driver died. Try pressing caps-lock a few times – if the keyboard’s LEDs don’t change, your inputs are dead and pretty much your only option is to power down or reset the computer. Most modern filesystems, like ext4 and btrfs (you likely have one of those) are very robust and can easily handle an ungraceful shutdown. When you start the OS again, it’ll run fsck on the root partition and get it into a functional state. Data loss can still occur if the computer dies while a process is still writing a file, but I think it was inevitable the moment the OS froze.

Unfortunately I can’t offer much advice other than to use a numbered Proton version instead of experimental, and to try again at a lower quality setting. You should also try Gamemode to temporarily optimize your system for running games.

I’ve played RDR2 on a weaker system than yours. It’s a very intensive game to run in terms of memory usage, streaming from mass storage, and CPU/GPU. Install it on an SSD to give it the best chance, and use a system monitor like bpytop or htop to check the RAM/CPU stats and temperatures.

TranquilTurbulence, 25 days ago

That capslock idea was pretty good. Next time I’ll start with that to see if all hope is lost.

EccTM, 26 days ago

• For what I assume is a security precaution, SysRq is disabled by default in Fedora, and you need to go enable it if you want to be able to recover when shit like this happens - the link shows how, and explains what each letter does.
• It’s honestly hard to say what caused it, but you could check your system logs and see what looks suspicious around the time of the crash. journalctl, dmesg and your steam logs (in ~/.steam/steam/logs usually) could be worth a look, or worth showing someone else at least if you aren’t sure whats going on in there.
• I’d avoid actively trying to cause it, but if it happens again you handled it exactly how I’d try handle it. Having SysRq enabled would let r-e-i-s-u-b handle it more gracefully than a forced shutdown at least!

Successful_Try543, 26 days ago

The linked article is really helpful in learning why Raising Elephants IS Utterly Boring ;-)

swab148, 25 days ago

I like the Arch wiki’s version: Reboot Even If System Utterly Broken.

Successful_Try543, 25 days ago

Nice. Both versions easily get stuck in the head.

Ti_Ka, 25 days ago (edited 24 days ago)

Thanks for the link! I managed to set up sysrq with it, which might have saved me from reinstalling steam when the crash occurred the second time (see the update in my post).

yukijoou, 26 days ago

you probably got a kernel panic, which froze the system. it’s like a BSOD on windows, except on linux, there isn’t a proper stack to handle them when they happen while you have a graphicam session running, so it kinda just freezes

i don’t think reisub would do anything, because the kernel was probably already dead

you don’t risk corrupting much data by hard-reseting your pc on linux – journaling filesystems, like ext4 or btrfs, are built to be resilient to sudden power loss (or kernel crashing). if a program was writing a file at thz time the kernel crashed, this one file may be corrupted, because the program would get killed before it finished writing the file, but all in all, it’s pretty unlikely. outside of fs bugs, which are thankfully few and far between on time-tested filesytems like ext4, you shouldn’t have to worry much about sudden power loss!

unfortunately, figuring out the cause of these issues can be challenging – i’ve had many such occurences, and you have no logs to go off of (because the system doesn’t have time to save them), so you’d most likely need to figure out a way to send your kernel logs onto another system to record them

as general mitigation steps, you should try monitoring your cpu temperature a bit closer - it could be high temperature tripping the safeties of your motherboard/cpu to avoid physical damage to them - in which case, try installing a daemon to control your cpu frequency, like auto-cpufreq, or something like thermald specifically made to throttle your cpu if it gets too hot (though i think that one is intel specific)

TheCheddarCheese, 26 days ago

I always thought a kernel panic ended the graphical session… Turns out I was wrong. Again.

gbin, 26 days ago

Your overall process is perfect: first try to solve it from the UI, then the console, then the magic sysreq key.

The fact that your kernel was not responding to the sysreq key could mean a couple things: is it enabled on your install? (cat /proc/sys/kernel/sysrq to check)

Before trying to understand why the kernel locked up, are you sure everything is solid on the hardware side? ie. Did you overclock anything? If yes did you burn test the PC on some GPU demo?