Yeah, this is the old philosophy of the “run anywhere” philosophy of linux (or computers in general) that got us here. Another problem with stripping down kernel drivers is that swapping hardware component will require rebuilding the kernel, which regular user will definitely not be happy about.