Sorry I only have this generic troubleshooting point to offer, but have you checked to see if NetworkManager might be modifying your IP routing table in unwanted ways during its operation?
From what you’ve described I’m under the impression that no Internet traffic needs to run through this system; perhaps NM is adding an unwanted default route?