As far as I understand, you only have to make your changes to the code available to users of your software. You are free to make any modifications as long as you keep them to yourself and don’t share the binaries (or access the service, in case of AGPL) with anyone. I might be mistaken, though.