Hm, I feel that it’s inaccurate to say “we wouldn’t be able to tell”. It’s not exactly a black box system — the app would have to run on an operating system, and if you are able to know what the operating system is doing, and what instructions are being executed by the CPU, then you can know exactly what the app is doing.
What the aforementioned bits of information provide is the ability to treat software as a black box and be sure of its safety without having to fundamentally audit it.