I think i understand the situation from their point of view(AI companies) and i think i understand what the regulations people are supposed to do, but i feel pessimistic about it being technicaly possible. So we will get this AI information apocalypse where nobody can predict or controll it and in a generation or two people will either stop understanding the world around them like birds watching cars, or we get some dunes book kind of wipe of AI like the buttlerian jihad and we forbid machines that think like people.
Perfect prediction may be impossible, but a narrower target is still useful: preserve enough evidence after each run to contain failures and assign responsibility. That will not stop an information flood, but it can keep a model action from becoming invisible or unauditable. The hard part is requiring that evidence before incidents scale, not after.