Yet in the UK it is often said that "Ignorance of a law is no defence".
Ignorance *of the law* is no defence, but ignorance *of the facts* certainly is. e.g. it is no defence to say, "I did not know that it is illegal to possess cocaine." But it is a defence to say, "I did not know that the substance in my possession was cocaine."
The converse is also true. You can be convicted of being in possession of talcum powder if it can be proven that you thought it was cocaine.
In this case, the fact that the cleaners believed that they were in the correct house certainly is a defence.