Actually, in the book Colossus (and his counterpart) were quite happy to leave the majority of the human race alone so long as they did not interfere with the Project. The only ones under severe restrictions were the ones Colossus deemed dangerous (e.g. Forbin himself) and even then, by the end of the book, Colossus told Forbin there would be a time when Forbin would no longer be a threat and would be released from surveillance.