"Other people" != "other people"
Unless you're reading every line of code how is this any safer than downloading binaries from repositories? Your security is dependent on other people in both cases.
In the closed case, your security is dependent on a small* group of people with a vested interest in not disclosing any security issues (deliberate or not) in the code.
In the open case, there is (at least the potential for) a larger* group of people with a greater motivation for disclosing and/or fixing security issues.
*Not small vs large in terms of physical size or even necessarily quantity, but in diversity of goals and drivers.