There are some very, very small and light fighter jets including the Gripen and Freedom Fighter. A lot of the reason for the size of the jet has to due with the mass fraction. They often want a better than 1:1 thrust:weight ratio for executing combat maneuvers, and it needs to offer this <em>while carrying a large payload</em> which is dramatically larger than a human and life support. Depending on class and mission this can include very large internal fuel stores, additional external fuel and multiple types of air-to-air and air-to-ground weapons for different scenarios.
Indeed, many UAVs are a lot <em>bigger</em> than some manned fighters, such as the Global Hawk next to an F-5.
But <em>by a mile</em> the most important concern is usage considerations.
Current UAVs are designed for and deployed against enemies who have no relevant defense technology, no ability to jam signals and no real competitiveness against the UAVs. They're designed assuming a state of air supremacy.
Fighter jets necessarily must be designed assuming a technologically competitive enemy who can both jam your drone's communication, find fault in its artificial intelligence and possibly blind it or otherwise subvert its senses.
If the machine has to operate without communications it must also make life-or-death decisions about what and when to shoot and even humans can screw this one up. There are friendly fire incidents all the time. <em>Key point; our observational technology hasn't always been adequate to differentiate a 747 and F-14, or a Brit on the ground from an Iraqi</em>. Can an autonomous, unintelligent machine judge context better and make mistakes less? <em>Could its inability, coupled with jamming or other manipulation, be subverted for propaganda purposes</em>?
All things considered, I don't see the cost-risk-benefit analysis weighing against the total elimination of a genuinely intelligent, well-trained, on-sight decision-maker. Humans are imperfect and a little heavy, but so are military-grade computers and sensors and not only are humans dramatically smarter at this point in time, but something can't be both intelligence and flawless anyway.