Linus calls Linux 'bloated and huge' Linux creator Linus Torvalds says the open source kernel has become "bloated and huge," with no midriff-slimming diet plan in sight. During a roundtable discussion at LinuxCon in Portland, Oregon this afternoon, moderator and Novell distinguished engineer James Bottomley asked Tovalds whether Linux kernel features were being …
I've used linux for over ten years, starting with the latest 1.2 - 1.3 kernels, and slowly climbing the ladder until now (fedora 10, 2.6.27). 1.2 kernels were terribly inefficient, but we didn't expected much of computers in those times, so it got away with it. But 2.0.x was really marvellous, albeit limited in functionalities. Then we got the 2.2.x series, that was rather experimental in spite of its alleged stable status, and performances sucked (big locks, many in-memory data copy etc.).
It was soon replaced by the 2.4 line. 2.4 is still rock stable today, and while it lacks many drivers for new hardware, it's pretty small and can run most of your exotic stuff (alpha, sparc smp). But in spite of being lean and stable, 2.4 is still full of the 2.2 conception quirks that are performance bottlenecks. Hence the need for 2.6.x, which originally aimed at streamlining the foundations of the kernel. Many O(n) algorithms were swapped out in favour of O(1) counterparts, and where it runs, 2.6, while generally much bigger than 2.4, runs way better and faster. That recently it begun to slow down a bit doesn't do justice to the extraordinary improvements it made over older kernels.
So maybe it's time to feature-freeze 2.6, and stabilize it while launching experimental 2.8 for new stuff. But in my books, 2.6 is still the most efficient kernel to date, and one of the most innovative, competing only with 2.0 in that area (2.0 brought a tremendous amount of novelties to linux, support for elf32 to begin with, and much, much, much more, and is still maintained for some embedded applications).
Now, I'm not a kernel hacker, but I've had the chance recently to play with NetBSD, I've used windows XP and Vista (not 7 yet), and a bit of MacOS X. While any of those can be marginally better than linux in a particular area, *after* spending an awful lot of time trimming it (just about like you could improve linux in the first place by tailoring a slackware to suit your needs), the conclusion is linux is the 4WD of OS. Runs everywhere, and pretty much goes over everything you can think of throwing at it. NetBSD ? Oh, you need to compile that kernel to get what you want (and I tell you, it's not for the faint of heart ; been compiling my linux kernel for ages without breaking a sweat, but NetBSD gives you chills down the spine). MacOS X ? Does everything, but random (x)thing will set you back another (y*10^3) $ - and forget about whatever computer you have, it's intel-apple only now. Windows ? ... I'm torturing myself to find a place where widows shines, and can't find none. It's a nightmare for everyone, from sys admins to users, albeit a familiar nightmare so they don't feel the need to wake up.
I am a software developer actually, in the HPC community. Cycles are expensive so we use them well. I'm sure some DB developers see where I'm coming from --- DB lookups should be done wisely, in the right places and if necessary cached - all too often you see people looking up the same bloody (non-volatile) thing in every iteration of a loop.
I'm still running (custom stripped down kernels) on hardware that XP won't even install on, with X, and it runs pretty decently. Mind you, I can't use any of the popular web browsers, thanks to 128mb of ram, but I can go from power on to El Red (in Kazeahkase) in about 30 seconds. I don't remember any sort of Windows doing that on a PII.
Not that the kernel couldn't use some optimizing, but it's a huge chunk of code that people actually depend on, and they've got to keep adding new hardware support.
Linux did accidentally remove a null pointer check in a couple versions, which lead to ReiserFS breaking tragically over and over on my fileserver. A full fsck takes about 12 hours on my larger arrays, and I was less than happy to be able to run them 2 or 3 times a week. The kernel isn't at all perfect, but it's the best that I can find (Because OpenBSD and I just don't get along well.)
The point is, you're supposed to put fences where as little as possible has to go through them -- not just where they happen to look pretty.
Drivers belong in kernel space (1) so that "ideal" hardware can be modelled (all things that do the same job should expose the same interface), and (2) so that hardware operations can be sanity-checked. Sure, you could implement a filesystem driver mainly in user space with just a straightforward bridge to the hardware in kernel space -- but how can you be certain then that it isn't going to attempt something silly? And then it becomes harder to enforce consistency of drivers between filesystems, which makes it harder to replace one with another.
"I wonder if they are stupid enough to use the performance gains of Windows 7 over Vista as proof of the superiority of Windows development and ignore the gigantic performance degradation of Vista over XP?"
They already have! There's all kinds of hype about how much faster 7 is, in actuality comparing it to Vista. I don't know how much Microsoft drummed up and how much is from users, well, using it.
Anyway, I do hope the developers speed the kernel back up if it's really slowed down that much. And I think they will. As for (size) bloat, I'm unconcerned -- they've added support for more and more hardware, and that is going to take more code. As several people said, I can build a custom kernel if I want to, slowness is a concern but size is not.
"Yes, that has always been Linux' problem... no plan, no direction, no engineering, no thinking... only people adding more and more stuff, then replacing some of that with new stuff changing the whole ABI from one release to the next... I wouldn't applaud that, it's kind of sad actually."
It's not great. But there are UNIXes that are more "plan first, do later", FreeBSD and OpenBSD come to mind. Good consequences are what you'd expect -- stability and consistency within a release series, good code quality, and so on. The bad, driver support and features are not thrown in as quickly.
What I think Linus really should do is finally start a 2.7-series kernel for any serious changes people are wanting to do; instead of just making more and more 2.6-releases, work on stabilizing and speeding up 2.6 (it's very stable for me, but it'd be easier to fix regressions if it wasn't a constantly moving target..) and put new features into a 2.7 development kernel on the way to a stable 2.8. The older stable-series kernels (1.0, 1.2, 2.0, 2.2, 2.4) you could more or less get a driver for one version and build it on another (within the same series.) The incompatible changes were in 1.1, 1.3, 2.1, 2.3 development series... unlike now where each 2.6.xx may or may not have major driver-affecting differences.
"This sort of thing shows you up as 'not a software developer' or if you are, please god don't let you be one that i have to work with."
What a load of old codswallop.
Lean, well written and designed code is something all good software guys aspire to.
Unfortunately, a lot of people of calling themselves "programmer" learned a bit of HTML at school and think they can program.
Programming is an art and a very precise skill and too many people think they can do it without the proper practice, training and experience.
The evidence m'lud:
- Coders don't check function return codes;
- They allocate memory expecting it never to fail so they don't take the trouble to detect NULL from malloc();
- They don't check their buffers and overrun them;
- They assume that the stack is unlimited and stick massive great structures on there
- The use "handy" STL templates without checking if they really are efficient or appropriate for the application.
Holy cow I could go on and on.
I have seen complex systems that were written in C to run on DOS boxes that run for years without a reboot (seriously).
When you have to write code that has to run ALL the time, like for military systems, or at the local phone exchange, then you can truly call yourself "programmer".
...tell me what the memory footprint is before you load any apps...
Linux boots for me into about 230Mb of memory tops, whereas my current install of Win XP runs at about 450Mb after a fresh boot with about 42 processes (it's fairly lean by most standards).
Real world programing requires trade off and compromises, and things are a hundred times more complex that what you see in a beginning programming book.
They will show you an simple program, lean an efficient. But in the real world ....
Well that program only covers 95% of cases ... adding additional code so that it works in all situations and the program is 10 times larger.
Now defensively verify all your input and check all your function calls for errors.
OK now you have to add system wide error handling to detect and recover from errors.
Don't forget security. You'll have to replace all that fast and simple c style string operations with data structures which are not vulnerable to buffer overruns, filter all incoming data ....
Oh and guess what ... that book taught you to program like your code was the only code running on the system, but you are coding in a multi-threaded multi processor envirtonment. So rewrite everything so that no two bits of code can ever step on each others toes.
Even at the level of writing a simple routine ... you can't just write "good lean code" ... its all about design trade offs and compromises.
When you go up to the high level design its the same thing. You can make the system more modular which will make it easier to understand, easier to test ... but then everytime a program wants to read from a file on the hard drive it has to pass through 7 layers of carefully tested and verified services then back up.
You can cut out all the overhead but then you have one big layer that is so interconnected and complex that you can't really test anything in isolation.
Or you can design everything in the most efficient way possible, but that design is inflexible. Every time you add a new feature you have to redesign from the ground up. Or you can build a flexible system whose design is less efficient but can easily be extended to accommodate change.
Then there are business concerns. Lets say there is a way to make your OS twice as efficient only its going to take 10 times as many programmers, and an extra 5 years and will break compatibility with 3rd party software.
Its all about balancing conflicting demands, design trade offs and market conditions.
"Lou Gosselin is right to put the blame on "feature-bloated monolithic kernels"
Actually he isn't right at all. World + dog knows that Linux hasn't been a true monolithic kernel since before I learned to mix metaphors. It is only monolithic in the sense that drivers run in kernel space - in combination with udev, etc., drivers are never loaded into the kernel unless required by hardware, so unless you can find a way to do away with device drivers for an ever-expanding plethora of hardware, protocols and legacy systems, you're essentially up shit creek.
The so-called "micro"-kernel concept is a documented failure (xnu, L3, mach, excepting QNX maybe) in terms of both code-bloat, performance and maintainability when compared to kernels that contain other critical subsystems than IPC, Memory management and scheduling in the core code. Sure it's a great and elegant idea conceptually, but an abject failure at doing anything else except being a conversation topic among the University chattering classes.
Now the argument of exactly what should go in a kernel can be argued over till we're blue in the face, and sure Linux could maybe do with a prune here and there, but devices are supported by code, not Scotch Mist.
Banks don't hire armies of assembler programmers, they hire armies of COBOL & RPG programmers, small numbers of assembler programmers. The investment in legacy support is huge but you won't find many large financial applications written in assembler (IBM 360 or otherwise). The folks who are hired also have to be good at JCL (not difficult), TSO (yes it is still used), CICS, and other ancient big-iron environments. ISAM, VSAM, DB2, and Oracle databases also co-exist in these environments and they all have to be able to use each other's data if required, and quite often you will find UNIX and TANDEM in the mix as well.
Its great that he's not complacent, and hey, maybe if he highlights it someone will provide a plan (worth a shot).
If someone did come up with a solution then we can have our cake and eat it too!
Sneaky Linus ;)
The Kernel is evolution in action - good stuff that is of value survives, the rest dies, rots and finally disappears.
From what I have seen lately re the file system if you can _prove_ your case then ppl listen. Its the scientific method applied to software development... Which is why FOSS will win in the end... it is a modern scientific, capitalist development versus the proprietary, authoritarian, monopolistic, command economy model (and we know how that one will end).
I think user kernel modules are a bad idea ! The kernel *HAS* to be able to fully control access to hardware resources. With user kernel modules, just how can the kernel reliably be expected to do this without checking, and allowing/blocking everything the user module is doing ? I can see this being even more messy and resource intensive than having a kernel with all the hardware support compiled in. And also aren't user kernel modules just another attack vector to be exploited ? Or a serious risk of system crashes due to deadlocking of a vital resource ?
So if you want to slim down that kernel, then recompile it for your system - at least you have the flexibility to do that, albeit with a bit of research required in preparation. But if you're concerned about performance, you most likely have the background to learn what you need to do this, and enjoy it at the same time.
I think Linus is right though to observe that the kernel is getting larger, and losing some performance, but as has been stated there will be more error traps in the code, more features, etc. This should give a better end user experience with a more feature rich and stable system. So whilst there is maybe room for improvement (everything suffers from this !), I don't think the problem is severe. If there is a more effecient way to do something, equally stable in comparison with the current way, then lets include it. If it's bleeding edge, and not quite ready, keep developing it by all means, but it should not be in the stable kernel.
Now I've used vista, and I didn't see any performance problems that have been mentioned, though the machine it was on was reasonable spec, and I didn't run xp on the same box as a comparison. There was masses of eye-candy as is to be expected, and I even liked the northern lights screensaver. I just didn't like vista, and some features really started to grate, particularly the endless dialogue boxes warning that I might harm the machine by running something. But it did work, just not in a fashion that I like. xp is better, but still not my favourite system.
So I'll be sticking with my friendly bloated warthog of a Linux kernel, the Operating System and all the other fine software that I can run on it, whilst thanking all who contribute towards it for what they offer to any of us who want to use it.
Pretty much all documentation on kernel options is in options(4). A few aren't but they're generally the more rarely used and experimental options..
Get source. Optionally update from CVS. copy and edit config file. run config. run make depend. run make. copy kernel into place. boot. If it fails boot old kernel in single user mode. fsck filesystems. copy back old kernel. reboot.
Or run build.sh and cross compile from just about any platform to any other platform. It's a heck of a lot easier than on other systems and properly documented to boot.
Alternatively, run OpenBSD. Users are strongly encouraged never to recompile the kernel unless they have a good reason (such as deliberately commented out Perc RAID array drivers, testing >4GB memory support or hacking the i2c probe code to be less aggressive (problem on a very limited set of motherboards. Unfortunately mine is one of them.)). OpenBSD's user community can be extremely harsh though, particularly if you haven't RTFM first. Being insulted by Theo is a rite of passage.
Or FreeBSD. That's strongly module based, a bit different and probably a bit more accessible.
Good feedback on this article. I moved to linux as it allowed me to do what I wanted, as against MS. With the more recent kernels I have been having a few grumbles from my wife from boot up times on her machine... But thats probably more to do with the cruft I have snuck on here system. Heartbeat and cluster services, to name a few. Am now trying ro come to grips with dkms for a kernel driver that I need. Personally I shudder at the thought of drivers not being compiled for the specific kernel. Keep the hardware lean an mean, please!
I have just had to download and install a driver for a logitech keyboard and mouse, for a client! 68MB! Non windows certified so you have to have another bl**dy keyboard and mouse to get it working. She unplugged to to move some cords, and the thing goes through the whole sequence again, requiring a separate keyboard. Oh and you can just plug on that you know is identified in as the prompt about a driver halts any autodetection of hardware, you have to pull the plug and put in the other keyboard and mouse. Don't get me started on HP printer drivers (~230MB!) plus another (69MB) required to install .NET. I have some clients who are still on dialup, and cannot get broadband, thank god it wasn't one of those!
The linux experience is in a whole different league. Most things just work, and I know if I want to get things going faster I can compile the thing myself...( I never do though ) A nice friendly system for auto compilation and optimization would be nice.
Tux, cause he's big monolithic, alive and happy!
Yeah, and you seemed to have missed the point. 'Linux', the kernel lest you forget, has developed much faster than any Microsoft product. And marketing? don't get me started. Marketing was invented by assholes with an exceptional ability to enhance the truth. Keep watching those happy happy joy joy Microsoft Vista television ads mate.
If I say "Linux is bloated and huge" people will say "no, you are a Troll". Then later when Linus T says the same thing, then what? Can not people think themselves? Must Linus T explain everything? If I say something true, it is not true until Linus T had confirmed. That is ridiculous. No critical thinking, no logical thinking. Dogmatic view.
Not everybody's using an x86 ; my test gear was a sparcstation 20-712 that came empty, OS wise. I've got already a 10-512 that's happily running the latest supported sparc32/debian (etch) so I was willing to try something else, because of sparc32 being EOLed. FreeBSD : no sparc 32 support. Out. OpenBSD : no smp support on sparc 32 (at least this is clear from the installation web page). Out. NetBSD : no particular caveat, brags about being compatible with about every arch out there. OK, let's burn that iso. Hum. There are two conflicting statements in the INSTALL notes at the iso root... Can it go smp or what ? Check the mailing list archive : in -CURRENT, sparc32 smp is broken. But hey, -4 still does smp ! Let's burn -4. Install, boot, post-config, ok, seems to work, let's D/L some stuff. pkg_add -v windowmaker (yes, I know, serial console etc..., but that's beside the point). "Kernel attempting to lock itself against a lock", break to OBP. WTF ??? A couple of random crashes later, let's try to slim down the beast. Remove every obsolete bits in default pertaining to 4, 4c, 4d arch, config, depend, make... wait... wait... link fail ! Would you believe it, there's an undocumented ref in 'default.smp' to some 4d stuff in 'default'. Neither config nor depend gave me warning about it ! Goto config, etc.
Verdict : it somehow works for some values of work. I've less random deadlocks (but I still got some on occasion). My linux 10 has served me well for years, acting as dhcp provider, dns relay, and pop3 from my isp to local imap in my home network. The NetBSD 20 wouldn't cope with that reliably at the moment. It makes for a fine X terminal, which is a pity. Superior BSD stability my foot.
Flames, obviously.
... just how religious some people are with regards to software. I've read a lot of comments on here that just fly in the face of the facts.
For instance, the fact that you can compile your own kernel if the source is too bloated - this is a non-argument and doesn't even begin to address the situation. The fact is, whether you Linux hardcore'ers like it or not, the majority of Linux users now would not have a clue how to go about this. Linux is still not a contender with MS in the desktop market but this isn't the geek only option it used to be. You need to get used to that and move on, develop a proper Windows Killer and not just copy it like all the smartphones copy the iPhone.
This argument: "Linux getting slower and bloated while Windows (7) getting faster and leaner." Made me LOL. Saying that Windows is getting faster because of the improvement of Win7 compared to Vista? WTF? Have you forgotten how fast W2K was on new hardware and how XP was reviled as bloatware when it was released? Then suddenly when Vista was released the same thing happened again... the hardware wasn't up to the pile of poo that was installed on it. Now Win7's out it's the saviour of MS, the best offering since (the once hated) XP. But only because it's running on quad cores with gigs of RAM.
It's a shame that people can't see that there is a major hardware/software imbalance here. As hardware improves, the software should evolve with it and run faster, but it's just not the case in today's bloatware infested world.
It never fails to amaze me how fanboyism can kill rational thought in otherwise intelligent human beings.
Clearly most posters above don't know anything about Linux or efficient code for that matter.
Linux might be bloated and huge relative to what it was, but that does not mean it is bloated and huge when compared to Windows.
Linux is modular which means that only the modules you actually need get loaded. Thus, the wifi drivers and tablet driver for some odd-ball machine are not actually loaded unless they are needed.
Code efficiency is very important for the majority of Linux devices (which are not PCs etc). Most Linux machines are phones etc and efficient code means better battery life and cheaper phones. As a phone software designer, try to ask the hardware guys to build 1Gbytes into the phone. Expect a lot of laughter.
The limitations constraining performance go through generations: CPU speed, memory availability, memory bandwidth, etc. The design choices that make sense at one time don't necessarily make sense at another time and you're always playing off memory usage against speed etc.
Stop me if I'm on the wrong track here...
There is a difference between the linux codebase and what actually gets loaded when you run it. The difference is influenced by two things:
1. Things not complied at all
2. Things compiled as modules and not loaded at all
Now, from what I understand it, you only need to compile in things that are required in order for linux to start up and start mounting your disks so that it can start loading modules. So this basically means filesystems and chipset drivers.
Fine. So what's wrong with compiling *everything* else as modules? That way, you have access to *everything* that the linux kernel supports, while at the same time only needing to load the modules required to support the things you need? Other than the disk space used (which I doubt is significant these days) I don't see any downsides.
That just leaves the requirement of compiling in things that are required to boot. These can't be modules because the modules are potentially stored in a different partition to the boot partition, which may be on a different controller and use a different filesystem. Fair enough, that is a problem as the number of filesystems and chipsets is only going to increase over time.
So, how's about a new type of module - a "boot module" - which gets stored on the boot partition along with the kernel itself. I doubt it would result in any massive increase in storage requirement, since the kernel would be roughtly correspondingly smaller as a result.
So that way, the kernel binary itself becomes as "micro" as it can, it loads only the boot modules required in order to boot, and then only loads other modules as and when they are required.
Anyone see a problem with that?
Linux has become bloated and huge.
Mac OS X has become much slimmer and faster. Mac OS X generally is faster with every iteration, not slower. With Mac OS X Snow Leopard, it has also become much much slimmer.
Perhaps, Linux developers can use Apple as an example of where to go.
There, I've said it. Linux used to be efficient. Now, thanks partly to people bunging on extra features with little or no real thought as to the usefulness of those features, and partly to the insistance on using drivers built into the kernal, it's not.
The problem Linux has is there is no overall person in charge. Both Microsoft and Apple have very definite plans for their various OSes, and both have made massive inroads to making those OSes more efficient. They have also worked to ensure that where they do add new features, they are useful and efficient.
Now, some people above have suggested that the user can recompile the Linux kernel. True, they can. Why should they though, when Linux fanbois talk about how efficient Linux is? Also, for 98% of the population (who wouldn't know one end of a c++ compiler fron the other), recompiling isn't an option.
There is a simple reason why people are not recompiling it for speed, and why it's bloating out of all proportion.
Ubuntu.
Linux used to be for the pro's. You spent months fighting with a command line to get X compiled only to blow your monitor up when you mis-typed the H & V Sync. You had to manually recompile your kernel to add / remove things to make it work. You ahd to know your shit.
Then Ubuntu came along to make "Linux for everyone". Unfortunately, linux for everyone removed all of the technical challenges form the install, It is lowering the IQ of the collective Linux Userbase. Face it. 90% of the "OMG LINUX ROX!!!11!!" commentards are Ubuntu users wo run it for 3 months, feel superior, then quietly limp back too Linux.
Real Linux users know it bloats, and have been recomiling their kernels for years. It's a non-issue. However it's a massive fail for "Desktop Ez-Mode Linux" as you end up falling into the traps that dog your "most Hated OS in the World".
Welcome to your bed. You made it. You sleep in it.
@Russ: You are wrong i'm afraid. BSD uses a modular kernel. Linux is Monolithic. It loads the whole damn lot.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
