back to article Ubuntu 16.04 LTS arrives today complete with forbidden ZFS

Canonical will today (April 21st) launch version 16.04 of its Ubuntu Linux distribution, Xenial Xerus, the new long-term-support version of the project. As the name suggests, long-term support versions of Ubuntu get, er, long-term support, a guaranteed five years from today to be precise. The Xenial Xerus will therefore be …

Page:

  1. Paul Crawford Silver badge

    Priorities?

    "frustrate those looking for cutting edge enhancements" is exactly who?

    I doubt many want yet-more pointless GUI dicking around, or Firefox moving its drinks cabinet 6 inches closer to Chrome in removing useful features. Most folk like stability, and like the look and feel to stay the same unless they deliberately change it.

    For me having ZFS is a major useful step in it provides both data checksums and file system snapshots. Though it remains to be seen if this is well-used (e.g. can you snapshot the OS partition/share before any update/upgrade to allow roll-back without also rolling back your own data? Is that the default option in the upgrade manager?)

    systemd is another sore point. If you read Pottering's blogs it all sounds a good idea, and many aspects are. But the endlessly growing interdependency with it is a very bad thing, as its their borging of things that an init process has no need for (e.g. NTP) presumably because they were bored and did not want to fix bugs or, more importantly, user complaints instead.

    1. wolfetone Silver badge
      Terminator

      Re: Priorities?

      "systemd is another sore point. If you read Pottering's blogs it all sounds a good idea, and many aspects are. But the endlessly growing interdependency with it is a very bad thing, as its their borging of things that an init process has no need for (e.g. NTP) presumably because they were bored and did not want to fix bugs or, more importantly, user complaints instead."

      systemd will have it's own word processor soon.

      There's nothing you can do.

      Assimilate.

      Resistance is futile.

      1. Sam Liddicott

        Re: Priorities?

        "systemd will have it's own word processor soon"

        Systemd could well be the decent text editor that the Emacs operating system has been waiting for

        1. phuzz Silver badge

          Re: Priorities?

          "systemd will have it's own word processor soon"

          systemd is the One Ring to rule over emacs and vi alike.

          I'm pretty sure that that makes nano a hobbit in this analogy.

          1. el_oscuro

            Re: Priorities?

            You do realize that EMACS and VI are opposites, and any attempt to merge them will result in the Universe exploding and being replaced by something even more bizarre and inexplicable?

            1. Destroy All Monsters Silver badge

              Re: Priorities?

              Can someone please get to work and write "systemd in Action"...

        2. Tim99 Silver badge
          Joke

          Re: Priorities?

          @Sam Liddicot

          Why would I need a word processor when I can cat into a file, and then sed any changes I need from the terminal?

      2. Warm Braw Silver badge

        Re: Priorities?

        I suppose maybe that's why the article describes it as a "boot loader" - perhaps it will be eventually...

        1. NB

          Re: Priorities?

          it already is. https://wiki.archlinux.org/index.php/Systemd-boot

          1. Anonymous Coward
            Anonymous Coward

            @NB - Re: Priorities?

            I especially loved this paragraph:

            Note: Windows 8.x+, including Windows 10, will overwrite any UEFI choices you make and install itself as the priority boot choice after every boot. Changing the boot order in the UEFI firmware will only last until the next Windows 10 boot. Know what the Change Boot Option key is for your motherboard.

            To make Windows 8.X and above respect your boot order, you must enter a Windows group policy and have it execute a batch (.bat) file on startup.

            How lovely!

            1. davidp231

              Re: @NB - Priorities?

              I've never seen that happen... mind you I use rEFInd as my boot loader. The only thing it defaults to is that last OS it booted.

      3. Voyna i Mor Silver badge

        Re: Priorities?

        "systemd will have it's own word processor soon.

        There's nothing you can do.

        Assimilate."

        next version: systemdemacs

    2. Anonymice

      Re: Priorities?

      "For me having ZFS is a major useful step in it provides both data checksums and file system snapshots. Though it remains to be seen if this is well-used (e.g. can you snapshot the OS partition/share before any update/upgrade to allow roll-back without also rolling back your own data? Is that the default option in the upgrade manager?)"

      In this respect, just as with LVM snapshots - you can only snapshot whole partitions. However that shouldn't be an issue if you partition your disks properly. User data & sysem data are best kept to their own FS'.

      1. Anonymous Coward
        Anonymous Coward

        Re: Priorities?

        IANAFSE, but I didn't think that there was any concept of a partition in ZFS...

        1. Anonymous Coward
          Anonymous Coward

          Re: Priorities?

          Yyou can create multiple filesystems ("datasets") using "zfs create" and mount them on different mount points.

          Then at the individual dataset level you can set filesystem options, create snapshots etc.

          So you have the same sort of separation as you'd get with partitions or logical volumes with separate filesystems, but in a more flexible way.

          1. Adam Inistrator

            Re: Priorities?

            zfs isnt as flexible as btrfs IMO eg Cant remove physical extents and rebalance. btrfs is the default fs in suse. and apt-get automatically snapshots every upgrade and you can rollback or co-exist even version upgrades although it requires command line to do so. I doubt ZFS even comes close on that score. Pretty awesome.

            1. future research

              Re: Priorities?

              zfs isnt as flexible as btrfs IMO eg Cant remove physical extents and rebalance. btrfs is the default fs in suse. and apt-get automatically snapshots every upgrade and you can rollback or co-exist even version upgrades although it requires command line to do so. I doubt ZFS even comes close on that score. Pretty awesome.

              ZFS can now remove devices

              http://blog.delphix.com/alex/2015/01/15/openzfs-device-removal/

              at least 6 years ago opensolaris, you could do a upgrade, it would automatically snapshot the rpool and reconfigure grub for the new OS, but give the current one its own boot option for the kernel and a clone of the root filesystem pre upgrade that you could easy boot back into as if nothing have happen. (only gotcha was then manually upgrading the zpool verison on the disks that could not be understood by the old system).

              So ZFS can do the automatic snapshot and stuff, just needs to be implemented by the distribution.

      2. Bronek Kozicki Silver badge

        Re: Priorities?

        In this respect, just as with LVM snapshots - you can only snapshot whole partitions.

        that's not correct. In ZFS you can create multiple (nested or independent) filestems within single pool, and snapshot then separately. You can also create multiple independent volumes (block devices) and also snapshot them independently - these can be used to host other filesystems, while providing them with snapshotting and checksumming features implemented in the underlying block device i.e. ZFS.

        However native ZFS filesystems is often a better match for snapshotting, since it allows you to run a regular text diff against old and new files (old files are automatically mounted as read-only filesystem, as you access .zfs/snapshot hidden subdirectory). Very useful for /etc directory, if you are running ZFS on root filesystem. Oh as I wrote above, this /etc directory could be a separate filesystem with its own as-often-as-you-wish snapshots and filesystem options, without the need to create new ZFS pool (with data partitions etc).

    3. Anonymous Coward
      Anonymous Coward

      Re: Priorities?

      Getting the right time today is not optional. A lot of important things (i.e Kerberos...) won't work if your machine has not the correct time since the very beginning. Logs too need to have the correct timestamps since the beginning if you need to correlate them. Unless you have an hardware device to feed the correct time, you'll need NTP to get the time ASAP.

      It looks to me many complains about systemd is from people who wish to manage systems as if it was still 1970 and nothing changed in the past forty years. There are a lot of things that were optional back then, and are no longer.

      1. Anonymous Coward
        Anonymous Coward

        Re: Priorities?

        By all means the init system can *start* ntpd, but it shouldn't *be* ntpd.

        Having an entirely separate NTP daemon allows you to choose from multiple NTP implementations, developed and maintained separately. Indeed there is much useful work in this area (e.g. ntimed) and not having the option to swap this out is a major backwards step.

        The same applies for many other aspects of system operation - logging for example.

        See:

        http://www.muylinux.com/wp-content/uploads/2014/08/funny-systemd.gif

        1. Anonymous Coward
          Anonymous Coward

          Re: Priorities?

          Basic services needs to be bolted inside. Having twelve ntp daemons and thirty logging systems is utterly stupid. The system needs to know which those basic services are, start them early, know their state, to be able to rely on them. Not all daemons have equal importance.

          As I said, its a matter of people unable to understand Unix is forty years old and what was good forty years ago not is no longer good. There's a Bible/Quran/<put your religious book here> approach to Unix design that hinders true innovation. Unix was never created perfect. Today, it's mostly an old system based on outdated assumptions that requires a modernization. Just, too many people are afraid of the new, it rocks their quiet outdated world and required them to learn new things...

          1. EvadingGrid

            Re: Priorities?

            No !

            Replacing old with new does not automatically mean better.

        2. Chris Fox

          Re: Priorities?

          "By all means the init system can *start* ntpd, but it shouldn't *be* ntpd."

          Reading through various threads and bug reports about ntpd and systemd, it seems that some people experience various cyclic dependencies and race conditions, the very issues that systemd was supposed to fix. Some fixes for such dependency hell involve adding hardcoded delays, the very kind of hacks that systemd was supposed to avoid. It seems the only way that systemd can live up to its hype is by taking over everything, and ceasing to be the very thing it was meant to be, an init system. It's adoption by Debian and its derivatives seems premature to say the least. And the every increasing frivolous dependencies on systemd and its libraries is most unfortunate (if you switch a Debian installation to another init sytem, why on earth does installing CUPS or, even more bizarre, GIMP *require* you to reinstate systemd?)

          For those who want a clean, elegant init system, with scripts that are usually just a few lines long, I recommend runit (as used in Voidlinux). In my experience it has faster start up and much lower memory and CPU load than systemd (which is helpful on a constrained system like an Rpi), and, unlike systemd, it is relatively easy to debug if things do go wrong. Unfortunately it can be hard to switch to runit on Debian and Ubuntu etc., given all the wierd systemd dependencies. So much for "preserving init choice".

      2. SolidSquid

        Re: Priorities?

        From what I saw, most of the complaints were more methodology related rather than functionality. People were annoyed that it was one massive bundled binary rather than a lot of smaller components which could be maintained independently

        1. Destroy All Monsters Silver badge
          Mushroom

          Re: Priorities: "Busyware" vs "Stableware"

          People were annoyed that it was one massive bundled binary rather than a lot of smaller components which could be maintained independently

          Annoyed?

          ANNOYED?

          This seems to say more about the Clueless Tweeter "ain't doing no reading of dusty literature" Generation than about the "product" in question.

          I would be mad as hell. It would be enough to call for a good KKK-style expedition complete with wild night rides and burning torches in order to adjust attitudes.

        2. BitDr

          Re: Priorities?

          "One massive bundled binary".

          Exactly! Zero modularity. A massive potential for single point of failure. Systemd is a fine example of good intentions that aren't. To the person who commented about it not being 1970, why are you using *NIX then? Because it is better! Why is it better? Because you have (had) control, it is stable, it is robust and fixable with knowledge and a text editor. Detailed information about what is going on and troubleshooting can be had/done with the mark 1 eye-ball and a system console that, at the very least, speaks ASCII and has an RS232 port on it (finding a modern computer with an RS232 port is another issue).

          IMHO systemd is a solution in search of a problem, yes I CAN use it, and because I'm forced to I WILL use it, but I'm now pretty much through flight-testing BSD. If we can't move-forward with a Linux distro that is systemd free then BSD will be the OS of choice for our servers and will also be the direction in which all of our customer's servers are headed.

          1. Anonymous Coward
            Anonymous Coward

            Re: Priorities?

            I'm forced to use Linux because it is free, and people don't want to pay for software. Every day I have to face how much outdated Linux design is, how outdated its tools are, and how much workarounds I have to devise to make it work as I need. The whole POSIX standard is a big outdated mess as well.

            It's just like Emperors new clothes, nobody dares to say the emperor is actually naked...

            1. Destroy All Monsters Silver badge
              Facepalm

              Re: Priorities?

              I'm forced to use Linux because it is free, and people don't want to pay for software.

              Astroturf-kun, please.

            2. Striped Lungi

              Re: Priorities?

              "Nobody dares to say the emperor is actually naked"

              I will..just give me a platform to say it from.

            3. keithpeter
              Coat

              Re: Priorities?

              "Every day I have to face how much outdated Linux design is, how outdated its tools are, and how much workarounds I have to devise to make it work as I need. "

              What are your needs?

              Can you give an example of a kernel/OS that you consider to be less outdated?

              Coat: mine's the one with a 9front .iso in the pocket

            4. John Sanders

              Re: Priorities?

              TROLL

          2. PeteA

            Re: Priorities?

            Interesting ... I'm also in the process of migrating away from Linux and onto FreeBSD largely because of (1) ZFS and (2) systemd. And yes, I'm well aware that I can run ZFS on Linux, but there are significant differences in the encryption story that (IMO) make the BSD version much more compelling.

        3. DrXym Silver badge

          Re: Priorities?

          "People were annoyed that it was one massive bundled binary rather than a lot of smaller components which could be maintained independently"

          The thing is, it isn't a massive bundled binary. It consists of dozens of executables that have well defined purposes relating to their own area of concern and run with minimal privileges. And systemd doesn't implement an NTP daemon, but it does have a service that allows it to synchronise the local date & time with a remote NTP server during bootup.

          This is bad how exactly?

          1. Paul Crawford Silver badge

            @DrXym Re: Priorities?

            "allows it to synchronise the local date & time with a remote NTP server during bootup. This is bad how exactly?"

            Syncing time during boot-up is a good thing. We already have ntpdate that is typically used to step-adjust during booting to correct any CMOS clock errors, so why re-implement it?

            My point is not that the goals of systemd are bad, nor that init scripts have problems, but it is the growing number of systemd dependencies that mean you can't choose what and where.

            If we want another rant, as one commentard has mentioned systemd will put a failed process in to maintenance mode, so it has to be manually restarted even after the next boot. With init a future reboot gives it another chance, which is a better thing for overall reliability (if in doubt || watchedog timeout => reboot!).

        4. John Sanders
          Linux

          Re: Priorities?

          Why does everybody keep saying it is a massive binary:

          ls -lh /lib/systemd/

          -rwxr-xr-x 1 root root 1.5M Jan 27 15:23 systemd

          -rwxr-xr-x 1 root root 15K Jan 27 15:23 systemd-ac-power

          -rwxr-xr-x 1 root root 51K Jan 27 15:23 systemd-activate

          -rwxr-xr-x 1 root root 91K Jan 27 15:23 systemd-backlight

          -rwxr-xr-x 1 root root 47K Jan 27 15:23 systemd-binfmt

          -rwxr-xr-x 1 root root 99K Jan 27 15:23 systemd-bootchart

          -rwxr-xr-x 1 root root 352K Jan 27 15:23 systemd-bus-proxyd

          -rwxr-xr-x 1 root root 264K Jan 27 15:23 systemd-cgroups-agent

          -rwxr-xr-x 1 root root 87K Jan 27 15:23 systemd-cryptsetup

          -rwxr-xr-x 1 root root 292K Jan 27 15:23 systemd-fsck

          -rwxr-xr-x 1 root root 71K Jan 27 15:23 systemd-fsckd

          -rwxr-xr-x 1 root root 31K Jan 27 15:23 systemd-hibernate-resume

          -rwxr-xr-x 1 root root 328K Jan 27 15:23 systemd-hostnamed

          -rwxr-xr-x 1 root root 272K Jan 27 15:23 systemd-initctl

          -rwxr-xr-x 1 root root 299K Jan 27 15:23 systemd-journald

          -rwxr-xr-x 1 root root 336K Jan 27 15:23 systemd-localed

          -rwxr-xr-x 1 root root 612K Jan 27 15:23 systemd-logind

          -rwxr-xr-x 1 root root 39K Jan 27 15:23 systemd-machine-id-commit

          -rwxr-xr-x 1 root root 51K Jan 27 15:23 systemd-modules-load

          -rwxr-xr-x 1 root root 736K Jan 27 15:23 systemd-networkd

          -rwxr-xr-x 1 root root 123K Jan 27 15:23 systemd-networkd-wait-online

          -rwxr-xr-x 1 root root 35K Jan 27 15:23 systemd-quotacheck

          -rwxr-xr-x 1 root root 35K Jan 27 15:23 systemd-random-seed

          -rwxr-xr-x 1 root root 51K Jan 27 15:23 systemd-remount-fs

          -rwxr-xr-x 1 root root 31K Jan 27 15:23 systemd-reply-password

          -rwxr-xr-x 1 root root 508K Jan 27 15:23 systemd-resolved

          -rwxr-xr-x 1 root root 308K Jan 27 15:23 systemd-resolve-host

          -rwxr-xr-x 1 root root 71K Jan 27 15:23 systemd-rfkill

          -rwxr-xr-x 1 root root 135K Jan 27 15:23 systemd-shutdown

          -rwxr-xr-x 1 root root 71K Jan 27 15:23 systemd-sleep

          -rwxr-xr-x 1 root root 91K Jan 27 15:23 systemd-socket-proxyd

          -rwxr-xr-x 1 root root 51K Jan 27 15:23 systemd-sysctl

          -rwxr-xr-x 1 root root 1.3K Jan 18 20:54 systemd-sysv-install

          -rwxr-xr-x 1 root root 332K Jan 27 15:23 systemd-timedated

          -rwxr-xr-x 1 root root 135K Jan 27 15:23 systemd-timesyncd

          -rwxr-xr-x 1 root root 431K Jan 27 15:23 systemd-udevd

          -rwxr-xr-x 1 root root 272K Jan 27 15:23 systemd-update-utmp

          -rwxr-xr-x 1 root root 31K Jan 27 15:23 systemd-user-sessions

          Stop talking about something you haven't used or know much about.

      3. Dazed and Confused Silver badge

        Re: Priorities?

        It looks to me many complains about systemd is from people who wish to manage systems as if it was still 1970 and nothing changed in the past forty years.

        Most of the criticism of systemd that I see are from people who would like their systems to work and not have key things broken for them by a bit of SW that has decided to shutdown some key service because it feels like it. The Unix approach of doing one thing and doing it well has proved useful for 40 something years because it appears to be the best way of doing things. It's like writing SW using functions rather than trying to put all your code in one place and using goto's. There are many good ideas incorporated into systemd, but they would still be massively better if they were separated out into a series of individual bits that did what they were supposed to do and did that job well.

        1. Anonymous Coward
          Anonymous Coward

          Re: Priorities?

          "The Unix approach of doing one thing and doing it well has proved useful for 40 something years because it appears to be the best way of doing things."

          Until the one thing you're trying to do is dependent on another thing, which is in turn dependent on a few other things, meshing into an awful mess because each one is maintained by a different coder with a different methodology and none of them keep track of each other. End result? Mysterious and non-obvious incompatibilities that send your system to a screeching halt.

          Meanwhile, hardware is becoming much more versatile and interdependent in nature, resulting in highly-dynamic systems, something Init wasn't built for because hardware back then was more static in nature. Was the concept of hot-swapping, for example, even practical back then? Thus you end up with maintenance nightmares like network interfaces coming up in the wrong order (rather an important thing when one interface is outward-facing and one inward). Sounds to me like an argument over whether something's broken or not, with one side claiming it's not broken and the other saying it is.

          1. Dazed and Confused Silver badge

            Re: Priorities?

            > Sounds to me like an argument over whether something's broken or not, with one side claiming it's not broken and the other saying it is.

            No most of the complaints about systemd aren't from people saying that init can't be improved upon, it can and it needs to be. It's just that systemd has taken the wrong turn in it's attempt to fix things.

            There are other attempts to produce a better init, a better init is needed sure. But an init that thinks it is also an NTPd is not what the world needs.

            1. John Sanders
              Linux

              Re: Priorities?

              Oh for FCUK sake!!!!

              systemd-timesyncd is a daemon that has been added for synchronizing the system clock across the network. It implements an SNTP client. In contrast to NTP implementations such as chrony or the NTP reference server this only implements a client side, and does not bother with the full NTP complexity, focusing only on querying time from one remote server and synchronizing the local clock to it.

              Do you guys bother to read any documentation ever? What are you point and click sysadmins?

        2. Anonymous Coward
          Anonymous Coward

          Re: Priorities?

          Unix was never proven useful It's just people believing it's perfect by definition (in an almost religious way) and refusing any idea that changes their way of working - and utterly blind to the many issues Unix shows today. Unix is exactly a bunch of spaghetti processes invoked by shell gotos.

          If Linux weren't free, and thereby a way to save a lots of money, Unix would have not go anywhere. The downside it's true innovation is stifled by people still thinking a design for isolated system with a few users is still good today.

          1. Roland6 Silver badge

            Re: Priorities?

            and utterly blind to the many issues Unix shows today.

            People were blind to the issues Unix showed back in the 70's and 80's - being free and able to run across a large range of differing vendors kit was deemed more important as it broke the near stranglehold of proprietary systems and introduced the world to interoperable IT, particularly when many systems shipped with, the then new Ethernet 2.0 connector and (TCP/IP) network stack... Without these, I doubt the Internet would have taken off in the way it did in the mid 90's...

          2. RegGuy1

            systemd: [Unix]'s true innovation is stifled

            Unix is a god-awful operating system. It is so flexible. It will let you experiment with it to see if you can make it worse by replacing the ageing init process, that has served millions for donkeys years perfectly well.

            What sort of operating system is that?

            Give me Windows any day where I have absolutely no control over how I can configure it -- indeed I can't even control the version; I installed version 8 but I went for a coffee an came back and it had grown into version 10. Those clever dudes at Microsoft -- let me give them my bank account details as well. What could go wrong?

        3. John Sanders
          Linux

          Re: Priorities?

          "The Unix approach of doing one thing and doing it well"

          You do realise that many things like SystemV do more than one thing and not very well?

          Are you confusing the forest for the trees?

          I have been using Systemd for two years now and found it superior in every way to SystemV/Upstart (which is what RH/CentOS/Ubuntu were using before)

          This comes from someone who was very skeptic at the beginning, I started using it reluctantly (was forced) just to realize that not only it is good, but bloody well designed.

          It is perfect? Probably not, but trust me it is way better than SystemV at managing services and dependencies between services.

          Most of the extra functionality is optional, you use it if you want it, pure and simple.

      4. Paul Crawford Silver badge

        @AC Re: Priorities?

        You missed the point completely, and others have already done my bit to point out that systemd starting NTP is fine, but reimplementing it in a manner that won't work outside of the systemd universe is a pointless waste of effort.

        It is this aspect, of making everything dependent on systemd for reasons that are often seem like no more than vanity, that is the issue.

        1. Anonymous Coward
          Anonymous Coward

          Re: @AC Priorities?

          You miss the point that critical sub-systems need to be started early and their state fully known, and as long as they are outside you have an issue because you have far less control over them.

          If a machine is the average useless hosting webserver you may not care much about time, but in many other situations the correct time is critical.

      5. Androgynous Cupboard Silver badge

        Re: Priorities?

        > It looks to me many complains about systemd is from people who wish to manage

        > systems as if it was still 1970 and nothing changed in the past forty years.

        The complaints are from people who didn't think the old system was that broken, and question the need to throw it out for a new and relatively untested system, designed by a guy who has a track record of designing for quite a narrow segment of the Linux market, of not listening to criticism, and failing to fix bugs. Having tried, and failed to get pulseaudio working properly on numerous headless boxes I have run into enough incorrect or incomplete documentation that I distrust his approach and dislike his hubris. Especially for an init system.

        1. Missing Semicolon Silver badge
          Unhappy

          Re: Priorities?

          yes... and systemd still fails hard when a service can't start due to permissions problems (or selinux restrictions). If the service writes no log, systemd does a poor job of capturing output from the service executable so you can find out why it's dying.

          1. John Sanders
            Linux

            Re: Priorities?

            Gosh... (facepalm)

            Well, probably you are talking about a very old problem in the Selinux policies of fedora version 16 or something like that, and it was a Selinux caused problem that prevented messages going into the Journal.

            You could do this: StandardOutput=syslog+console and StandardError=syslog+console on your system unit

            The systemd.exec(5) man page explains the options you can set to control your daemon logging:

            StandardOutput= controls the redirection of stdout. Value can be one of inherit, null, tty, journal, syslog, kmsg, journal+console, syslog+console, kmsg+console or socket

            StandardError= controls the redirection of stderr. Values are the same as above with a slight difference for inherit explained in the man page.

        2. Anonymous Coward
          Anonymous Coward

          Re: Priorities?

          > track record of designing for quite a narrow segment of the Linux market

          Like the comments about opaque/wireframe move&resize in metacity, which boiled down to "my laptop works fine, if you're system can't keep up that's your problem not mine" which can only possibly have been written by someone who's never used a WAN in their lives.

      6. disgustedoftunbridgewells Silver badge

        Re: Priorities?

        I'm not anti-systemd, I think init was awful.

        However NTP is a solved problem and was solved before systemd was a glint in the postmans eye.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019