437 posts • joined 15 Oct 2008
Interlan roaming is a GOOD idea
"Mandatory roaming would reward operators who invested the least in their own rural networks, and increase intra-company haggling."
This is patently not true. Operators will charge each other. Yes, there will be haggling, but ultimately, a small operator that wants increased coverage will end up paying a larger part of their subscription fees to other operators that carry their calls. If an operator can accurately work out what their infrastructure cost is per call minute (which they most certainly do), then they can charge a few % more to the operators that are roaming calls to their network.
It means there is still incentive to have your own network with good coverage, and virtual operators, although they have lower costs, will also end up with even lower profits. This is how "cloud" provision works. You get between 1/3 and 1/2 of bare metal performance due to overheads, but you can spin virtual infrastructure up and down easily. The total amount you pay over medium term is considerably more than having your own hardware would have cost you, but there's no capex, only opex. It's a tradeoff, and if anything roaming would increase competition by introducing more virtual operators (which already exist, e.g. GifGaf or Virgin Mobile).
How is this different from what MySQL's InnoDB buffer pool has always done?
How long before...
... auto-troll feature is implemented in online forums and games? And how will we be able to tell it's not humans trolling?
Re: It all depends on whether you'd prefer to pay 80% tax or be hung from a lamppost
I guess it hasn't occurred to you that emigrating is a very real and viable option for most of those that would be required to pay the 80% tax rate.
Re: deja vu
"80% tax for the rich is so France 2012, I think Hollande's aides must have read the book."
And look how well that is working out.
Re: I mainly agree but...
"The tax and national insurance that you might have to pay during your life can be considered to be a very large liability."
By that measure the inequality is even smaller because those on high incomes have enormous liabilities in terms of lifetime tax and national insurance contributions.
Re: 4K screen prices are already dropping
Having tried it in the past, 40" is waaaay too big for desktop use. I find 30" is the comfortable limit. Nowdays I have a pair of 22" 3840x2400 monitors on my desk (separate VMs), as by far the best compromise available.
RBLs generally work in two ways as far as removal goes:
1) Removal requests (many require a payment to process removals)
2) Time based auto-removal
Just about all have 2), and most have 1).
You can try to chase 1) where available, but ultimately some will retain the IP until 2) takes place. So all you can really do is wait 2 weeks or so for the blacklisting to expire.
Re: Gullible Twat Dribbles into Beard
ChromeOS may be of diminished usefulness, but as has been mentioned in several places on this thread, there is nothing preventing you from putting a fuller flavour of Linux on one.
I run RedSleeve on my Exynos Chromebook, and it is at least as good as any other laptop that size, with better battery life on top.
"I put Linux on an old laptop"
Old laptop will have a battery life several times shorter than the ARM Chromebook. For some of us the purpose of a small, lightweight laptop is disconnected use while away from our desks.
I put full fat Linux on my Exynos Chromebook, and while not quite up to the spec of my ThinkPad T60 (2048x1536 screen was hard to give up), the weight saving and 6x the battery life made for a clear win for the Chromebook in the end.
"Linux can be installed to a memory stick"
Good enough for some uses (e.g. small server/firewall), but for general laptop/desktop use USB sticks (with the exception of the ones that are full fat SSDs with Sandforce controllers in a memory stick form factor, with an astronomical price tag) simply aren't up to the task - they make the user experience quite painful due to terrible random-write performance.
I look forward to seeing your numbers, then, when you reproduce the test I originally mentioned. We can then debate observations with some more numbers to compare.
"Again, you assert "worst case" that is, to be blunt...dated. I run huge databases virtualised all the time. Ones that pin the system with no ill effects and no noticeable difference to metal."
Whatever happened to your previous statement that it is only by pushing the system to the redline that we learn tings? Which of the two assertions isn't true? :)
"I also strongly disagree with your assertion that you cannot give up an erg of performance in the name of convenience; that may be your personal choice, it certainly isn't mine."
I said that "you don't necessarily have the luxury of being able to sacrifice any performance for the sake of convenience." I didn't say it is always the case, I said it isn't necessarily the case. To give you some real-life examples, if you are already paying £100K+/month on rented bare-metal servers from someone like Rackspace (I have several clients I do database consultancy for that pay at least that much for their server renting from similar providers), losing 40% of performance would also crank up your costs by a similar amount. That's not an insignificant hit to the bottom line.
"Specifically that "features" within most hypervisors to optimize RAM usage create a dramatic overhead on the system and they need to be weeded out."
If you speak of ballooning and deduplication, I always test with then disabled. I mostly use Xen, and with that what happens is that the domU memory gets allocated when the domU is created, and if ballooning is not enabled there will be no memory shuffling taking place. The domU memory is allocated and never freed or in any way maintaned by the dom0 - it's all up to the domU kernel to handle.
"There is also the issue that many virtualised systems = many OSes caching to RAM."
Again, I am not sure what difference that would make, since the caching is done within the guest, and the disks are typically not shared.
I'm not saying that memory I/O isn't the problem - I'm saying that I have not yet hard an explanation for it that makes sense. IMO the biggest difference comes from context the increase in the cost of context switching. This has been documented by several people who looked into it. I'm sure you can google it, but here are a few links for a start:
http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html (finds context switching is 2-3x more expensive on ESX)
The databases most of my customers present me with almost always start of badly disk I/O bound, and running so poorly that the entire system is falling apart. By the time I'm done with them with making sure they are appropriately indexed for the queries run against them, some query rewriting to work around the more eggregious unoptimizable cases, usually using materialized views, and a handful of other tweaks, they are typically running in the region of 10-20x faster, and are purely CPU limited (possibly memory I/O limited, but this is quite difficult to differentiate).
As for my data being out of date - I'm happy to admit that I have not re-tested the virtualization performance with the MySQL load since November 2012 (ESXi 5.0 or 5.1, I am not 100% sure which).
But the most important point I would like to make is this: "Don't listen to my numbers - produce your own based on your workload." By all means, use my methodology if you deem it appropriate, and/or point out the flaw in my methodology. But don't start with the assumption that the marketing brochure speaks the unquestionable truth. Start with a null hypothesis and go from there. Consensus != truth, and from what you said I think we both very much agree on this.
"I would have to conduct my own testing. My lab results consistently show an ability to saturate RAM bandwidth on DDR2 systems. Your results smell like an issue with RAM bandwidth, especially considering that's where you're pulling your I/O. I will look to retry by placing the I/o on a Micron p420M PCI-E SSD instead."
This implies you are asserting that running virtualized causes a substantial overhead on memory I/O, otherwise saturating memory I/O shouldn't matter. I'm open to the idea that the biggest overhead manifests on loads sensitive to memory bandwidth, although measuring memory bottleneck independently of the CPU bottlenecking isn't trivial.
"I also disagree with your assessment regarding near/far cores on NUMA setups. Just because the hypervisor can obfuscate this for guest OSes doesn't mean you should let it do so for all use cases. If and when you have one of those corner case workloads where it is going to hammer the CPUs ina highly parallel fashion with lots of shared memory between then you need to start thinking about how you are assigning cores to your VMs."
In some cases you may not have much of a choice, if you need more cores for a VM than a single physical socket has on it. For other cases, maybe you could get a little more mileage out of things by manually specifying the CPU socket/core/thread geometry - if your hypervisor supports that. I'd be interested to see your measurements on how much difference this makes on top of pinning the cores.
"Hypervisors can dedicate cores. They can also assign affinity in non-dedicated circumstances. So when I test something that I know is going to be hitting the metal enough to suffer from the latency of going across to fetch memory from another NUMA node I start restricting where that workload can play. Just like I would in production."
Sure you can, but testing with a simple base-line use-case where you have one host and one big VM seems like a good place to start assessing the least bad case scenario on the overhead. As you add more VMs and more arbitration of what runs where, the overhead is only going to go up rather than down.
I'm not even saying that the overhead matters in most cases - my workstation at home is dual 6-core Xeon with two GTX780Ti GPUs (and an additional low spec one), split up using Xen into three workstations, of which two are gaming capable. With the two gaming spec virtual machines having a dedicated GPU and pinned 3 cores / 6 threads, both on the same physical socket (but no overlap on the CPUs). The performance is good enough for any game I have thrown at it, even though I am running at 3840x2400 (T221). So clearly even for gaming type loads this kind of a setup is perfectly adequate, even though it is certainly not overhead-free. It is "good enough".
But in a heavily loaded production database server you don't necessarily have the luxury of being able to sacrifice any performance for the sake of convenience.
"Frankly, I'd also start asking pointed questions about why such workloads are running on a CPU at all, and can't I just feed the thing a GPU and be done with it?"
That's all well and good if you are running custom code you can write yourself. Meanwhile, the real world is depressingly bogged down in legacy and off-the-shelf applications, very few of which come with GPU offload, and most of which wouldn't benefit due to the size of data they deal with (if you PCIe bandwidth is typically lower than RAM bandwidth, so once your data doesn't fit into VRAM you are often better off staying on the CPU).
"That makes me very curious where the tipping point between my workloads and your simulation is."
Databases are a fairly typical worst-case scenario when it comes to virtualization. If you have a large production database server, you should be able to cobble together a good test case. Usually 100GB or so of database and 20-30GB of captured general log works quite well, if your queries are reasonably optimized. Extract SELECT queries from your general log (percona toolkit somes with tools to do this, but I find they are very broken in most versions, so I just wrote my own general log extractor and session generator that just throws SELECTs into separate files on a round-robin basis). You will need to generate at least twice as many files as you have threads in your test configuration (e.g. 24 files for a single 6-core/12-thread Xeon). You then replay those all in parallel, and wait for them to complete. Run the test twice, and record the time of the second run (so the buffer pools are primed by the first run). Then repeat the same with a VM with the same amount of RAM and same number of CPU cores/threads (restrict the RAM amount on bare metal with mem= kernel parameter, assuming you are testing on Linux). This should give you a reasonably good basis for comparison. Depending on the state of tune of your database, how well indexed your queries are, and how much it all ends up grinding onto disks, I usually see a difference of somwhere in the 35%-44% ball park. Less optimized, poorly indexed DBs show a lower performance hit because they end up being more disk I/O bottlenecked.
As I said, the I/O saturation was a non-issue because the write caching was enabled, the data set is smaller than the RAM used for testing, and the data set was primed into the page cache by pre-reading all the files (documented in the article). The iowait time was persistently at 0% all the time.
I am glad you agree that the machine needs to be pushed to the redline for testing to be meaningful. Many admins aren't sufficiently enlightened to recognize that.
On the subject of leaving resources dedicated to the host/hypervisor, that is all well and good, but if you are going to leave a core dedicated to the hypervisor, then that needs to be included in the overhead calculations, i.e. if you are running on a 6-core CPU, and leaving one core dedicated to the hypervisor, you need to add 17% to your overhead calculation.
In terms of migrations and near vs. far memory in NUMA, if you have, say, a 2x6 core dual socket system, an you dedicate one core to the hypervisor and the other 11 cores to the test machine, you are still facing the same problem of hiding the underlying topology so the guest OS kernel is disadvantaged by not being able to make any decisions on what is near and what is far - it all appears flat to it when in reality it isn't. While pinning cores will still help, the situation will nevertheless put the virtualized guest at a disadvantage.
Heavily parallel loads suffer particularly badly when virtualized because of the extra context switching involved, and the context switching penalty is still a big performance problem on all hypervisors.
I guess you didn't look hard enough. If you Ctrl-F and search for "esx" it shoudl find you the relevant part of the page. Including the first line in the article. ESXi scored second least bad, after PV Xen.
Hyper-V I don't use, so cannot comment on it.
Xen isn't that configuration error-prone - there isn't that much to configure on it. The only things that make any appreciable difference are pinning cores and making sure you use PV I/O drivers. In the case at hand, the I/O was negligible since everything was done in RAM with primed caches, so PV I/O made no measurable difference. Either way, you wanted a reproducible test case - there is a reasonably well documented one.
A few notes on the test case in question:
1) Testing is done by fully saturating the machine. That means running a CPU/memory intensive load with at least twice as many threads as there are threads on the hardware. For example, if the machine has 4 cores, that means setting up a single VM with 4 cores, and running the test with at least 8 CPU hungry threads.
2) Not leaving any cores "spare". If you have, say, a 6-core system, and you leave a core dedicated to the hypervisor (i.e. you give the only VM on the system 5 cores), you are implicitly reducing your capacity by 17%. Therefore, in that configuration, the overhead is 17% before you ever run any tests.
3) Pinning cores helps, especially in cases like the Core2 which has 2x2 cores, which means every time the process migrates, the CPU caches are no longer primed. This is less of an issue on a proper multi-core CPU, but the problem comes again with extra vengeance on NUMA systems, e.g. multi-socket systems with QPI where if your process migrates to a different core, not only do you not have primed CPU caches, but all your memory is 2-3x further away in terms of latency, and things _really_ start to slow down.
However, many VM admins object to core pinning because it can interfere with VM migration. It's a tradeoff between a bigger performance hit and easier management.
You may find overheads are slightly lower on more recent hardware (e.g. if you are using a single socket non-QPI system), since the above tests were done on a C2Q which suffers from the extra core migration penalty if the target core isn't on the same die as the source core).
On something like a highly parallel MySQL test the results tend to be worse than the compile test implies, but you'll have to do your own testing with your own data (replaying the general query log is a good thing to test with) as I don't have any publicly shareable data sets, and I haven't tested how well the synthetic DB benchmarks reflect the real-world hypervisor overheads.
As a matter of fact - I do.
I'll PM you later today with more details on other good test cases, but as a first pass you might want to take a look here:
"I personally see two conflicting assertions here: that VSAN is so much better than fully virtualised server SANs because it runs in the hypervisor, and that the VSAN-less hypervisor is so awesome it can easily handle any workload – like, er, fully virtualised server SANs."
The hypervisor isn't so awesome that virtualized workloads are overhead-free. At full hardware saturation point the performance hit from running virtualized can be up to 40% on some workload/hardware combinations.
"The Seattle-based server was demoed powering a full LAMP stack, running Red Hat Linux, Apache web server, MySQL, PHP, WordPress, and – this being an event for press and analysts – the obligatory self-congratulatory video."
I suspect you mean Fedora rather than RHEL. The latter is not officially available. The only available EL6 port available is Red Sleeve, and EL7 ports are being worked on by Red Sleeve and CentOS. RH made no announcement about RHEL7 on ARM yet, as far as I am aware.
Mean time to failure
It's not the mean time to failure that's the problem - it's rebuild time after a failure.
Re: I may be wrong but...
For a good overview of the biggest problem with lead-free solder, google "tin whiskers". It is a major problem, especially for hardware that has to have a long live expectancy.
Re: I may be wrong but...
"Well, on price they're about as good as nVidia for gaming, which is the usual use for a graphics card."
I am inclined to agree, if your use-case is a simple single-monitor non-virtualized setup which is what the vast majority of gamers use.
Having been up to my eyeballs in multi-monitor virtualized setups for the past year and a half, my finding is that the Nvidia drivers handle more complex configurations much better. They generally "just work", whereas AMD cards are problematic on most levels.
Sounds suspiciously like...
... 8xxM series is more or less a relabeled 6xxM series.
880M = 1536 shaders, i.e. 680MX
I do't see anything there to imply actual new GPUs coming out, at best it looks like a minor bit of fiddling with power management.
Cleversafe? You must be joking. They lost all credibility when they disappeared their earlier open source implementation from their website. And even if they hadn't the dispersed data storage only works when you have an incredibly fast interconnect between the nodes which demolishes most use cases they were touting for it.
For real-world use GlusterFS is a far more sensible solution.
"I guess IBM is talking about a similar thing to ZFS where it it only rebuilds used blocks on the disc."
While that works when the FS is mostly empty or contains only large files in large blocks, rebuilding a mostly full vdev full of small files can take much longer than rebuilding a traditional RAID because you go from linear read/write to largely random read/write (150MB/s linear speed vs 120 IOPS which could be 480KB/s on 4KB blocks). Then again, if your RAID is under load during the rebuild you are going to end up in the random IOPS limit anyway.
You do know Nvidia released the source for their Tegra GPU drivers, right? And they had done so months ago.
Re: Add another layer of indirection...
"Not possible if you are already running under a hypervisor..."
I guess you haven't noticed that several hypervisors have been shipping for years with support for nested virtualization.
Re: This way the virt did fly!
"Our testing showed only about a 9% maximum performance drop versus native tin"
On what RDBMS/OS/hypervisor? Throughput of carefully tuned MySQL on Linux under ESXi at full saturation (100% CPU usage with at least twice as many concurrent active threads as there are CPU cores, hot pre-primed caches, read-only) drops by about 40% when you run virtualized on the same hardware with full hardware virtualization support. KVM is a little worse, Xen is a little less bad, but the difference is in the low single-figure % points.
Did your comparison involve pushing the server to the limit with a highly concurrent load or were you just measuring latency under low load?
Re: Real world experience "does not count"
I have no idea what Hitachi's support is like (never needed to use it), but Seagate's RMA process is very smooth and problem free. I can vouch for it as I have exercised it very extensively.
Re: Curse of the bad model
Unfortunately it's not about a single unreliable model - it's about multiple consecutive unreliable generations. Backblaze figures actually excluded some Seagate models because the failure rate was so ridiculously high that nobody would have taken the results seriously.
The other problem is that most manufacturers are having bad models and even lines increasingly frequently. WD green drives with their aggressive spin-down leading to vastly premature spindle motor and bearing failures are such an example, not to mention firmware bugs that lead to gems such as pending sectors that are either not found by extended self-tests (WD, Seagate) or are located at sector numbers that exceed the LBA limit without HPA enabled (Samsung).
Does that mean the drive encrypts everything by default and just throws away the key to "erase" the data?
Re: ARM needs standards
Funny you should say that. A standard is exactly what has been established recetnly. At around the same time AMD announced their "maybe available to developers in the second half of this year" Opteron A series, there was also an announcement of a standard way (broadly similar to BIOS/EFI on x86) for all the hardware on ARM to be presented to the software layer (e.g. memory maps, I/O ranges, etc.).
Re: The other way around?
That is pretty much the size of it. I wasted a number of days getting various ATI cards to work fully before I eventually caved in and bought a Quadro 2000 for testing. As if by magic, everything started to "just work". Modifying Nvidia cards isn't too difficult if you just want an Nvidia card that works in a VM.
Re: Data storage for shared systems
Indeed, ZFS is very much the way forward.
Just FYI - you can to PCI passthrough on ESXi as well, and many people have successfully gotten it to work with modified Nvidia cards.
This, however, is probably not a particularly suitable solution because you would need another machine (e.g. a laptop) to run the VM management tools from, whereas if you run KVM or Xen the management can be done from the local machine.
Re: The other way around?
I'm not sure what features KVM has and supports, I use Xen because it is far more mature and performs considerably better.
Recently the Xen guys have been working on adding an additional reset method - bus reset. This may or may not make it into the Xen 4.5 release, I'm pretty sure it isn't going to be in the upcoming 4.4 release, so you are looking at at least 6-12 months before the feature is in the release branch and available pre-packaged for your distro. That is a long time to be hanging on for something that might but is not proven to solve the problem. The Nvidia solution works perfectly now.
It is also not the only issue I have had with Radeons - there are many others. For example, the XP drivers have utterly broken and you cannot glue together multiple monitors into a spanning resolution above 3200x1600, which is completely useless when I need to stitch together two 1920x2400 stripes to get my IBM T221 to work properly. There are other issues as well that I am not going to go into now since they are off topic, but suffice to say that Nvidia suffers from none of those problems.
I would strongly advise you to stick with proven hardware and software. Anything that is bleeding edge and unproven is going to put you at a very high risk of running into bugs and regressions in hardware, firmware and software. This is another reason why I strongly recommend you get one of the Citrix approved workstations for VGA passthrough. In terms of software, something like EL6 + Xen RPMs from the source I mentioned is a good, stable, proven choice, and since you aren't going to get RH support for anything involving Xen you might as well go with CentOS or Scientific Linux, or see if you can get a vaguely reasonably support package on XenServer (based on CentOS).
Re: It's nice to dream...
Actually, you can pass through the GPU better than the CPU. Unlike with the CPU which you virtualize, you can pass the GPU device completely to a VM. Inside the VM, the GPU driver loads as it would if it was running on bare metal and provided you have a GPU for which the drivers work virtualized - ATI technically works but is too broken to be usable in a serious way, Nvidia Quadro cards work without any issues, as do GeForce cards modified into corresponding Quadros, but unmodified GeForce cards don't work because the driver has a whitelist of PCI device IDs which it will allow to boot up virtualized.
Trust me on this - I am typing this on a triple-seat virtualized rig, of which 2 seats are for heavy-duty gaming (with modified GTX780Tis) , and the 3rd is my Linux workstation.
Dual Booting Same Instance of Windows native and Virtualized
This generally doesn't work particularly well. All the underlying drivers will be different, and it is akin to replacing the motherboard with a completely different one - Windows doesn't handle this gracefully at all, and you will more often than not find that instead of greeting you with the login screen it will greet you with a BSOD when it finds that the paravirualized SCSI controller it was expecting to be finding it's C: partition on doesn't exist on bare metal.
Maybe he just doesn't have the space (he already said he has no desk space for a 2nd keyboard). Seriously creating a setup like this is not difficult if you have hardware that isn't buggy. I have a 12-core (24-thread) physical machine, with two VMs given 4 cores (8 threads) each, and I can still run big software builds in Linux dom0 while having two L4D2 or Borderlands 2 gaming sessions on the go on the same physical machine.
Re: The other way around?
There are not one but two options for passing through USB devices in Xen. You can use PCI passthrough to pass the USB controller through, of you can use USB passthrough to pass a specific device through. The former is usually a little more efficient, but the latter is more flexible (e.g. if multipler ports are on the same USB hub and you need to pass different USB devices on the same PCI USB controller to different VMs. For example, I have 2 VMs with a mouse/keyboard passed to each one via PCI passthrough, and it works extremely well.
Re: Apologies in advance since this is going to be long
Forgot to mention - RH only support KVM, so you are out of luck support-wise with Xen. KVM support for PCI passthrough is nowhere nearly as mature as Xen's, so your chances of success with KVM may be diminished. If you really want support with Xen, you can probably get something from Citrix for XenServer (which recently went free / open source, and the most recent version is based on CentOS 6, i.e. EL6).
Forget BTRFS - it doesn't matter whose distro you use, it is not going to make a turd into a diamond. If you want a similar FS that works, use ZFS (look at the ZoL port). I wouldn't touch BTRFS with a barge pole. If you use ZFS, you can put your VM's disk block device on a ZVOL and get a lot of advantages such as performance overhead free snapshots. Again, this is the setup that I use on my similar system.
Finally - you would probably be a lot better off asking a question like this on the Xen users mailing list rather than here.
Apologies in advance since this is going to be long
I have been running a setup like this for the past year or so. It is pretty easy if you get the right hardware. It is frustrating to the extreme if you have buggy hardware. When it works, it works fantastically well.
The setup I use is triple-seat. EVGA SR-2 with 96GB of RAM dual 6-core Xeons, 3 GPUs, 3 monitors, 3 mice, 3 keyboards. EVGA SR-2 was a terrible choice - it uses Nvidia NF200 PCIe bridges which have broken IOMMU support. I had to write a patch for Xen to work around it, but now it works like a dream. If you are not averse of spending more on hardware I would strongly advise you to buy one of the (bare-bones) HP or Dell machines certified by Citrix for VGA passthrough use and build it up from there. Having a reasonably bug-free motherboard is essential if you want it to "just work".
I use EL6, with the Xen and kernel rpm packages from http://xen.crc.id.au/.
If you get a machine with on-board graphics, use that for your host (dom0). Once you have configured it all you can just not have a console plugged into it. Alternatively invest into whatever cheap GPU you can get your hands on for dom0 - it won't matter much what you use. My advice would be to get something like an Nvidia GeForce 8400GS since it is passively cooled.
For your domUs don't even bother with ATI - they work right up to the point where you need to reboot domU, and then the whole host will need rebooting.
Go with Nvidia. A Quadro 2000, 5000, 6000, K2000, K5000 or K6000 work beautifully, but if you are aiming for something more than a Quadro 2000 they are ridiculously expensive. Instead you can modify a GeForce card. My advice would be to pick one of the following three, according to your performance requirements:
GTX480 and modify it into a Quadro 6000. This requires a minor firmware patch, no hardware modification required. Details on how to modify it are here.
GTX680 and modify it into a Tesla K10. This requires a simple hardware modification, removing one resistor off the back of the PCB. Make sure you get a non-Gainward model (they exhibit a weird limitation in what video modes they can display in domU - my modified Gainward 680 and 690 only work in SL-DVI modes. My MSI 680 works fine with DL-DVI mode).
GTX780Ti and modify it into a Quadro K6000. This requires a hardware modification, adding one resistor across specific two pins on the EEPROM. This mod is easy to reverse, but requires taking off the heatsink which on most models means voiding the warranty.
For details on how to carry out the hardware modifications on the Kepler series cards (680, 780) see the thread on the forum here: http://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/
Whether the Nvidia card will work in a domU is purely down to the whitelist hard-coded into the driver, specifying which device IDs to initialize if the driver detects that it is running on a virtualized machine. The modifications described above simply modify the device ID, which makes the driver agree to initialize the card in domU.
Other than that, my setup is exactly like what you describe - pass specific PCI devices (USB, GPU, audio) to domU and it should all just work. With Nvidia GPUs you can reboot the domU as many times as you like and it will work fine. The only thing you will not get is the VM's BIOS POST and loading splash screen, but as soon as it goes into the GUI, you will get output on the monitor and it will work fine from there. As I said I run a triple setup, with two modified 780Tis for two virtual gaming rigs and they work beautifully even at 4K resolutions. The 3rd console is dom0 Linux.
If you are running Linux on Linux you might as well just use VServer (or LXC or OpenVZ). No need for full virtualization sucking away the performance.
I'd just like to point out there are several ARM boards available with SATA ports, including the recent SheevaPlug, GuruPlug and DreamPlug, TonidoPlug2, CompuLab SBC-A510, Cornfed i.MXQ board, Higher end SolidRun CuBox-is, TrimSlice (note: it's SATA port is actually running via a built in USB->SATA adapter because CompuLab couldn't get PCIe SATA working properly for some reason on Tegra2) just off the top of my head.
On the point of how long it takes - I'm pretty sure it would take me substantially less time to set up a DreamPlug with Linux from scratch to act as a DHCP, DNS and NTP server that it would take to install Windows Server and configure the services on an even remotely similarly specced Atom machine. The process for the DreamPlug would be:
1) Extract rootfs to disk (one-liner)
2) Copy modules/firmware to rootfs and the kernel/initrd to bootfs (2 lines)
3) Configure uboot to tell it where to get the kernel from and tell the kernel where the rootfs is (2 lines)
4) Reboot into new OS
5) yum install dhcpd bind bind-utils ntp ntpdate
6) dhcpd would need 3-4 lines in the config to tell it IP ranges DNS and gateway to serve plus any specifics for more fancy things, e.g. assigning specific IPs to specific MACs. NTP should need no extra configuration, and Bind comes configured by default to act as a caching name server.
All of that would take little more time than it did to type this message.
Re: High performance ARM
How are you meaningfully comparing revenue in a meaningful way between Oracle and a FOSS package?
Also how are you comparing the "volume" between MS SQL and a freely downloadable FOSS database? If we are comparing the number of downloads of MySQL (and don't forget to include the distro ISOs rather than just the deb/rpm/tgz as well as summing the totals for MySQL, Percona and MariaDB) with the number of copies of MS SQL shipped/downloaded from MS I suspect that the "volume" of MySQL deployments is some orders of magnitude bigger than MS SQL.
If you are going to compare statistics then make sure those statistics are meaningful in the context of all the items in the comparison.
This was discussed to some extent back when soft-float/hard-float issue on arm was being considered. In the end, the conclusion was that rather than coming up with a dynamic linker bodge that could handle both and rolling fat binaries that contained the payload for both, this taking up double the memory (or requiring awkward strapping that would purge the non-executable payload from memory after mmap()-ing it) it was more sensible to have a clean break. And you know what? This isn't such a big deal because the vast majority of software people run on ARM is FOSS.
The question is really why would you want a single hugely bloated executable that runs on multiple platforms? What problem does it solve that isn't much better solved by just having different binaries? You have to compile it once for each platform before gluing them all together anyway, so what does it actually gain you that shipping 4 different binaries (or a zip file, or shell archive containing all 4) wouldn't do just as well?
Re: "SUSE / Redhat for Linux stuff"
Aarch64 has not been in any official Fedora build thus far. Last time I checked there were only preview developer builds because there were still a few important packages that were broken on aarch64 due to package and toolchain bugs.
GCC took quite a while (a year or two) after ARMv7 hardware was easily available before the hard-float support was working sufficiently well to be suitable for full distro building. It would not be surprised if ARMv8 support at toolchain level takes a similar length of time to become sufficiently stable after the easy general availability of hardware..
Re: High performance ARM
It's funny you should say that Vendor only supports "enterprise class" DBs, and then you mention Oracle - when Oracle now owns MySQL. Oracle DBs don't even feature in the top 10% of biggest databases by size or throughput I have seen in the field, every one of those has been MySQL. Oracle is down there in the noise with PostgreSQL. I'm not saying that PostgreSQL isn't as good a database - far from it; I'm merely saying it doesn't seem to be as popular.
I'll let Lew answer the exact cost part (I had no idea he was reading the reg forums :)). I had already mentioned that seeing a product without a price tag was offputting.
Thankfully, for once the lack of a pricetag does not equal to a lack of a product.
Re: High performance ARM
Have you heard of ARM's big.LITTLE achitecture? It gives you 4 Cortex A15 CPU cores and 4 Cortex A7 CPU cores. The A7 cores are a little less powerful, but a LOT less power hungry, so depending on the load the system is on, it will use one or the other set of cores (or all 8 cores on the recent Exynos CPU systems such as the Arndale board). So ARM already provides this kind of functionality on a single slab of silicon.
As for your apps - if they are based on closed, proprietary technologies you are indeed screwed, but that was always going to be the case, starting with the cheque you write to the vendor for the licences every year.
Re: Database file compatibility
You can already do this with both MySQL and PostgreSQL. The on-disk formats and wire protocols are exactly the same, completely compatible, and endiannes independent. I do this sort of thing all the time.
- Updated HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
- Peak Apple: Mountain of 80 MILLION 'Air' iPhone 6s ordered
- Students hack Tesla Model S, make all its doors pop open IN MOTION
- BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
- PROOF the Apple iPhone 6 rumor mill hype-gasm has reached its logical conclusion