370 posts • joined Wednesday 15th October 2008 17:34 GMT
"£125m had been spent on software code and a further £27m on software licences."
£125m on software development isn't that outrageous for a system of this scale that has to be as bulletproof in terms of security as possible.
£27m on software licences is, however, completely ridiculous. Whatever happened with the recent noises made about using open source where possible/appropriate?
Re: Any drive
375PB isn't enough for you? Really? What are you using to generate that much transient data? For argument's sake, I have a medium load server (moderate MySQL load, Apache, several websites of various descriptions, mail, etc.) that has been running for a couple of years, and according to dumpe2fs, it has averaged 800GB of writes per year. That gives a figure of a little under about 467,000 years life expectancy (give or take a few centuries).
Anybody telling you that write endurance is an issue today either doesn't have a clue what they are talking about, or they are trying to sell you something (or both).
Re: iptables is your friend
I didn't change the point at all - if you read the iptables rules I proposed you will see that it was specifically about throttling connections to port 25 to 3/minute. The elaboration on the TTL part for mitigating source IP spoofing DoS vector was there to clarify why that part is included at all, but it was never the primary point of the exercise. If you'd read and understood the iptables rules that would have been clear to you.
If the OP's problem is that he is getting spam floods from a small number of IPs, then the iptables rules listed will alleviate the resource pressure on the MTA by dropping the TCP connections to port 25 from any source IP that has exceeded 3 new connection attempts in any 60 second period. The OP's problem is that his mail server is running out of resources due to excessive SMTP spam traffic. Dropping a substantial chunk of that traffic via iptables at TCP before the packet ever gets as far as the SMTP layer in the stack will reduce pressure on the MTA and help the server not run out of resources.
A similar technique can also be used to mitigate brute force dictionary attacks on any service (simply tune the port number, connection attempts, and time period variables to suit), within reason (obviously it won't help as much against a distributed attack via a botnet with lots of IPs). The point here is that the OP didn't specify whether this was a heavily distributed flood or a flood from a small number of IPs. Without any indication to the contrary, I assumed (correctly or otherwise) that this was caused by a relatively small number of IPs. Still, even if it is a hevily distributed attack, you can whittle away a lot of traffic by using a method like this - potentially enough to make a difference between a usable system or a complete DoS.
Re: iptables is your friend
You're missing the point - if the SMTP traffic is coming from one server (or a small number of servers), that server's connectivity would get limited at TCP level. Connections over the 3 new connections/minute limit would get dropped at TCP level, which means they would never deliver SMTP payload. Processing SMTP payload is expensive, while dropping TCP connections is nearly free. So if he were to drop most of the offending traffic before it ever got past TCP layer to SMTP layer, the server wouldn't get overloaded.
Re: iptables is your friend
The TTL consideration is to stop a malicious party from performing the DoS where they try to fool the throttling into thinking that an IP has exceeded it's new connections limit. In classing terminology terms:
Alice periodically sends email to Bob. Bob throttles new connections from everyone to 3 per minute. Trudy spoofs Alice's source address and starts issuing new connections to Bob. Trudy doesn't care that the 3-way handshake never completes, beause all she is trying to do is fool Bob's throttling system into thinking Alice has opened too many connections in the given time period. Thus, if Alice then tries to send email to Bob, Bob's throttling system will deny Alice's legitimate connection. If we add the TTL check, and if Trudy's discance from Bob is different from Alice's (pretty decent chance), the Bob's server will not treat this as the same connection (it sees a connection source as (IP,TTL) tuple, rather than just IP), and this will allow Alice's connection despite Trudy's DoS attempt.
Of course Trudy can try to figure out what Alice's TTL to Bob is and try to attack it harder, but that's an extra hurdle to cross and we are not really looking at fighting an escalating arms race, just provide a good enough first-pass solution that covers the basic defence.
The iptables throttling fixes the issue by making no server able to open more than 3 (or some arbitrary number, up to the OP to decide what they want to set it to, but 3 is a probably a good start for a personal mail server where very high bandwidth miling lists aren't used) TCP connections to port 25. If it wants to send more than that, it'll have to time out, wait and retry later. Since MTA won't see any subsequent connection attempts from the source server in that minute, it won't be chewing through all the resources it is chewing through at the moment (because it doesn't have to accept the connection or receive the payload to analyze it). Therefore the little SheevaPlug won't be DoSed and will continue working as it's owner requires. The only caveat is that the occassional email from a relatively high bandwidth mailing list might get delayed once in a while.
iptables is your friend
Have you considered something like the following:
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -m state --state NEW -m recent --set --name filter_$myip_25 --rsource
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -m state --state NEW -m recent --update --seconds 60 --hitcount 3 --rttl --name filter_$myip_25 --rsource -j DROP
iptables -t filter -A INPUT -d $myip -i eth0 -p tcp -m tcp --dport 25 -j ACCEPT
What this will do is limit your traffic to port 25 to 3 new connections per minute per source IP given fixed TTL (to prevent source IP spoof based DoS-ing).
On the odd occassion that regular email gets caught in this, the legitimate server will simply retry a few minutes later. But it should stop endless spam spewing that is DoS-ing your server with a negligible impact on legitimate mail.
If you want to be nastier and you have the TARPIT rule compiled into the kernel, you can use -j TARPIT instead of -j DROP, but only do this if you are very confident that legitimate mail servers will foul of this very, very infrequently.
Pretty much sums up our experience from every time we've used IE
Is it the scorched earth backdrop at the end of each use that is the most analogous?
Re: Wouldn't a vacuum be better?
"2: ZFS RAIDz3 (Yay!) - which should be good up to about 100Tb drives."
Yes and no. With 6TB disks based on unproven bleeding edge technology, I wouldn't want to push my vdev geometry past 4+3 in RAIDZ3. If you really meant 100Tb (as opposed to 100TB), I'd say you were overly optimistic.
But can it run Crysis?
Sure it can. Grid K2 is essentially a GTX690 without the video output ports (I have a GTX690 modified into a Grid K2 in the ESXi test box under my desk). How well it will run it depends on how many clients you are running off a single GPU.
As a rough comparison, a single Quadro 6000 (essentially a clocked down and shader reduced GTX480) on ESXi vSGA managed about 6 simultaneous 800x600@25fps Borderlands sessions (rendering and encoding into video). Of course, you need a client end that can decompress the desktop video stream coming off the server in realtime.
Re: Wouldn't a vacuum be better?
No, it wouldn't. Vacuum is not thermally conductive for convection, and the platter surfaces are cooled by the gas (air, helium, whatever) inside the drive. Helium is more thermally conductive than air in addition to being less dense, hence less drag and better cooling allow more densely packed platters.
What really surprises me is the 5 year warranty on the model. This is normally reserved for the expensive enterprise grade drives. Does that mean the He6 will be substantially more expensive per TB than the air-filled 4TB model? Or is Hitachi offering the extra long warranty as a sweetener to help customers swallow the technology that hasn't been tested for long-term reliability yet?
Either way, the size is just getting silly. The RAID rebuild times with drives like this are inevitably going to be waaaaay outside anything sensible.
Re: Biiiiiiig Changes
"Except that the DDR-RAM part won't forget what it stores when power is removed."
RAM doesn't forget when the power is removed for quite a while. The contents are recoverable for minutes afterwards using equipment that is cheaply available. This almost certainly extends to hours with specialized equipment.
XP 64-bit at 0 on the chart? That's pretty secure.
Shorting the Housing Market
"Shiller has pointed out that in the US housing market there wasn't (and still isn't, not easily) any method of going short on the value of housing."
While it is true that this has serious impracticalities if done by the actual owners (moving is a pain in the backside), banks do it all the time - you can tell by the amount of deposit they require. The deposit requirements reflect the risk perceived by the lender of the house prices reducing. They are effectively saying that the risk of:
(house price reduction + inflation) =< (deposit + repayment + interest charged)
So perhaps this is something that should be kept in mind by those arguing that the houses are really increasing in value (as opposed to price) at the rate the statistics show.
Re: Maybe it's the filesystem?
If you are getting hung up about performance when assessing ZFS you are completely missing the most important point. For small systems handling data that isn't particularly valuable, and where you are cash-strapped to the point where you absolutely cannot aford any more hardware, you wouldn't be using ZFS.
If your data's correctness is of significant value, however, alternatives simply do not cut it. I suggest you
The particularly relevant quote is: "Now for comparison, take a look at, say, Greenplum’s database software, which is based on Solaris and ZFS. Greenplum has created a data-warehousing appliance consisting of a rack of 10 Thumpers (SunFire x4500s). They can scan data at a rate of one terabyte per minute. That’s a whole different deal. Now if you’re getting an uncorrectable error occurring once every 10 to 20 terabytes, that’s once every 10 to 20 minutes—which is pretty bad, actually."
Layout Optimization - Humans vs. Automated Process
"It also encourages people to believe that they can do a better job of laying out an array than an automated process."
You can bitch about this all you like, but the fact is that no technology is available where an automated process does a better job than a good sysadmin who knows what they are doing.
That is not to say that the automated process cannot be made to be as good - it is merely to say that storage vendors haven't created one yet. They have a vested interest in doing a crap job here - storage is generally priced per GB, so they just want to sell you more storage to make up for a shortfall in IOPS. they are also geared toward creating kit that panders to the ignorant sysadmin that doesn't understand how the storage stack works (everything between the application and the spinning rust or NAND) - and there's a reason for that, too; they advertise to the PHBs as having a product that even an idiot can administer, therefore there is no need to hire expensive people who actually know what they are doing.
The LUNs are there for a reason. In traditional RAID, they allow you to partition your disks into RAID arrays and ensure that the expected workload's blocks (both file system and application) align appropriately with the RAID blocks. Doing your homework on this can make a very substantial difference to performance (in some cases multiples). With more advanced *AID technologies (such as ZFS) this changes somewhat, but there are still plenty of low level aspects to consider that will affect the performance.
Unfortunately, we are still a long, long way from any cheap, poorly optimized performance system can provide any required performance, no matter how poorly managed.
That's what I was just thinking. All the GPUs released are straight relabeling of the 7xxx series GPUs - for the 2nd generation in a row. The only new addition is the R290 - and that is as yet unreleased. Worse, the R290 only just roughly matches the shader count of the Nvidia Titan - and Titan has been out for months.
I was really looking forward to AMD finally coming up with a worthwhile improvement, but they have really dropped the ball. Again. The only real benefit from the re-relabeling is that the prices are being pushed down slightly.
Re: CDDL and GPL not compatible
"BTW: pi's already have BTRFS."
BTRFS is the most useless pile of steaming manure that has ever disgraced Linux with it's inclusion into the kernel tree. It's feature were _intended_ to rival ZFS but after years of development it has failed to even match the usability of ancient vanilla file systems like ext*.
It is also telling that EL7 will ship with XFS as the default FS rather than BTRFS.
If I didn't know better I might suspect that Oracle's continued pushing of BTRFS is nothing more than an attempt to dissuade people from using ZFS on Linux and thus assist them in pushing Solaris as having ZFS as the killer feature.
Pi support and 32-bitness
It's not the Pi support per se that is the limiting factor on the Linux implementation, it's generally the support for 32-bit platforms. ZFS was designed for a 64-bit platform with a very robust kernel virtual memory subsystem. Linux's kernel virtual memory is somewhat crippled (it's use is generally discouraged, as there are usually better ways to do things), and when you combine that with generally memory starved 32-bit platforms you run into problems.
The FreeBSD implementation works much better if you are stuck with 32-bit hardware. Or if you really want to run Linux on a 32-bit platform with ZFS, zfs-fuse works very well.
ARM Micro Devices
Only a matter of time...
Re: Bring back 5¼"
If the average error rate (which has remained constant for the past 20 years or so) is still rated at one unrecoverable error in 10^-14 bits (~11TB), that would mean that during RAID rebuilding you are on average going to lose 1-2 sector of data which you will only get back if you have backups and the file in question hasn't changed since the last backup.
4TB drives are enough of a liability as it is. Making everything bigger and heavier will also make it much, much slower (5.25" disks spun at 3600rpm) and less reliable (bigger platters are going to wobble more and the whole drive will be more sensitive to vibration).
No doubt with WD's usual lying SMART
Thanks, but no thanks. I would not touch WD or Samsung drives with a bargepole. They consistently do things like log a number of pending sectors, but when you overwrite them (which should cause a remap), the pending count goes to 0, and reallocated count always stays at 0.
That means that either the drive is just re-using sectors that have demonstrably gone bad in the past (and I doubt it checks them again after the write - WD disks don't have the Write-Read-Verify feature), or equally bad, it remaps them but doesn't say how many it has remapped. No doubt to reduce warranty claims.
Seagate (for WRV feature), Hitachi (for reliability) and Toshiba for me, thanks.
Re: Intel's compilers are targeted at marketing folks
ICC has always produced code that runs significantly faster than GCC, and despite having had a decade and a half to catch up and AMD backing, GCC has failed to close the gap by much. Every x86 processor since the Pentium MMX has had vectorization capabilities, which means that for sensibly written code (tall order for most programmers, but bear with me) you will get code that runs 2-8x faster (MMX can process two 32-bit ints in parallel, SSE can process 4 floats or ints or 2 doubles in parallel) than non-vectorized code.
The 8x faster extreme is a bit of an oddity from back in the Pentium 4 days - it broke up operations into micro-ops in a way that didn't work too well without good compiler support, so code generated by most compilers ran slower, but ICC generated code faster. Pentium 4 flopped because it needed a decent compiler to make it run code faster clock-for-clock than the Pentium 3, but with a decent compiler it was actually faster clock-for-clock (by about 20%) for suitably written C or C++ code.
Last I checked ICC generated code ran no slower than GCC code in the worst case (pure pointer chasing, no processing), usually 30-50% faster (e.g. MySQL, google for the pre-Oracle paper on the subject), and up to 4-8x faster on tight number-crunching loops (e.g. custom curve fitting functions I was writing a while back).
I'm an open source supporter and contributor, but it's important to give credit where it's due and not let FOSS prejudice cloud judgements. GCC's vectorization support was years late to the party and is still after all this time nowhere nearly as good as ICC's.
Re: Why helium ?
In a word - heat. Platter and actuator surfaces require some degree of cooling to stop them from overheating. Helium is more thermally conductive than air, thus a better coolant.
Vacuum has terrible thermal properties - there would be no convection heat exchange at all. Vacuuming the disks would reduce the possible density, not increase it.
"Unless there is something that I haven;'t considered.........."
Only that disk manufacturers want your disks to fail as soon as the warranty runs out so you go and buy new ones. Data growth only goes so far toward a sustainable business.
There are three issues I see with this:
1) In any even remotely realistic use-case, you will run out of another resource (CPU or network) long before you reach the level of sequential I/O performance this device can allegedly deliver.
2) The test graphs only cover sequential I/O, not random I/O. What is the 4KB random-write performance on this device after it has been "primed" with a full fill from /dev/urandom?
3) Being based on mSATA SSDs, this is little more than an 8-port SATA card.
So what is the big deal? You can easily achieve similar performance using a decent 8-port SAS card and 8 similar SATA SSDs (e.g. from Intel or Kingston).
Any improvements in there...
... that didn't also go into the LibreOffice code tree as well? No? Didn't think so.
Re: Oracle has to be there
More likely OpenJDK than Dalvik.
Jazelle was deprecated after ARMv5, on which it was optional. ARMv5 is so old that the latest ARM Fedora dropped support for it. IIRC the "replacement" for it the ThumbEE instruction set which is completely generic, not in any way Java specific, and targetted at compiled code (applicable to Java only in JIT context).
Re: Compiler Matters, but it's a fair comparison
You clearly haven't tried leveraging vectorization in your C and C++ code, have you; otherwise you would know that the performance and completeness of the implementation is lacking, to put it kindly. GCC is simply not yet up to the job of approaching performance of ICC generated code. Run some proper tests of your own on real world code you wrote and fully understand, rather than spewing nonsense about things you know nothing about.
Also, the tests used, from what I can tell, were biased toward gaming and graphical workloads which benefit significantly from vectorization. Whether that is the sort of load the average user runs on their phone may be questionable, but that isn't exactly cheating.
Compiler Matters, but it's a fair comparison
The main thing this proves is that GCC sucks compare to ICC, at least on x86. This is not news - we have known this for years.
Does this make the test suddenly unfair? Hell no! Intel have a fantastic compiler, and it is free for non-commercial and non-academic use (e.g. free for open source projects). If ARM have a compiler that produces similarly improved results over GCC on their processors, and they are prepared to make it available for free on the same terms, I'm sure a lot of projects will use it. If they don't,, tough shit - they were beaten on the basis of both processors being used with the best available compiler.
Im a big fan of ARM but it is important to give credit where it's due.
I have a Chromebook Pixel, and it's _fantastic_ - now that I've put full fat Linux distro on it.
Re: "most trusted services in the world if they actually desire to do so."
For some reason this seems like a particularly apt quote in response to Evil Auditor @ 18:54 08/07/2013:
"The reason it's better to have it done in hardware instead of by the compiler is it makes for less work for the CPU,"
How do you figure that? If both compression and decompression is done in hardware and the initial code is uncompressed, then the CPU has to burn power to compress the code in the first place, then de-compress it just-in-time to execute it.
If the compression is done by the compiler you load the compressed code directly at run-time, and you only have to decompress it just-in-time to execute. By having the compiler do the compression once (and it can spend a lot more optimizing the compression since it doesn't have to be done in real-time at run-time) you are saving at least half of the run-time work, probably more since compressing is typically slower than decompressing.
Re: Begs the question
A compiler in hardware? Whasn't that one of the many too-smart-by-half ideas that Java was supposed to bring?
NO TELLY for 90,000 Brits
I sense the average IQ rising already.
Re: Spinning rust will die
It's not the transfer rate - it's the average access time, fundamentally dictated by the spindle speed, which fundamentally dictates the number of operations per second you can do. On a 7200 rpm disk, you can only do 120 IOPS.
If you do the maths, you can infer from this that to read all the sectors (assuming 512 byte) in random (i.e. non-sequential) order on a 1TB disk would take more than 188 days. If the sectors are 4KB, the figure goes down to "only" a little under 24 days. Quadruple for a 4TB disk. Suffice to say for anything but bulk sequential access (e.g. one's movie collection transferred to NAS because DVDs were taking up too much shelf space), spinning rust simply isn't a sensible solution purely based on performance. And that is before we get into the issues of reliability.
Reliability-wise, my maths says that with consumer grade 4TB disks, anything less than 4-disk RAID6 is a liability unless the data is worthless or something like a paranoia-mandated tertiary backup the loss of which doesn't really affect anything important.
HP Microserver: ~£160 after cashback for the 2.2GHz one (probably about £100 for the older 1.5GHz one).
4TB disks: £160 (£640 for 4)
£780 seems a bit steep for a Windows licence.
Waste Oil, Not Fresh
The fundamental flaw in the way the biofuels directive is being implemented is in the fact that fresh vegetable oil is turned straight into biodiesel. This is, quite frankly, retarded. A far more sensible way to do this is to ONLY use waste vegetable oil. That way you are recycling and getting a useful benefit from something that has already been used up for it's original purpose and you are thus reaping an environmentally free secondary benefit.
Seen as Mali is going to be giving away free domains anyway?
Re: Wear-levelling works against you?
I think you are exaggerating a worst-case scenario based on small writes with big stripes. If your writes are small, you should be using small RAID chunks.
You speak of disks from a somewhat more reliable era for disks. My observation of the reliability track record of 1TB+ disks is that they, quite frankly, suck. The failure rate per year among my server estate (granted, only about 35 1TB+ disks, but it's enough to get the gist of the reliability you can expect) shows a failure rate of nearly 10%/year over 3 years (the worst model of the disk I have has seen a failure rate of about 75% over 3 years - yes, some really are that dire).
Of course, some makes/models are better than others, but the trend is pretty poor, both in terms of the number of complete disk failures and in terms of the disk defects (bad sectors) that arise. And bear in mind that every time you get a bad sector you are liable to lose that data.
The primary cause of flash failure is wear - and that is predictable and trackable. Spinning rust can suffer an electro-mechanical failure at any time, and often does with little or no notice. Early warning and predictability are important if you value your data.
Re: It's whether the degree is *hard* or *soft*
"I had lots of sex at Uni, and I still came out with a 2.1"
You do realize that isn't what *hard* vs *soft* in the subject line is about, right? :)
Full marks for eloquence
"Naturally these are a long way from the sort of jobs once dished out to graduates in the days when a degree was a rare badge of distinction, rather than something which fell out of your cornflakes packet following three years of alcohol-fuelled fornication."
I can see this sentence being recycled many times.
Re: Gordon Gordon Tom Maddox Gordon Phil Gordon Gordon AC Destroyed All .....
Matt, you have exposed yourself as an opinionated ignoramus beyond whom it is to even aspire to mediocrity. I would invite you to stop embarrasing yourself in public, but I am reluctant to do so for you might heed it and withdraw such an excellent source of Monty-Python-esque comedic amusement.
Facebook runs CentOS on their servers, not supported RHEL.
Facebook was also using GlusterFS since before it was bought by RH (not that that a company running CentSO would care).
Google runs their own "unsupported" version of Linux (IIRC derived from Ubuntu or Debian).
Those two between them are worth more than a non-trivial chunk of the FTSE 100 combined. Note - FTSE 100, there is no such thing as FTSE 1000 (although I'm sure you'll backpaddle, google your hilarious blunder and argue you meant one of the RAFI 1000 indexes not weighted by market capitalization - which considering that there are, for the sake of comparison, only about 2,000 companies traded on the London Stock Exchange, isn't going to be all that heady a bunch).
One of the biggest broadcast media companies in the country (not to make it to specific, but the list of possibles isn't that big, you can probably figure it out) runs various "unsupported" desktop grade MLC SSDs (Integral, Samsung, Kingston) in their caching servers (older Oracle SPARC or HP DL360 and DL380 boxes). They also run a number of unsupported software on their RHEL (supported) machines including lsyncd and lnlb.
One of the biggest companies in mobile advertising (after Google) runs their real-time reporting database systems (biggest real-time MySQL-or-derivative-thereof real-time deployment in the country) on MariaDB 10.0 alpha, because official MySQL wasn't workable for them due to the Tungsten replicator (commercial support available, for all good that will do for you when the product is a pile of junk) being far too unreliable.
This is just the first four that I can think of off the top of my head (having either worked there or having worked with someone who worked there) that completely obliterate your baseless, opinionanted rhetoric. You really should quit while you're behind.
Re: Gordon Tom Maddox Gordon Phil Gordon Gordon AC Destroyed All Braincells.....
## "....what is the value and usefulness of an engineer that is only capable of regurgitating the very limited homework that the vendor has done for them?...." It's called implementing tried and tested technology.
No, Matt. That's what you keep telling yourself because you don't know how to do anything that hasn't been done for you and pre-packaged for bottle feeding.
Re: Tom Maddox Gordon Phil Gordon Gordon AC Destroyed All Braincells.....
Seriously guys - ignore this guy's trolling. As the proverb says: "Do not teach a pig to sing - it wastes your time and it annoys the pig." His illiterate squealing is starting to grate.
For all his private parts waving with the supposed "FTS 1000" [sic] company credentials and having a RH support contract, there are almost certainly more people at RH that have heard of me than him (ever ported RHEL to a different architecture?). Not to mention that he has been talking about cluster file systems to someone who has been rather involved in their development and support (ever deployed a setup with the rootfs on a cluster file system? Using DRBD+GFS or GlusterFS?) without realizing how much this exposed his pittiful ignorance of the subject.
The blinkered view of "it's not supported by the vendor so of course I haven't tried it", had it always prevailed, would have ensured that the fastest way to travel still involves staring an an ox's backside. Seriously - what is the value and usefulness of an engineer that is only capable of regurgitating the very limited homework that the vendor has done for them?
Let it go. Let his ignorance be his downfall as he hides his ineptitude behind support contracts. Incompetence comes home to roost eventually. Jobs of "engineers" that only know how to pick up the phone to whine at the vendor get off-shored to places that provide better value all the time.
- Xmas Round-up Ghosts of Christmas Past: Ten tech treats from yesteryear
- Special Report How Britain could have invented the iPhone: And how the Quangocracy cocked it up
- Analysis Microsoft's licence riddles give Linux and pals a free ride to virtual domination
- Massive! Yahoo! Mail! outage! going! on! FOURTH! straight! day!
- Bring it on, stream biz Aereo tells TV barons – see you in Supreme Court