Drobo has jumped in to the enterprise storage market with the B800i and the B1200i iSCSI appliances designed to be the simplest devices of their kind. For the past month, I have put the larger of the two - the 12-disk Drobo B1200i - to the test. It has both served in my test lab and been pressed into production; I've tormented …
You're testing a drive array with Windows file copy? If you needed some information on how to properly test these things all you needed to do is ask. Any fool can saturate a link with Windows file copy because it mostly uses memory/cache for the copy operations, but that says nothing about the performance of the disk or array whatsoever. All you did there was test the NIC. To properly test disk you need a utility like SQLIO which bypasses any Windows cache and doesn't use files in the write operations (it does use files though). IOMeter can also work but has several issues and is no longer being updated. You also need to test with differing IO sizes to show how different workloads will be affected - for instance Windows 2008 and above use 8KB blocks by default rather than the 4KB ones in 2003, SQL does things a little differently as does Exchange, and Hyper-V is different again. Finally, you need to pump enough IO through to saturate the SAN cache to get good readings. If the SAN has 4GB cache you'll want to feed it a good 6GB of chuff before testing to test the actual disk subsystem, otherwise you'll find that all arrays saturate the links.
Once you know what you're doing you can then find the weak spots and bottlenecks of any system. We tested the 500k IOP Violin memory box and managed to find bottlenecks which are unsurprisingly missing from their literature. We also managed to saturate 2*8Gbps fibre links and get the latency up to over 100ms to show the customer that the advertised low latency is not a magic fix all after all, but will help with some kinds of workload (same with any flash technology) :)
Actually, I used over a dozen tools from iometer to yes, windows file copy. You can turn off the caching, but thanks for playing, chap.
Re: feeding it adequate amounts of data, I did. Some huge transfers, some small, some mixed, some random, some sequential. I ran 50 VMs off the thing then did OS image level backus on half while defragging the other half and storage vmotioning things around.
It may even shock you to know that I was more interested in real world performance than synthetics. Amazingly, that's the bit that matters to me. More critically, it is what matters to most readers, I would think. How does the widget perform for its intended purpose?
You have been quite interested in poking your nose in to storage comment threads, spreading FUD based on assumptions and claiming Deep Knowledge instead of asking questions.
Of course, it would be so much easier to put your mind at ease with a totally detailed breakdown of every test I've run and what configs they used, but the truth of the matter is that they don't give me that kind of space. (I already caught hell for the legnth of this review, and there is more to discuss than raw performance.) I honestly wish I had the chance to put more in. Especially with regards to performance with the different tools under various load conditions and testing various aspects of the storage system. Unfortunately, I don't get a choice except to present it essentially as a summary.
So in the future, ask questions. I'll answer. Or, more interestingly, answer the more interesting question: just what is it you are selling? You seem to have an angle, why not just put it on the table and we can all judge? Surely it's good, you seem knowledgable enough.
Can you give an example of FUD that I've spread? I don't have anything to sell, I'm vendor independent and just posting to help my fellow reg forum members better understand storage. As it happens I do have a deep understanding of storage technology which is why I'm talking about latency and IO size rather than posting screenshots of file copy operations. I also tend to do all of the Mathis behind these things before testing so that if I get a result which doesn't fit, I know that something is going a little wrong and that further investigation is required in the Violin experiment I mentioned, the customer was expecting 500000 IOPS because that's what the box said. I then had to explain why that isn't possible from a single server with 16Gbit storage bandwidth when using Windows default block size. I also have had to explain why NetApp can deliver considerably higher sustained write IO than other vendors for the same number of disks, or why the tiering in Compellent is so clever. I also spend a fair bit of time explaining the difference between enterprise and SMB, for instance in the case of this Drobo.
The FUD you are spreading is related to making assumptions that what you see is all that is tested, or that your assumptions about a vendor/solution are truth.
A great example is the BYOTL feature where you started in on how I wasn't going to feed 10GbE off of the spinning disks discussed in the article, etc. The article didn't talk about feeding 10GbE. I discussed it in the comments section as it was going to related to a future article and you immediate set in with "the storage you discussed in the article won't do that." Of course it won't. The storage in that article was designed to meet the needs of that article.
I can't process the nightly input of the Square Kilometre Array with the MicroSD card in my cellphone either, but that doesn't make the shiny new Class 10 I tossed in there any less fit for purpose.
In the case of this Drobo you are prancing about discussing bottlenecks, access patterns and tests as though you know from firsthand lab experience that it will have issues in various places. If you have results that say so, put then here for all to see. I will test them before I put it back in the box and/or gladly point the Drobo guys at your results and see if they can verify. I promise you they are entirely interested in characterising their hardware for all use cases, knowing where it is inappropriate to use and sharing that information freely with customers. They are one of the few storage vendors I know who don't try to fleece you.
No storage solution is perfect for all use cases, and I never claimed the Drobo was, nor does Drobo make such claims.The performance and reliability characteristics are such that the unit itself strikes me as "entry enterprise class." It is beefier than all but the more niche SMB offerings, and really does compete against some of the lower-end Netapp stuff on offer.
I could see the Drobo being used at a branch office to back-end local systems there, or for video editing, etc. Anything where the loss of the information on the unit isn't critical because relevant backups exist elsewhere and a time delay of a few hours between the last backup and what was on the storage unit isn't the end of the world. That actually covers a lot of scenarios; even in modern enterprises. (Domain controllers are a good example of this. They typically exist in pairs, and Server 2012 actually can tell if it is a VM that has been rolled back or spawned from a template.)
Synchronous high availability isn't a requirement for absolutely everything; must as systems administrators want it to be. It is, however, increasingly considered a baseline requirement and the ability of one of these B1200i units to keep itself in block-by-block lock-step with another one will be "required functionality" 3 years from now. IF Drobo isn't working on it for inclusion in the next generation of these devices, then the B1200i will have been an expensive experiment for them.
Does the Drobo B1200i have the raw IOPS required to compete against a Tintri or Violin solution? No. Could it reasonably see use in the enterprise in place of lower-end enterprise Netapp, Dell or HP offerings? Yes.
Now, if you want to have a little semantic quibble about where you define the cutoff between SMB storage and Enterprise storage, that's fine. State your precise requirements to be considered Enterprise instead of SMB and I'll tell you if it meets those. Lots of people have different definitions of the spaces involved – though of course, everyone thinks their definition is correct, and some spend a great deal of time and effort trying to convince others of that – but the lines are still pretty damned arbitrary.
I have worked with SMBs for most of my life. The Drobo B1200i would be complete overkill for 75% of them. It would fit comfortably in most of the rest of them and I can envision several scenarios in which if could serve my enterprise clients quite ably.
To me, that makes it entry-level enterprise gear. I'd like to see certain functionality added – block level synchronous N-to-1 replication for one example – added before I would put it up as "critical systems" gear, but it provides enough oomph to run many workloads in a modern enterprise.
So yes, I do call FUD, sir. At first blush, I read your statements across multiple comment threads as indicative of someone who believes in IOPS Uber Alles and specifies nuclear aircraft carriers in order to go round the block and pick up groceries, rather than tailoring solutions.
Of course, for me to level that as an accusation – rather than say "this is what it seems like, please confirm/deny – would indicate that I took a small amount of information provided then added an assload of assumptions in order to create a perception that matched my prejudices and desired spin on the situation. Right now, I don't know quite enough about you, but you really do strike me as someone who falls broadly in to one of two categories.
The first: a fanboy with an axe to grind: enter the conversation with a particular set of solutions in mind that you feel are "better than all others" and will slowly poke context-inappropriate holes in competing (or even not really competing in the same area) products until their solution is revealed to be "the best." Bonus points if you manage to take one set of requirements then completely ignore them by imposing your own requirements in their place, setting straw men to demolish in proving that the discussed solutions are irrelevant because they don't meet your requirements, which have nothing at all to do with the original requirements under discussion.
The second: a vendor. I've seen a disturbing uptick of late in astroturfing, and it has made me paranoid about vendors who spend way too much time and effort tearing on the competition in tech sites.
Let me be clear; sharing information is good. I like the sharing of information. But it is stupid comments like "as it happens I do have a deep understanding of storage technology which is why I'm talking about latency and IO size rather than posting screenshots of file copy operations" that make me thing you are either full of shit, or a just a douchy dong. That statement of itself is laced with a whole fuckload of completely inaccurate assumptions which you basically take as gospel and run with. That's FUD.
In the context of this particular comment, it I didn't post any screenshots. Oh, I took some screenshots. I took several of various things and put them all into a Dropbox folder and shared it with my subeditor. He then chose which to add to the article. I haven't had access to the CMS to do things like "add my own screenshots" until well after this article was handed in.
Secondly, your derision indicates that you find testing things using Windows file copy as a tool to be inappropriate. While I agree that it would be were it the only tool employed, I call several layers of bullshit on any so-called "storage expert" who doesn't bust out real-world use cases like windows file copy as part of their larger testing suite.
Synthetics are good for some things, but real world testing is critical too. IT is all about workloads, and sweet holy fuck, some people use storage to do things like copy files. In fact, it is one of the most likely use cases this Drobo model will see.
Making such basic mistakes as assuming things like "all tests performed are discussed in the article" or even "all screenshots taken are posted" makes me feel that the rest of your arguments regarding the technical aspects of storage are questionable. "Writers are subject to the whims and mercies of the editors and their restrictions" is a pretty basic, fundamental bit of knowledge to have about tech magazines, newspapers or the conveyance of information in the writer form in general. You either lack that knowledge, or you choose to ignore it in order to impose your views and prejudices.
You continue forward as though omission of information - or chosing to include some things instead of others - is concrete evidence that no other tests or assesments were performed. "What you see if what you get" taken to an extreme, despite the information being presented in tone and in legnth as a summary. What's more, the snarky comebacks - though oh, so witty to you I am sure - border on (but don't quite spill over to) ad hom attacks which rather undermine the highly technical nature of your attempted information dissemination.
What does that say about your approach to storage testing? Or whether or not I should trust your assessment of the utility or positioning of a product you haven't handled?
FUD, sir. This is how I interpret your comments. If you intended them any other way, please, do enlighten me. I am not seeking to be hostile or mean with my words here – I'm trying new things in 2013 – but I do feel that your approach and your words are lacking in both context and candor.
Unless of course, you are simply trolling. In which case, please do e-mail me. I have a few "how to troll people even more effectively" tips I love to share with greenhorns.
Wow you seem to have some issues. At no point have I said anything bad about the Drobo box on test, nor have I implied it has any flaws for what it is used for.
As for using "real world tests" a proper expert in the field of storage will test the SAN as a disk solution and the file server as a file server solution. Since you were apparently testing iSCSI connectivity you should concentrate on disk based tests rather than inappropriate protocol based ones. Appologies if you did submit those but I/We know nothing of your editorial process and assume the author is the one controlling the article and content. If editors are making your articles so poor in your opinion then I suggest you take it up with them. I actually thought it was a fine article for what it's worth and knowing things such as the rack kit can be invaluable. As you said, you have worked with SMBs mostly, and this comes across in your writing hence my initial offer of help in how to properly test disk solutions since benchmarking and low level testing is a big part of what we do in large enterprise environments. This is what gives rise to my comments about disk versus protocol, an enterprise customer (and any wise SMB) would care quite a bit if a 2GB memory upgrade would give a similar performance enhancement to adding another shelf of disk since the cost of each is wildly different. When I spec up a new SAN costing nearly a million pounds you can bet your ass that someone will want to know precisely why the current storage cannot be improved for a similar result. The vast majority of IT people I come across know nothing about the intricacies of storage, so I figure the odd comment on here about such things may be seen as helpful when the article contains none of the information. I appologise for any offence, none was intended and I'll try to avoid posting on your articles in future if I can.
If you really do work with storage on a professional basis then you know full well that iSCSI is more than "just a protocol." There are layers to how it is implemented and there is more to it than traditional "disk based tests." This is doubly true when you are dealing with a unit like the Drobo with is doing block-based tiering.
The unit as a whole item has to be considered. That means the disks, whatever controllers are driving them, whatever layers of cache exist, how the tiering/promotion works, how the iSCSI target software responds, how the network on the device is set up (remember, buffers can be very bad in layers!) and finally what targets and workloads you are layering on top of those disks.
There is more to that than IOPS. There are layers and layers of software, hardware and configuration here which may very well end up treating data accesses differently. In fact, given this is a tiering unit, we know for a fact that access patterns matter.
Indeed; one of the biggest takeaways from this was that using the Drobo's LUNs for traditional windows file transfer was a non-optimal scenario given the specific design considerations implemented in this unit. (An interesting result.) If I were simply using raw disk storage – no networking, iSCSI software, tiering, layers of caching, etc – I would have expected completely different performance characteristics than the unit evidenced.
The reason I didn't focus on disk tests is precisely because this isn't a million-pound array. Fully loaded, the thing is less than $15K. It isn't a unit designed for IOPS or targeted at those live and breathe IOPS. Thus things like "reliability, stability and fitness for common use cases" trump synthetic benchmarks.
Regarding "making articles poor" and my supposed "issues," I think you're rather out to lunch here. While I would love to include a billion data points in my articles – mostly to fend off bitchy internet piranhas – I don't remotely believe what ended up in the article is "poor." My editors don't make decisions about what words I use. They make decisions about how many words I get per article, and they choose which pictures go up.
I then make a choice: what to include in my articles, and I maintain that I chose the right things to include. So let me try to explain this clearly: I normally aim my articles at what I consider to be the widest bulk of readers our there: SMEs. Enterprises buying million-pound storage are a tiny chunk of the market. Even those enterprises aren't deploying their million-pound storage arrays to cover every use case.
The kinds of organisations that buy million pound storage arrays will get a demo unit shipped to them and test them in house. I seriously doubt most of them read reviews on The Register, Ars Technica or so forth. They don't get paid to read reviews, they get paid to do reviews.
Given the above, I don't see a problem, really, with the editor's decision to put up screenies with Windows file transfers. Nor to I get why you get your panties in a bunch that protocol-level testing is part of the suite of tests I consider normative for any storage I encounter. I do both.
You saw protocol-level screenshots and made the assumption that this was all I did. You didn't ask; you assumed. You then proceeded to comment such that – while you may have felt you were offering yourself as a helpful resource – certainly came across as rude bordering on ad homeniem.
You may not see it that way – in fact it seems fairly obvious you aren't able to see that about your own comments – but others have. (I was in fact alerted to your comments in both threads by e-mails from other readers.)
Comments on the intricacies of storage are welcome; education is good. It is when you tie them to assumptions about which tests have/have not been performed (rather than asking) and/or make assumptions based on – of all things – flavour images that the whole thing falls apart.
If you had for example said something to the effect of "hey Trevor, I am an enterprise storage guy; in my world we use these tests for these reasons. Did you run those? What did you get? If you didn't run them, why not? Could you? I am looking for this info."
Put like that, I'd have cheerfully dug up whichever statistics I had to hand, or even taken the time out on Wednesday to light the unit back up before I shipped it back and run some additional tests to provide you/other readers whatever additional info was desired.
"Did you know that" and "more info please" generally go down a lot better than "if you knew what you were doing you'd" or "No True Scotsman…"
Trevor, you write some nice stuff sometimes. But your comment handling is sometimes off target. This is one of those occasions. When you re-visit this page in a month or two, I feel sure you'll agree.
You did all that, but couldn't be bothered to actually put it in the review?
Then you attack someone who dares to challenge you on it?
That's shocking behaviour from a professional.
I did a lot more than what ended up in the review. I had the thing for a month and ran a whole barrage of tests. Not all of them were properly recorded, screenshot, etc. (Though many more were than ended up in the article.) A lot of tests - at least initially - were to see where the bottlenecks were, what performance was like...basically to "get a feel for the unit."
The majority of my benchmarking work was done with iometer, sqlio, vdbench, crystal disk mark, passmark's performance test, hdtune pro and window's resource monitor. What came out of the benchmarks was that this device wasn't going to win any IOPS contests.
When writing the article I felt that taking up a page or so with a pile of statistics that can be boiled down to "the performance is okay for the disk loadout, but ultimately is rather middling, and heavily reliant on the tiering to make up the IOPS on the transactional side" was not really a great plan. It wouldn't let me address the other aspects of the unit or discuss the tiering in depth.
I decided that I would include the bits of performance info on the Drobo that I felt most admins likely to buy a Drobo would care about: throughput, not IOPS. My theory – and obviously I was quite wrong about this – was that if anyone was interested in more info, they'd ask. That's what normally happens; people ask questions about extended information in the comments and – assuming I have it – I provide it.
Well, apparently when I am wrong, I am spectacularly wrong.
So what results is ultimately a clash of perception. I perceived – and still do, frankly – Lusty's comments to border on ad homeniem, with a soupcon of "ultimately, you need to buy X product." I didn't – and, rereading his comments I still don't – get "wants to be helpful/educate" from his words. Then again, I am human, I can be – and at least once a day I am – wrong. If so, then Lusty, I apologise.
I don't mind being challenged. I do mind being attacked. There is a difference between a friendly challenge and an ad homenim; my interpretation places Lusty on the wrong side of that line.
My response – while somewhat irritable – was not meant to be "attacking" Lusty in any way. I certainly did – and do – want to explain why things get written as they do, what the choices made were and why they were made. I also wanted to find out what he wanted, what info I could provide him.
So no, I have no problems with someone who wants more information, or would prefer I included X information in an article. Criticism - and believe it or not, I do consider Lusty's comments valid criticism, not mere opinion – helps me determine what to put in future articles. Or, in this case, is putting more emphasis on my need to get a secondary site up where I can post "raw numbers" details for such reviews that I can then link to from a main article.
How one goes about expressing their desires for such information, however, will determine how your requests and thoughts are received by the individual you are addressing. Collectively, we internet denizens seem to have an arrogant opinion of ourselves and a sense of opinionated entitlement that is patently unreal. We demand the #deity-given right to attack writers on the internet personally and professionally yet we take notable umbrage when those same writers should dare to ask "so who exactly are you, and what do you want?"
The writer, we say, should serve as a psychic sink for our own narcissism and desire for personal recognition; they must accept beratement and chastisement from any and all quarters without questioning those who question them. They should accept us – anonymous blobs of text – as experts on all subject matters, especially when we disagree.
Agree with me, we demand, not him or her or them! Be polite to me, writer man, even if I am utter trash to you. This, we call behaving like a professional; a one sided concept as soon as the internet is involved. To be professional on the internet, you see, is to treat even the vilest of anonymous commenters with respect, patience and – preferably – even some fawning adoration. It is – apparently – deeply unprofessional to respond to someone in kind; never challenge the assumptions, motives or tone of the anonymous block of text!
The worst part is…both sides of that particular argument are right. If you bend over backwards and mollycoddle every douchebag on the internets you're going to end up wasting a lot of time and quite possibly killing yourself in a drunken, depressed stupor. The internet is a cyclone of shrieking trolls; many of whom take great delight wasting your time and ruining your day.
By the same token, you do want to be polite, respectful and – dare I say it – professional to as many of these folks as you can. Many of them are decent, hardworking types who are looking to make a contribution to the world and hoping you'll help them do so.
Sorting the shrieking troll (who ultimately is going to be a net drain on your psyche and time) from the indelicate nerd (who ultimately would be a great guy to know personally) is hard. Hard enough that we all get it wrong from time to time.
Reading people, interacting professionally, negotiating, helping and basically doing that "transfer of information in a social context" thing is a hell of a lot easier in person (or even over the phone) than it is in text. You get queues there that you don't in text form. So how things are presented in text makes a huge difference.
Most writers – especially tech writers – refuse to interact with commenters. They won't read the comments to heir own articles and they certainly won't respond. General douchiness of commenters is the biggest cited reason, but "inability to tell if serious, trolling or just horribly socially inept" is the second most frequent one I've heard.
Thus far, I've chosen not to be one of those folks. I've had such positive feedback from many of my readers about the fact that I take the time to answer questions and interact that I have chosen to maintain the practice. It could be that I was wrong to make that choice; I will be giving it serious reconsideration.
I have always taken a "give as good as I get" approach to comments. I have felt that anything else would generally lead to insanity. Commenters are right to expect some respect – and maybe even the benefit of the doubt – from writers. There seems to be distinct disagreement here over whether or not a writer has the "right" to expect the same of their commenters or not.
Is professionalism a one-way street on the internet? If so, then the other tech writers are correct and I have been wrong all this time: I should never answer comments at all. Nobody can reasonably be expected to maintain absolute decorum while being the psychic whipping boy for 7 million readers who are not held – externally or by the community to which they belong – to any similar standard.
The whole topic is – obviously – something I've given a lot of thought to. Beyond the traditional commenttard "you've got problems" response - trotted out by the internet's most brilliant minds whenever something is discussed in depth - I am interested in the thoughts and feedback of all and sundry regarding the above.
Whether you believe it or not I was genuinely trying to get you to include better testing. If you look at your article from a storage expert perspective, the first thing you notice (and it screams at you) is the screenshot of a windows copy operation. I've just had another look and it's clear you did have IOMeter running so I appologise for assuming you didn't know about it. As I've mentioned, this "throughput" figure is worse than useless even in your percieved "real world" scenarios because in your limited space you can't possibly justify why the number is what it is. In that screenshot, we see a point in time figure for throughput to the SAN without context. More useful in future would be a perfmon session graph with throughput, average seconds per transfer (latency) and and disk transfers per second over a 30 minute period running on the physical disk object related to the SAN drive. You say you ran lots of different benchmark software, but generally only one is necessary if you have well planned tests to find for instance throughput, random IO, sequential IO but before all of these, the benchmark tool needs to be used to configure the system optimally because without driving IO and knowing what number to expect you won't know if it's even configured properly.
I realise by now that you have some kind of issue with IOPS - I don't see these as the be all and end all of disk, but the cold reality is that when a SAN has finished its cache, the resulting performance is based on IOPS, RAID and IO size and always will be. Latency is a by product of some of these things, along with number of hops and IO queues at the client, and even simple mistakes like using the native switch VLAN for iSCSI traffic rather than separating the switch management traffic. If a single SATA drive can do 125 random IOPS and you have a default Windows 2008 server install and formated disk then that SATA drive can give you around 1MB of sustained throughput (125 IOPS x 8KB (IO size) = 1000KB throughput. A SATA drive can manage around 10x the IOPS in sequential, so 10MB/sec sustained throughput. If you mirror two drives you get the same performance, stripe them and it doubles, RAID 5 and is's roughly 1/4 per drive so 4 drives will give you the performance of a single drive.
When you move to SSD, this is more important because they are (usually) capable of effectively unlimited IOPS, and are often limited by the bus. If you use the same 8KB IO on a 3Gbps SATA link then you'll get around 50000 IOPS from the disk if the controller is up to it.
The important thing to keep in mind with all of this testing is what the limit of each component is, and therefore which bottleneck you're hitting. If you're seeing throughput of around 100MB/s then it's likely a gigabit ethernet bottleneck. If you're seeing that same number on a SAN with 4 SATA drives then you know you're writing to cache and that eventually you'll get a massive slowdown if the data continues to flow. Even with the SSD tiering, you'll still hit this point eventually but you'll need to fill the cache first and then the SSD tier, although the cache will be emptying while you write data at the speed of the disk so it's like a bucket with a hole in the bottom - a 1GB cache won't fill with 1GB of data written.
It's also important to note with iSCSI how the MPIO works since the iSCSI session and connection management allows many variations. With standard MPIO using 2 x 1Gbps sessions (default if you let Windows set up the connection) then you'll get 1Gbps throughput since it will likely be failover only. If you muck about and set up a single session with 2 load balanced connections (not LACP) then you can get 2Gbps throughput. This is where you'll want the massive sequential write testing scenario to determine how the links saturate, often the DSM will be buggy and you'll need to choose a "non optimal" load balancing algorithm because the optimal one won't actually load balance.
Most important of all is to know, and to point out to customers and readers alike that B and b are different beasts. The vast majority of IT staff won't know that 6Gb SAS can "only" do around 750MB/s and 3Gb can do 375MB/s. You and I will know that these are the same numbers, but it's surprising how many people think the lower numbers are down to some kind of "protocol inefficiency" just like when formatting a 1TB drive Windows doesn't actually use up hundreds of MB of space, it just uses a different unit to measure the same space.
I don't have a "problem" with IOPS at all. I simply didn't feel it was as relevant in this particular context as it was in others. You'll note that IOPS figures were included and discussed in the Hyper-X review because I felt that in that context, they mattered.
I ran lots of bench marks because I was personally curious. I simply threw things at the unit, collected data and then sat down at the end of it and decided what would make a good article that would appeal to the majority of my intended audience.
That said, for future reference – and as an item I hope you take the time to reflect on – you will not get a writer to "include [x] in their articles" with borderline ad homenim posts. If you take the time to read my interactions with other readers, I think you'll find that I am open to providing information as I can, and even work to include the better suggestions into my articles. (Or write articles on requested topics.)
Most writers don't take well to attempts to berate them into subservience, no matter the technical ability of the individual typing the comments.
I don't disagree that with you at all that the things you talk about are important. You are elucidating critically vital information to convey to the readers, however I believe that the context is inappropriate. The topic at hand is completely inappropriate to include as part of the review of this device; it should be a feature unto itself. "Here be the basics of storage: learn it, motherfuckers" or some such.
I think it is an important topic, and despite the animosity of the presentation, I believe it is worth pitching the features editor for an article later in the year. I will add it to the list.
Trevor, I still think your initial response was on the... "hasty" side (I didn't see it as brow-beating you into using IOPS, but rather asking for something more than a screenshot of a Windows file copy dialogue box) but I have to commend you for the time and effort you've put into responding to the issue. Thank-you. Having done a few reviews myself, I certainly understand that sometimes it seems it's all criticism and no gratitude.
@Chz My initial response may well have been on the hasty side. (To err is human. To err a lot is what happens when the coffee runs out.) I've reread it again, and I still see him as attempting to berate yours truly into submission. But if I am going to be a douche on the internets, I am going to be a verbose douche that explains himself so you can judge the totality of my douchiness with all the relevant context.
I am, however, sad that the conversation re: "is the expectation of professionalism on the internet a one-way street wherein those who write are expected to maintain a superhuman level of stoicism and those who comment are encouraged to be soulless self-respect stripping piranha-harpies" never got born. I think it has a real impact on what we – as readers of this fine ball of redtop satire – want this site to ultimately be.
Ah well, that's my problem though, eh? I actually worry about the underlying philosophical issues and long term results that are the inevitably born from the choices we make (good and bad) in our interactions with one another today. Ah well, off to write other things…
... rack mount servers held in just using the bottom two bolts before - makes my screwdriver hand start twitching as I try to work out which patch panel (usually held in with 4) donate a few spare ones, but probably not something that would cause immediate worry... how heavy are these storage things? Most servers, SANs, are still only held in by a few bolts even if they are in a sliding mechanism.
Not quite enterprise level installations, but at the bottom of each of the cabinets in systems I've managed or been involved have been properly heavy things like UPS - definitely not something that would go at the top of the rack!
Interesting report as usual...
Re: I've seen...
It's actually not as heavy as it would seem at first glance. I don't have the shipping weight on hand - that's at the office - but it is far lighter than a fully loaded Chenbro SR-107, that's for sure. Drobo have put rather a lot of effort into designing the thing; it would probably hold up. I just...couldn't make myself do it. IT was just heavy enough that one bit of metal fatigue in one ear...
...well, onto the bottom of the rack it went. If I remember - I'll try, honest! - then when I get to the office on Wed and we have to pack this thing up, I'll see if I can find the packing wieght for you.
Re: I've seen...
I've seen rack mount servers held in just using the bottom two bolts before - makes my screwdriver hand start twitching as I try to work out which patch panel
Considering all the weight of drives are at the front, the cantilever effect of the bottom two screws works quite well. The top screws are to help if a bolt snaps. That said, I always use four bolts.
I see a lot of people testing VMs with ISCSI. Why? (and I'm honestly wondering)
I've only ever used iSCSI for backup stores, ie BackupExec, DPM, Veeam, or ShadowProtect backup destinations, and used Fibre Channel or NFS for VM stores to make vMotion simpler to manage.
Re: I've seen...
"I see a lot of people testing VMs with ISCSI. Why? (and I'm honestly wondering)"
Honestly - cost.
I work for a medium sized business and the last SAN install I did for our head office where we have around 80 VM's / 4 large hosts was an EMC Clarrion. All fibre channel, dedicated network, full mesh topology, no single point of failure (host, HBA, switches, cables, storage processors etc.). Beautiful.
That was 4 years ago. Our EMC is now dropping IO's like the government drops elections promises and we're pretty much out of capacity.
2 weeks ago I was told we have a budget of 40k for a SAN. Actually two SAN's if I want to keep the same level of redundancy / replication I have between the two server rooms. Tried getting at least 10Tb usable storage, on 15k RPM discs that's certified for VMWare, has 3 years mission critical support, replication, dedupe and works on fibre channel. For under 40k.... Impossible I tell you.
So I'm left with no choice. It's cheaper to bulk up the Ethernet core with some more redundancy, slap in another redundant path and buy a HP P4500 (two of in a cluster - virtual IP). This scares the crap out of me as our Ethernet does NOT have redundant paths, nor redundant switching. In fact a single cable cut or a single switch failure at top of rack level over our 6 racks in the two rooms would take the whole Ethernet core down. (I had 8k for 10 switches.... FFS!). Can whack in a replacement switch (have a couple kicking around) or replace a knackered uplink / fibre in a few minutes. That's okay for voice and data traffic as it's a few minutes downtime where servers and clients can't talk.
However if that happened with iSCSI then I presume my VM's would all shit themselves as they can't access their VMDK's anymore. BSOD's all around I'd presume...
I'd love to just slap in a couple of FC SAN's and be done with it. Uses our existing, Enterprise-Grade FC infrastructure. However for the budget I've got there's just no chance. For £50k I can add in a redundant link for our Ethernet topology, run RSTP and create a new VLAN with the highest priority.
Not what I would want, but best I can do with the resources given.
Re: I've seen...
"I'd love to just slap in a couple of FC SAN's and be done with it. Uses our existing, Enterprise-Grade FC infrastructure. However for the budget I've got there's just no chance. For £50k I can add in a redundant link for our Ethernet topology, run RSTP and create a new VLAN with the highest priority."
That 50k includes the 2 SAN nodes, discs, support and the above topology changes. (Mainly civils in getting new fibre runs in place.)
Re: I've seen...
Drobo's web site says 47 pounds (~21 kg) without drives or packaging.
There's a reason why I have in our test lab at work a handful of unused 'universal' rack mount rails left over from a UPS deployment we did a while ago for that reason- a couple of the servers we have lost their rails a while ago, so this is the only way to keep them installed without the fronts ripping themselves out. :)
On an unrelated note, scary is seeing a refrigerator-sized core router (with about a mil or two worth of line cards installed in it) being held to the rack by four screws instead of the 8-10 that it preferred. :)
Re: I've seen...
"On an unrelated note, scary is seeing a refrigerator-sized core router (with about a mil or two worth of line cards installed in it) being held to the rack by four screws instead of the 8-10 that it preferred. :)"
I am still trying to get pas "a mil or two worth of line cards." Alas...
How good is the protection?
Having read the article, it is clear the author like the thing, even though it has a dumb-ass design of needing OS-specific management software. Compared to the various NAS I have used which have a web interface, that is just such a backward move and limits your choice of OS.
But the linked article told us nothing about how "BeyondRAID" works, only the sort of thing it is supposed to do better. Further more some of the points made, such as disk swapping/order, have not been an issue on hardware or software RAID for years now. When anyone makes up a fancy marketing name like this, and you see little about what it actually does, my BS detector starts to twitch.
So far (I guess?) you have had no HDD fail so presumably you don't know how well it handles real-life failures? Has the unit got support for periodic disk scrubbing? Have you tested it with double parity and pulling/replacing one disk, and during the rebuild pulling and replacing another? (with something like ZFS on the iSCSI allocations so you can check the file's integrity afterwards)
Re: How good is the protection?
Yep. I don't trust black boxes at all. I don't understand BeyondRAID because I don't know the algorithms used to move blocks around. Makes me very nervous. So I did what you suggested: pulled drives iut during a rebuild. Pulled drivers out in the middle of high I/O, etc.
Long story short: it offers RAID-6 class protection. It survives a disk pull during peak I/O. It survives a disk pull during rebuild. It survives instantainious 2-disk failures. It DOES NOTA like 2 disk faillures during RAID dxpansion (after adding a new disk). RAID expansion being different from rebuild.
Which puts it in line with my LSI or Adaptec hardware RAID cards.
Incidentally, do you know of anyone who makes HDD with programmable faults (bad sectors, silent bad sectors, read/write faults, etc) to test RAID systems more easily?
I know there is an option in Linux software RAID to put in a fault layer for debugging RAID stuff, but it would be nice to have an HDD (or SAS/SATA in-line thing) that could emulate typical hardware faults to test 'black box' systems.
Re: Testing HDD?
No. I wish. I purposefully keep around "dodgy" hardware to test those scenarios, myself. The Drobo did about as well as an LSI 1078-based card at handling them; properly detected them in most ciecumstances, went pear-shaped on a rebuild - dropping the bad disk - when it hit a non-recoverable read error, etc.
Re: Testing HDD?
You could try Xyratex near Portsmouth, they provide test rigs to many of the SAN manufacturers. Not sure they do "small" orders but it's worth a phone call.
Avoid OCZ for RMA
I recently had some ram fail that cost me £85 (inc VAT) so I got in touch with OCZ to raise an RMA.
Now I know that OCZ no longer make ram but they came back saying they would give a market value cash payment.
They offered me £14!
I asked them to show me where I could get similar ram for £14 and they pointed me to a website showing ram for £14 but less VAT and P&P. So basically £25 ram.
They wrote back saying they don't take into consideration VAT and P&P in valuations (how handy for them).
Anyway I pushed my point harder and in the end they gave in and gave me what I asked for (just the full cost of a standard 4GB DDR3 kit).
But I have to say I have done RMAs with many other companies (Corsair/BFG etc.) and none of them tried to palm me off with next to nothing like OCZ did. They all provided a like for like or similar if no longer available.
Re: Avoid OCZ for RMA
I've had two consumer grade OCZ SSD's fail on me, after the second RMA they upgraded me to later model SSD and both replacements were returned within a couple of weeks. Not great because of the failures but OCZ were actaully pretty good in their response. Unfortunately due to previous experience I actually don't trust the new driveand it's still in it packaging.
Like the auto-tiering software idea and general ease of use. There is definitely a market niche for one-shelf iSCSI arrays, both the hp P2000 and EMC VNXe are really designed for multi-shelf implementations and you don't get the real benefit of either without several shelves of disk. Does it have a replication ability? A simple replication ability for DR and backup would make it a killer product, even if it was just some form of backup/replication to a cloud service.
This is 2013, manufacturers. Get with the program.
> For me - with my Linux desktop...
I'm starting to not understand the extreme incompetence exhibited by hardware makers [and I'm looking at AEG UPS here, btw] to not ship adequate *nix software, or, if they do, ship it with a JDK attached (possibly an 1.6.18 or something to unlock that ghetto feel). It's not hard to do and one could forgive the missing polish, but it should work.
Re: This is 2013, manufacturers. Get with the program.
We have some old-ish MGE UPS and they are also incompetently designed.
For example, there are some features/parameters you can set using the Windows software that you can't do on the Mac or (now missing?) Linux versions, and what is more stunningly stupid is you can't do it by network at all! Yes, so unless you have a Windows box directly connected by RS232 to the head, your UPS in the far away data centre is crippled by design for network control/monitoring even though it runs a web server.
Re: This is 2013, manufacturers. Get with the program.
We've just taken delivery of a couple of Netapp boxes. Now, although the management software is available for win/linux/mac the setup software (that does essential things like set the unit's IP) is Windows only.
Fortunately for me, I use Win7, so that wasn't a porblem, instead I've been sat here all day waiting for netapp to sort out our login details so I can download the setup software.
Seriously netapp, if we were your competitors trying to get a look at your software we'd just buy one of your products, and the software is useless without the hardware, so why can't I just download the software without a login?
Re: This is 2013, manufacturers. Get with the program.
"so why can't I just download the software without a login?"
because they want to spam you. APC do the same crap. You just have to sign up to use our software its real easy BAM have some spam sucker!!!.
Do your customers mind?....
"In real world use I've run a domain controller, a moderate use exchange server with 100 users, a high transaction financials database (which was in the process of doing some fairly intensive integrity checks), 10 VDI instances, five Apache servers and eight image-rendering VMs all off the Drobo at the same time."
...do customers mind this sort of thing? I mean, what would have happened if the review unit happened to have a hardware failure of some kind? Just curious about the ethics side of this.
The tramp: not windows, just hangover. Happy new one.
Re: Do your customers mind?....
"......do customers mind this sort of thing?....." Depends on the SLAs agreed in the contract for the service provided. Top-line hosted support may guarantee you 24x7 access, backup, etc., but not tie the provider to a particular model or vendor's kit. If it's not top-of-the-line service then you may actually have accepted up to 24-hours downtime, plenty of time to recover to another device. If he can fit several clients' business into a one-shelf array it implies they are not giant conglomerates or corporations, therefore probably not top-of-the-line services.
Re: Do your customers mind?....
Not really; the customer in question has zero money. They don't have highly available, totally redundant infrastructure. They live off of periodic backups, templates and things like DFSR and rsync.
If their primary goes down and they are too poor/cheap to have an HA config or a working cold spare, what's the alternative? Wait a week for the local ultra-low-cost vendor to get a replacement part? The Drobo at least had drives that wrren't 6 years old. Plus I was test-driving a Unitrends backup appliance at the same time; way - way outside my customer's price range - which between the two setups - and when combined with VMWare DRS - gave them a more reliable and protected IT infrastructure than they were able to manage off the archaic crap they are consistently unwilling to upgrade.
I will trust a completely unknown Drobo NAS with some known-good backup solutions running on my brand new Ivy Brdige compute node a hell of a lot more than I trust their ancient Santa Rosa compute nodes with RAID 1 WD Raptor 300s that constantly fail. (For that mattrr, the Santa Rosa systems are detriorating as well...)
So no. The customer doesn't mind. They might not have made it through the season otherwise, and are too cheap to buy new gear.
Re: Do your customers mind?....
"Not really; the customer in question has zero money. "
So you are working pro-bono? Excellent, and a happy new year to you, Sir.
Now, first reply to this with a list of Mr Potts' customers wins a virtual beer. I'm sure their marketing people love his public comments on their employers IT policy.
Re: Do your customers mind?....
@kiethpeter there are several customers using my corporate infrastructure, and yes…most of them are pro bono. I'm not a complete asshole; I try to provide charities, churches, friends, family and so forth with capacity (and even services) when and where I can. The tradeoff is that these services are provided on a "best effort" basis. I am continually attempting to acquire new hardware, software and expertise to ensure that "best effort" is better next week than it was the last.
My infrastructure is also (sometimes) shared by my largest customer; the customer in question reads does in fact read The Register. It isn't always black and white though; when I say they have zero money for IT spend and I mean it. Staffing is covered, but capital spend requires making some tough choices.
Do you refresh your servers/desktops/etc which seem to be doing the job and could be stretched out another two years or do you buy another $industry_specific_widget that could increase your production capacity 10% for the next 2 years, paying itself off and the new IT gear you need, if you can only baby the existing stuff along that little bit farther?
The customer in question chose the second route. I have reservations about it, but ultimately gave the thumbs up as "doable" largely because the IT equipment for my consulting company lives in their datacenter. If the brown stuff hits the rotating air circulation mechanism, then I can spot them spare capacity for long enough to see them though whatever issues arise.
(Does that make me a cloud provider? What kind of cloud? What if it's just "I have spare servers and light things up on hardware? Labels, categories and names, oh my!)
My relationship with the customer is solid; they aren't going to screw me and I am committed to making sure that they have what they need to get the job done. They provide me a rack's worth of space – and the fibre optic internet that goes with it – in their datacenter. In exchange, I make sure that if something goes wrong with their aging, finicky infrastructure it will get fixed, ASAP, even if that means putting my own equipment into play.
The datacenter is 20 ft from my office. Work is 15 minutes (8 at night) from my house. For now, it'll do.
Re: Do your customers mind?....
"there are several customers using my corporate infrastructure, and yes…most of them are pro bono."
I meant the excellent, no sarcasm. I have worked in the charitable sector.
"My relationship with the customer is solid; they aren't going to screw me and I am committed to making sure that they have what they need to get the job done. They provide me a rack's worth of space – and the fibre optic internet that goes with it – in their datacenter. In exchange, I make sure that if something goes wrong with their aging, finicky infrastructure it will get fixed, ASAP, even if that means putting my own equipment into play."
Just all seems a bit ad hoc for a business critical system. Best of luck and hope there isn't a disaster.
Re: Do your customers mind?....
@keithpeter totally ad hoc. And we've had a few "disasters." (I.E. primary system caught fire, dead disks, power outage, fibre provider + backhoe, you name it.) In ten years, I've seen a bit of everything.
Fortunately, I am wildly paranoid. Wildly, world-endingly, crazy-like-a-fox paranoid. My file servers are specced and configured to run double duty (in an emergency) as virtual hosts. I have replicated and backed up traffic offsite here, there, everywhere. Certain endpoint systems are bought specifically to match the hardware of the servers in use so that A) the end user has a bitching system as they need for their job and B) in an emergency I can cannibalize it to get a more-critical system up and running.
When I build personal PCs - or PCs for friends/family/etc - they are build to similar specification as those systems built for my customers. That way as these people upgrade their systems - which occurs on a more rapid schedule, generally - I have a steady influx of gently-used parts that can be recertified and put on the shelf as spares for those brown-pants moments.
Every single thing I design or build is designed so that it can – in a pinch – serve any of a half dozen roles. Everything is tested to serve those roles before the first generation hits the ground and some of these systems will be torn out of use (for example as servers) and repurposed (for example as desktops.)
It isn't "proper." It isn't "best practices." But for many of these clients it is the difference between "being able to run a business and not being able to run a business." Doing things according to a whitepaper would bankrupt some of these organisations just to do the upfront costs, that's before we even get into the whole "refresh when the hardware warrantee runs out, or Microsoft releases Yet Another Version."
I sure catch a lot of crap from commenters for talking about such things. (It's blasphemy.) There are a lot of assumptions that because I am forced – yes, forced by necessity, something people can never seem to understand – to work with next to no budget for hardware or software that nothing is redundant or backed up. I think that these folks would be shocked to see exactly how far you can push modern equipment, and exactly how many layers of redundancy you can build into your network just by ensuring that your equipment can serve dual roles if absolutely need be.
I really like the ones who presume I must prefer things this way, or that I am somehow choosing this course. It is s simple matter of "presenting the arguments for more money" they claim; argue and you will get budget. It's not the case. I am generally privy to enough of the financials of these businesses that I know damned well how much money is coming in, where it is going and why.
My job is to make do with what's there until the money is available. It is to reassess the situation every so often and inform the business how close to the red line we are. What is the worst-case scenario? How do we recover from that? What are the risks of the various scenarios coming to pass? What downtime will occur whilst implementing backup plans?
The owners of the company then weigh the options and make the call about who gets how much money that year. So by necessity, everything for these types of folks is ad hoc. It isn't good for one's sanity, maybe…but you do learn rather a lot, very quickly.
And you learn very quickly to understand concepts like "return on investment," "total cost of ownership," "capital expenditure," "operational expenditure" and "operational paranoia" at a truly intuitive level. Every single choice and decision goes through cost, risk analysis, "how many layers of redundancies can we build for that" and many more filters.
It is ad hoc, but a carefully considered, meticulously constructed sort of ad hoc. Layers and layers of it, all interconnected and highly redundant. Fortunately, there looks to be only a couple more years of it until I am finally, mercifully free of the last of it. I can't wait.
Re: Do your customers mind?....
"I sure catch a lot of crap from commenters for talking about such things."
Well, next time, just link to your answer above and save a lot of typing.
Fortunately, there looks to be only a couple more years of it until I am finally, mercifully free of the last of it.
Good luck with all that. I sort of miss working for a charity/NGO but when I get sentimental, Ruth digs me sharply in the ribs and reminds me about the sleepless nights.
Re: Do your customers mind?....
Slowly coming to an end...I got a whole 9 hours last night!
It was magical.
Why does it move commonly accessed blocks to the SSD, rather than duplicate them there? If it duplicated them, presumably you wouldn't need redundancy in the SSD set (if one died, you still have the original block on the spinning rust disks) so you could triple the amount tiered. And I'd imagine changes to SSD blocks could be journalled (sequentially - so nice and fast) on the spinning disks, so a failure during writing would be recoverable too.
I wish I had an answer to this, honestly. I've wondered the same thing. I was initially quite worried about exactly this issue; triple mirroring on 3x SSDs doesn't make give me any warm fuzzies if those three SSDs are OCZ. IF any company's SSDs are going to drop 3x SSDs at the exact same time, my personal experience says "it will be OCZ." So I installed several layers of backups on all production VMs that ended up living on this array – I am Le Paranoid, but don't quite (yet) have the hardware to achieve it – and soldiered on.
I suspect that a lot of it has to do with their thought process on the tiering. They refer to it as "transactional tiering" or "the transactional layer." They want really heavily hit stuff to live there – heavy access database blocks, basically – and that would be as much for write speeds as read. That makes the "move rather than copy" element of their tiering (sort of?) make sense; writing in a read-promotion-only scenario would leave your writes bounded by the cache. (Anything after cache fill would but straight-to-spinning rust and thus dead slow.)
I am not sure I agree with the decision, but I am not sure I disagree either. IT is a technology. It works in this fashion. I would find it adequate – even quite good – for several types of workloads, but not for others.
As with anything really, you have too look at your workloads and see if they fit. It most certainly isn't a good fit for everything, but I can think of several places where it is just fine. Ultimately, I'd like the ability to toggle between the "promote-read-only" behaviour pattern and the current "promote-read-write".
10GbE below $300/port
$300/port is fairly high if you want low cost 10GbE, below $200/port has been available quite readily for some time (over a year now), from vendors with better equipment than Supermicro. The key is to look for gear that doesn't have the PHY built in, this does limit the equipment on cable distance(no multi-kilometer runs), and in the switches I have it also removes the ability to have 40Gbps uplinks. But a 48x10GbE for under $9500 should be easily attainable. You don't have to be google to get those prices either, this is for small unit quantities. I'd bet the likes of google probably doesn't pay more than $4k for a 48 port 10GbE switch at high volume.
Re: 10GbE below $300/port
I'd love to know which vendors you are referencing! Links? Contact info? Are we talking about the cost of a single unit, or are you disucssing volume pricing here? Please share with the class...
Give it six months
Then retest the I/O. Every drobo we've put in has exhibited a severe performance drop off over time. Cure was to reformat and copy everything back, did that twice before giving up and using them as backup mirrors.
I've also seen random disk drop outs and rebuilds, storage space go missing until rebooted, and an esata drobo that only connected to the dashboard for setup and health checks after you powered everything down and switched the esata for FireWire or USB.
Haven't tested one for a few years so maybe they're better now, let us know how you get on.
Re: Give it six months
Wish I could; sadly, it has to go back ASAP. I had heard (3rd and 4th hand, mind you) that the more recent Drobos - starting with the 8 disk units - stopped having those issues. I think it was a software tweak they made. In any case, I remember hearing exactly those same complaints about the old 4-disk models that were USB only type affairs. The same people now claim that this isn't the case with the 8-disk network attached varieties.
Entirely anecdotal, however, please take with truck full of NaCl.
Like a snake needs shoes
This is a headless NAS. There are only two possible reasons why instead of browser-independent web management it requires a Windows / Mac client app to be installed.
Either: the developers are incompetent. They go the long way around because it's the only way they know.
Or: they've some motivation to induce in their customers a dependency on Windows / Mac they may not already have. I.E. the main thrust is not only to provide a great product.
Re: Like a snake needs shoes
Redundant comment maybe? The article did note that the main selling point of the device was that it could be used by pointy-haired bosses, your mother or mine, or video-editing folk- rather than people with the technical literacy that is normally associated with desktop Linux users. This would suggest that the developers aren't incompetent for the reason that you suggest, but rather they are aiming at specific markets- leaving their competitors to cater to people who do understand RAID and the like. You were never in Drobos sights as a customer. That has always been Drobo's selling point; their first products weren't even NAS, but rather "just plug this thing into your computer as you would a USB HDD but now your data is redundant across disks as long as the light is green". Easy. That their newer kit also includes Thunderbolt should be another clue as to who they intend to sell to.
This is the simpler way to explain why the Drobo client is for Windows / Mac... considering some conspiracy is a bit too much effort. Besides, Microsoft and Apple can both be considered to be competitors to Drobo in some respects.
I'm not recommending Drobo since I haven't used them- and even if I had done and my experience was positive, potential buyers would want, ideally, to hear from many existing users.
Re: Like a snake needs shoes
The most obvious 'some other motivation' being Windows 90%+ desktop market share....
- iPad? More like iFAD: Now we know why Apple ran off to IBM
- Apple orders huge MOUNTAIN of 80 MILLION 'Air' iPhone 6s
- +Analysis Microsoft: We're building ONE TRUE WINDOWS to rule us all
- Climate: 'An excuse for tax hikes', scientists 'don't know what they're talking about'
- Analysis Nadella: Apps must run on ALL WINDOWS – on PCs, slabs and mobes