Posts by Nate Amsden

Getting good at storing files doesn't have to be basecamp's business.

People seem to jump to the end conclusion, either you build everything yourself, or you use a public cloud, and I just don't understand why. There is a massive chasm of space in between those two options in the form of packaged solutions from vendors like HPE, Dell, and others. Many different tiers of equipment hardware, software and support.

8PB is a lot, but it's not really for object storage. HPE Apollo 4510 is a 4U server that can have up to 960TB of raw storage(so ~10PB per rack, assuming your facility supports that much power/rack). Depending on performance needs - 60 x 16TB drives may not be enough speed for 960TB by itself. Probably want some flash in front of that(handled by the object storage system). Of course you would not be running any RAID on this, data protection would be handled at the object layer. Large object storage probably starts in the 100s of PB, or Exabytes.

There's no real need to use CEPH which is super complex(unless you like that). Probably better to be using something like Cohestity or Scality (no experience in either), both available for HPE Apollo(and other hardware platforms I'm sure). There are other options as well.

I think I was told that Dropbox leveraged HPE Apollo or similar HPE gear +Object storage software when they moved out of Amazon years ago. As of 2015 Dropbox had 600PB of data according to one article I see here.

I'm quite certain it would be easy to price a solution far less than S3 at 8PB scale, even less scale. You also don't need as much "protection" if you choose proper facilities to host at. Big public cloud providers cut corners on data center quality for cost savings. It makes sense at their scale. But users of that infrastructure need to take extra care in protecting their stuff. Vs hosting it yourself you can use the same model if you want, but if you are talking about 8PB of data that can fit in a single rack(doing that would likely dramatically limit the number of providers you can use to support ~20kW/rack? otherwise split into more racks), I would opt for a quality facility with N+1 power/cooling. Sure you can mirror that data to another site as well, but no need for going beyond that (unless geo latency is a concern for getting bulk data to customers).

Re: Open source

You got a bunch of down votes but you are right for the most part. A lot of the early open source models was release the source for free and then have a business around supporting it. Not everyone would sign up as customers but the good will from releasing the source would attract users. It worked well for several companies, and of course public cloud is taking that away from a lot of these orgs, which is unfortunate. And as El Reg has reported several such companies have been very vocal about this situation.

Obviously in many(maybe all?) cases the license permits this usage(at the time anyway, some have introduced licensing tweaks since to prevent it), but I'm quite sure if you went back in time ~15ish years and asked the people making the products did they anticipate this happening they would probably say no in almost all cases(perhaps they would of adjusted their licenses if they viewed that possibility as a credible threat). At the end of the day the big difference with these cloud companies and earlier generations of "mom & pop" ISPs that were using Apache or whatever to host their sites, is just massive scale.

Those licensing their code in BSD licensing or similarly completely open licensing probably wouldn't/shouldn't care anyway.

Similarly for the GPL, a trigger of sorts to making the GPLv3 was the TiVo "exploiting" a loophole in the GPLv2. So GPLv3 was made to close that hole(and perhaps others). There's even a Tivioization term they made

https://en.wikipedia.org/wiki/Tivoization

"In 2006, the Free Software Foundation (FSF) decided to combat TiVo's technical system of blocking users from running modified software. The FSF subsequently developed a new version of the GNU General Public License (Version 3) which was designed to include language which prohibited this activity."

Re: Hiring impact

I think you are incorrectly confusing SaaS and IaaS in your statement.

Mom and Pop shops that have minimal IT needs likely will have almost zero IaaS, because they can't manage it. IaaS (done right) IMO requires more expertise then on prem, unless you have a fully managed IaaS provider. But the major players don't really help you with recovery in the event of failure, it's on the customer to figure that out. vs on prem with vmware for example if a server fails the VMs move to another server, if the storage has a controller failure or a disk failure there is automatic redundancy. Doesn't protect against all situations of course but far more than public cloud does out of the box. If Mom & Pop shop just have a single server with no redundant storage etc, if that server has a failure, they can get it repaired/replaced generally with minimal to no data loss. Vs server failure in the major clouds is generally viewed as normal operations and the recovery process is more complex.

I've been calling this model "built to fail" since 2011, meaning you have to build your apps to handle failure better than they otherwise would need to be. Or at least be better prepared to recover from failure even if the apps can't do it automatically.

SaaS is a totally different story, where the expertise of course is only required in the software being used, not any of the infrastructure that runs it. Hosted email, Office, Salesforce, etc etc..

On prem certainly needs skilled staff to run things, but doing IaaS public cloud(as offered by the major player's standard offerings) right requires even more expertise(and more $$), as you can't leverage native fail over abilities of modern(as in past 20 years) IT infrastructure, nor can you rely on being able to get a broken physical server repaired(in a public cloud).

Re: Cloud Vs On-Prem

Should do the math for how bursty is bursty. At my last company I'd say they'd "burst" 10-30X sales on high events, but at the end of the day the difference between base load and max load was just a few physical servers(maybe 4).

IMO a lot of pro cloud folks like to cite burst numbers but probably are remembering the times of dual socket single core servers as a point of reference. One company I was at back in 2004 we literally doubled our physical server capacity after a couple of different major software deployments. Same level of traffic, just app got slower with the new code. I remember ordering direct from HP and having stuff shipped overnight (DL360G2 and G3 era). Not many systems, at most maybe we ordered 10 new servers or something.

Obviously modern servers can push a whole lot in a small (and still reasonably priced) package.

A lot also like to cite "burst to cloud", but again have to be careful, I expect most transactional applications to have problems with bursting to remote facilities simply due to latency (whether the remote facility is a traditional data center or a cloud provider). You could build your app to be more suited to that but that would probably be quite a bit of extra cost (and ongoing testing), not likely to be worth while for most orgs. Or you could position your data center assets very near to your cloud provider to work around the latency issue.

Now if your apps are completely self contained, or at least fairly isolated subsystems, then it can probably work fine.

One company I was at their front end systems were entirely self contained, no external databases of any kind). So scalability was linear. When I left in 2010 (company is long since dead now) costs for cloud were not worth using vs co-location. Though their front end systems at the time consisted of probably at most 3 dozen physical servers(Dell R610 back then, at their peak each server could process 3,000 requests a second in tomcat) spread across 3-4 different geo regions(for reduced latency to customer as well as fail over). Standard front end site deployment was just a single rack at a colo. There was only one backend for data processing that was about 20 half populated racks of older gear.

Re: Time to dump Cisco

Curious can you name any such products especially in the networking space? I've been doing networking for about 20 years and haven't heard of any vendor/product remotely approaching 15 years of support after end of sale, at most maybe 5 years?

Re: Not us then

Microsoft would say the same if you use Office 365, backups are the responsibility of the customer.

https://learn.microsoft.com/en-us/azure/security/fundamentals/shared-responsibility

Welcome to cloud.

(myself I have self hosted email for 24 years, and haven't been responsible for corporate email since 2002 at that point ran email with postfix/cyrus imap which is what I use for home still)

Re: Right.

I have read people claim ransomware can sit waiting for upwards of 90+ days before striking. I used that justification to finally get a decent tape drive approved a few years ago for my last company's IT dept, then they used Veeam to backup to tape.

I suppose in theory if you restored old data onto a server WITH A CLOCK SET TO THE RIGHT TIME (not current time), then perhaps it could be fine, but of course systems don't often behave well when their clocks are out of sync.

So for example if you restored data from 45 days ago, onto a server with it's clock set to 45 days ago then you may be ok, if you restore it to a server with current time then perhaps the existing ransomware will see the strike time has passed and activate again. I've never been involved in ransomware myself so don't know how fast it is acting.

I'd assume this was a highly targeted attack against rackspace not a drive by thing.

Also read years ago on multiple occasions claims from security professionals that on average intruders had access to a network for roughly 6 months before being detected, first saw this claim reported by the then CTO of Trend Micro, saw a presentation from him at a conference, normally I hate those kinds of things but that guy seemed quite amazing. I was shocked to see him admit on stage that "if an intruder wants to get in, they will get in, you can't stop that". And not try to claim his company's products can protect you absolutely. I posted the presentation in PDF form(probably from 2014) here previously, though it loses a lot of it's value without having the dialog that went along with it

http://elreg.nateamsden.com/TH-03-1000-Genes-Keynote_Detect_and_Respond.pdf

Re: So where are the backups?

at my first system admin job back in 2000 one of the managers there(not someone I reported to) would on occasion ask me to restore some random thing. I thought it was a legitimate request so I did (or tried, sometimes could not depending on the situation). Later he told me he didn't actually need that stuff restored he was just testing me. Which I thought was interesting. I wasn't mad or anything. Never have had a manager do that again or at least never admit to it.

One company I was at we finally got a decent tape drive and backup system in place. I went around asking everyone what do they need backed up, as we don't have the ability to backup EVERYTHING (most of the data was transient anyway). Fast forward maybe ~6-9 months we have a near disaster on our only SAN. I was able to restore everything that was backed up, some requests did come in to restore stuff that was never backed up and I happily told them, sorry I can't get that because you never requested it be part of what was backed up. In the end minimal data loss from the storage issue but there was several days of downtime to recover.

My first near disaster with storage failure(wasn't on the backend team that was responsible) was in 2004 I believe, double controller failure in the SAN took out the Oracle DB. They did have backups, but they knowingly invalidated the backups every night by opening them read/write for reporting purposes. Obviously the team knew this and made the business accept that fact. So when disaster struck, it was much harder to restore data as you couldn't simply copy data files over, or restore the whole DB as the reporting process was rather destructive. Again multiple days of downtime to get things going again, and I recall still encountering random bits of corruption in Oracle a year later(would result in a ORA-600 or something error and the DBA would then go in and zero out the bad data or something).

My most recent near storage disaster was a few years ago at my previous company. Their accounting system hadn't been backed up in years apparently, IT didn't raise this as a critical issue, if they had raised it with me I could have helped them resolve it, it wasn't a difficult problem to fix just one they didn't know how to do themselves. Anyway, the storage array failed, again double controller failure. End of life storage array in this case. They were super lucky that I was friends with the global head of HPE storage at the time and after ~9 hours of our 3rd party support vendor trying to help us I reached out to him in a panic and he got HPE working around the clock, took about 3 days to find and repair the metadata corruption, minimal data loss(no data loss for the accounting folks). Was quite surprised when I asked for a mere $30k to upgrade another storage system so we could move the data and retire the end of life one, and the same accounting people who almost lost 10 years of data with no backups told me no.

Re: IaaS loves to blame the customer

dug up the power issue from Amsterdam, which was previously a Telecity data center that Equinix acquired by this point(2018):

"DESCRIPTION: Please be advised that Equinix and our approved contractor will be performing remedial works to migrate several sub-busbar sections back from there temporary source to the replaced main busbar which became defective as reported in incident AM5 - [5-123673165908].

During the migration, one (1) of your cabinet(s) power supplies will be temporary unavailable for approximately six (6) hours. The redundant power supply(s) remains available and UPS backed. "

But this power incident wasn't critical for me since everything was redundant on my end, I'm not a power expert so certainly can't say for sure if a better power design could of allowed this kind of maintenance to be done without taking power circuits down to customers. But I can say I've never had another facility provider need to take power offline for maintenance for any reason in almost 20 years. Perhaps this particular activity it would be impossible to avoid I don't know.

After Equinix acquired Telecity I noticed the number of customer notifications went way up, Telecity had a history with me at least of not informing customers of stuff. I hated that facility and staff AND policies so much. Only visited it twice, before Equinix took over, according to my emails looks like we moved out less than 3 months after the above power issue (move was unrelated to that).

Re: IaaS loves to blame the customer

I don't agree there at all. Good infrastructure management is good management. Having a properly designed facility is a good start. Well trained, knowledgeable staff is also important. Having and following standards is also important.

That Fisher plaza in Seattle at the time as far as I recall had issues such as:

* Staff not replacing UPS batteries before they expired

* Not properly protecting the "Emergency Power off" switch (which was one power incident a customer pressed it to find out what would happen, after that all customers required "EPO Training")

* Poor design led to a fire in the power room years after I moved out which caused ~40 hours of downtime and months of running on generator trucks parked outside. A couple years later I saw a news report of a similar fire at a Terremark facility, in that case they had independent power rooms, and there was zero impact to customers.

* Don't recall the causes of other power outages there if there were any other unique causes.

Another facility I was hosted in Amsterdam had an insufficient power design as well, and poor network policies

* The network team felt it was perfectly OK to do maintenance on the network, including at one point taking half of their network offline WITHOUT TELLING CUSTOMERS. They fixed that policy after I bitched enough. My normal carrier of choice is Internap, which has a 100% Uptime SLA, and has been excellent over the past 13 years as a network customer. Internap was not an option in Amsterdam at the time so we went with the facility's internet connection which was wired into the local internet exchange.

* At one point they told customers they had to literally shut off the "A" power feeds to do something, then the following week they had to shut off the "B" power feeds to do that thing to the other side, don't recall what it was, but obviously they didn't have the ability to do maintenance without taking power down (so am guessing no N+1). No real impact to either event on my end, though we did have a few devices that had only 1 PSU(with no option on those models for a 2nd), so we lost those, however they had redundant peers so things just failed over. In nearly 20 years of co-location only that facility ever had to take power down for maintenance.

One company I was at moved into a building (this was 18 years ago) that was previously occupied by Microsoft. We were all super impressed to see the "UPS Room", it wasn't a traditional UPS design from what I recall, just tons of batteries wired up in a safe way I imagine. They had a couple dozen racks on site. Wasn't till later the company realized most/all of the batteries were dead so when they had a power outage it all failed. None of that stuff was my responsibility, all of my gear was at the co-location.

My first data center was in 2003, an AT&T facility. I do remember one power outage there, my first one, I recall I was walking out of the facility and was in the lobby at the time when the lights went out. I remember the on site staff rushing from their offices to the data center floor and they stopped to assure me the data center floor was not affected(and it wasn't). Power came back on a few minutes later, don't recall if it was a local issue to the building or if it was a wider outage.

My first server room was in 2000. I built it out with tons of UPS capacity and tons of cooling. I was quite proud of the setup, about a dozen racks. Everything worked great, until one Sunday morning I got a bunch of alerts from my UPSs saying power was out. Everything still worked fine but about 30 seconds later I realized that while I have ~45min of UPS capacity I have no cooling right now so I rushed to the office to do graceful shutdowns of things. Fortunately things never got too hot I was able to be on site about 10 mins after the power went out. There was nothing really mission critical there, it was a software development company and the majority of the gear was dev systems, the local email server(we had 1 email server per office) and a few other things were there as well.

There are certainly other ways to have outages, I have been on the front lines of 3 primary storage array failures in the last 19 years, arrays which had no immediate backup so all of the systems connected to the arrays were down for hours to days for recovery. And I have been in countless application related outages as well the worst of which date back 18 years ago an unstable app stack being down for 24+ hours and the developers not knowing what to do to fix it. At one point there we had Oracle fly on site to debug database performance issues too. I've caused my own share of outages over the years though I probably have a 500:1 ratio of outages I've fixed or help fix vs outages I caused.

My original post, in case it wasn't clear, was specific to facility availability and to a lesser extent network uplink availability.

Re: We make a rod for our own backs...

oh yeah, that's right, sorry was a long time ago!

Re: This is not how data centers work

Be sure to deploy your own environmental sensors.. Most good PDUs have connections for them. I have at least 4 sensors (2 front/2 back) on each rack(2 PDUs * 2 sensors each)). They monitor temperature and humidity.

I remember the first time my alarms for them tripped, I opened a ticket with the DC asking if anything had changed. It wasn't a big problem (the humidity had either dropped or exceeded a threshold I forget which), I was just curious the dramatic change in readings. They responded that they had just activated their "outside air" cooling system or something which was the cause of the change in humidity.

Had major thermal issues with a Telecity facility in Amsterdam, no equipment failures just running way too hot in the cold isle. Didn't have alerting setup for a long time, then when I happened to notice the readings it started a several months long process to try to resolve the situation. Never got resolved to my satisfaction though before we moved out.

I remember another facility in the UK at another company that was suffering equipment failures(well at least one device, their IDS failed more than once). The facility insisted the temperature was good, then we showed them the readings from our sensors and they changed their tune. They manually measured, and confirmed the temps were bad and fixed them. Never was on site at that facility so not sure what they did perhaps just opened more floor panels or something.

But just two facilities with temperature issues over the past 19 years of using co-location.

Power and cooling, two things most people take for granted when it comes to data centers(myself included). Until you've had a bad experience or two with either then you stop taking them for granted.

Re: We make a rod for our own backs...

Lots of gear can come with DC power supplies. For a while companies like Rackable who was popular in the hyperscale space back before 2010 had rack based AC-DC systems. They were bought by SGI then HP bought SGI those product lines are long dead. I was interested back in 2009 in one of their products called "CloudRack", which was neat in theory, never got to see it in action though. Built for hyperscale, the servers had no power supplies or fans, there was rack level of both that supplied the servers(sample server picture from my blog at the time http://www.techopsguys.com/wp-content/uploads/2011/05/c2-tray.jpg). I wanted to get it for a Hadoop build out at the company I was at. I wouldn't dare use them for anything mission critical of course.

I think going beyond rack based DC distribution is likely to be wasteful/inefficient with the loss in energy over the distance? Thought I was told/read something along those lines at one point. Also I think I was told/read that DC is much more dangerous than AC.

Another efficiency gain is increasing the voltage. I've never seen it used myself but the PDU vendor I use (ServerTech) at one point was pushing 415V (https://www.servertech.com/solutions/415v-pdu-solutions). Unsure how much savings that higher voltage can bring.

Re: Turn down the lights!

at least where equipment is at I'd assume almost all data centers use motion activated lighting. I remember the first data center I visited in 2003 was an AT&T facility, really liked that place. I don't recall if they used motion sensors or simple timers on their floor lights(which was one giant warehouse floor with a huge raised roof). Heard stories about that place when it first opened, there was so little equipment that gear was running too cold they actually put space heaters in for some customers to get the temperature to more normal levels. By the time I saw it they had plenty of customers and no heaters to be found.

The practice is so annoying that I install my own LED lighting(utility clamp lamps from a hardware store). I don't go on site often(currently in probably my longest stretch haven't been on site in just over 3 years), but there have been times when I was on site for a dozen hours straight. Not only did my LED lights provide much better lighting but I didn't have to walk around the cage every hour(or more) waving my arms to trigger the motion sensors.

Nothing was more annoying than a Telecity data center I dealt with in Amsterdam for a few years though, the only place I've been at with hot/cold isle isolation. Sounded neat in theory but it was so annoying to have to walk ~120 feet between the front and the back of the rack. Almost as bad as they required you put these little elastic "booties" on your shoes before walking on the raised floor, another practice I've never seen anywhere else(and completely crazy). I hated that place. Ironically Equinix acquired them eventually(after my org cut ties with them). Then it took another 3 years to get off their mailing lists.

But of course the power provided by lights is probably a rounding error given how efficient LEDs are.

Re: Business Continuity

I'd be willing to best most(90%+) of the customers did not take their own backups, just like most likely most office 365 customers don't take their own backups. Quite surprising really (maybe I shouldn't be surprised).

Mirrored sites can still be compromised, if anything it may be easier, compromise one site and the replication automatically compromises the other site(s) for you (depending on how it was compromised and what kind of replication). Failures can also replicate, data corruption can destroy multiple sites as fast as your replication can send it.

ISP going down and security compromise are very different things. Myself I have been involved with 3 primary storage array(SAN) failures in my career, all of them took multi day recovery efforts, all lost some data with a risk of total data loss, and in all cases the company did not have good backups, ALSO in all cases the company chose not to immediately invest in better protection going forward following the near disaster. All 3 situations were the most scary of my career as well, and in the two I was directly involved with I pulled an unbelievable amount of monkeys out of my ass to get the systems working again. The first one was early in my career and I was on the ops/app team not the backend team so I just waited while they worked to fix the issue. But I was the one to report the issue to everyone, will always remember the Oracle DBA telling me he almost got into a car accident when he read my emergency alert sent to everyone on that Sunday at around lunch time(with output from the HPUX Oracle systems showing "I/O error" on several mount points from the df command). Spent about 32 hours on a conference call for that, probably my longest ever conference call.

I've been fortunate never to have been involved in a serious security incident(have had to deal with a few stupid hacks from unmaintained systems that I was asked to help with over the years).

I run my stuff pretty well, though nothing is perfect, the best strategy (if possible) is try not to be a tempting target. Rackspace, hosting a lot of customer stuff is obviously not in a position to do that, so they have to deal with a lot more things than I.

Re: Business Continuity

Curious what you mean by this. Rackspace has been managing exchange systems for well over a decade at this point so they obviously have a lot of experience there. I assume they still operate their own data centers in many/most cases?(I know their business has changed quite a bit in the past decade). In this particular situation I would say they are expected to provide service levels associated with mission critical stuff for Exchange. That's what the customers are likely paying for anyway. Just not sure what you mean by "no third party hosted service can". Do you mean that only doing it yourself can you provide true mission critical services? Or do you mean only Microsoft can provide mission critical Exchange?(obviously far from the only mission critical app stack out there) or both or other?

Sounds like Rackspace's communication was poor on this, but taking down everything was a good response assuming they did it right away after they determined it was a security issue.

I'm very much pro on prem for everything, at least everything I know(mostly Linux based and I do infrastructure too). I've been operating mission critical internet facing infrastructure for 20 years(as of March 2023), and non mission critical internet facing infrastructure since 1997.

I don't know Exchange(bulk of my Windows expertise dates back to NT4 era) and I find it interesting that so many self proclaimed Exchange experts/admins advocate using Office 365 over hosting it themselves, I guess MS did a really poor job with that software stack, or the average Exchange expert/admin is an idiot (or both). I remember seeing some cool "time machine" based backup systems for Exchange ~12-13 years ago, backed up stuff in real time and you could roll back to any moment with a click of a button(or at least that was the marketing, never saw it in action). Forgot any of their names. Probably put out of business by Office 365(because there are fewer customers running Exchange) though which from what I understand doesn't have anywhere remotely that ability.

I've never been a Rackspace customer myself, every time I looked at the pricing(last time was 12 years ago) it didn't make sense given what I can accomplish with regular co-location. But there are probably lots of customers that really need their hands held on everything, so probably a good solution for them.

Re: Take This With a Grain of Salt

I was running servers 20 years ago with linux with 3 to 4G of ram and all of it ECC.

Unsure why logic running linux on 4G back then would be different now. Regular ECC is barely adequate anymore. HP's Advanced ECC came out in the 90s and is far superior. IBM came up with Chipkill at a similar time(never used IBM servers myself). Dell never came up with anything but presented an "Advanced ECC" option in their Xeon 5500 systems I remember. It was a different technology vs HP. Dell's advanced ECC came from Intel and while it did the job it removed a third of the memory capacity of the system if I recall right. HP's has no overhead by default.(but can have overhead if you enable even greater protection from dimm failures including online spare memory and memory mirroring).

That said I just got my first laptop with ECC last week. Lenovo P15 with 8 core xeon and 80G of ram. Was kicking myself for not going ECC on my 2016 Lenovo P50 which I still use today, currently with 48G. Not that I've noticed any instability. It runs 24/7. Both laptops run linux mint 20.

Per linus wonder why he didn't just remove the faulty dimms. HP, and Dell perhaps others too can tell you which dimm is bad. I remember the pain involved when I used Supermicro years ago which could not do that, tracking down which was bad was annoying. I'd assume he has at least a half dozen dimms and can lose 2 or 3(maybehave to remove a full bank) and still have a more functional system than a laptop.