Posts by Nate Amsden

when might this end?

Seems like we have been getting reports for the last ~5 years or more about SSD firmware bugs that brick drives after X period of days from a wide range of manufacturers.

For me these firmware bricking bugs are the biggest concern I have with SSDs on critical systems. Fortunately I have never been impacted(as in had a drive fail as a result of these) yet. But have read many reports from others who have other the years.

Even the worst hard disks I ever used (IBM 75GXP back in ~2000 and yes I was part of the lawsuit for a short time anyway) did not fail like this. I mean you could literally have a dozen SSDs fail at exactly the same time because of this. It's quite horrifying.

I have a critical enterprise all flash array running since late 2014, no plans to retire it, all updates applied(with no expectation of any more firmware updates being made for these drives anymore), oldest drives are down to 89% endurance left so in theory endurance wise they could probably go another 20-30 years, though I don't plan to keep the system active beyond say 2026 assuming I'm still at the company etc.

pretty crazy

Came across this a few days ago, comments regarding network equipment lead times - https://www.reddit.com/r/networking/comments/n644ux/lead_times_through_the_roof/

My org hasn't needed to buy much in the past year so haven't been affected by the situation. Most critical network has enough capacity for the next 2-3 years without requiring anything new. Oldest critical stuff(everything is redundant) has been in service 3,348 days(9.4 years today) and goes EOL by the vendor 6/30/22. Probably can get 3rd party support after.

Servers don't have any new needs at this point before 2024 if things keep going the way they have the past ~3 years anyway.

Nothing exciting, but runs super stable almost zero issues on everything.

bad app and DNS

Here's a great example of a bad app. Java. I first came across this probably in 2004. I just downloaded the "reccomended" release for Oracle Java on linux (from java.com) which is strangely 1.8 build 291 (thought there was java 11 or 12 or 13 now? ) anyway...

peek inside the default java.security file

(adjusted formatting of the output to make it take less lines)

# The Java-level namelookup cache policy for successful lookups:

# any negative value: caching forever - any positive value: the number of seconds to cache an address for - zero: do not cache

# default value is forever (FOREVER). For security reasons, this caching is made forever when a security manager is set. When a security manager is not set, the default behavior in this implementation is to cache for 30 seconds.

# NOTE: setting this to anything other than the default value can have serious security implications. Do not set it unless you are sure you are not exposed to DNS spoofing attack.

#networkaddress.cache.ttl=-1

I don't think I need to explain how stupid that is. It caused major pain for us back in 2004(till we found the setting), and again in 2011(4 companies later, couldn't convince the payment processor to adjust this setting at the time had to resort to rotating DNS names when we had IP changes) and well it's the same default in 2021. Not sure about newer than Java 8 what the default may be. DNS spoofing attacks are a thing of course(I believe handling them in this manor is poor), but it's also possible to be under a spoofing attack when the jvm starts up resulting in a bad dns result which never gets expired per the default settings anyway.

At the end of the day it's a bad default setting. I'm fine if someone wants to for some crazy reason put this setting in themselves, but it should not be the default and in my experience not many people know that this setting even exists and are surprised to learn about it.

But again, not a DNS problem, bad application problem.

shocking

Well maybe I shouldn't be shocked, but I am still. Not at the security issue but looking at that MX server survey I had no idea that Exim and Postfix combined had that high of a market share, and that Sendmail was at 40% ~15 years ago and is now at under 4%. I really expected nobody to have more than say 20-25% market share. Personally I have been using Postfix since about 2001 I think. It was suggested to me for a anti virus solution I was looking to deploy at the time and just haven't had a need to look at anything else.

I went off to look at sendmail.org, and wow they are old school(except they seem to be operating under the "ProofPoint" brand not sure when that happened), just read the stuff under the "Contact us" section. Also it's the first reference to a FTP server I have seen on a website in a long time(I have nothing against ftp myself other than it is funky to work through firewalls).

I still prefer text email myself and my personal email server does strip html off of incoming emails automatically which can sometimes make things difficult (and in very rare occasions impossible as in the entire message is empty) to read. But it certainly brings back memories of an earlier era(an era that was much more fun for me computing wise anyway).

For work my org uses office 365 (and hosted exchange at rack space prior), MS introduced breaking changes in the OWA client which I use for most of my mail which makes text based email composing impossible. Reported it almost 2 years ago and last I checked it was still broken (the behavior being new line characters are broken making the entire email be one long line, in many cases totally unreadable. Message is fine in the "outbox" and only gets mangled once it gets beyond that level).

I guess they are going to miss their SLA?

https://www.theregister.com/2021/01/06/four_nines_azure_active_directory_sla/

agree with the other posters

This CEO is an idiot. Don't care what the EULA says don't be paying such tiny amounts of money for such a massive amount of storage, the math doesn't work out, not even close. It's like people wanting to leverage google drive for a few dollars a month and storing hundreds of gigs to tens or hundreds of TBs and somehow think that is a reasonable thing to do. It's so beyond absurd I don't even have words.

And what kind of small business has that much data? Really sounds fishy. Maybe this CEO is trying to disguise his hoarding habits by saying it's for his "Small business".

To add insult to injury the guy is a CEO of a storage company. He has no right to be angry he should be embarrassed for being so stupid about this, and then doubling down and going public about it. Hell he could of fired off an email to people on his IT team and said "hey I'm thinking about using this for X, what do you think?" Or maybe he did and he didn't agree with their response.

kill openSLP

FYI you can run this command to see if the SLP service is even being used (at least on vSphere 6): esxcli system slp stats get

to see stats about SLP, assuming it is accurate, when I disabled SLP on my clusters back in October the stats indicated no hits to the service since the system started(assuming some kind of health check that ran when the hypervisor booted, time stamp of the event matches boot time exactly).

vmware suggested in the past to disable SLP if you are not using it as a "workaround" though they implied(as of Oct 2020, though looking now looks like they have removed that language, tried checking archive.org for the older page but page was just a blank page) that may break stuff so it's not a long term solution(for me based on the stats of that command, it is a long term solution don't think that feature has ever been used at the orgs I have worked at):

https://kb.vmware.com/s/article/76372

as an extra check I ran nmap against the hosts to verify the port was closed after making the change.

now if they only let consumers buy LTSC windows

That would be a nice improvement. I know enterprise customers can get LTSC, but consumers should be able to as well. MS is supporting the LTSC windows regardless so it's not as if it would be much effort. Slap a premium price on it, that's fine. I'll pay double, even triple the price for that peace of mind without much hesitation.

I purchased a copy of Windows 10 LTSC for a work VM last year, and the cost was almost $500(had to buy win10 pro then a LTSC upgrade license, note company paid not me of course). Possible the vendor quoted sub optimal part numbers I am not sure. Support until 2029 so that's good.

My main desktops/laptops have been linux since 1998, any windows systems at home still run 7(most are off as I don't need them), no plans to upgrade them at this point. AV software still supported, and I haven't had a known security issue with any of my personal systems since the early 90s.

don't understand

As someone who manually compiled and installed many kernels from 2.0 until late in the 2.2 series(fond memories of 2.2.19 for some reason), I never ever even one time had to delete any files as a result of a kernel update (outside of MAYBE the /boot partition if it was low on space). I had to check the article again to make sure it was referring to Linux and it seems to be. Since with 2.4.x onwards(basically when they stopped doing the "stable" and "development" kernels) I rely on the distro packages for kernel updates again, no files need be deleted for such updates.

Could this story somehow be referring to an OS upgrade rather than a KERNEL upgrade? Of course in linux land these can and typically are often completely unrelated(even in 2021, as much as the linux kernel is terrible about binary compatibility with it's own drivers many people know that whether you run kernel 3.x or 4.x or 5.x provided it supports your hardware there isn't much difference for typical workloads). But even with an OS upgrade I don't see a need to delete (m)any files. I also spent hundreds of hours back in the 90s compiling things like gnome, kde, X11, even libc5 and glibc, among dozens of other packages I can't remember anymore.

maybe this was a thing before 2.0 kernels (my intro to linux was slackware 3.0 with 2.0.something I believe back in 1996), but I suspect it was not.

this whole story just doesn't make any sense.

ServU FTP

Wow that brings back memories. I knew a guy(online only never met) back in the late 90s who distributed a "hacked" ServU which was popular for a certain file sharing people back then. People used it because it didn't need a license key, but he also inserted his own backdoor account(s).

systemd and DNSSEC ?

wtf? I wouldn't be surprised if systemd is doing DNS these days but isn't DNSSEC a server-to-server thing not a client to server thing? If so wtf is systemd doing with it?

on the topic of DNSSEC I came across this blog a while back and found it informative, rips into DNSSEC https://sockpuppet.org/blog/2015/01/15/against-dnssec/

"In fact, it does nothing for any of the “last mile” of DNS lookups: the link between software and DNS servers. It’s a server-to-server protocol."

Been running DNS myself since about 1997(both hosting authoritative BIND9 servers as well as hosting domains with Dyn in the last decade or so), though no DNSSEC.

Same network speed as current system?

This article says the network links are 200 Gigabits on the new system.

However apparently their current system has 25 GB/s:

https://www2.cisl.ucar.edu/resources/computational-systems/cheyenne

"Partial 9D Enhanced Hypercube single-plane interconnect topology Bandwidth: 25 GBps bidirectional per link"

25 Gigabytes * 8 = 200 Gigabits.

Would of thought things would be faster on the newer system. That bidirectional statement may imply the current system is just 100Gbit in each direction and the new system is perhaps 200Gbit in each direction? that would be a good boost.

why phone number required

I installed signal again just now to verify the experience. First thing it wants me to do is send me a text message. it also wants access to contacts. So i deleted it. I don't use the other apps mentioned in the article either.

I think i tried it a couple years ago with one of those virtual burner phone services but it didn't work.

I don't get why they don't have an email sign up option. I assume if you never grant access to contacts you can manually add your friends.

I do have line installed. It too wanted a phone number to install. Account was created while i was overseas with another phone and local sim. I decided to install it on my newer phone in 2019 and it wanted a sim card too. Also didn't work with virtual phone texting i think. So i bought a prepaid sim. Used it to register the app then switched back to email authentication and removed phone rights from the app and changed back to my normal sim.

With signal i think(memory is hazy this was a while ago), i did the same process only after verifying I could not find any way to switch to email authentication. So i nuked it before removing the prepaid sim.(possible i am confusing signal with another chat app in this situation i tested at least a couple)

I used the line chat app with nothing but wifi on a dedicated phone for 2 years and it worked fine( it had a sim originallywhen it was installed). I can search for friends by thier username in the app or if they are with me in person i think there is a QR code function take a picture with the app and it adds the friend.

Are there chat apps (make it simple) that work from signup on a wifi only device? I just think if you're really concerned about privacy then you'll want the option to use a dedicated device on wifi. ( cheaper than maintaining a prepaid sim ongoing). Before I moved line to my main phone if i traveled i took both phones and used tethering.

I couldn't find any last i checked.

I wouldn't be surprised if signal is more "secure" than line i just don't want to give them my phone number. So i don't use it. I'd like to though have heard good things.

users may be more inclined to update apps

If the developers (of both OS and apps) would develop versions that just have the security fixes not new features. Also sorely lacking is the ability to easily roll back as well.

Was burned too many times early on after switching to Android many years ago I have had auto app updates off and only update when I really have to.

Two cases in point. Ironically both Weather apps.

Weather.com perhaps before their app was sold to IBM had a pretty good Android app. Then they improved it I guess and wrecked it pretty royally in my opinion anyways. Fortunately I had a backup of the older version(v4.2) and it continued to work for several years(some MINOR things broke, but 85% of the app was workable which was better than the new official version). I was actually quite impressed how long the older version lasted. I powered up my Note 3 just now with that app and it does not work anymore(no errors, just no weather data), but it did at least up until July 2019 when I moved to a newer phone as my daily driver.

On my newer devices I switched to Accuweather, which too had a really nice(in my mind anyway) user interface worked well, paid for the no ads version. Then recently they revamped it. Wrecked it again (check google reviews MANY complaints). Fortunately again I had a backup and reverted to the older version. For whatever reason since I downgraded the notification bar doesn't update automatically anymore no matter what I do, I have to click a little icon on the bar to get it to update. But it works otherwise and again is better than the alternative of using their new app. They started sending popups in the app to get me to upgrade but have ignored them. Not sure if I will get lucky enough to be able to use this older version in the years to come, or if I'll need to find another weather app.

zfs in ubuntu since at least 16.04

I'm not sure if it was considered "supported" back in 16.04(2016) but am currently running several 16.04 systems with zfs with packages from ubuntu.

Perhaps the 19.10 innovation with zfs was installer support? I recall reading news about that but don't remember the version specifically.

https://packages.ubuntu.com/xenial/zfsutils-linux

No special setup, just using zfs in some cases where the compression is helpful, as the back end SAN storage is old enough that it doesn't support inline compression(no zfs on root, just extra file systems after system was installed already)

Haven't personally used any Ubuntu that wasn't LTS since 10.04 so don't know how much further back than 16.04 built in zfs got.

Most probably aren't affected

It seems according to the advisory a workaround is to remove the USB 3.x controller. As far as I know this is not added by default, none of the ~850 Windows and Linux VMs I manage have it. I had to go and add a USB controller to see the option even appear. Have never needed USB 3 otherwise.

Even my vmware workstation at home which I use every day is using USB 1.1 controller.

score one for good defaults I suppose.

(vmware customer since 1999)

no printer at home since 2004

I rarely print. Last time I printed regularly at home was 2003, i would print out my resume along with mini CD labels for business card CDs with a bunch of samples on them, attach to my paper resume at snail mail it to job applications (in addition to applying online), i figured it was a good way to get noticed at the time. Anyway i got a new job in 2003 and my printing needs sort of stopped. When I needed to print i printed at the office.

Fast forward a few jobs and many years I shifted to fully remote work in 2016. Was working from home prior to that but the office was close by(about a mile). In 2016 i moved 90 minutes away.

I started using fedex office for my printing needs. Have to drive 15min each way to get the print outs and there's a $1 minimum for submitting jobs through their website but it works well. I probably go on average 5 or 6 times per year and spend probably on average $8 to $12 per year for those jobs.

There is a UPS store that is much closer and they claim to do online printing too but last time I checked I could not find a way to submit a simple 1 page job(or a few 1 page jobs). It seemed geared towards project level stuff but maybe that's different now.

quite confused

The article references a "peak" share price of about 8, but links to a site which shows the current share price in the low 30s

https://www.investegate.co.uk/CompData.aspx?CPI

Looking at the past few years of performance it seems as if they've probably done several reverse stock splits? It seems back in 2015 they peaked at around $800 according to finance.yahoo.com.

I think the article should be updated to reflect the split adjusted(?) pricing.

not difficult to optimize cost for Oracle in VMware

I did it back in about 2007-2008 time frame. When I was hired the company was undergoing an Oracle audit. They were licensed for Oracle SE One, however their DBA consultants had installed Oracle EE. I pushed to move to SE but manager didn't trust me yet, assured us after we paid the fees everything was all cleared up. The following year we had another audit and we failed again. This time I was able to lead the charge to move to Oracle SE. Which at the time(I assume still now) has a per socket charge (max 4 sockets in a system) rather than per core.

So of course moving to Oracle SE was huge, as it turned out really the only reason the consultants installed EE was their monitoring system used partitioning, which of course hit us with another license breach. I recall buying new CPUs for our Oracle systems going from high speed 2 core (better licensing for EE) to lower speed quad core (better licensing for SE). I ran into a limitation on the DL380G5 systems that we had our system boards were too old to run quad core not even HP knew at the time (they later updated the spec sheets to reflect that), they replaced the boards and installed the new chips.

From a VMware perspective we did two things - we limited where we used Oracle (didn't allow it to run on just any VM host), and we also ran our DL380G5s with a single socket, cutting licensing further. This officially wasn't supported by VMware (ESX 3.5) at the time though I believe their intention was they didn't support running a single cpu, I had no doubts a single quad core cpu would work fine. VMware sold licenses at the time in pairs of sockets so we used 1 socket license on one system and one socket on another saving costs even more. (in hindsight perhaps this was a VMware license violation I never looked into that aspect). We never ended up needing VMware support, and Oracle had no issues with our new setup. I remember specifically having to educate our Oracle rep(s) on the licensing of Oracle SE being per core. Our production OLTP servers ran on physical single socket DL360G5, partly because I wanted no performance impact from VMware, but also because Oracle's support policy at the time was "reproduce on physical hardware before we support you in a VM". Didn't need that extra risk for the production OLTP. That and we just had VMware standard, so no VMotion, no HA etc. Just basic hypervisor.

Obviously Oracle SE has far fewer abilities than Oracle EE, the biggest one we missed at the time was Oracle Enterprise Manager with the query reports. Though at the time (10gR2 I think??) you were still able to install OEM on SE. It was easy to install and delete in the case they came to audit again(they did not). Newer versions of Oracle from what I could tell made it impossible to install OEM on SE (or at least it wasn't dead easy like it was back then).

Though I'm sure that Oracle SE is very cost effective and far easier to migrate to than MariaDB - not only that Oracle SE has much more abilities than MariaDB even.

Unfortunately there is nothing in the MySQL world in 2020 that comes close to those Oracle query reports I had access to in 2008. Our DBA does log many queries and can run query reports but it is a very time consuming process and can have quite a lot of overhead (if your not careful your query logs can be multi GB per hour easily which means query reports can take hours to get results back). I have used tools like ScaleArc and Heimdall which are MySQL aware proxies which come with real time analytics, though they are limited in that they can just see the queries and the response times, they can't get the in depth metrics of what is going on inside the DB for a given query.

I do get tons of internal metrics on each MariaDB server we have even query response times(can't see WHAT queries just # of queries at given response time thresholds), probably 200-250 metrics per server per minute or something. But still pales in comparison to the internal query level metrics available in Oracle in real time.

In a more modern world if I needed to run Oracle on a larger scale in VMware I'd build a dedicated VMware cluster for nothing but Oracle DB. Run the apps on other servers, limit licensing impact. I did run a very small Oracle system at my current org for many years it was for our vCenter 5 backend database. Given the choice was MSSQL, Oracle, or IBM DB2, the internal vCenter DB cannot scale very high at all. For me the choice was obvious, running Oracle on Linux. I used named user plus licensing(so not per CPU or per core) on Oracle SE and it was dirt cheap just a couple hundred a year or something. Eventually retired it almost 2 years ago after finishing migration to vCenter 6.5 on VCSA which uses an internal Postgres database.

Had many conversations over the years with Oracle they never cared about auditing us(I was ready regardless). Though we did undergo our first VMware audit last year, and we failed. I wasn't aware we couldn't move 3 nodes of essentials plus licensing from our UK office which had permanently shut down to our California office. The licensing was only in use for a few months but still had to pay the fees(which was buy a new essentials license which we used for 1 month before transitioning to Enterprise plus licensing as I retired some older systems from our primary datacenter and gave them to our HQ office with the VM licensing intact).

Fortunately we didn't get hit with a VMware audit a couple years earlier, the VAR that sold us our licensing for our datacenter stuff in EU bought everything in the U.S.(through HP) and shipped it to EU, would of had to pay probably more than 10x the fees for that, though that location for us was permanently closed 2 years ago (moved systems to U.S. for EU customers), then the business decided to close all EU operations entirely last year.

Having a region lock on a license code is just so stupid. I can understand region locks for things like on site support(had to jump through a lot of hoops(too many and took too long) to ship our HP gear from Amsterdam to the U.S. and get it under support again), but otherwise makes no sense to say you can't use this license code you bought in the EU on a U.S. system even though the EU system is permanently shut down. Not as if this was a site license, just a one off license purchase.

don't forget cheap

Customers wanted the easy to use, the scalability and they didn't want to pay VMware etc for licensing. Perhaps Openstack will get a big boost from the new EU cloud project or whatever.

The move from utility computing(what I prefer, not an often used term) to (infrastructure) cloud computing comes at an astronomical relative cost, and in my opinion in most cases isn't justified. Huge cost increases come from huge increases in complexity etc.

reminds me..

reminds me of the basic packet exploits against windows systems back in the 90s, I think teardrop was one, then there was a "ping of death" and others though at the time those just caused crashes, not sure if they were able to execute code as well.

"No subscription equals your office hardware bricked"

I think I have read that Cisco has some product lines that do this? (new product lines which have been quite controversial with the new subscription model). I don't know specifics as I have generally tried to avoid anything Cisco for at least 15 years now.

The org I work for uses Aerohive for corporate wireless which is a cloud based management product(company was purchased by Extreme Networks last year I think, I've been an Extreme Networks ethernet customer for 20 years now). Last year our subscription lapsed. I asked support in advance what would happen as we were just about to expire and our vendor was struggling to get a quote. They said no issues just can't manage the devices. We ended up getting the order in on time but it got canceled for some reason(never got notification, I was assuming once the order was processed Aerohive would automatically update the subscription on their end and things would just keep working) and we ended up with no subscription for several months(took me that long to need to login again to poke at the interface before I realized the order never went through as our vendor confirmed it was received by Aerohive and being processed several months before). At the end of the day there was no impact, network stayed up and fine. The only reason I had to access the management UI at the time was to a quarterly "rogue AP scan" for PCI, otherwise it was not for at least another 7 months before I actually had a need to make a configuration change(adjusted radio strengths).

Even without a subscription I would wager that I would be able to login to the command line of the Aerohive access points and make changes there if needed. I mean I can already login and make changes, I don't think the AP would phone home and say "hey I don't have a subscription so you can't make any changes", but it's technically possible I suppose. I logged into the CLI of the APs many times while cleaning up the configuration last year. I monitor each individual AP via SNMP with our general monitoring tool and even without subscription there was no issues gathering metrics either.