UK ICO, USCourts.gov... Thousands of websites hijacked by hidden crypto-mining code after popular plugin pwned Thousands of websites around the world – from the UK's NHS and ICO to the US government's court system – were today secretly mining crypto-coins on netizens' web browsers for miscreants unknown. The affected sites all use a fairly popular plugin called Browsealoud, made by Brit biz Texthelp, which reads out webpages for blind …
@AC
And also of why dynamically loading scripts instead of using a local copy is a bad idea. You end up with multiple dependencies completely outside of your control.
True, but on the other hand, massive amount of work for you making sure your local copies are always up-to-date, and have the latest fixes applied for that nasty little zero day.
We can't really win!
"True, but on the other hand, massive amount of work for you making sure your local copies are always up-to-date, and have the latest fixes applied for that nasty little zero day."
But when you discount all the non-essential stuff like multiple analytics scripts and calls to left-pad scripts you could write yourself and typefaces that have just that special version of the letter "i" your designer wanted and all the "social media" scripts you link too to show their icons, just how many are left to track and update?
Use a local proxy to cache all your remotely collected scripts. Have that proxy run a comparison check against the last known good version for all external scripts.
If the code changes, don't update the cache until it has been signed off as safe, at that point you can update your 'known good' version and carry on serving it to your clients.
Ok, so if there's a problem with a valid script and it needs to be updated then that fix might be delayed until you can sign off the update, but that's a lot better than taking the chance of feeding your customers compromised scripts.
This avoids the need to micro-manage all the scripts internally, but injects a safeguard against compromised remote script updates such as the one in this story.
Or does that sound too hard?
> Or does that sound too hard?
No, but if that solution didn't bring its own problems, it'd be a common design.
> Use a local proxy to cache all your remotely collected scripts. Have that proxy run a comparison check against the last known good version for all external scripts.
If you locally proxy then you need to deal with all your traffic. Troy Hunt's blog about this story mentions for his site today that this would be 500GB of minified http data just for jQuery if he didn't offload it to cloudflare CDN. That's costly. You can make all sorts of arguments about how people shouldn't use all these libraries, that we should jump right into notepad to develop our static pages, but that kinda misses the whole point.
Another downside of your local cache model is that your performance will be crap if I'm not geographically close to your server. Most of those problems disappear if your are delivering off googakaflare. Even if you put your own site on a CDN, it's still a download overhead to your first time visitors.
You haven't specified how you differentiate a new good version Vs a new not so good version. Your technique tells you when a script changes but so does SNI, and you can do that in 30 seconds.
The problem with local caches or SNI is that they solve this problem, but block solution to the counter problem. Imagine that instead of being pwnd, this library vendor was privately reported an XSS bug that allowed your (and any other site using the same library's customers) private data to be exposed. Without SNI and with everyone referencing a common version, that XSS flaw could be immediately fixed across millions of websites. With SNI and private caches, you can't do patch management in that way. That is the crux of why it is a hard problem.
> True, but on the other hand, massive amount of work for you making sure your local copies are always up-to-date, and have the latest fixes applied for that nasty little zero day.
Not to mention the underlying assumption that your own security is going to be any better than the original host's or relevant CDN.
1) If a site I depend on gets hacked, I'm not going to know until they tell me or I read about it here.
2) If a site I depend on gets hacked, I have no tools or ability to fix the problem.
3) If a common site gets hacked, then everyone who depends on it gets hacked, per this example. Since crypto-jacking is about making money, this is a particularly important point.
Of course, 1 can be partially mitigated if I'm running a round-trip test from the outside.
The idea of including off-site scripts that have not been digitally signed (recursively) is nuts. That doesn't fix everything, of course, but it is a start.
To paraphrase something in the current edition of The Econimist, in an article about Elon Musk's SpaceX and Tesla companies):
"There will always be complexity. You can choose to have that complexity outside your business, where you have little or no control over it, or you can bring that complexity in-house, where you can see it, measure it and control it."
The two problems with hosting* a local copy of dependencies.
- You don't benefit from the browser cached version that almost certainly exists due to the user previously visiting a site with that plugin.
- You have to pay for that bandwidth**
At the end of the day, you have a trade-off because you need to decide whether to trust a third party to manage risk on your site or whether you want to vet everything. SRI would have prevented this specific hack, true, assuming though you didn't have website authors who saw the error in the console, got the new hash, then blindly applied it to put out the "our customers can't login" fire. It also means that where there is a legitimate vulnerability in the framework, your site cannot be fixed automatically.
*Making a copy of any version that you deploy is important if only to deal with one of these vendors disappearing without notice.
**The irony isn't lost on my me.
"Another good demonstration of why ad blockers and script blockers are essential."
We...ell... there are people who have poor (or no) eyesight. My guess is that they would tend to trust that specific plugin - or at least grudgingly accept it (though I believe you would maybe best run a dedicated text to speech engine locally, but many websites are just sh*t when it comes to being accessible - just try to open them in lynx... my guess is if it does not work there you are out of luck). Maybe most of us who do run NOscript (or so) do have some exceptions that are more or less mandated by the sites we use? Or that make a website usable? Sure, I allow only a few exceptions, and only temporary, but that's all it would take.
So: yes, script blockers are needed and very sensible, but in some cases we poke the holes into that layer of security - out of necessity. And don't give me the "then don't use those websites if they are unusable without scripts"-thing. Some sites mentioned are university homepages. Imagine being a part of that university and you have to organise your work, teaching, learning, outreach, travel claims, .... through their system, which does pull in such a script. In the present case you were lucky if you are sighted and thus could just refuse to run that script - next time it is a script that is actually needed for the operation of the website.
"We...ell... there are people who have poor (or no) eyesight. My guess is that they would tend to trust that specific plugin - or at least grudgingly accept it"
True. But that doesn't mean everyone else also has to trust that specific plugin and have it running in the background every time they visit a site. You're correct that perfect security isn't possible and that at some point we have to trust something, but there's a huge difference between allowing some exceptions where necessary, and expecting every user to blindly load hundreds of external resources just in case a tiny proportion of people might need them at some point.
That's why script and ad blockers are essential. Yes, you need to make some exceptions and so can't eliminate all possible vulnerabilities. But a big problem with the internet in its current state is that the default expectation is that everyone will always trust and run everything regardless of whether there is any reason to do so. Until the opposite is true and the default is to access resources only when actually needed, blockers are necessary to enforce that state.
Yes, but if you take the very useful NoScript, a typical website calls on 8-10 js domains.
It's not always clear what scripts are need-to-have vs eye candy/ad serving. Then, as you enable some, they in turn want to load more.
Obfuscated urls may look malicious but often are just how CDNs work.
(not to mention js-only sites that render nothing until enabled)
It's a massive obnoxious mess and often results in me leaving a site on arrival.
In addition to checking vs last-seen versions, as suggested here, maybe it's time browsers/plugins look at behavior and profile of scripts, including stuff like api calls being made, cpu use and clipboard access. Known-good script hash checks vs a central registry. Possibly uploading never seen code up for analysis.
No need for the equivalent of flashlight apps accessing contacts, for example.
I don't see this as a JS fail per se - any language holding its role in our current web architecture would cause this. But our trust model feels a lot like when we were sending each other EXEs with funny contents in the mid 90s. I don't see it lasting another 20 years.
"Just about every non-trivial website on the planet loads in resources provided by other companies and organizations"
"Another good demonstration of why ad blockers and script blockers are essential."
And a reason to apply a CLUEBAT to those web site authors that propagate this nonsense, who seem to have NO clue at all...
"Another good demonstration of why ad blockers and script blockers are essential."
Until someone pops a miner into one of the adblockers - which happened to one of the Chrome youtube adblockers about two weeks ago.
Noscript caught it - but it was difficult to trace as it only attempted activation on some (but not all) non-https sites and never on the popular ones, pointing the finger at those sites instead of the adblocker.
Just don't do it. It's not worth it. We're seeing reports now nearly every day that third-party scripts, usually ad platforms, get hijacked and that high-profile websites start dropping malware or running coin miners.
Besides, I question the practice of government websites connecting to third-party domains. If you're running a gov site, security is a top-tier priority. This time we had a script being hijacked for coin miners, but what if it got hijacked by credentials-stealing code? Gov sites deal with highly sensitive information, and as such shouldn't run any code that its maintainers aren't 100% what it does. Concretely, what this means, is that they should host their own instance of the service and serve the scripts from their own domain. That this isn't already established policy amounts to sheer lunacy.
Agree about not loading off-site (3rd party) scripts, or anything else.
The other non-first party resources are frequently trackers which don't belong on a true public/gov't site.
Laziness and perhaps some personal gain are the main reasons that developers link out to 3rd party resources.
When a site relies on these it is also exposing itself to DoS and multiple failures as well as slower loading times.
"The other non-first party resources are frequently trackers which don't belong on a true public/gov't site."
Yup.
Google analytics might be "cool" in terms of the statistics it can generate but there are standard stats packages for that which don't use JS, don't slurp up all your personal details to Google - and don't miss people like me who block that website.
It would be interesting to see the ICO asked why they expect visitors to run JS from Twitter and Google, given their focus on data security and personal information privacy.
Banks are equally guilty. They've all opened themselves up to the third party being compromised, their DNS hijacked, or slurping (why the would govt or banks need Google Analytics or fonts loaded from Google or whatever). And then we're all repeatedly taken by surprise when stuff like this happens.
Fine in theory, but in practice this can cause sites not to work correctly.
Solution, keep it in house, audit all third party code and stop being a muppet (yes you so called IT experts out there)
FFS WAKE UP
re
Just about every non-trivial website on the planet loads in resources provided by other companies and organizations – from fonts and menu interfaces to screen readers and translator tools.
@AC
"Solution, keep it in house, audit all third party code and stop being a muppet (yes you so called IT experts out there)"
I wish I was obscenely rich too!
Although, if I was, would I be wasting my life in IT?)
Code needs to be 'fit for purpose'. If the 'purpose' is e.g. a website to advertise a product that will earn your company £50K p.a. you can't afford code audits of JQuery, Ruby or whatever the current flavour of the month is. If you insist on the audit, we'll either go bust or we'll forget the 3rd party stuff and dig out the old copy of Dreamweaver and hand code it in plain HTML.
Other than taking extreme care of sensitive data, there's no perfect solution.
"If the 'purpose' is e.g. a website to advertise a product that will earn your company £50K p.a. you can't afford code audits of JQuery, Ruby or whatever the current flavour of the month is."
If, for want of a proper audit - or reducing the amount of flavour of the month - the consequence is that you end up damaging your would-be customers the loss of reputation, damages and maybe fines is also something you can't afford.
Security may be expensive. Lack of it can cost more.
> Security may be expensive. Lack of it can cost more.
Can you offer any actual examples from your own experience to illustrate the content of your post?
I am curious to know if you really did suggest to your organisation that the entire stack should be audited and analysed before running their latest "buy one get one free" campaign.
Have you ever heard of "ALARP"?
To give you an example from my own field, which is aviation: there are various scenarios during a take-off sequence in which an engine failure may, or inevitably will, depending on the scenario, lead to a crash. As will, rather more obviously, a sufficiently large loss of propulsion at a certain point after the start of the take-off run. We know this. We have actually quantified that risk. What do we do? Do we give up on aeroplanes? Or do we "just" mitigate this to a point where you say "fuck it, beyond this it just wasn't your lucky day".
Happy flying! :-)
"What do we do? Do we give up on aeroplanes? Or do we "just" mitigate this to a point where you say "fuck it, beyond this it just wasn't your lucky day"."
...I'm a scaredicat. Yes, I gave up on planes....and bungi jumping, as well as the 4th rung on ladders :)
...If only I could give up on government websites I'd be a happy boy :)
@Pen-y-gors
Sure, and when all your customers have gone, or no one wants to use the website/service because its not trusted, your profits will go down and you will go bust.
Don't piss off the customer/client as they provide your wages. ferengi rule of acquisition 4098
> Other than taking extreme care of sensitive data, there's no perfect solution.
Yup. Reading through these comments you can easily tell the difference between those posters who feel entitled to an opinion on the basis that they read stuff from websites¹, and those who actually write websites.
I found your other comments very sensible too.
¹ Including those of us who code websites as a hobby outside an organisational context.
" those who actually write websites"
or worse, those tasked with fixing someone else's crap-web-code that uses a bizarre dependency tree of embedded script...
Sometimes the simplest solution is to include the script elements yourself, within the page where it's used, via copy/pasta, and then use 'STYLE' assignments in the HTML tags where appropriate. But, then again, I _HAND_ _CODE_ all of my HTML, and *despise* those dependency trees... and it was SO amazing to see that my changes were correctly reflected on the phones and slabs (when I did it _MY_ way) where they'd cached the included script files unnecessarily and there was no obvious way to flush the cache and get them to re-load... stupid browsers not re-loading the included script. The PC browsers re-loaded it, why can't the Android devices?
yeah - script dependencies NOT working consistently.
>Code needs to be 'fit for purpose'.
+1. This.
We have a problem. We have a bunch of suggestions. I am not an expert but it seems like we need to tally up both the constraints and the issues. Anytime you veer on security extremes at the cost of usability, users go elsewhere or bypass your safeguards. This is a well known factor with security, nothing new to the web.
- it is not feasible to run websites built with only your own code. i.e. javascript as it is known to all browsers, without any code written by anyone else. That puts us in what, pre-2015 JS? No modules, no imports, just good ol'd <scripts src=myself>? Hey, no JS would be even safer.
- if you wish to do so, your competitors will eat you alive.
- complex code written by yourself is not necessarily the most secure code. Yeah, I know some of you will claim otherwise... I don't. I can do my best and then have other people review it, but that's about it.
- hosting JS libraries locally makes it easier for me to trust your site, rather than 20 JS domains, but doesn't change whether the original code is malicious or not. In a way, I'd tend to trust a well known CDN's version of Jquery more than yours, for example.
- Once you accept that some of your code will be based on someone else's code, that brings in security issues. Whether the script from a 3rd party domain or hosted on your server doesn't really matter. The little LPAD quip someone made is funny, but naive... do you really think that lpad is http'd in from the web? No it is merged in by some kinda of bundler. Whether the user sees it or not, it's still code coming from somewhere that was liable to be hijacked at some point.
So we need better ways to deal with potentially harmful code not written by the people running a website. That runs the gamut from the company running the website auditing its library stack for known vulnerabilities to npm checking uploaded code for hacks to your browser becoming even more paranoid than it is.
There are no easy solutions here, just hard problems. That's what collaborative development, based on merging different libraries together means. And that's been around for a long time and is not a web-only/JS-only issue - as soon as you don't control your compiler, you're already at risk - can't remember who demonstrated that, but it was at least 30-40 yrs ago.
Also, let's not confuse our dislikes with actual security risks. I have Google Analytics on perma-block in Noscript, but a _hacked_ Google Analytics slurping more crap than Google usually does - which can justifiably be totally unacceptable to many - is vanishingly unlikely.
Basically agree that govt sites should probably get 3rd-party plugins from a government secure CDN. But even that has problems.
Given the way government IT works, it would take at least nine months, at a cost of £250K to get an updated copy of a JS library installed on the CDN, which is a pain if three new versions have since come out, each of which fixes an important hole.
Be cheaper to write your own library - at least the holes would be your own!
So what's the alternative, exactly?
1. Write, maintain and test everything in house. Oh, and remember to document it too, because otherwise you're just storing up trouble for next week. And even then you'll still have dependencies - on browsers, on server platforms and scripting languages - and vulnerabilities will still creep in. I'm not really seeing the business case for that.
2. Make sure every resource is fully audited, and can't be amended without appropriate hoop jumping. This is marginally less work than (1) (and commensurately slightly less secure), but frankly it's still a shedload of effort for very small return.
3. Avoid scripts entirely. Congratulations, now you spend your whole life saying "no" to the marketing department. Good luck keeping your job, even if your company can survive.
Or 4. Accept that the occasional breach is part of your normal operational costs. Just like you expect employees to pinch some of your stationery, you expect customers to duck out on some of their bills, so you also expect hackers to disrupt some of your transactions. Accept it, model it, budget for it. Move on.
"You say that like it's a bad thing."
I've never worked anywhere where the IT department got a veto over anyone, let alone marketing, so saying "no", even to the most wrong-headed idiocy, can cost your job.
I've always suspected that marketing is a lot less effective than the marketeers like to make out, but they are undeniably good at keeping their jobs and budgets.
> I've always suspected that marketing is a lot less effective than the marketeers like to make out
The larger your marketing department, the less pound for pound effective they are, but make no mistake: good marketing is what makes or breaks your organisation.
A billionaire acquaintance of mine told me very clearly once: "the first employee I hired was a salesman", only the third was a developer (I forget what the second was).
One thing I would recommend to anyone is to take a short marketing course. I found that it really helps to make more sense of our world, and make better buying choices once you can see through the marketer's spells.
now you spend your whole life saying "no" to the marketing department.
Definitely a good idea. Every business should appoint someone full time to do just this. Come GDPR time, now only a few months away, they're the ones most likely to bring big fines down on your company.
> Concretely, what this means, is that they should host their own instance of the service
That messes with Browsealoud's business/licensing model - and it's far from a cheap product. Browsealoud pre-dates the (W3C) Web Speech API which is supported by all current browsers, trivial to implement and should have made the product redundant anyway.
I think the point is you shouldn't need JavaScript in a web browser in the first place.
JavaScript was created when HTML was primitive, as a means of extending a web sites capabilities, and usability etc.
You 'should' now be able to do everything in HTML/CSS, and that includes anything dynamic. If there is something that you can't do in HTML/CSS currently, that needs JavaScript, then HTML/CSS should be extended to add that functionality, assuming it's valid of course.
I'd love to see a time when all the wiizzy stuff can be done without needing to revert to JavaScript, at that point JavaScript could be deprecated, and not just disabled in web browers, but actually removed from the code completely.
AC as I work in a dev area that heavily uses JavaScript, and they know me on here :-)
"I think the point is you shouldn't need JavaScript in a web browser in the first place."
I really wish I could do this 100% of the time. But then, if the customer needs 'google maps' embedded on the page, for whatever reason, you can just have a *teensy* bit of script to support it.
So I guess I'd re-state that: "You shouldn't need any more than a tiny bit of JavaScript, if any at all, in a web browser in the first place"
If there is something that you can't do in HTML/CSS currently, that needs JavaScript, then HTML/CSS should be extended to add that functionality, assuming it's valid of course.
It's Monday, so forgive me if I sound a bit dense. I'm charged with developing a web application that, among other things, has various fancy chart gizmos. These charts are expected to change their appearance at the touch of a button - switch from column charts to bar charts, provide popups for every data point on hover, change the confidence level curve when the user wants to switch from 90% to 91% to 92% confidence, allow the user to change the colors used, etc. The application basically loads all the needed data in the browser, and then the entire interface is controlled with Javascript. Is there any feasible way to do this kind of thing with html and css?
(Some of you may think this is not the appropriate use of a browser. I don't disagree. But clients don't want to install desktop applications. It's been this way for almost 20 years. Screamed at for using Flash, and I'm pretty sure I'll be screamed at soon for using Javascript. )
Most folk who use no script do not block each and every JavaScript. They rather block by default, then whitelist either sites you trust, or specific scripts or whatever. In this specific circumstance, a lot of the sites would have been trusted, and even going through the details, is a JavaScript for a known screen reader going to set off any alarm bells?
> Just about every non-trivial website on the planet loads in
> resources provided by other companies and organizations
Really? OK, adverts. But other than that? Surely at least many of them are self-contained. I hope.
If you are going to use 3rd-party code, you've got a difficult decision to make: import it from the 3rd party when the page loads and you're vulnerable to the 3rd party going down, getting hacked etc. But on the other hand, if a security issue is found then they may be able to fix it without you having to take any action. Copy the code to your own server and you'll find you've not kept up with updates and you get hacked....
"Copy the code to your own server and you'll find you've not kept up with updates and you get hacked."
Why are so many updates required (it seems to be a given in a number of comments)? If it's because the code is a bundle of bugs you'd be better off not having it. If it's because of new "features" (did someone say Agile?) then those updates may be adding more vulnerabilities, not removing them.
If you've tested your site and it functions correctly why would you auto deploy updated dependant files? How do you schedule regression testing? Watch all the third parties for when they change? May as well host the files yourself. You can't be serious about your site is you'll let multiple third parties update bits of it independently...
An interesting idea.
If a scumbag-in-the-middle is 'enhancing' code as it flies down the fibre from the CDN to your server, then signatures could be useful. But if the miscreant has managed to corrupt an official upgrade somehow then you're still stuffed.
There are times I have happy dreams of going back to debugging 10,000 lines of COBOL in the daily accounts overnight batch job. Life was simpler then...
Apparently, you don't understand how signatures work. The keys are referred to as the "crown jewels", and held MUCH more tightly than what they sign. Of course, a complete compromise of your network will (probably) get them, but not all hacks get that far. In particular, you can hack the site that hosts my code all day long, you won't get anywhere near my hosts with my keys.
When China got insiders and exflitrated data from Google, the crown jewels were not touched. And that was before Google started to talk seriously about security.
Much though the fans of crypto currency would like to ignore it - and setting aside the security issues that lead to this particular incident - isn't stuff like this an almost inevitable consequence of the way crypto currencies work?
If you create a system that is based on the premise of "swap processor time for currency" then there are going to be a lot of people who will try to find ways to grab time on other people's processors, for their own gain. Whether it's hacks like this one, emailed worms, or something else, the incentive is going to motivate many people to have a try, and get "free" money.
"If you create a system that is based on the premise of "swap processor time for currency" then there are going to be a lot of people who will try to find ways to grab time on other people's processors, for their own gain. "
This was one of the biggest worries of the camram-spam mailing list around 20 years ago - and in that case the currency (hashcash) was "merely" the ability to deliver email.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
