Loading 100GB of data and analysing 100 years worth of climate data
158 posts • joined 18 Sep 2014
Loading 100GB of data and analysing 100 years worth of climate data
I am sitting in a talk right now where a researcher is analyzin biodiversity around the world.
OR how about the production risk model at a major insurance company?
Or using ulia to model pharmacokinetics?
Joe W, you have it exactly. You can change to using Julia for every one of those use cases, except LabView probably.
Give it a try - but at the Reg says use 0.7 for the moment. You may end up liking it!
The discussion on naming Julia comes around n the Julia discourse regularly.
It is not named after anything in particular.
Those keyboard did make a racket though when you were typing.
Wow. That brings back memories.
As a graduate student in high energy physics, I got my very own 3270 terminal, connected to the mainframe downstairs by twinax cables.
Best keyboard I have ever had.
Many joyful days running virtual machines under CMS. I remember the REIPL CMS command to reboot (Initial Program Load). It probably would all come back if I got a 3270 in front of me!
In later days I got a FALCO terminal at CERN. VT220 emulation, plus a hotkey to Tektronix graphics emulation. I used a 3270 emulator on DEC VMS to access the mainframes there.
Great column and some great stories. Brightened up my Fridays as beer o'clock approaches.
You cram them in a rack because they will likely to be accessed remotely.
OK, these ones are said to be quiet. But powerful workstations, as the article says, are noisy and produce heat.
So you use Teradici PCOIP terminals or Citrix to access them
These guys are good at this, and I have worked with them.
Low power, silent terminal on your desktop, can be on the back of your monitor.
Secure encryption for spooky sites.
Workstations are secure in the data centre, your data never leaves there.
Workstations are on a fast network near the data.
If a workstation fails, you just connect another one in the rack up to the desktop, No crawlin about under desks ad the engineer gets up and running faster.
Fantastic blog from an experienced chemist:
What if the AI is stupid enough to try brewing up FOOF for instance?
In the blog Derek describes compounds so reactive that if they go on fire and you throw the sand bucket on them.. the sand catches fire...
I'm no chemist, but I have seen notices in chemistry departments about compunds which smell so bad you have to notify people. What if the AI fancied a little batch of one of them?
Living in Copenhagen, I can say that the Carlsberg Brewmasters IPA is not half bad.
thinking of that, I might have a tin or two watching the sun go down over the harbour this evening.
I prefer Jacobsen Brown Ale actually, which I find very good. Burp.
David, I believe the target for a realistic Exascale machine is 30MW - not as in target to get above 30, but to get below.
Yes indeed, having worked for several HPC integrators the running costs are just as important as the hardware. So any innovative schemes to improve the PUE, or indeed to reduce the power per transistor switch cycle, are of interest.
None of these designs are low power because at the end of the day a transistor is a transistor.
Well said. There is work going into the power consumption per instructions 'picowatts per flop'.
I saw one commen tthat the advent of GPU computing taught a generation that double precision floating point is not needed for everything. I think there will be more effort put into choosing the appropriate precision for calculations, saving power by making the actual algorithms more power aware.
In the Oxford Supercomputer centre they have a similar system. An inert gas is used to keep the oxygen content in the server room below the point at which ignition will occur, but you can still breathe.
Very sensible safety precautions there are never to work alone.
I do recall feeling a bit woozy after being in the server room for a few hours though.
Thanks AC. I have installed and configured my fair share of 48 port switches, most recently Mellanox 10Gbps switches.
The cabinet looks like a standard wall mount comms cabinet to me, and as it is mounted high up I have never really been curious enough to go poking in it. 200 CAt5 cables is quite bulky. I guess I only need to count the cables coming out of the box.
There is a standard sized wall-mounted cabinet in the basement. I have never had a close look at what is in there.
The babinet is about 10U high, so you are right it probably does not have a whopping big enterprise router in there.
Actually the cabinet is in the underground car park and cables go off up the cable risers to each floor. I would say you are correct, and there must be some small switch in each riser, so each set of apartments is connected to a leaf switch.
The apartment building wher eI live has Hyperoptic as the provider. Blooming marvellous, compared to the awful ADSL service which went before, and would stop working on hot days. Far too many connections multiplexed onto Docklands infrastrucure which went all the way back to the Bermondsey exchange. I saw the excuse was that in the Docks era all the warehouses had alarm lines via telephone. These could not have junction boxes in the streets, as the crims would bypass the alarm.
Anyway - Hyperoptic runs a fat fibre connection to the basement of the building. Every flat gets a Ct5 cable back to the router in the basement. Works great and I was able to stream the 4K test signal from the BBC.
Thanks Stephan. I have been using Virtualbox for years and am a big fan.
I will give it a spin.
I see now what the problem is - Windows 10 Home will not allow the installation of HyperV. Ho hum.
Sorry, but I paid nigh on 2000 quid for a high end laptop, which I love. HP Spectre.
I can boot it into Linux and do all the development work I want.
I paid Micorosft for windows 10 Home, which I actually quite like. But I want to run some Docker containers? Oooh noo mate... shell out some more cash.
IF anyone from Microsoft reads this, you are not encouraging anyone to dabble in Docker with Windows. Not relvant to Dev Spaces of course.
You will find that on downloading Docker for Windows it wil lnto run with Windows 10 Home edition. It needs the Professional Edition.
That is not a great way to encourage anyone to try Docker out.
I think this really flags up that passwords are an outdated concept. Mock as much as you like abotu writing passwords down, but why in the 21st Century are we relying on a series of characters pressed out on what is quaintly known as a 'keyboard'.
I don't have the solution myself, however I rather do like Microsoft Hello and facial recognition.
I realise there will be many responses saying 'this is not secure enough'
For centuries, people have placed great store in physical keys. You can still see keys for ancient castle doors for instance, so we have a cultural appreciation of keys.
I really dont see why more companies dont use smartcards for authentication. You normally have a smartcard on a lanyard, and this is used to open doors within the building.
OK, for the home user and e-commerce sites you wont have a company smartcard. But increasingly we see two factor authentication using a one-time code sent to a mobile phone.
ArrZarr, do some Googling on Chernobyl. It is the opposite with the water. The RBMK reactor has a positive void coefficient. When the water boiled and steam bubbles appeared this caused an INCREASE in power output from the reactor. I also dont believe that the operators added any more water - indeed how can you add more water in what is highly likely to be a closed loop system.
Amazon Kindle - the network is known as Whispernet.
Louis, ASML of course. I worked with them last year on their HPC cluster.
HPC being used to design the next generation of machines.
The interconnect for the Transputer ended up being the Quadrics high speed interconnect.
Have worked with some of those guys, whi are still working in HPC.
AC, you have a point. However, in order to 'push the envelope' and create a new class of machines you do need centrally funded research. You then find the advancements trickle down to the systems which industry uses and pays for.
No European makers of disk or solid-state drives of significance. The same is true for interconnect technology.
Welllllll.. Israel is in the European Song Contest.
So I think Mellanox gets in there too...
Chris, clearly you have been at ISC and the article starts well.
But think of this like an Apollo Moonshot program. When Kennedy announced that the US would land men on the moon before the end of the decade.. well NASA did not have Saturn 5 rockets sitting in a big shed somewhere....
Europe should be pushing the envelope, and constructing an exascale system. We already have outstanding centres in Germany, in Barcelona, and in Switzerland.
The UK of course has EPCC, Daresbury and RAL labs, Met Office
(I would say ECMWF but they're off to Italy) Plus the ARM based Isambard project in Bristol.
I do get your points about the lack of CPU foundries and DRAM foundries in Europe.
but the chips themselves don't have to be made in European territory.
Remember the titanium for the SR-71 was purchased from the Russkies!
I fully think that Europe can bring new ideas to Exascale. Both in hardware and importantly in the software.
And Exascale is power hungry. Maybe we should be looking at locating this thing somewhere near a green energy resource too.
Phil, USB-C handles not only the serial signalling. It handles power input, with various power profiles for different devices.
10 Gbps Ethernet
HDMI graphics up to 4K AFAIK
So you can have a lightweight laptop which plugs into an external full feature GPU (or indeed any other PCI Express card)
Lusty, I agree. If you need a USB storage drive they have USB-C connectors now, and if I am not wrong some models have both that and normal USB.
If you need to use normal USB peripherals buy a USB-C hub which is the size of a packet of chewing gum. My Hperdrive adapter is for all intents and pusrposes a desktop docking station, which you can happily carry with you in the laptop case.
I dont have a Surface. I do have an HP Spectre though and USB-C is the dog's dangly bits.
HP Spectre also usefully has a normal USB-3 connector on the other side, with charge on sleep.
So you can charge your phone, or use a USB powered peripheral from the 'normal' USB port.
I can laso say that the Hyperdrive adapter I have is lovely. USB-C to the laptop. Then you get an HDMI graphics port out, two USBs and a USB-C charging port.
Korev, I agree. I was a bit hesitant to say Gridengine there.
But the Univa variety is strongly being developed. I guess that is not Open Source.
I ike SLurm actually.
Indeed. The dual support model is becoming more common.
Look at PBSPro - they opensourced, and have a dual model of
*) you can't justify paying for support or you do not want to? Fine! Use the open source and you will help to keep the community on a wide base and help with enthusiasm on mailing lists (etc)
*) You want a supported product? Well here you are. And here's a number to call when you need us.
Look at Redhat / CentOS also. I don't think that Redhat have lost much if any revenue with CentOS. They have kept many big name institutes 'in the fold' so to speak, whi were never going to spring for thousands of RHEL or Redhat Desktop licenses.
> and, at least, this tech. wasn't shelved.
And that's why Open Source.
OK, this deal was for DDN to acquire personnel with skills and maintenance contracts.
But in general having Open Source means that a project can be revived if someone thinks the effort justifies it.
I would also mention Gridengine here.
ps. I'm no financial expert but I always though Mellanox did pretty well in the market.
Their products are good and thats why people buy them.
I have installed and configured Mellanox kit for many years. Excellent company and good guys, both on the sales side and the technical support side.
I really, really hope whitepines is wrong. Todays high performance networking gear thrives on openness. On the Infiniband side Mellanox make improvements to their OFED stack which are available on their website but which also filter down to the OFED components which the Linus distributions use.
AC, I completely agree. This is a stupid proposal.
Smooth Newt - are you serious? A huge amount of effort has gone on over the years in networking scanning equipment such as this. They use a standard called DICOM
How would you propose to do this? Writing DICOM studies to removeable media then a radiographer puts the media in another terminal and reads it in to the PACS system?
What a waste of time and sheer drudgery for someone. Hopefully in your scheme the tags for the patient ID etc. are automatically read in.. rekeying anything like that is an invitation to a mixup.
Also what removeable media? I don't know the lifetime of MRI scanners these days I must admit.
When I worked in PET scanning we archived to DAT tapes and gave patients a copy of the scans on Sony Magneto Optical disks. That was many years ago, but I doubt you would be able to get a reader fro these MO disks today.
Plesae have a look at Julia. The "two languages" problem you describe is exactly what Julia is designed to overcome. Code in a modern language, with a REPL and Juyputer notebooks. Runs comparably fast as C.
The HP product page says Mellanox ConnectX-5
"One of my first acts when joining a company now is to book a repeating appointment of 45 minutes between 12:00 - 1:00 every working day for lunch and setting the calendar to refuse all requests during this time."
I am sorry but that is bad. I hve worked in several organisations. In one very well known organisation, all the team went down together for lunch. It was great to just sit down at lunch and talk about what was going on in the world.
In a more recent organisation I worked for people regularly went outside to sit in their cars, alone, at lunchtime. One colleague who I enjoyed working with and respected mightily asked me not to join home at lunch, as this was his 'alone time' as you do. That guy was one of th emost overworked people I have ever met.
I have alarm bells ringing for any organisation where people are either so overworked or so fed up with their co-workers that they want to be along for 45 minutes.
I don't know there... there is a lot more choice these days. The article quite rightly points out that there are more alternatives these days. Fails to mention BeeGFS though which is gainign in popularity. Als CEPH is gaining in the more throughput oriented field, such as particle physics. There was a post of the CEPH mailing list today from a researcher at Lawrence Berkely Lab, who compared to Lustre. SO I think a lot of 'big iron' shops are contemplating or moving to CEPH.
Can someone explain to me ... why in the blue blazes would a supermarket have its own date format? I guess a database expert can explain.
But more importantly why is a format like that not documented and given to all developers?
I know it is not a comparable situation, but I have a great respect for ISO 9001 standards for documents. ISO standards might seem boring in the extreme. However there as a case of a person receiving a radiation overdose because the dose calculation was performed using a superseded document. Far more serious than loyalty points.
I rather thought 1metre positioning was needed if you are aiming a cruise missile at Saddams underground bunker. Or some such.
Nick, so true.
Just go and live in any EU country. You have freedom of movement after all.
After three months you can be thrown back out of you do not have a job or proof that you can support yourself. But the UK press conveniently forgot that one in the hysteria about immigrants.
The UK press also convenienty conflate people from outwith the EU with people from the EU.
I would really appreciate some responses to this, positive or negative.
As someone who has lived out of the UK, the UK does seem to have a very self-centred view of how good it is in industry.
The main unique industries I see in the UK are Rolls-Royce for jet engines, bespoke high end car makers like Jaguar LandRover, Aston Martin. And Formula One.
Yup, I acknowledge that the bespoke car makers are foreign owned, but they still operate in the UK.
We also have world renowned universities and research centres.
But for other industries? Mass car manufacturing is an assembly job which can be moved to any country. Computer manufacturing has been gutted, with Silicon Glen in Scotland being a shadow of its former self.
Referring to Xeon Phi, remember that this descended from Larrabee which was Intels multi-core graphics card https://en.wikipedia.org/wiki/Larrabee_(microarchitecture)
So it is kind of ironic that Intel is coming full circle here.
One thing that is interesting is for large visualizations Intel is pushing CPU based rendering, using Ospray https://www.ospray.org/
I believe that will remain very relevant at the high end.
Looking at Nvidia, there success in the market is not an overnight thing, or an accident. Years ago they made conscious efforts to move into the compute market, and sustained that with Nvidia centres of excellence, continued developments of CUDA, NVlink etc. etc.
So are we in the future going to see the rather bizarre situation of Nvidias chips being bought for compute mainly, and Intel producing the most popular graphics chips?
Time will tell!
The UK currently has 15 systems in the November 2017 Top500 List.
I think that's pretty good really.
The UK is indeed investing in supercomputers.
Look at the Snowden computer at Daresbury Lab.
Look at the Brunel supercomputer at Bristol - built with ARM processors and there is a lot of interest in that. The Goverment recently announced funding for machine learning, as reported on El Reg.
UK Met Office is currently at Number 15, which is pretty respectable.
So what are they doing?
UK - weather forecasting. Materials simulations. Jet engine simulations (Rolls Royce).
CFD (Formula 1, Bloodhound, Rolls Royce etc etc)
AWE - can imagine that would be atomic weapons design then.
GCHQ - hello!
US National Labs - nucelar stockpile stewardship. Simulating how materials age with radiation, and are the bombs safe to store, and are they going to go off when used.
NOAA - weather simulations
NASA - astrophysics and all the other stuff NASA does
Texas TACC - simulations of the human heart etc etc.
Swiss - probable lots of interesting research. But the Swiss centre also houses their meteo centre, so weather forecasting at least. And weather forecasting with the Swiss terrain needs a lot of detail.
China - no idea!
Playstations were indeed used for this in the past.
If I'm not wrong Sony disabled the method to load Linux onto them, so they are now useless for this purpose.
I went to a talk about the official Linux distro for Playstation several years ago.
Chris, that would be a damn good plot. A histogram of initial position in Top500 with a Y-axis of years in the Top500.
Cost isnt given AFAIK so you could not do a cost per year on Top500
I guess I can answer this question myself, but are all the CSV files downloadable from the site?
I guess I also have to assume that when a system is in the list for a second year it retains an identity you can track.
Biting the hand that feeds IT © 1998–2018