If the quants in financial services get tools for running their models more quickly, will the economy get better or worse? Who knows? But thanks to Microsoft, we're all going to find out. At the SC09 supercomputing trade show in Portland, Oregon, this week, Microsoft is trumpeting the parallel programming smarts its baking into …
Four hours to do what exactly?
I call BS. Upon installing a cluster it takes more than five hours just to validate hardware configuration and correct operation. And that's assuming the validation doesn't find any errors in the cable routing, which there always are. So the notion that you tilt up the racks and four hours later you are running is just nonsense.
If they're talking about installing the software, then four hours is an age. Most clusters these days network install on demand, so you can run differing versions of libraries and the like for differing customers. A four hour gap between paying customers is unacceptable.
Replacing hideous Excel macros is a job description
Curiously one of the first contracts I had many, many years ago was replacing an Excel model that took over an hour to run. It was core to this business' oppperations, replaced it with a smal DBI Perl app and stuck on their app server it took under 2 minutes to run. Job too 2 weeks, most of which was spent unraveling years of Excel macros conflated into a hideous mess in order to derive a spec for the code. it's now internally supported and running fine.
If you need a supercomputer to run your macro, consider writing a program ;)
"Curiously one of the first contracts I had many, many years ago was replacing an Excel model that took over an hour to run. It was core to this business' oppperations, replaced it with a smal DBI Perl app and stuck on their app server it took under 2 minutes to run. Job too 2 weeks, most of which was spent unraveling years of Excel macros conflated into a hideous mess in order to derive a spec for the code. it's now internally supported and running fine.
If you need a supercomputer to run your macro, consider writing a program ;)"
True but thats generally not an IT issue is down to business managers to come up with the cash to fund IT to do feasibility and get the work done. The Business Manager has to jump through a commitee to get the cash . . . commitees tend to look at yet another request for their cash for an IT project and drag their feet.
Its a problem I've seen in all the companies I've worked for that workers are forced to continue to work with substandard solutions because getting any cash for improvements out of the business is a real headache - or senior managers have already spent any available money on their own priorities.
Misuse of spreadsheets
Another great example of people misusing Excel. Don't people realise there is a time when they should look at a proper system, based on a database and not an Office product??? There really is no excuse when we have desktop systems like FileMaker/Access right up to SQL and others.
Fools. No wonder these spreadsheetards got us into the economic shit we're in now.
Racehorse pulling a plough
The title says it all really. People with huge datasets need something better. I use to handle datasets of 2 million records in JMP ( running on Linux or Windows workstations) and do complex manipulations very rapidly (seconds)
For complex macro models I guess a fortune awaits anyone who write a tool to automatically convert an Excel model into a compiled program.
@AC 10:06 Misuse of spreadsheets
Couldn't agree more. Excel is the most misused program ever made. People use it for all sorts of stuff, in 99% of the cases NOT for actual spreadsheet work.
Worst case I've seen is some stupid bint using her calculator to put numbers in her Excel-sheet...
No wonder the markets collapsed
End User Computing rules were (supposedly) put in place years ago to prevent end users creating "applications" in Excel on which business decisions are made and Auditors are supposed to highlight any such apps as findings. It doesn't mean Excel can't be used necessarily, but the processes for creating them must be the same as for any other application. Business case, specification, development, testing, documentation etc. And the spreadsheet must then be locked down so that end users can't buggerise it.
So MS has basically developed a product that allows financial companies to more effectively break standard accounting practices. Bravo!
My favourite quote of the day is...
"Fools. No wonder these spreadsheetards got us into the economic shit we're in now."
Ha ha! That is the precise reason why, in the engineering economics class I am taking, I wrote a Python module to run the financial calculations. I don't even use Excel for spreadsheeting!
Genius! I've actually had JPG's and typed document's sent to me embedded in Excel files. It's bad enough that people send images embedded in Word docs...
As a "spreadsheetard"...
.. I can safely say that the reason why you end up with Excel being misused is because it takes a lot of complicated programing and puts it in a simple, adaptable interface. Even when you're at the stage of having to write macros to complete the functions, it's still easier and cheaper than crossing into programing.
Case in point, I wrote our company's stock prediction model using Excel and an addon to retrieve & format data from our main database. Took about 2 hours to write the first draft and from there on out, it was mostly tweaking variables to get it to work properly.
If we'd hired a programer to perform the same task, or purchased the "off the shelf" add on module & had it customised, cost would have run in thousands of pounds and to save what? 10 or 15 minutes per time it's run...
All in all, you can understand why Microsoft has taken this step, it does fill some people's needs!
Once, long ago, people knew that 'computers' were hard to do, complex things, and thus hired people with a proven track record, and academic qualifications, all was well.
Now any arse with a two day course and word-generated certificate is a "expert"
Place head between legs and kiss
@ Paul 87
"If we'd hired a programer to perform the same task, or purchased the "off the shelf" add on module & had it customised, cost would have run in thousands of pounds and to save what? 10 or 15 minutes per time it's run..."
That's where you're so wrong. Just because something belongs in a database doesn't mean you need to either be a "programmer" or hire one. FileMaker, for example, is a lot easier to use than macros and costs a couple of hundred quid.
Whatever happened to using the right tool for the job? Waste all those gigaflops doing things a 486 could do in the same time with a little bit of real code?
How about we apply this logic to building too?
Lego is well known as something that makes it easy to build little models with. Countless people have experience in building with lego and many know exactly how to make a plane from 5 standard (2x4) bricks. So, let's make super-size packs of lego available, that make building a real house possible. That way everyone can design their houses quickly and easily and get them in nice bright colours too. The ultimate in DIY building technology! No more messy concrete! No need for expensive and time-consuming structural engineers and architects. The demand must be there!
Get coat, leave quickly...
Excel being pushed into yet another place it shouldn't go. I hate it when people load stuff into spreadsheets, where the software does its best to second-guess the raw data, mangle it, and so on. I hate it doubly when people want stuff in "Excel format" even though they probably just mean CSV format which they'll then load into Excel and mangle royally, before shitting it back in my direction.
Agreed with the person who coined the term "spreadsheetard": you want a database or just about anything that isn't fiddling with columns in Excel and pretending that you're "understanding" the data. Don't get me started on the shit user interface, cell contents (edited in a tiny text box) that overflow onto other cells, spreadsheet "form" design as if graphical user interface technology just stopped in the early 1980s.
And agreed with the "spreadsheets caused the crisis" assessment, too. Microsoft's stimulus package to the rest of the planet would involve reining Excel in (ultimate wish: taking it away for good), not proliferating the damned thing.
Its not misuse of spreadsheets its missuse of computing
35 years ago when I started in IT there were a nicely worked out set of rules on how to programming.
Along cam MS and told you you don't need them just drag and drop.
Now they need a super computer and a billion lines of code to do what a couple of thousand valves could do in 1958!
Excel propels our business forward
Since I started at my current company I have automated many laborious processes and improved efficiency dramatically by "misusing" excel. Yes, I suppose I could've used any other office product with VBA built in, but excel is my weapon of choice. (I also make and run PHP apps on a little web server too and really that's my real weapon of choice, but it's beside the point)
Why? Because as Paul 87 pointed out, the cost of creating these applications externally or as a compiled program would've been tremendous and as Mark 110 said, this would've also taken forever to get agreed and pushed through, meanwhilst we repeatedly carry out the same monotonous time wasting tasks that make work such a pain in the arse.
The best misuse of excel I've seen is someone that resized all the cells to be small squares ( a few pixels square ) and wrote a macro to flood fill the cells with colour then animated a rotating cube! - http://www.gamasutra.com/view/feature/3563/microsoft_excel_revolutionary_3d_.php
BUT.... If you write something in excel that can take 45 hours to run then you've gone in the wrong direction. You don't need a supercomputer, you need your head examined. Similarly the muppets who embed uncropped screenshots in a spreadsheet and then make a text box to explain what is going on need to be shot.
Microsoft, you've gone in the wrong direction. If you want to attach supercomputers to Office then why don't you do something useful like maintain responsiveness whilst saving autorecover information, or improving the fidelity of the print preview so you don't get random thick gridlines everywhere, or allowing interactive collaboration on the same document by implementing row/cell/sentence level locking.
Can't be arsed to learn a programming language?
Red Hat got you seeing red?
Think Matlab is "too hard"
Now you too can be a scientist. With Microsoft Excel HPC edition, you'll dragging and dropping formulae in no time! While your colleagues are struggling with their Bash and Python scripts, writing parallel coded, Cuda, Buddah, what the heck is all that mumbo-jumbu anyway? You'll be fancy-free, sipping coffee, waiting for your results!
Need to do some climate science to debunk those greenies? There's a Wizard for that!
Need to do some climate science to debunk those Global Warming Denialists? There's a Wizard for that!
Need to simulate hundreds of millions of events? Um, let us get back to you on that one...
Ah that's the solution!
Instead of make Excel faster, throw more CPU at it, perfect!
I briefly contracted some IT work at a company that had Excel pulling data from a rather large DB and then doing some rather simple (add, subtract, etc) math on the results. The rows weren't especially huge, as far as databases go, but they were only able to do ~10 rows at a time, because even that would take Excel over 30 minutes to run, and 15 rows never finished ever, because 100k of data would eventually get Excel to churn through 2gb of memory, and once it started enthusiastically swapping, that was the end. The solution offered at the time by the actual IT staff was "We'll get x64 Windows and Office so you can have 8gb of ram!"
old diehards on here
"Now they need a super computer and a billion lines of code to do what a couple of thousand valves could do in 1958!"
Yeah, how about saving my powerpoint as a movie, or creating a mailshot in word, or a billion applications in excel. Kilburn et al had great fun crowded around atlas 2 watching youtube clips, didn't they?
Can we not move on from the facts that:
a) Microsoft Office does a lot of things well
b) it does not require a 50 year regression in information technology to solve the things which it cannot do.
How long till some old fart replies and says powerpoints a load of new fangled nonsense as well.
The 'Just build a database people' on here are the type of people who, when asked for directions would retort in a nasally voice 'well I wouldn't start from here' and then go home and tell their wife what a wit they are while she thinks of her fitness instructor.
I think I receive what must be the most retarded of retarded documents, its an Excel spead sheet from a version belonging in a past century, embedded in a word document, attached to an email.
Of course, the sender knows best
As another "spreadsheetard"
I've got a load of VBA that builds things on top of the data that comes out of our BW. I work in country finance in a large multinational with a central BW. There's things we need to see which other countries do not. Making the change in the BW would require a change request that would take time and money, whereas writing a bit of VBA to add an extra few columns/split a few rows takes a LOT less of each.
Excel dominates because we're talking about things which have grown up in finance departments in small to medium companies, not IT departments in large companies. These are people who do VBA coding to help with their finance job, it simply wouldn't work to have people who do a bit of finance to go along with their programming job. Getting the code a bit wrong so it takes 20 minutes instead of 5 minutes once a day? Not too much of a problem. Getting the understanding of what the figures represent wrong so that you provide the wrong input into a business decision? Can be a really major problem.
Excel and VBA work well for certain levels. The problem is that it's only to a certain level - 45 hour runs are well beyond that, but until someone invests the time and money to get someone in to re-write it, those books will keep being run.
@ Nexox Enigma
"The rows weren't especially huge, as far as databases go, but they were only able to do ~10 rows at a time, because even that would take Excel over 30 minutes to run, and 15 rows never finished ever, because 100k of data would eventually get Excel to churn through 2gb of memory, and once it started enthusiastically swapping, that was the end."
Oh dear ... sounds very like you couldn't find ANYONE who knew how and when to stop or step out of a loop when necessary. I see this kind of badly coded bullshit all the time. Its sad because VBA is easy and this is just the sort of thing its quite good at if coded properly.
I've managed to get Excel to bandy back and forth thousands of rows in seconds from a live database just fine. Network latency is a real issue here, though.
But seriously, as previously mentioned here, If you have to attach supercomputers to Office files you've kind of missed the point, don't really understand what your trying to do and should just step back. IT probably isn't really you.
As the old (but true) addage goes, "when the only tool you have is a hammer then everything looks like a nail". Never were truer words spoken.
"How long till some old fart replies and says powerpoints a load of new fangled nonsense as well"
About an hour and a half. I must confess to the nasal voice.
Excel is fine in its place. Its place is not as a database, for it lacks enforcement of constraints and data types, among other things. As a place to old small lists, sure--why not?
As a "spreadsheetard"...
I use MS Access and ODBC connections a lot. It is great for identifying tables/fields, relationships and to get the general flow. I can usually come up with a working model of a report or process in a very short time. However, I ALWAYS tell people these are only proof of concept or prototypes.
Unfortunately, these end up as the production solution more often than not. The problem is that there is often nobody available/capable of turning these quick sketches into "proper" programs. I can think of many programming projects dragging on for months that were abandoned in favor of using the "tinker-toy ms office" prototypes. This is especially the case in smaller businesses that can't afford to employ a full-time dedicated programmer.
Instant Legacy solutions
The problem with Excel spreadsheets is that they tend to create instant legacy solutions where only the originator can maintain the model. By all means develop the original model in Excel, but if it becomes production status - look at getting it written in a compiled language along with decent documentation. A well crafted compiled version will inevitably run a lot faster and if done properly make proper use of multiple processors.
There are of course Excel Compilers which go some way to handling those issues.
As a retired APL programmer I hold up my hand and confess "Poacher turned Gamekeeper"
PowerPoint and old farts
"How long till some old fart replies and says powerpoints a load of new fangled nonsense as well."
Already done. Edward Tufte noted the use of PowerPoint rather than a technical report was one of the factors in the faulty decision making surrounding the tile damage to the space shuttle Columbia. That poor decision making lead to the loss of the ship with all hands. The Investigation Board quoted Tufte's conclusions in their report.
See Tufte's "The cognitive style of PowerPoint". Tufte is a data analysis old fart, his "The visual display of quantitative information" being a key book in the field.
Well I never
Excel is too slow, so use perl, python or bash (according to the commentards above).
Does no-one know how to use a compiler any more?
This is a hate of mine, and I know it.
One of my school projects was to build an application (ticketing system) in Excel.
It was a NIGHTMARE, we were meant to do it by recording every action, pah... open up the source, sorted.
So so slow for the most basic things, its not flexible... it's shit.
Folks build huge inefficient spread-sheets for job security. They like having to be kept around to "tweak" the piles of shit as they crash a lot.
How hard can it be? It's mostly VB6 based so it's pretty simple. You'd just have to simulate the Excel calls and compile it as a VB6 program.
I agree in principle with people who say these should be proper programs or database-based apps rather than spreadsheets, but if you think it likely you've never worked in an office.
My last task at my previous job was using VBA code in Access to automatically reformat and import a billion and one Excel spreadsheets- then every subsequent one. Why couldn't the data be fed straight into Access, or the forms be changed to make it easier? "It's been like this for 15 years, why change it?".
Then more VBA was required to run the results of an Access query through a string of Excel spreadsheets (can't just incorporate the code into Access or change the Excel code to make it easier as "it's always been like this") before dumping the results using VBA into a template form in word (again, set in a decade's worth of stone) and telling Outlook to put it into a (standard) email. They select the form they want to process, click "process" and get presented with the finished form for checking- even the standard email's written for them.
A complete masterpiece of code if I'm honest- took bloody ages but it's all documented, commented, fault-tolerant and as efficient as it can be given what I had to work with. It took the whole process a minute to run rather than an hour manually typing and re-typing information- but I've been told it's now being ignored and they're doing it manually again because "it's always been done like that" and they "don't trust the computer to do it right" even though they get to verify the form as correct (haven't had any problems so far). Plus they cleared a lot of their departmental work backlog, meaning that there's less work- meaning there'll probably be layoffs as they just don't need as many people.
That's what you've got to compete with if you're proposing moving things to a proper system- decades of frozen-solid company dogma, people throwing ice over any unfreezing efforts to try and keep their technically unnecessary jobs, and Excel being "free" (it's always on their computers but this new technology means paying money. Why'd you do that when excel's free?).
Any man who can push through a full database and proper program based system for a well established company- and keep them using it- is truly either a man to be reckoned with.