Forget Silicon Valley's talent shortage. The real tech talent wars are being waged beyond the spiritual home of high-tech - and everyone is losing. In fact, the greatest threat to the adoption of industry-changing technologies like Hadoop and Node.js may well be the dearth of talent capable of deploying them effectively. It's no …
The difference between a job and a hobby
> open source ... it gives recruiters a deep view into an engineer's capabilities without the need for an interview.
But you also need to be aware that a lot of people who have written some OSS software have done it for non-financial rewards. Those are completely different motivations from being paid and don't necessarily mean that the individual who wrote a screamingly good piece of software would be prepared to buckle down for the long term to the disciplines of change, deadlines, designs, testing and debugging. Likewise, the second-rate coder might be a far better fit to a product-orientated outfit than a superstar hacker would.
In fact you still need the interview. Not necessarily to assess the technical skills of a candidate but to check on their maturity, self-discipline, reliability, ability to work with others (a team needs a lot more than a bunch of developers) and for those others to check that they can get on with the developers. Some of that can be done over a phone or online, but ultimately nothing beats at least one face-to-face meeting if you're going to enter into a long term business relationship with someone.
you hit the nail on the head.
the key issue is training.
Grads leave uni not knowing about much of the technology that they will be using later in their careers. Companies wanting to hire someone fresh out of uni who is an expert in X, only get their wish is they have influence on the syllabus.
A company that is prepared to invest in its staff will have a constantly improving workforce, none (less) of this 'do the minimum or you're sacked'. As long as they don't get over-skilled and search for pastures new... and that's the danger.
Companies don't want to spend money improving in house staff when they can hire for a specific skill. That way, they don't have to pay to have the staff member not doing productive work while they are being trained, only to have to give them a raise.
Then they whinge about a skills shortage, and want cheap imported labour to fill the 'skills gap' instead of plugging the 'wages gap' or the 'training gap'
HR box ticking
Assume those smart programmers you are hiring can learn.
We have decided to use Hardoop for our next product cos it's Kewl. So we need to only hire Hardoop programmers. We need good ones so we want senior programmers with 10years of Hardoop, intermediate programmers with 5years and junior programmers with 1 year - well those are the numbers we used to use with SQL so it should be OK.
Of course we wouldn't consider hiring a 10year SQL/Java veteran in the hope they can learn - cos they wont be adding value while they are learning.
Oh dear we couldn't hire any 10-year Hardoop programmers we better stick to doing the next app in Foxpro as usual - we have lots of 20years experience Foxpro programmers.
It's no longer safe to ask for x years experience as you risk falling foul of age discrimination legislation. You should only really be asking for specific skills.
I would like to point a difference between Google and JP Morgan
Google and Facebook do not ask applicants to already have 5 years experience in Hadoop. They know that people can learn.
But Google do send people on training courses BEFORE they are employed, to see if they can do things the Google way. This is part of their famously hard interview process - maybe they could actually interview properly, get skilled workers and not ask bloody stupid questions (many of which are available on the net)
The company for which I work require skilled programmers with storage knowledge, or an ability to learn storage, building a unique product, we don't feel the need to have 12+ rounds of interviews and our staff are perfectly skilled enough. If you can't tell if you should employ someone after one or two interviews, this says more about you than about them.
Everyone is dancing around the 800 lbs. (~57 st.) burning gorilla in the room.
There would be Hadoop or insert tech of the day here engineers if punters felt there would be demand for those skills.
Since the gilded executives that run our fair oligarchy fell in love with cheap labor no matter what the actual cost is, it became apparent to anyone with eyes that there was no future in any IT major for anyone unfortunate enough to be born in the West.
You want tech talent?
Show the people working at it that you are not going to give their job away to a visa holder with made up skills because his head shop is cheap and you can take it off your taxes.
That is indeed the case. I'd be willing to bet many of these entrepeneurs and C level execs are quite happy to get rich themselves on the backs of these tech talents they complain cost too much. Boo hoo they might have to start to take less profit themselves in order to distribute the wealth their companies earn.
IT degree not necessarily useless
I'm a very experienced and competent software designer and developer and, when I can be bothered, the contract work I do is very well paid.
My computer science degree certainly wasn't a waste of time.
However, I do have both an interest in and an aptitude for the subject. Many people don't. If you struggle through a computer science degree yet still expect to be handsomely paid, you're going to be disappointed. Which is equally true for lawyers and other professionals: a few are very well paid but the median salary isn't actually very good.
Don't enter a career because you heard it pays well. Find something you enjoy and become good at it.
Haven't really done your homework.
Not too long ago we had a recruiter mouth off, tell the assembled techies they were useless because they couldn't write a CV just the way he liked it. Turns out just about every recruiter wants it slightly differently. And it goes way downhill from there.
Me, I think the jobs market is highly inefficient, not least caused by recruiters, but HR is equally guilty. Regularly you see adverts asking for more years of experience in a technology than said technology exists. The very fact this can happen means a massive disconnect in hiring.
It's worse than that. A good unix generalist will take most any unix and make it dance in no time. Yet hiring is usually after a specific flavour and a specific version. A generalist versed in database technology could equally well learn, and learn quickly, to adapt to something new like hadoop, and maybe in a while we'll see different technologies building on the same ideas; perchance we'll invent something that'll do for big data what unix does for operating a system and small data while at it. This isn't chauvinism; pipes and filters are a bit primitive but general enough to get by for most small data related tasks (and tricks like "xml" are actually not that helpful an idea there). Other systems focus on other things. Want a whole store of cookie-cutter type apps with an eye-candy interfaces? You know where to find it.
Anyhow. Technology generalists have their place, even if they're hard to assess. And, of course, because it often takes quite a bit of time for someone to grok the underlying patterns in computing and become that generalist. "Hiring" is still looking for cookie-cutter people, preferrably young ones that can be thrown at the problem like hardware. So I submit that most HR people and their "best practices", like the execs letting them go about it in that way, are as outdated and as useless any random tech recruiter. Knowledge work really is different from factory work. So sorry. Of course I'm biased, and by now I'm too bitter to find employment ever again. But that doesn't change that even the hiring process is far less about skills than about, well, doing what dogs do when they meet: Sniff each other. See if the new guy would fit in.
And that too is important. Some people get very petty when you don't, even if they ought to know better. (Telling people who work double your hours for less than half your pay they're "not committed" does not earn you brownie points, you silly veep, you.) But it is still more important to get the thing done. You want high expertise? You find it, you train it, you make do with generalists, all of the above. Bitching about it been done, never helped. Fire some of those useless chair 'taters in HR, get rid of your recruiter chums, git off yer arse and try again. And if you fail, well, that too is part of entrepeneurship, now innit?
how about this?
spend less time caring about the rock stars, primadonnas and shiny resume holderns, and more about the guys in your wiring closets and server farms who've proven to be the guys who "made it work" time and time again?
All the fancy apps don't mean sh*te if your infrastructure isn't reliable.
Essentially, stop the emphasis on "think outside the box" and start "thinking ABOUT the box"
big tech jobs, I don't believe it
You mentioned hadoop. I had a look.
Gumtree: 1 job in entire uk
Monster: 36 in 'all jobs'
dice: 599 in the entirety of the US.
You talk about these frankly exotic jobs, dan olds go on about analytics being the next big thing, blather about gpus etc, I'd be damn glad of a chance to do something other than bog standard data crunching, tech support and whatever. Erlang anyone? model checking? some mathematics? (which for 1st time *ever* I have a chance to do at my place, except I'm too busy doing the daily stuff that keeps the company going to get involved). Formal methods? Data mining & MDX etc? Hows about learning some snort, would make a change. Or some actual work reason to write a modest compiler. Haskell or functional stuff generally? Scala, even?
No. Great majority is for Java/mysql/web. The rest is equally mundane. MSSQL is about as exciting as it gets, and then it's just mostly trivial tuning. And some regexps + xslt [*] to break the monotony. I've seen exactly one unusual job (erlang) in five years.
A real challenge would be nice. I just don't see them.
[*] xslt breaks the monotony in the same way breaking a bone stops you being bored.
Maybe they could just use proper databases
Plenty of people know how those work.
Oh no, not again ..
"I recently spoke to a London-based friend who is building a Hadoop-related company. "
This is the problem right there. You don't build hadoop companies any more than you build windows companies. They are tools that you use to solve problems for customers in ways that make them happy to pay you. Perhaps if your friend actually focused on his value proposition rather than the technology, he'd be able to hire people who would help him meet those goals irrespective of the currently fashion trends.
Oh dont be a pedant
His friend was probably building a company in which he needed to map/reduce style large scale number crunching as a primary business reason and had ended up deciding to use hadoop.
Or to put it another, shorter, way: a hadoop company.
Snot pedantry ..honest
That's all very well except the thrust of the article was that this dude couldn't resource his hadoop company. Perhaps if he'd looked for general map/reduce skills he might've come up with a better answer. ... I read it more like someone trying to ride the latest fashion.
I think I get your meaning now. Your earlier post made it sound (to me at least) that you were disparaging the company for choosing hadoop in the first place rather than commenting on their focus for "Hadoop" people rather than "Map/Reduce skills" people.
For the sake of balance
Can we now have an article about the upside of tempting us with Googlesque perks?
I'd like some of those.
Open source devs
Look forward to seeing the documentation then.
Hard getting people who know how to deploy Hadoop? They're not likely to be the people that write code in various open source projects - they're going to be sysadmins/DBA types. How's open source going to help you find them?
That said, not forcing people to move to the cultural vacuum of silicon valley is no bad thing. Companies might learn a thing or two about localisation that way too ;-)
Sorry but that's half the problem.
You need people w a diverse and somewhat divergent skillset.
You need two types of people ... SME and solution architects.
The SME role is the data scientist. The solution architect is the guy who can stand up the hardware and infrastructure as well as tune the cluster and write code.
Your typical DBA types tend to muck things up. (speaking from experience)
Dear Anonymous Cowbert
Perhaps you shouldn't be let near databases then. For an actual admin knows that buzzword soup with TLA vermicelli doesn't go well with getting things done. There's a reason you're not supposed to bring food to the dino pen, you know.
I recently spoke to a London-based friend who is building a Hadoop-related company. He told me that his biggest challenge has been hiring qualified Hadoop engineers, to the point that he may well be driven to put off development of his product business to focus on a training business... that provides Hadoop training. In other words, his solution is to build a pool of qualified Hadoop engineers, since he can't "buy" them.
That's one option, but unlikely to fly at Best Buy or JP Morgan Chase. For these companies, another option is to avoid competing with Silicon Valley and other tech hubs such as New York City and Seattle altogether. The analysts at 451 Group have analysed where skills for Hadoop and also NoSQL congregate: by a huge margin, engineers with these talents live in Silicon Valley. But given the open-source nature of these technologies, it is possible to find them elsewhere, including the Ukraine and (gasp!) Middle America.
Actually Matt you are dead wrong on this point.
Companies like JPM and Target can build the tech competence in house.
The issue is that it takes time and effort that doesn't directly reflect on a tangible ROI. You first have to build the skills and then build the solutions using this tech. It takes about 6 months to train senior developers and architects to the point where they are capable of going off on their own. Of course one could argue that one could use pig or hive to reduce that runway. You can but you still need a core team who can lead. The lower the quality of skilled resources, the longer the runway.
The problem in the non-tech companies like the ones mentioned is in how they view their tech staff and how they treat their tech staff. For these companies to succeed in using Hadoop, they need to start and rethink their organization. Tech savvy companies can more easily adopt this technology and hold the advantage....
But hey, what do I know? ;-)
We've built an Agile Perl team of 12 from scratch this year delivering high quality software for NTT but based in Cornwall, UK.
The quality of candidates has been astounding with CPAN author/contributors, O'Reilly authors as well as many other very talented individuals joining the team.
There are various key issues with regard to motivating people to want to join a company and the typical ones are quality of professional work level, quality of other team members, agile and realistic approaches to deliverables, as well as the other normal ones.
For us, however, we're encapsulating a further value: Quality of living. Allowing people to be able to live outside of a city, a few minutes from the beach, overlooking beautiful rivers and being surrounded by nature is a major factor in attracting high quality applicants.
We are finding many experienced professionals have hand enough of commuting or living squashed together. If anyone would like to discover more about what's happening in Cornwall with HeadForwards (dot com) then do get in touch. I'd be happy to show you around.
this guy must have an MBA
What is a "hadoop developper" anyway?
In my previous company, where I was recruiting the developers, we were building software for all mobile platforms of the day + maintaining a website with a database + building Mac and PC client software too.
There was no question of having "palm" or "SQL" or "Java" or "anything" developer. The skillset required was:
- good skills in algorithmics (this must come from school training - cannot be grafted on later)
- good analytical mind for problem solving
- good interpersonal skills for teamwork, learning and adapting
- ability to demonstrate mastership in some tech speciality (the math behind 4G network protocols for one hire - whatever, just show us you are good at something)
additional plusses were:
- be a team driver on methodology
- be a team driver on new tech (have useful knowledge to share and skills to do so)
- hackers (try to force a hacker into a 1.5 year dev project ...)
- script kiddies (great superficial knowledge in one area, no depth - out!)
London & Hadoop
As currently UK's sole Hadoop committer, I tend to get all the linked In job invites related to Hadoop.
Every so often something hits me related to things in London, but if these are the one's Matt is talking about, the recruiters suck. Things like "exposure to some of the following, Linux, Java, Hadoop, Ocaml, Python, Haskell. ". Or some idiot phoning me up at work -interrupting my coding- to discuss something about spotify. No way to win friends there.
Ignore the technologies: get the statisticians.