Not to heap more misery upon Nokia employees, but if you are a depressed Finnish employee, we recommend you stop reading now and turn to this adorable story about a rabbit adopting some kittens. What you are about to read throws Elop's challenge to the company into a whole new light. Earlier this week Microsoft issued, and then …
It's not hard to make an unbrickable phone
Even Openmoko managed that one FFS! Stick a recovery bootloader/firmware updater somewhere that can't be overwritten(*). Updating over USB and the cable falls out half way through? No worries, hit the button sequence for recovery mode and start again.
* Ok, it can, but you need to take the phone apart and attach a JTAG debugger to do it. And if the user can do that they probably don't need to worry about bricking in the first place.
Makes one assumption
Tthat assumes the recovery bootloader actually does what it's supposed to do. Such a rarely touched feature could easily be DOA with no one noticing or bitrot over time, especially if the firmware format changed, or the signing key, or flash partitions got shifted around during a product's lifetime.
It may even be that even with a recovery mode, that the mode is not visible to the user. Or it may be you're meant to stick a pin in a hole while rebooting to retrigger it. Things that might be non-obvious to a user and require a return to store.
Anyway I suspect Microsoft's problem is the more phones they have, the more headaches they'll have trying to coordinate an update. Are they going to do a big bang update of every phone model, on every network or a rolling barrage based on phones coming out of the QA process. And what does that mean if phones later in QA expose bugs that could affect other phones. Is there going to be a second round of updates.
...but then you could try hacks, break the phone, recover, repeat and eventually get access. Then you could pirate stuff (SHOCK!)
Make it do things MS didn't think of (HORRORS!)
Not have to upgrade so often (CALAMITIES!)
The world would end I tells yah!
The mere fact you could use such a system to reverse out a cock-up such as this is of no consequence to MS and they hold all the cards. They ruined PCs and they seem poised to do the do the same again on phones. It is a real shame. After netbooks, smartphones was the last chance for F/OSS. Unless they pull something out of the hat quick, they'll fail there too.
Things that might be non-obvious to a user and require a return to store.
Well, they *could* always read that little book that comes with the 'device'...
From what I understand the bootloader does have the ability to be forced into recovery mode. Only the one on the majority of Omnia 7's has a bug which stops it from being started in this mode so the phone is unusable after a failure.
Damn valid point and it IS Samsung's fault
When you develop a BSP (board support package) for a mobile phone, it is your responsibility to implement the boot loader and the recovery mechanism for it.
Step 1: define a region for the main boot loader for the device
Step 2: define a region for the "shit hits the fan" boot loader
Step 3: Append a checksum/crc for each of these regions
Then you make a really really simply boot loader which does the following :
1) If no buttons are pressed by the user tries to verify the main boot loader and if it passes CRC check for that region, execute the main boot loader. That boot loader should also be able to recover the device if the "shit hits the fan" bootloader fails CRC.
2) If the user is pressing a specific button combination OR the main bootloader fails CRC check, then execute the "shit hits the fan" bootloader which contains enough code to restore the device via a USB connection.
If nothing else works on the device, this needs to be flawless. The program code to make this work is insanely simplistic and any idiot can program it if they think of it.
If you happen to be Samsung on the other hand, then you make your own boot loader which doesn't take the possibility of data loss or the need to replace either of the boot loaders into effect. As a result, the first time there's a failure, your phone is bricked and requires JTAG programming of the Flash memory which is why on new phone models, it's critical to expose the JTAG port of the device for programming through a docking connector or easily accessed test pins. This way, people who mail ordered the phone and can't get to a service center easily can be a bit enterprising, build a $10 cable and fix it themselves.
This screw up WAS NOT ... I repeat WAS NOT Microsoft's fault. The device should have been recoverable by the user NO MATTER WHAT! The developer of the board support package is the one at fault here and even if it was a subcontractor from Microsoft who made it for Samsung, it was still Samsung's fault for not having verified that the BSP functioned as defined.
My experience with working with Nokia on similar projects is... this would NOT happen to them. They are the most anal retentive assholes regarding these issues and they know to test this thoroughly before it ever leaves the factory. If you think you can just pay someone to design a phone for you and not bother testing it before it ships, you shouldn't be trusted in this business.
How it's done
Having implemented firmware upgrade for an embedded device (it wasn't a phone, but the principle is the same), I can tell you precisely how you make this robust.
Not so much a recovery boot-loader, but a boot-loader that is capable of performing recovery when the usual boot image fails. It doesn't need to do much, and you should never need to upgrade it. In our case, the boot-loader was a small application statically linked to a heavily stripped-down kernel and device drivers. I think we got all of this in about 250KB, including full network stack.
The worst-case recovery scenario is that the flash filing system you are about to boot off is hosed, so the bootloader must be able to integrity check it make a judgement call to reformat and recover.
You either lock the flash sectors containing the image, or for maximum security, use a completely different storage device which is either one-time programmable, or which cannot be programmed in circuit. We used the first of these two option, and for good measure ensured that the boot sector was mapped into read-only pages in the kernel as well as using the processor's write inhibit for the flash-bus.
In case you wonder how the boot device is programmed in the first place, it's either via TJAG, or you use the services of a bulk programming service which tapes and reels the parts for assembly.
So, with this in mind, we can only assume a massive design cock-up, involving one or more of the following:
1) First stage bootloader not capable of recovery
2) First stage bootloader not write protected (and something accidentally overwrote it)
3) Failed Attempt to upgrade the first-stage bootloader
As with the previous poster, I suspect that the problem is the lack of recovery tool that is capable of being used by the customer. Writing one of these is not terribly hard.
The shocking part..
The shocking part of this update was the lack of a need for it.
There was practically nothing in it.
You only push updates if they are REALLY necessary and as you point out, you'd better be bloody sure that it is OK.
This sounds like a major problem; one of the things I liked about the old "Pocket PC" (Windows CE) was that it was almost impossible to brick.
I gave my old Dell Axim X50v to my twelve year old son, and he's changed the O/S on it several times (Pocket PC 2003 all the way up to Windows Mobile 6.5 and back) without bricking it. He even had a few problems along the way, but was able to recover from them and get back to a working device.
perhaps he'd like to go to work for Microsoft
You'd have to sign the child labor agreement, though.
Reading a bit too much in between the lines?
Firmware updates have been happening for years and years on smartphones, and OTA updates are just firmware updates over 3G/wifi rather than over a cable. So Microsoft screwed up with one? This may not be unheard of but it's generally quite rare and certainly not an indication of deep problems with Microsoft's OS.
I also fail to see how a Nokia bureacracy has any relevance at all.
Nokia will become just another Windows Mobile OEM, and will be running Microsoft's software like everyone else - that's their problem, not some imaginary firmware update methodology flaw.
What an utter load of misinformation in this article - please get the facts right before you attempt some Microsoft Bashing.
>>Earlier this week Microsoft issued, and then withdrew, a software update for its Windows Phone devices.
They've suspended the update for samsung only phones!
>>It's a nightmare for the manufacturer, because it requires an expensive return to a repair centre, and the customer, once burned, is much less likely to become a repeat purchaser. That "small number" may be as high as ten per cent.
Yes 10% of updates failed, but over half of the updates were caused by a lack of disk space or a poor internet connection. The update process backs before patching, and a lack of space to back up will prevent the update from taking place - though this doesn't brick the device.
>>Bizarrely, the patch didn't contain any new features. It was a notification about new features to come, the equivalent of that little alert you get in XP telling you that there's a new version of Software Update available
The patch was an update to the updating process - not a notification, and was probably purposely done this was as a trial roll out - which is another reason the patch is being rolled out slowly rather than a big bang approach. Surely it's better to roll out a small update to test the process first before a major update in march!
>> OTA - Over the air updates
You only get a notification an update is available over the air - the update is performed via a fixed connection through zune of the mac windows 7 phone connector, not over the air.
10% is a huge percentage. It's like a 10th!
These are phones that have to be fixed by the phone manufacturer that installed the WP7 badness. Their cost not Microsoft's unless Sammy ask for their money back.
This was a massive fail.
To say otherwise is to try and spread Harmony. Only someone paid to do so would do so.
10% failure rate is inexcusable
Look, there are obviously reasons for why the update failed. That doesnt excuse them happening though. If there wasn't space to backup files, some kind of "I don't have enough space to proceed" message is required. If the internet connection broke halfway through or the firmware was corrupted, then why wasn't the file signed so it could be checksummed?
It isn't acceptable to brick phones except in the rarest of circumstances. Even there it would make sense to provide a mode for the user to be able to recover from that state without an expensive return.
So no phones were bricked?
Or were they?
It was indeed a massive fail...of Samsung's.
The most reported cause of the truly troubled phones (not those that just failed to update, and were fixed by powering off and back on - which was the most common problem) by those who actually studied the fault have all been pointing towards certain firmware versions for the devices...particularly those ones designed for the mobile networks specifications - like T-Mobile UK, whose bootloader is locked to make sure you can't update it yourself, you have to take it to someone with the right jig (I built one), ie a repair centre or store, and pay money to get them to do the update.
My Omnia 7 updated without the slightest hitch because I updated my firmware to a non-T-Mobile branded proper version.
Nope, still Microsoft's Fail...
Nokia can update the same OS over what must be around a hundred different models/form factors, each one with a standard profile or with operator customisations, different regions/languages, etc...
The fact that WP7 can't cope with this isn't Samsung's fault. They did, after all, customise their phones' firmware within what MS allows (what little they let manufacturers customise) and even then it blows up.
Surely nobody was surprised by this?
Popcorn sales say not!
If the Nokia-Microsoft deal amounts to a "special relationship" significantly above that which HTC, Samsung et al have with Microsoft (and I've read hints that Nokia will be able to customise WP7) then in theory they'll be able to have better control over the update testing.
Even iPhone updates cause problems occasionally, but the close tie between hardware and software definitely helps. Hopefully NoWin can do that too.
a very special relationship
the Nokia-Microsoft deal does amount to a "special relationship".
Elop IMHO has huge conflict of interest. Check the "Top 10 Other Holders: MSFT" here:
Ownership Breakdown: MSFT
as of 2/24/11
1. Gates III (William H)
2. Ballmer (Steven A)
3. Marquardt (David F)
4. Daiwa Securities Group Inc
5. Bach (Robert J)
6. Turner (Brian Kevin)
7. Hastings (Reed)
8. Elop (Stephen A) <-----
You cant tell me that hes not throwing all of nokias investment in linux in the bin and going microsoft because its the best thing for the company, when he personally stands to make a small fortune on his microsoft shares.
My last mate who still has a nokia is about to go samsung galaxy s. The linux platform on the n900 was really good and on time. How can this happen?
Microsoft bricking lesson bodes badly for Elop's Brave New Nokia
"But it does shed light on the magnitude of the task Nokia's new CEO Stephen Elop has set the company. I'm not sure the risk has been fully factored in by analysts"
I don't think Elop really cares. If Nokia succeeds at selling Windows phones then great. If the company fails, then MS will buy them out ala Novell.
Re: Microsoft bricking lesson bodes badly for Elop's Brave New Nokia
"I don't think Elop really cares. If Nokia succeeds at selling Windows phones then great. If the company fails, then MS will buy them out ala Novell."
Indeed. MS get their grubby mitts on "intellectual property" to extort cash from others and threaten the competition, whichever way it goes. Anyone who flushes the high-margin product pipeline in such an apparently cavalier fashion and trashes the company's publicly-estimated value is either clueless or wants to bring the company to a state of desperation where an acquisition just looks easier for everyone involved.
And that wouldn't be the first time Elop has been involved in getting a company acquired.
"A small number"
Microsoftian for people affected by their worst mistakes.
Does register get paid by Microsoft competitors
I am still to find an article on Register which somehow doesn't twist a story about Microsoft to make MS look really really bad.
Actually you don't need to twist stories at all
Microsoft does look bad when they try to (think of Danger Sidekick data loss as an example). Oh, and if you want positive stories about Microsoft (aka Get the Facts) just go on Microsoft web sites, nothing will trouble your esteem for them.
What part of "biting the hand" don't you understand?
By your logic, most of the El Reg commentards are being paid by Microsoft competitors too...
MS screwed up!
The story is that they screwed up... They did screw up so I cannot see any other way of reporting the story without making them look bad... No twisting required.
Who are you more critical of? Your Government, or the opposition?
If you want one facing the other way, look at
Replace Microsoft with Apple and...
You sound like all the Apple fanbois who complain about how negative El Reg is to Apple. Maybe they just call them like they see them no twisting needed.
re: make MS look really really bad
No need for twisting - it's just that MS rarely manages to look good, if you take an objective view. It's a huge company with massive resources and no vision, that consistently delivers poorly tested incompatible software, behind schedule and usually by imitating another developer's product.
i bet all the people that 'down voted' your post own (multiple) linux boxes! ha!
why, yes. I do. I have exposure to M$ too, but who needs them at home?
I don't own multiple Linux boxes
...but I am the proud owner of two bricked Windows Mobile phones (Motorola Q and a Samsung Ace) that my 2 year old loves playing with, and I'm currently using an HTC Touch Pro 2 that I'd throw into the toilet if my company wouldn't just give me another one of the same.
I've used Windows phones for the last 6 years and am quite content with the move from XP to W7 - but trust me, you don't have to twist this or be a Mac/Linux fanboi for this to raise significant concern.
Usual tactics of those either malicious, dim-witted, or losing an argument: attack the messenger rather than discuss the point.
Microsoft are to be applauded.
They have managed to get the full desktop experience of using Windows onto a mobile device, other manufacturers will be so jealous... or not.
No they have really.....
Looks like the full desktop experience to me, slow boot up, plenty of crashes and updates making a mess of stuff that was working before.
I dare say if anybody manages to keep their phone for a year you'll find it running at 1/2 of the speed of when you got it and wondering where the bloody hell all the disk space has gone and wishing you had bought something else.
Oh, and it will be full of viruses just by a website drive by....
Mines the one with the android in the pocket....
have a beer.
Thanks for my one good laugh of the day.
The difference being . .
that customers of mobile phones have more choice today than those available to desktop and laptop buyers in the past. The one big thing going in Microsoft's favour is that most consumers have very short memories and don't give a fig for the technicalities - they just want new, hip and shiny. (By "most" I do not include the denizens of this forum of course).
Way to spin a story!
Engaget had this story 2 days ago. It only effects a few Samsung handsets with specific firmware revisions works fine with other vendors. The update isn't just a notification it updates the update code.
The so called 'bricking' of the device can apparently be fixed by removeing the battery, putting it back in and turning on the phone.
Loved the self reference...
I don't even know what a change management paradigm is (remember)
Yes I do remember. Thanks for the lulz!
So how many WP7 deployments?
So MS haven't given any figures on WP7 deployments yet, but we know that "small number" of users were affected... and now also know that that's 10% of WP7 users. LOL! Clearly these two messages bypassed MS Public Relations.
Shat bricks myself last night...
OK, setup. N8, creaky old Ubuntu/XP-2 IBM T30 dual-boot laptop, which has been staggeringly reliable over the last, oh, 8 years or so. Mobile internet connection (no, it loads onto the PC first, _then_ transfers, so no worries).
I decided to upgrade because when I tried to put the music player as a main panel shortcut, it insisted in putting the radio there instead. I had two radios on the main panel.
OK, go S/W update. Yep, update available, but at 152 megs, only way is through OVI Suite.
Link all together, go OVI, and find there's : Updates for OVI, N8, and some apps.
Now, this *might* be a bit of my fault, as the lappie only has 256M of ram. (Stop sniggering - I only use it in XP mode for OVI) As a result, this Alzeimer's suffering 'device' takes a full 20-30 minutes to stop thrashing its disk on startup, with NO apps running. Not that I can get an app running in that time, the mouse cursor is as nimble as a slug. Ubuntu 10.10, it thrashes 2 minutes.
Offered the choices:
1) update OVI
2) Update N8
3) update apps.
Fearful I'd need the cursed "Restart Windows to complete..." stuff, I (foolishly?) decided to skip 1) as I wanted to do something else with my life that evening. After all, the version I had worked, it's only an upgrade/reflash innit?
So, choose 2) and start the 1 hour download. Download OK. Start the upgrade to the N8. Numerous pop-up messages telling me "This device can perform better if connected to a USB 2.0 port. Click here for a list of available ports, which instantly gives the message "there are no availble 2,0 ports".
Aside - Why the Fuc*king hell couln*t the stupid twat who wrote that bit of code not have got it to check first" Alice-in-Wonderland "It's rude to offer what you don't have". I digress.
OVI lied. This process will take about 15 minutes. Who'se minutes? Anyway 3/4 of the way through = ½ hour, while the N8 is now displaying 'test mode', OVI tells me there's been a problem, disconnect the USB, reconnect, and click "here". OK, Mummy knows best, so I did. B'stard started from the beginning again. Did it twice more, at which point I'd had enough, take it back to the shop, and they can reflash the fuc*ker. All I had was a white screen with NOKIA written on it.
Turn off, turn on, and - I'll be jiggered, it groaned into life, with umpteen messages. The bug I started this process for is now fixed beautifully. Does seem more responsive, too.
Daft thing is, I did upgrade OVI afterwards, and - it doesn't require a reboot.
Start all above at 3:30, finally finish (ok, loaded latest maps for Finland afterwards) at 9:00. B'stard made me miss 'happy hour'.
An aside, I went to the Nokia.fi site last night. After a few seconds, this big blue banner appears. Clicking 'Sulje" - close didn't. Goto Nokia.com. Same banner, went away obediently on clicking 'close'.
Try with Firefox, no problems.
Now, as it only fucks up in the Nokia.fi site*, and only with Explorer, do you think Elop is trying to tell the Finnish Nokians something...
*I expect it's been fixed now.
Sorry, can't remember what version of explorer I have, and I can't be arsed to endure ½ hour of disk-thrashing to find out, nor go down 4 flights of stairs to retrieve the T30's bits from the car park, and put it back together again...(joke - it's only 3 flights...)
Mat, 'cos it kinda looks like a brick.
So what you are trying to say is that you ignored all the instructions and minimum machine specs and are now upset that the update did not go smoothly? Nokia software may be shite, but for this one I think you only have yourself to blame. On a plus note, an OTA update via wireless would have saved you a lot of hassle.
Nope. Not available. Commands me to go to OVI suite. Not sure minimum machine specs come into this with only one prog. running. Ovi does everything else. You're probably correct that I should've upgraded OVI first (it only took 2 hours, but no reboots), as it tells that this fixes certain connection problems between PC and N8. But I'm buggered if I'll do another upgrade on this setup.
Assuming there ever will be one.
Forgot to mention
The bozo HW/Cost-cutting dept. at Nokia decided to scrimp and save and give 256 megs. for system memory. OK, oodles for porn^Wmedia and apps. in the main memory, but... there's no room for a fallback if the download is 157 smegs and goes tits-up. Assuming the "E:drive" isn't used as a temp.
Jeepers, wouldn't have added more than a dollar to the build price to double it. 50c. if I were the buyer...No bricks!
A botched update was expected of course, and up to 10 percent failures? Delightful. To blame both the users and the handset makers was a stroke of genius. Over the top. Please wait 4-6 weeks for a direct ship replacement for your phone? A work of art.
A shame there's no way to way to blame the carriers as well for the trifecta. That would have been perfect. Hopefully there's a way to work that into the main update.
Mirosoft still manages to do software
In their unimaginative, clunky way (let's leave their business ethics for another day) But Nokia can't do software at all. They bought Symbian and destroyed it. They bought Trolltech and were about to finish it as well... Elop is a trojan horse, but one that the Nokia board and the previous boss Jorma Ollila brought in knowing very well what they were getting.
Lot of other opinions ...
We understand your charter includes speculative conclusions, but a lot of other opinions are being withheld pending a formal deal and the subsequent details, and actions, that would follow.
Fields of green...
lot of 'turf in here....
sadly not directly
Unfortunately at El Reg many of the readers that defend M$ aren't directly paid by M$ but are hack clueless help desk monkey MCSEs who are deathly afraid of having to ever use the command line. Point and click only baby.
- Product round-up Ten excellent FREE PC apps to brighten your Windows
- Hi-torque tank engines: EXTREME car hacking with The Register
- Review What's MISSING on Amazon Fire Phone... and why it WON'T set the world alight
- Product round-up Trousers down for six of the best affordable Androids
- Why did it take antivirus giants YEARS to drill into super-scary Regin? Symantec responds...