The inner workings of solid state storage devices are so fundamentally different from traditional hard drives that forensic investigators can no longer rely on current preservation techniques when admitting evidence stored on them in court cases, Australian scientists said in a research paper. Data stored on Flash drives is …
The smell of acid in courtrooms
I had this image of a label saying "Press firmly here to release acid to destroy data chips".
Indeed the two articles don't conflict as you point out "... can't be reliably deleted or *preserved* ...". So now we get to have wondrous fun in courtrooms... arguing over whether these bits are virgin, faithful, or have been shopped around some.
when did hdds die????
Who is saying drive tech will be dead anytime soon?
...if you're up to no good, get an SSD
Seems to be somewhat at odds with http://www.theregister.co.uk/2011/02/21/flash_drive_erasing_peril/
So which is it: Impossible to wipe, or at risk of getting wiped while you're not looking?
Seems to be nothing of the sort
Read all of the article and you will see that this is explained.
I seem to have missed the last 2 paragraphs somehow, which do indeed explain it
I think the point is that the SSD is only garbage-collecting its own data structures and duplicate copies of blocks; it's not doing filesystem-level garbage collection.
When you delete a file on your PC, the OS just updates the directories and FAT (or equivalent). There is no signal to the drive that the blocks which contained the file data are no longer needed, and so those blocks will persist. This sort of feature *is* just coming available in high-end SANs, so that thinly-provisioned space can be reclaimed when files are deleted, but it needs support both in the OS and the device.
However, this article does suggest that overwriting your file blocks with zeros *might* actually have some value for flash drives, because the previous copies of the blocks are then eligible for garbage collection and the drive might erase them in the background at some point in the future. Or it might not, as it sees fit.
"When you delete a file on your PC, the OS just updates the directories and FAT (or equivalent). There is no signal to the drive that the blocks which contained the file data are no longer needed"
Apparently someone hasn't been keeping up on what TRIM is all about...
"However, this article does suggest that overwriting your file blocks with zeros *might* actually have some value for flash drives"
Yes, smart SSDs will dedup the zero-padded blocks, and thus, not actually fill up your HDD with zeroes....thus the "can not be deleted through traditional means" bit of the other article...
When will commentards and "scientists" in general stop treating SSDs like a traditional spinning disk and realize it for what it is? SSDs and their data are as much of a moving target as RAM with an OS using address randomization. Files aren't even stored in sequential blocks! Go read a wiki page at the very least.
RE: Garbage collection...
If someone else hasn't already said, you don't need to write zeros. In fact, when you write zeros, if the drive doesn't think the block is reliable, your zeros might get written somewhere else.
The drive itself, when a file is deleted, will then "garbage collect" and actually erase the data to prep the block for a faster write.
Both issues are present in SSDs - if you try to wipe or overwrite a specific file, your new data might get written to another location and the old data left alone and still sitting there. And if you delete a file, the data may actually go away later as the drive does its thing in preparation for the next write.
As someone remarked on Slashdot, if you have time, just rm -r * and the data will mostly all actually go away without the need for a lengthy patterned wipe that might not actually wipe all the data. And if you are writing things that you really don't want anyone else to recover, write it encrypted. That way if anything is recovered, it has to be decrypted to be of any use.
Next gen hard drives will do that as well
Overlapping recording will most likely use similar techniques and a non-overlap area or a flash cache as a buffer. So even hard drives will stop being a "forever" evidence.
Investigation will move to the Cloud
Online storage and backup will give many clues. Centralised infrastructure for both business and personal will be the norm.....
Plus let's not forget there will still be plenty of evidence left behind that is not deleted files.....
They will not be able to charge people with "making" all the porn images they find in the cache that were never saved just looked at once.
So if they take my HDD, they can do a sector scan and find everything that was ever on it, and say with reasonable surety that I did it.
What do they do with the cloud? Do a sector scan of PBs of storage? How do they know who saved what? If I delete something from the cloud and the storage blocks are released, how can they ever be associated with me?
Although I do like the sound of the police taking all the HDDs of Google away for months of analysis because someone accused me of taking a photograph of a building! That might put some pressure on the police to act a bit more reasonably.
they just ask google.
google knows all you did on the internet.
Great idea - let's allow someone to tamper with the drive before forensics start looking at it. I'm sure the defense team would just let that one slip by!
Right track, wrong solution
I have put much thought into this since I've been developing data recovery algorithms since the days when 20 megabyte drives were considered a luxury ;)
The concerns involved at this point are more related to the fact that on power up of the drive, possibly before the controller itself becomes active, the drive begins to flush its "mark for deletion" cache. This occurs by sending the delete block command to the Flash memory chips for the blocks that are to be readied for writing. This is an important thing to do as you would want to flush these sectors are quickly as possible to avoid issues related to power losses. After all, if the mark for deletion table is run multiple times, then extra writes will occur thereby degrading the sectors more rapidly. It's just a good algorithm to follow.
The drawback to this approach with regards to forensics is that simply powering up the drive, even attempting to burst a "do no write" or "enter forensics mode" command over the controller probably would not issue quickly enough to avoid the drive being written to and therefore rendering the drive as altered and invalid with regard to evidence requirements.
Forensics MUST copy the drive unaltered, sector by sector to an image which can be used to recover the media, leaving the original drive in tact before any analysis occurs.
The correct solutions would be :
1) require a jumper to be present on all devices to block garbage collector from initializing.
This solution sucks because it would take another year or two before the jumper is present and therefore would only be ok for newer drives.
2) require that the controller of the drive is flash programmable from a JTAG circuit so that the firmware can be altered in an environment where the flash chips themselves are not powered up. This also sucks since if I were the one being prosecuted, I would argue that this modification also counts as altering the contents of the drive.
3) program a controller chip separately, then desolder the original controller chip and solder the forensics controller chip to the board. This solution is great, but requires that the forensics firm physically "damage" the device in question. I'd imagine that very quickly the quality of the forensics companies' surface mount rework technicians would come into question and would quickly become an issue as to whether they altered the data accidentally.
For part suggestion 2 or 3, the next huge problem is, can you reliably keep track of all those chips and firmwares. There will be thousands of different models and revisions in the future.
4) access the JTAG ports for the individual flash chips (they should probably be accessible on all devices), then power the chips up and perform a JTAG read of all the data. This is slow and boring, but it is 100% reliable and it is 100% guaranteed to not alter the physical device or the data on the device. Therefore leaving the confiscated device in pristine order.
As a data recovery "expert". I recently recovered "all recoverable data" from a RAID 0 stripped set which someone had actually formatted and installed a fresh copy of Windows XP on... one of the drives. Having done this by imaging both drives and developing a tool to detect the raid parameters and reassembled the striped set into a single linear image from which I reconstructed as much of the original MFT as possible and then recovered images through algorithmic steps followed by a brute force method of scanning the image for JPEG jfif signatures and then reassembled photos by using simple linear reassembly as well as scanning "likely places" for missing sectors. I feel pretty comfortable calling myself a hard drive data recovery "expert" which means, I'm pretty good at it, at least good enough that I'll recover more data than most people will from a drive unless needing a clean room.
At least in my "expert" beliefs the only correct method of performing Flash based forensics is to :
1) create an image file via JTAG of each individual flash chip on the device.
2) copy these images to an identical device
3) find out from the device manufacturer how to read the sector mapping table from the controller via JTAG.
4) upload the sector mapping table to the duplicate device.
5) image the device in question to a forensics workstation
6) work from the image files.
Anyone analysing a drive can insert data on to it, you will never escape the fact that you have to trust the authorities, its not like a material sample where you could split it and give half to the defence - and even then a corrupt prosecutor could substitute-in something incriminating...
The solution - for flash drives - would seem to be to separate the controller from the commodity flash chip - most USB sticks use a simple parallel-interface Flash without any firmware on it. Similarly, Flash drives based on SATA or any other standard will rack-up inexpensive dedicated flash, with just one instance of the controller, in a different chip.
This misses the point
Issues of reliability of evidence aside, in many cases, the drive will have harvested deleted blocks well before the device is seized and a forensic examination can take place.
The thing with magnetic media such as spinning disks is that even overwriting data won't stop someone from reading the old data. Why? The old data still has a residual magnetic signature remaining that can be detected. This is why the DoD and the like incorporate 7-pass (or better) random overwrites for the whole drive. This is also where the idea of a "Shred" delete came in: overwrite the file A LOT of times with random garbage in an effort to purge the latent signature of the old data.
Contrast this to flash, which doesn't have such a problem, since the data is represented by a contained charge, rather than the polar position of a magnetic bit. Once a TRIM has been issued and the cells purged of a charge, one won't be able to determine if there was a charge there to begin with. Issuing a "secure erase" on a drive may purge the allocation tables, and eventually the flash will get garbage-collected and purged, but the time that it takes to accomplish this is indeterminate. This being said though, I'd still rather have my virii and hax0r tools on a small garbage-collection-plus-TRIM capable SSD with a panic-button script that would "secure erase" my drive and then start spewing junk to it in hopes of finishing off the actual data. With a UPS, SWAT would have physically unplug the machine to stop it, and by then it's too late. At least using spindle drives, police had a chance of stopping the overwrites before much of the drive could be overwritten even just once, let alone a full 7+ times. If you think the tools written by black hats are cool, you'd love their unreleased "protective measures."
Of course, the knee-jerk reaction of Law-Enforcement Officials...
... will be to propose legislation to make mass-storage garbage collection illegal, even if most of them don't understand -- from a technical standpoint -- why these devices work the way they do.**
Much the same way that the RIAA and MPAA proposed legislation to force ADC (Analog-to-Digital Converter) and DAC (Digital-to-Analog Converter) chip makers to include copy protection technologies in every chip, regardless of sampling resolution, rate, and intended use. Never mind that your average hardware hacker with a basic knowledge of electronics can build a workable device on his own using standard, off-the-shelf components purchased from his/her local Radio Shack or other parts catalogue.
**Not all, though: Robert Morris and various other colleagues of his were quite well educated in the Sciences, and helped Clifford Stoll nab a KGB flunkie trawling the nascent Internet for US military secrets.
Pretty soon it will be illegal to open any electronic device unless you are a professional service technician working for a third party client and are CRB checked. Obviously for our safety, as [make up number] of people die annualy from playing around with teh leccy boxes.
That's if you can even find the special dodecahedral holographic screwdriver that you will need to open the next iPhone, which will probably spray you with coded security ink if you do manage to crack it open.
If they have nothing to hide, they have nothing to fear. What are they planning on putting inside their gadgets that we won't be allowed to see? Some kind of modern day Woodpecker signal?
I for one am doing a roaring trade in tinfoil.
Too little too late.
First off, garbage collection is not an option. It's a requirement. Disabling garbage collection on the device will create serious flaws in the technology and set it back by 3-5 years.
Garbage collection, write balancing and sector remapping are the key factors which made flash storage available to begin with. It was even more important than the price of Flash itself. Flash has a VERY short life span if you don't run these algorithms.
You could require that all drive manufacturers delay startup of their garbage collectors until after the controller has had the opportunity to signal whether it should be enabled or not, but it's a waste of time.
1) Anyone worried about forensics can ebay a pre-regulation flash drive already. So, it'll be 10 years or longer before this had any real positive impact. The criminals that are too dumb to do this in the first place, well drive forensics is probably a waste of money for them. They'll incriminate themselves without it.
2) Another major limiting factor of the law is that there are many different types of drive controllers. Unlike traditional hard drives which were linked through a standard cabled interface like IDE, S-ATA, SAS, SCSI etc..., flash devices are often USB direct to the controller, PCIe, mini-PCIe, DoC, compact flash, cardbus, PCI, well... dozens if not as many as 100 different types of interfaces. I have even seen one device which was only connectible as a WIFI device and the Flash controller itself was part of the microcontroller, so there would be absolutely no way of reading the data through the controller without altering the data.
3) Flash isn't like hard drives. You can actually buy a $100 7 port USB thumb drive raid. It as probably made by jimbob in his basement using off the shelf chips. Hard drives required respectable companies to make with massive clean rooms and such. Flash drives can be built by designing and manufacturing a $2 card and soldering it together with a BGA rework station in your basement. So you can always order a drive from someplace which doesn't regulate. The government would have to start treating all flash chips as controlled substances to avoid this.
I can go on for a long time, but in short, regulation for Flash devices isn't good enough. The only real requirement should be "all flash chips on all flash drive devices should expose the JTAG port" and this should already be the case since jtag is pretty much always wired up to test before shipping.
Thanks for sharing your experience and thoughts; they were as worthwhile as the article itself.
A2D D2A (encryption protection)
Wasn't that Biden that suggested it? I could of sworn it was. I was absolutely blown back this morning when I read what you typed. I remember when this was proposed, and I remember pitching the exact same argument as you do today, back then as I was working with chips building a digital audio delay from a pair of AD7569JN 's. And at the time I was rat-wiring (okay soldering) a plethora of wires, for the ram chips.
My argument back then was that these jackass officials of the establishment have no electronics background and should not be allowed to make laws on things which they don't fully understand. I'm from the generation who knew, if you can completely tear a TV set apart and figure out how to build your own, there would be no law saying, you couldn't technically clone with your own parts, it and slap your own name on it. Everyone I know who had learned electronics knew this. Yes the TV set (a the highest level of assembly) has a copyright, but the chips, caps, resistors, and coils don't.
I also mentioned back at the time, the lack of a2d d2a books for Americans. I had to buy the little pink book from the UK (i was pissed at the US's Lack in this area, not the UK) to fucking learn it back then. I was so pissed at the time. Oh yeah there were a couple $1000 books at UC Davis. (a college I could never afford to go to, yet live just down the street from) I could have bought from their overpriced library. Amazing, there was actually more knowledge in Analog Devices tech manuals back then, which you could simply get for free from AD. Even providing schematics, with working examples in a basic circuit.
I always thought it funny, that American electronics as taught in college, barely scratched A2D D2A topic, I nearly got burnt out at the time trying to create my "digital delay" at home with crap scraps of vague useless information (bucket brigade method mostly), while trying to finish classes in whatever ... transmitters or whatever it was at the time. The pink book was a godsend to me at the time, putting together just enough concepts that I finally understood A2D and D2A at a basic functional level. Keep in mind computers were 8086 / 286's back then.
Was my stupid project with dual AD7569JN 's a project in frustration and market deception at the expense of really trying to learn? Perhaps. FYI - I eventually succeeded, but after some time, a wire broke, and I accidentally burned up one half of the delay, but one half still worked. Anyway. It sounded about as crappy as those phone answer chips, but I always knew, I made the delay, not some manufacturer like lexicon.
I was actually trying to clone a lexicon model 97. I never was able to pulse width modulate the signals properly, to reach my goal of creating a cloned model 97 e.g. Lexicon Super Prime Time Digital delay
Now my eyesight is so fucked up it will never happen, but I have the basics under my belt, I learned a ton, about things I never would have known like the DIG216's (digitally controlled analog switches) and I have since then been able to service my own Lexicons. Even without the fucking logic analyzer.
I can only wonder if the same situation exists today in our schools.
PS: radio shack (a great symbol actually) has now degraded into a fucking mobile phone dealer -IMO Sure you can still get a few parts there, but it's nothing like it was for learning logic, or analog, or tx / rx.
@pengwyn: "...should not be allowed to make laws...
...on things which they don't fully understand."
Yeah, but if we enforced that kind of rule, government would be totally crippled and ineffective.
Oh, wait... It already is, scratch that.
An 80GB drive?
Although I imagine it is not that much more difficult to extract data from 2TB magnetic drives, you should at least research it. We aren't going to be using 80GB drives for long, whereas we probably will be using 2 and 3TB hard drives for a while now, so not even bothering to test one shows a stunning lack of insight.
Though I suppose it is the findings regarding SSDs that they were actually interested in, but some dumb-ass college professor insisted that they also test a hard drive from his old clunky piece of shit computer, so that they can pretend that they are doing proper scientific research. The time would have been better spent testing multiple brands and types of SSD but let's not get technical or anything.
I find this news to be satisfying purely from the perspective that technology not only makes it easier for law enforcement officials to do their jobs, but simultaneously harder too. Although I have no particular interest in breaking the law myself, and a strong self-interest in being protected from criminals, I still can't seem to muster a shit in favour of the forensic investigators. Reality bites.
I love the idea that technology is a sneaky motherfucker betting on both teams, resisting any and all attempts at partisanship. On the one hand cops can now spy on everyone from the comfort of their easy chairs, on the other hand we have access to encryption that the cops can never break. Maybe life is fair after all.
Drive controllers independent of OS drivers
What's happening here is that the SSD controllers are independent of the OS drivers. Thus, the drive firmware is doing what it wants on its own time. The "file system" presented to the OS is virtual in the first place, and the firmware is only interested in presenting a reliable facade. Writing to sector 214 on the drive is not always going to be chip block 0x2e9f, and just because you arent using the drive doesn't mean that it isn't going through some unkown optimization, like moving data from a block with 10% reliability to a block with 90% reliability.
You want reliable forensics? Remove the chips from the device, and then read them.
Nice one for the criminals though!
1. Government organisation misplaces drive (Crazy concept, right?!) with legally grey, yet probably reliable personal info on it
2. Criminal gets hands on it and abuses info
3. Police want to use hdd as evidence in court but can't because the info is in a legally grey area.
(Lack of tinfoil hat/doomsayer icon so picked closest)
Digital Evidence Suspect Anyway.
Everything from a suspects hard drive is 'suspect' anyway. Hard drives don't keep evidence, they keep bits, and these bits can be fiddled with by anyone with access to the hardware or administrative access to the software. Some of these people may not like you http://www.net-security.org/secworld.php?id=9090 . If so, you can only hope they're as dumb as that guy.
I offer 'privacy protection' service for my clients computers. As part of the service I have a program like eraser over write the free space on the drives with zeros. I document when I start this service for them so the lawyers can't say they just started this for the purpose of destroying evidence. Haven't done it for any SSD drives, needless to say software will have to improve before ssd's can be trusted to get rid of your data.
The sooner deleted means completely and forever gone off the drive, the better humankind will be.
"The sooner deleted means completely and forever gone off the drive, the better humankind will be."
I'm the other person in this universe who thinks the phrase "marked for deletion" is at best gibberish and at worst doublespeak. There used to be more of our kind, but I don't know, mass extinction by comet or something. In any case, I'm glad you are alive.
I took a somewhat different different approach to 'privacy protection' for SSD drives - Redaction Unless Static Text (RUST). Rather than writing zero's, web documents contain real working links, to nowhere. http://www.rustprivacy.org/ (http://purl.org/pii/terms/)
"Corrosion of Evidence" Used To Be Called...
Heat Death of the Universe, etc, your data is not gonna get MORE correct.
Conflicts with another study
Another research study (link) suggest recovering data from SSD's is easier than traditional HDD's. Quite frankly it may just depend on how you look at it, the legal standard may be higher than what I would be worried about beign recovered...
Yes, it does.
And if you check out http://hardware.slashdot.org/story/11/03/01/1740240/SSDs-Cause-Crisis-For-Digital-Forensics , You'll see, in the comments, that the papers' author attempts to explain this.
The universe just LOVES chaos. Everything degrades over time due to expansion at the molecular level. We and everything else in the universe is flying apart. I know. I know...you are gonna mention blackholes being the opposite right?
Anyhow one wonders if the recent case against a suspected terrist that took 9 months to crack his encryption used an SSD drive or your bog standard magnetic.
Best way to kill data? Icon related.
...expansion at the molecular level...
That explains me expanding waistline !
I call dibs on that flash drive when they're done with it!
I just want a fast pocket drive!!!
Not a new problem.
This is common problem when looking at raw blocks of journaled file systems but with another level of indirection. The data isn't useful when you attempt to recover it for legitimate uses but it is useful for criminal activity.
Gone or Forever? Yes and No
You have no control over SSDs internal intelligence. It's a black box that you trust with your data. You cannot guarantee that you will be able to recover data, or that it was deleted in the first place.
How lucky are you feeling?
Why should personal use equipment be optimised for Plod?
Plod can take the lumps and go figure out their own solutions for technical challenges. there is no need to add complexity to already complex storage technologies just because MI whatever or others want to read your private files.
It is high time people stood up and said : 'Enough!'. Be it eve's dropping on conversations, geo-location on smartphones or checking electricity consumption for potential pot growers the Plod is everywhere. They don't make people remove insulation from homes so they can use their thermal imaging systems more easily, so why do we have to?
A pox on the lot.
So what they're *really* saying is that nothing was ever lost, or nothing was ever *going* to get lost, but there is enough doubt to make that "grey area" data useless in court.
I call bullshit on this one.
If you get found coming into the country with a bunch of iffy porn on your forensically at risk of 'corrosion of evidence' flash drive, I'm willing to bet YOUR particular drive will be stated to be immune to all those problems. In court.
Its a new storage media - so that means new methods will have to be found to analyse it forensically. So what?
That's what forensics is for.
Oh and the legal system will have to play catch up as well.
Imagine how it must have been when an investigative reporter discovered that people could store files on one of these new hard disc things and then delete them without having to use a shredder or fire to seemingly destroy them.
Get a grip and start reporting news, not this rubbish
arm your harddrive with thermite
and you never have to worry about a sudden raid and then finding anything...of course, it will burn through the computer, drive, table, floor, and cellar, but who cares when they don't have evidence and you can even give them your passwords.
Thanks to your use of thermite...
the cops now have ground for "terrorist conspiracy" and other funny stuff like this.
How about something simpler...
How about something related to the context...
...like destruction of evidence?
ref: arm your harddrive with thermite
Actually sometimes stuff nastier than thermite gets used.
Police in pakistan have in the past called in the bomb squad when dealing with computers they wished to recover from a suspected terrorist's house.
The large lump of plastic explosive found inside the case, kind of eliminated the suspected part of the investigation.
The device was rigged to go off if the case was lifted, and would have probably killed everybody in the room if it had gone off.
This does however raise some questions about how our police apprently just grab computers, with the assumption that a bad back and a bit of HSE paper work is the worst that could happen to them.
A bit unfortunate for plod ...
... but if it was easy then there wouldn't be such a field as 'forensics'. They'll just have to work a bit harder to come up with the methods and logical arguments to make use of the information that survives.
On the plus side, it is that very uncertainty (because the flash drive is doing its own thing) that prevents people from destroying all evidence of past activities.
One point, though : if the user can't actually erase everything, can't it be argued in court that the user could not sanitise it when it was bought and so the dodgy material was arguably already there - and they had no way of knowing or doing anything about it?
How the world works - part 7,234.
1) Researchers publish paper showing that data on flash drives may be available forever.
2) Organised crime collectively shits itself and calls its lawyers. Well-stuffed envelopes change hands.
3) Legal loophole loving lawyers commission research to prove that, while data is retrievable, it isn't usable in court due to a complex legal process known as "Technological Bullshit Using Long Words".
Next week: New research sponsored by the world's law enforcement agencies proves retrieved flash data *is* evidence.
Some time later: A group of lawyers replace the entire Forbes rich list as a marathon case arguing the toss goes into its 25th year.......
I kind of wish....
Legalities and technicalities etc.. aside:
I kind of wish that hard drives - especially of the spinning rusty disk variety were able to erased in "light bulb on // light bulb off" types of speeds....
The current system of software driven "drive cleaning" through erasure and overwriting takes ages - and it's an especially SLOW process when wanting to dispose of perfectly good drives by gifting them to others, when the drives have years of running time left in them.
Just a thought.