To be fair ...
... most if the Arts' students data remaining on the thumb drive was pretty random to begin with.
Art students – or at least those at the University of Northern Iowa – are the most likely to know how to permanently delete data from USB drives, while business or humanities students don't even try. That's one amusing conclusion of a serious study, led by the US university's Sarah Diesburg, into how people treat the security …
You should dd from /dev/full since NAND flash erases with 1s not 0s. Writing 0s adds an extra erase cycle for every block when compared with writing 1s. After writing 0s the drive is "full" from the FTL's point of view, so not only are you wearing it out faster subsequent writes will also be slower.
Used to have fun resuscitating files on magnetic media. It was also useful at times Windoze decided to fall over and die, at the time when the temp files were often readable.
Must look into how to do it on NAND flash memory, have precisely one photo accidentally erased, a happy scene outside Tokyo station.
The article is pretty crap, but the OP's comment about wear-sharing, it makes sense that a device won't really erase the blocks until there is no space where they are not LRU. Must check!
With flash mem., I think from too much of a writer-controller, not end-product level.
You should be able to bring back some deleted files from flash memory, since most filesystems just mark them as deleted. The catch is that on magnetic media you can recover them unless those specific blocks have to be overwritten, while on flash/SSD you can only recover them if fewer than some critical number of blocks anywhere on the device have been written.
On average if N percent blocks are in use and therefore 1-N perfect blocks are free, once 1-N blocks have been rewritten anywhere on the device the blocks for your file will be gone. But since the controller isn't perfectly LRU and looking to coalesce used filesystem blocks to the larger NAND blocksize in practice you'd probably need a filesystem with fairly low write activity or catch your "oops I deleted a file I didn't mean to" pretty soon after the oops!
The nice thing is that once the blocks with critical data have been erased there is no way to bring it back. You don't need to worry about multiple overwrites to 'really' erase it per DOD standard any more than you need to write multiple times to RAM to erase it. Well, as far as I know. Maybe the NSA has figured out that a NAND cell retains 6% of charge after erase from a 0 but only 2% after erasure of a 1 and has some way to gather that info....if so an extra cycle or two of rewrites can't hurt if you want to erase data that if discovered could put you in jail for the rest of your life :)
You don't need to worry about multiple overwrites to 'really' erase it per DOD standard any more than you need to write multiple times to RAM to erase it
You needed to worry about multiple overwrites when disks were in the multi-megabyte size. Pretty sure there is paper about that floating around the Internet.
Good luck getting your STM microscope out to recover my once-zero data from remanent magnetic zones.
Flash devices need more internal capacity than the number printed on the outside. They need a map of logical to physical sectors that has to be stored somewhere. They need to keep track of how often each block has been erased. Some sectors do not work on new chips and some will fail while in service so there have to be spares. Finally, the ware levelling algorithm can make better choices if it has lots of unused sectors to choose from. I have found devices where the capacity of the chips add up to 50% more than the advertised capacity.
There used to be a problem with second hand chips. Old devices were recycled leading to new devices that started with a large number of bad sectors, and those that did work had already gone through a large number of erase cycles. Under provisioning is still popular. The device will work fine if you only use a quarter or perhaps a half of the nominal capacity.
A full format that writes zeroes to all the sectors not used for filesystem metadata will identify many under provisioned drives. Some of the more cunning drives will try to identify the file system, and forget the contents of unallocated sectors to increase the pool of available blocks (or to hide under provisioning).
If 90% of your drives survive a full format then you have found a supplier who works hard to detect and demand money back for under provisioned drives before they reach customers (or you picked a file system type supported by some excellent firmware).
Using dd like 1980s_coder is close to a good answer. Drive firmware is likely to avoid storing duplicate data, so half a sector might store all the zeroes, and a few more would map lots of logical sectors to that compressed sector. For a while, some of your illegal porn and bombing plans will be stored on blocks scheduled for erasure, and the firmware will get around to that in due course.
I would love to use the trim command. The latest versions of the SATA, SDHC and USB command sets all include trim or an equivalent. SATA support is common and it even works on some devices (modern Linux kernels have blacklists and whitelists). A few USB devices claim to support trim. I have yet to come across a USB enclosure that forwards trim commands to a drive.
I like to write a sequence of random numbers to a new drive, and try to read them back. That spots under provisioning. Two or three complete drive writes of random numbers will probably erase my terrorist plans. One day deleting a file could result in trim commands that are promptly and reliably obeyed, but for the next decade, the only secure erasure strategy I have real confidence in is fire.
"my experience is that if you fully format a pen drive you have something like a 10% chance of permanently corrupting it, making it unreadable".
*My* experience tells me you should get better quality UFDs. Or at least stop using the promotional ones you get for free. Many of those are laughably *under*-provisioned: a 16GB UFD actually containing a 512MB flash chip...? Yeah, I can see how well that will work! :-D
No UFD -- indeed, no non-ROM storage medium of any kind or capacity -- enters my service without at least a couple of full drive write/reads with integrity checking, if I can help it. Very few fail this. But at least those that do take no data with them.
You could effectively "TRIM" them yourself if you identify the no longer used blocks and write all 1s to them. Unfortunately since you generally don't know the internal blocksize of the flash you wouldn't know how much to write. Writing 1s to a less than blocksized block would be worse than doing nothing.
my experience is that if you fully format a pen drive you have something like a 10% chance of permanently corrupting it, making it unreadable
In which case you have identified a crappy unreliable pen drive before storing anything you wanted to keep on it, saving yourself much grief later. Buy a different make next time. And remember Sturgeon's law.
I used to fill my criminal drives with MP3s after formatting them as I had more songs than disk-space. Then I started I started getting raided on bogus terrorism excuses and I built a forge, better than a hammer.
There's a really good, if irrelevant, NS article just online, Memory recall works twice as fast as the blink of an eye
When I was a four year old I used to test how fast I could think by throwing my self off a small flight of steps and trying to think something before I landed. I never could think anything mid-air except, "Think something" which didn't count as I'd already been thinking that. I concluded I was a slow-thinker, and as I grew older others certainly were more 'quick-witted'. They tend to get in a lot more trouble earlier on though, it's a common-difference in brain function that leaves them open to impulsive short-termism and leaves me more open to brain-freezing in emergency situations.
Computer magazines and websites have speed-tests for machine components, processors and systems, I hope someone develops something like that for humans. There are seemingly four stages to human memory, remembering it, recalling it and I forget the other two. Not my field of study. Still, I'm in a court case just now that mostly relates to events from decades ago, and I seem to be the only person who remembers anything, and I remember those past events too well if anything. Being able to forget, to wipe memory, must be as much of a blessing. I wish there was a Darik's Boot And Nuke for the mind, like Eternal Sunshine, but everyone seems intent on memory augmentation implants.
For dlash drives, SD cards, etc : use a two-step process.
1) Smash with hammer
2) use hot soldering iron on remaining exposed chips.
I've tried dissasembling hard drives and it's a pain to remove the screws. Once you have the lid off though, then neodymium magnets are your friend.
Arts – nobody left undeleted data on the drives, 44.4 per cent had run a "quick" format, and 33.3 per cent had run a full format.
I think this is far too generous to arts students. The reality is that none of these students had stored anything on the drives in the first place. 33.3% of them had never even plugged their drive into a computer before.
A while back for grins and giggles I bought a couple of self-encrypted drives. Ones with keypads on them, so if I needed, I could use them with something other than Windows. The drives are slow. Sure, the manufacturers claim that the new models are faster, but there's a price to be paid for your data being encrypted before it gets written to the drive.
A problem with the Windows drives is that each of them wants to load a utility into Windows to access the data on the drive. So your Windows machine is going to wind up with a zillion utilities in it for all of those drives. And then what happens if your OS goes titsup? Bye bye bytes!
Currently both Windows and Linux have encryption for removable file systems. The user just has to be aware of them, and put them to use.
There needs to be a proper standard (ISO etc) that defines removable drive encryption.
Make it modular and extendible, i.e. ability to add/update/depreciate algorithms etc.
Add that to the OS, Win/OSX/Linux/iOS/Android etc. Probably as a 3rd party install initially, then as a core OS component later.
Then you'd be able to plug the drive into any machine, and as long as you have the correct credentials/key, you can access the drive.
Perhaps have a read-only option as well i.e. one key/credentials gives full read/write access, another key/credentials gives only read access.
Definitely agree. We use FIPS 140-2 USB memory sticks at work - pricey enough, but I had to order a few even more expensive ones that dual-booted to both Mac and Windows for staff with Apple kit. The software is a bit better (they've got an admin password, for example, whereas the cheaper ones don't). However, when someone turned up with a Chromebook... yeah, having a standard would help a lot.
When you say "Amazon" do you mean "Dispatched from and sold by Amazon EU S.a.r.L. ", or some random third party trader selling through Amazon Marketplace? There really is a massive difference!
Regardless, if you trusted the only copy of irreplaceable and precious media files to a single storage device (with no backup) you're crazy! Even genuine drives fail.
Learned this the hard way when I bought "genuine original" ink blocks for my Xerox from a marketplace seller once, only to find out when I opened them that they were nothing of the sort, and not even the same colour as the originals. Said seller refused to refund for the open pack and Amazon were less help than might be hoped.
Regarding memory sticks and the like, the thing that annoys me is that nominal sizes vary not only between manufacturers but between batches from the same manufacturer - I run a fleet of 20-odd Raspberry Pis at work as video players and use dd (or similar) to "clone" the SD cards. The first batch of Transcend cards were slightly bigger than the second batch and both were bigger than the Sandisk cards. The Official "Samsung" Pi-logo cards were somewhere in between, but the unbranded Pi logo cards I've just bought from RS are a bit smaller.
All this means that I have to identify the smallest card to make my "original" for copying, a bit of a pain if I buy new cards that are smaller.
The first batch of Transcend cards were slightly bigger than the second batch and both were bigger than the Sandisk cards
I ran into this exact problem when I tried to write back a Pi SD image to a new card after the previous (Transcend I think) one went bang and ran out of space despite both being "8Gb"...
"All this means that I have to identify the smallest card to make my "original" for copying, a bit of a pain if I buy new cards that are smaller."
I hit this same problem with some supposedly identical model number replacement HP SCSI disks many years ago. That was when I learned to provision a RAID array a few % smaller than the HDD capacity.
Since my Pi images never contain "real" data anywhere near 8GB, next time I need to do this I'm going to use dd - which will write until it fails unlike some other utilities which refuse to write at all - and then fsck (GParted) to "fix" the partition size. Might work, you never know!
next time I need to do this I'm going to use dd - which will write until it fails unlike some other utilities which refuse to write at all - and then fsck (GParted) to "fix" the partition size
It didn't work for me, resulting truncated image wouldn't boot, but managed to faff about mounting the image via loopback and shrinking the filesystem down slightly so the resultant new image was small enough.
Wasn't any actual data on it, but would have taken longer to get the boot environment configured back how it was supposed to be than it took to mess about with the filesystem. At least I had an image of the SD card from a few months before it failed to go back to.
If you could point out a camera that has multiple card slots used in RAID mode, I will go out and buy a dozen.
Sold by X but fulfilled by Amazon; and someone who has been selling for a good number of years; it was not my first card from him, so I suspect he didnt know some fakes had slipped through.
I sympathise about "sold by x, fulfilled by Amazon" as I use this facility myself. Sometimes it's just a rubbish product though - I have had some genuine USB sticks that have "just failed". Integral brand - not going to buy them again.
If you could point out a camera that has multiple card slots used in RAID mode, I will go out and buy a dozen.
There seem to be several with dual slots, and although none have "RAID", I suspect that a simple backup could be the answer you need. For example, the Nikon D7100.
To quote from the manual (p69):
"When two memory cards are inserted in the camera, [...there are several options, including...] Backup (each picture is recorded twice, once to the card in Slot 1 and again to the card in Slot 2),"
I know that there are Canon and Pentax models with dual slots, probably others too. Dual slots started with video cameras I think, though it was often a twin hard drive and often used simply to increase continuous recording time, along with dual battery slots.
My Canon 5D Mark 3 has CF and SD card slots. A dozen of those might cost you a few quid though and that money would be better spend getting memory from a reputable seller instead. It's not the only Canon to offer this and other manufacturers offer it too. As for operating in RAID - I suppose it's RAID 1 since the card slots operate in parallel. I doubt you'll find RAID 5 in a camera unless you have it routed to a nearby NAS box via Wifi...
If a camera of that calibre is outside your budget, then test the card before use. Reading your original comment it sounds like you added media over several days to the same card before realising it had failed. Surely you would have checked the stuff you'd captured at least once a day if only a quick browse? At least then you'd have spotted the failure sooner. Also the camera ought to have detected a problem with the card. One of my CF cards failed once, taking with it 5GB of RAW files, but the camera at least told me there was a problem and I was able to swap cards.
Fulfilled by Amazon means nothing regarding how reliable/genuine the item is. They're acting only as the warehouse/logistics agent - the Marketplace seller is responsible for the product itself. Fakes may slip through - but this is less likely with Amazon supplied items and in that case you'd at least have comeback against Amazon themselves.
The only reliable ways to ensure no data can be retrieved from a flash memory device are:
1) only ever put well-encrypted data on the device. "well encrypted" means at least AES-128 and at least a sixteen-character pass phrase, the longer the better.
2) completely destroy the device. Only two effective ways: use a hammer or equivalent to reduce the chip(s) to grain of sand size; or chuck it in a really, really hot fire - think crematorium or scrap metal man's furnace
This was official advice from an agency whose job it is to know
Serious question: does microwaving USB sticks work? (For destroying them, not for erasing, obviously.)
On a non-related note: don't knock art students. Personally I think that everyone that applies to any art college should be accepted. That way we keep getting interesting pop bands and have the chance to avoid the occasional genocidial dictator.
I wonder if... that old set up involving a wound copper kettle element, an open frame lab-grade ferrite core transformer, two coils and a long piece of flex would do the job. You know... the set up that the physics teacher used to demonstrate electro magnetism with by turning dissection pins into short-lived arc lamps.
Hmmm people put sensitive data on tiny, easy to lose things that you can plug in any system? *shudder*
I'll stick to having stuff stored on my own server, on my own property that I can access anywhere via ssh and other secure methods and if need be can copy to a different location.
My uses for USB flash drives:
- random bootable linux distro 1
- random bootable linux distro 2
- random bootable linux distro 3
Installers/etc... for stuff I might need to be using in an enviroment without good online connection,
err... random music/vids/etc....
I was retiring good old fashioned hard disks the other day. I was very disappointed to find that when you peel off the label that covers the "inspection hatch" while they were running there was no satisfying noise of total destruction, just a lot of wind noise, and the buggers just kept on working fine till I stuffed something into the hole.
And there was me hoping for the noise of a good old fashioned head crash.
And there was me hoping for the noise of a good old fashioned head crash.
Please explain in 100 words or less why exactly you expected this to happen.
Coz manufactures go on and on about how the disk heads need perfect conditions to keep flying just above the surface of the disk and why we used to have to go to such care back when hard disk packs used to be removable and all the bollocks we were fed when removable packs were phased out in favour of sealed for life units.
De-duplication and hidden block management could always bite you.
Attempting to physically destroy the device will just make people try harder to find what you are hiding.
An effective approach is to fill the drive with legal but embarrassing porn, and do a sloppy job of deleting it. The pseudo-random nature of video will avoid block de-duplication, very little of the previous contents will remain on the hidden blocks, and anyone forensically examining the drive will stop looking after they "find" the deleted porn.
Sometimes letting people "win" is the best way to succeed.
Biting the hand that feeds IT © 1998–2019