Quick and dirty file-clustering-for-idiots DFSR was one of Server 2003 R2’s hidden gems. It is more robust in Server 2008 R2, but the fundamentals haven’t changed. You mostly use it for collecting data from shares on branch sites to a centralized site for backups. Alternately, it is a method of providing centrally published information to branches. That’s what the …
Obvious problem, if the server with the RAID array has gone unexpectedly titsup in full cry, there's a real likelyhood of the RAID set being compromised. A variety of failures could cause this, not least of which is the RAID card itself deciding to have a bad day.
Presumably the replication side works on a "known good" basis so any partially written or corrupted cack from server A that just went pear-shaped never makes it into the copy on server B.
Why aren't you just using shares in the DFS namespace (e.g. \\domain.com\share as opposed to \\fileserver\share)?
When you publish shares through the namespace this failover is handled automatically when one server falls offline. No need to mess around with changing hostnames, which also leaves you in a tricky position for recovering the failed server.
I'm using DSFR with 2008 R2 as a "resilient" file share for web servers. There have been times where replication stops working and we had to reboot the file servers to start it up again. Other times it's taken a few minutes to replicate files (only 1MB) for some weird reason. But 99.9% of the time it's great.
When we've simulated a dead file server in the DSF group it takes a minute or two for DSF to realise that and to start fulfilling requests from one of the other file servers. I don't have to mess around for 15 minutes like the author of this article does, it's quite automatic but obviously not as fast as I'd like it to be, but having two SANs is a lot more money.
DFSR = Distributed File System Replication. A Microsoft tool to quite literally replicate files from point A to point B. It can even replicate in a mesh.
http://msdn.microsoft.com/en-us/library/bb540025(VS.85).aspx
DFS = Distributed File System. Essentially the ability to post multiple shares from multiple computers under the same namespace. (Used for resilience and aggregation.)
http://technet.microsoft.com/en-us/library/cc787066(WS.10).aspx
DRBD = A neat Linux tool for block-level replication of volumes.
http://www.drbd.org/
Anything else I can help you with? I do take the time to read and respond to comments here for exactly such reasons. My editor won’t let me write really long articles, so a lot of information has to be added in the form of comments, rather than the article itself. Left to my own devices, I write articles around 1500-1750 words long. He keeps trying to get me to hand them in closer to 500-750 words. Some things get missed, or are purposefully omitted. For this I sincerely apologise.
Using DFS-R to replication between datacentres is a reasonable solution to simply acheive some resilience.
However it has a number of limitations which can cause issues:
No file locking - to me this is the major issue. I know you don't have the shares active in the secondary location, but if you need automated failover using multiple DFS targets, this is a big issue in terms of data integrity, probably the no.1 most important thing with any file sharing system.
Scalability - you are replicating 10TB. It is nice to know you are not having issues with replicating that much data, but this is well beyond what Microsoft have tested which was 1TB (see http://blogs.technet.com/b/filecab/archive/2006/02/09/419181.aspx for an explanation). Sure in theory it will work, but the jet database can grow very large and if you have an issue you could have a huge resync occur, especially since you only have 1 replication group.
I am also pretty concerned abot something you mentioned:
All of the shares are a subfolder of E:\Local Shares
If I understand correctly you have 10TB in a single volume? All I can say is god help you if you have an NTFS corruption, as your disk check will take several weeks with a 10TB volume. You could be using mount points I suppose, but you didn't mention that and it wouldn't make much sense to run everything under E: if you don't need to.
As others mentioned, your failover method is overly complicated - it would have been much better to use a DFS namespace with 2 targets - one pointing to your primary server and one to your secondary. You can leave the secondary disabled to get around the file locking issue, and then if you want to failover you simply enable one target and disable the other. 4 clicks instead of messing around with DNS and renaming servers, forcing replication etc.
The next article deals with the limitations of DFSR, NTFS, etc. Suffice it to say: no. I don't have it all on one partition. A single RAID array, (due to cost limitations,) but not one file system. DFSR won't handle 60M files in a single replication group; not even close.
For many people, simply replicating everything from a single volume would be fine; they don't have 60M files and 10TB of data. For myself, I have a folder under E:\ called “Shares.” On smaller sites, where I only have 4M or less files to worry about, this is a single volume, single replication group. They are usually less than about 2.5TB and work just fine.
On the main site, where there are 60M files and around 10TB to replicate, each of these shares is a replication group of its very own. On this site, each of these shares is actually a separate partition mounted into the “shares” folder. A little overly complex perhaps…but easy enough to administer as a single entity. (The backup crawlers for example simply see the whole lot as descendants of “E:\”)
As to the Jet database, it’s not the only issue. Remember that the DB behaves like an absolute turd if it can’t stuff itself entirely into RAM; additionally Bad Things Can Happen if DFSR is in the middle of replication, the Jet DB *has* managed to put itself entirely into RAM and the power goes out. You are correct as regards the file locking issue: it’s a problem.
The above are the reasons I view DFSR as “hobo high availability.” The /proper/ way to HA for file storage is to use two SANs that replicated at a block level. No questions about that. Not everyone has that luxury; however SANs are rapidly dropping in price. For the moment, buying 10 NASes to cover my 5 logical sites is still nearly an order of magnitude more expensive than my current “hobo high availability” solution.
If it makes you less concerned, my partner in crime is a strictly by-the-book systems administrator. His job is largely to restrain me. I MacGyver a solution to whatever problem is a burning fire on my desk at that particular moment. Where something might take hours, days or weeks to get operational or repaired “by the book,” my job is to make that happen in minutes. My fellow sysadmin then frown cannons me, and sets about poking holes in all of my theories and experiments.
We poke away at them and eventually arrive at workable, reproducible configurations. They are rarely best practice because we simply don’t have the resources to set up our network that way. They work though; and they work reliably. Tested through various failures both in the lab and in practice. DFS namespace for example is one of those that didn’t. (See post further down the thread on that.)
It’s why I write what I write; theory versus practice. In theory there is one way to do everything, all the time, and it works for everyone. In practice…I find that real world requirements and resource availability are different almost every single time.
Hi Trevor, thanks for the response.
I completely undertstand where you coming from, I used to be a admin constantly fire fighting and installing networks with little budget, and DFS is indeed a great solution in some cases - it is bundled with the OS, simple to install and setup, and newer versions offer some nice efficieciences such as cross-file Remote Differential Compression (with Enterprise) and bit level differencing. I used it for several customers and generally it can be left alone shipping files back to the main office or whatever you are using it for. The main issue is file locking and the time it takes to replicate - you never quite know how up to date it is, and sometimes it seems to get stuck on a small file for a long time. If you do have a disaster you know you have most of the data, but probably not all of it.
I am currenly building a 12 node dual site 2008 R2 file cluster which has taken about 6 months so far thorugh various test environments, so your solution is certainly a lot quicker and cheaper. We are replicating about 40TB at SAN level using HP EVA synchronous replication, so if a file is written the operation does not complete until written at the other site - you therefore know that your data is always up to date. It takes about 30 seconds for a resource group to failover and popup in the other datacentre, the HP cluster extension software handles the read/write status of the LUNs and makes the whole process automated. There about 50 pairs of disks on the cluster, so mount points become pretty essential - I can see it could be easier for you when setting up backups etc, assuming the software supports following mount points.
The points is that there are different requirements for different clients - mine is automated, highly available and expensive (although being a Windows solution is a lot cheaper than e.g. Netapp), yours has a manual failover process, is moderately available and cheap - as you say no solution is ever one size fits all.
I actually asked the Microsoft DFS product manager about replicating 30TB from one cluster (you can now replicated from cluster resource groups in 2008 R2) and there was a worrying silence, so it is good to hear from someone who is replicating a decent amount of data with no major issues.
Cheers.
My "rules of thumb" for dealing with NTFS and DFSR are laid out here: http://www.theregister.co.uk/2010/09/28/sysadmin_ntsf_file_limits/
For all intents and purposes, if you follow those guidelines you can replicate any amount of data you want. Basically, cut the replication groups into smaller chunks and use Muchos Big Time Noodles*.
*Noodles = RAM. How? Plural of box = Boxen. Plural of RAM = Ramen. Ramen noodles. RAM = noodles. I think this is why they say my mates and I speak our own (nonsensical to outsiders) language...
I don't use the DFS namespace because certain Linux boxes don't play well with it. Since I host some of the files my Linux web servers host from my Windows file servers, it's easier by far simply to rename the system.
I should also add in that I have found things go really sideways about 20% of the time when adding a repaired server back into the DFSR pseudo-cluster. The only real way to get the systems back in sync has been to delete the replication group, delete the copy of the files on the destination server and rebuild from scratch. Now normally you could make a lot of this go away by pre-staging your files…but even pre-staging your files has some significant holes.
What it boils down to is that if you absolutely require your users to be using the “always most up to date fileset” but still require a constant backup, then using DFSR without the DFS namespace is a quick and dirty way to get the job done. That way your backup set might be a little out of sync, (DFSR isn’t instantaneous,) but you will only every be using it in case of emergency.
It should be pointed out though that DFS Namespaces in conjunction with DFSR can be really, really cool. They can solve a few problems, and provide a great way to centrally distribute certain kinds of information. For the curious, I do use DFS namespaces for some shares. A great example is my “Backups” share. Here lies all the RARs, BKFs, VHDs etc. from my automated backup systems. This replicates these backups to file servers in all sites. Cheap and easy off-site backup. (Well, cheap and easy because we have uncapped internets…)
My primary uses for DFSR however aren’t the traditional “whitepaper uses”. Almost nothing I do is a whitepaper use. I specialise making the best of the very limited resources I have, under the weird pro-Microsoft-but-no-real-budget constraints my CTO makes me live in. In this case DFSR isn’t being used to provide distributed access to files. It’s being used for what I call “hobo high availability;” a poor man’s substitute for a SAN that does true block-level mirroring. In this role, it works well.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
