[<<]Message[>>] [<<]Author[>>] [<<]Subject[>>] [<<]Thread[>>]
Number : 3481 Date : 2003-01-08 Author : Yuki Taga Subject : Re: EXTREME alarm Size(KB) : 19
Hi Kan, Wednesday, January 8, 2003, 3:13:26 PM, you wrote: KY> Yuku: Yuki ^_- KY> First of all, let me say that it is our worst fear come true to KY> hear that a user loses a lot of files using XXCOPY. Well, I appreciate that. Of course, what I lost were backups, not original data. The damage ended up being one of time, not data. I did lose some 'incremental' backups, but they are of no consequence, really. We need to get to the bottom of this, however, because it could have been real data, and almost surely *will* be read data for someone. There is a terrible problem, someplace. KY> We put a very high priority in the design to prevent any KY> accidental loss of files. If the users are annoyed by the KY> exceedingly annoying warning prompts everywhere, I have no KY> apology for that. At least, until the user learns the trick of KY> adding the proper switch to suppress the warning (hopefully after KY> reading and understanding what the warning is trying to say), the KY> user will be protected from an instantaneous loss of files. That KY> is why we discourage people from using /YY. I understand this. And I think it's good policy, except that when a command has been written to a batch file, and carefully tested, I think it's fine to use the /YY switch. Would you not agree? That was the case here. There was no "typo", the command was run from a batch file (.cmd), and the file had been unmodified for many months. The command had *never* produced even the slightest hint of a problem before. KY> Before giving you a concrete answer, I was trying to figure out KY> what could have gone wrong with your command. I have gone KY> through a lot of scenarios. (This delay of my response was a KY> result of trying to give you something informative and trying to KY> avoid speculation as much as possible). I appreciate that, and I appreciate your thoroughness. I'm not going anywhere, and I'm in no hurry. ^^_^^ KY> One thing I did not hear (which is always very vital in KY> troubleshooting) is which Windows version that you are running. KY> It is nearly as important as the XXCOPY version. That is, a very KY> substantial portion of our problems are results of very subtle KY> differences in behavior between the Windows OS. The big divide KY> is between the 9X/ME family and the NT/2K/XP family. Even though KY> Microsoft calls them "Win32", the two environments are quite KY> different in small but significant ways. Earlier today, Melissa KY> reported some funny error condition with the missing server (I KY> will address it in another message). That is also related to KY> Windows version (in Melissa's case, I believe it is Win9X's bug). KY> So, please tell us the Windows version in every problem report. You are right. I neglected to mention this. I run a dual boot box. The system partition (c:) is XP. The other system is Windows 2000 Pro, which has its boot partition on drive L. So the two systems here are in the same basic family (NT derivatives). Personally, I'm sure this was not an OS fault, and as you read on, I think you will agree. KY> OK, now, based on my preliminary check, I still cannot come up KY> with a plausible scenario for how you lost everything in the T: KY> volume. From my understanding of how things work with XXCOPY, KY> the section in the T: drive which will be wiped out will be KY> confined to the directory that you specified: "T:\Daily KY> Essential\Equis72\". I cannot figure it out, either Kan. But I can tell you that after initially losing the contents of t:, I continued to experiment, and every time I ran the command, a command I had been running countless times in the past few months, it wiped out anything on t:. Every time. Now, the key point that I was explaining to Dan Anderson, that you might not have read yet: 1) Yesterday, I did what I thought was a safe upgrade. I went from 2.82.7 to the 2.83.4. I did not expect the behavior to be any different with this simple batch file between the two versions of xxcopy, but I was wrong. Finally deciding I *must* be xxcopy, and it must be the upgrade, I pulled all of xxcopy, and reinstalled the 2.82.7 version. Immediately, I had no problem, and the command ran as expected, with safe and sound output. So it was version 2.83.4, and that is where you need to look. I will take a moment here to digress, and mention something I have always felt was amiss with the xxcopy install.bat routine. You recommend dumping all the files in the Windows system folder, and that comes up default on the install, with your recommendation. I have *never* felt comfortable about this, and I don't allow it. I have a discreet directory for xxcopy, and that is where it lives. This makes uninstalls, or rollbacks, very easy. I know I got *everything*, whereas I cannot be sure with all the stuff in the system directory. I suggest you change this default recommendation, for this reason. I am sure you do it because the directory needs to be in the PATH, but I am also sure that 99.99 percent of the people who use xxcopy know how to add its home to their PATH statements. ^_- But I digress. ^_^ Let's get on with the other comments. KY> Let me start with the source (as you will see, this matter is not KY> very significant to the subject of how you lost your data). KY> Your command line had the "built-in" ambiguity as to the source KY> specifier. KY> Source: m:\2000ap~1\equis72 KY> Destination: "t:\Daily Essential\equis72" KY> Here, the source can be interpreted in two ways. This KY> not-so-optimal characteristics of XXCOPY command line KY> is a direct result of making XXCOPY fully compatible KY> with Microsoft's XCOPY. That is, your source specifier KY> is ambiguous and I cannot tell which of the following KY> two ways to interpret it (nor can XXCOPY until it examines KY> the directory at m:\2000ap~1\). If, there is a directory KY> named "equis72" in m:\2000ap~1\, then, you are trying KY> to copy everything inside the m:\2000ap~1\equis72\ directory KY> (including all subdirectories) to the destination. Correct. I was trying to copy the directory m:\2000ap~1\equis72 and EVERYTHING in it to the destination. Something I've done probably 1,000 times without a problem. KY> The second possible interpretation of your command line is when KY> there is no such directory as m:\2000ap~1\equis72\. In that case, KY> XXCOPY will interpret (just as XCOPY would) as the following: KY> Source base directory: m:\2000ap~1\ KY> Lastname pattern: equis72 KY> In this scenario, XXCOPY will try to copy a file named "equis72" KY> in every subdirectory under the m:\2000ap~1\ directory. KY> You should avoid such an ambiguous command line if you mean KY> the source specifier is the name of the directory by adding KY> the trailing backslash. Kan, I believe you are saying above that the *only* way this can be ambiguous is if the complete source, as specified, does not exist. Am I correct? In any case, that was not a problem here. The complete source existed, and still exists. Simply rolling back to the old version of xxcopy allowed the batch to run perfectly. KY> But, the source directory ambiguity in this case makes no KY> difference in how a possible damage could be done to the files in KY> the system. That is because XXCOPY does not alter the files in KY> the source directory unless you have the /RS or /RC switches. KY> The damage is nearly always at the destination side. So, let us KY> look at the destination specifier. Yes. To be clear, there was no source damage. The damage was only at the destination. KY> "T:\daily Essential\equis72" KY> In the destination side, XXCOPY does not allow any filename KY> pattern (nor any wildcard character) which completely eliminates KY> any possible ambiguity on the destination (we pay modest price on KY> this by not being able to do a rename-while-copy operation with KY> XXCOPY --- but I strongly feel that this decision is a correct KY> one). In the destination case, the trailing backslash (unlike KY> the source specifier counterpart) makes a relatively small KY> difference. If the trailing backslash is present, it is KY> equivalent to having the /I switch (create the destination KY> directory unconditionally if not present already). Since the KY> command used here did not have the trailing backslash in the KY> destination specifier, it would prompt the user by "Create new KY> directory XXXX (Yes/No) ?". In this case, the /YY switch KY> suppressed this as if /I were present. But, in either case, the KY> difference is minor. KY> I'm not in the position to know whether the destination directory KY> had been present before the doomed XXCOPY operation began. But, KY> here's the heart of my analysis. It was present, I can assure you. This is a daily essential backup to a USB drive. Takes only a few minutes, and is done *every* day. This is the first problem I've ever had with it. The destination directory was there, for sure. KY> So, let us study the two possible scenarios: KY> 1. If the "T:\Daily Essential\equis72\" directory existed: KY> Depending on what was in the source directory, some or KY> nearly *ALL* of the contents in the destination directory KY> could be "wiped out". That is by design and that is what KY> /CLONE operation is all about. Nothing to panic except if KY> you did not have a valid directory in the source as KY> expected, you may see XXCOPY do its job correctly by KY> removing all files and directories from the destination KY> directory. But, that's nothing to complain here. KY> 2. If "T:\Daily Essential\equis72\" directory did *NOT* exist. KY> In this case, xxcopy will not perform any delete operation. KY> Since there was nothing in the destination, there was KY> nothing to delete from the destination, nor there was any KY> file to be overwritten. Therefore, in this case, there KY> were absolutely no files to lose. KY> ---------- KY> By this analysis, the worst case scenario is that you may KY> lose some or *ALL* of the files that were already present KY> in the T:\Daily Essential\equis72\ directory. But, under KY> no circumstances, XXCOPY will touch anything outside of KY> the destination directory that you have specified (this KY> fact is based on the assumption that there is no serious KY> bug with respect to the destination --- we are not aware KY> of any bug of this sort). KY> In short, the damage should have been confined to inside KY> the destination directory. If you had misspelled the KY> destination directory (and the wrong directory did not KY> exist), then, no harm would be done (this would be KY> similar to Case 2 above). I understand this, Kan. And I agree with you about what *should* have happened, and what we would expect to have happened based on these switches and how they work. However, I can tell you that the drive was wiped. I had about 30 GB of partition images there, outside the "daily essential\" folder, and they were gone, as was every other folder or piece of data outside that folder. What was there, was "daily essential\equis72\the entire root structure of m:\2000ap~1. Actually, the entire structure was not there, because I stopped the batch from completing when I realized there was a problem. I tried this several times before rolling back xxcopy versions, and the behavior was always the same. KY> So far, we have no confirmed behavior ever that the KY> /Z function ever harmed directory contents other than KY> the specified destination directory. I was running KY> quite a few tests today on both WinXP and Win98SE KY> environments and I have not observed any XXCOPY KY> behavior that would alter contents outside the KY> destination directory. KY> This is the end of my analysis on the XXCOPY itself. KY> ========================================================= KY> Let me add my thoughts on other issues that are KY> outside XXCOPY's activities. KY> I'm not sure how to interpret your report. First of all, KY> what I don't understand is what procedure you took to KY> "slammed it shut". The proper way to terminate an ongoing KY> XXCOPY operation is to hit Ctrl-Break. This is the best KY> way to end a session because XXCOPY will gracefully KY> terminate the action immediately after the handling of KY> the current file is completed. Under no circumstances, KY> XXCOPY will abort a file I/O operation in the middle and KY> keep an open file open. Alternatively, you may click KY> the Window-Close button (the top-right corner [X]), and you KY> click [YES] to the confirming dialog. The second method KY> is much inferior to the first (Ctrl-Break) method, because KY> you are forcing an active program to be aborted. I was not very graceful about this, I admit. We ladies sometimes panic, and when I realized there was a big problem, I hit the panic button. I clicked the close box on the DOS window and terminated. I did read to the bottom, and I apologize for not being more clear about this. I didn't yank the cable or throw the power switch. ^_^ I'm not THAT bad. ^^_^^ Well, I guess I've said about all I can say regarding this, Kan. Essentially, what we have here is a batch file that has worked for months upon months. Never a problem. Big problem after the install of the new xxcopy version yesterday. Problem disappeared like magic after the rollback. I don't know what else to say. Best, Yuki KY> In some cases, such an action will leave unclosed file KY> in the volume that may be very resilient to an attempt KY> to remove --- I once had to reformat the volume where KY> I had a huge allocated (corrupted file) space in order KY> to reclaim the space. When an application program is KY> abruptly terminated by a "brute force", then, the resulting KY> mess (open files, etc.) should still be gracefully KY> dealt with by the operating system (But, we have seen KY> not so graceful aftermath). The second method still allows KY> the device driver the chance to do a few closing procedures KY> of pending operation on the drive. KY> If you did not follow any of such preferred method, from KY> your terse description of "slammed it shut", I imagine you KY> either yanked the USB cable that connects the external KY> drive unit to your computer, or you removed the power from KY> the USB unit. My guess is that if you remove the USB KY> cable connection in the middle of a disk access KY> (as much as half such accesses may be for write access KY> in this case), I imagine the host system will immediately KY> notify XXCOPY that a fatal write error has occurred KY> and XXCOPY will report the result accordingly. Since KY> the external drive has its own USB interface and so on, KY> I imagine that even if the physical connection is lost KY> in the middle, the on-board microprocessor should perform KY> a sensible thing and close the file with no serious KY> harm to existing volume structure. (Note: since I KY> have never designed an USB-hard disk nor do I own one, KY> this is my educated guess). KY> If, you removed the power from the external unit (This KY> is the worst thing you can do to your drive), then, KY> as far as XXCOPY is concerned, the result is nearly KY> identical to the case of USB-cable removal. But, the KY> data on the external volume (T:) may be corrupted. KY> I imagine it would be very similar to removing power KY> from a running hard disk when it is recording something. KY> My guess is that even if the drive is busy writing KY> to the media at the time of power removal, the loss KY> of data on the entire volume is a very rare event. KY> That is because the data on a disk volume is scattered KY> to everywhere. There is a FAT section (and its KY> backup copy), then, the root directory, and the rest KY> of the sub directories and so on. Since the disk KY> can write only one place at a time, when the power KY> failure to the unit occurs, it must be at only one KY> place. And, I imagine 99% of the cases, you will KY> have "functioning" disk volume with a possible KY> corrupted data structure --- that is what the scandisk KY> utility routinely handles. It is hard to imagine that KY> the disk with a losing power can damage the MBR or the KY> boot sector or both of the FAT areas (or whatever KY> applicable to the USB-disk case). In short, I estimate KY> that it should be extremely rare to lose everything in KY> one stroke. KY> I wish you had used a lot more precise wording in KY> describing your catastrophic experience. The KY> action you took was not well explained, and the KY> result of the damage is also so vague I cannot make KY> much guess work based on your description. If the KY> external hard disk behaved as if it were a completely KY> new and clean (freshly formatted) condition, it would KY> be very odd, in light of what you did to it (and I KY> don't even know what you exactly did to your drive). KY> Since you must have "aborted" in the middle (probably KY> at the beginning), the remaining volume structure KY> is probably half destroyed --- but probably left KY> partially erased ???? I really need more description KY> to reconstruct what has happened. KY> Please read this very carefully and give us a KY> detailed report on exactly what happened (what you KY> did) and what is the appearance of your extranal KY> hard disk. How many directories are there in the KY> root directory? KY> I apologize for this rather long reply. KY> But, since we take the matter like this very seriously, KY> I wanted to cover all possibilities. That is, if KY> some important files on the drive are still there KY> and can be retrieved, we should try that first. KY> But, more importantly, we would like to pinpoint the KY> cause of this event so that we can prevent the same KY> thing from happening. I believe this is my best KY> answer that I can give you based on your reports. KY> If you have lost *ALL* and have nothing more to KY> lose, that is even better for the sake of our KY> analysis. You may try to reproduce the incident KY> and give us all the detailed observation of what KY> has happened. KY> I appreciate if you make a more descriptive report. KY> Kan Yabumoto KY> ====================================================== KY> At 2003-01-07 05:03, you wrote: >>Hi, >> >>I installed 2.83.4 over the previous recent version today. >> >>Then I ran a normal backup. The first command on the set: >> >>ECHO OFF >> >>xxcopy m:\2000ap~1\equis72 "t:\Daily Essential\equis72" /CLONE/YY /WV0 >> >>I knew something was wrong pretty quickly, as the external USB drive >>(t:) started writing like crazy, but the DOS window in XP was just >>sitting there not giving me any output. >> >>So I slammed it shut as fast as I could. >> >>Too late. Drive t: was wiped. 120GB drive, and xxcopy was busy >>cloning my entire m: drive to it. This is NOT the command, and no >>other command was run except what you see above. >> >>This has NEVER happened before. Obviously, if it happens again, >>xxcopy and I are through. I used to think this was a fantastic >>program. I am not so sure now. I am in a terrible position now, >>needing to redo a tremendous number of partition images, some with >>incremental backups that CANNOT be duplicated. >> >>Any ideas why xxcopy went xxcrazy? >> >>Best, >> >>Yuku KY> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ Best, Yuki mailto:yukitaga@t...
This message if part of XXCOPY's message Archive. The archive contains all the messages posted at Yahoo!Groups: XXCOPY.