[<<]Message[>>] [<<]Author[>>] Subject[>>] [<<]Thread
Number : 4466 Date : 2003-04-28 Author : Kan Yabumoto Subject : Clarification on the /CL switch Size(KB) : 6
When Garry suggested the use of a /CL switch as a "Hail Mary" technique to profacaurus's problem in the excessively slow file transfer case. ------------------------------------------------------------- Sorry for an American Idiom here. In the game of football, (no, not Soccer, the American Football), Hail Mary is a low-success-rate touch-down pass in a "nothing-to-lose" situation --- usually in the last seconds of a losing game. ------------------------------------------------------------- I was very skeptical on the idea of its use because the obscure /CL switch was added about two years ago (almost as an experiment to a rare situation that had something to do with performance) for a customer but the effect of its use was never really confirmed. I was not sure about its effect. But, by default, the switch would become in effect only when the file size is 1 GB or larger. It has been basically a forgotten feature --- I was almost ready to throw this obscure feature away. Since it was somehow documented and existed, it was left alone. Now, Garry dug it up and miraculously, it seemed to have had a "positive" effect. At the first glance when I read profasaurus's report, I thought I saw a rare touch down pass that was spectacularly performed. But wait a minute, as everyone was at a loss of how to explain this odd phenomena, I went back and examined the effect at the source code level. Lo and behold. It made a lot of sense to me. But, not in the "intended" way. It turned out to be more of an embarrassing bug that was "fixed" by an accident --- nothing to celebrate. It was not a touch down pass. It was a very rare case of an accidental discovery of go-around of a mysterious bug by an insightful suggestion by an experienced XXCOPY user who pays close attention to a strange documentation on a strange feature. Here's how it went wrong... XXCOPY's help text provides the following explanation: ----------------------------------------------------------- /CL Sets the cache limit (file size upper limit, default = 1G). /CL0 No cache limit (a file-write always uses cache). /CL Sets cache limit (file size upper limit for file-write operation). When a huge file is copied, excessive cache activity (known as thrashing) may occur. A proper virtual memory setting should avoid the thrashing. A /CL setting usually alleviates the thrashing with the expense of reduced performance without cache. ---------------------------------------------------------- The source of the trouble began when this documentation was produced. Because the /CL0 was stating the exact opposite of what it was doing. The doomed /CL switch sets a limit on the write-behind-cache. By default, the limit was set to a very high value (1 GB). That is, up to 1 GB, a file will be written as usual, using the system cache (XXCOPY does not do anything). When the /CL1G is in effect (by default), the ill-conceived code adds the "FILE_FLAG_WRITE_THROUGH flag in the CreateFile() function when the destination file is opened. This is one of the very rare instances where XXCOPY tries to manipulate how the file I/O operates ( as I often assert, XXCOPY does not access devices using any low-level device access function.). But, when we find the standard file I/O functions ---- CreateFile() as the pivotal function for XXCOPY, with myriad of flags and options, we try to take advantage of them as much as possible. Going back to the /CL case, the use of /CL0 will remove the upperlimit of the cache-enable range. That is, contrary to the documentation, XXCOPY was allowed to use the write-cache even beyond the default 1 GB limit. This explains why a few users recently reported their problems with an excessively low file transfer performance with very large files. As a matter of fact, it was profasaurus's alertness that brought this to light. So, yes, /CL0 works. It works almost by an accident. It removed the bug in a very funny, unintended way. I hope you followed my explanation now. Mystery solved. But what now? ================================ It is time to remove the obscure "feature" which was nothing but trouble. In retrospect, the terse Microsoft documentation fooled me and somewhere, our documentation was saying the opposite of what it was doing. More careful reading of Microsoft documentation made me aware of the proper usage of the flag. As a remedy to this bug, I decided to introduce a new feature with relation to the cache control. /CA // cache control (enable all by default) /CA // n is a bit-mapped parameter /CA is equivalent to /CA7 (all cache enabled) The /CA parameter (n) controls the following three independent cache-related settings: bit2: enables Source file read cache bit1: enables Destination file write cache bit0: enables Destination file read cache An XXCOPY file copy operation, has four phases: 1. Read from the source file. 2. Write to the destination file. 3. Read from the source file (in /V2 case) 4. Read from the destination file (in /V2 case) (the last two are for a verify-after-copy.) Summary of the /CA variations for cache control. switch src-rd dst-wr dst-rd ------------------------------------------- /CA on on on /CA7 on on on /CA6 on on off /CA5 on off on /CA4 on off off /CA3 off on on /CA2 off on off /CA1 off off on /CA0 off off off We will leave the /CL switch as a legacy feature. /CL0 is now equivalent to /CA (all cache enabled) and /CL1 will be treated as /CA5 (write-cache disabled). So, even after the introduction of the new /CA switch, profasaurus's xxcopy scripts will continue to work. But, I rather want everyone to forget this nightmare and alter the legacy /CL0 switch to /CA. Now, I was rather shocked to find that presence of the write-cache makes a huge performance difference on file transfer over a network. The reason why I put extra efforts on the /CA feature was that we have heard many CD-R users who wanted to disable cache in order to perform a true data verify operation. Although the performance may suffer, either disabling the dst-wr cache (/CA5), or dst-rd (/CA6) cache, (or even disabling both /CA4), the user may force the CD-R drive to perform the real verify-after-write operation. Since it is hard for us to predict which cache operation would be most useful, we will let the user determine whichever one to use. It's an empirical approach. The users should find out what works best rather than we force our hard-coded solution based upon our speculation to the end user. This feature will debut hopefully in a few days as the new betatest version (v.2.84.1). Kan Yabumoto
This message if part of XXCOPY's message Archive. The archive contains all the messages posted at Yahoo!Groups: XXCOPY.