[ Back to Table of Contents ]
[ << ]
[ >> ]
XXCOPY TECHNICAL BULLETIN #41
From: Kan Yabumoto tech@xxcopy.com
To: XXCOPY user
Subject: File Compression on the NTFS Volume
Date: 2008-03-15
===============================================================================
Introduction:
One of the advantages in NTFS is its file compression capability.
Starting with v 2.95.0, XXCOPY adds features that relate to
file compression on the NTFS volume.
In many instances, the file system treats the compression state
of a file somewhat like an additional bit in the file attributes.
On the other hand, there are many applications (notably, Windows
Explorer and the command-line utility, ATTRIB.EXE to name a few)
that treat the new C-bit (Compression bit) differently. Furthermore,
the way files and directories are created in a directory is
dictated by the directory's compression setting. In short,
the C-bit in the NTFS volume is not like other ordinary file
attributes.
Compression, Pros and Cons:
The NTFS design with regard to the compression setting is
quite elegant and very easy to use. For example, when you
initialize a new NTFS partition in Windows XP, the Disk Management
tool (DiskMgmt.msc) provides a checkbox for volume-wide compression
setting immediately before a volume format operation. Or, you
may select any individual directory and enable or disable the
compression setting through its property sheet. Thereafter,
its contents are created using the parent directory's compression
setting. You may later change the compression setting of any
file or directory regardless of the parent directory's setting.
With the property sheet, it is quite intuitive and easy to use.
The best part of this scheme is that a compressed file behaves
exactly like an uncompressed file from application program's
point of view. It is transparent to most programs. Ultimately,
file compression can save substantial storage space.
The down side of the file compression is, of course, some
penalty in access speed. The trade off between the speed and
the space saving should be carefully weighed when you implement
the file compression on your NTFS volumes. If you strike a
good balance between the two factors, there are a lot to gain
by using the file compression feature.
It seems that the relative obscurity of the file compression
may be due to the lack of efficient control of the settings.
First of all, the most common way to manipulate the setting
is by Windows Explorer's property sheet for the file/directory
object which requires a second level of dialog in the Advanced
Attributes setting window. It takes six mouse button clicks
per file to compress an individual file.
XXCOPY to handle the compression settings:
Starting with v.2.95.0 (released 2007-03-01), XXCOPY supports
the file attributes C-bit (Compression). Although the C-bit
behaves somewhat differently from the rest of the common
file attributes, XXCOPY will treat it uniformly. That is,
when you would like to convert a regular file to a compressed
file, you set the C-bit of the file attributes using XXCOPY
(the standard ATTRIB.EXE utility can handle only the A, H, S,
and R bits).
Remember how XXCOPY made files hidden and readonly?
xxcopy c:\mydir\*.doc /AS:HR // set H and R bits
Then, the following is how to clear all AHSR bits:
xxcopy c:\mydir\*.doc /AR:AHSR // reset all four bits
----------------------------------------------------------
In addition to setting and resetting the file attributes
bits, the /AN switch can be used to force a new value of
any combination of the A, H, S, or R attribute bits
xxcopy c:\mydir\*.doc /AN:AR // force a new value
Please note that the /AN operation does not support the
C-bit just like the ATTRIB.EXE utility whose scope of
operation is confined int the the A, H, S, or R bits.
----------------------------------------------------------
Changing the compression state of files uses the same command:
xxcopy c:\mydir\*.doc /AS:C // set the C-bit
xxcopy c:\mydir\*.doc /AR:C // reset the C-bit
This is how you can compress a group of files in one breath
and also how to uncompress them. A change in the C-bit in the
file attributes involves a compression or decompression operation
of the file data which goes through file read/write accesses that
are obviously much slower than normal changes in the file attribute
bits. If the operations would not change the existing C-bit value,
XXCOPY's file attribute operation skips the assignment of the C-bit
value for efficiency's sake.
The relationship among the /AS, /AR, and /AN switches:
Although the /AS (Set) and /AR (Reset) operations are listed
in the same group in the documentation and related switches
are typically treated as mutually exclusive (and if so, the
last one always prevails), XXCOPY accepts both the /AS and /AR
switches in one command, allowing the combination of the set
and reset operations in one step as a special exception to
the switch-grouping rule.
--------------------------------------------------------------
When contradicting set of parameters are specified with
/AS and /AR, then, the last value prevails. For example,
the following two commands achieve exactly the same result:
xxcopy *.doc /AS:H /AR:AHC // H-bit to be set, then reset
xxcopy *.doc /AR:AHC // H-bit will simply be reset
--------------------------------------------------------------
While both the /AS and /AR operations work as a bit-wise
manipulation of the existing file attributes value, the third
method, the /AN operation assigns a whole new value to the
file or directory object. Therefore, the /AN switch and the
other two (/AS and /AR as a pair if you will) are treated as
mutually exclusive group as usual (the last one prevails).
Compression Setting of Directories:
Although XXCOPY's original design was mostly on files, the
NTFS file compression feature demands XXCOPY to manipulate
the directory attributes (known as "File Attributes" of
directory). So, the /AS and /AR functions are extended with
the /ASD and /ARD functions respectively to handle directories
as opposed to files. To avoid complexities in the command
line rules, we chose not to mix the two types of operations
into one.
That is, for file attributes, use /AS or /AR and for directory
attributes, use /ASD and /ARD. The required parameter
(a combination of letters A, H, S, R, and C) works exactly the
same for /ASD and /ARD.
E.g.,
xxcopy c:\mydir\ /ASD:C // set the C-bit on mydir
xxcopy c:\mydir /ASD:C // set the C-bit on mydir
----------------------------------------------------------
Note that the trailing backslash is not needed because
the /ASD switch disambiguates the meaning of the source
specifier (c:\mydir). However, we strongly suggest that
you form a good habit of always furnishing the trailing
backslash whenever you specify the name of a directory.
----------------------------------------------------------
The important thing is that when the C-bit of a directory is
set, the effect is limited to the directory's attributes setting
itself. The current contents of the directory, if present,
will not be compressed by this action. Files and subdirectories
created thereafter in the directory will automatically inherit
the compression setting of the directory (files will saved in
compressed mode).
If you want to compress the whole directory and its existing
subdirectories, then, you should specify /S as usual.
xxcopy c:\mydir\ /ASD:C /S // mydir and its subdirs
Since the above command does not touch the files inside,
you need a second command to convert the content files.
xxcopy c:\mydir\ /AS:C /S // files inside every level
Note that XXCOPY does not perform a conversion in either
direction (from compressed to uncompressed and vice versa)
when the file is already in the right state of compression.
In other words, there will be no performance penalty in
letting XXCOPY compress a file that is already stored in
compressed mode and vice versa.
Suggested Usages:
As experienced XXCOPY users know, the power of XXCOPY is its
ability to select files and directories. The file compression
feature of the NTFS volume is an excellent arena to show off
XXCOPY's prowess.
The following example is to crate a backup directory in compressed
mode, then, uncompress certain files that does not benefit by
compression.
mybatch1.bat
-------------------------------------------------------------------
cd d:\backup // create directory
xxcopy d:\backup\ /ASD:C // set C-bit
xxcopy c:\mydir\ d:\backup\ /S // copy all files compressed
xxcopy d:\backup\*.jpg /in:*.mp3 /AR:C /S
-------------------------------------------------------------------
The second example below copies files into a non-compressed
destination first, and then compress all the files except certain
classes of files that that are known to be highly compressed
already.
mybatch2.bat
-------------------------------------------------------------------
xxcopy c:\mydir\ d:\backup\ /bu // normal backup
xxcopy d:\backup\ /AS:C /X:*.jpg /X:*.mp3 // avoid dense files
-------------------------------------------------------------------
Afterthoughts:
It is quite natural for a user to wish a more direct control
of file compression which is carried out Anthe. That is,
with the current version of XXCOPY, you need to copy files
into the destination using the default compression mode set
by the parent directory once (say, in uncompressed mode), and
then, to compress the files in a second pass. Instead, it
would be nice to directly write the selected file in compressed
mode as a part of the file-copy operation when the destination
directory is not set for compression. We will explore such
a feature in the future.
© Copyright 2010 Pixelab, Inc. All rights reserved.
