XXCOPY
[ Back to Table of Contents ] [ << ] [ >> ]

XXCOPY TECHNICAL BULLETIN #41


From:    Kan Yabumoto           tech@xxcopy.com
To:      XXCOPY user
Subject: File Compression on the NTFS Volume
Date:    2008-03-15
===============================================================================

Introduction:

    One of the advantages in NTFS is its file compression capability.
    Starting with v 2.95.0, XXCOPY adds features that relate to
    file compression on the NTFS volume.
    
    In many instances, the file system treats the compression state
    of a file somewhat like an additional bit in the file attributes.
    On the other hand, there are many applications (notably, Windows
    Explorer and the command-line utility, ATTRIB.EXE to name a few)
    that treat the new C-bit (Compression bit) differently.  Furthermore,
    the way files and directories are created in a directory is
    dictated by the directory's compression setting.  In short,
    the C-bit in the NTFS volume is not like other ordinary file
    attributes.


Compression, Pros and Cons:

    The NTFS design with regard to the compression setting is
    quite elegant and very easy to use.  For example, when you
    initialize a new NTFS partition in Windows XP, the Disk Management
    tool (DiskMgmt.msc) provides a checkbox for volume-wide compression
    setting immediately before a volume format operation.  Or, you
    may select any individual directory and enable or disable the
    compression setting through its property sheet.  Thereafter,
    its contents are created using the parent directory's compression
    setting.  You may later change the compression setting of any
    file or directory regardless of the parent directory's setting.
    With the property sheet, it is quite intuitive and easy to use.
    The best part of this scheme is that a compressed file behaves
    exactly like an uncompressed file from application program's
    point of view.  It is transparent to most programs.  Ultimately,
    file compression can save substantial storage space.

    The down side of the file compression is, of course, some
    penalty in access speed.  The trade off between the speed and
    the space saving should be carefully weighed when you implement
    the file compression on your NTFS volumes.  If you strike a
    good balance between the two factors, there are a lot to gain
    by using the file compression feature.

    It seems that the relative obscurity of the file compression
    may be due to the lack of efficient control of the settings.
    First of all, the most common way to manipulate the setting
    is by Windows Explorer's property sheet for the file/directory
    object which requires a second level of dialog in the Advanced
    Attributes setting window.  It takes six mouse button clicks
    per file to compress an individual file.


XXCOPY to handle the compression settings:

    Starting with v.2.95.0 (released 2007-03-01), XXCOPY supports
    the file attributes C-bit (Compression).  Although the C-bit
    behaves somewhat differently from the rest of the common
    file attributes, XXCOPY will treat it uniformly.  That is,
    when you would like to convert a regular file to a compressed
    file, you set the C-bit of the file attributes using XXCOPY
    (the standard ATTRIB.EXE utility can handle only the A, H, S,
    and R bits).

    Remember how XXCOPY made files hidden and readonly?

      xxcopy  c:\mydir\*.doc  /AS:HR     // set H and R bits

    Then, the following is how to clear all AHSR bits:

      xxcopy  c:\mydir\*.doc  /AR:AHSR   // reset all four bits

      ----------------------------------------------------------
       In addition to setting and resetting the file attributes
       bits, the /AN switch can be used to force a new value of
       any combination of the A, H, S, or R attribute bits

          xxcopy  c:\mydir\*.doc  /AN:AR   // force a new value

       Please note that the /AN operation does not support the
       C-bit just like the ATTRIB.EXE utility whose scope of
       operation is confined int the the A, H, S, or R bits.
      ----------------------------------------------------------


    Changing the compression state of files uses the same command:

      xxcopy  c:\mydir\*.doc  /AS:C      // set the C-bit
      xxcopy  c:\mydir\*.doc  /AR:C      // reset the C-bit

    This is how you can compress a group of files in one breath
    and also how to uncompress them.  A change in the C-bit in the
    file attributes involves a compression or decompression operation
    of the file data which goes through file read/write accesses that
    are obviously much slower than normal changes in the file attribute
    bits.  If the operations would not change the existing C-bit value,
    XXCOPY's file attribute operation skips the assignment of the C-bit
    value for efficiency's sake.


The relationship among the /AS, /AR, and /AN switches:

    Although the /AS (Set) and /AR (Reset) operations are listed
    in the same group in the documentation and related switches
    are typically treated as mutually exclusive (and if so, the
    last one always prevails), XXCOPY accepts both the /AS and /AR
    switches in one command, allowing the combination of the set
    and reset operations in one step as a special exception to
    the switch-grouping rule.

     --------------------------------------------------------------
      When contradicting set of parameters are specified with
      /AS and /AR, then, the last value prevails.  For example,
      the following two commands achieve exactly the same result:

        xxcopy *.doc /AS:H /AR:AHC  // H-bit to be set, then reset 
        xxcopy *.doc       /AR:AHC  // H-bit will simply be reset
     --------------------------------------------------------------

    While both the /AS and /AR operations work as a bit-wise
    manipulation of the existing file attributes value, the third
    method, the /AN operation assigns a whole new value to the
    file or directory object.  Therefore, the /AN switch and the
    other two (/AS and /AR as a pair if you will) are treated as
    mutually exclusive group as usual (the last one prevails).


Compression Setting of Directories:

    Although XXCOPY's original design was mostly on files, the
    NTFS file compression feature demands XXCOPY to manipulate
    the directory attributes (known as "File Attributes" of
    directory).  So, the /AS and /AR functions are extended with
    the /ASD and /ARD functions respectively to handle directories
    as opposed to files.  To avoid complexities in the command
    line rules, we chose not to mix the two types of operations
    into one.

    That is, for file attributes, use /AS or /AR and for directory
    attributes, use /ASD and /ARD.  The required parameter
    (a combination of letters A, H, S, R, and C) works exactly the
    same for /ASD and /ARD.

    E.g.,

      xxcopy  c:\mydir\    /ASD:C       // set the C-bit on mydir
      xxcopy  c:\mydir     /ASD:C       // set the C-bit on mydir

        ----------------------------------------------------------
         Note that the trailing backslash is not needed because
         the /ASD switch disambiguates the meaning of the source
         specifier (c:\mydir).  However, we strongly suggest that
         you form a good habit of always furnishing the trailing
         backslash whenever you specify the name of a directory.
        ----------------------------------------------------------

    The important thing is that when the C-bit of a directory is
    set, the effect is limited to the directory's attributes setting
    itself.  The current contents of the directory, if present,
    will not be compressed by this action.  Files and subdirectories
    created thereafter in the directory will automatically inherit
    the compression setting of the directory (files will saved in
    compressed mode).

    If you want to compress the whole directory and its existing
    subdirectories, then, you should specify /S as usual.

      xxcopy  c:\mydir\    /ASD:C /S     // mydir and its subdirs

    Since the above command does not touch the files inside,
    you need a second command to convert the content files.

      xxcopy  c:\mydir\    /AS:C  /S     // files inside every level


    Note that XXCOPY does not perform a conversion in either
    direction (from compressed to uncompressed and vice versa)
    when the file is already in the right state of compression.
    In other words, there will be no performance penalty in
    letting XXCOPY compress a file that is already stored in
    compressed mode and vice versa.


Suggested Usages:

    As experienced XXCOPY users know, the power of XXCOPY is its
    ability to select files and directories.  The file compression
    feature of the NTFS volume is an excellent arena to show off
    XXCOPY's prowess.

    The following example is to crate a backup directory in compressed
    mode, then, uncompress certain files that does not benefit by
    compression.

      mybatch1.bat
     -------------------------------------------------------------------
      cd      d:\backup                   // create directory
      xxcopy  d:\backup\   /ASD:C         // set C-bit
      xxcopy  c:\mydir\   d:\backup\ /S   // copy all files compressed
      xxcopy  d:\backup\*.jpg  /in:*.mp3  /AR:C /S
     -------------------------------------------------------------------


    The second example below copies files into a non-compressed
    destination first, and then compress all the files except certain
    classes of files that that are known to be highly compressed
    already. 
        
      mybatch2.bat
     -------------------------------------------------------------------
      xxcopy  c:\mydir\   d:\backup\ /bu           // normal backup
      xxcopy  d:\backup\  /AS:C /X:*.jpg /X:*.mp3  // avoid dense files
     -------------------------------------------------------------------


Afterthoughts:

    It is quite natural for a user to wish a more direct control
    of file compression which is carried out Anthe.  That is,
    with the current version of XXCOPY, you need to copy files
    into the destination using the default compression mode set
    by the parent directory once (say, in uncompressed mode), and
    then, to compress the files in a second pass.  Instead, it
    would be nice to directly write the selected file in compressed
    mode as a part of the file-copy operation when the destination
    directory is not set for compression.  We will explore such
    a feature in the future.



© Copyright 2016 Pixelab All rights reserved.

[ XXCOPY Home ] [ Table of Contents ] [ << ] [ >> ]

Join the XXCOPY group