In other words, if you use a glass box, Schrödinger's Cat always dies.

Multi-threaded Asynchronous File Copy Class for C#

It used to be that, since file I/O was your biggest bottleneck, copying files on multiple threads was inefficient, even slowing down the file copy.  Multiple simultaneous copy jobs would spend the most time just positioning read heads around disk sectors.  Modern disks, however, have vastly improved read-ahead caching.  SSD drives eliminate much of the time physical disks would spend positioning the read head.  If you have multiple disks in a RAID array or advanced SAN, this reduces the I/O bottleneck for simultaneous file copies.

In testing, I have found that multi-threaded copying cuts the time to copy multiple files anywhere from twenty to fifty percent over copying the files individually.  This holds true whether copying locally, across a LAN or WAN.  The TestCopyForm project linked below has a sample application with this class where you can adjust buffer size and specify whether to use multi-threading to copy files or copy them individually.  You can test with various combinations to see what works best for your environment.

Multi-Threaded Async File Copy Class

To use this class, add it to your project.  When you initialize an instance of the class, pass it either a single file source path and destination path, create a Queue of KeyValueObjects to pass multiple files, or use a simple new object initialization and add the files through the CopyList queue.

FileCopyMTStream fcs = new FileCopyMTStream();

string SourcePath = "c:\\somedir";
string TargetPath = "c:\\someotherdir";
string targetfile;

foreach (string sourcefile in Directory.GetFiles(SourcePath))
{
    targetpath = Path.Combine(TargetPath, Path.GetFileName(filepath));
    fcs.CopyList.Enqueue(new KeyValuePair<string, string>(filepath, targetpath));
}

Next, add event handlers to your code to handle status updates:

fcs.FileCopyStarted += Fcs_FileCopyStarted;
fcs.ProgressChanged += Fcs_ProgressChanged;
fcs.FileCopyComplete += Fcs_FileCopyComplete;
fcs.FileCopyException += Fcs_FileCopyException;

When you write the event handlers, make sure that you use Invoke to interact with the UI thread - the event handlers will be running under the context of one of the FileCopyMTStream threadpool threads.  If you try to access a control from that thread, it will throw an error.

private void Fcs_ProgressChanged(string SourceFile, int Percentage, long CopiedSize, long FileSize)
{
    Invoke(new MethodInvoker(delegate 
    {
        if (lvFileList.Items.ContainsKey(SourceFile))
        {
            lvFileList.Items[SourceFile].SubItems[3].Text = CopiedSize.ToString("N0") + " copied";
            lvFileList.Items[SourceFile].SubItems[4].Text = Percentage.ToString() + "%";
        }
    }
    ));
}

When you are ready to kick off the copy, call the following method:

fcs.StartCopy();

Here is a test project that includes the class:

TestCopyForm.zip

Here is the class.  Enjoy!

using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;

// FileCopyMTStream - File Copy Multi-Threaded Stream
// Adrian Hayes
// http://www.bearnakedcode.com
//
// A class which asynchronously copies one or more files using multiple
// threadpool threads.  It allows file copies without locking UI thread and
// provides status reports via events.
//
// License:  You are free to use this code in any of your projects as you wish.
//           It is offered without warranty.  Use at your own risk.
//           If you publish its source (or any portion thereof), please include
//           a reference back to http://www.bearnakedcode.com.
//
// Enjoy!

namespace BearNakedCode
{

    //Delegates for events
    public delegate void FileCopyStartedDelegate(string SourceFile, long FileSize);
    public delegate void FileCopyExceptionDelegate(string SourceFile, Exception Error);
    public delegate void CompleteDelegate(string SourceFile, bool Canceled);
    public delegate void ProgressChangedDelegate(string SourceFile, int Percentage, long CopiedSize, long FileSize);

    /// <summary>
    /// Copies a queue of files asynchronously with multi-threading.
    /// </summary>
    public class FileCopyMTStream
    {

        #region PublicVariables

        /// <summary>
        /// A queue of files to copy.  Elements are KeyValuePairs with Key = Source / Value = Destination.
        /// </summary>
        public Queue<KeyValuePair<string, string>> CopyList = new Queue<KeyValuePair<string, string>>();


        /// <summary>
        /// Will copy using multiple threads, managed by threadpool(default).
        /// If false, copy is limited to 1 thread.
        /// </summary>
        public bool MultiThreaded = true;


        /// <summary>
        /// Size of the copy buffer.  Default is 4 KB buffer.  
        /// </summary>
        public int BufferSize = 4 * 1024;


        /// <summary>
        /// When reporting progress, it will wait until it has copied this amount of
        /// bytes before raising the ProgressChanged event.  This is to reduce flicker
        /// in controls from overly-frequent status updates.  Default is 128K bytes.
        /// </summary>
        public long ReportInterval = (128 * 1024); // report only on each 128K block

        #endregion

        //provide cancellation method
        private static bool CancelRequested = false;


        #region EventHandlers

        /// <summary>
        /// Event that fires when progress has changed on a file copy.  
        /// </summary>
        public event ProgressChangedDelegate ProgressChanged;
        private void OnProgressChanged(string SourceFile, long CopiedSize, long FileSize)
        {
            int pct = (int)((double)CopiedSize * 100 / (double)FileSize);
            //int pct = (int)(Math.Round((double)CopiedSize / (double)FileSize) * 100);
            if (ProgressChanged != null)
                ProgressChanged(SourceFile, pct, CopiedSize, FileSize);
        }


        /// <summary>
        /// Event that fires when a file copy has started.
        /// </summary>
        public event FileCopyStartedDelegate FileCopyStarted;
        private void OnFileCopyStarted(string SourceFile, long FileSize)
        {
            if (FileCopyStarted != null)
                FileCopyStarted(SourceFile, FileSize);
        }

        /// <summary>
        /// Event that fires when a fatal exception occurs on a file copy.
        /// </summary>
        public event FileCopyExceptionDelegate FileCopyException;
        private void OnFileCopyException(string SourceFile, Exception Error)
        {
            if (FileCopyException != null)
                FileCopyException(SourceFile, Error);
        }

        /// <summary>
        /// Event that fires when a copy job finishes or is cancelled.
        /// </summary>
        public event CompleteDelegate FileCopyComplete;
        private void OnFileCopyComplete(string SourceFile, bool Cancelled)
        {
            if (FileCopyComplete != null)
                FileCopyComplete(SourceFile, Cancelled);
        }
        #endregion

        #region Initializers

        public FileCopyMTStream()
        {

        }

        /// <summary>
        /// Initializes a new instance of FileCopyMTStream for a single file copy.
        /// </summary>
        /// <param name="Source">Full path to the source file.</param>
        /// <param name="Destination">Full path to the destination file, including file name.</param>
        public FileCopyMTStream(string Source, string Destination)
        {
            CopyList.Enqueue(new KeyValuePair<string, string>(Source, Destination));
        }


        /// <summary>
        /// Initializes a new instance of FileCopyMTStream for multiple file copies.
        /// </summary>
        /// <param name="SourceDestinationList"></param>
        public FileCopyMTStream(Queue<KeyValuePair<string, string>> SourceDestinationList)
        {
            CopyList = SourceDestinationList;
        }

        #endregion

        /// <summary>
        /// Set to True to request all remaining copy jobs cancel at their current progress, close their streams and exit.
        /// </summary>
        public void CancelAll()
        {
            CancelRequested = true;
        }


        /// <summary>
        /// Begins the asynchronous file copies.
        /// </summary>
        public void StartCopy()
        {
            WaitCallback callback;

            KeyValuePair<string, string> job;

            if (MultiThreaded == false)
            {
                new Thread(delegate () { SingleThreadCopyJob(CopyList); }).Start();
            }
            else
            {
                while (CopyList.Count > 0)
                {
                    job = CopyList.Dequeue();
                    if (CheckCancelRequested(job.Key))
                        return;

                    callback = new WaitCallback(CopyJob);
                    ThreadPool.QueueUserWorkItem(callback, job);
                }

            }
        }

        /// <summary>
        /// Copies a file.  Used in thread pool operations to copy multiple files simultaneously. 
        /// </summary>
        /// <param name="job">A source/target pair to copy a file.</param>
        private void CopyJob(object job)
        {
            //copyjob - KVP.  Key = source file.  Value = destination file
            KeyValuePair<string, string> copyjob = (KeyValuePair<string, string>)job;

            if (CheckCancelRequested(copyjob.Key))
                return;

            CopyFileStream(copyjob);
        }

        /// <summary>
        /// Copies all files, one at a time, on the same thread.  
        /// Use where multiple I/O streams would present a bottleneck.
        /// </summary>
        /// <param name="JobQueue">A queue of source/target pairs of files to copy.</param>
        private void SingleThreadCopyJob(Queue<KeyValuePair<string, string>> JobQueue)
        {
            KeyValuePair<string, string> job;
            while (CopyList.Count > 0)
            {
                job = CopyList.Dequeue();
                if (CheckCancelRequested(job.Key))
                    return;
                CopyFileStream(job);
            }

        }

        /// <summary>
        /// Checks if the CancelRequested flag has been raised.  
        /// Sends copy Complete event notification if cancelled.
        /// </summary>
        /// <param name="SourceFile">The file currently set to be copied.</param>
        /// <returns>True if the cancel flag set.  False if OK to copy the file.</returns>
        private bool CheckCancelRequested(string SourceFile)
        {
            if (CancelRequested)
            {
                OnFileCopyComplete(SourceFile, true);
                return true;
            }

            return false;
        }


        /// <summary>
        /// Uses a FileStream to copy a file from source to destination.  
        /// Sends event notifications on progress and completion.  
        /// </summary>
        /// <param name="CopyJob">
        /// A Key Value pair of Source and Target paths for the file to copy.
        /// </param>
        private void CopyFileStream(KeyValuePair<string, string> CopyJob)
        {
            byte[] buffer = new byte[BufferSize]; // 16K buffer

            long reporttally = 0;

            using (FileStream source = new FileStream(CopyJob.Key, FileMode.Open, FileAccess.Read))
            {
                long filelength = source.Length;

                OnFileCopyStarted(CopyJob.Key, filelength);

                using (FileStream dest = new FileStream(CopyJob.Value, FileMode.Create, FileAccess.Write))
                {
                    long totalBytes = 0;
                    int currentBlockSize = 0;

                    while ((currentBlockSize = source.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        if (CheckCancelRequested(CopyJob.Key))
                            return;

                        dest.Write(buffer, 0, currentBlockSize);

                        totalBytes += currentBlockSize;

                        reporttally += currentBlockSize;
                        if (reporttally >= ReportInterval)
                        {
                            OnProgressChanged(CopyJob.Key, totalBytes, filelength);
                            reporttally = 0;
                        }

                    }
                }
            }

            OnFileCopyComplete(CopyJob.Key, CancelRequested);
        }
    }
}

Regular Expression Tester

I "regularly"  (nyuk nyuk!) use regular expressions in my .Net code.  However, I find that I spend no small amount of time designing them, troubleshooting why they aren't working and trying to work out the escape characters correctly.  So much so that I decided to write a Regex testing application.

Regex Tester (zipped .exe)

C# Source Code

This tool allows me to rapidly test regular expressions against sample text with or without options.

It shows me matches, splits and replaced text.

If the Regex phrase is invalid, it will display the error text:

When I'm satisfied with the phrase and options, I click Copy Regex Initializer to copy it to the clipboard:

I can paste the copied text directly into my code after declaring the Regex object:

I hope you find it useful.

Win 2008 Terminal Server Network App Crashing

How many Microsoft Developers does it take to change a lightbulb? None. They just make darkness the new standard.

You can no longer safely run applications from a network location in Windows 2008 & R2 terminal server. When a user opens a program across the network and logs off, it will crash the program for all other users running it on the same server.  It appears that Microsoft is no longer supporting functionality that has been present for 20+ years and is not going to fix the bug due to "Architectural Changes" in Windows 2008.

http://support.microsoft.com/kb/2536487

Here is a summary of the issue:

1. When a user opens any network application in a 2008 or R2 Terminal Server environment, the OS creates a File Control Block (FCB). The FCB is a handle that the OS uses to access the program file loaded into memory.
   
   
2. If another user on the terminal server opens the same program, the OS will give the second user access to the first user’s FCB and access to that part of the original user’s memory space that stores the executable.
   
   
3. When the first user logs off, all their FCB’s are dropped and become inaccessible to other users that were sharing them.
   
   
4. The next action the remaining users perform in the program fails and it crashes the application because it cannot access the program files.
   

You can see this reported in the Application event log on two entries with the same time stamp:

• Application Error, Event ID 1000 – “Faulting application xxxxx.exe, version xxxxx…”
  
  
• Application Error, Event ID 1005 – “Windows cannot access the file for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program XXXXX because of this error.”

Microsoft says the "New Standard" is to load applications locally or use a WebDAV share. Local install works ok in one or two-server environments, not so well in 400+ server environments. Applying program updates to multiple servers is an administrative nightmare and a waste of resources as your environment grows. And WebDAV? Really?

We've gotten past Microsoft's Offshore Support Defense Forces and we're finally in discussions with high-level US engineers at Microsoft. We're attempting to convince them of the gravity of the issue and to resolve it. The engineers have acknowledged that they have received multiple reports of the issue but that MS development is refusing to fix it.

They said that the bug is coded deep in the 2008 OS and would require architectural redesign, so they are reluctant to make any changes. It looks like they tried to fix it in 2008 R2, because it now shares the FCB of the last user who opened the program file instead of the first, but it still crashes when they log off before another user grabs the ball.

It seems like they forked in the wrong direction and locked themselves in a faulty design.

Update - 2016.11.03 - The Microsoft KB article 2536487 states that Windows Server 2016 fixes this issue.  If you have installed Windows 2016 and verified that this is fixed, I would love to hear from you in the comments.  Thanks!

Update - 2012.05.01 - Multiple responders have reported that accessing the network files over a DFS share eliminates the FCB errors.  See comments below.

Update - 2011.09.23 - We have had some success running the programs from a UNC path instead of using the mapped network drive.  This is still in testing, but I believe you may also need to remove the mapped drive to eliminate the application crashes. If this works in your environment, please post back in the comments.

Connect to a UNC path with security credentials

I had an ASP.NET site where I wanted to access network resources, but did not have sufficient share permissions because the code ran under the ASP user. I also had a service that copied files every night between file shares in two different domains. I wanted a way to access remote resources without opening up security holes by changing permissions or running as a privileged user. Surprisingly, there is a lack of .Net managed code that will allow you to do this.

I have seen many coders who get around this problem via LogonUser() - programmatically log onto the box as a user who has the necessary privileges and perform the required task. This works fine if you grant the user local logon rights on the box. I'm not comfortable doing that on a public-facing box. I have also encountered code that shells out and performs a net use command. Tsk. So uncivilized. :)

The NetUseAdd() and NetUseDel() commands of NetApi32.dll provide a way to add access to a network resource with network credentials. This will allow you to run the process from whatever user you wish, but still access protected network resources.

The UNCAccessWithCredentials.cs class wraps these functions in an IDisposable class that allows you to encompass your code in a using block. When you encounter the end of the block, the class disconnects from the resource and the connection is closed. If you need a persistent connection, call the NetUseWithCredentials function, but remember to clean up your mess with the NetUseDelete function.

Example usage:

 
using (UNCAccessWithCredentials unc = new UNCAccessWithCredentials())
{
  if (unc.NetUseWithCredentials(UNCPath, UserName, Domain, Password))
  {
      //Code that accesses the remote resource
  }
  else
  {
      MessageBox.Show("Failed to connect to " + UNCPath + "\r\nLastError = " + unc.LastError.ToString(),
                      "Failed to connect",
                      MessageBoxButtons.OK,
                      MessageBoxIcon.Error);
  }

}
 

UNCAccessWithCredentials.cs:

 
using System;
using System.Runtime.InteropServices;
using BOOL = System.Boolean;
using DWORD = System.UInt32;
using LPWSTR = System.String;
using NET_API_STATUS = System.UInt32;

namespace ConnectUNCWithCredentials
{
 public class UNCAccessWithCredentials : IDisposable
 {
     [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
     internal struct USE_INFO_2
     {
         internal LPWSTR ui2_local;
         internal LPWSTR ui2_remote;
         internal LPWSTR ui2_password;
         internal DWORD ui2_status;
         internal DWORD ui2_asg_type;
         internal DWORD ui2_refcount;
         internal DWORD ui2_usecount;
         internal LPWSTR ui2_username;
         internal LPWSTR ui2_domainname;
     }

     [DllImport("NetApi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
     internal static extern NET_API_STATUS NetUseAdd(
         LPWSTR UncServerName,
         DWORD Level,
         ref USE_INFO_2 Buf,
         out DWORD ParmError);

     [DllImport("NetApi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
     internal static extern NET_API_STATUS NetUseDel(
         LPWSTR UncServerName,
         LPWSTR UseName,
         DWORD ForceCond);

     private bool disposed = false;

     private string sUNCPath;
     private string sUser;
     private string sPassword;
     private string sDomain;
     private int iLastError;

     /// <summary>
     /// A disposeable class that allows access to a UNC resource with credentials.
     /// </summary>
     public UNCAccessWithCredentials()
     {
     }

     /// <summary>
     /// The last system error code returned from NetUseAdd or NetUseDel.  Success = 0
     /// </summary>
     public int LastError
     {
         get { return iLastError; }
     }

     public void Dispose()
     {
         if (!this.disposed)
         {
             NetUseDelete();
         }
         disposed = true;
         GC.SuppressFinalize(this);
     }

     /// <summary>
     /// Connects to a UNC path using the credentials supplied.
     /// </summary>
     /// <param name="UNCPath">Fully qualified domain name UNC path</param>
     /// <param name="User">A user with sufficient rights to access the path.</param>
     /// <param name="Domain">Domain of User.</param>
     /// <param name="Password">Password of User</param>
     /// <returns>True if mapping succeeds.  Use LastError to get the system error code.</returns>
     public bool NetUseWithCredentials(string UNCPath, string User, string Domain, string Password)
     {
         sUNCPath = UNCPath;
         sUser = User;
         sPassword = Password;
         sDomain = Domain;
         return NetUseWithCredentials();
     }

     private bool NetUseWithCredentials()
     {
         uint returncode;
         try
         {
             USE_INFO_2 useinfo = new USE_INFO_2();

             useinfo.ui2_remote = sUNCPath;
             useinfo.ui2_username = sUser;
             useinfo.ui2_domainname = sDomain;
             useinfo.ui2_password = sPassword;
             useinfo.ui2_asg_type = 0;
             useinfo.ui2_usecount = 1;
             uint paramErrorIndex;
             returncode = NetUseAdd(null, 2, ref useinfo, out paramErrorIndex);
             iLastError = (int)returncode;
             return returncode == 0;
         }
         catch
         {
             iLastError = Marshal.GetLastWin32Error();
             return false;
         }
     }

     /// <summary>
     /// Ends the connection to the remote resource 
     /// </summary>
     /// <returns>True if it succeeds.  Use LastError to get the system error code</returns>
     public bool NetUseDelete()
     {
         uint returncode;
         try
         {
             returncode = NetUseDel(null, sUNCPath, 2);
             iLastError = (int)returncode;
             return (returncode == 0);
         }
         catch
         {
             iLastError = Marshal.GetLastWin32Error();
             return false;
         }
     }

     ~UNCAccessWithCredentials()
     {
         Dispose();
     }

 }
}
 

Change Height of TextBoxes Using Font Size

Download sample code

I had a c# project that required a single-line text box with adjustable height. I found many examples of how to adjust the width, but nothing on how to dynamically change the height of a text box. I did not want to use multi-line because I wanted to use the auto-complete features of the single-line text box.

Single-line textbox height is set by the size of the font, not the TextBox.Height property. This makes it difficult if you are looking for an exact height. Luckily, the font property uses a float for the font size (emSize). You can use fractions of fonts to fine-tune the textbox height.

The calculation the textbox uses to determine its height is:

Height = ( Font Size * Font Line Spacing / Font Em Height ) + 7

• Font Size - It is easiest to measure font in pixels so you do not have to factor in screen dpi.
• Font Line Spacing - The distance, in design units, between two consecutive lines of text.
• Font Em Height - height, in design units of the font's widest letter - typically the letter M.

Text boxes have a 3-pixel lower and 4-pixel upper white space around the font height. Therefore, the calculation adjusts the height by 7 pixels.

We can reverse this calculation to obtain the font size needed for a desired height:

Font Size = ( height - 7 ) * Font Em Height / Font Line Spacing

This method will return a font object that will set the size of your text box:

private Font GetFontForTextBoxHeight(int TextBoxHeight)
{
    // What is the target size of the text box?
    float desiredheight = (float)TextBoxHeight;

    // Set the font from the existing TextBox font.
    // We use the fnt = new Font(...) method so we can ensure that
    //  we're setting the GraphicsUnit to Pixels.  This avoids all
    //  the DPI conversions between point & pixel.
    Font fnt = new Font(textBox1.Font.FontFamily,
                        textBox1.Font.Size,
                        textBox1.Font.Style,
                        GraphicsUnit.Pixel);

    // TextBoxes never size below 8 pixels. This consists of the
    // 4 pixels above & 3 below of whitespace, and 1 pixel line of
    // greeked text.
    if (desiredheight < 8)
        desiredheight = 8;

    // Determine the Em sizes of the font and font line spacing
    // These values are constant for each font at the given font style.
    // and screen DPI.
    float FontEmSize = fnt.FontFamily.GetEmHeight(fnt.Style);
    float FontLineSpacing = fnt.FontFamily.GetLineSpacing(fnt.Style);

    // emSize is the target font size.  TextBoxes have a total of
    // 7 pixels above and below the FontHeight of the font.
    float emSize = (desiredheight - 7) * FontEmSize / FontLineSpacing;
    
    // Create the font, with the proper size to change the TextBox Height to the desired size.
    fnt = new Font(fnt.FontFamily, emSize, fnt.Style, GraphicsUnit.Pixel);

    return fnt;
}

Whenever you have to set the textbox size, set the font property using the above method:

TextBox.Font = GetFontForTextBoxHeight(int TextBoxHeight)

Good Hex Editor

Every once in a while, I need a hex editor to poke at a file. There are a lot of editors out there, and a lot of chaff among them. I have a few requirements for what constitutes a good hex editor:

• Free - I don't use hex editors enough to justify a $40 editor.
• GUI
  
• Unicode capabilities
• byte-level editing
• Cursor tracking between hex and text frames
• hex and ASCII/Unicode text searches
• File comparison - what has changed?

I found one that covers most of these requirements, plus a few more neato features. If you haven't checked out XVI32, give it a try. I particularly like the wildcard search and replace function.

The only thing I would like to see is a file comparison feature. It helps in hacking another program: save a copy of some program's data, then make a change through the program and compare with the reference file. There may be a way to do this through the scripting interface (another neat feature), but it would be nice to see that in the application.