Microsoft Azure File Service API and Synchronizing Files with Local Machine

09/06/2014

One of the new features that was release for public preview during TechEd 2014, was the Microsoft Azure File Service. The file service enables multiple virtual machines hosted in the same Azure Region to connect to a network share that lives in Azure storage. This is nice feature as previously you had to make your own network share on a virtual machine if you wanted to share data between several machines via SMB.

2015-05-22: Updated the blog post to reflect the GA version of the Files API and not the preview

What is even better the Azure File Service comes with a REST API very similar to blob storage API which you can leverage to put things into your network share even if you are not on a Azure virtual machine. I was hoping that it was somehow possible to mount the network share even from machine outside of Microsoft Azure, but that is currently not possible, as you can only connect to a network share on a storage account within the same region as your virtual machine. But with the REST API we can build our own file sychronization tool that can keep files in a local folder insync with the files of the network share, but more on that later.

Get started with Azure File Service

Okay so you want to test it out, here is how it works.

  1. Visit the Azure Preview portal and signup for Azure Files. Be warned that there quite a waiting list, it took me around 14 days to be accepted into the preview program.
  2. Once accepted (you will receive an email), create a new the storage account you want your network share to live in. (Don’t auto create the storage account by creating a new machine from the Azure Portal, because for some reason the associated storage account created in this way won’t have the Azure File service enabled). If the storage account is created correct you will see the following in your dashboard for the account.
    azurefiles
  3. Now create a virtual machine in the same region as your storage account and log on to it
  4. Before you can connect to your Azure Files network share, we have to create one. The commands are now part of the general Azure Powershell that can be downloaded by following this guide.Currently that is a little more involved than it will be in the future, the reason is the Azure Files API is only available in the Micorsoft.WindowsAzure.Storage 4.0 library, which is not the one used by the general Azure Powershell scritps. So we have to download a special powershell package for working with the Azure Files, the package can be found here
  5. Once the Azure Powershell is downloaded and installed, we are ready to create a new Azure File network share
  6. Open the Azure Powershell command prompt, where you are located in the folder as where you unzipped AzureStorageFile.zip, and then use the commands to create a new network share (It is similar to the concept of creating a new BLOB storage container).
    # Create a context for account and key
    $ctx=New-AzureStorageContext [STORAGE_ACCOUNT_NAME] [STORAGE_ACCOUNT_KEY]

    # create a new share
    $s = New-AzureStorageShare myshare -Context $ctx

    You have to replace [STORAGE_ACCOUNT_NAME] and [STORAGE_ACCOUNT_KEY] with the respective values for your storage account.

  7. To mount the network share on your Azure Virtual machine, you use the following command net use z: \\[STORAGE_ACCOUNT_NAME].file.core.windows.net\myshare /u:[STORAGE_ACCOUNT_NAME] [STORAGE_ACCOUNT_KEY]
    Then you end up with a nice network share with 5TB of data (the maximum currently supported)
    myshare
  8. Creating directories, uploading files or downloading files to your newly created Azure Files network share can be also done with the following powershell commands (most likely you will just mount the network share and use Windows Explorer, but it is also possible with powershell)

    $ctx=New-AzureStorageContext [STORAGE_ACCOUNT_NAME] [STORAGE_ACCOUNT_KEY]

    ## create a new share
    #$s = New-AzureStorageShare myshare -Context $ctx

    $share = Get-AzureStorageShare myshare -Context $ctx

    # create a directory in the test share just created
    New-AzureStorageDirectory -Share $share -Path sjkp

    # upload a local file to the testdir directory just created
    Set-AzureStorageFileContent -Share $share -Source D:\upload\testfile.txt -Path sjkp

    # list out the files and subdirectories in a directory
    Get-AzureStorageFile -Share $share -Path sjkp

    # download files from azure storage file service
    Get-AzureStorageFileContent -Share $share -Path sjkp/testfile.txt -Destination D:\download

    # remove files from azure storage file service
    Remove-AzureStorageFile -Share $share -Path sjkp/testfile.txt

Now you are ready to use the Azure Files network share in your application, or maybe just on your Azure hosted development machines. Remember that you can also use Azure Files in your Cloud Services Instances, so it could serve as an easy way to share data between instances. Just be sure to understand the limitations of Azure Files the most important are:

For more details see the MSDN blog post from the dev team

Synchronize Local Files with Azure File Service

As I mentioned in the beginning I was a little disappointed that there was no seamless way to integrate with Azure Files from my local machine. Ideally I would have loved to be able to mount the network drive like I can in Microsoft Azure. I don’t know if it’s a feature that is going to be added or if it somehow already can be achieved with smart use of VNets in some sort of hybrid solution.

Instead I decided to take F# for a spin (I’m not a F# developer, infact this is my first attempt to build anything useful with F#), and tried to build a simple file synchronization program, that can sync an Azure File network share with a local folder, by using the Azure Files REST API that is part of the Nuget package WindowsAzure.Storage 4.0.1.

My solution is a simple console program that maps a local folder to a Azure File share. Every 10 sec the program polls the Azure File storage via the REST API to see if any new or updated files/folders have appeared in Azure Files, and if so it downloads them. Also if a file or directory is added to the local folder a FileSystemWatcher event will trigger and the file will be uploaded to Azure Files.

The code is at this moment very much proof of concept, as it is still feature incomplete and have a complete lack of error handling. The fact that this is my first F# program will probably be reflected in some strange use of the language, but please bear with me (I know I have mutable in there, but I just wanted to see it working).

If you after this warning still want to take a look at my code, then take a look here: https://github.com/sjkp/AzureFileSync