How to Transfer Files from Local system Folder to Azure Storage via SFTP

Introduction

Hi friends, in this post I am going to show you how you can securely trasfer files from your local filesystem to Azure Storage using SFTP protocol. This is a multipart series, in future we are going to extend this solution by adding queues and transfer the files from Azure Storage to SharePoint document library using PnPCore SDK.

Creation of Azure Storage Account

For creating Storage Accounts, you need to have a proper Azure Subscription. You can follow the below steps to create the Storage Accounts with SFTP enabled.

On the Basics tab screen, you should be able to configure the below
- Subscription – Choose the right subscription if you have multiple
- Resource group – You can create a new resource group, if you are testing it. Later you can delete the resource group which will delete all the services associated with the resource group
- Storage account name – make sure you note down the name and all the letters should be in lowercase.
- Region – Choose a region where would you like to host the storage.
- Performance – I chose Standard since this is the demo.
- Redundancy – I chose LRS and you can choose based on your requirement

On the Advanced tab, below are the settings that is important for us and the rest are all you can leave it to the default selection. Below are to be done at the time of Storage account creation. Few cannot be modified later like SFTP settings.
- Enable hierarchical Namespace – This is mandatory for enabling SFTP protocol to the storage account
- Enable SFTP – Enable so that we securely transfer the files
- Access Tier – I chose Hot since we will access the storage frequently

On the Networking tab, make sure you enable the public access so that you can access the storage account from any IP else you should be in a specific IP range to access the storage.

The rest of the settings are based on own requirements and once done, you can review the settings and create.
Once the storage account is created, there are few settings we need to verify or update before creating the container.
Go to the Networking menu on the left nav inside your Storage account and make sure the settings are configured correctly like the screenshot below

Go to the Configuration menu in the Storage account and make sure the settings are same like the screenshot below

Now let us create a container to store the files. Go to the Containers menu on the left navigation and click on the + Container to create a new container

Once the container is created, you should be able to upload the files directly in Azure.

Now we need to create a local SFTP user and password and map the container permission for us to access and upload the files to the container via Console application.

The below screenshot shows on the permissions assigned to the container. For demo purpose I gave all the permission to the user account which is like a super admin account, but you can restrict based on your requirement.

Once you click on Add, a automatic generated password will be generated and make sure you copy the password because once you close the below window, you won’t be able to retrieve the password again but you can regenerate a new password.

.Net Console App

Create a Console App project using Visual Studio 2019 or 2022 and follow the steps as shown in the below screenshots

Give a project name and browse to the location to save all the project files.

Choose the .Net framework. Here I chose .Net 6.0

Once the project is created and loaded, create a new json file named appsettings.json to store the settings for Azure Storage. You have to make sure the json file is copied to output directory whenever we change or build or publish. Follw the screenshot below

Create a AppSettings section inside the appsettings.json file and update the below mentioned variables.

{
  "exclude": [
    "**/bin",
    "**/bower_components",
    "**/jspm_packages",
    "**/node_modules",
    "**/obj",
    "**/platforms"
  ],
  "AppSettings": {
    "FilePath": "",
    "SFTPConnString": "<resource group name>.blob.core.windows.net",
    "SFTPUsername": "<resource group name>.<container name>.<username>",
    "SFTPPassword": ""
  }
}

FilePath: Location of the files to be imported to Azure Storage
SFTPConnString: <resource group name>.blob.core.windows.net
SFTPUsername: <resource group name>.<container name>.<username>
SFTPPassword: Auto generated password

Now let us add some Nuget packages that are required to get the configuration and connect to Azure Storage. You can install the versions highlighted in the below screenshot or even the latest version.

Microsoft.Extensions.Hosting
Microsoft.Extensions.Configuration
SSH.Net

Next step is to work on the actual code to import the files to Azure Storage.

Let us create a IModel.cs file to create a interface for appsettings. Below are the properties that will be received from the appsettings json file.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

namespace FTLocalToAzure
{
    public class ISettings
    {
        public string FilePath { get; set; }
        public string SFTPConnString { get; set; }
        public string SFTPUsername { get; set; }
        public string SFTPPassword { get; set; }
    }
}

All the codes shown below must be added to Program.cs file.

Lets add the required using statements

using FTLocalToAzure;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Renci.SshNet;

Let us get the configuration from the appsettings.json file. Make sure you name the json file properly and refer the name below.

// To get the configuration from appsettings.json file
var builder = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
IConfiguration config = builder.Build();
var configSettings = config.GetSection("AppSettings").Get<ISettings>();

Configure the default logger to log to console. You can also use Serilog or any other open source logger package to log to the file.

// Configure the logger
using ILoggerFactory factory = LoggerFactory.Create(builder => builder.AddConsole());
ILogger logger = factory.CreateLogger("LocalFilesToAzure");

Create a method to check for settings. To make sure all the settings are given before we execute the main method. In the below method, we are checking all the 4 variables from the appsettings.json file and the main method will execute only when all the 4 values are set.

/* Method to check all the settings are configured or not */
bool checkForSettings()
{
    bool ret = true;
    if (string.IsNullOrEmpty(configSettings?.FilePath)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPConnString)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPUsername)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPPassword)) ret = false;
    return ret;
}

Create a method to upload the files from local to Azure Storage. Below are the steps that are performed in the below method

Passing a list of filenames to process
Checking whether config settings is null or not
Create a SFTP connection using the connection string, username & password
Iterate through the filenames and below steps are executed for each file
Try to read the file as FileStream
Using the UploadFile method from the SFTP client to upload the file to the Azure Storage Container.

/* Method to upload the files to Azure via SFTP */
async void UploadFilesViaSFTP(IEnumerable<string> files)
{
    try
    {
        logger.LogInformation("Started to upload files to Azure.");
        IEnumerator<string> filesEnum = files.GetEnumerator();
        if (configSettings != null)
        {
            using (var sftpClient = new SftpClient(configSettings.SFTPConnString, configSettings.SFTPUsername, configSettings.SFTPPassword))
            {
                sftpClient.Connect();
                while (filesEnum.MoveNext())
                {
                    Console.Write($"Trying to upload the file - '{filesEnum.Current}'");
                    string strFilename = string.Format("{0}", Path.GetFileNameWithoutExtension(filesEnum.Current), Path.GetExtension(filesEnum.Current));
                    using (var uplfileStream = System.IO.File.OpenRead(filesEnum.Current))
                    {
                        if (!sftpClient.IsConnected) sftpClient.Connect();
                        sftpClient.UploadFile(uplfileStream, strFilename, true);
                        Console.WriteLine("\t Uploaded");
                    }
                }
                sftpClient.Disconnect();
                logger.LogInformation("All files uploaded!");
            }
        }
        else
        {
            logger.LogCritical("No config settings found");
        }
    }
    catch (Exception ex)
    {
        logger.LogError(ex.ToString());
    }
}

Create a main method which will be executed when we run the console app. Below are the steps that are performed in the main method

Call the checkForSettings method to check whether the settings are given or not
To make sure the location of the folder configured exists or not
Enumerate the files in the folder
Filter any of the unwanted files and store just the filename in a collection. Here I had filtered out the .exe file.
If the filtered collection count is > 0, then call the UploadFilesViaSFTP passing the filename collection as the parameter.
Call the main method at the end.

/* Main method to execute to start the upload process */
async Task ProcessFilesToUpload()
{
    logger.LogInformation("Starting the upload process.");
    try
    {
        logger.LogInformation("Checking for settings.");
        if(checkForSettings())
        {
            // Getting the configured directory in a string variable
            string strDir = configSettings?.FilePath;
            // Checking if the directory exists or not
            logger.LogInformation("Checking whether directory exists or not.");
            if(Directory.Exists(strDir))
            {
                logger.LogInformation("Enumerating the files.");
                // Enumerating the files and storing in the files object
                var files = from objFile in Directory.EnumerateFiles(strDir) select objFile;
                logger.LogInformation("Filtering the files.");
                // Filtering some of the irrelevant files
                IEnumerable<string> finalFiles = files.Where(f => Path.GetExtension(f).ToLower() != ".exe");
                if (finalFiles.Count() > 0)
                {
                    UploadFilesViaSFTP(finalFiles);
                }
                else logger.LogInformation("No files to upload.");
            }
        } else
        {
            
            logger.LogCritical("Configuration missing!");
        }
    }
    catch (Exception ex)
    {
        logger.LogError(ex.ToString());
    }
    Console.ReadLine();
}

Below is the complete Program.cs file.

using FTLocalToAzure;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Renci.SshNet;

// To get the configuration from appsettings.json file
var builder = new ConfigurationBuilder()
                .SetBasePath(Directory.GetCurrentDirectory())
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
IConfiguration config = builder.Build();
var configSettings = config.GetSection("AppSettings").Get<ISettings>();

// Configure the logger
using ILoggerFactory factory = LoggerFactory.Create(builder => builder.AddConsole());
ILogger logger = factory.CreateLogger("LocalFilesToAzure");

/* Main method to execute to start the upload process */
async Task ProcessFilesToUpload()
{
    logger.LogInformation("Starting the upload process.");
    try
    {
        logger.LogInformation("Checking for settings.");
        if(checkForSettings())
        {
            // Getting the configured directory in a string variable
            string strDir = configSettings?.FilePath;
            // Checking if the directory exists or not
            logger.LogInformation("Checking whether directory exists or not.");
            if(Directory.Exists(strDir))
            {
                logger.LogInformation("Enumerating the files.");
                // Enumerating the files and storing in the files object
                var files = from objFile in Directory.EnumerateFiles(strDir) select objFile;
                logger.LogInformation("Filtering the files.");
                // Filtering some of the irrelevant files
                IEnumerable<string> finalFiles = files.Where(f => Path.GetExtension(f).ToLower() != ".exe");
                if (finalFiles.Count() > 0)
                {
                    UploadFilesViaSFTP(finalFiles);
                }
                else logger.LogInformation("No files to upload.");
            }
        } else
        {
            
            logger.LogCritical("Configuration missing!");
        }
    }
    catch (Exception ex)
    {
        logger.LogError(ex.ToString());
    }
    Console.ReadLine();
}

/* Method to check all the settings are configured or not */
bool checkForSettings()
{
    bool ret = true;
    if (string.IsNullOrEmpty(configSettings?.FilePath)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPConnString)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPUsername)) ret = false;
    else if (string.IsNullOrEmpty(configSettings?.SFTPPassword)) ret = false;
    return ret;
}

/* Method to upload the files to Azure via SFTP */
async void UploadFilesViaSFTP(IEnumerable<string> files)
{
    try
    {
        logger.LogInformation("Started to upload files to Azure.");
        IEnumerator<string> filesEnum = files.GetEnumerator();
        if (configSettings != null)
        {
            using (var sftpClient = new SftpClient(configSettings.SFTPConnString, configSettings.SFTPUsername, configSettings.SFTPPassword))
            {
                sftpClient.Connect();
                while (filesEnum.MoveNext())
                {
                    Console.Write($"Trying to upload the file - '{filesEnum.Current}'");
                    string strFilename = string.Format("{0}", Path.GetFileNameWithoutExtension(filesEnum.Current), Path.GetExtension(filesEnum.Current));
                    using (var uplfileStream = System.IO.File.OpenRead(filesEnum.Current))
                    {
                        if (!sftpClient.IsConnected) sftpClient.Connect();
                        sftpClient.UploadFile(uplfileStream, strFilename, true);
                        Console.WriteLine("\t Uploaded");
                    }
                }
                sftpClient.Disconnect();
                logger.LogInformation("All files uploaded!");
            }
        }
        else
        {
            logger.LogCritical("No config settings found");
        }
    }
    catch (Exception ex)
    {
        logger.LogError(ex.ToString());
    }
}

await ProcessFilesToUpload();

Below are the screenshots based on my execution.

Files to be uploaded to Azure Storage.

Console output on the execution.

Azure Storage container after the files are imported.

Use-Case

The above solution seems to be very simple but it can be widely used in many scenarios or situations. The above solution can be configured in a Scheduler to upload the files daily from a particular folder. In the future article, I am gonna walk you on how we can transfer files to Azure Storage and then from there we are going to upload to SharePoint document library using the Azure Queues.

Reference Link

Github repo

Conclussion

I hope you had learned something new and in future article we will see some different scenarios. I also welcome if you have any ideas or something that need to explore.

Happy Coding…