Adding Queue Message When a File is Uploaded to Azure Storage

/ Sudharsan K...

Introduction

Hi friends, this post is the continuation of How to Transfer Files from Local system Folder to Azure Storage via SFTP. In our previous post, we saw how we can transfer files from local to Azure Storage via SFTP. But, the files are not gonna stay their for a while, we have to transfer to SharePoint. For that we are going to use Azure Queue.

Why Queue?

For transferring files from Azure Storage, my preferred way to do is to design a Azure Function with blob trigger. When we use the blob trigger, it will trigger a function whenever a file is uploaded to the configured Azure Storage. If the files are uploaded on timely manner like 1 file at a time and the next file will take time to upload, then we can write the code to get the file and upload to SharePoint inside the blob trigger itself. But in our case, we are using console and we are gonna upload lot of files within seconds or minutes. If we follow the above route, then we may hit with locks and other issues with SharePoint.

To avoid the issues we need to use Azure Queue. We add the queue message with just the filename whenever the file is uploaded to Azure Storage. Since this is a very minimal function we can execute it on the blob trigger itself which won’t cause any issues. Later on in the next post we will see how we can use the queue message to upload the files from Azure Storage to SharePoint document library.

Below is the steps to configure Azure Queue and to create Azure Function.

Azure Queue creation

Navigate to the Azure Storage we created in the previous post. Navigate to the Queues from the left navigation and create a queue as shown in the below screenshot.

Once the queue is created, navigate to the Access Keys section from the left navigation in the same Azure Storage. You should see 2 set of keys and make sure you make a note of the Connection string from either. We will be using this connection string in our Azure Function code to connect to the Storage and the Queues. Please refer the screenshot below

Create Azure Function using Visual Studio

We are going to create a Azure Functions using Visual Studio. I am using VS 2022 version and you can also use the same or 2019 version. I also used C# code. Please refer the screenshot below.

Give a name to the project and choose the location to store the Azure Functions project.

Choose the information like the screenshot shown below in the next step.

Once the project is created, right click on the project and add new item and choose Azure Function and give a name. Please refer the screenshot below

On the next screen, choose the Blob trigger as shown below.

Below is the default function created when the trigger is created.

Create settings named ConnString and use the Connection string values copied from the Access Keys section.

Now we need to change the existing code to match our Storage and Queues. Below are the changes that needs to be done. Please refer the screenshot below.

  • Change the default function name
  • Update the trigger settings. Change the storage name and add the Connection string variable.
  • Add the Queue connection string and the name
  • Add the filename as a message to the queue.

Press F5 to run the function app. You should be able to see the below screen. The HttpTrigger is just a sample trigger that was created by default. You should be able to see the blob trigger.

using System;
using System.IO;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Microsoft.Extensions.Logging;

namespace AzureToSharePoint
{
    public class BlobFunction
    {
        [FunctionName("AddQueueMessageForFiles")]
        public void Run([BlobTrigger("testcontainer/{name}", Connection = "ConnString")] Stream myBlob, string name, ILogger log,
            [Queue("testqueues"), StorageAccount("ConnString")] ICollector<string> queueMsg)
        {
            log.LogInformation($"C# Blob trigger function Processed blob\n Name:'{name}' \n Size: {myBlob.Length} Bytes");
            queueMsg.Add(name);
        }
    }
}

Now the function is locally running. Now execute the console application to copy the files from local to Azure Storage. Once the files are uploaded to the storage, you should see the blob triggered the function and the queues are added.

Conclussion

I hope you had learned something about the queues and how to trigger the blob whenever the files are uploaded to the blob and then add to the queue message. In the next post we will see how to use the queue to transfer the files from Azure Storage to Azure Queues.

Happy Coding…

Leave a comment