Azure Function for Metadata

Decube can leverage a customer-hosted Azure Function to retrieve file schemas, minimizing file egress from the customer's environment to Decube's compute. While this approach is generally applicable to the SaaS deployment model, it can also be beneficial for self-hosted solutions where the Data Plane resides in a different region than the Storage Container.

Prerequisite

• Create an Azure Function with Custom Handler as the Runtime Stack

• Networking between Decube Data Plane (either SaaS or customer own) to the Azure Function

Creating Azure Function

1. Go to your Azure Portal and navigate to Function App and click on `Create`

1. Choose either Consumption (recommended), Functions Premium or App Service

1. Fill in the appropriate field

   1. Function App Name - a unique name for this function

   2. Runtime stack - Custom Handler

   3. Region - Closest to your ADLS storage account region

   4. Operating System - Linux

1. Under Networking, choose Enable Public Access - On

   1. See Azure documentation if alternate networking is required

1. Under Deployment, disable Continous Deployment

2. Storage, Monitoring, Tags are to be set up according to customer need

3. Create the Azure Function

Deployment

Visit https://github.com/DecubeIO/adls-azure-function to start

1. Clone the repo to a local machine

2. Run func azure functionapp publish $functionAppName --custom

   1. $functionAppNameis based on the Azure Function name used in Creating Azure Function

Using the Azure Function

1. When creating/modifying an ADLS source, enable Use remote Azure Function

2. Fill in the Azure Function URL in this format: https://myfunction.azurewebsites.net

3. Fill in the Azure Function Key. This can be found in your Portal Azure > Go to created function above > Functions > App Keys > Either choose an existing host key or create a new host key