Rescuing Your REST API Data With Azure Logic Apps
Its Codeless, Its Serverless and It Scales Automatically
Did you ever realize how REST APIs cant really be relied on to retain all your data?
Inherent in API calls lies the risk of the underlying application or logic failing to persist your data, happily discarding it with a “Bad Payload” or even an “OK” reply.
Unless you already have a reliable mechanism for retaining the data in those failed calls you may never see it again. Are you losing those important customer orders?
A common approach is to introduce a service bus to retain the messages, but that also means they become largely inaccessible unless you use special tools to retreive them.
Service Buses also dont retain your data for very long, usually discarding them after only two weeks. Is anyone minding them or do they just go poof?
In this article I will show you a real-world example of how to use Logic Apps and Azure Data Lake to retain all those messages permanently.
All the messages will be stored as text files with a json extension which means they can be handled with ease in the same way as plain files.
Lets go about securing our data.
Azure Data Lake can be thought of as a great big filesystem which will save your data in 3 locations simultaneously. For most intents and purposes it eliminates any risk of data loss out of the box. Besides that its rather cheap. Lets use that.
Creating the Data Lake to Hold the Messages
First we need to create our data lake. Here is a Microsoft video showcasing just that. Go ahead and follow it to set up your data lake.
Create a New Container and Name It “Apimessages”
In your data lake you need a folder to be the root folder for all messages.
Under this folder all the APIs you choose to connect will create their own subfolders to store their messages. Like so: apimessages/orders/createorder
Creating the Logic App
Add a HTTP step to your Logic App and give it a placeholder Schema like below. The HTTP POST URL you will use later to put in the API Policy to tell the API manager where to send the messages.
Add an “Initialize Variable” step to hold the filename we wish to use for the saved message.
Make sure to type the formula: concat(base64(triggerBody()),'.json')
in the expression field when you enter the Value for the filename variable.
We are using the body of the message as filename encoded in the Base64 format.
This will ensure the message is named after its body content, and will make it unique. (Same message stored only once, or overwritten with the same name)
Add a step to check how long the file is and shorten the filename to 600 characters if needed. (max length is 1024 characters, but long filenames may cause problems when deleting)
In the expression field enter this formula: length(base64(triggerBody()))
If the filename is longer than 600 characters we will shorten it, otherwise we will let it be. In the left (True) field lets add a “Set variable” action to shorten the filename.
Set the expression in the Value field to: concat(substring(base64(triggerBody()),0,600),’.json’)
This shortens the filename to 600 characters and adds the .json suffix.
The first 3 steps of the Logic App are now finished and your App should look like this:
Now its time to add actions to save the message:
Add a “Create Blob” activity to your App. Enter the same values as below.
The blobpath variable is a header sent by the API-policy we are adding to the API in Azure API management.
Adding the folder path can be a bit tricky. Add the following formula in the
expression field that pops up when you click the folder path textbox: /apimessages/triggerOutputs()[‘headers’][‘blobpath’]
Our Logic App is now complete and it should look like this:
Connecting Your API to the Message Store Logic App
In order to capture the messages coming in we will use API Manager Policies. This policy will send the messages to our Logic App every time a message is received on the API.
Be sure to put it between the inbound tags as shown here
<inbound>
<base />
<send-one-way-request mode=”new”>
<set-url>@(“https://the address found in your logic app http call window</set-url>
<set-method>POST</set-method>
<set-header name=”blobpath” exists-action=”override”><value>orderservice/neworder</value>
</set-header>
<set-body>@(context.Request.Body.As<string>(preserveContent: true))</set-body>
</send-one-way-request>
<set-variable name=”bodystored” value=”@(context.Request.Body.As<string>(preserveContent: true))” /></inbound>
Message Secure
Sit back and watch your data being safely stored. The API policy will create subfolders for each api and save the messages inside.
Stay tuned for part 2: “Keeping Track of Those Failed API Messages”
We will see how to easily keep track of failed versus successful messages so that all failed messages are put in one folder for review and reprocessing.
