GURURAJAN's Blog

#AWS - Increase your Quicksight SPICE data refresh frequency.

#aws_tips_and_tricks

GURURAJAN's photo
GURURAJAN
·

2 min read

Table of contents

AWS_Quicksight.png

Scenario:

Let us say I want to fetch the data from the source (Jira) and push it to SPICE and render it in Quicksight Dashboards.

Requirement: Push the data every 30 Mins once.

Quicksight supports the following:

  • Full refresh
  • Incremental refresh

Full refresh:

  • Process - Old data is replaced with new data.
  • Frequency - Every 1 Hr once
  • Refresh count - 24 / Day

Incremental refresh:

  • Process - New data get appended to the dataset.
  • Frequency - Every 15 Min once
  • Refresh count - 96 / Day

Issue:

  • We need to push the data every 30 Min once.
  • It is going to be a FULL_REFRESH
  • When it comes to Full Refresh Quicksight only supports Hourly refresh.

Solution:

We can leverage API support from AWS.

  • Package - Python Boto 3
  • Class - Quicksight.client
  • Method - create_ingestion
  • Process - You can manually refresh datasets by starting new SPICE ingestion.
  • Refresh cycle: Each 24-hour period is measured starting 24 hours before the current date and time.

Limitations:

  • Enterprise edition accounts 32 times in a 24-hour period.
  • Standard edition accounts 8 times in a 24-hour period.

Sample code:

Python - Boto for AWS:

import boto3
client = boto3.client('quicksight')

response = client.create_ingestion(
    DataSetId='string',
    IngestionId='string',
    AwsAccountId='string',
    IngestionType='INCREMENTAL_REFRESH'|'FULL_REFRESH'
)

awswrangler:

import awswrangler as wr
wr.quicksight.cancel_ingestion(ingestion_id="jira_data_sample_refresh", dataset_name="jira_db")

CLI:

aws quicksight create-ingestion --data-set-id dataSetId --ingestion-id jira_data_sample_ingestion --aws-account-id AwsAccountId --region us-east-1

API: 

PUT /accounts/AwsAccountId/data-sets/DataSetId/ingestions/IngestionId HTTP/1.1
Content-type: application/json

{
   "IngestionType": "string"
}

Conclusion:

Using this approach we can achieve 56 Full Refreshes for our dataset also we can go one step further and get the peak hours of our source tool (Jira) and configure the data refresh accordingly. This way we can even achieve a refresh frequency of 10 Min once.

Ref:

aws_tips_and_tricks