What is Incremental modeling?

Incremental modeling is the process of importing and running only the new rows of data based on the conditions criteria. The new rows are appended in your already imported/created dataset.

Y42 provides you the ability to append new data in your pipeline in a faster and cheaper way continuously.

Why use Incremental modeling?

Let's suppose you have millions of records in your Paypal integration. You use a SQL model to model those raw data inputs and create your desired report.

Every day new records are added to your Paypal records, so you build an Orchestration and schedule it to run every day, to generate those new records in the Paypal import and then to run the query of the SQL model.

Doing this process costs because you are writing the data in your warehouse and the process takes time also.

Here is when Incremental models comes in the rescue.

By putting a condition like for example created_at > max(created_at) you will run the query only for the rows that satisfy the matching criteria and then append the new rows to the old dataset of the model.

To start using incremental step by step visit this article.