Incremental data strategies

Incremental data strategies refer to approaches used where only the changes or updates to existing data are processed and incorporated, rather than reprocessing the entire dataset from scratch.

OpenAPI specification offers extensions that extend the capabilities of the OpenAPI Specification (OAS) beyond its core functionality.

• These extensions are used to provide more detailed documentation, specify custom behavior, or include metadata that is not covered by the standard specification.

• Extensions are denoted by keys prefixed with x-adeptia in the OpenAPI document.

 Adeptia introduces the following extension to define the Incremental data strategies in the standard Open API specification.

Supported incremental data strategies

Adeptia supports the following incremental data strategies:

1. ModifiedSince:

   • The ModifiedSince approach, commonly known as conditional requests, is a technique implemented in REST APIs for efficiently verifying if a resource has been altered since a specified timestamp.

   • This method aids in minimizing redundant data transmission and enhancing overall performance. It typically employs the If-Modified-Since header within HTTP requests.

2. FromTo:

   • A timestamp-based strategy, known as the FromTo approach, is employed to monitor modifications by utilizing timestamps to record the latest modification time of a resource.

   • This method bears resemblance to the ModifiedSince strategy; however, rather than depending on HTTP headers like Last-Modified and If-Modified-Since, clients and servers directly exchange timestamps through API-specific Headers or Query Parameters.

3. QueryExpression:

   • The QueryExpression strategy for monitoring modifications entails utilizing SQL-like queries to articulate conditions concerning modifications.

   • With this approach, clients can transmit tailored criteria to retrieve modified resources based on specific conditions or expressions.

4. Cursor (or ChangeId):

   • The Cursor or ChangeId strategy for monitoring modifications entails employing a unique identifier linked to a particular version or alteration of a resource.

   • Clients utilize this identifier to ascertain whether the resource has been modified since a specific change, by incorporating the Cursors or ChangeId in their requests.

Defining ModifiedSince strategy

Field name	Description	YAML
name	The name of the parameter to be injected into the request.	<ch:codesample> </ch:codesample>
injectAs	Specifies the method for injecting the parameter into the request.	Possible Values are either "header" or "query". <ch:codesample> </ch:codesample>
sinceHeader	The name of the HTTP header that carries the Last-Modified timestamp is "Last-Modified" from the last response from API.	<ch:codesample></ch:codesample>

Example: ModifiedSince strategy definition for the Xero application

Defining FromTo strategy

Field name	Description	YAML
fromField	The field name containing the previous timestamp value.	<ch:codesample> </ch:codesample>
toField	The field name containing the current timestamp value.	<ch:codesample> </ch:codesample>
injectAs	Specifies the method for injecting the parameter into the request.	Possible Values are either "header" or "query". <ch:codesample> </ch:codesample>

Example: FromTo strategy definition for the Shopify application

Here {trigger.cursorField} refers to a placeholder for a field or a value associated with the trigger (in this case cursorField value defined in the trigger) that is dynamically determined or provided at runtime. Here the cursorField represents a point in time or a unique identifier used to track the last processed record or event. For more details, refer to the Triggers page.

Defining QueryExpression strategy

Field	Description	YAML
name	The name of the parameter to be injected into the request.	<ch:codesample> </ch:codesample>
injectAs	Specifies the method for injecting the parameter into the request.	Possible Values "header" and "query". <ch:codesample> </ch:codesample>
append	This indicates whether the value should be added to the existing parameter value or replace it entirely.	Possible Values true false <ch:codesample> </ch:codesample>
valueExpression	The expression for appending a value to an existing value typically involves concatenating the new value with the existing value using a predefined separator.	<ch:codesample> </ch:codesample>

Example: QueryExpression Strategy definition for QuickBooks application

Here is an explanation of the different placeholders being used here: {trigger.cursorField} refers to a placeholder for a field or a value associated with the trigger that is dynamically determined or provided at runtime. Here the cursorField represents a point in time or a unique identifier used to track the last processed record or event. Please refer to Triggers | Extension: Define Trigger for more details. {event.previousTimeStamp} refers to a placeholder for the timestamp associated with the previous occurrence of the event. It is substituted with actual values or timestamps when the event is processed within the application. {event.currentTimeStamp} refers to a placeholder for the timestamp associated with the current occurrence of the event. It would be replaced with the timestamp of the current event when processed within the application.

Defining Cursor Strategy

Field	Description	YAML
operationId	The operation ID for the endpoint intended to retrieve the Cursor.	<ch:codesample> </ch:codesample>
cursorPath	The JSON path for the Cursor value in the response body depends on the structure of the JSON data. If the Cursor value is located within an object named "cursor" at the top level of the response body, the JSON path would be simply "$.cursor". However, if it's nested within other objects or arrays, the path would reflect that structure. For example, if the Cursor value is nested within an object named "data" within the response body, the JSON path would be "$.data.cursor". It's crucial to inspect the actual JSON response structure to determine the accurate JSON path for retrieving the Cursor value.	<ch:codesample> </ch:codesample>
pagination	The pagination strategy to retrieve the latest cursor from the last page. In this approach, the client requests the last page of results and retrieves the latest cursor from the last page. For more details, please refer to Pagination Strategies	<ch:codesample> </ch:codesample>

Example: Cursor Strategy definition for Dropbox application