Data archival and cleanup
The Adeptia platform automatically manages the lifecycle of runtime data generated by your integrations, archiving older records to a separate store and purging data that has exceeded its configured retention period. This keeps your operational database lean and performant while preserving historical information for audit, troubleshooting, and compliance purposes.
What this capability does
As your integrations run, the platform accumulates execution logs, transaction records, B2B EDI interchange data, SAP IDoc records, Business Rules execution results, migration logs, and related repository files. Without lifecycle management, this data grows unboundedly and degrades system performance. The archival and cleanup service addresses this by:
- Archiving records that have aged past a configurable threshold, copying them to a dedicated archive store before removing them from the live database.
- Cleaning up records and associated files that have exceeded their retention period, freeing storage in both the database and the file system.
- Pruning orphaned directories left behind by aborted or incomplete executions, so the file system stays tidy even when no database row owns a given folder.
When archival is enabled, the service copies data to archive tables before deleting it from the live tables, so no historical record is lost. When archival is not enabled, the service deletes aged records directly from the live tables.
Data covered by archival and cleanup
The service manages retention across several categories of runtime data:
| Data category | Description |
|---|---|
| Transaction logs and data | Execution-level log entries and activity data produced by process flows |
| Transaction execution logs | Summary records of each integration execution, including source, destination, status, and timing |
| B2B EDI data | Interchange, group, transaction, and acknowledgment records for X12 and EDIFACT messages |
| SAP IDoc data | SAP TID and IDoc records generated during SAP integrations |
| Business Rules execution data | Execution summaries, individual rule results, and associated result files written to shared storage |
| Audit trail logs | Audit entries recording platform activity |
| Event logs | Platform event records |
| Migration logs | Records produced during data migration operations |
| IDP logs | Log entries associated with IDP processing |
| Usage information | Platform usage tracking records |
| Repository files | Files written to the integration repository during execution, including EDI files, webservice attachments, DI repository files, and reprocessing recovery files |
How archival jobs work
The service runs three distinct job modes, each suited to a different operational need:
| Job mode | What it does |
|---|---|
| Log cleanup with archival | Copies aged live records to the archive store, then deletes them from the live tables. Also moves associated repository files to the archive location. |
| Archive data cleanup | Removes records from the archive store that have exceeded the archive retention period. |
| Force cleanup | Deletes aged records directly from the live tables and repository, bypassing archival. Used when you need to reclaim space immediately. |
Each job processes data in configurable chunks and can be tuned for throughput and database load. The service uses retry logic for transient database errors and can skip records that match configurable error codes, so a single problematic record does not block the entire cleanup run.
For Business Rules result files stored on shared storage, the service copies each file to the archive location and verifies the copy before deleting the source. If a file is already present and verified in the archive, the copy step is skipped, making the operation safe to re-run after an interruption. After file-level cleanup, the service also removes any empty directories left behind under the results storage area.
Retention periods you can configure
You can set independent retention periods for different categories of data, giving you fine-grained control over how long each type of record is kept in the live database and in the archive store.
| Retention setting | What it controls |
|---|---|
| General log retention | How many days live log records are kept before archival or deletion |
| Archive log retention | How many days archived records are kept before being purged from the archive store |
| Business Rules log retention | Retention period for Business Rules execution logs in the live store (falls back to the general log retention when not set) |
| Business Rules archive retention | Retention period for Business Rules records in the archive store (falls back to the general archive retention when not set) |
| B2B log retention | Retention period for B2B EDI records |
| Migration log retention | Retention period for migration log records |
| IDP log retention | Retention period for IDP log records |
| Usage information retention | Retention period for usage tracking records |
Email notification on failure
The service can send an email notification to the system administrator when an archival job encounters a failure. This requires a mail server to be configured in the platform. Notification behavior is configurable and can be enabled or disabled independently of the archival jobs themselves.
Why this matters for your integrations
Keeping runtime data under control directly affects the performance and reliability of your integrations. A growing log database slows down queries used by monitoring, reporting, and reprocessing features. Regular archival and cleanup ensures that:
- The operational database remains fast and responsive.
- Historical records remain available in the archive store for audit and troubleshooting.
- File system storage used by integration repositories does not grow without bound.
- Business Rules result files are preserved in a verified archive copy before the live copy is removed.
By configuring retention periods appropriate to your compliance and operational requirements, you can balance storage costs against the need to retain historical data.