Benchmarking
Adeptia conducts extensive performance testing with each release to ensure improving and better performance of its product. Some performance benchmarks are provided here for reference:
- Test A - Large Number of Small EDI Files
- Test B - Large Sized EDI Files
- Test C - Large File Processing with Large Data Ingestion
- Test D - UI Load Testing with Concurrent Users
- Test E - Advanced Positional File
- Test F - API Load Testing Results
- Test G - Schema
- Test H - Miscellaneous
Test A - Large Number of Small EDI Files
System Configuration
- Cluster = 2 nodes (Kernel + WebRunner)
- Processor = Intel Xeon CPU E5-2686 2.30GHz
- Number Of Cores = 4
- Total RAM = 16GB
- OS = Windows Server 2012 R2 64-bit
- Backend / Log Database - AWS SQL RDS
- Kernel = 8GB
Result - All tests ran successfully
| Scenario | File Size x # of Files | Total Time | Throughput |
|---|---|---|---|
| Inbound File Type – EDI 4010 Orders Mapping – Medium Complex # of ST = 1 | 2KB x 3000 | 13m | 4 files/sec |
| Inbound File Type – EDI 4010 Orders Mapping – Medium Complex # of ST = 1 | 2KB x 6000 | 26m | 4 files/sec |
| Inbound File Type – EDI 4010 Orders Mapping – Medium Complex # of ST = 1 | 2KB x 20,000 | 1h 46m | 3.14 files/sec |
| Inbound File Type – EDI 4010 Orders Mapping – Medium Complex # of ST = 44 | 37KB x 1000 | 1h 12m | 4 files/sec |
| Outbound EDI from IDOC File Type – EDI 4010 Invoice Mapping – Medium Complex | 11KB x 3000 | 12m 08s | 4.12 files/sec |
Test B - Large Sized EDI Files
System Configuration
- Single node (no cluster)
- Processor = Intel Xeon CPU E5-2686 2.30GHz
- Number Of Cores = 4
- Total RAM = 16GB
- OS = Windows Server 2012 R2 64-bit
- Backend / Log Database - AWS Oracle RDS
- Kernel = 8GB
Result: All tests ran successfully
| Scenario | File Size x # of Files | Max CPU % | Max RAM % | Max. Kernel Heap | Total Time |
|---|---|---|---|---|---|
| Inbound File Type – EDI 4010 ISA : 1 GS : 1 Transaction Set : 5000 ~ 25-35 segments in each transaction set | 2.79MB x 1 | 42% | 38% | 1.1GB | 34s |
| Inbound File Type – EDI 4010 ISA : 1 GS : 13 Transaction Set : 1 ST in each GS ~ 140000 segments in each transaction set | 40.1MB x 1 | 88% | 46% | 2.3GB | 5m 22s |
| Inbound File Type – EDI 4010 ISA : 1 GS : 1 Transaction Set : 10000 ~ 50-103 segments in each Transaction set | 13.6MB x 1 | 72% | 43% | 2.2GB | 1m 31s |
Test C - Large File Processing with Large Data Ingestion
System Configuration
- Single node (no cluster)
- Processor = Intel Xeon CPU E5-2686 2.30GHz
- Number Of Cores = 4
- Total RAM = 16GB
- OS = Windows Server 2012 R2 64-bit
- Backend / Log Database - AWS Oracle RDS
- Kernel = 8GB
Result: All test ran successfully
| Scenario | File Size x # of Files | Number of Records per file | Max CPU % | Max RAM % | Time |
|---|---|---|---|---|---|
| File Type – XML Mapping – Simple 1-to-1 # of Files = 1 | 25GB x 1 | 696895 | 27% | 34% | 23m 48s |
| 50GB x 1 | 1393690 | 27% | 34% | 47m 30s | |
| 200GB x 1 | 5574760 | 27% | 34% | 3h 10m | |
| 500GB x 1 | 13936900 | 27% | 34% | 7h 52m | |
| File Type – XML Mapping – Simple 1-to-1 # of Files = 50 parallel | 25GB x 50 | 696895 | 100% | 37% | 10h 7m (total) 8h 51m (avg/file) |
| 50GB x 50 | 1393690 | 100% | 37% | 19h 45m (total) 17h 13m (avg/file) | |
| File Type – XML Mapping – Complex (Functions used: Math, String, DB Look, If Condition) # of Files = 1 | 25GB x 1 | 696895 | 29% | 36% | 33m 30s |
| 50GB x 1 | 1393690 | 29% | 36% | 1h 7m | |
| 200GB x 1 | 5574760 | 29% | 36% | 4h 27m | |
| 500GB x 1 | 13936900 | 29% | 36% | 11h 13m | |
| File Type – XML Mapping – Complex # of Files = 50 parallel | 25GB x 50 | 696895 | 100% | 39% | 9h 59m (total) 8h 22m (avg/file) |
| 50GB x 50 | 1393690 | 100% | 39% | 20h 2m (total) 15h 53m (avg/file) | |
| File Type – TEXT Mapping – Simple 1-to-1 # of Files = 1 | 25GB x 1 | 696895 | 45% | 36% | 6h 26m |
| 50GB x 1 | 1393690 | 45% | 36% | 12h 40m | |
| 200GB x 1 | 5574760 | 45% | 36% | 51h 30m |
Test D - UI Load Testing with Concurrent Users
System Configuration
- Single node (no cluster)
- Processor = Intel Xeon CPU E5-2686 2.30GHz
- Number Of Cores = 4
- Total RAM = 16GB
- OS = Linux 64-bit
- Backend / Log Database - AWS Oracle RDS
- Kernel = 8GB
- WebRunner = 4GB
- Tool used = JMeter
Result: All tests ran successfully
| Scenario | Home Page | Dashboard | Network | Partners | Transactions | Max CPU % | Max RAM % |
|---|---|---|---|---|---|---|---|
| # of Partners and Transactions in Back-end DB = 1138 # of Concurrent Users = 25 # of Loops (iterations) = 50 | 4.2s | 1.1s | 0.8s | 0.4s | 0.6s | 53% | 64% |
| # of Partners and Transactions in Back-end DB = 1638 # of Concurrent Users = 25 # of Loops (iterations) = 50 | 4.3s | 1.0s | 0.9s | 0.3s | 0.4s | 49% | 59% |
| # of Partners and Transactions in Back-end DB = 2200 # of Concurrent Users = 25 # of Loops (iterations) = 50 | 2.3s | Overview page : 4.3s | 9.0s | 0.7s | 2.7s | 64% | 59% |
| Transaction page: 0.4s | |||||||
| Trigger page: 0.5s |
Test E - Advanced Positional File
System Configuration
- Single node (no cluster)
- Processor = Intel Xeon CPU E5-2686 2.30GHz
- Number Of Cores = 4
- Total RAM =64GB
- OS = Windows Server 2012 R2 64-bit
- Backend / Log Database - AWS Oracle RDS
- Kernel = 20GB (minimum), 40GB (maximum)
Result - All tests ran successfully
| Scenario | Number of Records per file | File Size x # of Files | Total Time |
|---|---|---|---|
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 3 million | 234 Mb x 1 | 45m 39s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 5 million | 391 Mb x 1 | 1h 19m 31s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 10 million | 782 Mb x 1 | 2h 31m 49s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 10 million | 782 Mb x 3 | 2h 13m 4s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 10 million | 782 Mb x 5 | 2h 12m 37s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 10 million | 782 Mb x 10 | 2h 42m 45s |
| File Type – Text to Advanced Positional Mapping – Simple 1-to-1 | 10 million | 782 Mb x 15 | 3h 56m 36s |
Test F - API Load Testing Results
The API load testing results are documented in Test_Result_WebService_LoadTesting_V3.5.pdf.
Test G - Schema
| Sl. No. | Test Case Description | Data Set (File Size) | Execution Time |
|---|---|---|---|
| 1 | To test processing of simple Excel file (xls) through File Event, File Source and File target Performance_Process_flow_Excel | 100 MB | 1 min 28 sec Kernel CPU: 25% WebRunner CPU: 0.00% Kernel Mem: 5.5GB WebRunner Mem: 854MB |
| 2 | To test processing of simple Positional file through File Event, File Source and File target Performance_Process_flow_Positional | 100 MB | 59 sec Kernel CPU: 27% WebRunner CPU: 0.00% Kernel Mem: 7.21GB WebRunner Mem: 855MB |
| 3 | To test processing of simple JSON file through File Event, File Source and File target Performance_Process_flow_JSON | 100 MB | 25 sec Kernel CPU: 3% WebRunner CPU: 0.03% Kernel Mem: 8.7GB WebRunner Mem: 855MB |
| 4 | To test processing of simple Adv Positional file through File Event, File Source and File target Performance_Process_flow_Adv_Positional | 100 MB | 1 min 15 sec Kernel CPU: 29% WebRunner CPU: 0.00% Kernel Mem: 8.2GB WebRunner Mem: 830MB |
| 5 | To test processing of simple Text file through File Event, File Source and File target Performance_Process_flow_Text | 100 MB | 2 min 00 sec Kernel CPU: 28.5% WebRunner CPU: 0.00% Kernel Mem: 8.5GB WebRunner Mem: 855MB |
| 6 | To test processing of simple Adv. Text file through File Event, File Source and File target Performance_Process_flow_adv_text | 100 MB | 1 min 49 sec Kernel CPU: 25% WebRunner CPU: 0.00% Kernel Mem: 8.6GB WebRunner Mem: 856MB |
| 7 | To test processing of simple xml file through File Event, File Source and File target Performance_Process_flow_Xml | 100 MB | 7 sec Kernel CPU: 19% WebRunner CPU: 0.3% Kernel Mem: 8.85GB WebRunner Mem: 855MB |
| 8 | To test Database Event along with Adv Database Source when the number of records to pick up are large in the table. Use one to map to write data to Adv DB target with different batch/commit size options. Performance_Process_flow_Adv_Database | 50000 records to pickup Batch size=5000 Commit Size=2500 | 1 min 36 sec Kernel CPU: 19.8% WebRunner CPU: 0.00% Kernel Mem: 3.56GB WebRunner Mem: 841MB |
| 9 | To test Database Event along with Adv Database Source when the number of records to pick up are large in the table. Use one to map to write data to Adv DB target with different batch/commit size options. Performance_Process_flow_Adv_Database | 50000 records to pickup Batch size=5000 Commit Size=5000 | 1 min 40 sec Kernel CPU: 20.2% WebRunner CPU: 0.00% Kernel Mem: 3.34GB WebRunner Mem: 839MB |
Test H - Miscellaneous
| Sl. No. | Test Case Description | Data Set (File Size) | Execution Time |
|---|---|---|---|
| 1 | To test concurrent processing of flat files of different sizes with moderate mapping functions and one to one conversion. | File Type: CSV File Size: 1MB Concurrent Files: 500 | 2 min 54 sec Kernel CPU: 5.20% WebRunner CPU: 0.00% Kernel Mem: 3.33GB WebRunner Mem: 824MB |
| 2 | To test concurrent processing of flat files of different sizes with moderate mapping functions and one to one conversion. | File Type: CSV File Size: 1MB Concurrent Files: 1000 | 9 min 32 sec Kernel CPU: 5.82% WebRunner CPU: 0.00% Kernel Mem: 3.34GB WebRunner Mem: 834MB |
| 3 | To test concurrent processing of flat files of different sizes with moderate mapping functions and one to one conversion. | File Type: CSV File Size: 100KB Concurrent Files: 5000 | |
| 4 | To test processing of flat files located at File through non-streaming Mapping with pre-built Mapping functions applied (string, conditional, date, math) and generate the Target file. | 100MB Plain CSV Text file Non-streaming Mapping mode | 1 min 40 sec Kernel CPU: 2.5% WebRunner CPU: 0.00% Kernel Mem: 3.37GB WebRunner Mem: 836MB |