Data load performance
Data load performance is the time taken to persist a loaded dataset to storage measured from the start of the 'process data import' operation to when its successfult completion is reported. An excerpt from a FusionRegistry.log logfile illustrates the princple:
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Process data import
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - acquire lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - lock acquired on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.672 INFO Thread-8 com.metadatatechnology.free.manager.data.publish.impl.DataSourceManagerImpl - Provision 'urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=ECB:EXR_MT_MT(1.0)' links to data source with Id 'IN_MEMORY'
2021-08-16 15:39:56.635 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 1.0000107% complete
2021-08-16 15:39:56.636 INFO Thread-10374 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 1%
2021-08-16 15:39:56.734 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 2.0000215% complete
... Import process progress
2021-08-16 15:40:07.180 INFO Thread-10568 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 98%
2021-08-16 15:40:07.250 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 99.00107% complete
2021-08-16 15:40:07.251 INFO Thread-10570 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 99%
2021-08-16 15:40:07.318 INFO Thread-10572 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 100%
... Build constraints and other post import tasks
2021-08-16 15:40:07.358 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - release lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:40:07.361 INFO Thread-10574 com.metadatatechnology.free.manager.data.publish.impl.MatrixDataImportManagerImpl - Data Imported Successfully
Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)
3633 series
1,207,654 observations
Datastore: Registry Managed Database
| Registry |
Database |
Registry environment |
Datastore database environment |
Load time
|
| 10.7.4 |
MySQL 8.0 |
Linux, Core i5, 16 GB, SATA HDD 7200rpm storage |
(same platform as the Registry) |
424 seconds
|
| 10.7.4 |
MySQL 8.0 |
Linux, Core i5, 16 GB, SATA SSD storage |
(same platform as the Registry) |
64 seconds
|
| 10.7.4 |
MySQL 5.7 |
Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage |
(same platform as the Registry) |
23 seconds
|
| 10.7.4 |
MySQL 5.7 |
Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage |
(same platform as the Registry) |
(tbc)
|
Datastore: Fusion Datastore (in memory datastore persisted to the operating database)
| Registry |
Database |
Registry environment |
Datastore database environment |
Load time
|
| 10.7.4 |
MySQL 8.0 |
Linux, Core i5, 16 GB, SATA SSD storage |
(same platform as the Registry) |
11 seconds
|
| 10.7.4 |
MySQL 5.7 |
Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage |
(same platform as the Registry) |
6 seconds
|
| 10.7.4 |
MySQL 5.7 |
Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage |
(same platform as the Registry) |
(tbc)
|
