Performance benchmarks

Data load performance

Data load performance is the time taken to persist a loaded dataset to storage measured from the start of the 'process data import' operation to when its successfult completion is reported. An excerpt from a FusionRegistry.log logfile illustrates the princple:

2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Process data import
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - acquire lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - lock acquired on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.672 INFO Thread-8 com.metadatatechnology.free.manager.data.publish.impl.DataSourceManagerImpl - Provision 'urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=ECB:EXR_MT_MT(1.0)'  links to data source with Id 'IN_MEMORY'
2021-08-16 15:39:56.635 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 1.0000107% complete
2021-08-16 15:39:56.636 INFO Thread-10374 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 1%
2021-08-16 15:39:56.734 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 2.0000215% complete
... Import process progress
2021-08-16 15:40:07.180 INFO Thread-10568 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 98%
2021-08-16 15:40:07.250 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 99.00107% complete
2021-08-16 15:40:07.251 INFO Thread-10570 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 99%
2021-08-16 15:40:07.318 INFO Thread-10572 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 100%
... Build constraints and other post import tasks
2021-08-16 15:40:07.358 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - release lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:40:07.361 INFO Thread-10574 com.metadatatechnology.free.manager.data.publish.impl.MatrixDataImportManagerImpl - Data Imported Successfully

Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)

3633 series
1,207,654 observations

Datastore: Registry Managed Database

Registry	Database	Registry environment	Datastore database environment	Load time
10.7.4	MySQL 8.0	Linux, Core i5, 16 GB, SATA HDD 7200rpm storage	(same platform as the Registry)	424 seconds
10.7.4	MySQL 8.0	Linux, Core i5, 16 GB, SATA SSD storage	(same platform as the Registry)	64 seconds
10.7.4	MySQL 5.7	Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage	(same platform as the Registry)	23 seconds
10.7.4	MySQL 5.7	Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage	(same platform as the Registry)	(tbc)

Datastore: Fusion Datastore (in memory datastore persisted to the operating database)

Registry	Database	Registry environment	Datastore database environment	Load time
10.7.4	MySQL 8.0	Linux, Core i5, 16 GB, SATA SSD storage	(same platform as the Registry)	11 seconds
10.7.4	MySQL 5.7	Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage	(same platform as the Registry)	6 seconds
10.7.4	MySQL 5.7	Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage	(same platform as the Registry)	(tbc)