Difference between revisions of "Performance benchmarks"

From Fusion Registry Wiki
Jump to navigation Jump to search
(Created page with "=Data load performance= Data load performance is the time taken to persist a loaded dataset to storage. ==Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)== 3633 series<br>...")
 
 
(6 intermediate revisions by one other user not shown)
Line 1: Line 1:
 +
[[Category:How_To]]
 
=Data load performance=
 
=Data load performance=
  
−
Data load performance is the time taken to persist a loaded dataset to storage.
+
Data load performance is the time taken to persist a loaded dataset to storage measured from the start of the 'process data import' operation to when its successfult completion is reported. An excerpt from a <code>FusionRegistry.log</code> logfile illustrates the princple:
 +
 
 +
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Process data import
 +
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - acquire lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
 +
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - lock acquired on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
 +
2021-08-16 15:39:55.672 INFO Thread-8 com.metadatatechnology.free.manager.data.publish.impl.DataSourceManagerImpl - Provision 'urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=ECB:EXR_MT_MT(1.0)'  links to data source with Id 'IN_MEMORY'
 +
2021-08-16 15:39:56.635 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 1.0000107% complete
 +
2021-08-16 15:39:56.636 INFO Thread-10374 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 1%
 +
2021-08-16 15:39:56.734 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 2.0000215% complete
 +
... Import process progress
 +
2021-08-16 15:40:07.251 INFO Thread-10570 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 99%
 +
2021-08-16 15:40:07.318 INFO Thread-10572 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 100%
 +
... Build constraints and other post import tasks
 +
2021-08-16 15:40:07.358 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - release lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
 +
2021-08-16 15:40:07.361 INFO Thread-10574 com.metadatatechnology.free.manager.data.publish.impl.MatrixDataImportManagerImpl - Data Imported Successfully
  
 
==Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)==
 
==Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)==
 
3633 series<br>
 
3633 series<br>
−
1,207,654 observations
+
1,207,654 observations<br>
 +
[[File:ECB Exchange Rates.zip|Download: ECB Exchange Rates Structures and Data - ZIP]]
  
 
===Datastore: Registry Managed Database===
 
===Datastore: Registry Managed Database===
 
{| class="wikitable sortable"
 
{| class="wikitable sortable"
 
|-
 
|-
−
! Database !! Registry Environment !! Datastore Database Environment !! Load time
+
! Registry !! Database !! Registry environment !! Datastore database environment !! Load time
 
|-
 
|-
−
| MySQL 8.0|| Linux, Core i5, 16 GB, SATA HDD 7200rpm storage|| (same platform as the Registry)|| 424 seconds
+
| 10.7.4 || MySQL 8.0|| Linux, Core i5, 16 GB, SATA HDD 7200rpm storage|| (same platform as the Registry)|| 424 seconds
 
|-
 
|-
−
| MySQL 8.0 || Linux, Core i5, 16 GB, SATA SSD storage|| (same platform as the Registry)|| 64 seconds
+
| 10.7.4 || MySQL 8.0 || Linux, Core i5, 16 GB, SATA SSD storage|| (same platform as the Registry)|| 64 seconds
 
|-
 
|-
−
| MySQL 5.7 || Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage || (same platform as the Registry)|| 23 seconds
+
| 10.7.4 || MySQL 5.7 || Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage || (same platform as the Registry)|| 23 seconds
 
|-
 
|-
−
| MySQL 5.7 || Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage || (same platform as the Registry)|| (tbc)
+
| 10.7.4 || MySQL 5.7 || Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage || (same platform as the Registry)|| (tbc)
 
|}
 
|}
  
−
===Datastore: Registry Managed Database===
+
===Datastore: Fusion Datastore (in memory datastore persisted to the operating database)===
 
{| class="wikitable sortable"
 
{| class="wikitable sortable"
 
|-
 
|-
−
! Database !! Registry Environment !! Operating Database Environment !! Load time
+
! Registry !! Database !! Registry environment !! Datastore database environment !! Load time
 
|-
 
|-
−
| MySQL 8.0 || Linux, Core i5, 16 GB, SATA SSD storage|| (same platform as the Registry)|| 11 seconds
+
| 10.7.4 || MySQL 8.0 || Linux, Core i5, 16 GB, SATA SSD storage|| (same platform as the Registry)|| 12 seconds
 
|-
 
|-
−
| MySQL 5.7 || Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage || (same platform as the Registry)|| 6 seconds
+
| 10.7.4 || MySQL 5.7 || Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage || (same platform as the Registry)|| 6 seconds
 
|-
 
|-
−
| MySQL 5.7 || Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage || (same platform as the Registry)|| (tbc)
+
| 10.7.4 || MySQL 5.7 || Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage || (same platform as the Registry)|| (tbc)
 
|}
 
|}

Latest revision as of 22:11, 10 September 2023

Data load performance

Data load performance is the time taken to persist a loaded dataset to storage measured from the start of the 'process data import' operation to when its successfult completion is reported. An excerpt from a FusionRegistry.log logfile illustrates the princple: 

2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Process data import
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - acquire lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.671 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - lock acquired on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:39:55.672 INFO Thread-8 com.metadatatechnology.free.manager.data.publish.impl.DataSourceManagerImpl - Provision 'urn:sdmx:org.sdmx.infomodel.registry.ProvisionAgreement=ECB:EXR_MT_MT(1.0)'  links to data source with Id 'IN_MEMORY'
2021-08-16 15:39:56.635 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 1.0000107% complete
2021-08-16 15:39:56.636 INFO Thread-10374 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 1%
2021-08-16 15:39:56.734 INFO Thread-8 com.metadatatechnology.fcee.engine.data.writer.PercentageCompleteWriterEngine - 2.0000215% complete
... Import process progress
2021-08-16 15:40:07.251 INFO Thread-10570 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 99%
2021-08-16 15:40:07.318 INFO Thread-10572 com.metadatatechnology.free.engine.data.publish.impl.ImportProcessorEngine - Percentage Complete: 100%
... Build constraints and other post import tasks
2021-08-16 15:40:07.358 INFO Thread-8 com.metadatatechnology.free.model.data.publish.DatabaseLock - release lock on database tables for flow: urn:sdmx:org.sdmx.infomodel.datastructure.Dataflow=ECB:EXR(1.0)
2021-08-16 15:40:07.361 INFO Thread-10574 com.metadatatechnology.free.manager.data.publish.impl.MatrixDataImportManagerImpl - Data Imported Successfully

Test Dataset: ECB Exchange Rates (ECB, EXR, 1.0)

3633 series
1,207,654 observations
File:ECB Exchange Rates.zip

Datastore: Registry Managed Database

Registry Database Registry environment Datastore database environment Load time
10.7.4 MySQL 8.0 Linux, Core i5, 16 GB, SATA HDD 7200rpm storage (same platform as the Registry) 424 seconds
10.7.4 MySQL 8.0 Linux, Core i5, 16 GB, SATA SSD storage (same platform as the Registry) 64 seconds
10.7.4 MySQL 5.7 Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage (same platform as the Registry) 23 seconds
10.7.4 MySQL 5.7 Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage (same platform as the Registry) (tbc)

Datastore: Fusion Datastore (in memory datastore persisted to the operating database)

Registry Database Registry environment Datastore database environment Load time
10.7.4 MySQL 8.0 Linux, Core i5, 16 GB, SATA SSD storage (same platform as the Registry) 12 seconds
10.7.4 MySQL 5.7 Windows 10, Core i7, 32GB, PCS PCIe M.2 SSD NVME (2000 MB/R, 1100 MB/W) storage (same platform as the Registry) 6 seconds
10.7.4 MySQL 5.7 Linux, Core i7, 16GB, EVO PLUS M.2, PCIe NVMe (up to 3500MB/R, 3200MB/W) storage (same platform as the Registry) (tbc)
Retrieved from "https://wiki.sdmxcloud.org/index.php?title=Performance_benchmarks&oldid=6003"