Apache Kafka integration - Revision history

Vmurrell at 17:04, 13 October 2025

2025-10-13T17:04:31Z

Vmurrell at 17:03, 13 October 2025

2025-10-13T17:03:13Z

Vmurrell at 12:51, 13 September 2023

2023-09-13T12:51:05Z

Vmurrell at 12:50, 13 September 2023

2023-09-13T12:50:37Z

Vmurrell at 15:40, 4 September 2023

2023-09-04T15:40:08Z

Vmurrell at 14:50, 4 September 2023

2023-09-04T14:50:07Z

Mnelson: /* Events */

2021-01-14T13:59:46Z

‎Events

Mnelson: /* Registry Database Tables */

2020-12-03T09:37:59Z

‎Registry Database Tables

Mnelson: /* Registry Database Tables */

2020-12-03T09:37:38Z

‎Registry Database Tables

Mnelson: /* Events */

2020-12-03T09:20:56Z

‎Events

@@ Line 1: / Line 1: @@
-[[Fusion Registry Configuration]]
+[[Category:Fusion Registry Configuration]]
 [[Category:How_To]]
 =Compatibility=

@@ Line 1: / Line 1: @@
-[[Category:Fusion Registry Install]]
+[[Fusion Registry Configuration]]
 [[Category:How_To]]
 =Compatibility=

@@ Line 1: / Line 1: @@
-[[Category:Fusion Registry Install
+[[Category:Fusion Registry Install]]
 [[Category:How_To]]
 =Compatibility=

@@ Line 1: / Line 1: @@
-Category:Fusion Registry Install
+[[Category:Fusion Registry Install
 [[Category:How_To]]
 =Compatibility=

@@ Line 1: / Line 1: @@
-[[Category:Functions]]
+Category:Fusion Registry Install
 [[Category:How_To]]
 =Compatibility=

← Older revision		Revision as of 13:59, 14 January 2021
Line 140:		Line 140:

	The kafka logs are only set to SENT when the Fusion Registry has been notified by the kafka server that it has successfully received the message.		The kafka logs are only set to SENT when the Fusion Registry has been notified by the kafka server that it has successfully received the message.
		+
		+	=Check Structure Notification Status=
		+	http(s)://[server]:[port]/[webapp]/ws/secure/settings/kafka/producer/status/structure
		+
		+	Response:
		+	{
		+	Running : boolean
		+	}

	=Events=		=Events=

@@ Line 289: / Line 289: @@
 | producer_id || the id of the producer
 |-
-| tx_status|| transaction status <br>'''0''' Pending
+| tx_status|| transaction status
                                   <br>'''1''' Sent Sucessfully
                                   <br>'''2''' Was not sent due to local failure, message did not leave Fusion Registry, typically message fails local checks with respect to connection settings

← Older revision		Revision as of 09:37, 3 December 2020
Line 236:		Line 236:

	= Registry Database Tables =		= Registry Database Tables =
		+
		+	The folowing database tables are used in the Fusion Registry to store Kafka information
		+
		+
		+	'''kafa_producer_config'''
		+	This table is used to link a Fusion Registry Producer (i.e. the structure producer) to a topic, and message format
		+	{\| class="wikitable"
		+	\|-
		+	! Column !! Description
		+	\|-
		+	\| id \|\| Id of the Registry Kafka Producer
		+	\|-
		+	\| topic \|\| The kafka topic the producer sends messages to
		+	\|-
		+	\| format \|\| The format the message is sent in
		+	\|}
		+
		+	'''kafa_producer_config_properties'''
		+	This table is used to store any additional properties for the Fusion Registry Producer
		+	{\| class="wikitable"
		+	\|-
		+	! Column !! Description
		+	\|-
		+	\| id \|\| Id of the Registry Kafka Producer
		+	\|-
		+	\| line_number\|\| properties are stored as a list, this is the index in the list
		+	\|-
		+	\| property \|\| The property, in the format '''prop=value'''
		+	\|}
		+
		+
		+	'''kafka_connection_settings'''
		+	This table is used to store any additional properties for the Fusion Registry Producer
		+	{\| class="wikitable"
		+	\|-
		+	! Column !! Description
		+	\|-
		+	\| conn_key\|\| kafka connection property e.g. batch.size
		+	\|-
		+	\| conn_value\|\| the value of the property e.g. 1000
		+	\|}
		+
		+
		+	'''kafka_producer_tx'''
		+	The producer transaction table
		+	{\| class="wikitable"
		+	\|-
		+	! Column !! Description
		+	\|-
		+	\| tx_id \|\| the id of the transaction (see '''sdmx_transaction''' table)
		+	\|-
		+	\| producer_id \|\| the id of the producer
		+	\|-
		+	\| tx_status\|\| transaction status <br>'''0''' Pending
		+	<br>'''1''' Sent Sucessfully
		+	<br>'''2''' Was not sent due to local failure, message did not leave Fusion Registry, typically message fails local checks with respect to connection settings
		+	<br>'''3''' Sent and a notification of error received from kafka server
		+	<br>'''4''' Unexpected Fusion Registry error
		+	\|-
		+	\| tx_last_updated \|\| the last time this transaction record was updated (UNIX Time milliseconds)
		+	\|-
		+	\| tx_size \|\| size of the transaction in bytes (before compression)
		+	\|-
		+	\| tx_messages \|\| number of messages in the transaction
		+	\|}

← Older revision		Revision as of 09:20, 3 December 2020
Line 234:		Line 234:
	* Only publish 'public' structures - i.e. those without any rules restricting access		* Only publish 'public' structures - i.e. those without any rules restricting access
	* Publish changes for each Content Security group (and 'public') on their own separate Kafka topics - applying Kafka Security to the topics would then ensure the intended access restrications are enforced		* Publish changes for each Content Security group (and 'public') on their own separate Kafka topics - applying Kafka Security to the topics would then ensure the intended access restrications are enforced
		+
		+	= Registry Database Tables =