EdgeServerCompiler - Revision history

Vmurrell at 15:31, 11 September 2023

2023-09-11T15:31:30Z

Vmurrell at 13:41, 16 August 2023

2023-08-16T13:41:06Z

Mnelson: /* Compile New Dataset into Environment */

2022-07-05T14:26:07Z

‎Compile New Dataset into Environment

Mnelson: /* Compile New Dataset into Environment */

2022-07-05T13:53:54Z

‎Compile New Dataset into Environment

Mnelson: /* Multiple Compiles without impacting Last Updated in Delta Mode */

2021-10-20T09:37:13Z

‎Multiple Compiles without impacting Last Updated in Delta Mode

Mnelson: /* Delete Observations or Series in Existing Dataset */

2021-10-20T09:36:34Z

‎Delete Observations or Series in Existing Dataset

Mnelson: /* Delete Observations or Series in Existing Dataset */

2021-10-20T08:56:47Z

‎Delete Observations or Series in Existing Dataset

Mnelson: /* Delete Observations or Series in Existing Dataset */

2021-10-20T08:55:11Z

‎Delete Observations or Series in Existing Dataset

Mnelson: /* Update/add new observations/series to existing dataset */

2021-10-20T08:54:55Z

‎Update/add new observations/series to existing dataset

Mnelson: /* Command Line Arugments */

2021-10-20T08:38:26Z

‎Command Line Arugments

← Older revision		Revision as of 15:31, 11 September 2023
Line 1:		Line 1:
	[[Category:Fusion Edge Server]]		[[Category:Fusion Edge Server]]
		+	[[Category:How_to_FES]]
	= Fusion Edge Compiler =		= Fusion Edge Compiler =

← Older revision		Revision as of 13:41, 16 August 2023
Line 1:		Line 1:
		+	[[Category:Fusion Edge Server]]
	= Fusion Edge Compiler =		= Fusion Edge Compiler =

@@ Line 263: / Line 263: @@
 = Worked Examples =
 == Compile New Dataset into Environment ==
 <p>'''Description''': The Edge Server is running against a environment with datasets published to it.  A new dataset is ready for publication, for a Dataflow which is not part of the current live environment.  The file system has been built and contains the dataset and associated structure file for the data.  The remainder of the files in the file system are either not present, or are currently in the live environment from a previous compile.</p>

@@ Line 270: / Line 270: @@
 <p>'''Compiler Arguments:'''</p>
 <code>
-compileFileSystem.bat " -src "MyFiles" -tgt "CompiledEnvironment"-lgr "LiveEnvironment  -sgn "password"
+compileFileSystem.bat -src "MyFiles" -tgt "CompiledEnvironment"-lgr "LiveEnvironment"  -sgn "password"
 </code>
 <br><br>

@@ Line 341: / Line 341: @@
 <p>'''Technique 2''':</p>
 <p>The second technique is to always run the buildFileSystem command passing in an explicit date for updatedAfter, and not referencing the ledger at all, this will ensure that the data query has a known timestamp from which to pull the data from.</p>
 <code>
 buildFileSystem.bat  -api "https://yourorg.org/sdmx"   -tgt "/home/compiler/deltadata" -df "all" -upd “2020-01-30T00:00.00”

@@ Line 307: / Line 307: @@
 <p>'''buildFileSystem.bat''' is used to pull the full dataset from the web service, and place it in the target directory.  There is no need to provide a ledger location, as the pull request should NOT include information about pulling data updated after a given timestamp.  The '''df''' argument provides the reference to the dataset to pull.</p>
 <p>'''compileFileSystem.bat''' is used to compile the file in the  replacedata folder.  The reference to the ledger is used so that a copy of the existing environment can be taken, and the environment version can be updated accordingly.  The '''rd''' argument is used to ensure that the dataset being compiled into the environment is NOT merged into the existing dataset, but used as a replacement of the dataset in the existing environment.</p>
 == Ignore file timestamps in Filesystem ==

@@ Line 297: / Line 297: @@
 buildFileSystem.bat  -api "https://yourorg.org/sdmx"   -tgt "/home/compiler/replacedata" -df "ACY:DF_ID(1.0)"
 </code>
 <code>
 compileFileSystem.bat " -src "/home/compiler/ replacedata" -tgt "CompiledEnvironment"-lgr "LiveEnvironment " -sgn "password" -rd
@@ Line 302: / Line 303: @@
 <br><br>
 <p>'''Explanation''':</p>
-<p>The only way to delete observations and series from a dataset is to recompile the full dataset (minus the observations that are no longer required).  The first step is to ensure the full dataset is in the file system, ready for compile.   In this example, it is assumed that the dataset is imported into the file system is built using the web service API, however, it is possible to manually create/edit a dataset in the file system in which case the buildFileSystem step can be skipped.</p>
+<p>The only way to delete observations and series from a dataset is to recompile the full dataset (minus the observations that are no longer required).  The first step is to ensure the full dataset is in the file system, ready for compile.   This example shows how to pull the full dataset from a web service API using the buildFileSystem script.  However, it is possible to manually create/edit a dataset in the file system in which case the buildFileSystem step step can be skipped.</p>
 <p>'''buildFileSystem.bat''' is used to pull the full dataset from the web service, and place it in the target directory.  There is no need to provide a ledger location, as the pull request should NOT include information about pulling data updated after a given timestamp.  The '''df''' argument provides the reference to the dataset to pull.</p>

@@ Line 276: / Line 276: @@
 <p>The default behaviour of the compilation process is to merge in any new information from the local file system into the live environment.  A copy of the live environment is downloaded from the -lgr location, and this forms the basis for the new environment which is put into the -tgt folder.  The structures read in from the local file system are merged into the copy of the live environment, and the new dataset is compiled and also included in the copy of the live environment.  This live environment copy is written into the CompiledEnvironment folder, as designated by the -tgt argument.  All other files in the LiveEnvironment remain unchanged in the new CompiledEnvironment, so when the CompiledEnvironment is deployed, the Edge Server will contain all the same information as before, plus the new dataset.
-== Update/add new observations/series to existing dataset ==
+== Update or Add Observations or Series into Existing Dataset ==
 <p>'''Description''': The Edge Server is running against a environment with datasets published to it.  There is some new data for publication for one of the existing datasets.  The new data contains some observations which already exist in the live environment and is to be overwritten, it also contains some new observations which do not yet exist.</p>
@@ Line 288: / Line 288: @@
 <p>'''Explanation''':</p>
 <p>The default behaviour of the compilation process is to merge in any new information from the local file system into the live environment.  A copy of the live environment is downloaded from the -lgr location, and this forms the basis for the new environment which is put into the -tgt folder.  The data file in the MyFiles folder could either contain all the data (existing plus new data) or simply contain the deltas (just the new observations and those being edited).  Either way, the compiler will detect that this dataset already exists in the live environment and read it in, it will then read the data from the file system and merge the contents into the copy from the live environment, before writing it back out into the CompiledEnvironment folder, as designated by the -tgt argument.  All other files in the LiveEnvironment remain unchanged in the CompiledEnvironment, so when the new environment is deployed, the Edge Server will contain all the same information as before, plus the new and edited observations.</p>
-−
 == Ignore file timestamps in Filesystem ==
 <p>'''Description''': The default behaviour of the compilation process is to read the timestamp from the ledger (if provided) and use this as a filter on all files in the source file system.  Files whose last updated timestamps are before the timestamp on the ledger are deemed to have already been compiled, and as such are ignored by the Compilation process.  So even if the source file system contains 100 files, if only 1 of those files was modified since the last compile, the only 1 file will be included in the next compilation process.  As the default behaviour of the compiler is to take a copy of the live ledger files and merge into it, the compiler does not need to re-read files it has already processed.</p>

@@ Line 213: / Line 213: @@
 |}
 ==  Properties File ==
-==  Properties File ==
+<p>If providing configuration parameter via a properties file, it must be in JSON format, and have the following structure </p>
 <pre>
+{
@@ Line 230: / Line 229: @@
+}
 </pre>
 = Worked Examples =