NetarchiveSuiteNOTE: This description is in the early stage of being written
Assignment - Rewrite deploy based on settings overwrite
Action(print,Printer friendly version) TableOfContents
References
Reference documents
- ["Glossary"]
Dependencies
- All the tasks below need to be done in the order they are specified.
- all of the tasks is required to be done in the same iteration, although they can be implemented and tested in parallel with other developments.
Terminology
- ...
Bugs
...
Feature Requests
[#1520] Deploy of more than one Bitapplication per server
Basic idea behind new deploy
History of deploy
In the previous NetarchiveSuite there have already existed a deploy module. However this module had the following inconviniences:
It was tightly connected to the setup of the Danish intallation of NetarchiveSuite
It was based on a configuration file consisting of XML where the connection between configuration settings and NetarchiveSuite settings were far from obvious.
It was based on the assumption that all settings had to be specified in a settings file for NetarchiveSuite
Now since the assumption of specified in a settings file for NetarchiveSuite is no longer valid becuse of resent changes. The NetarchiveSuite has now been changed to have in-build default settings where only local overwrites of defaults has to be set for the individual applications.
Furthermore, resent analysis and assignment description (assignment B2) for the archive has revealed the most obvious connections to the setup of the Danish intallation of NetarchiveSuite. This is mainly based on the interpretation of Location, which have both meant "location of bitarchive to be used" and "physical location of this instance of this application". As part of the assignment B2 as well as part of this assignment is a change of naming of the old location settings into distinc terms of physical location and replica. This change will mean that the new deploy no longer needs to be connected to the Danish intallation of NetarchiveSuite.
Lastly, the definitions in the new configuration file will in the new deploy be named precisely the same way as settings for NetarchiveSuite in cases where there are a direct connection. Only few definitions will be for deploy only, and in these cases the definitions will follow the naming convention of being prefixed with "deploy_".
Override settings structure
The idea is that the new deploy will be based on the default settings in NetarchiveSuite. A configuration file is then used for specifications of where the different applications are placed and which settings overrides is needed for the applications.
The default settings can be overwritten at different levels in the configuration settings file. this is illustrated in the below figure:
attachment:layers.gif
Documentation
In existing documentation
Deployment Manual with reference to existing sections in Installation Manual
- Preparation
- make it-config file
- get netarchivesuite zip file
- get and adjust configuration files
- deploy
- how to
- what is happening
- installation
- how to (including modifications)
- what is happening (incl. makedir)
- start and kill
Settings documentation should only consist in references to xml files in repository. Explation of settings overwrite structure and special deploy configuration settings (levels, install dir etc.)
New deploy documentation
The new deploy works in three steps:
- prepare folders for deployment
- Install folders on machines for all physical location
- Start all installed applications
Preparation of folders for deployment is illustrated in the below figure
(consider whether NetarchiveSuite zip file should be given as parameter here and be placed in install-dir) attachment:deploy_step1.gif
Installation of folders on machines for all physical location is illustrated in the below figure
(consider whether NetarchiveSuite zip file should be as parameter in previous step, and then be taken from install dir here)
attachment:deploy_step2.gif
Start all installed applications is illustrated in the below figure attachment:deploy_step3.gif
Changes in NetarchiveSuite apart from deploy
System state GUI
The current columns "Organisation" and "Port" must be replaced by
- physicalLocation for "Location" (for all)
replica for "Replica" (for BitArchiveMonitors and BitArchives only)
http.port for "HTTP" (for GUIApplication and ViewerProxyApplication only)
harvesting.queuePriority (for HarvestControllerApplication only)
Final look must be agreed with users. An idea could be introduce a hiding mechanism at the same level and in the same way as the "Show all" functionality
In order to pass data to System state GUI, the data must be made available via the SingleMbeanObject.
Split of location concept
Today "location" covers both physical location and bitarchive replica. This must change in order to make deploy more general.
The following settings are changed:
locations -> replicas
location ->
replicaId (added replicaType & replicaName)
deployApplicationInstanceId (for BitarchiveMonitorApplications)
locations.batchLocation -> useReplicaId
thisLocation ->
archive.thisReplicaId for bitarchive & monitor (Channels)
thisPhysicalLocation for others (SingleMbeanObject)
useReplicaId for getFile/get in archive.tool, viewerproxy & arcrepository?! Used for getFile/get tool
Channel name definitions
Must rely on thisReplica and applicationInstanceId instead of thisLocation and http.port
thisReplicaId for Channels (only used for ba & bamon)
The following setting is new:
- applicationInstanceId (replacing use of port in channel names)
HarvestController
ViewerProxyApplication (may not be necessary if use of IP-address is enough and only 1 on each machine)
BitarchiveMonitorApplication (may not be necessary if we accept deploys naming instances from thisReplicaId)
Differentiate applications on instance id
Differentiate applications on instance id istead of http.port number or thisLocation. ...
The following setting is new:
- applicationInstanceId (replacing use of port in channel names)
HarvestController
ViewerProxyApplication (may not be necessary if use of IP-address is enough and only 1 on each machine)
BitarchiveMonitorApplication (may not be necessary if we accept deploys naming instances from thisReplicaId)
New definition of IT-config
explation of settings overwrite structure and special deploy configuration settings (levels, install dir etc.)
New special deploy settings
<deployClassPath>
- Defines a class path to be added for an application
- Note: several additional class paths can be specified within a scope, but new definitions in inner scopes will overwrite outer scopes.
<deployGlobal>
Defines a deploy global 1. level scope where settings can be set to overwrite setting defaults from the NetarchiveSuite software.
<deployInstallDir>
- Installation directory for a deployMachine
<deployJavaOpt>
- Defines a jave option for an application.
- Note: several additional jave options can be specified within a scope, but new definitions in inner scopes will overwrite outer scopes.
<deployMachine>
- Defines a deploy machine 3. level scope where common settings for the machine and the applications running in the macine can be set. These settings will overwrite 1. and 2. level settings
<deployMachineUserName>
- Defines the user name for a deployMachine
New settings
<applicationInstanceId>
Defines identification of a single application instance (e.g. suffix for application specific scripts, suffix for directory to place files etc.). This is needed in cases where there are more instances of the same application are placed on the same machine (e.g. BitarchiveMonitors) in case more instances of the same application are placed on the same machine
- Will for instance replace port in definition of channels.
Settings indirectly set
Named tags that result in implicit setting og settings:
<thisPhysicalLocation name="PL"> in <deployGlobal> scope
- sets settings.common.thisPhysicalLocation to PL
<applicationName name="AN"> in <physicalLocation> scope
- sets settings.common.applicationName to AN
- NOTE: that the applicationName is NOT USED today, it is automatically set at startup. However this is also one of the reasons that we cannot reload new setting for an application. Thus this new deploy will also help solving the reuoload issue at a later stage.
New it-config example
The below must replace the existing it-config-example file.
<?xml version="1.0" encoding="UTF-8"?>
<!--
* File: $Id: it_configuration_example.xml $
* Revision: $Revision: $
* Author: $Author: $
* Date: $Date: $
*
* The Netarchive Suite - Software to harvest and preserve websites
* Copyright 2004-2007 Det Kongelige Bibliotek and Statsbiblioteket, Denmark
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
-->
<!--_________________________________________________________________________-->
<!-- DEPLOY SPECIFICATION LEVELS AND SCOPES -->
<!---________________________________________________________________________-->
<!-- There are four level of scopes within deploy setting:
1. level: defined by <deployGlobal>
Defines a deploy global scope
2. level: defined by <thisPhysicalLocation name="XX">
Defines scope for a physical location (overwirtes level 1 defs.)
3. level: defined by <deployMachine>
Defines a deploy machine scope (overwirtes level 1 & 2 defs.)
4. level defined by <applicationName name="XX">
Defines scope for an application (overwirtes level 1-3 defs.) -->
<!--_________________________________________________________________________-->
<!-- SPECIAL DEPLOY SETTINGS (apart from NetarchiveSuite settings -->
<!--_________________________________________________________________________-->
<!-- New deploy settings:
<deployGlobal>
Defines a deploy global 1. level scope where settings can be set
to overwrite setting defaults
<deployClassPath>
Defines a class path to be added for an application
Note: several additional class paths can be specified within a
scope, but new definitions in inner scopes will overwrite outer
scopes.
<deployMachine>
Defines a deploy machine 3. level scope where common settings for
the machine and the applications running in the macine can be set.
These settings will overwrite 1. and 2. level settings
<deployJavaOpt>
Defines a jave option for an application
Note: several additional jave options can be specified within a
scope, but new definitions in inner scopes will overwrite outer
scopes
<deployInstallDir>
Defines the installation directory for a deployMachine
<deployMachineUserName>
Defines the user name for a deployMachine -->
<deployGlobal>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<deployJavaOpt>-Xmx1536m</deployJavaOpt>
<settings>
<common>
<environmentName>TEST</environmentName>
<applicationInstanceId></applicationInstanceId>
<remoteFile>
<class>
dk.netarkivet.common.distribute.FTPRemoteFile
</class>
<serverPort>21</serverPort>
<retries>3</retries>
</remoteFile>
<jms>
<class>
dk.netarkivet.common.distribute.JMSConnectionSunMQ
</class>
<broker>kb-dev-adm-001.kb.dk</broker>
<port>7676</port>
</jms>
<jmx>
<passwordFile>conf/jmxremote.password</passwordFile>
<timeout>120</timeout>
</jmx>
<indexClient>
<!-- from dk.netarkivet.archive.indexserver.distribute.IndexRequestClientSettings.xml -->
<indexRequestTimeout>43200000</indexRequestTimeout>
</indexClient>
<replicas>
<!-- The names of all bit archive replicas in the
environment, e.g., "nameOfBitachiveOne" and "nameOfBitachiveTwo". -->
<replica>
<replicaId>SB</replicaId>
<replicaName>SBB</replicaName>
<replicaType>bitArchive</replicaType>
</replica>
<replica>
<replicaId>KB</replicaId>
<replicaName>KBB</replicaName>
<replicaType>bitArchive</replicaType>
</replica>
</replicas>
<!-- Default bit archive to use for batch jobs (if none is specified) -->
<!-- or get and getFile oprations, (as well as bitArchive monitor) -->
<useReplicaId>KB</useReplicaId>
</common>
<monitor>
<jmxUsername>monitorRole</jmxUsername>
<jmxPassword>test</jmxPassword>
</monitor>
<!-- directory for install -->
<archive>
<bitpreservation>
<baseDir>bitpreservation</baseDir>
</bitpreservation>
<arcrepository>
<baseDir>.</baseDir>
</arcrepository>
</archive>
<!-- tempDir corrected from ./tests/commontempdir -->
<tempDir>tmpdircommon</tempDir>
<!-- viewerproxy.baseDir is set below -->
<!-- harvester.harvesting.serverDir is set below -->
</settings>
<thisPhysicalLocation name="KBL">
<!-- installation directories only used by deploy -->
<deployInstalldir>/home/test</deployInstalldir>
<deployMachineUserName>test</deployMachineUserName>
<settings>
<common>
<remoteFile>
<serverName>kb-dev-har-001.kb.dk</serverName>
<userName>ftptestuser</userName>
<userPassword>ftptestpasswd</userPassword>
</remoteFile>
<mail>
<server>examplesmtpserver.netarkivet.dk</server>
</mail>
<notifications>
<class>dk.netarkivet.common.utils.EMailNotifications</class>
<receiver>example@netarkivet.dk</receiver>
<sender>example@netarkivet.dk</sender>
</notifications>
<useReplicaId>KB</useReplicaId>
</common>
</settings>
<deployMachine name="kb-dev-adm-001.kb.dk">
<applicationName name="dk.netarkivet.common.webinterface.GUIApplication">
<deployClassPath>lib/dk.netarkivet.harvester.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<settings>
<common>
<http>
<port>8076</port>
</http>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
</settings>
</applicationName>
<applicationName name="dk.netarkivet.archive.arcrepository.ArcRepositoryApplication">
<settings>
<common>
<jmx>
<port>8101</port>
<rmiPort>8201</rmiPort>
</jmx>
</common>
</settings>
</applicationName>
<applicationName name="dk.netarkivet.archive.bitarchive.BitarchiveMonitorApplication">
<applicationInstanceId>KBBM</applicationInstanceId>
<settings>
<common>
<jmx>
<port>8102</port>
<rmiPort>8202</rmiPort>
</jmx>
</common>
<archive>
<thisReplicaId>KB</thisReplicaId>
</archive>
</settings>
</applicationName>
<applicationName name="dk.netarkivet.archive.bitarchive.BitarchiveMonitorApplication">
<applicationInstanceId>SBBM</applicationInstanceId>
<settings>
<common>
<jmx>
<port>8103</port>
<rmiPort>8203</rmiPort>
</jmx>
</common>
<archive>
<thisReplicaId>SB</thisReplicaId>
</archive>
</settings>
</applicationName>
</deployMachine>
<deployMachine os="windows" name="kb-dev-bar-010.bitarkiv.kb.dk">
<deployMachineUserName>ba-test</deployMachineUserName>
<deployInstallDir>c:\Documents and Settings\ba-test</deployInstallDir>
<deployJavaOpt>-Xmx1150m</deployJavaOpt>
<applicationName name="dk.netarkivet.archive.bitarchive.BitarchiveApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<archive>
<thisReplicaId>KB</thisReplicaId>
<fileDir>q:\bitarkiv</fileDir>
</archive>
</settings>
</applicationName>
</deployMachine>
<deployMachine os="windows" name="kb-dev-bar-011.bitarkiv.kb.dk">
<deployMachineUserName>ba-test</deployMachineUserName>
<deployInstallDir>c:\Documents and Settings\ba-test</deployInstallDir>
<deployJavaOpt>-Xmx1150m</deployJavaOpt>
<applicationName name="dk.netarkivet.archive.bitarchive.BitarchiveApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<archive>
<thisReplicaId>KB</thisReplicaId>
<fileDir>q:\bitarkiv</fileDir>
</archive>
</settings>
</applicationName>
</deployMachine>
<deployMachine name="kb-dev-har-001.kb.dk">
<deployClassPath>lib/dk.netarkivet.harvester.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<applicationName name="dk.netarkivet.harvester.harvesting.HarvestControllerApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<harvester>
<harvesting>
<queuePriority>LOWPRIORITY</queuePriority> <!-- on http port-->
<heritrix>
<guiPort>8190</guiPort>
<jmxPort>8191</jmxPort>
</heritrix>
<serverDir>harvester</serverDir>
</harvesting>
</harvester>
</settings>
</applicationName>
</deployMachine>
<deployMachine name="kb-dev-har-002.kb.dk">
<deployClassPath>lib/dk.netarkivet.harvester.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<applicationName name="dk.netarkivet.harvester.harvesting.HarvestControllerApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<harvester>
<harvesting>
<queuePriority>LOWPRIORITY</queuePriority>
<heritrix>
<guiPort>8190</guiPort>
<jmxPort>8191</jmxPort>
</heritrix>
<serverDir>harvester</serverDir>
</harvesting>
</harvester>
</settings>
</applicationName>
</deployMachine>
<deployMachine name="kb-dev-acs-001.kb.dk">
<settings>
<viewerproxy>
<baseDir>viewerproxy</baseDir>
</viewerproxy>
</settings>
<applicationName name="dk.netarkivet.archive.indexserver.IndexServerApplication">
<settings>
<common>
<jmx>
<port>8101</port>
<rmiPort>8201</rmiPort>
</jmx>
</common>
</settings>
</applicationName>
<applicationName name="dk.netarkivet.viewerproxy.ViewerProxyApplication">
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<settings>
<common>
<http>
<port>8076</port>
</http>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
</settings>
</applicationName>
</deployMachine>
</thisPhysicalLocation>
<thisPhysicalLocation name="SBL">
<!-- installation directories only used by deploy -->
<deployInstalldir>/home/netarkiv</deployInstalldir>
<deployMachineUserName>netarkiv</deployMachineUserName>
<settings>
<common>
<remoteFile>
<serverName>sb-dev-bar-001.statsbiblioteket.dk</serverName>
<userName>ftptestuser</userName>
<userPassword>ftptestpasswd</userPassword>
</remoteFile>
<mail>
<server>examplesmtpserver.netarkivet.dk</server>
</mail>
<notifications>
<class>dk.netarkivet.common.utils.EMailNotifications</class>
<receiver>example@netarkivet.dk</receiver>
<sender>example@netarkivet.dk</sender>
</notifications>
<useReplicaId>SB</useReplicaId>
</common>
</settings>
<deployMachine name="sb-dev-har-001.statsbiblioteket.dk">
<deployClassPath>lib/dk.netarkivet.harvester.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.archive.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.viewerproxy.jar</deployClassPath>
<deployClassPath>lib/dk.netarkivet.monitor.jar</deployClassPath>
<applicationName name="dk.netarkivet.harvester.harvesting.HarvestControllerApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<harvester>
<harvesting>
<queuePriority>HIGHPRIORITY</queuePriority>
<heritrix>
<guiPort>8190</guiPort>
<jmxPort>8191</jmxPort>
</heritrix>
<serverDir>harvester</serverDir>
</harvesting>
</harvester>
</settings>
</applicationName>
</deployMachine>
<deployMachine name="sb-dev-bar-001.statsbiblioteket.dk">
<applicationName name="dk.netarkivet.archive.bitarchive.BitarchiveApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
</common>
<archive>
<thisReplicaId>SB</thisReplicaId>
<fileDir>/netarkiv/0001</fileDir>
<fileDir>/netarkiv/0002</fileDir>
</archive>
</settings>
</applicationName>
</deployMachine>
<deployMachine name="sb-dev-acs-001.statsbiblioteket.dk">
<applicationName name="dk.netarkivet.viewerproxy.ViewerProxyApplication">
<settings>
<common>
<jmx>
<port>8100</port>
<rmiPort>8200</rmiPort>
</jmx>
<http>
<port>8076</port>
</http>
</common>
<viewerproxy>
<baseDir>viewerproxy</baseDir>
</viewerproxy>
</settings>
</applicationName>
</deployMachine>
</thisPhysicalLocation>
</deployGlobal>
Rewrite deploy
Note that the newest version of the NetarchiveSuite code has eliminated use of the SideKick application. Therefore the special handling in starting and stopping this process is not necessary anymore.
Reuse deploy code from kb-doms (Royal Library Digital Object Managemnt System code):
- machine definition class (for different os)
- hirarchy modules at machine and application level
Reuse (and expand) SimpleXmlTree from the current NetarchiveSuite software.
parameter changes compares to old deploy:
- jmxremote.password must be generated from scratch instead of reading a file
- security.policy file must be given explicit as parameter
- log.prop file must be given explicit as parameter
NetarchiveSuite-package (zip file) must be given explicit as a new parameter
- Settings file is no longer needed as parameter
- Environment is no longer needed as parameter, it is readen from setting environmentName
Use following design:
- check input
- read and set settings from global level
- make physical location objects with from global level in it-config overwritten by specific physical location settings
- for each physical location
- read and set settings from physical location level
- make machine objects specified for the location correspong to the mashine OS
- for each machine
- read and set settings from machine level
- make application objects specified for the machine
- for each application
- read and set settings from applciation level
- use methods on objects in hierarchy to produce various scripts
When deployed is rewritten, the scripts for making multi-user test platform must also be updated
Order of implementation
- Location impl. + test could with big advantage be implemented first
- New deploy could be started as parallel activity, but will depend on
- changes in location settings
- (to some extend) introduction os applicationInstanceId setting