Follow-up on settings design discussions

We discussed our settings structure based on the presentation given by KFC

We reached no common consensus on the results of our settings discussions, but some common themes were recurring, and I will try to enumerate and comment on them.

Discussion topics

Default settings

A possible way to make settings files easier to understand and maintain is to have resonable standards packed with the NetarchiveSuite distribution, and only override a subset of settings for the individual applications as needed.

We discussed a few ways we could make default settings, and there seemed to be general agreement that having a default xml file compiled in the jar files might be a reasonable idea

Module splitting

The discussion on how to move settings from the common module to the modules where the settings were actually used didn't touch many specifics, but there was agreement that something needed to be done.

Note that the idea above indicates that we would need to compile a settings file in each module, and somehow use all these default settings.

Settings format

Although everyone agreed that the corrent format of settings wasn't very good, there were no new ideas presented on how to structure our settings.

The major problem we have is that there is no concept of pairing of settings (e.g. these ports belong to this host).

The idea of adding attributes was briefly discussed, but we didn't actually get the idea very far.

Pluggable settings

The idea of extending our settings structure with settings needed by plugins was again discussed without reaching a conclusion. We discussed the solution of Heritrix of a module defining its own settings, and saw pros and cons. The major problem is that any setting needed by a plugin would be hard to validate before the plugin was already loaded.

Changing settings at runtime

It was generally agreed that changing settings with JMX while the applications were running was desirable. However, it is also difficult, since the behaviour is not always obvious if we change the settings at runtime.

This is both due to poor usage of settings (for instance reading them once and storing them in a static field or a field in a singleton) and due to situations where it is not obvious what should happen (for instance what should happen if you change the JMS broker while waiting for a reply on a message).

It would require review of each individual use of a setting to reach this goal in general.

Reaching a common base

Module specific settings backed by defaults

It seems the "easy" first step is to have separate settings files for each module, with a default settings file packed in the jar file, and *one* overriding settings.xml file, that replaces relevant settings. We should probably support referring to multiple settings files, but encourage having one per application.

The required work will be the following:

SettingsStructure.java needs to look for settings first overridden on command line, then in the file system, and then in the classpath.

The settings.xml file needs to be split up into settings for each module.

The settings constants defining settings must be moved to each module. Probably, constants defining plugins' setting names should be moved to the plugin classes.

The settings.xsd file will probably be deprecated, the new settings structure would be difficult to describe in XML schema in any reasonable way.

Deploy needs to be rethought to only generate a minimum settings extension for each application.

Making settings settable with JMX

It would be fairly easy to both be able to realod settings if the files are changed on disk, and update them using JMX.

The difficult part would be to review the code for settings that should be updatable but are poorly programmed.

The settings where the meaning of updating them is unclear should simply be documented not to support update on runtime.

Moving on

It is still unresolved how we handle groupings of settings, and if a schema defining settings in a manner extensible by plugins is possible.

Evaluation of existing configuration frameworks

Spring has been suggested on several occasions as a configuration framework that solves a lot of the problems that we have with settings and deployment. LC has done some research into its capabilities, and is of the opinion that it is not a good match for our needs. Using Spring well would entail some major restructurings, and would actually cause more complex factory setups than are currently required.