Yellow Notes for Potential Future Developments

These are the notes from the "potential future developments" board at the [:AgendaWorkshopSeptember2007:workshop 2007]. Points noted as being important for some participants are marked by being bolded

Points that needs further explanation is marked with ?

Roadmap

• Java upgrade to Java 1.6
• Encrypting password for bit archive

• New Heritrix integration

Quality Assurance

• Automatic filtering of collected URLs
• Use scope in QA ?

Extra Metadata

• Contractual Information
• Naming of jobs
• Look up of WHOIS data to add to metadata
• Access to Heritrix logs via Harvest definition interface
• A way to show count of stored bytes (after deduplication) and information on deduplication
• Subdomains from logs

Harvest Tuning

• Possibility to inactivate domains

• Removing seeds from database

• Automatically check seeds

• Exclude domains from snapshot harvest

• Alias detection tool integrated in system

• Domain implementation

• QueueAssignmentPolicy part of plugin ???

• Global crawlertraps

• Count inline images in the same domain

Heritrix

• Pause/stop harvest from User Interface

• Check email address using Heritrix method
• Internationalisation of Heritrix (not seen as important)
• Operator in metadata
• Heritrix timeout challenge
• Meta language for Heritix templates

• Sharing of crawlertraps (Heritrix)

• Arcreader and Arcwriter separately

• Heritrix split
• Use Heritrix deduplcation module

• Arcfile naming

Infrastructure

• Take down parts and start them again
• Use of SPRING for deploy
• Automatic start of application that is down

Database

• Specify wanted backup directory
• Elaborate on update procedure when database has changed
• Oracle implementation as plugin

• All SQL out in DBSpecific for plugin

Open Source Procedure

• Control on formats for patches
• What about new translations (in release information)
• User traning material
• Documentaiton on listen to queues

User Interface

• Neater user interface

• Click pattern

• Filter job lists by category

• Password & user nameto control settings

• Calendar view of jobs

• For people with special needs
• Provinance

End User access