Differences between revisions 67 and 68
⇤ ← Revision 67 as of 2010-07-08 12:58:34 →
Size: 45071
Editor: SoerenCarlsen
Comment:
← Revision 68 as of 2010-08-16 10:24:49 → ⇥
Size: 45186
Editor: localhost
Comment: converted to 1.6 markup
Deletions are marked like this. Additions are marked like this.
Line 12: Line 12:
 * [http://kb-prod-udv-001.kb.dk/twiki/bin/edit/Netarkiv/SupportNetarchiveSuite Support] of released !NetarchiveSuite (http://netarchive.dk/suite).  * [[http://kb-prod-udv-001.kb.dk/twiki/bin/edit/Netarkiv/SupportNetarchiveSuite|Support]] of released !NetarchiveSuite (http://netarchive.dk/suite).
Line 14: Line 14:
 * Support of Wayback in the Netarchive.dk production site. See [:IntegrationOfWaybck:List of tasks] and [:AssignmentWaybackIntegration:Assignment] for Wayback Integration  * Support of Wayback in the Netarchive.dk production site. See [[IntegrationOfWaybck|List of tasks]] and [[AssignmentWaybackIntegration|Assignment]] for Wayback Integration
Line 18: Line 18:
 * Implementation according to [http://netarchive.dk/suite/Development implementation methodology]
 * Implementation and release test mainly in [http://www.google.com/calendar/render?gsessionid=tjZgbhGt6eNBB1mrlNwt3A intensive period]		  * Implementation according to [[http://netarchive.dk/suite/Development|implementation methodology]]
 * Implementation and release test mainly in [[http://www.google.com/calendar/render?gsessionid=tjZgbhGt6eNBB1mrlNwt3A|intensive period]]
Line 26: Line 26:
||1. [http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/SupportNetarchiveSuite Support] of released !NetarchiveSuite ||<style="text-align: center;">2 ||<style="text-align: center;">'''All (Google calender)''' ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Ongoing || ||1. [[http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/SupportNetarchiveSuite|Support]] of released !NetarchiveSuite ||<style="text-align: center;">2 ||<style="text-align: center;">'''All (Google calender)''' ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Ongoing ||
Line 31: Line 31:
||<bgcolor="#cccccc" style="text-align: center;">Prioritized bugs according to [https://gforge.statsbiblioteket.dk/tracker/index.php?group_id=7&atid=105 list] of priority 4 and priority 3 tasks. ||<bgcolor="#cccccc" style="text-align: center;">'''Total 5''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''!SubTotal 0''' ||<bgcolor="#cccccc" style="text-align: center;">.. ||<bgcolor="#cccccc" style="text-align: center;">'''-''' || ||<bgcolor="#cccccc" style="text-align: center;">Prioritized bugs according to [[https://gforge.statsbiblioteket.dk/tracker/index.php?group_id=7&atid=105|list]] of priority 4 and priority 3 tasks. ||<bgcolor="#cccccc" style="text-align: center;">'''Total 5''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''!SubTotal 0''' ||<bgcolor="#cccccc" style="text-align: center;">.. ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||
Line 33: Line 33:
||<style="vertical-align: top;">'''5 Module harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1856 Bug 1856]'' Schedule problem after first start on NAS 3.10.0. No schedule started ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">Bug seemingly fixed with patch release 3.10.2 ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''6 Module monitor?:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1895 Bug 1895]'' Running checksum gives Garbage Collector OutOfMemoryError and schedule stops ||<style="text-align: center;">1 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">Max heap for Bitarchive monitors raised to 1936MB in prod. Awaiting upgrade of java in production environment ||<bgcolor="#cccccc" style="text-align: center;"> ||		 ||<style="vertical-align: top;">'''5 Module harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1856|Bug 1856]]'' Schedule problem after first start on NAS 3.10.0. No schedule started ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">Bug seemingly fixed with patch release 3.10.2 ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''6 Module monitor?:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1895|Bug 1895]]'' Running checksum gives Garbage Collector OutOfMemoryError and schedule stops ||<style="text-align: center;">1 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">Max heap for Bitarchive monitors raised to 1936MB in prod. Awaiting upgrade of java in production environment ||<bgcolor="#cccccc" style="text-align: center;"> ||
Line 38: Line 38:
||<style="vertical-align: top;">'''8. Module harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=688 Feature request 688]'' hosts-report should be IDNA decoded when writing harvestInfo to the DB ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''9. Module Access:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=823 Bug 823]'' No index = Internal server error ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''10. Module Monitor:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1756 Bug 1756]'' JMX status page does not update when a new application is started on previously used JMX port ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''11. Module Archive:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1782 Bug 1782]'' Same datetime repeated many times, while logging batch checksum of files ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''12. Module Documentation:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1779 Bug 1779]'' Improve documentation of the additional tools ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">FIXED ||
||<style="vertical-align: top;">'''13. Module Archive: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1708 Bug 1708]'' bitpreservation logic offers "add to archive" for file that is not in either location ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''14. Module Documentation:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1732 Bug 1732]'' !LocalArcRepositoryClient not documented ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">Fixed in System Design 3.12 manual ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''15. Module Archive:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1260 Bug 1260]'' Too much and wrong feedback information on "Missing pages" ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''16. Module Monitor:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1205 Bug 1205]'' Security policy for unit tests contains hardcoded path to development environment ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''17. Module Archive:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1193 Bug 1193]'' Exceptions from !FileBatchJob stop batch job processing ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||		 ||<style="vertical-align: top;">'''8. Module harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=688|Feature request 688]]'' hosts-report should be IDNA decoded when writing harvestInfo to the DB ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''9. Module Access:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=823|Bug 823]]'' No index = Internal server error ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''10. Module Monitor:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1756|Bug 1756]]'' JMX status page does not update when a new application is started on previously used JMX port ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''11. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1782|Bug 1782]]'' Same datetime repeated many times, while logging batch checksum of files ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''12. Module Documentation:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1779|Bug 1779]]'' Improve documentation of the additional tools ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">FIXED ||
||<style="vertical-align: top;">'''13. Module Archive: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1708|Bug 1708]]'' bitpreservation logic offers "add to archive" for file that is not in either location ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''14. Module Documentation:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1732|Bug 1732]]'' !LocalArcRepositoryClient not documented ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">Fixed in System Design 3.12 manual ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''15. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1260|Bug 1260]]'' Too much and wrong feedback information on "Missing pages" ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''16. Module Monitor:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1205|Bug 1205]]'' Security policy for unit tests contains hardcoded path to development environment ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''17. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1193|Bug 1193]]'' Exceptions from !FileBatchJob stop batch job processing ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
Line 49: Line 49:
||<bgcolor="#cccccc" style="text-align: center;">Prioritized Feature Requests according to [:TaskTableFromMay2009Workshop:list] of priority 4 and priority 3 tasks ||<bgcolor="#cccccc" style="text-align: center;">'''Total 21''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''!SubTotal 21''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">'''-''' || ||<bgcolor="#cccccc" style="text-align: center;">Prioritized Feature Requests according to [[TaskTableFromMay2009Workshop|list]] of priority 4 and priority 3 tasks ||<bgcolor="#cccccc" style="text-align: center;">'''Total 21''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||<bgcolor="#cccccc" style="text-align: center;">'''!SubTotal 21''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">'''-''' ||
Line 51: Line 51:
||<style="vertical-align: top;">'''18. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1696 Feature request 1696]'' Ingest domain seed URLs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''19. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1688 Feature request 1688]'' Monitoring broad crawls. ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152. ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''20. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1134 Feature request 1134]'' Filter job lists by category '''''' ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21. Module Harvester: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1668 Feature request 1668]'' Paginate and make sortable and searchable the list of jobs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21a. Module Harvester: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1924 Feature request 1924]'' Allow to search a domain in active jobs (in case of webmaster complain) ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21b. Common: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1925 Feature request 1925]'' PostgreSQL connectivity (using the PostgreSQL driver version 8.4 - JDBC 4) ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21c. Module Harvester: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1926 Feature request 1926]'' Ability to disable the inactivity check ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Waiting for review ||
||<style="vertical-align: top;">'''21d. Module Harvester: '''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1927 Feature request 1927]'' Delay job end to allow Heritrix report generation ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21e. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1928 Feature request 1928]'' Ability to easily resubmit a selection of failed jobs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">? ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''21f. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1929 Feature request 1929]'' 15 second level TLD related to the .fr and .re domains ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''21g. Module Harvester:''' ''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1930 Feature request 1930]'' Ability to implement a different crawl control loop via HeritrixLauncher / new Heritrix JMX controller ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''22. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1813 Feature request 1813]'' An extra resubmit button to make it visible which jobs have already been handled ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed . ||
||<style="vertical-align: top;">'''23. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=929 Feature request 929]'' Documentation needed for how we split jobs (incl. maybe additional splitting modularity) ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''24. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1774 Feature request 1774]'' Stop using the JMS queues for queuing snapshot harvests ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''25. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1896 Feature request 1896]'' Crawl of password protected FTP-sites ||<style="text-align: center;">2 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High Priority ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''25a. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1951 Feature request 1951]'' Upgrade to Heritrix 1.14.4 ||<style="text-align: center;">2 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High Priority ||<bgcolor="#cccccc" style="text-align: center;">'''Fixed ''' ||		 ||<style="vertical-align: top;">'''18. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1696|Feature request 1696]]'' Ingest domain seed URLs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''19. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1688|Feature request 1688]]'' Monitoring broad crawls. ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152. ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''20. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1134|Feature request 1134]]'' Filter job lists by category '''''' ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21. Module Harvester: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1668|Feature request 1668]]'' Paginate and make sortable and searchable the list of jobs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21a. Module Harvester: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1924|Feature request 1924]]'' Allow to search a domain in active jobs (in case of webmaster complain) ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21b. Common: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1925|Feature request 1925]]'' PostgreSQL connectivity (using the PostgreSQL driver version 8.4 - JDBC 4) ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-151 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21c. Module Harvester: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1926|Feature request 1926]]'' Ability to disable the inactivity check ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Waiting for review ||
||<style="vertical-align: top;">'''21d. Module Harvester: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1927|Feature request 1927]]'' Delay job end to allow Heritrix report generation ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''21e. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1928|Feature request 1928]]'' Ability to easily resubmit a selection of failed jobs ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">? ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''21f. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1929|Feature request 1929]]'' 15 second level TLD related to the .fr and .re domains ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas/Sara ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''21g. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1930|Feature request 1930]]'' Ability to implement a different crawl control loop via HeritrixLauncher / new Heritrix JMX controller ||<style="text-align: center;">? ||<style="text-align: center;">Nicolas ||<style="vertical-align: top;"> ||<style="vertical-align: top;"> ||<style="vertical-align: top;">http://kb-prod-udv-001.kb.dk:8060/cru/NS-152 ||<bgcolor="#cccccc" style="text-align: center;">Review and followup done ||
||<style="vertical-align: top;">'''22. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1813|Feature request 1813]]'' An extra resubmit button to make it visible which jobs have already been handled ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed . ||
||<style="vertical-align: top;">'''23. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=929|Feature request 929]]'' Documentation needed for how we split jobs (incl. maybe additional splitting modularity) ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''24. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1774|Feature request 1774]]'' Stop using the JMS queues for queuing snapshot harvests ||<style="text-align: center;">? ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''25. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1896|Feature request 1896]]'' Crawl of password protected FTP-sites ||<style="text-align: center;">2 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High Priority ||<bgcolor="#cccccc" style="text-align: center;">Fixed ||
||<style="vertical-align: top;">'''25a. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1951|Feature request 1951]]'' Upgrade to Heritrix 1.14.4 ||<style="text-align: center;">2 ||<style="text-align: center;">SVC ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High Priority ||<bgcolor="#cccccc" style="text-align: center;">'''Fixed ''' ||
Line 68: Line 68:
||<style="vertical-align: top;">'''26.Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1774 Feature request 1774]'' Stop using the JMS queues for queuing snapshot harvests '''''' ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed '''''' ||
||<style="vertical-align: top;">'''27. Module Harvester''':''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1681 Feature request 1681]''''' '''Add seed to DB via webservice (via Browser Extension/Rich Client)''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''28. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1682 Feature request 1682]'' Statistics (DB access, scripts, batch jobs ....) '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''29. Module Harvester''':''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1683 Feature request 1683]'' Util for regenerate admin.data file''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''30. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1684 Feature request 1684]'' Activity when domain is to be crawled. One table for seed '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''31. Module Archive''':''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1743 Feature request 1743]'' When accessing Bitpreservation this takes really long time ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''32. Module Harvester:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1120 Feature request 1120]'' Crawlertrap info should be shareable between institutions '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">SVC will add comments to this FR. Might be an easy solution to share Crawlertraps by emailing files with crawler trap informations. ||<bgcolor="#cccccc" style="text-align: center;">Redundant (Copy of ??) ||
||<style="vertical-align: top;">'''33. Module Harvester''':''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1066 Feature request 1066]'' Show whether seed URL existed''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''34. Module Archive:'''''[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1809 Feature request 1809]'' Write assignment for improving batchjob interface '''''' ||<style="text-align: center;"> ||<style="text-align: center;">JOLF ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">Hign priority ||<bgcolor="#cccccc" style="text-align: center;">Done ||		 ||<style="vertical-align: top;">'''26.Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1774|Feature request 1774]]'' Stop using the JMS queues for queuing snapshot harvests '''''' ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed '''''' ||
||<style="vertical-align: top;">'''27. Module Harvester''':''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1681|Feature request 1681]]''''' '''Add seed to DB via webservice (via Browser Extension/Rich Client)''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''28. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1682|Feature request 1682]]'' Statistics (DB access, scripts, batch jobs ....) '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''29. Module Harvester''':''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1683|Feature request 1683]]'' Util for regenerate admin.data file''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''30. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1684|Feature request 1684]]'' Activity when domain is to be crawled. One table for seed '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''31. Module Archive''':''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1743|Feature request 1743]]'' When accessing Bitpreservation this takes really long time ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''32. Module Harvester:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1120|Feature request 1120]]'' Crawlertrap info should be shareable between institutions '''''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">SVC will add comments to this FR. Might be an easy solution to share Crawlertraps by emailing files with crawler trap informations. ||<bgcolor="#cccccc" style="text-align: center;">Redundant (Copy of ??) ||
||<style="vertical-align: top;">'''33. Module Harvester''':''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1066|Feature request 1066]]'' Show whether seed URL existed''' ''' ||<style="text-align: center;"> ||<style="text-align: center;">Andreas ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''34. Module Archive:'''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1809|Feature request 1809]]'' Write assignment for improving batchjob interface '''''' ||<style="text-align: center;"> ||<style="text-align: center;">JOLF ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">Hign priority ||<bgcolor="#cccccc" style="text-align: center;">Done ||
Line 81: Line 81:
||<style="vertical-align: top;">'''36. WARC:''' Finalize [:AssignmentHarvester2:Assigment] for Harvester for support of WARC format ||<style="text-align: center;">? ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed || ||<style="vertical-align: top;">'''36. WARC:''' Finalize [[AssignmentHarvester2|Assigment]] for Harvester for support of WARC format ||<style="text-align: center;">? ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
Line 83: Line 83:
||<style="vertical-align: top;">'''37. Archive: '''Implement [:AssignmentGroupB2:Assignment B.2.3] - Use segments in bitarchives ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''38. Archive: '''Implement [:AssignmentGroupB2:Assignment B.2.4] - Write !BitPreservation scheduler ||<style="text-align: center;">5 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''39. Archive: '''Implement [:AssignmentGroupB2:Assignment B.2.5] - Write !BitPreservation webinterface ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||		 ||<style="vertical-align: top;">'''37. Archive: '''Implement [[AssignmentGroupB2|Assignment B.2.3]] - Use segments in bitarchives ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''38. Archive: '''Implement [[AssignmentGroupB2|Assignment B.2.4]] - Write !BitPreservation scheduler ||<style="text-align: center;">5 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
||<style="vertical-align: top;">'''39. Archive: '''Implement [[AssignmentGroupB2|Assignment B.2.5]] - Write !BitPreservation webinterface ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">Postponed ||
Line 89: Line 89:
||<bgcolor="#cccccc" style="text-align: center;">[http://netarkivet.dk/netarkivet/index.php?title=Kendte_problemer Crawl-problems] (Netarchive.dk) . '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''''''''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">- '''''' || ||<bgcolor="#cccccc" style="text-align: center;">[[http://netarkivet.dk/netarkivet/index.php?title=Kendte_problemer|Crawl-problems]] (Netarchive.dk) . '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''''''''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||
Line 91: Line 91:
||<style="vertical-align: top;">40. [http://netarkivet.dk/netarkivet/index.php?title=Dinby.dk dinby.dk] 2009-02-17 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' ||
||<style="vertical-align: top;">41. [http://netarkivet.dk/netarkivet/index.php?title=Kino.dk Kino.dk] 2009-03-25 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review '''''' ||
||<style="vertical-align: top;">42. [http://netarkivet.dk/netarkivet/index.php?title=Webmuseum.re-cph.com Webmuseum.re-cph.com] 2009-08-04 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review ||
||<style="vertical-align: top;">43. [http://netarkivet.dk/netarkivet/index.php?title=Epn.dk Epn.dk] 2009-08-30 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' ||
||<style="vertical-align: top;">44. [http://netarkivet.dk/netarkivet/index.php?title=statstidende.dk Statstidende.dk] ||<style="text-align: center;">0.5 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review '''''' ||
||<style="vertical-align: top;">45. [http://netarkivet.dk/netarkivet/index.php?title=seoghoer.dk seoghoer.dk] ||<style="text-align: center;">0.5 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review'''''' ||		 ||<style="vertical-align: top;">40. [[http://netarkivet.dk/netarkivet/index.php?title=Dinby.dk|dinby.dk]] 2009-02-17 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' ||
||<style="vertical-align: top;">41. [[http://netarkivet.dk/netarkivet/index.php?title=Kino.dk|Kino.dk]] 2009-03-25 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review '''''' ||
||<style="vertical-align: top;">42. [[http://netarkivet.dk/netarkivet/index.php?title=Webmuseum.re-cph.com|Webmuseum.re-cph.com]] 2009-08-04 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review ||
||<style="vertical-align: top;">43. [[http://netarkivet.dk/netarkivet/index.php?title=Epn.dk|Epn.dk]] 2009-08-30 ||<style="text-align: center;">1 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;">1 ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' ||
||<style="vertical-align: top;">44. [[http://netarkivet.dk/netarkivet/index.php?title=statstidende.dk|Statstidende.dk]] ||<style="text-align: center;">0.5 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review '''''' ||
||<style="vertical-align: top;">45. [[http://netarkivet.dk/netarkivet/index.php?title=seoghoer.dk|seoghoer.dk]] ||<style="text-align: center;">0.5 ||<style="text-align: center;">CSR ||<style="text-align: center;">SVC ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">Awaiting review'''''' ||
Line 98: Line 98:
||<style="vertical-align: top;">46. [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1881 Feature Request 1881] Quality assurance through a batchjob interface ||<style="text-align: center;">0.5 ||<style="text-align: center;">JOLF ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">In progress || ||<style="vertical-align: top;">46. [[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1881|Feature Request 1881]] Quality assurance through a batchjob interface ||<style="text-align: center;">0.5 ||<style="text-align: center;">JOLF ||<style="text-align: center;">CSR ||<style="text-align: center;"> ||<style="text-align: center;">High priority ||<bgcolor="#cccccc" style="text-align: center;">In progress ||
Line 106: Line 106:
||<bgcolor="#cccccc" style="text-align: center;">Converting old Web collections to Netarchive.dk. See [http://udvikling.kb.dk/cvsshadow/digiliv/ProjektDokumenter/omkostninger%20ved%20indsamling%20af%20gammelt%20materiale-3.doc proposal]. These task will be independent of !NetarchiveSuite code-freeze. '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''''''''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">- '''''' || ||<bgcolor="#cccccc" style="text-align: center;">Converting old Web collections to Netarchive.dk. See [[http://udvikling.kb.dk/cvsshadow/digiliv/ProjektDokumenter/omkostninger%20ved%20indsamling%20af%20gammelt%20materiale-3.doc|proposal]]. These task will be independent of !NetarchiveSuite code-freeze. '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||<bgcolor="#cccccc" style="text-align: center;">Total x '''''''''''' ||<bgcolor="#cccccc" style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">- '''''' ||
Line 122: Line 122:
||<style="vertical-align: top;">66. Prepare [http://netarchive.dk/suite/Iteration43Releasetest release test] ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">OK || ||<style="vertical-align: top;">66. Prepare [[http://netarchive.dk/suite/Iteration43Releasetest|release test]] ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;">6 ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">OK ||
Line 126: Line 126:
||<style="vertical-align: top;">67. Execute [http://netarchive.dk/suite/Iteration43Releasetest release test]. ||<style="text-align: center;">12 ||<style="text-align: center;">TLR ||<style="text-align: center;">All ||<style="text-align: center;">12 ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">In progres || ||<style="vertical-align: top;">67. Execute [[http://netarchive.dk/suite/Iteration43Releasetest|release test]]. ||<style="text-align: center;">12 ||<style="text-align: center;">TLR ||<style="text-align: center;">All ||<style="text-align: center;">12 ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">In progres ||
Line 133: Line 133:
||<style="vertical-align: top;">70. Define goals for [http://netarchive.dk/suite/Iteration44TaskList Iteration 44 task list] ||<style="text-align: center;"> ||<style="text-align: center;">CHH ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' || ||<style="vertical-align: top;">70. Define goals for [[http://netarchive.dk/suite/Iteration44TaskList|Iteration 44 task list]] ||<style="text-align: center;"> ||<style="text-align: center;">CHH ||<style="text-align: center;"> ||<style="text-align: center;"> ||<style="text-align: center;"> ||<bgcolor="#cccccc" style="text-align: center;">.. '''''' ||

Task list and timetable for iteration 43

Status

OK/Not Ok

1. Highlights approved

OK

2. Assignment of tasks

OK

3. Task list and time table approved

OK

4. Implementation phase started

OK

5. Release test phase started

OK

6. Assignment phase for next iteration started

OK

7. Iteration 43 completed

Highlights for Iteration

  • Integrate BNF features in main branch

  • Support of released NetarchiveSuite (http://netarchive.dk/suite).

  • Implement prioritized bugs and feature requests

  • Support of Wayback in the Netarchive.dk production site. See List of tasks and Assignment for Wayback Integration

  • Migration of old Web materials to Netarchive.dk
  • Iteration 43 is planned as a development release.

Development procedure

Table of tasks

Tasks for iteration 43. Updated 9. June 2010

Estimate md

Main responsible

Reviewer

Remaining md at 9. June 2010

Comments

Status

Implementation phase (task x-n)

Open Source release + bugs and feature request

Total 3

-

-

Total 3

-

Support of Open Source Release

1. Support of released NetarchiveSuite

2

All (Google calender)

Ongoing

2. Implement translateprocess. Adjustment to Open Source partners.

1

CSR

SVC

-

3. Maintain French Translation files.

1

Nicolas/Sara

SVC

See also Task 22

-

4. Maintain Italian and german Translation files.

1

Andreas/Eleonora

SVC

See also Task 22

-

Bugs and Feature requests

Prioritized bugs according to list of priority 4 and priority 3 tasks.

Total 5

-

-

SubTotal 0

..

-

Priority 5 bug

5 Module harvester: Bug 1856 Schedule problem after first start on NAS 3.10.0. No schedule started

1

CSR

SVC

Bug seemingly fixed with patch release 3.10.2

Fixed

6 Module monitor?: Bug 1895 Running checksum gives Garbage Collector OutOfMemoryError and schedule stops

1

SVC

CSR

Max heap for Bitarchive monitors raised to 1936MB in prod. Awaiting upgrade of java in production environment

Priority 4 bugs

7..

.

Priority 3 bugs

8. Module harvester: Feature request 688 hosts-report should be IDNA decoded when writing harvestInfo to the DB

Postponed

9. Module Access: Bug 823 No index = Internal server error

Postponed

10. Module Monitor: Bug 1756 JMX status page does not update when a new application is started on previously used JMX port

Postponed

11. Module Archive: Bug 1782 Same datetime repeated many times, while logging batch checksum of files

Postponed

12. Module Documentation: Bug 1779 Improve documentation of the additional tools

FIXED

13. Module Archive: Bug 1708 bitpreservation logic offers "add to archive" for file that is not in either location

Postponed

14. Module Documentation: Bug 1732 LocalArcRepositoryClient not documented

Fixed in System Design 3.12 manual

Fixed

15. Module Archive: Bug 1260 Too much and wrong feedback information on "Missing pages"

Postponed

16. Module Monitor: Bug 1205 Security policy for unit tests contains hardcoded path to development environment

Postponed

17. Module Archive: Bug 1193 Exceptions from FileBatchJob stop batch job processing

Postponed

..

Prioritized Feature Requests according to list of priority 4 and priority 3 tasks

Total 21

-

-

SubTotal 21

-

Priority 4 Feature request

18. Module Harvester: Feature request 1696 Ingest domain seed URLs

?

Nicolas

SVC

Postponed

19. Module Harvester: Feature request 1688 Monitoring broad crawls.

?

Nicolas/Sara

SVC

http://kb-prod-udv-001.kb.dk:8060/cru/NS-152.

Review and followup done

20. Module Harvester: Feature request 1134 Filter job lists by category

?

Nicolas/Sara

SVC

http://kb-prod-udv-001.kb.dk:8060/cru/NS-151

Review and followup done

21. Module Harvester: Feature request 1668 Paginate and make sortable and searchable the list of jobs

?

Nicolas/Sara

SVC

http://kb-prod-udv-001.kb.dk:8060/cru/NS-151

Review and followup done

21a. Module Harvester: Feature request 1924 Allow to search a domain in active jobs (in case of webmaster complain)

?

Nicolas/Sara

http://kb-prod-udv-001.kb.dk:8060/cru/NS-151

Review and followup done

21b. Common: Feature request 1925 PostgreSQL connectivity (using the PostgreSQL driver version 8.4 - JDBC 4)

?

Nicolas

http://kb-prod-udv-001.kb.dk:8060/cru/NS-151

Review and followup done

21c. Module Harvester: Feature request 1926 Ability to disable the inactivity check

?

Nicolas

http://kb-prod-udv-001.kb.dk:8060/cru/NS-152

Waiting for review

21d. Module Harvester: Feature request 1927 Delay job end to allow Heritrix report generation

?

Nicolas/Sara

http://kb-prod-udv-001.kb.dk:8060/cru/NS-152

Review and followup done

21e. Module Harvester: Feature request 1928 Ability to easily resubmit a selection of failed jobs

?

Nicolas/Sara

?

Postponed

21f. Module Harvester: Feature request 1929 15 second level TLD related to the .fr and .re domains

?

Nicolas/Sara

Fixed

21g. Module Harvester: Feature request 1930 Ability to implement a different crawl control loop via HeritrixLauncher / new Heritrix JMX controller

?

Nicolas

http://kb-prod-udv-001.kb.dk:8060/cru/NS-152

Review and followup done

22. Module Harvester:Feature request 1813 An extra resubmit button to make it visible which jobs have already been handled

?

SVC

CSR

Postponed .

23. Module Harvester:Feature request 929 Documentation needed for how we split jobs (incl. maybe additional splitting modularity)

?

SVC

CSR

Postponed

24. Module Harvester:Feature request 1774 Stop using the JMS queues for queuing snapshot harvests

?

SVC

CSR

Postponed

25. Module Harvester:Feature request 1896 Crawl of password protected FTP-sites

2

SVC

CSR

High Priority

Fixed

25a. Module Harvester:Feature request 1951 Upgrade to Heritrix 1.14.4

2

SVC

CSR

High Priority

Fixed

Priority 3 Feature request

26.Module Harvester:Feature request 1774 Stop using the JMS queues for queuing snapshot harvests

Postponed

27. Module Harvester:Feature request 1681 Add seed to DB via webservice (via Browser Extension/Rich Client)

Andreas

Postponed

28. Module Harvester:Feature request 1682 Statistics (DB access, scripts, batch jobs ....)

Andreas

Postponed

29. Module Harvester:Feature request 1683 Util for regenerate admin.data file

Andreas

Postponed

30. Module Harvester:Feature request 1684 Activity when domain is to be crawled. One table for seed

Andreas

Postponed

31. Module Archive:Feature request 1743 When accessing Bitpreservation this takes really long time

Andreas

Postponed

32. Module Harvester:Feature request 1120 Crawlertrap info should be shareable between institutions

Andreas

SVC will add comments to this FR. Might be an easy solution to share Crawlertraps by emailing files with crawler trap informations.

Redundant (Copy of ??)

33. Module Harvester:Feature request 1066 Show whether seed URL existed

Andreas

Postponed

34. Module Archive:Feature request 1809 Write assignment for improving batchjob interface

JOLF

Hign priority

Done

Roadmap tasks

Total 52?

-

-

Total 8,5

-

Tasks from ...

35. QA: Assignment for enhanced QA tools

2

SVC

CSR

High priority

Postponed

36. WARC: Finalize Assigment for Harvester for support of WARC format

?

Postponed

37. Archive: Implement Assignment B.2.3 - Use segments in bitarchives

6

Postponed

38. Archive: Implement Assignment B.2.4 - Write BitPreservation scheduler

5

Postponed

39. Archive: Implement Assignment B.2.5 - Write BitPreservation webinterface

6

Postponed

-

..

..

Crawl-problems (Netarchive.dk) .

Total x

-

-

Total x

-

Focus on following crawl-problems

40. dinby.dk 2009-02-17

1

CSR

SVC

1

High priority

..

41. Kino.dk 2009-03-25

1

CSR

SVC

1

High priority

Awaiting review

42. Webmuseum.re-cph.com 2009-08-04

1

CSR

SVC

1

High priority

Awaiting review

43. Epn.dk 2009-08-30

1

CSR

SVC

1

High priority

..

44. Statstidende.dk

0.5

CSR

SVC

High priority

Awaiting review

45. seoghoer.dk

0.5

CSR

SVC

High priority

Awaiting review

45.b http://netarkivet.dk/netarkivet/index.php?title=Berlingske.dk

2

CSR

High priority

Awaiting review

46. Feature Request 1881 Quality assurance through a batchjob interface

0.5

JOLF

CSR

High priority

In progress

Wayback/Nutchwax tasks independent of NetarchiveSuite code-freeze.

Total x

-

-

Total x

-

Tasks from ...

47. Wayback: Implement Indexer

11

CSR

SVC

High Priotrity

Done

48. Wayback: Implement Index Aggregator

4

MKS

JOLF

High Priority

In progress

58.Wayback: Documentation of Indexer/Aggregator

2

CSR

JOLF

1

High priority

Postponed

59. Wayback deploy: If not possible with Jetty then test of Wayback deploy using Tomcat

2

JOLF

CSR

2

High priority

In progress

..

Converting old Web collections to Netarchive.dk. See proposal. These task will be independent of NetarchiveSuite code-freeze.

Total x

-

-

Total x

-

Tasks from ...

60. Old Web collection: Old KB Webarchive

SVC

JOLF

High priority

In progress

61. Old Web collection: Old Webarchive from Niels Brugger collected by HTTrack

HBK

SVC

High priority

In progress

62. Old Web collection: Prepare ingest of extracted data from Internet Archive into Netarkivet.dk

SVC

HBK

Wait for IA correction

63. Old Web collection: Ingest received data from Internet Archive into Netarkivet.dk

CLO

SVC

-.

Common tasks calculated as implementation tasks

Total x

-

-

Total x

-

Others

Total x

-

-

SubTotal 2

-

64. Upgrade: New KB-PROD-UDV

5

SVC

TLR

Postponed

65. Batch: Create/execute a batch test script specified by 1 or 2 researches

2

JOLF

TLR

Postponed

..

..

-

..

..

Prepare release test

Total x

-

-

SubTotal 12

-

66. Prepare release test

6

6

OK

Available man-days for implementation phase

Total x

-

-

Total x

-

Release test phase (task ...)

Release test

Total x

-

-

Total 12

-

67. Execute release test.

12

TLR

All

12

In progres

'

..

Release notes

Total x

-

-

Total 0,5

-

68. Write release note

0,5

SVC

Awaiting end of code freeze

Available man-days for release test phase

Total x

-

-

Total 10

-

Assignment phase for next iteration (task ...)

69. Component bug/feature fix/management

QA

..

70. Define goals for Iteration 44 task list

CHH

..

71. Presentation of goals and tasks for Iteration 43. Achieve a common understanding of the purpose of the iteration and each task on status meeting

SVC

..

72. Assignment of tasks, bugs and feature request

QA

..

73. Update release test procedure

TLR

..

Available man-days for assigment phase

Total x

-

-

Total 22

-

Timetable

Timetable iteration 43. Updated 4. May 2010

Start time

End time

Responsible

Baseline 4. May 2010. Start time

Baseline 4. May 2010'. End time

1. Implementation of decided tasks

4. May 2010

31. May 2010

4. May 2010

31. May 2010

2. Code freeze. Create the build for release test and notify when build is ready

9. June 2010

SVC

1. June 2010

3. Release test

9. June 2010

11. June 2010

TLR

1. June 2010

3. June 2010

4. Code unfreeze

14. June 2010

SVC

4. June 2010

5. Assignments, bug components and bug fixes

10. June 2010

11. June 2010

2. June 2010

3. June 2010

Iteration43TaskList (last edited 2010-08-16 10:24:49 by localhost)