== Task list and timetable for iteration 40 == ||'''Status''' ||'''OK/Not Ok''' || ||1. Highlights approved ||'''OK''' || ||2. Assignment of tasks ||'''OK''' || ||3. Task list and time table approved ||'''OK''' || ||4. Implementation phase started ||'''OK''' || ||5. Release test phase started ||'''OK''' || ||6. Assignment phase for next iteration started ||'''OK''' || ||7. Iteration 40 completed || || === Highlights for Iteration === * [[http://kb-prod-udv-001.kb.dk/twiki/bin/edit/Netarkiv/SupportNetarchiveSuite|Support]] of released !NetarchiveSuite (http://netarchive.dk/suite). * Enhance !NetarchiveSuite wiki according to [[UpdateNetarchiveSuiteWiki|decided structure]]. * Implement prioritized bugs and feature requests * Enhancement of Batch support * Review of WARC Tools Phase III SRS * Support of Wayback in the Netarchive.dk production site. See [[IntegrationOfWaybck|List of tasks]] and [[AssignmentWaybackIntegration|Assignment]] for Wayback Integration * Migration of old Web materials to Netarchive.dk * Iteration 40 is planned as a development release. === Development procedure === * Implementation according to [[http://netarchive.dk/suite/Development|implementation methodology]] * Implementation and release test mainly in [[http://www.google.com/calendar/render?gsessionid=tjZgbhGt6eNBB1mrlNwt3A|intensive period]] * Target release: December 2009 === Table of tasks === ||'''Tasks for iteration 40. Updated 15. December 2009''' ||'''Estimate md''' ||'''Main responsible''' ||'''Reviewer ''' ||<10% bgcolor="#cccccc" style="TEXT-ALIGN: center">'''Remaining md at 15. December 2009''' ||<20% bgcolor="#cccccc" style="TEXT-ALIGN: center">'''Comments''' ||'''Status''' || ||||||||||||||||||||||'''Implementation phase (task x-n)''' || ||'''Open Source release + bugs and feature request''' ||'''Total 3''' ||'''-''' ||'''-''' ||'''Total 3''' || ||'''-''' || ||||||||||||||||||||||'''Support of Open Source Release''' || ||1. [[http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/SupportNetarchiveSuite|Support]] of released !NetarchiveSuite ||2 ||'''All (Google calender)''' || ||2 || ||Ongoing || ||2. Implement translateprocess. Adjustment to Open Source partners. ||1 ||CSR ||SVC || || ||- || ||||||||||||||||||||||'''Bugs and Features requests''' || ||Prioritized bugs according to [[https://gforge.statsbiblioteket.dk/tracker/index.php?group_id=7&atid=105|list]] of priority 4 and priority 3 tasks. ||'''Total 5''' ||'''-''' ||'''-''' ||'''!SubTotal 0''' ||.. ||'''-''' || ||||||||||||||||||||||'''Priority 4 bugs''' || ||'''3. Module Access:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1758|Bug 1758]]'' UrlCanonicalizerFactory falls back to default value silently ||0,5 ||CSR ||SVC ||0 ||High Priority ||OK || ||'''4. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1777|Bug 1777]]'' Add event seeds only accepts a very short list of seeds ||3 ||SVC ||HBK ||0 ||High priority ||OK || ||||||||||||||||||||||'''Priority 3 bugs''' || ||'''5. Module Access:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1796|Bug 1796]]'' Configurations manual doesn't include anything about setting up wayback ||1 ||HBK ||SVC ||0 || ||OK || ||'''6. Module Access:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=823|Bug 823]]'' No index = Internal server error || || || || || || || ||'''7. Module Monitor:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1756|Bug 1756]]'' JMX status page does not update when a new application is started on previously used JMX port || || || || || || || ||'''8. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1782|Bug 1782]]'' Same datetime repeated many times, while logging batch checksum of files || || || || || || || ||'''9. Module Documentation:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1779|Bug 1779]]'' Improve documentation of the additional tools || || || || || || || ||'''10. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1764|Bug 1764]]'' Poor information on failed batch job || || || || || || || ||'''11. Module Documentation:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1732|Bug 1732]]'' LocalArcRepositoryClient not documented || || || || || || || ||'''12. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1727|Bug 1727]]'' Poor error message in RunBatch || || || || || || || ||'''13. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1708|Bug 1708]]'' bitpreservation logic offers "add to archive" for file that is not in either location || || || || || || || ||'''14. Module Deploy:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1705|Bug 1705]]'' Make jmxremote.access writable before overwriting it (install script) || ||JOLF ||SVC || || ||OK || ||'''15. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1619|Bug 1619]]'' Potential NullPointer exception in RemoveAndGetFileMessage.getData() || || || || || || || ||'''16. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1260|Bug 1260]]'' Too much and wrong feedback information on "Missing pages" || || || || || || || ||'''17. Module Monitor:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1205|Bug 1205]]'' Security policy for unit tests contains hardcoded path to development environment || || || || || || || ||'''18. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1193|Bug 1193]]'' Exceptions from FileBatchJob stop batch job processing || || || || || || || ||'''19.''' || || || || || ||.. '''''' || ||Prioritized Feature Requests according to [[TaskTableFromMay2009Workshop|list]] of priority 4 and priority 3 tasks ||'''Total 21''' ||'''-''' ||'''-''' ||'''!SubTotal 21''' || ||'''-''' || ||||||||||||||||||||||'''Priority 4 Feature request''' || ||'''20. Module harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1116|Feature request 1116]]'' Global crawlertraps ||6 ||CSR ||SVC ||0 ||Assignment . See also FR 1120 Review decides whether some of the tasks will be implemented. Backend. To be finished in Iteration 41. Decision: Only assignment. ||OK || ||'''21. Module harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1696|Feature request 1696]]'' Ingest domain seed URLs ||5 ||Nicolas ||SVC ||0 || ||Postponed || ||'''22. Module harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1688|Feature request 1688]]'' Monitoring broad crawls. ||5 ||Nicolas ||SVC ||0 ||FR under discussion on mailing list. ||'''Postponed''' || ||'''23. Module archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1736|Feature request 1736]]'' Monitoring batchjobs ||2 ||JOLF ||HBK ||0 ||New feature request for restructuring the Batch architecture has been made. This FR only deals with improved logging. ||OK. || ||'''24. Module monitor:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1757|Feature request 1757]]'' Need a way to remove an application from lists of monitored applications ||2 ||HBK ||SVC ||0 ||High priority . ||OK || ||||||||||||||||||||||Priority 3 Feature request''' ''' || ||'''25.Module Harvester:''' [[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1774|Feature request 1774]] Stop using the JMS queues for queuing snapshot harvests || || || || || || || ||'''26. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1681|Feature request 1681]]'' Add seed to DB via webservice (via Browser Extension/Rich Client) ||? ||Andreas || || || ||Started Postponed || ||'''27. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1682|Feature request 1682]] Statistics (DB access, scripts, batch jobs ....) '' ||? ||Andreas || || || ||? '''''' || ||'''28. Module Harvester:''' [[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1683|Feature request 1683]] Util for regenerate admin.data file ||? ||Andreas || || || ||? '''''' || ||'''29. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1684|Feature request 1684]]'' Activity when domain is to be crawled. One table for seed ||? ||Andreas || || || ||? || ||'''30. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1743|Feature request 1743]]'' When accessing Bitpreservation this takes really long time ||? ||Andreas || || || ||? || ||'''31. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1120|Feature request 1120]]'' Crawlertrap info should be shareable between institutions ||? ||Andreas || || ||SVC will add comments to this FR. Might be an easy solution to share Crawlertraps by emailing files with crawler trap informations. ||Redundant (Copy of 20) || ||'''32. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1066|Feature request 1066]]'' Show whether seed URL existed ||? ||Andreas || || || ||? || ||'''33. Module Archive:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1809|Feature request 1809]]'' Write assignment for improving batchjob interface ||? ||JOLF || || || ||.. '''''' || ||'''34a. Module Harvester:''' ''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1134|Feature request 1134]]'' Filter job lists by category '''''' ||? ||Nicolas/Sara || || || ||.. '''''' || ||34b. Module Harvester: '''''[[https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1668|Feature request 1668]]'' Paginate and make sortable and searchable the list of jobs ''' ||? ||Nicolas/Sara || || || ||.. '''''' || ||Roadmap tasks '''''' ||Total 52?''' ''' ||-''' ''' ||-''' ''' ||Total ''''''8,5 '''''' || ||-''' ''' || ||||||||||||||||||||||Tasks from ...''' ''' || ||'''35'''. Review of [[http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/PlanningRollen#WARCtestPhaseIII|WARC Tools Phase III SRS]] ||5 ||SVC ||HBK,CSR,TLR || ||High priority ||'''OK''' || ||36. Assignment for enhanced QA tools ||2 || || || || ||Postponed || ||37. Finalize [[AssignmentHarvester2|Assigment]] for Harvester for support of WARC format ||? || || || || || || ||38. Finalize assignment for [[AssignmentGroupB2|Assignment group B.2.2]] ||0,5 || || || || || || ||39. Implement [[AssignmentGroupB2|Assignment B.2.2a]] - Generalise replica to include all checksum voters ||8 ||JOLF ||SVC ||0 ||High priority ||OK || ||40. Implement [[AssignmentGroupB2|Assignment B.2.2b]] - Store bit preservation information in a database ||8 || || || || ||In Progress || ||41. Implement [[AssignmentGroupB2|Assignment B.2.3]] - Use segments in bitarchives ||6 || || || || || || ||42. Implement [[AssignmentGroupB2|Assignment B.2.4]] - Write !BitPreservation scheduler ||5 || || || || || || ||43. Implement [[AssignmentGroupB2|Assignment B.2.5]] - Write !BitPreservation webinterface ||6 || || || || || || ||44. Finalize assignment for [[AssignmentGroupB4|Assignment group B.4.4]] - Yet more better infrastructure ||2 || || || || || || || || || || || || ||.. '''''' || || || || || || || ||.. '''''' || ||Wayback/Nutchwax tasks independent of !NetarchiveSuite code-freeze.''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total ''''''x '''''' || ||-''' ''' || ||||||||||||||||||||||Tasks from ...''' ''' || || || || || || || ||-''' ''' || ||45 [[TaskOverviewNutchWAX|Evaluation of !NutchWax]]. ||2? ||HBK ||CSR || ||High priority ||'''OK ''' || || || || || || || ||In progress '''''' || || || || || || || ||.. '''''' || ||Converting old Web collections to Netarchive.dk. See [[http://udvikling.kb.dk/cvsshadow/digiliv/ProjektDokumenter/omkostninger%20ved%20indsamling%20af%20gammelt%20materiale-3.doc|proposal]]. These task will be independent of !NetarchiveSuite code-freeze. '''''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total ''''''x '''''' || ||-''' ''' || ||||||||||||||||||||||Tasks from ...''' ''' || ||46. Investigation in dataformat as well as methods ||? ||SVC ||HBK || || ||In progress '''''' || ||47. Prepare ARC/WARC document for basis of decision to Steering group. Must old data be ingested into Neetarkivet.dk using ARC container or WARC container. ||? ||HBK ||SVC,TLR || ||High priority ||In progress '''''' || ||48. Generic converter prototype ||? ||HBK ||SVC || ||High priority ||.. ''Done Committed in CVS needs code review'' || ||49. Old KB Webarchive ||? ||SVC ||HBK || || ||In progress '''''' || ||50. Old Webarchive harvested with ARC-Httrack ||? ||HBK ||SVC || || ||.. ''Under dev.'' || ||51. Old Webarchive harvested with Wget ||? ||HBK ||SVC || || ||In progress '''''' || ||52. Old Webarchive harvested with !NedLib ||? ||SVC ||HBK || || ||In progress '''''' || ||53. Old Webarchive from Niels Brugger in waf format ||? ||HBK ||JOLF || || ||.. ''Done Waiting for Code Review'' || ||54. Old Webarchive from Kurt Vest Nielsen (Ingeniøren from 1995) ||? ||JOLF ||HBK || || ||Postponed '''''' || ||55. Webarchive from the library of The Danish Parliament ||? ||SVC ||HBK || || ||Postponed '''''' || ||56. Old Webarchives from Net-papers ||? ||SVC ||HBK || || ||Postponed '''''' || ||57. Digital publications of The Danish Law Gazette from the missing period ||? ||SVC ||HBK || || ||Postponed '''''' || ||58. Old Webarchive from Niels Brugger collected by HTTrack ||? ||HBK ||SVC || || ||Postponed '''''' || ||59. Prepare ingest of extracted data from Internet Archive into Netarkivet.dk || ||SVC ||HBK || ||High priority.Output will be a document showing what has been made of choice and an instruction to the daily manager of Netarkivet.dk of how to ingest the data. ||In progress || ||60. Ingest received data from Internet Archive into Netarkivet.dk || ||CLO ||HBK || ||High priority ||Awaiting document from task 59. '''''' || ||Common tasks calculated as implementation tasks''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total x''' ''' || ||-''' ''' || ||Others''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||!SubTotal 2''' ''' || ||-''' ''' || ||61. Setup of new KB test system ||2 ||TLR ||SVC ||2 ||High priority ||In progress '''''' || ||62. Test of 64 bit version of KB-PROD-ADM ||2 ||TLR ||SVC ||2 ||High priority ||.. '''''' || ||63. Architectual consideration: Move harvesters close to the backbone of the research network. ||2 ||SVC ||HBK ||2 ||High priority ||.. '''''' || ||64. Create/execute a batch test script specified by 1 or 2 researches ||2 ||JOLF ||HBK ||2 || ||.. '''''' || ||65. Prepare joint face to face meeting with UDV and Pligt/Natinal ||1 ||CHH ||CSR ||1 || ||.. '''''' || ||66. Prepare action plan for development activities ||1 ||CHH ||CSR ||1 ||High priority ||.. '''''' || ||67. Status of [[http://netarkivet.dk/netarkivet/index.php?title=Kendte_problemer|Crawl-problems]] (Netarchive.dk) ||1 ||CHH ||CSR ||1 || ||.. '''''' || ||Prepare release test''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||!SubTotal 12''' ''' || ||-''' ''' || ||68. Prepare [[http://netarchive.dk/suite/Iteration40Releasetest|release test]] ||6 || || ||0 || ||OK || ||Available man-days for implementation phase''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total x''' ''' || ||-''' ''' || ||||||||||||||||||||||Release test phase (task ...)''' ''' || ||Release test''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total 12''' ''' || ||-''' ''' || ||69. Execute [[http://netarchive.dk/suite/Iteration40Releasetest|release test]]. ||12 ||TLR ||All ||12 || ||In progress '''''' || || || ||' || || || ||..''' ''' || ||Release notes''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total 0,5''' ''' || ||-''' ''' || ||70. Write release note ||0,5 ||SVC || || || ||Awaiting end of code freeze '''''' || ||Available man-days for release test phase'''''' '''''' ||Total x'''''' '''''' ||-'''''' '''''' ||-'''''' '''''' ||Total 10'''''' '''''' || ||-'''''' '''''' || ||||||||||||||||||||||Assignment phase for next iteration (task ...)'''''' '''''' || ||71. Component bug/feature fix/management || ||QA || || || ||..''' ''' || ||72. Define goals for [[http://netarchive.dk/suite/Iteration41TaskList|Iteration 41 task list]] || ||CHH || || || ||..''' ''' || ||73. Presentation of goals and tasks for Iteration 41. Achieve a common understanding of the purpose of the iteration and each task on status meeting || ||SVC || || || ||..''' ''' || ||74. Assignment of tasks, bugs and feature request || ||QA || || || ||..''' ''' || ||75. Update release test procedure || ||TLR || || || ||..''' ''' || ||Available man-days for assigment phase''' ''' ||Total x''' ''' ||-''' ''' ||-''' ''' ||Total 22''' ''' || ||-''' ''' || === Timetable === ||Timetable iteration 40. Updated 13. November 2009''' ''' ||Start time''' ''''''''' ||End time''' ''''''''' ||Responsible''' ''' ||Baseline 13. November ''2009''. Start time''' ''' ||''Baseline ''13. November ''2009. End time'' '''''' || ||1. Implementation of decided tasks ||16. November 2009 ||11. December 2009 || ||16. November 2009 '''''' ||11. December 2009''' ''' || ||2. Code freeze. Create the build for release test and notify when build is ready ||14. December 2009 || ||SVC ||14. December 2009 '''''' || || ||3. Release test ||14. December 2009 ||16. December 2009 ||TLR ||14. December 2009 '''''' ||16. December 2009 '''''' || ||4. Code unfreeze ||17. December 2009 || ||SVC ||17. December 2009 || || ||5. Assignments, bug components and bug fixes ||16. December 2009 ||18. December 2009 || ||16. December 2009 ||18. December 2009 '''''' || eration39TaskList here.