Task list and timetable for iteration 38
Status |
OK/Not Ok |
1. Highlights approved |
OK |
2. Assignment of tasks |
OK |
3. Task list and time table approved |
OK |
4. Implementation phase started |
OK |
5. Release test phase started |
|
6. Assignment phase for next iteration started |
|
7. Iteration 38 completed |
|
Highlights for Iteration
[http://kb-prod-udv-001.kb.dk/twiki/bin/edit/Netarkiv/SupportNetarchiveSuite Support] of released NetarchiveSuite (http://netarchive.dk/suite).
Enhance NetarchiveSuite wiki according to [:UpdateNetarchiveSuiteWiki:decided structure].
Implement prioritized bugs according to [https://gforge.statsbiblioteket.dk/tracker/index.php?group_id=7&atid=105 list] of priority 4 and priority 3 tasks
- Enhancement of QA
- Enhancement of Batch support
Finalize the support of Wayback in the Netarchive.dk production site. See [:IntegrationOfWaybck:List of tasks] and [:AssignmentWaybackIntegration:Assignment] for Wayback Integration
- Migration of old Web materials to Netarchive.dk
- Start of task according to roadmap
Module Archive
- Enhanced support for Batch
Module Harvester
- ...
Module Access
- Support for Wayback
- Test of Nutchwax
Module Common
- ...
- Bug fixes according to updated prioritized bug list
- Iteration 38 is planned as a development release candidate.
Development procedure
Implementation according to [http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/TemplateImplementationTask implementation methodology]
Implementation and release test mainly in [http://www.google.com/calendar/render?gsessionid=tjZgbhGt6eNBB1mrlNwt3A intensive period]
- Estimated ... for implementation.
- Estimated ... for release test.
- Estimated ... for assignemnt of tasks for iteration 39.
- Target release: Medio September
Table of tasks
Tasks for iteration 38. Updated 11. September 2009 |
Estimate md |
Main responsible |
Reviewer |
Remaining md at 9. September 2009 |
Comments |
Status |
||||
Implementation phase (task x-n) |
||||||||||
Open Source release + bugs and feature request |
Total ? |
- |
- |
Total x |
|
- |
||||
Support of Open Source Release |
||||||||||
1. [http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/SupportNetarchiveSuite Support] of released NetarchiveSuite |
2 |
All (Google calender) |
|
|
|
Ongoing |
||||
2. Implement translateprocess. Adjustment to Open Source partners. |
1 |
KFC |
ELZI |
|
|
.. |
||||
Bugs and Features requests |
||||||||||
Prioritized bugs according to [https://gforge.statsbiblioteket.dk/tracker/index.php?group_id=7&atid=105 list] of priority 4 and priority 3 tasks. |
Total 5,5 |
- |
- |
SubTotal 1 |
.. |
- |
||||
Priority 5 bugs |
||||||||||
Module Archive: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1547 Bug 1547] Wrong synchronization in the IndexRequestServer and the FileBasedCache let two processes generate Index at the same time, and one of them fails |
2 |
KFC |
SVC |
0 |
|
OK |
||||
[https://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/Iteration37ReleaseTest Patch 3.8.2 release test] |
2 |
TLR |
All |
0 |
|
OK |
||||
Priority 4 bugs |
||||||||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1073 Bug 1073] resubmitting jobs redirects the browser to the list of all jobs |
2 |
HBK |
SVC |
0 |
|
OK |
||||
Module Archive: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1721 Bug 1721] Batch timeout is not configurable |
2 |
HBK |
SVC |
0 |
|
OK |
||||
Module Common: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=555 Bug 555] JMS connections cannot reconnect. [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1218 Bug 1218] Exception while adding listeners to JMSConnection. [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1275 Bug 1299] Network I/O errors shuts down JMSConnection. [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1645 Bug 1645] JMS connections very unstable. [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1275 Bug 1275] The message limit (maxNumMsgs) of 100000 has been reached. |
2 |
KFC |
SVC |
1 |
|
Committed. Awaiting sanity test and review |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1690 Bug 1690] Keep track of order XML changes |
2 |
KFC |
SVC |
0 |
|
Postponed |
||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1172 Bug 1172] password protected domain was not harvested |
1,5 |
CSR |
JOLF |
0 |
|
OK |
||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1336 Bug 1336] Harvester job dies suddenly |
2 |
HBK |
SVC |
0 |
|
Postponed. Waiting for more log information. |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1174 Bug 1174] Poor error message on dead job |
0 |
CSR |
JOLF |
0 |
This should be fixed by fixing bug 1188. No further work is required |
-- |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1188 Bug 1188] Heritrix side exceptions on JMX calls are ignored |
3 |
CSR |
JOLF |
0 |
|
OK |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1680 Bug 1680] Broad harvest stability (Job fail) |
? |
Andreas |
SVC |
0 |
SVC will close this bug as it is a symptom on other bugs. |
OK |
||||
Module Common:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1661 Bug 1661] Too many warnings logged when looking up Heritrix running state |
? |
KFC |
SVC |
0 |
|
OK |
||||
Module Archive:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1719 Bug 1719] Batch Job cannot instantiate loaded class |
? |
KFC |
SVC |
0 |
Invalid |
OK |
||||
Priority 3 bugs |
||||||||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=688 Bug 688] hosts-report should be IDNA decoded when writing harvestInfo to the DB |
2 |
|
|
|
We will need a domain name normalizer that both unmangles IDNA names and lowercases. This will take more than 1 MD. This and 596 must be solved together |
.. |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1069 Bug 1069] How to setup an apache proxy used to control access to the GUI and viewerproxy servers is missing from the Installation manual |
? |
|
|
|
|
.. |
||||
Module Archive:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1260 Bug 1260] Too much and wrong feedback information on "Missing pages" |
1,5 |
|
|
|
This bug will automatically be solved if we chose to implement feature request #1380 "Avoid double initiations of commands by doubble click" |
.. |
||||
Module Archive:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1193 Bug 1193] Exceptions from FileBatchJob stop batch job processing |
? |
|
|
|
|
.. |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1729 Bug 1729] Remove use of deprecated ARCWriter.write() method |
? |
|
|
|
|
Awaiting review |
||||
Module Harvester:: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1730 Bug 1730] The prefix to the messages is thrown away |
? |
|
|
|
|
Awaiting review |
||||
Prioritized Feature Requests according to [:TaskTableFromMay2009Workshop:list] of priority 4 and priority 3 tasks |
Total 21,5 |
- |
- |
SubTotal 1 |
|
- |
||||
Priority 4 Feature request |
||||||||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1298 Feature request 1298] Set JMXConnection timeout, if possible ' |
2 |
KFC |
SVC |
0,5 |
... |
Committed awaiting review |
||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1678 Feature request 1678] Make CDX-entries for the deduplicate entries in the crawl.log, and append to the other CDX-entries. [http://netarchive.dk/suite/ImprovedIndexing Analysis] |
8 |
CSR |
SVC |
7 |
.. |
In progress. Release test not dependent on this task |
||||
Module Common: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1687 Feature request 1687] French translation. |
2 |
Sara |
KFC |
? |
Postponed |
|||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1688 Feature request 1688] Monitoring broad crawls. |
5 |
Sara |
SVC |
0 |
SVC reviewed assignment. Release test not dependent on this task |
|||||
Module Harvester: [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1689 Feature request 1689] Managing crawls using object number. |
? |
Nicolas |
KFC |
|
Committed awaiting review |
|||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1641 Feature request 1641] It should be possible to turn off deduplication completely. |
2 |
SVC |
Nicolas |
0,5 |
|
Committed awaiting review |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1691 Feature request 1691] Configure which Heritrix reports to include in metadata ARC file. |
? |
Nicolas |
KFC |
|
|
In progress |
||||
Priority 3 Feature request |
||||||||||
Module Access:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=623 Feature request 623] We need to normalize URLs when browsing data |
5 |
|
|
|
Lighter solution |
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=680 Feature request 680] Cannot browse harvested password protected material |
10 |
|
|
|
At least partly solved by wayback. Investigations by collections sections ongoing. |
.. |
||||
Module Documentation:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1288 Feature request 1288] Batch and and use of Tools must be described |
? |
|
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1066 Feature request 1066] Show whether seed URL existed |
2,5 |
|
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1112 Feature request 1112] Automatic checks of seeds when entered in the harvest definition interface |
? |
|
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1120 Feature request 1120] Crawlertrap info should be shareable between institutions |
? |
|
|
|
|
.. |
||||
Module Archive:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1285 Feature request 1285] Storage of processed batch classes |
? |
|
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1482 Feature request 1482] Harvest information for job must report if there are problems in getting information |
? |
|
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1511 Feature request 1511] Thousand separators requested in user interface |
1 |
HBK |
KFC |
|
|
Done |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1681 Feature request 1681] Add seed to DB via webservice (via Browser Extension/Rich Client) |
? |
Andreas |
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1682 Feature request 1682] Statistics (DB access, scripts, batch jobs ....) |
? |
Andreas |
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1683 Feature request 1683] Util for regenerate admin.data file |
? |
Andreas |
|
|
|
.. |
||||
Module Harvester:' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1684 Feature request 1684] Activity when domain is to be crawled. One table for seed |
? |
Andreas |
|
|
|
.. |
||||
Module None' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1677 Feature request 1677] Enable WARC file writing and handling in the NetarchiveSuite |
? |
Soeren |
|
|
|
.. |
||||
Module None' [https://gforge.statsbiblioteket.dk/tracker/index.php?func=detail&aid=1116 Feature request 1116] Global crawlertraps |
? |
Soeren |
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
Roadmap tasks |
Total 52? |
- |
- |
Total 8,5 |
|
- |
||||
Tasks from ... |
||||||||||
[:AssignmentWaybackIntegration:Task Access 2.2] Ant target for deployable wayback |
2 |
CSR |
JOLF |
0,5 |
|
OK |
||||
[:AssignmentWaybackIntegration:Task Access 2.3] Create a PROPER version of NetarchiveResourceStore |
2 |
HBK |
CSR |
1 |
Unit tested |
Sanity test phase |
||||
Assignment for enhanced QA tools |
2 |
KFC |
SVC |
0 |
|
Postponed |
||||
Finalize [:AssignmentHarvester2:Assigment] for Harvester for support of WARC format |
? |
SVC |
KFC |
|
|
.. |
||||
Finalize assignment for [:AssignmentGroupB2:Assignment group B.2.2] |
0,5 |
JOLF |
KFC |
0 |
|
In progress |
||||
Implement [:AssignmentGroupB2:Assignment B.2.2a] - Generalise replica to include all checksum voters |
14? |
JOLF |
KFC |
0,5 |
|
Committed. Sanity test. Review must be completed before release test |
||||
Implement [:AssignmentGroupB2:Assignment B.2.2b] - Store bit preservation information in a database |
8 |
JOLF |
KFC |
0,5 |
|
Committed. Sanity test. Review must be completed before release test |
||||
Implement [:AssignmentGroupB2:Assignment B.2.3] - Use segments in bitarchives |
6 |
JOLF |
KFC |
|
|
.. |
||||
Implement [:AssignmentGroupB2:Assignment B.2.4] - Write BitPreservation scheduler |
5 |
JOLF |
KFC |
|
|
.. |
||||
Implement [:AssignmentGroupB2:Assignment B.2.5] - Write BitPreservation webinterface |
6 |
JOLF |
KFC |
|
|
.. |
||||
Finalize assignment for [:AssignmentGroupB4:Assignment group B.4.4] - Yet more better infrastructure |
2 |
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
Wayback/Nutchwax tasks independent of NetarchiveSuite code-freeze. |
Total x |
- |
- |
Total x |
|
- |
||||
Tasks from ... |
||||||||||
|
|
|
|
|
|
.. |
||||
|
5 |
|
|
|
|
.. |
||||
[:AssignmentWaybackIntegration:Task Access 2.4] Deduplicated CDX Indexing (Technical investigation) |
1 |
CSR |
SVC |
|
|
In progress |
||||
Evaluation of NutchWax. |
2? |
HBK |
CSR |
|
|
In progress |
||||
Technical decision on type of production HW for Wayback and Nutchwax. |
2? |
CSR |
CLO |
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
Converting old Web collections to Netarchive.dk. See [http://udvikling.kb.dk/cvsshadow/digiliv/ProjektDokumenter/omkostninger%20ved%20indsamling%20af%20gammelt%20materiale-3.doc proposal]. These task will be independent of NetarchiveSuite code-freeze. |
Total x |
- |
- |
Total x |
|
- |
||||
Tasks from ... |
||||||||||
Investigation in dataformat as well as methods |
? |
SVC |
HBK |
|
|
.. |
||||
Generic converter prototype |
? |
SVC |
HBK |
|
|
.. |
||||
Old KB Webarchive |
? |
SVC |
HBK |
|
|
.. |
||||
Old Webarchive harvested with ARC-Httrack |
? |
HBK |
SVC |
|
|
.. |
||||
Old Webarchive harvested with Wget |
? |
HBK |
SVC |
|
|
.. |
||||
Old Webarchive harvested with NedLib |
? |
SVC |
HBK |
|
|
.. |
||||
Old Webarchive from Niels Brugger in waf format |
? |
HBK |
JOLF |
|
|
.. |
||||
Old Webarchive from Kurt Vest Nielsen (Ingeniøren from 1995) |
? |
JOLF |
HBK |
|
|
.. |
||||
Webarchive from the library of The Danish Parliament |
? |
SVC |
HBK |
|
|
.. |
||||
Old Webarchives from Net-papers |
? |
SVC |
HBK |
|
|
.. |
||||
Digital publications of The Danish Law Gazette from the missing period |
? |
SVC |
HBK |
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
|
|
|
|
|
|
.. |
||||
Common tasks calculated as implementation tasks |
Total x |
- |
- |
Total 2 |
|
- |
||||
Others |
Total x |
- |
- |
SubTotal x |
|
- |
||||
Setup of new KB test system |
|
TLR |
SVC |
0 |
|
.. |
||||
Setup open Crucible server |
|
KFC |
SVC |
0 |
|
.. |
||||
Prepare release test |
Total x |
- |
- |
SubTotal 2 |
|
- |
||||
Prepare [http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/Iteration38ReleaseTest release test] |
6 |
TLR |
|
2 |
|
Need more information from developers |
||||
Available man-days for implementation phase |
Total x |
- |
- |
Total x |
|
- |
||||
Release test phase (task ...) |
||||||||||
Release test |
Total x |
- |
- |
Total 6 |
|
- |
||||
Execute [http://kb-prod-udv-001.kb.dk/twiki/bin/view/Netarkiv/Iteration38ReleaseTest release test]. |
6 |
TLR |
|
6 |
|
.. |
||||
|
|
' |
|
|
|
.. |
||||
Release notes |
Total x |
- |
- |
Total 0,5 |
|
- |
||||
Write Release Notes |
0,5 |
KFC |
|
|
|
- |
||||
Available man-days for release test phase |
Total x |
- |
- |
Total 10 |
|
- |
||||
Assignment phase for next iteration (task ...) |
||||||||||
Component bug/feature fix/management |
|
QA |
|
|
|
.. |
||||
Define goals for [http://netarchive.dk/suite/Iteration39TaskList Iteration 39 task list] |
|
CHH |
|
|
|
.. |
||||
Presentation of goals and tasks for Iteration 39. Achieve a common understanding of the purpose of the iteration and each task on status meeting |
|
SVC |
|
|
|
.. |
||||
Assignment of tasks, bugs and feature request |
|
QA |
|
|
|
.. |
||||
Update release test procedure |
|
TLR |
|
|
|
.. |
||||
Available man-days for assigment phase |
Total x |
- |
- |
Total 22 |
|
- |
Timetable
Timetable iteration 38. Updated 11. September 2009 |
Start time |
End time |
Responsible |
Baseline 3. August 2009. Start time ' |
Baseline 3. August 2009. End time' |
1. Implementation of decided tasks |
3. August 2009 |
21. September 2009 |
|
3. August 2009 ' |
21. September 2009' |
2. Code freeze. Create the build for release test and notify when build is ready |
24. September 2009 |
|
KFC |
21. September 2009' |
|
3. Release test |
24. September 2009 |
28. September 2009 |
TLR |
21. September 2009 ' |
22. September 2009' |
4. Code unfreeze |
29. September 2009 |
|
KFC |
23. September 2009' |
|
5. Assignments, bug components and bug fixes |
29. September 2009 |
30. September 2009 |
|
23. September 2009 ' |
25. September 2009' |