From nch@roe.ac.uk Fri Apr 21 13:47:00 2006 Date: Fri, 21 Apr 2006 12:35:27 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly project meeting minutes, 21st April 2006 Minutes of WFCAM Science Archive meeting: 21st April 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: JB, MAR, ETWS, NCH, RSC Apologies: NJC, AL, JPE, JMS, PMW, MSH DONM: 10am, Friday 28th April 2006 Actions discharged this week: ----------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. Discharged by stealth; see Hardware below. ACTION: NCH to install the top-level curation metadata for the new UKIDSS survey Discharged with some help/advice from JB. Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 7/04/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES ACTION: NJC to close out the data modelling for arbitrary multiple epoch surveys - CONTINUES ACTION: JB to see that the WSA homepage is incorporated into the MyEd portal. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH summarised the fall-out from last week's VDMT (ETWS, JB and NJC attended at various times in addition to the usual suspects from the Edinburgh end). The monthly/quarterly report was delivered verbally along with the usual summary performance charts. Otherwise, major issues discussed included the fixing of the external UK review date to October 2006, discussions on data transfer issues (see below) and finally a push from CASU to get basic survey data products (i.e. pipeline processed flat FITS files) released rapidly from the archive end. On this last point, NCH summarised the WFAU point of view, viz. that the effort required to do this is not commensurate with any gains in scientific usefulness, and that there is no big push from the community for such access anyway. WFCAM update: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 6/4/2006; issues regarding data transfer were discussed at length in the VDMT meeting and JB & PSB are pushing forward with tests, at the same time keeping the UKLight option moving forward. Networking: JB reported a number of experiments with the CASU/WFAU transfer procedure, during which a record-breaking transfer of 100 GB at 15 MB/s was achieved using five-fold multi-threaded scp. However, the rate with 5 threads is quite variable and yields a mean value slightly worse than 8 threads; some experiments with the scp code are being done in addition. In parallel with this, JB and PSB are pushing forward with UKLight. WSA Operations: JB reported 05B transfer complete (CU1); jpeg and ingest (CU2 & 3) for the same data should be finished by the end of the weekend possibly barring some of the last 4 days of data. 05A reprocessing has started at CASU, and one night's data has been made available as a test to make sure nothing nasty crops up. The exact timescale for reprocessing/retransfer is a little unclear at present but we should know more at the time of the next meeting. Hardware: The team discussed the current flat-file storage situation. NCH raised the possibility of removing ancestor frames (i.e. "normal" frames), as one way to reduce the required volume, if it can be shown that these are not generally used by the community and that this note doesn't raise any major objections from those to whom thes minutes are circulated. The team felt that at this stage this should only be considered an option for reprocessed data sets. In the medium term, the option of NAS boxes has been discussed between JB and JNTD as a way of expanding in a more cost-effective, rapid, space-saving and low maintenance manner. After the current file storage node (khafre) has been ordered and installed, a tentative plan to investigate NAS and obtain a single system as an experiment was floated by JB. PMW returns next Monday, when we will push forward on ordering khafre and the small amount of kit needed for the UKLight tests. Software: NCH reported reviewing the CU7 software (frame set association, source merging and overlap region seaming) in the light of experience, user feedback, and changes in the Survey Definition Tool tiling algorithm to make sure that this curation procedure is bug-fixed/enhanced/and otherwise generally ready for DR1. A small number of low-level bugs have been identified that results in poor choice of frame sets, incorrect source merging and sub-optimal overlap seam flagging in very rare situations. This is being debugged and checked against the 05B LAS data. RSC reported: "I've installed a patched high-performance version of SSH for file transfers on djoser, and have placed the installation instructions on the TWiki to aid installation on the other servers. I've ensured the dependencies are correctly described in the wfcamsrc C++ code to aid the build process. Also, miscellaneous bug fixes and further work/documentation on an object oriented framework design for curation scripts to make maintenance easier and developer quicker." MAR reported: "Carried on working on query queue. Now got an initial working version implemented via a servlet that uses a QueueManager class to start and maintain query threads and add/update/remove queries from the queue. The queue is held in a database table so some persistence is built in in case of downtime. Want to implement this on SSA as well as WSA but need to think how to implement balance." ETWS and NJC are or have been on leave over the last week or so. Finally, NCH pointed out a small issue with linux/windows file share sweeping as is currently implemented in the curation software, and asked RSC to look into a robust solution to avoid deletion of important files. ACTION: RSC to look into a robust solution to linux/windows share sweeping by the CU software Survey Data Release: As noted previously, the DR1 release is now fixed at 14th July. NCH outined a tentative schedule for the 6 weeks immediately prior to this with a view to expanding as more becomes known about retransfer/reingest/re-QC of 05A and QC of 05B. At present the schedule looks like this, working backwards from release: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts which implies archive QC1 needs to be complete by the first week of June and hence 05A needs to be retransfered and reingested by the end of May; source extraction for the is UDS also a potential worry. Anyway, the schedule will be revised and reviewed each week over the coming weeks to try to relieve as much stress & pressure as possible in the run up to DR1. Non-survey Data Release: JB noted a couple more registrations/enquiries concerning non-survey datasets and asked what the priority was relative to survey work. NCH advised that the surveys take the priority, and that non-survey releases will simply have to be done on a best-efforts basis with current resources. Astrogrid deployment: Nothing new to report this week. Miscellaneous: Nothing else this week.