From nch@roe.ac.uk Fri Apr 28 15:00:33 2006 Date: Fri, 28 Apr 2006 14:11:01 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 28th April 2006 Minutes of WFCAM Science Archive meeting: 28th April 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: JB, MAR, ETWS, NCH, RSC, NJC, PMW Apologies: AL, JPE, JMS, MSH, RGM DONM: 10am, Friday 5th May 2006 Actions discharged this week: ----------------------------- ACTION: NJC to close out the data modelling for arbitrary multiple epoch surveys Discharged; see Software below. ACTION: RSC to look into a robust solution to linux/windows share sweeping by the CU software Discharged Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 21/04/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES ACTION: JB to see that the WSA homepage is incorporated into the MyEd portal. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted that there is a UKIDSS survey heads telecon next week (Thurs May 4th at 1630) with agenda items including progress for DR1 and the data access policy. NCH will report back at next weeks meeting. PMW noted that a forward-look planning session is needed for Q3 work and we should review where we are with Q2. ACTION: NCH & PMW to have a VDFS work planning session on Thurs 4th at 2pm. WFCAM update: Nothing new to report this week. Comments and issues arising from CASU fortnightly minutes: No new minutes as of this morning. Networking: JB reported continuing testing of the CASU/WFAU transfer rate, with some progress using enhanced scp. WSA Operations: JB reported good progress on operations, despite one or two issues precipitated by a major power outage on Monday. A full WSA backup was secured overnight on Wednesday night. NCH noted that 7 weeks has elapsed since the last full backup, and that this was an uncomfortably long timescale. Various issues had resulted in the latest backup being delayed by several weeks. The team discussed the backup policy, and agreed that we should endeavour to secure at least one full backup each month; NCH suggested we experiment with a differential backup soon to see what the issues are in using intermediate differentials within the monthly full backup period. Hardware: With reference to last week's discussion concerning pixel data storage, NCH reported a conversation with SJW who suggested that we should endeavour to keep online all non-redundant pixel data, and that if pushed for space the data to consider scrubbing would be the individual microstep frames of interleaves, since the former can be recovered by unstitching the latter if needed. This will be done for 05A if we hit the buffers on disk storage when retransfering the reprocessed 05A data prior to the arrival of the next storage server. The next storage server (khafre) has been ordered; and JNTD is making good progress in setting up the new high capacity catalogue server (thutmose). Software: NJC has completed a revision of the top-level data modelling for arbitrary multiple epoch surveys (applicable to calibration data and the new UKIDSS Planetary Transit Survey). NCH suggested a brainstorming session to close out this and to check that all bases are covered with respect to this style of survey. ACTION: NJC, RSC, NCH, ETWS & MAR to meet Wed 3rd May, 2pm to finalise arbitrary multi-epoch survey data model. ETWS reported: "Updated the transfer check to use the debian sarge ls at CASU. Updated the schemas with the new attributes fro the nightly zero points from CASU. Included a check into CU4 to run only on catalogues where the pixel data has been ingested by CU3 beforehand. Installed the high performance scp on djedefre." MAR reported: "Went back to working on the external survey query builder incorporating the extra features that ETWS had made to the parsing script to generate the checkbox forms. Further work on the queue system (listing options) and updates to webpages. Added multiple multiframeid upload to Achive Listing page, beta made available to Steve Warren." NJC reported: "No major updates this week - getting back into things, some work done on calibration. Started looking into using S-Extractor, setting up TestWSAnjc after updates." NCH noted the preliminary look at S-Extractor for the UDS in DR1, and that this would likely be discussed at the telecon next week. NCH reported further work on CUs 3 & 4 (checking the efficacy of image/ catalogue deprecations on ingest of reprocessed data) and CU7 (enhancing and debugging frame set association, source merging and overlap seaming). RSC reported schema updates and schema updater enhancements. Survey Data Release: Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts NCH asked NJC to start looking at creation of the UDS mosaic using the CASU toolkit codes in order to be well prepared for any issues that arise in advance of DR1. NJC noted that several options and enhancements would be tested out in the process. ACTION: NJC to make a start on the DR1 UDS. Non-survey Data Release: NCH noted that a summary of projects observed by WFCAM has been received from JAC, and that JB has checked these off against releases from the WSA. In 05A, 8 projects have not registered for access; 12 others have registered and 11 have been released (the 1 unreleased only registered as recently as February). In 05B, 28 projects were observed. Of these, 13 have registered for access and 2 have so far been released. Astrogrid deployment: Nothing new to report this week. Miscellaneous: NCH noted that the ADASS XVI preliminary announcment has gone out, and PMW encouraged team members to think about attending (at least one, and preferably two attendees would be good). Dates are 15-18th and location Tucson, AZ. NCH noted that although this was around the time of the putative VDFS UK review, JPE was aware of this and would be avoiding the dates, and all the hard work would have to be done in September anyway.