From nch@roe.ac.uk Thu Jun 8 17:31:47 2006 Date: Fri, 2 Jun 2006 12:01:32 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes 2nd June 2006 Minutes of WFCAM Science Archive meeting: 26th May 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: JB, MAR, NCH, NJC, PMW, RSC, RGM Apologies: AL, JPE, ETWS, MSH, JMS NB: DONM: 10am, Friday 16th June 2006 (NCH away in Barcelona wrt GAIA) *NOTE FROM NCH TO AL/SJW/AA/JPE: a few worrying problems are noted below concerning data release schedule and fundamental infrastructure and it's affect on archive operations and uptime. Please read carefully and send me your reactions.* Actions discharged this week: ----------------------------- ACTION: JB to retransfer and reingest the bug-fixed data. Discharged (that is for the 26th October data) Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 26/05/06 meeting: ---------------------------------------------- ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES: Note that this is on hold until after DR1. ACTION: JMS to contact JPE regarding progress on October UK VDFS review organisation - CONTINUES Specific points and new actions: -------------------------------- Project management: PMW noted that it is time once again to fill in the progress charts (for May). This will be done offline. ACTION: PMW to take around the progress charts for filling in work done for May. WFCAM update: Nothing new to report this week Comments and issues arising from CASU fortnightly minutes: No new minutes as of this morning. Networking: JB has submitted the UKLight bid for bandwidth - thanks to JB, PSB and all concerned in preparing this. WSA Operations: NCH gave an update on QC. Some "preliminary" 05B eyeballing has been done for the DXS, while nothing has been received so far from the UDS. NCH suggested that we simply apply NJC's eyeball deprecations for the UDS in order to enable progress on this at least. 05A QC must of course await retransfer/reingest of the reprocessed 05A data (the full extent of the processing bug noted last week in 05A V3 is not known...) NCH and JB noted that MJI has sent a "heads up" note concerning some quirks in the 06A data, and how CASU are handling the processing in the unexplained absence of detector frames in the batches of data received from JAC. We thank CASU for the warning; we will worry about this after DR1. Hardware: An exasperated NCH noted that the perennial infrastructure problems of power supply and heat dissipation have resurfaced this week. Tuesday saw a heat overload in the server room C1 due to the combination of the main aircon unit developing a fault coupled with an unusually warm (for Scotland - i.e. temperature in double figures) day. Because there is no spare capacity in the room, the units are fully on continually and get no chance for the defrost cycle which is necessary for efficient operation. The situation was fixed rapidly, but the question of installing sufficient capacity in the room to allow for the defrost cycles and future expansion seems to be continually ignored. In addition, there was a site-wide power cut for two hours at 9am on Wednesday morning, which obviously affected archive up time for that interval, but also had a knock-on affect of absorbing much effort and disrupting general operations for the rest of that day. Premises are blaming WFAU in part for installing new equipment. WFAU aspires to provide a high quality service for European astronomers, but cannot do so if the fundamental infrastructure of the site limits activities. We have tried to give an honest estimate to premises of our annual requirements; admittedly we may have underestimated in the past the need for storage. However, the fact remains that infrastructure upgrades are reactive rather than proactive, and we are apparently at constantly at the limit of what it is possible to do on this site. This is not good enough for high availability of IT services. Software: JB noted that ETWS has produced further checking/monitoring software for non-survey data. Survey Data Release: NCH noted the following home truths with regard to releasing DR1 - powers that be take note and communicate your responses asap. NCH has emphasised time and time again (particularly at consortium meetings) that it is only reasonable to allow WFAU a 4- to 6-week period of grace in between the last items of data (that are to be included in that given data release) arriving at the archive and the actual release data itself. This period is to allow us to do all the various curation fettling and prepare a high quality product, and is a reasonable estimate of the time required when everything is working under "steady-state" conditions. The situation at the moment is that VDFS has not quite reached "steady-state" and worse still we are now 6 weeks away from the putative DR1 release data with *no* V3 05A reprocessed data ingested yet (for various reasons, not only because of the processing bug noted in last week's minutes). If DR1 is to include the reprocessed V3 05A data, then it *cannot* take place on the 14th July. Delaying a couple of weeks will not help hugely, and there are significant staff absences for the first two weeks of August. Hence a delay to the end of August looks more likely, unless we decide to go with the current processed version of 05A. Reiterating the DR1 preparation schedule as it currently stands: DR1 release at: 14/7 (NB: databases to be DR1 & DR1PLUS !!) Copying/transfering/ 7/7 backups etc. start UKIDSS pre-release 1/7 checks start Final CUs: 16 (incl. 23/6 SDSS DR3 hopefully), 18,19 start CUs 2,3,4,7 for DXS 20/6 and UDS start Archive QC2 starts 14/6 CU7 for wide/shallow 7/6 surveys starts QC1 for 05B remains to be closed out; NCH will prioritise the UDS QC to enable NJC to progress UDS DR1 preparations. NCH is unable to complete the rest of 05B QC because his presence is required in Barcelona next week for a GAIA meeting and stack eyeballing has yet to be completed. Non-survey Data Release: MAR reported: "Finished off work on enabling flat file access (archive listing) for non-survey users. Eleven non-surveu programmes have beem informed that they now have this access. Astrogrid deployment: No news this week. Miscellaneous: Nothing else this week.