From nch@roe.ac.uk Tue Oct 16 09:07:41 2007 Date: Mon, 15 Oct 2007 14:21:47 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 12/10/07 Minutes of WFAU VDFS Science Archive meeting: 12th October 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, NJC, LGR, MAR, MSH, PMW Apologies: BCW, JPE, AL, RGM, JB, JMS, ETWS NB: DONM: 10am, Friday 19th October 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: ETWS to ascertain what the status of any prepared catalogue product from GALEX is. Discharged: GR1 was a data release (.5TB in volume) in 2004/5. GR1 will no longer be distributed by MAST when GR4 becomes available in late 2007. In 2006 it was followed by GR2 (1.5TB) which superceded the GR1 in quality and sky coverage. In January 2007 they released GR3 (1.5TB) as a supplement to GR2. A breakdown of observations in GR2/GR3 by survey type will be as follows: GR3 # of GR2 # of Total # of Survey Acronym Tiles Tiles Tiles ------------------------------------------------------------- Medium-Deep MIS 652 365 1017 Deep DIS 12 110 122 Nearby Galaxy NGS 38 258 296 Calibration CAI 20 0 20 "Guest Investigator GII 288 0 288 Survey" All-Sky AIS 8644 7077 15721 Spectra Grism 29 10 39 ETWS couldn't find any DB product download page so unless RGM finds one this does not look viable at present. ACTION: JB to open a trac ticket for a logging software enhancement Discharged ACTION: MSH and NCH to check out the SSA loading situation on hatsehpsut Discharged; loading continues ... Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration after DR3. ACTION: RSC to progress implementation of automated replication of the ingest DB file metadata on a public server. Continues; RSC has located and fettled the appropriate scripts, and now awaits restoration of the ingest DB for test purposes. Actions carried forward from 05/10/07 meeting: ---------------------------------------------- ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). ACTION: JB & ETWS to restore SegueDR6 on thutmose at their convenience (i.e. no hurry given the present circumstances) NCH noted that it'll almost certainly have to be restored under SQL2005 rather than 2000, so new server hatshepsut will be needed rather than thutmose... ACTION: NCH to bug site services concerning unacceptable infrastructure problems. ACTION: ETWS to look into the examples of unavailable jpegs. Specific points and new actions: -------------------------------- Project management: PMW noted that it is that time (again!) when we need to complete the end of month reports; furthermore we need to finalise the official Q4'07 plan for future progress reporting. ACTION: NCH and PMW to meet on Mon 4pm to attend to various management admin. WFCAM & VISTA updates: Nothing new this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the most recent meeting; there were no comments (congrats on a most successful ADASS, btw). Networking: Nothing new this week. WSA Operations: Operations have consisted entirely of continuing to recover from the major disk array failures last week. The WSA DB has been put back online and consistency-checked; we believe we know where the problem pages are and continue with fixes... A couple more requests for minor fixes to the DR3-processed data have come in from CASU: MJI stumbled across a problem with some of the PV2_3 values in the FITS image headers for a few files with associated catalogues. The problem only affects certain files on two nights 20061228 and 20061229. STH pointed out a minor problem with the Y-band nightly (average) zero-points - it is necessary to update the Y-band average nightly zero-point with the 0.08 offset. Hardware Further disk array failures this week have not helped WSA recovery. JB and NCH have clocked the disks on load server ahmose back from U320 to U160, and in the process discovered that the poor thing was running in hybrid 160/320 mode which probably hasn't helped matters. ACTION: NCH, MSH and JB to work with Eclipse on stabilising the servers JB noted: "Apart from being a hardware/DB monkey this week I've been investigating Solid State Drives/Disks, a new page on the twiki can be found with a summary of the current state of affairs here: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SolidStateDrives It's rather technical but it boils down to: SSD's are more reliable and faster but cost more than our current SCSI discs and are definitely no match for SATA discs price wise. The only relationship not in the orders of magnitude is SSD's are twice the price of SCSI's." Software: NJC reported low-level bug fixes; RSC has the UDS crosstalk flagger shaken down for DR3 (plus some general quality error bit flag work); MAR has investigated the NVO footprint web service and broken it with the DR2 LAS frame set definition (evidently it's not particularly scalable). Survey Data Release: Schedule is still on hold due to major hardware failure (see above): Weeks What: 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. LAS/GPS eyeball results now all in. NCH noted that he has tentatively suggested repeating the DR2 trick (excluding the GPS initially to get the release out asap) to both AL and SJW, who are both supportive of the idea. MAR and LGR have worked on supplying a DR2 GPS source attribute subset for use externally in a cluster finding code (not the kind of thing that can be "coded" up in SQL...) Non-survey Data Release: MAR noted that half a dozen new non-survey registrations have come in within the last week or so, and also that we should be a little careful about verifying registration info supplied as accurate as there was a recent case of a legitimate PI asking for access to the wrong project ID. Probably need to cross-check against JAC/UKIRT webpages for PATT, but some uncertainty as to where Japanese and UH details may be... ACTION: MAR to contact AA to ask for his advice on PI-programme registration verification Astrogrid deployment: No news this week. Miscellaneous: Nothing else this week.