From nch@roe.ac.uk Fri Aug 15 17:37:27 2008 Date: Fri, 15 Aug 2008 14:29:32 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 15/8/08 Minutes of WFAU VDFS Science Archive meeting: 15th August 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, ETWS, RSC, NJC, MAR, LGR, PMW, MSH Apologies: JPE, JMS, AL, BCW, RGM DoNM: 10am, Friday 21st August 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: MSH to chase JNTD for final SAN specs & quotes Discharged; awaiting response Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB to test out attaching a normal database as an MS SQL "reporting database" in readonly mode on thutmose/hatshepsut RPB noted that some tests with attaching databases as readonly on read-write OS filesystems were inconclusive; in any case, a spare volume on public server thutmose needs to be used to set up a readonly volume for further tests. Unfortunately thutmose is ill (see below) so this has stalled. ACTION: NCH to assemble all feedback from VISTA PSP PIs concerning survey design in a TWiki page Continues; NCH has started a new topic at http://apache.roe.ac.uk/twiki/bin/view/WFAU/VSABaselineSchemas Actions carried forward from 08/08/08 meeting: ---------------------------------------------- ACTION: RPB to formulate backup plan arising from meeting Continues; currently wholesale new backups running; formal policy to be documented. NCH noted that he has made a TWiki page with information on the legacy SuperCOS dataset backups, and this could be used as a model. Specific points and new actions: -------------------------------- Project management: RGM/PMW gave an update on continuing Rolling Grant conversations with STFC: more reassuring noises, and a promise that something substantive will appear within the next month (and Scotland Yard will become London's next airport...) WFCAM & VISTA updates: NCH & ETWS have been communicating with JPE and Magda Arnaboldi in ESO/Garching to let them know that ETWS will attend the Public Surveys phase-2 workshop. ETWS will prepare a little presentation around archiving info and baseline survey schemas, so hopefully there will be time allocated in the final programme which is to appear at http://www.eso.org/sci/observing/policies/PublicSurveys/PIworkshop2008.html Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 6th August. NCH welcomed particularly the new flagging of summit-rejected MSBs: this will be extremely useful: please let us know the relevant new FITS keyword! The team also noted the work on processing VISTA commissioning frames, and look forward to seeing a few examples to test against the soon-to-emerge VSA schemas. Networking: ETWS noted that he and PSB are communicating over setting up transfer links with CASU's new VISTA hardware, and that some tests with processed VISTA data will then be possible. WSA Operations: NCH noted that UKIDSS Survey Head Alastair Edge has pointed out some blank jpegs in the 08A data. Turned out to be a problem in the jpegger at the WFAU end, and we suspect this may have been caused by a combination of recent NAS box trouble and 64-bit shenanigans with Debian4.0 and Python. Anyway, ETWS is on the case: ETWS noted - Finished list driven CU2 compressed image creation for the wfcam_fud and UDS data and startet CU3 metadata ingests. - Created JPEG thumbnail overview pages to track down all the broken 08A compressed images. It seems to be that the recent hardware failures caused these problems. - Worked with Pete Bunclark (CASU) on the passwordless connection for VISTA. - Worked with MSH on the installation of our third party software on horus. Hardware MSH gave advanced warning of significant (~1 day) of disruption to the archive servers to perform various bits of maintenance. This has been scheduled for the first-day-of-the-month at-risk period on September 1st. Servers requiring attention are thutmose, takelot and ahmose. Software: This week NJC has improved the astrometric errors calculation for synoptic tables. NJC has also found a few instances of poor quality intermediate stacks that were missed by quality control. While these were discovered by looking at the magnitude-rms plots produced for DXS and UDS images, NJC has also completed the recalibration mode of synoptic operations which will identify extensions where the photometry is significantly different from the expected value and will also recalculate zeropoints for good quality frames to improve the detection of low amplitude variables. RSC reported: "Prepared an initial version of the non-survey release script, which is limited to processing just specified programmes with multi-passband, shallow data at this stage. Completed refactoring of CU19 to allow for automatic release of non-surveys and committed changes along with some refinements to the bulk transfer code to make outgests slightly more efficient, benefiting CU7 etc. With the refactor of CU19 complete, I took the opportunity to also commit the changes required to curate the MergeSource tables from this development point into a separate dev_mergesource branch, where they'll stay unless we can make the procedure efficient enough to justify the change." MAR reported working on web interface stuff for the new non-survey support regime Survey Data Release: NJC has opened up the possibility of further refinements to DXS/UDS QC from examination of synoptic runs of data from the intermediate stacks. NCH suggested the best way to fold this in is probably in subsequent data releases, rather than redoing the stacking for the current one. Non-survey Data Release: NCH suggested that the team get together to integrate all the new developments done in support of non-survey datasets. ACTION: NCH, MAR, RSC, NJC and ETWS to get together Mon 2pm to work through integration of individual non-survey enhancements Astrogrid deployment & Data Analysis services LGR noted that he is continuing development a jsp front-end for his new pixel analysis tool. Miscellaneous: RGM again reminded everyone that today is the deadline for abstracts for ADASS XVIII (see http://adass.org/). RSC & NJC expressed interest in attending. Finally RGM noted that it would be good if some of the team can look over some LSST PDR documents concerning database schemas for data management. RMG will assemble relevant information on the LSST TWiki topic: http://apache.roe.ac.uk/twiki/bin/view/WFAU/LSST and then NCH suggested that he and NJC, and possible RSC, review the design and make comments for inclusion into some documented feedback for the project. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================