From nch@roe.ac.uk Mon Jul 14 10:20:06 2008 Date: Mon, 14 Jul 2008 09:30:46 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 11/7/08 Minutes of WFAU VDFS Science Archive meeting: 11th July 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RPB, MAR, ETWS, PMW, RSC, LGR, RGM, MSH Apologies: NJC, JPE, JMS, AL, BCW DoNM: 10am, Friday 18th July 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: PMW & NCH to meet next Tues (8th) at 12pm to flesh out the Q3 plan. Discharged ACTION: ETWS to send MJI a specific example or two of reprocessing confusion Discharged ACTION: MAR to co-ordinate with SJW over DR4 announcement(s). Discharged Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 4/07/08 meeting: ---------------------------------------------- ACTION: RPB to formulate backup plan arising from meeting Continues; currently wholesale new backups running; formal policy to be documented. ACTION: NCH, RSC, MAR, ETWS, NJC & RPB to meet next Friday 4th July 2pm to sort out non-survey curation automation Rescheduled to this afternoon (11th); much thanks to NJC and RSC who have been preparing a comprehesive review and proposal. A full summary will appear in next week's minutes. Specific points and new actions: -------------------------------- Project management: The team discussed the Q3 plan as formulated by PMW and NCH at the beginning of the week, and ratified it as sensible/doable. Of particular importance was the first implementation of the schema design for the VISTA Public Surveys so that it can then be discussed and iterated with the PIs in Q4 in advance of the surveys starting. WFCAM & VISTA updates: Nothing new this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week Networking: ETWS noted a brief outage in connectivity with CASU earlier in the week owing to thunderstorms down south. Transfer operations were not affected; we note that 08A processing is now complete and we await ok_to_copy. WSA Operations: ETWS reported another 5 months of GPS data have been ingested over the last week. Hardware MSH noted that, in conjunction with IT Support, we've had a conflab with hardware suppliers concerning a possible low-cost SAN (preferably fibre, or possible iSCSI networked) catalogue server with high capacity and IO throughput. Software: RSC noted: "Continuing work to switch CU7 to the curation of the MergeSource table, which is essentially the Source table without photometric attributes. This avoids the need for sources to be re-merged following a photometric calibration. Merging and ingest now work correctly, but more optimisations can and should be made. Seaming also now works, but further optimisations are required for scalability to large surveys, e.g. GPS, VISTA surveys. I've updated the C++ code to compile with the new 64-bit GCC compiler on khafre. Initial tests have revealed no problems as yet with the Python scripts and database connection. Some more testing is required to see how compatible our C++ curation codes are with 64-bit. A stable branch of the software, release_6, has been created as an operations fall-back in case new software features produce unexpected bugs. I've also been discussing automated non-survey curation with NJC." NCH and RSC have discussed possible optimisations for the source seaming under the new partitioned schema, and RSC is testing implementations. ETWS noted the 64 bit third party software is now installed on khafre; and working a bit more on enhancing CU1 for reprocessing shenanigans. This week NJC has mainly been looking at the automation of non-surveys and has written a script that will use metadata (with some registration information when available) to fill curation tables, setup schemas and ultimately create the non-survey tables, ready for CU6 to process the data. NJC has also improved CU6 so that it recognises shallow surveys and will work with these appropriately and has also added a function to calculate the next releaseNum (needed for deep stack naming) given a default value of 0. This will also help automation. NJC has also done some minor tidying up of old scripts, and finally noted that he intends to start testing some of the stacking/mosaicing tools under 64-bit on khafre now that it is working MAR has continued to deploy the user interface software on new 64-bit web server horus (core applications for the legacy SSS stuff are now working as well). Survey Data Release: The team wondered what the status of the UDS was ACTION: NJC to email Seb/Omar to see how they're getting on with the DR4 UDS ETWS noted that GPS preparations continue apace; MAR reminded us that we need to apply a couple of QC cuts prior to running source merging. NCH notes that special dispensation for STScI folks to access UKIDSS > DR1 has been given by the Board (the JWST folks want to look into guide star issues) and MAR is to arrange this and act as contact. Non-survey Data Release: Meeting this afternoon to thrash out automation of support for survey-like DBs. Astrogrid deployment & Data Analysis services No news on AG. LGR noted that he is looking into final optimisations of his web-based list-driven optical/IR photometry facility. LGR hopes to have it ready for beta-testing by the end of the month; in the meantime PMW has volunteered as an in-house tester and will try to break it next week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================