From nch@roe.ac.uk Fri Jun 27 16:11:21 2008 Date: Fri, 27 Jun 2008 15:31:00 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 27/6/08 Minutes of WFAU VDFS Science Archive meeting: 27th June 2008 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, NJC, RPB, MAR, ETWS, PMW, MSH, RSC Apologies: JPE, JMS, AL, BCW, LGR, RGM DoNM: 10am, Friday 4th July 2008 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to backup SegueDR6 (SQL DB from thutmose and flat-file backup files from hatshepsut) Discharged ACTION: PMW, MSH, NCH, ETWS, RPB, RGM to get together next Thurs (26th) at 2pm to review hardware requirements. Discahrged; see Hardware below ACTION: MSH to communicate his 64-bit Debian experiences to Horst in ITSG with a view to sorting out khafre. Discharged; no reaction as yet from Horst... ACTION: NCH to remember to discuss non-survey support at a future meeting when DR4 is better under control and RSC is back. Discharged; see Non Survey below Actions partly discharged but continuing: ----------------------------------------- ACTION: MSH (with metadoc help from ETWS if required) to publish SDSS SegueDR6 via AG DSA. Being done today. Actions carried forward from 20/06/08 meeting: ---------------------------------------------- ACTION: RPB to formulate backup plan arising from meeting Continues; currently wholesale new backups running; formal policy to be documented. Specific points and new actions: -------------------------------- Project management: NCH reminded *everybody* to get summary lines for progress each week to him promptly for inclusion in these minutes. Here endeth the first whinge. WFCAM & VISTA updates: Nothing new this week. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 23/6/08; the only comment was that we note the efforts underway to produce a standalone piece of code to help flat-file users apply full-blown intricate photometric calibrations themselves - sounds like a good idea as it's probably beyond the coding abilities of most users... Networking: Nothing of note this week. WSA Operations: ETWS reported: - Updated the parsing scripts to create browser pages for DR4. - Created browser pages for the SSA. - Processing of GPS recalibration has finished, data is waiting for ingest. - 08A April data is completely ingested and JPEGs are created. RPB reported: "cu7 and cu16 for LAS, DXS, and GCS. All finished, cu19 started yesterday after we fixed the problem with lack of space on ahmose causing a confusing error message when cu19 tried to create the database. Ahmose share folder is tidied up and tempdb has been shrunk. Last action there is to shrink the WSA transaction logs once cu19 has finished. Creating skinny tables of SDSS attributes in the WSA. This should speed up cu16, particularly for the LAS, in future. The only survey we actually ran with the new skinny tables, though, was GCS which doesn't normally take too long anyway. Progressing with taking a known good backup of all the existing databases. SegueDR6 done now. Reinstated weekly incremental backups of the WSA to commence this weekend. Copying TWOMASS db from hatshepsut to takelot in readiness for the network-mounted data testing discussed at the hardware meeting yesterday." Hardware MSH reported: "Emailed Horst with instructions on how to setup Khafre to run Starlink in 64bit mode. Reviewed hardware quotes for LSST/PanStarrs database cluster nodes. Helped develop new hardware design plan for the next 2 years of WSA and VSA operations." NCH noted this hardware conflab that took place yesterday. From the VDFS point of view, the feeling was that we need to try to move back to having a single DBMS instance addressing all the release DBs for ease of maintenance and also for generalised crossmatch functionality (e.g. the SSA will be temporarily isolated from all the other large DBs and so only limited crossmatch facilities are available). The plan is to test out some large attached storage solutions to benchmark trawl speeds of NAS/SAS solutions, and then to make a decision well in time for hosting large Vista DBs. Current hardware will be redeployed as test units, cannibalised for useful storage or retired and replaced by higher capacity storage as we move forward with this. Software: NJC reported: "NJC has completed testing of CU6, which pipelines the production of multi-epoch data after QC has been completed. This now runs through the setting up of controls which decide what tasks need to be completed (so that the pipeline can be restarted automatically), deep stack and catalogue production, deep stack and catalogue ingest, QB flagging, provenance table filling, source merging, neighbour table production and finally synoptic table production. This now runs all the way though for the DXS. NJC has also started working on a tool that automatically sets up requirements for multi-epoch programmes, such as stack products. This will be useful in particular for large deep surveys and the numerous non-survey programmes." RSC noted: "I've mostly been fixing up some minor software bugs that cropped up as DR4 release peculiarities: * Updated CU16 to use new skinny SDSS flag tables to speed up cross-matching with SDSS tables. Also working on a solution to avoid conflicts between test CU16 runs and release runs via unique file tags. * Fixed CU19 bug whereby frames with default programmeIDs have been included in survey releases for the past year without us noticing until NCH pointed it out last week. Also revised a rather confusing error message that occurred on database creation failure. I've tried to alleviate some niggles in updating the WFCAMPROPRIETY metadata mirror. Previously the update would fail to start if a user was accessing the database at that time, now it will go into holding loop for up to three hours. Also, users accessing the database during an update may be confused by incomplete datasets a solution to which is being investigated by MAR and myself, either by restricting public access during the update or else by maintaining the old copy of the database until the update is complete. I've updated CuPerformance on the TWiki with the latest DR4 curation software performance times." Survey Data Release: MAR, with advice from SJW, has closed out QC2 for DR4 (the data's not perfect, but it'll do...) CU19 (survey release DB creation) was invoked yesterday, fell over because of lack of disk space, and was then reinvoked following a bit of tidying up... ACTION: RPB to remember to shrink the WSA transaction logs once CU19 has finished running. We anticipate final checking this afternoon then placing online early next week. NCH asked MAR to freeze the DR3 webpages and create the usual temporary DR4 ones, and everybody to review the existing documentation for accuracy updates. (RSC and NJC have been keeping some release notes as curation has progressed). Non-survey Data Release: NCH asked the team to think about non-survey release DB automation in advance of a separate conflab: ACTION: NCH, RSC, MAR, ETWS, NJC & RPB to meet next Friday 4th July 2pm to sort out non-survey curation automation ETWS has updated all photometric calibrations, and is currently checking reprocess synchronisation with the CASU copies of all data in advance of release of a tranche of non-survey DBs. Astrogrid deployment & Data Analysis services MSH noted that the next Astrogrid Consortium meeting is being held at IfA next Mon/Tues. MSH noted attending a DCA meeting at ESO where he presented the DSA software and helped approximately 15 scientists install and deploy it (some using test data, others using their own datasets). Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================