From mar@roe.ac.uk Wed Jan 9 12:13:06 2013 Date: Tue, 8 Jan 2013 15:22:13 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Andrew Lawrence , Bob Mann , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Norman Gray , Stephen Warren , Tom Kerr , Tom Shanks Subject: WFAU VDFS Science Archives project team meeting minutes, 7 Jan 2013 Present: MAR, RSC, RPB, ETWS, NJC, RGM, MSH Apologies: JPE, AL, STV, NG, KTN, NCH DoNM: 10am Friday January 25th 2012 *** NB in the Villas meeting room *** NCH in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: RPB to set in motion q-bits for LAS/GCS once given the nod by MAR Discharged, DR10 almost ready for release ACTION: ETWS to check the DXS fileset (provided by RPB) for corruption Discharged. ACTION: RPB to set up WISE-ASS cross-neighbours for UKIDSS surveys to replace the old preliminary source catalogue matchings Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS (with help from NJC as necessary) to add FeII 1.644 micron filter details into the WSA. Continues; all done apart from Vega-to-AB and zeropoint attributes (the later still needs the latest data to be ingested). Actions carried forward from 30/11/12 meeting --------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: Nothing for the minutes. WFCAM, VISTA and VST updates: Nothing to note. Comments and issues arising from CASU minutes: No new minutes this time. Networking: Transfers are up to date. WSA/VSA/OSA/GES Operations: ETWS reported: - The 2008, 2009, and 2010 disk38 data has been fixed now. 2011 is still going. - WSA data has been CU3ed up till September 2012, and ingests are running. - FinaliseFlatfileIngest has finished on the VSA, but it looks like many MJD values are still missing. This is being investigated by NJC. CU13 is working its way through the VVV. The CASU VISTA re-processing prompted much discussion. RSC noted that we might need to employ another DB load server so that existing version and re-processed version operations could be run in parallel. It was thought that ramses9 might be suitable. The re-processed data would likely be ingested into a new load database but we'd first need to see what the re-processed data entailed. Hardware and Systems: There had been no machine or disk failures over the holidays! Eclipse had not yet got back with any results of their tests on the disks from /disk38. It was hoped that we could just use the 4Tb replacements disks when the time came. Given the uncertainty over the grant situation and hardware budget available it was thought to be a good time to look at hardware requirements over the next 15 months. ACTION: MSH, with input from ETWS, to estimate hardware requirements, timescales and costs up to end March 2014. Survey Data Release: UKIDSS DR10 - nearly ready for release. CU19 had stalled during gcsDetection outgest. This has happened previously and the cause is still not known. Manually running the same query over the holidays and outgesting to a temporary table proved equally fruitless. A subsequent re-working of the query into two stages yielded results in just over an hour. Following the freeing up of some space on the relevant DB servers and the installation of gcsDetection, DR10 is good to go. SDSS DR9 neighbours will be added shortly after. GPS to be added as and when. VMC - next release awaiting completion of CU3 and 4. VIKING - The ESO release had finished just before Christmas and many files had been uploaded to ESO over the holidays. Software: NJC noted that he'd mainly been working on the list driven code and associated environment/schema. Initially this would use the CASU code but could be readily adapted to use SExtractor if required (eg final release of DXS). RSC reported: - CU16 can now be stopped/started allowing resumption in processing in the case of failure (typically due to lack of disk space for the VVV) and has also been tweaked to bring about acceptable performance for VVV cross-matches with large matching radii against sparse (extragalactic / IRAS) catalogues. - All regular curation code that modifies detection tables should now have been updated to work against the monthly-split VVV detection tables. - The current VVV DR2 release curation logs are being closely monitored and analysed for overall performance, keeping an eye out for potential improvements in scalability. - CU19 now sends ETWS e-mails to announce the creation of new release databases as a reminder that new schema browser pages are required. - The missing bit 12 detection flags were fixed using a one-off script. - Investigations into the flagging of saturated detections in VISTA tiles revealed that detector-scale fluctuations in the average saturation level across a tile were typically as large as a factor of 5. This means that any simple modification, e.g. just flagging everything against the lowest average saturation level, would be unsatisfactory. Moreover, we also noted that CU4 presently performs a saturation correction on the flux of detections from tiles based on this unhelpful average saturation level value for the entire frame. Since we anticipate a re-ingest of all catalogues due to reprocessing in the near future, the current plan is to switch off saturation corrections for tiles in CU4 prior to ingesting these reprocessed catalogues. The detection quality flag for saturation will then be modified to account for the large spatial variation in tiles. Non-survey Data Release: None. Astrogrid deployment, VO & Data Analysis services: Nothing was raised. Miscellaneous: The "VISTA: a celebration" ROE workshop was next week (17-18th January). RGM asked that all ensure that their staff absences entries on the twiki are up to date so that the next WFAU quarterly meeting can be scheduled! -- Scanned by iCritical.