From nch@roe.ac.uk Sat Jun 9 11:51:45 2012 Date: Fri, 8 Jun 2012 16:07:35 +0100 From: Nigel Hambly To: Eckhard Sutorius , Nicholas Cross , Rob Blake , Ross Collins , Mike Read , Clive Davenhall , Mark Holliman Cc: Lorenzo Rimoldini , Bob Mann , Jim Emerson , Keith Noddle , Norman Gray , Stephen Warren , Tom Shanks , Simon Dye , Stelios Voutsinas , Tom Kerr , Mike Irwin , Andy Lawrence at ROE Subject: WFAU VDFS Science Archives team meeting minutes, 8/6/12 Present: MAR, RGM, RSC, NJC, RPB, ETWS, NCH, NG Apologies: JPE, AL, KTN, STV, MSH DoNM: 10am Friday June 22nd 2012 in the Vista Hut (subject to non-demolition); MAR in the chair. Actions discharged this time: ----------------------------- ACTION: ETWS to copy over the WISE all-sky DR flat files from IPAC Discharged; DB created on ramses2 but needs to be backed-up and then copied to live public server ramses10. Crossmatches (including a retrospective VHS crossmatch requested by the survey PI) will then be done. RPB noted that some disk shuffling is necessary to accomodate the external survey databases; the luxury of having double copies on disk for backup and load-balancing can no longer be afforded owing to the volume of the ensemble of external survey DBs (this may be reinstated at some point in the future subject to funding for hardware). Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to scope out the areas/situations where we'd like to test MonetDB i.e. where do the current bottlenecks lie? Continues; a stub of a TWiki page has been prepared with a general introduction and some initial thoughts on this. Requires further development and some input from operations regarding specific performance figures: http://apache.roe.ac.uk/twiki/bin/view/WFAU/DatabaseScalabilityPerformanceAnalys is so all should try to contribute facts and ideas here. RGM noted that the arrangements for getting a MonetDB engineer over to WFAU to work with us on some large scale experiments are not going quite as smoothly as hoped, but we hope still to have some expert help soon. Actions carried forward from 11/05/12 meeting --------------------------------------------- ACTION: RPB to set up WSA mirror/sync for flat file DBs using MS SQL replication. Continues. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. ACTION: RPB to complete the LTO-5 tape backups of VVV and SSA DBs and SSS flat file datasets. And VVV DR1. And WISE-ASS (snig). Continues; the VVV ingest loading DB has been done; new to the backup queue are the VVV DR1 DB and the WISE all-sky survey DB (particularly important since there are no disk duplicate copies for these). Specific points and new actions: -------------------------------- Project management: Main news this time is that STFC, in their infinite wisdom, have elected a) to get less science per pound by closing UKIRT before JCMT, and furthermore b) to flush several tens of millions of pounds down the bog by shelving WFCAM and SCUBA2 before they've had a chance to fulfil their science missions. Well done. This cast a pall of doom over last week's science from UKIDSS IV meeting, and also has impacted the operations plans in the current grant renewal outline. Anyway, we carry on despite Dame Fate vomiting in our laps once again. The fall-out from the meeting is discussed in several places below. WFCAM, VISTA and VST updates: Nothing to report this time, other than the deeply sad and frustrating news that UKIRT will likely cease survey operations at the end of September 2013. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 23rd May, in particular it was acknowledged that some of the recent slow network performance may be due high-volume data shuffling at both ends ... Networking: ... and tests are on-going at this end to try to identify where there may be issues affecting transfer speed. It was suggested by NCH and MAR that a wholesale reboot of the SQL Servers may help things since file transfer speed internally between these MS Windows machines over "infiniband" has reduced to an embarrassing kb/s dribble. WSA/VSA/OSA/GES Operations: ETWS noted: - Continuing transfers of the WFCAM April data, which is still very slow. - After RPB split up the VVV Astrometry and Photometry tables into monthly parts, I've re-written CU4 to ingest into these. This should speed up ingests for the VVV. RPB noted that all Vista P88 data is transfered, metadata ingested and catalogue processed (sans VVV) up to the end of February. The next iterations of GES and VST processed data are eagerly awaited. Hardware and Systems: Nothing of note this time. Survey Data Release: The schedule for WFCAM releases is crystalising, and consists of DR10 (sans GPS) sometime over the summer; GPS skipping DR9 and going straight to a DR10 release sometime in the Autumn of 2012; a final UKIDSS DR sometime before the end of 2013 (subject to availability of processed mop-up observations during the UHS observational campaign, and agreement on the precise contents in terms of reprocessed data and new features - see below). The scope and schedule for UHS releases still needs to be discussed and agreed. Vista release discussions were dominated by the requirements for the imminent(ish) delivery of phase-III catalogue products to the wonderous ESO-SAF as follows: General: o IAU-like unique(ish) identifiers o FITS metadata file empty catalogue file to go along with the more standard FITS tables o UCD1+ metadata descriptions ACTION: RSC/MAR to create a FITS metadata header file utility for ESO catalogue deliverables ACTION: NG/ETWS/RGM to get together to make sure UCD1+ developments are ready to fold in to phase-III Vista deliverables Specific surveys: VVV: the necessity of limiting the delivery to P85 products needs to be formally agreed with the survey co-PIs; VMC: MAR has been coordinating with the survey PI, who requires products to be available for validation by June 22nd. VIKING: following on from a telecon with the survey PI, last year's (!) phase-III deliverables are being regenerated again supplemented with standard band-merged (not list-driven) source lists. VIDEO: exact requirements to be discussed and agreed with survey PI next week during his visit to IfA VHS, UltraVISTA: no requirements as far as we're aware. ACTION: NCH to email VVV co-PIs to get agreement on scope of current phase-III catalogue deliverables ACTION: NJC to discuss with VIDEO PI the requirements for current phase-III catalogue deliverables Software: RSC noted: - Completed Python software updates necessary to prepare the GES database from the initial set of data files provided - Toying with ideas to speed up the ESO release script by making use of a RAM disk cache to optimise the I/O performance for steps that are necessarily sequentially reading/writing the same file, i.e. checksum, compression etc. The team discussed SW requirements/priorities, resulting from discussions at the recent UKIDSS workshop and recent direct communications with survey PIs, in advance of preparing the cases for continued support of archiving within the WFAU grant renewal: UKIDSS-LAS: main priority was specified as stacking the 2 J-band exposures for deeper J-band imaging and associated catalogues over the ~1700 sq. deg. of area covered by Jx2. Generalised list-driven photometry was stated as being a lower priority, although a better large-scale "batch" mode of the on-demand list-driven service was requested. UKIDSS-GCS: no new requirements (what a very nice and contented, easy-to-please survey head that guy is...) UKIDSS-GPS: main gripe is with detection incompleteness in high-density areas, which would require full multiple-simultaneous profile-fitted point source analysis a la DAOPHOT, in conjunction with the Moon on a stick... Slightly easier to service is a new requirement for K band difference images (where possible) using the 2 epochs of K, and the application of MJI's magic saturated photometry recovery recipe for WFCAM catalogues (currently only applied in the VSA). UKIDSS-DXS: survey head mentioned in passing the possibility of generating proper motions from the intermediate stack products... UKIDSS-UDS: deeper stacks generated by stacking intermediate stacks in groups of weeks and/or months springs to mind... not sure who mentioned this. UKIDSS general: reprocessing image data (especially EDR), SDSS-like uber-calibration employing overlaps etc, and inclusion of reprocessed catalogues (for bug-fixes and enhancements at both pipeline and archive end) were seen as lower priorities. UHS: the main issue here was the request for a much more rapid eyeball QC turn-around in order that gaps left by QC cuts can be filled in during the same observing season. Also mentioned was crossmatch with PanSTARRS, if possible; of course, seamless inclusion of LAS/GPS/GCS J and K is required. Vista priorities that we're aware of: VVV: again, full-blown multiple simultaneous iterative psf-fitted detection and photometry in the Galactic centre was suggested... grrrr. We suspect difference-image analysis at the archive end is probably still wanted. VIKING: list-driven photometry is the prime requirement here, both internally within the VISTA passbands, and ideally with the corresponding optical imaging dataset from VST-KIDS. VMC: not sure of any new requirements ACTION: MAR to attend next VMC teleconf on Thurs 14th at 9am BST. VHS: we assume crossmatch with VST-ATLAS is very much a requirement. VIDEO: we will consult with survey PI when he's up at the IfA next week. UltraVISTA: rumours of possible new requirement to serve data from the VSA for this survey... watch this space for more news. There was some uncertainty concerning the likely development of the CASU codes in servicing some of these requirements - we've heard rumours of PSF developments, but are not sure if this is application of a PSF model at standard detection positions (rather than a much more CPU-intensive DAOPHOT-like approach); similarly the applicability of the list-driven source photometer for extended sources, being based on circular apertures (?), was not clear for satisfying the extragalactic fraternity. Perhaps CASU could enlighten us?! NJC spent some time last week chasing down a rather strange bug at the archive end that turned out to be caused by accidental inclusion of a FITS BSCALE keyword in VSA-generated deep stack confidence maps. As far as we are aware this has only affected VMC K-band stacks with a corresponding knock-on effect in the catalogues derived from them. The next releases of VMC data will be corrected for this bug. Non-survey Data Release: MAR and NCH discussed with Prof Myungshin Im of CEOU Korea the requirements for creating an uber-survey comprising various U/11A/K projects. NCH demo'd the WSA, and it made a favourable impression. Astrogrid deployment & Data Analysis services: Nothing of note this time. Miscellaneous: That's quite enough for now. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.