From nch@roe.ac.uk Fri Jul 8 16:33:22 2011 Date: Fri, 8 Jul 2011 16:15:58 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project team meeting mins, 8/7/11 Minutes of WFAU VDFS Science Archive meeting: July 8th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, MAR, NJC, RPB, RSC Apologies: JPE, MSH, RGM, AL, KTN DoNM: 10am Friday July 22nd 2011 in the Vista Hut Actions discharged this time: ----------------------------- ACTION: MSH to set up ramses10 in the cluster rack and configure as a further ingest DB server. ACTION: RPB to look into automated RAID monitoring across all servers. Actions partly discharged but continuing: ----------------------------------------- ACTION: RPB to look into RAID monitoring on the NAS boxes Actions carried forward from 24/06/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... Specific points and new actions: -------------------------------- Project management: Nothing to report this time. WFCAM & VISTA updates: WFCAM is offline (UKIRT closed June 28th to July 20th inclusive). VISTA continues to pump out the data (and the Galactic bulge and centre are up... oh crikey) Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 22nd June; in particular it was noted that the VIDEO data is being reprocessed, and that new v1.2 keywords are now appearing in the Vista data headers. Networking: ETWS noted that the latest WFCAM data (Apr/May/Jun) has been released by CASU and is currently transfering up to the WSA. WSA/VSA Operations: ETWS reported: -- Finished upgrade of CU4 to split the daily ingest for VVV to speed up ingests and error recovery. -- Finished ingesting the astrometically re-processed GPS data. -- Started looking into parallelising CU13. RPB reported: "VHS v1.1 release database ready for release. "VMC and VIKING release databases hopefully imminent. "VVV finished source merging and started reseaming, though this looks as though it will take some time. Dicussion in the meeting of running the Synoptic Source code in parallel with reseaming to speed things up." The team noted the imminent Hertfordshire meeting and the need to have something ready in time to demo/discuss. All agreed to review progress on Mon 11th and make a decision as to what features of the prepared DB to prioritise for production during the remaining time available. Hardware and Systems: RPB noted that storage for the VVV catalogues is likely to become a serious issue very soon. All agreed that splitting it off onto it's own ingest and release DB servers was a sensible plan, and RPB noted that he would be discussing with MSH the options for expanding the storage to keep pace with the ballooning catalogues. ACTION: RPB and MSH to come up with a sustainable storage expansion plan to keep pace with the VVV Software: RSC reported: "A clarification has been made to the definition of the source tables' *Xi & *Eta offset positions for detections in each passband. They will now always be defined as offsets from the "master" detection that was used by pairing algorithm, thus allowing users to effectively reduce the pairing tolerance criterion in their own sample selections. Previously, they were defined as offsets from the merged source position (which for UKIDSS shallow surveys was effectively the same thing, but the proper motion solution now changes that where it is applied). These changes will be released imminently in a new release branch, which also includes the proper motion solution implementation for the UKIDSS shallow surveys." MAR noted that a tweak to the user interface searching facilities is needed to cope with VISTA images near the South Celestial Pole. ETWS and MAR have worked out a solution and are implementing. MAR also noted that the ESO gripe feedback on the first tranche of deliverables for the VMC has been dealt with by contributions from various team members and the resulting data regenerated and passed on to the PI for submission into the ESO-SAF. RPB noted that the old design feature of curation clients writing temporary files to NAS folders should be deprecated in favour of the speedy infiniband-connected shares between (some of) the curation servers and the DB servers. ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. ETWS noted that the newest curation client installations seem to seg fault during early CUs, but the source is mysterious since it only happens with some code branches and not others. RSC volunteered to help investigate ACTION: RSC and ETWS to investigate seg faults for early CUs when running on the newest curation client systems. NJC noted: "This week I have been working on several parts of the code. I have modified SynopticMerger so that it can be appended to and made modifications to SynopticMerger and TilePawPrintMerger so that they ingest after each frameset rather than at the end. I have fixed a few bugs in CU6 after the major refactoring that occured a few weeks ago and made a few improvements to the logic of Automator." Survey Data Release: Trial VISTA v1.1 release DBs for VHS, VMC and VIKING are nearing completion and will be made available to the respective survey teams within the next week. Preparations of the same for VIDEO will commence once the reprocessed intermediate tiles and final mosaic are available. Regarding the VVV, see comments under Operations above. RSC noted that the fettled replacement of the WFCAM 30th Jan with the corrected night's data needs to be ingested before any further UKIDSS DR9 preparations can take place. NCH noted that eyeball QC results are in for the DXS and GCS; we trust LAS eyeballing is still on track for the end of July to keep the DR9 release on track sometime in August. NCH requested that we do a dummy run of the GCS DR9 as a final sanity check on the proper motions code (but certainly not to enable creaming off any marvellous new science results prior to release to the ESO community). Non-survey Data Release: One new WFCAM non-survey is being prepared at the moment, for Dirk Froebrich (who spotted availability of his latest data through the archive listing, but couldn't understand why a survey-like release DB wasn't instantly available too). Before criticising somebody, walk a mile in their shoes... then when you do, you'll be a mile away, and you'll have their shoes. Astrogrid deployment & Data Analysis services: No news this time. Miscellaneous: Madrid WSA user Herve Buoy has requested a full set of DR8 GCS "normal" frames for his own psf-fitting purposes. MAR duly obliged and the HDD is on it's way south as this is being typed... ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.