From mar@roe.ac.uk Wed Nov 21 12:20:13 2012 Date: Wed, 21 Nov 2012 09:35:29 +0000 From: Mike Read To: VDFS Science Archive Team -- Clive Davenhall , Eckhard Sutorius , Mark Holliman , Nicholas Cross , Nigel Hambly , Rob Blake , Ross Collins , Stelios Voutsinas Cc: CCs for VDFS -- Andrew Lawrence , Bob Mann , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Norman Gray , Stephen Warren , Tom Kerr , Tom Shanks Subject: minutes of WFAU VDFS science archives project team meeting, 16 Nov 2012 Present: MAR, RSC, RPB, MSH, ETWS, NJC Apologies: NCH, RGM, JPE, AL, STV, NG, KTN DoNM: 10am Friday November 30th 2012 *** NB in the Villas meeting room *** NCH in the chair (room is booked). Actions discharged this time: ----------------------------- ACTION: MAR to fettle VVVDR1 so that rows/columns in vvvSource and vvvDetection, originating from bad Ks deep stacks, are reset to defaults. Discharged, and following consultation with Phil & Dante VVVDR1 is back online. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS (with help from NJC as necessary) to add FeII 1.644 micron filter details into the WSA. Continues; all done apart from Vega-to-AB and zeropoint attributes (the later needs the latest data to be ingested). Actions carried forward from 5/10/12 meeting --------------------------------------------- ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues; this one still has low priority. Specific points and new actions: -------------------------------- Project management: Nothing for the minutes. WFCAM, VISTA and VST updates: Nothing to report this time. Comments and issues arising from CASU minutes: Nothing of note other than they were lengthy. Networking: No transfers currently running due to the disk issues but recent speeds have been good. SDSS BestDR9 has now been transferred over. WSA/VSA/OSA/GES Operations: Some VMC/VIKING stacking/tiling had been done however further progress had been limited by the issues with disk37 and disk38. ETWS reported: -- Transferred SDSSBestDR9 files from Chicago (compressed, 5.6 TB). After being allowed to use scp transfer rates of up to 14.7MB/s were reached, averaging at ~8-9MB/s. -- Transferred latest OSA data up to September, created JPGs and ingested the metadata. -- Disk38 data problems: Copied the data from disk38 to disk41, which manifested the error in ~10% of the VSA files. Fixed the metadata (1% of all files affected) and determined all broken VSA pixel data on disk38 (17400 files) which took ~10 days. The pixel data fix is running since 3 days and might finish later today or earlier tomorrow. It was still not known if any of the catalogues on disk38 are corrupted, this would need to be looked into after the pixel files had been fixed. MAR thought that any corruption must be subtle as he'd managed to open all VSA catalogues on disk38 and read to the end of the table data. ACTION: ETWS/MAR to check disk38 catalogues for corruption CASU had recently produced some list driven catalogues for ATLAS. These had been transferred up but the decision was made to hold off ingesting these catalogues for now. Hardware and Systems: Disk37 was currently offline following the movement of the original disk38 to a new chassis for a recovery attempt. It was hoped that disk37 would be back later in the day but it could take a lot disk swapping in disk38 before the final verdict was in. Kakai's alarm was sounding so should be used with caution until it could be looked at. Survey Data Release: UKIDSS DR10 - QC1 underway. We have received the UDS release files. VMC and VIKING - some progress but currently awaiting the return of disk37. RGM asked about the status of the VIKING ESO release, NJC thought that deep products are still required as these would have been lost from disk38. ACTION: NJC to find out where we stand with regard to a VIKING ESO release. VVV - P87 and P88 have been CU4'ed but need to check for any problem files. It was thought that the next release would only include up to P87 but this would be verified. ACTION: MAR to check which semesters are to be included in the next VVV release. ATLAS - Following a trawl through the ATLAS metadata (thanks to Mike Irwin for lots of help with some of the issues this threw up) MAR had emailed Tom Shanks and Nigel Metcalfe with a suggested route for progressing the initial release. No response from Tom to date. Software: RSC reported work on VVV performance improvements and tests on quality bit flagging. In addition to CU13 fixes, NJC had been working on the list driven and source re-measurement code. Back in September Sebastian Gurovich reported differences between WFAU and CASU Ks photometry. Sebastian seemed to be comparing magnitudes derived from the CASU pipeline tiles to VSA deepstack magnitudes. At the time we'd been unable to identify how the differences arose. Earlier in the week Mike Irwin had prompted a re-opening of this cold case. The differences are around the 0.01-0.05 mag level and show a systematic pattern across a tile. [Following the meeting a missive from Mike Irwin informs that the differences likely originate from the fact that tiles pre 20101030 have not received detector-level magzpt adjustments as per full v1.1.] Non-survey Data Release: Nothing to report. Astrogrid deployment, VO & Data Analysis services: MSH reported that all the AG services are now hosted by WFAU. Miscellaneous: RGM suggested that once things were back to routine operations we should conduct some sort of hardware morbidity and mortality review to see what we can do in future to prevent similar issues. -- Scanned by iCritical.