From nch@roe.ac.uk Mon Jul 25 11:39:53 2011 Date: Mon, 25 Jul 2011 10:01:44 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting minutes, 22/7/11 Minutes of WFAU VDFS Science Archive meeting: July 22nd 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RPB, RGM, KTN, MSH Apologies: JPE, AL, MAR, RSC DoNM: 10am Friday August 5th 2011 in the Vista Hut Actions discharged this time: ----------------------------- ACTION: RPB to look into RAID monitoring on the NAS boxes Discharged; all fixed up with RPB and MSH beeing notified by email if any degredation. ACTION: RPB and MSH to come up with a sustainable storage expansion plan to keep pace with the VVV Discharged; see Hardware below. Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Actions carried forward from 8/07/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... ACTION: RSC and ETWS to investigate seg faults for early CUs when running on the newest curation client systems. Continues; Specific points and new actions: -------------------------------- Project management: NCH and RGM noted that a decision on spending residual staff resources from the WFAU grant is imminent (final decision next week) following discussions of various options. Likely outcome is to get an extra pair of hands temporarily to progress VST/ATLAS archive. WFCAM & VISTA updates: Nothing new this week (we're assuming UKIRT is reopening after the recent shutdown...). Comments and issues arising from CASU minutes: No new minutes this week, but NJC noted that he'd spoken with MJI at this week's survey science meeting in Hertfordshire (see below). Networking: WFCAM processed data all up to date (to June) and ingested for flat file availability. RGM noted that a GALEX SQL DB is on it's way from STScI, the deal being to send UKIDSS DR6 by return on the same HDD. RGM also asked about the status of SDSS DR8, and if there were any plans to incorporate this into the archive. ACTION: ETWS to enquire about availability of SDSS DR8 SQL DB from the States. WSA/VSA Operations: ETWS reported: - Transferred and ingested April, May, and June data into the WSA and May v1.2 data into the VSA. - Updated the browser pages for the latest VIKING and VVV release. - Finished processing (mainly updating the HTM indices to level 20) for the WISE data. - Started checking all WFCAM release DBs for inconsistencies in FITS and JPG file paths. - Worked on implementing code for finding the maximal RA and Dec of detectors in VISTA which have "bulging" sides due to the low Dec and/or the TAN projection. RPB reported: "VIKINGv20110714 release database created. VVVv20110718 release database created, albeit missing a few tables that will be added in later as they are created. VMC v1.1 release database has had a few problems with variability code (CU6), though hopefully this will be fixed soon and a release will be imminent. Problems with TilePawPrints (wrong data type, no default rows) fixed in VSA, as well as VHS release database. VIKING release database to be fixed shortly. WISE database created and in the process of being deployed onto the live servers." NCH noted that a user had complained this week about broken links to jpegs, and that shuffling of data was routinely the cause of such issues. The suggestion is to make it much easier to keep the release DBs up to date by writing a helper script to automatically detach, reattach in read mode, fettle the links, detach, update the logins etc, then reattach in read-only mode such that this process is much more streamlined. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. With regard to the VVV ingest bottlenecks, RPB noted that these are inevitable at some level since parallelisation is extensively used during creation of the ingest files, but they are loaded against one instance of MS SQL. However, it was agreed to go ahead with splitting the detection table by month and then recast the detection table objects as union views over those partitions, to expedite table operations on (much) smaller chunks and enable parallel operations on different time ranges. ACTION: RPB to try a proof-of-concept on detection table monthly partitions using the VHS to start. Hardware and Systems: MSH noted that the storage expansion upgrade plan centres on replacement of disks with 3TB units, and acquisition of new NAS boxes containing the same, to keep pace (mainly with VVV). A few tests on the performance of RAID6 arrays need to be done first, to ensure the write performance (in particular) doesn't slow things down too much. The intention is for such NAS storage to be used on both the mass file-store units and the DB servers. Software: NJC noted: "I have been working on speeding up and adding more breakpoints to code such as the SynopticMerger, TilePawPrintMerger and the variability and discovered a small error in the extNum type in the TilePawPrints table. I have attended the VVV/GPS meeting and met many of the Chilean side of the VVV team and gave a talk about the WFAU work and a demonstration of the VSA. I learnt a lot about the concerns of the team: that a lot of work may have to be done at the pawprint level for best results, about the difficulties of bright stars and the diffraction spikes in particular. I also learnt that CASU may soon be producing PSF photometry, so we need to be ready for reingestion of catalogues in the future." Survey Data Release: NCH welcomed the recent v1.1 releases of Vista survey DBs to the respective PSPIs over the last week (VHS, VIKING and partial VVV). It was suggested that we need to emphasise that these products are prototype "public releases" for the survey teams to sanity check for public consumption, as well as for their own science exploitation. It was agreed that this would be emphasised in future communications. ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation by emphasising that the recently prepared DBs are not simply for internal consumption. NJC reported back from the Herfordshire meeting earlier this week, noting useful conversations with the VVV and VIDEO teams. A demo of the most recent VVV (partial-) release was given to a sizeable bunch of interested astronomers. NJC noted several presentations on various aspects of VVV use (variability, DIA, crud flagging near bright stars). ACTION: NJC to nudge Will Sutherland concerning bright star and diffraction spike flagging algorithms as may have been developed for Vista images. Non-survey Data Release: Nothing new this week. Astrogrid deployment & Data Analysis services: No news this time. Miscellaneous: NJC noted that Istvan Dekany from Chile is visiting WFAU Mon/Tues next week, to gain experience in using the VVV DB and to speak to various folks. NJC noted also that local interest in the VIKING images for the purposes of photometric redshifts as a part of 3d weak lensing studies in conjunction with VST-KIDS is on the increase, and he would be helping them with their initial looks at the data. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.