From nch@roe.ac.uk Thu Aug 25 12:04:10 2011 Date: Wed, 24 Aug 2011 17:17:06 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting mins, 19/08/2011 Minutes of WFAU VDFS Science Archive meeting: August 19th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RPB, MAR, RSC Apologies: JPE, AL, RGM, KTN, MSH, C37, 89X DoNM: 10am Friday September 2nd 2011 in the VH Actions discharged this time: ----------------------------- ACTION: NCH to continue negotiations with VMC PSPI concerning a world release DB. Discharged; started off by proposing a solution to the field-delimited requirement (also for VIKING) Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Continues; system level tests being carried out by MSH ACTION: ETWS to enquire about availability of SDSS DR8 SQL DB from the States. Continuing; DB not available from the Chicago data centre, but Hopkins have offered a copy by ftp (5TB compressed, 12TB native!). ETWS is finding out over how many volumes the DB files are split while we look for somewhere to put it before initiating transfers... ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Work in progress (some problems with OS level permissions on r/o files - MS DOS strikes again) ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation by emphasising that the recently prepared DBs are not simply for internal consumption. VHS and VIKING nudged, with so far no response. The plan is to send one-month reminders, escalating to fortnightly, as any potential release dates approach... ACTION: RPB to review crossneighbour definitions for WISE (and at the same time) the SSA for all WSA/VSA surveys. Continues; WSA needs to be reviewed. ACTION: MAR to make release status and usage stats more visible on the VSA web pages. Done but not live (yet). Actions carried forward from 5/08/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... ACTION: RPB to try a proof-of-concept on detection table monthly partitions using the VHS to start. Continues Specific points and new actions: -------------------------------- Project management: Nothing of any consequence this time. WFCAM & VISTA updates: Nothing of note. Comments and issues arising from CASU minutes: No new minutes this week. Networking: GALEX has arrived from STScI for assimilation into the VSA/WSA. UKIDSS DR6 will be sent back by return on the 3TB external HD supplied. ETWS noted that the network speed has dropped recently to a paltry 10 MByte/s. Investigations are on-going. WSA/VSA Operations: ETWS reported: -- Enhanced CU3 to ingest data by given survey. This will enable us to ingest the reprocessed GPS data only and not interrupt the relelases of the other surveys, since the re-ingests would deprecate the old data. -- Enhanced CU1 to get only the re-processed data and copy it as version 4. -- Started the transfers. These showed up one day where we had lost the previously transferred data, so the full day will be re-transferred. It also showed up 2 days in 2006 that are flagged with "no data" at CASU, but the directories there have data and have also been reprocessed. Transfer rates are relatively slow, down to half the usual speed. This might still be related to the connection problems at Cambridge. -- Worked on the remote release database update, but encountered problems changing the file permissions remotely on the windows server side. -- Created new browser pages for the latest released VSA databases. -- Contacted Ani Thakar about the SDSS BestDR8 release. NCDM (Chicago) had some major problems with their storage infrastructure early this year and they are still recovering. This means they are still only delivering BestDR6 and 7. We can get BestDR8 from JHU via ftp, but this would take twice the time (~4 weeks) to transfer. Also I haven't heard back yet if the archive is a monolithic or a split version. It's 5.18 TB compressed and 12.63 TB uncompressed. Hardware and Systems: Flaky curation serve sahure has now had it's memory replaced by the maintenance contractor, so fingers crossed that this has cured it's recent sickness. Software: Testing of the proper motion codes is continuing with an LAS DR9 dry limited to the DR8 data (10A/B to be appended once the eyeball QC is in from SJW). Speaking of the UKIDSS CSS, a request has been sent to define a new errorbit for edge effects over and above those due to dither offset. A new prescription has been sent, which RSC will be incorporating into the qbits flagging code in time for DR9. On the same subject but wrt VSA, RSC noted a few issues over implementation of the VIKING PI's bright star crud flagging algorithm. Guided by the ensuing discussions, the indefatigable RSC will be implementing some of these features as well. NJC noted further optimisations of CU6, and that it's run on the VVV continues ... Survey Data Release: DR9 schedule is just about on schedule, despite continuing fettling of the pm codes, quality error bits and, lastly but not leastly, awaiting the LAS eyeball QC results. Non-survey Data Release: Nothing new this week. Astrogrid deployment & Data Analysis services: No news this time. Miscellaneous: Zut alors: ADASS have announced the opening of abstract submission for the 21st annual conference on Astronomical Data Analysis Software and Systems. The conference will be held in Paris, France, at the Paris Mariott Rive Gauche Hotel and Conference Center during 6-10 November 2010. http://www.eso.org/sci/meetings/2011/adass2011.html RGM reminds us all that there's plenty cash in the travel pot, so all who wish to participate should go. Presently intending to go from the VDFS team are NJC, RSC (abstract: pms in the WSA) and MSH. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.