From acd@roe.ac.uk Thu Jul 31 13:26:55 2014 Date: Thu, 31 Jul 2014 10:50:00 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tkerr@jach.hawaii.edu" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 25 July 2014 WFAU VDFS Science Archives progress meeting minutes, 25 July 2014 Present: RSC, RPB, MAR, STV, ACD Apologies: ETWS, NJC, RGM, NCH, KTN, AL, MSH Note: ETWS kindly supplied written comments in advance of the meeting. DoNM: 10am Friday 8 August 2014 in the Villas meeting room, MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: RPB estimate the number of additional tapes that would be needed to backup all the data on disk. Discharged (duplicate action). ACTION: NJC to circulate the relevant VPHAS related e-mail messages. Done. ACTION: RPB to sort out disk space for the new OSA ingest database. Done. Space was now available. ACTION: MAR to finish off GPS QC and add an entry to the forthcoming deadlines. Done. MAR had identified the issue causing the problem (or at least an issue causing a problem) and consequently two months of data were being retransferred from Cambridge. Once the new files are available MAR will need to apply the QC to them. Actions partly discharged but continuing: ---------------------------------------- None. Actions carried forward from 20/6/2014 meeting: ---------------------------------------------- ACTION: RPB to put all prices of additional items associated with the backup policy on the Twiki pages. Continues. RPB had obtained the prices but not yet put them on the Twiki page. In summary the costs would amount to a one-off expense of ~£7000 and a recurrent monthly expense of ~£130. ACTION: RGM to investigate the size and content (how does it differ from DR9?) of DR10 to see if it's worth us hosting and whether it is necessary to keep all of the other releases. Continues. RGM had started a wiki page for WFAU SDSS strategy http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan. ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Continues: this one still has low priority. ACTION: MAR to document issue with ATLAS unpaired sources and give examples of recovering matches. Continues. ACTION: ACD to arrange a follow-up VVV brainstorming meeting. Continues. RSC was organising the testing and the meeting should be organised ICW him. RSC noted (i) the meeting would not be held before RGM returned from annual leave in mid-August and (ii) further progress needed to made on the actions and assignments listed on the relevant Twiki page before it was worth holding a meeting. NEW ACTION: ACD to review Twiki page and decide who to chase about progressing their assigned actions. ACTION: RPB to write a script to backup the SVN repository. Continues ACTION NJC: update ESO VISTA releases on the deadlines web page. Continues, NJC not present ACTION: MSH to produce a costed list of the additional kit necessary to permit wholesale reprocessing of UKIDSS/VISTA. Continues. ACTION: ACD, RGM: review finances. Continues. ACTION: RPB to make the necessary Spitzer SMC/VMC neighbour table entries. Continues. Specific points and new actions: ------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report; the telescopes were believed to be operating normally. Comments and issues arising from CASU minutes: * No minutes had been received since the previous meeting. Review of Deadlines: * Due to slippage the following deadlines needed to be revised on the Twiki page: - 31/7/14: ATLAS DR2 release (P92) - 31/8/14: UKIDSS GPS release to DR10, * RPB noted that the deadline of 25/9/14 to start CU19 for VVV DR3 was unlikely to be met. * ACD noted that the much-delayed GES iDR2 release had happened and data for iDR3 were likely to become available to WFAU by the end of July. ACTION: ACD ensure that the Twiki deadline page is updated. Networking: * ETWS noted that he had: - Transferred VISTA May data. - Transferred all OSA data up to June 2014 as version 1.1 - Transferred the AllWISE Source Catalogue: 360GB compressed, 1.2TB uncompressed. This catalogue was being made available in response to a request by Nicolas Lodieu for WFCAM but the WISE catalogues are also used in VSA and OSA. WSA/VSA/OSA/GES Operations: * RPB reported that: - He was sorting out the AllWISE catalogue. - The VSA VVV ingest database was back on-line and fully backed-up (but the disks are not replicated). VVV detection objID updates have been restarted following the earlier rollback. * ETWS noted that he had: - Re-transferred and ingested some WSA user-requested data. - Split the CU3 operations for the OSA into MultiframeID update and metadata processing. This change made the files ready for CU2 earlier, since they need the missing CAMNUM keyword fixed before JPG creation. Consequently, CU2 has already finished and the metadata processing part should finish at the end of the week, roughly a week earlier than he had anticipated. There are six files that have not processed smoothly and need to be investigated. The list-driven pseudo-pixel file entries have also been created, so that CU4 ingests can already start next week. * ACD reiterated that the GES iDR2 release had happened and data for iDR3 may be available from the end of July. Hardware and Systems: * RPB reported: - A database node had arrived and was currently being set up. Once operational it should relieve the shortage of disk space. - Kakai had experienced a hardware failure which had be solved by replacing a RAID card. - There had been a disk failure on Pedubaste which had required a disk to be replaced and a subsequent reboot. Software: * MAR noted that a facility had been added to the GES Web site to allow access to some of the site's pages to be restricted to Consortium members. * ETWS reported that he had: - Written the schema for the AllWISE Source Catalogue and handed all the SQL scripts to RPB to ingest into the WISE database. - Fixed minor issues with the new code for the OSA ingests. - Done the bookkeeping for the observation days of OSA, WSA, VSA. Survey Data Release: * The only development was the GES iDR2 release previously mentioned. Non-survey Data Release: * No non-survey releases had been made since the last meeting. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * A consensus had emerged amongst all the interested parties except NJC that the best way to construct VVV ObjIDs was to use the table primary key. There were, however, some issues concerning the huge multiFrame IDs for data products (a discussion ensued about this point which was as protracted as it was unintelligible). In due course ETWS would have to implement the solution so any final decision should wait until he is present. -- Scanned by iCritical.