From acd@roe.ac.uk Wed Aug 3 17:32:33 2016 Date: Wed, 3 Aug 2016 16:32:21 +0000 From: Clive Davenhall To: Bob Mann , Andy Lawrence , "Read, Mike" , Eckhard Sutorius , Mark S. Holliman , Nicholas Cross , Nigel C. Hambly , Rob Blake , Ross Collins , Stelios Voutsinas , Dave Morris , Clive Davenhall Cc: "dante@astro.puc.cl" , James Dunlop , "j.p.emerson@qmul.ac.uk" , ktn keithnoddle.org , "Lorenzo.Rimoldini@unige.ch" , "mcioni@aip.de" , "matt.jarvis@astro.ox.ac.uk" , "mike@ast.cam.ac.uk" , "nigel.metcalfe@durham.ac.uk" , "norman@astro.gla.ac.uk" , "pwl@star.herts.ac.uk" , "rgm@ast.cam.ac.uk" , "Simon.Dye@nottingham.ac.uk" , "s.j.warren@ic.ac.uk" , "tom.shanks@durham.ac.uk" Subject: WFAU VDFS Science Archives progress meeting minutes, 29 July 2016 WFAU VDFS Science Archives progress meeting minutes, 29 July 2016 Present: RPB, RSC, MSH, NJC, ACD Apologies: ETWS, MAR, STV, DMR, KTN, AL, RGM DoNM: 10:00 am Friday 12th August 2016 in the Villas Meeting Room MAR in the chair (room is booked). Actions discharged this time: ---------------------------- ACTION: NJC to set-up WISExSCOS neighbour tables. Discharged. ACTION: RPB to restore VSA replication. Discharged. Actions carried forward from 15/7/2016 meeting: ---------------------------------------------- ACTION: RGM to investigate the size and content (how does it differ from DR9?) of SDSS DR10 to see if it is worth us hosting and whether it is necessary to keep all of the earlier releases. Continues. The first part had been done: the difference between DR9 and DR10 is largely the presence of additional spectra. The part of this action that continues is that RGM had not yet established which early releases could safely be scrapped. RGM also noted that that DR11 and DR12 were now available and now need to be looked at. For further details see the WFAU SDSS strategy wiki page: http://apache.roe.ac.uk/twiki/bin/view/WFAU/SDSSDRPlan We currently have space so it was decided to start transferring DR12 ACTION: ETWS to transfer SDSS DR12/13 Continues. ETWS had had contacted Ani Thakar, who had advised that SDSS DR13 would be released at the end of July (and that SDSS DR12 had recently been removed from the SDSS servers in preparation). RGM was happy to wait for DR13 but did not want further significant delays as one of his students planned to work with the data. ACTION: RPB, NJC to fix the errors encountered while trying to make non-survey releases U/14B/UA15, U/14B/UA16, U/14B/UA18 and U/14/BUA19. Continues. NJC believed that he had fixed the problems with the two releases that previously failed. RPB should attempt to re-run these releases. ACTION: ACD to liaise with DMR and STV to establish a set of deadlines for their work. Continues. ACTION: MAR to look at H-ATLAS documentation. Continues. ACTION: RPB (was RPB, NJC) to determine if the necessary OGLE neighbour tables had been set-up and to ensure that they appear in the next VMC release. Continues. NJC had set up the neighbour tables. RPB needs to ensure that they are included in the release. Specific points and new actions: -------------------------------- Project management: * Nothing to report. WFCAM, VISTA and VST updates: * Nothing to report. Networking: * Nothing to report; the networks were operating nominally. WSA/VSA/OSA/GES Operations: * RPB reported the following progress: - ETWS was believed to have ingested all VSA P96 data, - processing of the VMC P96 consortium release has started; currently the deep products are being created. - VVV is progressing and the TilePawPrints table is currently being updated. The task to set the ppErrorBit flags will be run once NJC has corrected the bug that previously caused it to crash. * ACD noted that Cambridge had reported that some GES abundances had been omitted. He had found and fixed the problem. Cambridge further advised that another node had belatedly provided a set of parameters and abundances and they would be forwarding these shortly. Hardware and Systems: * MSH noted that after a long period of no reported disk failures he had checked the settings on all the servers and discovered that 3 of the database nodes and 4 of the NAS boxes had somehow reset their email settings to a broken default, so failures were not being reported. All of the affected machines have similar RAID cards, so the fault may be a bug in the firmware. Henceforth RPB and MSH will check the settings monthly until they are sure that they have fixed the problem. One of the NAS boxes, shepsesre, is on the verge of collapse and has a large (57TB array) in a precarious state. Further to this, the disk activity/identification lights are not functioning (so MSH cannot identify which data is in danger) and the RAID card is intermittently reporting power problems. Consequently the dead disks cannot be replaced as any power fluctuation could cause a failure of the entire array. MSH is actively copying all 108 TB of data off the NAS and onto other hardware. The copy is up to about 90TB so far, and once it is complete he will start troubleshooting the broken array. Review of Software Release Deadlines: * Progress towards the software deadlines was reviewed briefly. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/PrepareSoft Other Software: * NJC noted that he needed to sort out the VVV programme setup. He knew how to address this issue and intended to test the setup on real VVV data. The new programme would allow for the merging of synoptic sources. The target list would be generated without computing parallaxes and would follow normal VSA practice. Review of Survey Release Deadlines: * Progress towards forthcoming survey deadlines was reviewed. See: http://apache.roe.ac.uk/twiki/bin/view/WFAU/ForthcomingDeadlines Survey Data Release: * None Non-survey Data Release: * RPB reported that the following WSA 15A non-survey projects had been released: U15AH01 U15ANA05 U15AUA05A U15AUA12A Releases had been attempted but failed for a further 5 non-survey projects. The reason for these failures needs to be identified and rectified. ACTION: RPB, NJC. RPB should examine the log files to identify the problem and NJC or RPB should then fix it. Finally MAR had a indicated that a set of WSA 15B projects were ready for release but RPB had not yet attempted to release them. Astrogrid deployment, VO & Data Analysis services: * Nothing to report. Miscellaneous: * NJC noted that Oscar Gonzalez had approached him about holding a VVVX meeting at the ROE in June 2017 and asked him to be one of the local organisers. This approach followed discussions between Oscar and Dante Minniti. The proposed meeting would be focussed on VVVX but would also include some discussion of MOONS. NJC should discuss the meeting with RGM before proceeding. RSC and ACD should be kept apprised of any developments relevant to MOONS. ACTION: NJC to discuss proposed VVVX meeting with RGM and to keep RSC and ACD apprised of developments pertaining to MOONS. * MSH noted that he had been called for jury service on Wednesday 3 August. ACD, 3/8/16.