From nch@roe.ac.uk Sat Aug 6 10:26:04 2011 Date: Fri, 5 Aug 2011 16:41:34 +0100 From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , "Noddle, Keith -- Keith Noddle" , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project team meeting mins, 5/8/11 Minutes of WFAU VDFS Science Archive meeting: August 5th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RPB, KTN, MSH, MAR, RSC Apologies: JPE, AL, RGM DoNM: 10am Friday August 19th 2011 in the Vista Hut Actions discharged this time: ----------------------------- ACTION: RSC and ETWS to investigate seg faults for early CUs when running on the newest curation client systems. Discharged; suspect hardware problem (see below) ACTION: NJC to nudge Will Sutherland concerning bright star and diffraction spike flagging algorithms as may have been developed for Vista images. Discharged; email followed up by phone call and we've been promised something next week. Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC to restore temporary ingest file writing to high bandwidth shares between curation clients and DB servers. Continues; after some discussion it was agreed to create a new high capacity partition on an infiniband crossmounted NAS box. ACTION: ETWS to enquire about availability of SDSS DR8 SQL DB from the States. Continuing; DB not available from the Chicago data centre, but Hopkins have offered a copy by ftp (5TB compressed, 12TB native!). ETWS is finding out over how many volumes the DB files are split while we look for somewhere to put it before initiating transfers... ACTION: ETWS, with RPB, to develop an automated helper script to keep release DB files links updated. Work in progress ACTION: RPB (or whoever communicates with the PIs) to keep up the pressure on world public release DB preparation by emphasising that the recently prepared DBs are not simply for internal consumption. VHS and VIKING nudged, with so far no response. The plan is to send one-month reminders, escalating to fortnightly, as any potential release dates approach... Actions carried forward from 22/07/11 meeting --------------------------------------------- ACTION: RPB to set up mirror/sync for flat file DBs using MS SQL replication. Continues; NCH noted that the MS SQL implementation may not be as streamlined as might be hoped... ACTION: RPB to try a proof-of-concept on detection table monthly partitions using the VHS to start. Continues Specific points and new actions: -------------------------------- Project management: NJC reported back from the recent VMC telecon, and noted that the next is scheduled for the end of September. NCH noted that there is another VHS ultra-cool dwarfs science working group telecon next Friday, which he will try to attend. WFCAM & VISTA updates: News from UKIRT (courtesy of Watson Varricatt): "After the engineering shutdown, we were back in operations on 2011-07-22 UT. During the shutdown period, WFCAM was opened up and the J filter mechanism, which was giving problem, was inspected. The microswitch was fine. There was a slight play seen in the J filter paddle handle, which is new or might have been there, but increased recently. Anyway, as an anticipatory measure, the locations of the much used J and less used Z filters are now swapped the 'open' positions are moved back a little bit for all filters. No tripping was noticed during the last one week's operations." This week ESO announced availability of Vista data through the ESO-SAF: http://archive.eso.org/wdb/wdb/adp/phase3_vircam/form ... so WFAU can all give up and go home then. Comments and issues arising from CASU minutes: No new minutes this week. Networking: MSH and ETWS noted that the UK Light connection is broken (while the standard Janet connection is not). MSH is liaising with local ITSG to see if it's a problem at our end. WSA/VSA Operations: Most of this week's meeting was taken up with a discussion about how to handle the latest reprocessing of WFCAM data at CASU. Our understanding is that GPS PI requested a rederivation of photometric zeropoints prior to DR8, and also the catalogue classifier has been rerun as part of this. Turns out that MJI has done literally everything (i.e. not just problematic GPS frames, not even all GPS, but *all* WFCAM going back to 2005) - what a trooper. This leaves us with something of a problem in either grabbing just the GPS and leaving everything else out of sync with CASU, or wholesale copying up of all the new stuff. After considerable wailing and gnashing of teeth, it was decided to grab everything again. *CASU please take note*: WFAU will badge this as version 4 ("v4") for folder naming up here (despite it not having an official CASU reprocessing label). So we kindly request PLEASE do not subsequently reprocess anything with label VERSION_4 at your end - move directly to 5. It's the only way we can keep track of things before (and after). ETWS reported: -- Got a reply from Ani Thakar regarding the SDSSDR8, it is 5.18TB compressed (12.63TB restored) and available per FTP download of backup files from JHU. -- Re-calculated max/min values for RA and Dec in the VSA and updated the database accordingly. This fixes these values for 'bulging' frames and also the errors in rotated frames across the 0h border. -- Updated the 3rd party software on khafre after the debian upgrade by MSH. -- Tried to fix TilePawPrints table issues with the help of MAR. NCH enquired about WISE integration into the WSA/VSA. RPB and ETWS noted that the DB is ready, but were uncertain as to the status of crossneighbour definitions for the various surveys ACTION: RPB to review crossneighbour definitions for WISE (and at the same time) the SSA for all WSA/VSA surveys. NCH also asked the RPB do a one-off special and copy WISE crossneighbs into the latest VHS release DB for the ultracool dwarfs science team. Hardware and Systems: MSH noted that sahure is sick with some kind of intermittent fault (possibly memory or otherwise motherboard related). All investigations have so far drawn a blank; he plans to continue testing until next Tues and then bounce it back to the suppliers. KTN voluntered to start the ball rolling on negotiating a replacement. ETWS and MSH noted that khafre is now updated with the latest Debian release and all software installations. MSH noted that a new 0.15 PB (native) NAS box is on it's way. Software: NJC noted: "I have been working on sorting out the TilePawPrints matching for VISTA and other issues such as the VVV best matching. I found that missing observations from tiles were not being correctly processed in the best matching and have revamped the TilePawPrint tables to do a better join with the Source tables so that every row in the Source table will get a match even if it is default. I have also made sure that VIDEO can be processed all the way through." RSC reported: "Following full-scale tests on the GCS the proper motion solution implementation has been further refined to reject frame sets with hopeless astrometry discrepancies, which will work well for the GCS release where only one multiframe (the Antares field) had such a problem, but further testing on the LAS/GPS should be done sometime soon to make sure we aren't rejecting too many frame sets for these surveys. CU19 (and SyncDb) has been updated to support the release of the new Tile<->PawPrint relation tables. I've also experimented in changing the location of temporary files, including ingest files, for curation scripts from the NAS space to the faster, infiniband connected, Windows catalogue servers' mounted share spaces. Concerns over disk space and additional complexity / reliability issues of changing the software have lead us to consider trying instead a straight swap of the NAS space location to a local curation server disk cross-mounted to all curation servers via infiniband. This could also be potentially mounted by the Windows catalogue servers to allow ingests to occur without an additional transfer step." MAR noted that he has worked on VSA UI usernames/communities/sessions so as to implement a request from VVV to manage their own communities. Survey Data Release: There was much debate this week concerning the status of prototype Vista world-public release DBs, and worries that after initial positive responses, it's all gone quiet again, other than at the last VMC telecon where the PI noted that v1.1 products haven't been delivered to ESO yet so our proposal and schedule are unacceptable. ACTION: NCH to continue negotiations with VMC PSPI concerning a world release DB. Generally, the worry up here is that this is a priority for nobody except WFAU, despite the fact that ultimately it's the only thing that will enable world community exploitation of the surveys. The team discussed the plans for keeping this on the PSPIs radar, and monthly reminders will be sent out to try to keep up the momentum. MAR suggested that some more visible release and usage stats on the web pages might help ACTION: MAR to make release status and usage stats more visible on the VSA web pages. Non-survey Data Release: Nothing new this week. Astrogrid deployment & Data Analysis services: No news this time. Miscellaneous: Nowt else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.