From nch@roe.ac.uk Fri Aug 31 12:22:05 2007 Date: Fri, 31 Aug 2007 12:07:20 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 31/08/07 Minutes of WFAU VDFS Science Archive meeting: 31st August 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, MAR, NJC, JB, RGM, JDT, MSH Apologies: BCW, JPE, AL, JMS, LGR, ETWS NB: DONM: 10am, Friday 14th September 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: MAR to substitute the world EDR announcement in place of the DR2 release announcement on the WSA homepage. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details; will now check WSA SRAD and VISTA URD for any mention of this. Actions carried forward from 24/08/07 meeting: ---------------------------------------------- ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). Specific points and new actions: -------------------------------- Project management: Next week's VDMT has been postponed until (probably) the following Monday. ACTION: PMW to assemble the usual reporting materials for the next VDMT during the coming week. NCH noted that a second iteration of the JAC/CASU/WFAU MoU has been received from the director JAC; this has been distributed to PMW, RGM and AL for comments. WFCAM & VISTA updates: VISTA news: JPE reports that a recent trip to ESO is for discussions about the (successful) bid to get EC funding to lay a 1 Gbps optical fibre link between Paranal and Antofagasta. When installed the 'joint research activity' he is organising is to demonstrate its utility by transferring VISTA (& VST) data to Garching and them (certainly for VISTA) to Cambridge. An email from Luca Rizzi (UKIDSS survey support scientist) at JAC communicates some not-so-good news: "Unfortunately, we will have a prolonged shutdown of UKIRT starting Friday August 31, and possibly until September 12th. WFCAM has shown signs of problems with its cold head, and will be taken off the telescope. We have tried to observe as long as possible before shutting down to ensure some completeness of the projects located at the beginning of the night, but had little luck, since the last two nights were plagued with worse than average seeing and computer problems. We will keep you posted on the evolution of the situation." We wish JAC all the best in tracking down and fixing the problem. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS received a note from Ani Thakar at JHU that there are no plans to distribute a split multi-volume version of SegueDR6. The team decided to delay a decision as to whether we transfer a copy of the single volume (and presumably single 0.5TB DB file) version from Chicago until October for possible crossmatching in UKIDSS DR3. JB noted that UKLight broke last weekend somewhere between Warrington and Edinburgh (something to do with the recent seismic activity around Manchester perhaps...?!): "Last weekend there was a (minimum) twelve hour break in the UKLight service due to a broken fibre between Leeds and Edinburgh leading to a slow down in transfers (and average speed). CU1 simply continued from where it left off after the link was restored." JB noted further that CU1 has almost completed transfer of 06B data with about a week of data left to go (including the three reprocessed nights). WSA Operations: JB reported: "Early curation continues with CU1 having almost completed its task, CU3 about a week behind at present and CU2 a week behind that and CU4 a week behind that. Sneferu is currently undergoing a software rebuild [so is not available to process and ingest data - Ed]" MAR reported further progress with QC for UKIDSS DR3; NCH noted that eyeballing checks should be complete by close of play today (GPS and DXS received so far). ACTION: NCH to send MAR the GCS eyeball check results, and forward the DXS results (received yesterday). Hardware: PMW asked about the plans to integrate the public catalogue server into the new full-height rack; the team decided that a good time to do this would be after the current telescope proposal deadlines in September, but before the release of DR3 to avoid any interuption to access (especially to the latter) at critical times. ACTION: MAR to put a note that some archive services will be offline for up to a day on Mon 1st Oct on the WSA website downtime page. JB reported: "One of khufu's disks is causing problems and I am going to get Eclipse to send a replacement and swap in a new drive. NJC has now been transfered to the new disk setup on Djoser (having sucessfully moved disk02 to the NAS system) which will mean his disk intensive work will perform better and cause less load on other machines. Disc speed tests continue though it is not very clear that this data will help much due to the plethora of processes running at the same time as the tests (it is a good test of the system under load though)." Software: RSC reported: "Following last weeks brief discussion in the meeting I wrote a CurationOverview TWiki article clarifying the order of running release CUs, the dependencies between them, how to run them, and what to do if the script is interrupted due to e.g. hardware failure. This proved to be a useful exercise as it identified a bunch of improvements that could be made to the CU7 command-line interface. I've now provided better support for semester-delimited date ranges, and an option to simply and automatically recreate a source table from scratch. Also, an option to skip source table reseaming, which will be useful when breaking up the source merging of the GPS, as it will allow the reseaming stage to be also broken down into chunks in the event of the script being interrupted. I've also being liaising with NJC on his improvements to CU13 and conversion from Numeric to Numpy, and have got in contact with "Astronomy Picture of the Day" to inform them about our new WSA gallery - no response yet." NJC reported: "I converted Cu13_stack and Cu13_mosaic into CuSession classes with the CLI interface so they now fit in with the new software architecture. I have fully updated cu13_mosaic and Statistics to numpy and added more options to Statistics. I have updated a lot of old SqlWrappers to use DbSession objects and have deprecated some of the sql fragments, still left around from the old architecture. General cleaning up, and making everything clearer." MAR reported: "Finally got Lorenzo's dynamic plotting to work without re-course to system calls (seems a shared library was missing from Tomcat's path). Updated gallery and other web pages as per various suggestions." JB noted that he and ETWS have been doing the usual maintenance of the operations software for small enhancements and bug fixes: "I have been working with Eckhard to remove obscure bugs from the early CU's as well as the usual bug fixing and addition of flexibility to the software." Survey Data Release: Revised schedule (weeks remaining) in the light of the past week's progress: Weeks What: 1.0 CU1 (transfer - assuming sustained 3MB/s) 1.0 CU2 (jpegs) 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. Note that the degree to which some of QC1 can run in parallel is a little uncertain and the above estimates are generally pessimistic; the worst-case scenario of is DR3 release by the end of November. Everything will be done to expedite and arrive sooner. NCH noted that it is important to notify CASU as soon as our transfers have finished so that final application of the photometric zeropoint calibrations can be done down south and the resulting complete list of updates sent north and applied at this end prior to QC. ACTION: NCH to enquire as to the status of the photometric zeropoint recalibration activities at CASU. Non-survey Data Release: Nothing new to report this week. Astrogrid deployment: Nothing new this week. Miscellaneous: Finally, JDT is departing soon to join Google Corp, and in order to send him off in a shower of beer: Very early doors at lunchtime today in the Braidburn (assemble at 1pm in the lodge). Standard early doors today at 6pm in the Old Bell after work. Advanced notice of supplementary early doors on Friday September 21st again after work (or maybe an all-day bender at this rate...) to finally say cheerio.