From nch@roe.ac.uk Fri Aug 24 15:10:12 2007 Date: Fri, 24 Aug 2007 14:03:43 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 24/08/07 Minutes of WFAU VDFS Science Archive meeting: 24th August 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, MAR, NJC, JB, LGR, RGM Apologies: JDT, BCW, JPE, MSH, AL, JMS DONM: 10am, Friday 31st August 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to backup the WSA ingest DB this weekend. Discharged ACTION: JB to check out the available disk space on ahmose/amenhotep to ensure there's enough space for DR3. Discharged; JB estimates that we have enough space for DR3; if push comes to shove on the public server amenhotep then the SSA and 2MASS can be moved away to make space. ACTION: MSH to liaise with JNTD on transfering the SSA ingest files from cosaxp6 to hatshepsut and to do some load tests. Discharged; MSH reported by email that JNTD has set up the full Gbit link to new catalogue server hatshepsut and the plan is to now ftp the SSA intermediate ingest files over for a stress-test of SQL2005. ACTION: ETWS to check WSA Python code for StringIO change. Dishcarged; no further problems found in extant Python code. ACTION: NCH and RSC to merge in latest speed enhancements for CU7 into SVN. Discharged; RSC noted that at the same time the modifications for master RA,Dec and epoch in the source tables have been made to make things easier for end-users Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details; will now check WSA SRAD and VISTA URD for any mention of this. Actions carried forward from 17/08/07 meeting: ---------------------------------------------- - none this week. Specific points and new actions: -------------------------------- Project management: With reference to the December ESO Workshop on UKIDSS, MAR noted he may also possibly attend; RGM asked what kind of meeting it was likely to be. NCH noted that the pre-registration form asks for delegates to request talks as well as offer them, so it will be tuned to what the community wants. RGM suggested LGR should go to demonstrate his latest analysis tools and to engage with the UKIDSS community. WFCAM & VISTA updates: NCH noted that WFCAM Semester 08A has been announced as start Feb'08 to end June minus 3 weeks somewhere towards the end for re-aluminising the primary. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 22/8/07; see below for clarifications. Networking: ETWS noted that UChicago and Ani Thakar at JHU have supplied more details concerning BestDR6 and SegueDR6 (which apparently does exist as a separate 0.5TB SQL database). Presently only single-volume copies are available at UC; JHU will be preparing multi-volume copies over the next month or so, and these will be available at some point from UC. NCH reiterated the priorities supplied by SJW were for Segue first if possible, and not to worry so much about DR6 as the gains are only incremental over DR5. The team agreed to monitor progress on the multi-volume SegueDR6 and integrate into WSA as and when it becomes available (possibly in time for UKIDSSDR3, but no promises...) JB and ETWS noted that CU1 transfers are now using single threaded ftp but Axel is available for testing multi-threads at some point and the NFS network is being rationalised so the hope is that transfer speeds will gradually improve. JB notes: "Automatic CU1 transfers on djedefre continue with most of December 2006 now transfered - transfer rates are approximately 3MB/s but climbing (with the introduction of ftp and removal of NFS crossmounts)." MJI/MR have proved details of the ~700 GPS files that have been reproduced for 06A following a fix in the crosstalk correction software in the pipeline. The team reiterated that it is very reluctant to delay DR3 any further than necessary by trying to insert these into the WSA by hand. WSA Operations: JB reported: "A backup of the WSA has been taken. CU2 has been run for all data from the 13thDecember onwards. CU3 was sucessfully tested on data from 19th December. Data continues to be moved to the NAS boxes to reduce NFS crossmounting and reduce PAN traffic load. The backup tapes are being rationalised. Shrank some databases on Amenhotep to make space for DR3 on it (I believe there is space on both Ahmose and Amenhotep for DR3 now). Removed 20070213 data in preparation for retransfer and ingest of fixed data." Hardware: JB reported that the specs for a new NAS box ae being considered. Software: NJC reported updates to schemas (particularly GPS source), some minor bug fixes and work on his galaxy photometry paper. RSC reported: "Finalised improvements to CU7 for DR3: Merged source positions are now the RA & Dec of the "master" detection (trac ticket:67). This allows us to now choose the MJD of the master detection as the "epoch" value of the merged source. Also, committed NCH's reseaming optimisation patches, and provided support to continue interrupted (e.g. due to a database connection loss, as often occurs with the long GPS runs) complete source table reseams from scratch (trac ticket: 91)." ETWS reported maintenance of CU1, in addition to using this and CUs2-4 for general ops. MAR reported: "Put gallery of WSA "pretty pics" online: http://surveys.roe.ac.uk/wsa/gallery/ Added coordinate readout and go to on GPS mosaic http://surveys.roe.ac.uk:8080/wsa/gps_mosaic.jsp Spent a couple of days trying to get LRG's dynamic data analysis plotting installed under Tomcat on thoth. Finally got it running but is currently resorting to a Java runtime call. The problem seem to be the cmobination of the JRE used by Tomcat and the "user environment" it's running under." JB noted contributing some fixes to CU1 and helped develop the ftp setup for it. Survey Data Release: SJW has requested that the recent world-release of EDR be flagged on the WSA archive pages ACTION: MAR to substitute the world EDR announcement in place of the DR2 release announcement on the WSA homepage. T0 is now confirmed as 22nd August. The team looked again at the release proto-schedule (units of weeks!): T0+ What: 2.0 CU1 (transfer - assuming sustained 3MB/s) 1.5 CU2 (jpegs) 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 0.0 CU13/14 for DXS/UDS (in parallel with shallow survey CU7s) 2.0 Final CUs. Note that the degree to which some of QC1 can run in parallel is a little uncertain and the above estimates are generally pessimistic; the worst-case scenario of is DR3 release by the end of November. Everything will be done to expedite and arrive sooner. Non-survey Data Release: JB reported that some new non-survey registrations await setting up in the archive; NCH reiterated that DR3 is the priority, but if possible to slot these in somewhere. JB noted that enhancements to non-survey DBs like 2MASS cross-neighbours are possible if done in the standard way as for surveys (making changes to WSA_InitiateArchive.sql, rather than simply poking curation-driving tables by hand). Astrogrid deployment: RGM noted that at this weeks VO project meeting the possibility of asking an ESAC VO representative to visit to help deploy SIAP/SSAP infrastructure was discussed, and this should be scheduled in the light of existing commitments. RGM also asked if it would be possible to restore BestDR1 for some test purposes. ACTION: JB to restore BestDR1 to thutmose from LTO tape at a convenient time (but no rush!). Miscellaneous: Nothing else this week.