From nch@roe.ac.uk Fri Mar 9 12:34:38 2007 Date: Fri, 9 Mar 2007 12:17:35 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 9/3/07 Minutes of WFAU VDFS Science Archive meeting: 9th March 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, MAR, MSH, JB, RGM Apologies: JMS, JDT, BCW, AL, MAR, JPE DONM: 10am, Friday 16th March 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH and RSC to refactor CU19 next week. Discharged ACTION: PMW to prepare the usual VDMT materials early next week. Discharged ACTION: NJC to communicate with JAC over the new filters being used in WFCAM, and input the necessary data into the WSA. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; BestDR2 is still being shrunk in preparation for moving to Thutmose while 2mass is a little too busy to take offline at the moment; will wait for a week or so until the clamour over DR2 dies down a bit before transfering it to Thutmose. ETWS discovered that the empty tables we have in BestDR5 are empty in the SDSS publically accessible version too. ACTION: RSC to profile CU4 and investigate possible optimisation Continues; on hold until parallelisation and other refactoring is complete. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; NCH, RSC and MAR discussed this in the week and because of some subtle complexities it will be put on hold until after DR2. RSC has made some code changes to make the curation applications robust against this very minor schema change when it does happen. Actions carried forward from 02/03/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: JPE to contact Tim Naylor concerning scheduling a VDUC meeting. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. Continues; some contact with JAC concerning likely timescale for fix-up of the late-Oct/Nov/early-Dec data. Specific points and new actions: -------------------------------- Project management: NCH noted the VDMT of this afternoon. PMW has prepared the usual materials. WFCAM & VISTA updates: No news this week Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB reported: "No news yet from UKLight, I have been in touch with Sam, in theory they should be in contact with us either today or tomorrow with a timetable for hooking us up (possibly in early April - i.e. too late to help much with the initial onslaught of 06B). There was a network glitch (our local network went offline) on Wednesday which caused GPS source merging some problems but the problem identified and fixed, this didn't stop the slightly less than necessarily robust CU7 from falling over (and taking two and a half days of processing with it) - see below for more details." WSA Operations: JB noted that we are now transfering five days of data initially that we will use to test for anomalies in the 06B data and the data flow system; data transfer is running at 3 to 4 MByte/s. System backups went as normal and ongoing maintenance (ie shrinking) of databases freed up a few Gigabytes of space. CU7 for the GPS has been successfully run for 05A, unsuccessfully run for 05B&06A (due to a bug that caused it to fail to deal with a temporary network glitch - this is being fixed now) and so is now rerunning for 05B - it is about half way through at present. The plan is to run CU7 one final time over the weekend to finish source merging for the GPS ready for DR2 release (phase 2). CU16 and CU19 still need to be run after CU7 to prepare the GPS for release and some of the code is new for these CUs (due to bugs having been fixed!). Hardware: JB noted: "Ahmose's hardware replacements from last Friday seem to have gone well as we've not had any problems from that area since (though the background re-initialisation slowed performance of Drive H for three or four days). Mark and I finished installing the new JBOD for Ahmose Thursday which Mark started Wednesday, this necessitated moving the KVM two racks along and thus also rerouting a number of cables. While doing this we also cleaned up the cabling a bit and things look a lot neater now (although we also developed a dislike for one of the types of rack we have - silly mounting design) The NAS hardware is up and running and has had its network configured satisfactoraly, it just remains to setup and format the disk arrays and that that (we also have the latest version of the NAS software now, courtesy of Eclipse)." Software: NCH asked the team to review the plan of work for the final month of the quarted as a check on priorities. ETWS will finalise CU5 (difference imaging) now that NCH has had a thought about how to define difference image pairs; RSC will continue work on CU7 for single-passband lists and synoptic surveys once some higher priority refactoring is out of the way. ACTION: ETWS to finish of CU5 by driving it from the relevant merge log for difference image pairs. ETWS reported: "Created metadata document of BestDR5 for Astrogrid. Investigated empty tables in BestDR3 and BestDR5: all tables are left intentionally empty, either to be used with a later release or filled up during general usage of the database." NCH and RSC finished work on refactoring CU19 to make it much less disk-hungry and scalable for the GPS, and also for VISTA. Testing is now in progress. JB noted: "Submitted a new bug in the new Trac system having found tracked down a potential couple of problems with CU7. Also tracked and reported a few other bugs and discussed some future features of the transfer and ingest software (with ETWS being the one doing most of the actual work!)" MAR reported: "Worked on refactoring the core of the Java classes used in the SQL queries to make them less WSA specific and more configurable looking towards VISTA etc. Looked at configuring the crossID service to allow users to specify a minimum distance as well as a maximum distance. Could implement via yet another option on the form." NJC reported some preliminary work on CU6 (synoptic survey source table generation) for Vista. Survey Data Release: Nothing new to report this week. Non-survey Data Release: Nothing to report this week. Astrogrid deployment: MSH noted: "The secure DSA for UKIDSS DR2 is now online and published to the main AstroGrid registry. We've tested it and the security is working as expected, and it is accessible by the same community accounts as DR1. The overall consensus I recieved was that it would be preferable to keep both DR1 and DR2 DSAs available. In the future I will remove the DSAs for previous releases when they are no longer accessible from WFAU's website. On a side note, GLIMPSE is now also available through a DSA here in Edinburgh." MSH and RGM are reviewing the WFAU TWiki page detailing our data holdings for the current DCA census. MSH will ask for info as required. Miscellaneous: NJC has been working on some WFCAM/UKIDSS publicity materials for SJW and AA. AL asked NCH to relay to the team that his demonstration of the WSA interface at the LSST Potsdam meeting impressed lots of US and ESO folks. Finally, NCH called early doors for this evening.