From nch@roe.ac.uk Fri Oct 7 13:33:40 2005 Date: Thu, 22 Sep 2005 14:35:01 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly project meeting: 22 September 2005 Minutes of WFCAM Science Archive meeting: 22nd September 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, JB Apologies: MCH, RGM, JPE, JMS, AL, NJC, MAR, JDT *NOTE* DONM: 10am, Wednesday 28th September 2005, plate library Actions discharged: ------------------- ACTION: MAR & RSC to assemble an advertisement-feature poster concerning the WSA for ADASS. Discharged: thanks to RSC for assembling an extremely snazzy poster. ACTION: JDT and MAR to continue sorting out AG access option functionality for saving WSA results to "MySpace". Discharged; functionality tested by MAR. ACTION: ETWS to email MR (mriello@ast.cam.ac.uk) and STH (sth@ast.cam.ac.uk), CCing to NCH and MJI, about the handshaking problems with reprocessed data. Discharged, a combination of emails from ETWS and NCH to MR/MJI and a face-to-face between MR and NCH at the UKIDSS SV meeting seems to have resolved the problems, and the CASU/WFAU interface is up and running again. ACTION: NCH submit a helpdesk to get khufu set up. Discharged; helpdesk ticket in, assigned to JBW and modified to specify liaison with ETWS/RSC regarding set-up (see Hardware below). ACTION: NCH to email PSB concerning UKLight. Discharged. Actions partly discharged but continuing: ----------------------------------------- None this week. Actions carried forward from 02/09/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH (prompted by PMW) welcomed new archive operator Johann Bryant (username JB) to the team. PMW noted that minutes and actions have been received from the last VDUC. Yet more feedback on the requirements doc has been suggested in the light of some changes. NCH reported back from yesterday's UKIDSS SV meeting; the priority for UKIDSS now is to get the reprocessed SV1+2 data released asap to try to close out science verification in time for an ESO-wide data release by the end of the year. Recognising that this timescale is likely to be tight, SJW asked when the next release may be. NCH threw caution to the wind and suggested 21st Oct, but this may be difficult given the large amount of reprocessing/retransferring/reingestion necessary (not to mention all the changed and bug fixes). The situation is further compounded by NCH's absence for the first 3 weeks of October (6 nights WFCAM then 2 weeks R&R). Apart from the inevitable pressure on the next release date, a few extras cropped up - see Software below. NCH noted his absence beginning (and including) Thurs 29th Oct and ending 23rd October - there will be a large amount of delegation of the next few days... WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: Minutes have just arrived... will discuss and minute next week. Networking: As noted above, the CASU/WFAU interface is now back up and running following communications with MR & MJI. SV1 data was readied for transfer as at 21st September; MR hopes that SV2 will be completely reprocessed by the end of next week (30th Sept) at the latest. After a bit of rearrangement at this end, transfers will begin in earnest. Hardware: NCH has delegated the set-up of new 8TB storage brick khufu to ETWS/RSC. It will be used as a test-bed for the latest Debian release and GNU compiler products. PMW suggested that he get a quote for a new JBOD for public catalogue server amenhotep, to get the ball rolling on upgrade of that system for larger DBs. ACTION: PMW to get quotes for a 16-bay U320 150GB disk enclosure with 2x 8-way backplanes Software: The following extra items arose out of discussions at yesterday's second UKIDSS SV meeting: 1) Batch upload of user-supplied lists to get catalogue data (as opposed to image thumbnails - now implemented thanks to MAR). ACTION: MAR 2) Documentation (probably a Q&A entry) on how to query the database to get dates of observations for sources ACTION: ? 3) A new mergedClassStat attribute that is analogous to the continuously- distributed classStat in the detection table made from some arithmetic combination of the individual class stats. ACTION: NCH 4) Liaise with UKIDSS DXS and UDS survey heads as to the efficacy of the standard,a rchive-driven default stacked/mosaiced products that are ingested for the community at large. ACTION: NJC (in the context of continuing work on CU13/14). 5) QC post-ingest: liaise with SV community on the details of any post-ingest QC control that flags data that are not to be propagated further into the database-driven science products. ACTION: ? (no clear plan was forthcoming from UKIDSS an how this might work). We should endeavour to at least have 1 & 3 done for the next release. RSC raised the issue of CVS change logs and the old fashioned way we are annotating changes in the software at the moment (i.e. comprehensive comments at the top of each file of source). RSC suggested that a better way to do this was to use the CVS log facilities exclusively, rather than duplicate the information in the source as well as the repository system. NCH gave a guarded go-ahead to this, provided that we continue with the old method for any files were there are comments in source (especially the SQL scripts since these are exported to the W2K3 side where there is currently no access to CVS). Any new source can use the CVS method if the developer so wishes, provided that log comments are not limited to one-liners from command-line commits. ETWS reported: "Enhanced the parsing script to write attributes in a single row into the glossary when they only differ in the table name. This makes the glossary much more concise and easier to read. On the other hand I've been revisiting CU1 to include versioning of downloaded files, since CASU will only keep latest versions on their server." RSC reported: "New Source Matching Code: * Impressive 5 hour benchmark for finding neighbours in the billion row SSA database. Speed improved as djoser is now running more than three times faster following it's reboot, and we're now using a more efficient filter for the source matching code (almost two times faster than dec-plane sweep). * Outgest for SSA data took approx 2 hours, ingest of 2.5 billion neighbours with minimal transaction logging took 1.5 hours. * Cross-catalogue matching is also now possible with the new code and a test finding neighbours to SSA sources from the 2MASS XSC database has proved to be reliable - finding the exact same set of neighbours and appears to be calculating the same separation distances between source pairs as the old code. * Now implementing schema driven features for the new code." NCH agreed that these results look fantastic, and will scale to 10 to 100 billion-row tables in a feasible way. Furthermore, comparative benchmarks need to be treated with caution, since the old method has only been run on a different IO subsystem (which was faster for disk writes than the current one) - hence the performance of the new code is even better than these numbers indicate. Finally NCH noted that the merged Source base schema now needs changing, and those changes propagated into every merged source table defined in the archive. NCH suggested he made the changes for lasSource, and then delegated the rest to somebody else (ETWS and/or NJC). Data Release: Pending; awaiting retransfer & reingest of SV1 & 2 data. Astrogrid deployment: Nothing new this week. Miscellaneous: Nothing else this week.