From nch@roe.ac.uk Tue Apr 3 14:14:49 2007 Date: Mon, 2 Apr 2007 11:10:00 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 30/3/07 Minutes of WFAU VDFS Science Archive meeting: 30th March 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, ETWS, RSC, LGR, NJC, JB, MSH, AL Apologies: JMS, JDT, BCW, JPE, MAR DONM: 10am, Friday 13th April 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: RSC to profile CU4 and investigate possible optimisation Discharged; declared done by NCH as he and RSC have added some bulk load optimisations. ACTION: PMW to get a quote for a new load server (hatshepsut?!) Discharged; quotes and options have been examined by PMW, JB and MSH and an order has been placed for a fancy new machine that should have plenty of umph for Vista. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: ACTION: JB to rearrange external catalogues onto batch catalogue server Continues; BestDR2 now ready to move over - just 2MASS to go. o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues ACTION: JB to set up the NAS as soon as ITSG have provided an IP address Continues; final RAID initialisations being done now (see below). Actions carried forward from 16/03/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES ACTION: ETWS to email SJW as soon as some 06B data are available for perusal in the WSA. - CONTINUES; AL noted that SJW is up in Edinburgh for the AAO Board meeting next week, so maybe time for a face-to-face pow-pow then. Specific points and new actions: -------------------------------- Project management: PMW has liaised with VDUC chair and the next VDUC will be held in Edinburgh on May 10th. NCH noted that SJW has floated the idea of a UKIDSS ESO-style workshop, date TBC (possibly in the Autumn?). AL noted that SJW is up in Edinburgh next week, so bums on seats please and best behaviour... WFCAM & VISTA updates: Some correspondence with JAC folks over header keywords: RQ_MINSB= '15.0 ' / [mag/arcsec**2] requested min J skybrightness RQ_MAXSB= '100.0 ' / [mag/arcsec**2] requested max J skybrightness RQ_MNSEE= '0.00 ' / [arcsec] requested min seeing RQ_MXSEE= '1.1 ' / [arcsec] requested max seeing RQ_MINCL= '0 ' / [percentage] requested min cloud coverage RQ_MAXCL= '0 ' / [percentage] requested max cloud coverage RQ_MNTAU= '0 ' / requested min tau constraint RQ_MXTAU= '2.00 ' / requested max tau constraint RQ_MINMN= '0 ' / [percentage] req. min illumination moon RQ_MAXMN= '100 ' / [percentage] req. max illumination moon ... these need to be added into Multiframe. ACTION: NCH to remember to action somebody to do this at the next meeting; perhaps write a schema-header comparison tool? Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB reported: "A UKLight connection is now up and running between Edinburgh and Cambridge, it appears that somewhere along the line the need for a connector at this end of the connection got dropped, however I have managed to borrow with a view to buying said equipment from Sam Wilson (from, the newly christened, Information Services). IT Support (Jonathan Dearden) will install this ASAP at this end so that we should then have a live connection from WFAU to CASU." ETWS noted receipt of a new tarball of CASU toolkit codes which have been incorporated into the archive end. NCH noted the usual gripes from RGMcM in Cambridge concerning lack of immediate availability of transfered data in the WSA. There's no pleasing some people ... WSA Operations: JB reported: "CU7 for the GPS completed early last week, CU16 was then run for it successfully. The rerun of CU7 for the LAS went ahead and completed early enough in the week to also run CU16 before weeks end. Having also ingested (CU3) the difference images from CU5 for the GPS this meant we were able to start running CU19 (creating database release product) over the weekend. Support requests continued to be answered and the regular backups continued as normal." Hardware: JB noted: "Progress on the NAS configuration is continuing with the RAID disks currently being configured, early next week should see them formatted if all goes well. A disk on Ahmose failed Friday but appears to have come back cleanly. The specification for the new server that will take over from Ahmose as primary load server has been decided and PMW is now ordering it. MSH and I went over what is currently happening with the hardware aspects of the system and I briefed him on the NAS and UKLight setups." Software: RSC reported: "Mostly helping with the refactor effort to make CU19 more efficient, reliable and less disk space hungry (see trac ticket 35 & 25). Further schema/database API development to enable automatic creation of database functions/views for CU19. Made CU19 more schema driven to ensure it releases the same tables as are listed in the schema browser, no more, no less. This lead to the creation of a new schema script, WSA_calSchema.sql to contain just calibration programme tables that formally lived in WSA_CalibSchema.sql (as well some more tables from this script being moved to more suitable schema scripts). Final testing and bug fixing to ensure released database is consistent with the old CU19 - updated the SoftwareTesting procedure on the TWiki for CU19. Started work on fixing the non-survey release functionality of CU19 to update it to the latest database schema by making the code schema driven. Deprecated CU18, and replaced with the helper script AddIndices.py, as CU19 creates all the required indices in release databases now. Also, upgraded exmeta to handle duff WCS values by filling the CurrentAstrometry table with a default row for that detector (see trac ticket 39). Created the fitsUtils.prepHeaders() function based on former contents of cu13/tidyUpStackMosaics.py, so that CU5 could use the same code as CU13 uses to prepare the provenance FITS header keywords correctly for use by ProvenanceFiller. Also helped Eckhard upgrade our local copy CASU's toolkit software." ETWS reported: "Got a new set of CASU programmes and commited them to the CVS. Ran CU5 successfully after Mike Irwin pointed me in the right direction to find associated H2 and K images for subtraction. 344 difference images and their jpgs were made and successfully ingested. MAR, JB and I tried to pre-release some new non-surveys and fixed some inconsistencies in the database. Otherwise general work for the release of DR2 part 2 went on as usual." JB noted: "Discussions on CU19 and schema details took place and a best course of action plan agreed, this entailed various changes, least of which were a few SQL modifications and fixes I made (with the majority of the leg work conducted by ETWS and RSC). ETWS and I also fixed a number of bugs that came to light when trying to fix CU21 (Non-Surveys)." NCH has been away for GAIA and hols in Italy; MAR is currently on hols (also in Italy, as it happens...) Survey Data Release: We seem to be on course to resurrect the GPS in DR2 by Easter ... Non-survey Data Release: JB noted: "CU21 was run allowing a number (six or seven) newer Non-surveys to have archive listing access to their data, the catalogues for these Non-surveys were also ingested however there are still a couple of bugs in the release code (CU19) for Non-surveys which will delay any releases until at least after my holidays." Astrogrid deployment: MSH noted: "AstroGrid is working on developing an implementation of the VO Space IVOA spec and upgrading the DSA component to make a number of important features available. These include "user databases" which will be user owned and operated, though they will be housed on the DBMS servers of data centers (like WFAU). Discussion is ongoing to determine how best to implement this." Miscellaneous: Nothing else this week.