From nch@roe.ac.uk Fri Apr 20 16:52:01 2007 Date: Fri, 20 Apr 2007 15:44:36 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 20/4/07 Minutes of WFAU VDFS Science Archive meeting: 20th April 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, RSC, LGR, NJC, MSH, MAR, JB, ETWS Apologies: JMS, JDT, BCW, JPE, RGM, AL DONM: 10am, Friday 27th April 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to rearrange external catalogues onto batch catalogue server Discharged as completed as far as possible (until SSA rebuilt on batch server, BestDR2 and 2MASS must stay on normal public catalogue server). ACTION: NCH & PMW to meet Thursday 2pm next week to flesh out the Q2 plan of work. Discharged; see Project Management below. ACTION: MSH to chase up JNTD in IT Support to ensure the UKLight hardware is installed asap. Discharged ACTION: NCH to email CASU concerning transfer/ingest of 06B data. Discharged ACTION: NCH to make contact with the new UKIRT/WFCAM Campaign PIs Discharged; of the five PIs, Pinfield (Transit) and Mouhcine (Coma survey) are interested in VDFS Science Archiving while Smail (HiZELS), Kerins (GABARIT) and Tanvir (GRBs) are unlikely to request any UKIDSS-like curation in the WSA. ACTION: NCH to send MSH an interesting cone-search and SQL query for the AG DSA web page documentation. Discharged; high-z QSO and Orion brown dwarf CMD given as "interesting" examples. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues ACTION: JB to set up the NAS as soon as ITSG have provided an IP address Continues; final RAID initialisations being done now (see below). ACTION: NCH to polish up CU19 for a few minor issues arising in the DR2 GPS run. Continues; mostly done, but needs finalising when RSC returns 'cos NCH is too thick to understand the Python OO framework. Actions carried forward from 13/04/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES ACTION: ETWS to email SJW as soon as some 06B data are available for perusal in the WSA. - CONTINUES; some (fixed) 06B data now in, but archive offline from 4pm today. ACTION: RSC to add new keywords into Multiframe, and check for any others. - CONTINUES ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: The team examined the Q2 plan, and with a bit of fettling accepted it as a good working plan. WFCAM & VISTA updates: NCH noted that AA from JAC has been in touch over backup projects which may start to push out data later in 07B with data flow ID strings "U/UKIDSS/B1" and 2. It is envisaged that flat-file data access will be more than enough for those data. JB suggested these could be set up as non-surveys; in any case, we'll deal with this when the data start to arrive. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 20/4/07 pm. Networking: JB and MSH noted the latest UKLight developments. It now seems that there may be a problem with the KB end of the KB-ROE link, or with one fibre within the link, since not enough photons are making it up the hill. Investigations are ongoing; in the meantime, spades at the ready... Despite this, thanks to PSB who has pushed things forward as far as possible by testing out the KB to CASU link, and it seems to be working. WSA Operations: ETWS reported: "Ingested first batch of 06B data (20-23/12/06). Started download of some more directories and re-transferred reprocessed 24/12/06 data. Postponed more transfers due to power cut this weekend." ETWS noted that single-thread network transfer rates are running at 2 to 3 MByte/s. NCH reported that a full backup of the WSA ingest DB was successfully completed last weekend. Hardware: JB reminded everyone that IT infrastructure is all due to shut down from 4pm today to enable electrical safety testing. All operations and archive access will therefore cease, probably until Monday when everything should be back up and running. PMW noted that the new LTO-3 drive is ready for installation in the tape library; JB also noted the new catalogue server is here. Some discussions ensued concerning DBMS and OS installations on this machine; likely that we'll try out SQL 2005, but stick to W2K3 rather than attempting to run Windows-Vista. JB and MSH noted the continuing saga over the 20TB NAS-box; several problems have occured resulting in several RAID rebuilds. In the meantime, ETWS has set up ~0.5TB transfer storage space on /disk09 to keep things ticking over. Erratum: in last weeks minutes, SCSI-attached SATA is technobabble; it should have read serial-attached SCSI. Software: NJC reported: "This week I have started the variability analysis software for CU6, updated the WSA with new schemas for synoptic data tables. I have also changed CU14 to recalibrate deep and intermediate stacks on a detector basis, rather than the whole frame. I have continued testing and bug fixing CU6 and examined the magnitude RMS plots." MAR reported: "To try and speed up user queries added several more spatial inidices to various source tables (eg l and b on gpsSource). Had problems initially as default fill factor on amenhotep had somehow got over-written. Browser has been updated to refect the changes and put back in some of the functions lost whene we re-factored some of the SQL scripts (thanks Eckhard). Did more work on re-factoring main interface query code to make it easier to configure and roll-out for VSA (and other archives) and work with queue system." ETWS reported: "Updated the browser web pages with missing functions and indices. Started installation of software upgrade on sneferu. If all goes as planned, I will update the other machines during the following weeks, so be aware that there will be some disruptions." NCH reported applying the latest schema changes to the WSA, and generally synchronising the WSA ingest DB schema with that in the CVS. RSC is whooping it up in Cyprus this week. Survey Data Release: Nothing new to report this week. Non-survey Data Release: Nothing new to report this week. Astrogrid deployment: Nothing to report. Miscellaneous: NCH noted that ADASS XVII is being held in London on 23-26 Sept, and encouraged all to start thinking about attending and what they may present. At this stage, JB, NJC, NCH and (presumably) RSC have all expressed interest. Finally, seeing as the LAN will be freezing up from 4pm, NCH called very early early-doors for later this afternoon.