From nch@roe.ac.uk Fri Apr 27 14:33:35 2007 Date: Fri, 27 Apr 2007 14:28:54 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU Science Archive weekly project meeting minutes, 27 Apr 2007 Minutes of WFAU VDFS Science Archive meeting: 27th April 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, PMW, RSC, MAR, ETWS Apologies: JMS, JDT, BCW, JPE, RGM, AL, JB, LGR, NJC, MSH DONM: 10am, Friday 4th May 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: JB to set up the NAS as soon as ITSG have provided an IP address Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues ACTION: NCH to polish up CU19 for a few minor issues arising in the DR2 GPS run. Continues; mostly done, but needs finalising with a bit of help from RSC. Actions carried forward from 20/04/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES ACTION: ETWS to email SJW as soon as some 06B data are available for perusal in the WSA. - CONTINUES; ETWS will discharge this one this afternoon when the first tranche of 06B jpegs have been created and ingest DB updated. ACTION: RSC to add new keywords into Multiframe, and check for any others. - CONTINUES ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted that the next VDMT follows almost immediately after this meeting; VDUC is now going ahead 11am to 5pm on 10 May; and both he and MJI have been called on to update the next UKIRT Board meeting (Durham, 17th May) on WFCAM processing and archiving status. WFCAM & VISTA updates: A telecon is being arranged with the VVV PI; looks like 3 May at 4pm(ish). Comments and issues arising from CASU fortnightly minutes: No new minutes as of 27/4/07 am. Networking: JB noted: "The UKLight connection, though up, is currently not setup with ssh correctly, this is causing problems (specifically it seems to be a key problem). Numerous 'fixes' have yet to avail so work is ongoing." WSA Operations: ETWS reported transfer and ingest more 06B data and creation of jpegs for the same. JB noted "Transfer, Processing and Ingest have continued for the 06B data, it was hampered by transfer speed and down time over last weekend due to the power outage. Network issues with the ROE network/servers have also caused the odd hiccup. Thanks goes to Eckhard for sorting most of this." Hardware: JB reported: " The NAS box is now setup and has had it's disks mounted on our other work machines, performance on it is comparable with the other machines (which is good!) A disk was giving errors on Khufu, it has been rebuilt and will be monitored for further issues (if they occur then it will be replaced). Installing the new server (I can't spell it yet) was to be delayed until early next week, except that Eclipse are now waiting on screws from the rack manufacturer so it will be late next week at the earliest. The LTO3 drive for the tape library has been installed though, because it would require a reboot of Ahmose (which has been busy), it has yet to be used in anger." Software: MAR reported: "Investigated speeding up cone search queries as noticed GPS was getting slow. Looked into HTM indices/functions again and installed the HTM v2 extended procedure and a few v2 functions. Some improvement could be gained with using covering indices but this was lost as soon as different selections were made. Conclusion is that not much can be done to speed up these searches on the current system/schema as we are up against a very high density of sources in the GPS and a large row count and length. Continued re-factoring UI code to allow easier configuration for existing and future implementations." ETWS reported: "Successfully upgraded the software on sneferu, will continue on the other pixel servers starting with djoser this afternoon. Refactored CU1 to be included into the DataBuilder." RSC noted that before he left, he revised CU16 for the new schema design that allows us to create cross-match neighbour tables between survey tables, rather than just with external survey tables. Survey Data Release: NCH noted that some concerns have been raised by users and the operations side concerning the GPS. In fact, NCH has found that a fairly simple real-world example query in the GPS at DR2 is to slow for any practical purposes. Contact has been made with the UKIDSS Survey Head (CC CSS) to try to rationalise the rather bloated GPS schema in the light of experience and the results of the Board's 2yr review. Discussions are ongoing as to which merged source table attributes can be dropped without compromising science usage. In the meantime, MAR has added a few new useful 1d indexes, and NCH has added a covering index to make the GPS views, and the example query, go much faster. Non-survey Data Release: MAR noted that 4 new non-survey registrations have come in this week, two of which have data to serve out. Furthermore, a dozen 06A non-surveys await DB preparation (albeit only one PI has been in touch specifically asking for a prepared DB). ETWS suggested that some streamlining of the procedure should expedite this kind of thing; NCH suggested that ETWS and MAR look at the procedure of the one DB specifically requested with a view to better automation from the operations point of view. Astrogrid deployment: Nothing to report. Miscellaneous: Nothing else this week.