From nch@roe.ac.uk Mon Sep 13 08:46:03 2010 Date: Fri, 10 Sep 2010 17:08:11 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive fortnightly meet mins, 10/09/10 Minutes of WFAU VDFS Science Archive meeting: September 10th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NJC, MAR, NCH, RSC, ETWS, RPB, MSH Apologies: JPE, AL, RGM, KTN DoNM: Friday September 24th in the Vista Hut Actions discharged this week: ----------------------------- ACTION: RPB to update WFCAMOPENTIME for 09A projects no longer inside their proprietary period. Discharged ACTION: MAR to add some web documentation for the GPS photometric calibration issues. Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to assign actions on the following next time: - Modify entire code base to work with Tiles (Partly discharged - but needs fully testing on the new V1 data). This includes checking code works here on deep stacks, checking quality control/ QBF, ingest, CU7, CU6. Also do we need to do better links between tile detections and pawprint detections or will Provenance etc be good enough. (RSC has added tile qbits; also suggested a preliminary test run of CU7 using shallow tiles first of all). - VVV SynopticSource (allowing both SourceXSynopticSourceBestMatch and SourceXDetectionBestMarch) NJC noted that he's working on this, with programme builder setup under test now, but some issues with CU13 (see below) Actions carried forward from 28/08/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; on hold until at least next PSPI teleconf, and possibly until "Phase 3" workshop at ESO. ACTION: NJC to suggest workshop at next PSPI meeting. NJC noted that he needs to liaise with JPE about the next teleconf. ACTION: NCH to remember to mention SSA crossneighbour production in WSA/VSA next week (it seems to be switched off?) Specific points and new actions: -------------------------------- Project management: Nothing new this week. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: The team noted the minutes of the meeting of 26th August. Regarding the need for tiling/filtering software at the archive end, NJC has been in touch with MJI/JRL to explain the reasons behind our request. Networking: All processed data transfers are up to date for WFCAM and VISTA. WSA/VSA Operations: ETWS noted -- I've implemented the changes to the C++ code to relax ingest of data with broken WCS info into CurrentAstrometry. All the values will have the maximal double value to make deprecation easier. This way uneffected detectors will still be ingested. -- Included checks to ensure the sequence number is linear and ascending to filter out broken VVV data. -- WFCAM data for 10A is now completely transferred, JPGs are created and metadata is ingested. -- All from CASU released VISTA data (up to July 23rd) is as well transferred, JPGs are created and metadata is ingested. Catalogues for all surveys but the VVV are ingested up to May 24th. Due to the fair amount of broken catalogues in the VVV and their sizes these ingests will take longer than expected. With regard to the last point, ETWS is assembling some stats and info on all the VVV broken catalogues and will be contacting CASU. NJC has been looking after Maria Ida Morretti, visiting from the VMC team for the last couple of days as she learned all about the SQL release DBs and how to use them. MAR noted that after much to-ing and fro-ing with the PI, VMC QC is complete up to the end of March so an interim release DB for the science team can go ahead full steam. Hardware and Systems: MSH noted that hardware procurements are going ahead for various new machines; first up is a new cluster node for VVV curation. RPB noted that in the meantime the venerable hatshepsut will be used as a proof-of-concepts platform to check that splitting off the VVV from the main VSA ingest DB causes no issues in the curation software and procedures. ETWS noted that SW upgrades on khafre have thrown up a small gotcha in that the DB interface middleware mxODBC is incompatible with the latest Python2.7 and our licenses are invalid for upgraded versions. We will continue with Python2.6.6 for the time being, and if it is necessary to move to 2.7 at some point in the future, NCH agreed to contact Egenix to plead poverty and scrounge updated licenses (you never know your luck...) Software: RSC noted: "A new SVN branch of the curation software has been made, release_18. Following some confusion over the status of existing branches it was unanimously agreed that we should revert to the original system of only performing operations (both WSA & VSA now) in the release branch and keeping the trunk for development of new features that are not yet required for operations. The key here is remembering when these new features need releasing, which will require better communication between operations & development. Any schema changes to either of the load databases (WSA & VSA) should always be reflected immediately in both branches. The recent curation runs for VSA releases has helped iron out any remaining bugs in the software for the handling of both tiles & paw-prints. I've updated the CuPerformance TWiki page to include a table for VSA releases, to help give us a better idea of how long a release will take to prepare and how well the software performance is scaling for each component." NJC has had a few problems with the latest pawprint stacking code version from CASU in that it bombs out if one or more extensions are everywhere zero confidence throughout the set being stacked (whereas the previous version seemed to handle this). Investigations are continuing... Survey Data Release: GPS DR7 has gone live this morning after copying on to the public DB servers. UKIDSS DR8 (LAS, GCS, DXS) was released last week (the UDS had to be deprecated and the catalogue tables emptied because of a low-level pattern noise problem identified by the Nottingham team; we anticipate reinstatement of the UDS by the end of September depending on delivery of a fettled mosaic). VHS interim release DB is nearly ready to be released to the PI; VMC is next in the queue. Non-survey Data Release: Once CU4 ingests of all 10A catalogues are complete, a tranche of survey-like DBs for 10A WFCAM projects will be prepared. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: ADASS abstracts from MSH/NJC are/willbe submitted (deadline is 15th Sept). NCH noted that PanSTARRS interest in optical/infrared crossmatching at the QUB science meeting last week has resulted in a plan to export non-proprietary SQL DBs into the Published Science Products System (PSPS) which is the Hopkins-designed PanSTARRS equivalent of the SDSS Catalogue Access Systems. RPB and NCH are looking after this, with UKIDSSEDRPLUS being used as a small tester DB to verify procedures. NCH noted that he will support one user who is asking for some bulk outgests of limited attribute sets in order to import into the PanSTARRS "Desktop VO" system for quick-and-dirty crossmatching. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================