From nch@roe.ac.uk Fri Nov 25 17:11:56 2005 Date: Fri, 25 Nov 2005 12:46:31 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes: 25 November 2005 Minutes of WFCAM Science Archive meeting: 25th November 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, NJC, JB, JPE, JMS, JDT Apologies: MCH, RGM, AL, MAR (aka Michael Schumacher) DONM: 10am, Friday 2nd December 2005, plate library Actions discharged: ------------------- ACTION: NCH and PMW to meet on Mon 31st to assemble the Q4'05 and Q1'06 VDFS work plans ACTION: NCH & PMW to have a planning meeting next week, Tues AM. Discharged ACTION: JB to continue to investigate the USNO-B problem by restoring flat-file backups. Discharged; it seems that no institution has a bug-free catalogue - there seem to be problems at LEDAS etc. and worries that the original hard disk from Flagstaff was not error-free. The team decided that there are higher science priorities (e.g. GLIMPSE) for external catalogues, so the question of USNO-B is deferred until a reliable source of the catalogue files can be found. ACTION: NCH to schedule installation of the khufu rack UPS at the end of Nov. Discharged ACTION: NCH, MAR & ETWS to meet this afternoon to review the state of readiness for release of non-survey DB products. Discharged; but actioned again (see below) Actions partly discharged but continuing: ----------------------------------------- ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Continues; MAR, JDT & NCH met to discuss some implementation issues. ACTION: JB to look at the GLIMPSE info and begin SQL implementation with help from ETWS and NCH. Continues; JB has started looking into the data model. Actions carried forward from 18/11/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH reported that he and PMW had met to discuss further planning issues, especially in the light of the latest round of user community feedback following the IR workshop and UKIDSS Consortium Meeting. The resulting wish-list has been prioritised/allocated as follows: 1/ Add in the GLIMPSE external catalogue and create cross-neighbour joins with the GPS ACTION: JB and ETWS to implement SQL DB 2/ World release to take place 18 months after ESO release, and not 18 months after observation dates (and similarly for non-survey data as well, presumably, but with 12 month proprietary period) Discharged by NCH: Note that proprietary period clocks now start ticking at the point of release as opposed to the point of observation; hence, for example, UKIDSS data will be released to the world 18 months after the proprietary release; similarly non-survey PI data will be released to the world 12 months after release to the proprietors. Presumably anybody that has any mojar objections to this will contact NCH having read these notes. 3/ Link to CASU survey progress information from WSA pages ACTION: To be done asap as part of continual updates to web pages 4/ Quality control: flagging of residual cross-talk images, 128 pix either side of v. bright / saturated objects in single-passband catalogues 5/ Application of 2d photometric correction maps of systematic calibration errors (of order a few %) measured from all 2MASS standard residuals, averaged as a function of position on frames 6/ New data model required for poor-seeing time programme (multi-epoch photometry in search of planetary transists) All to be inserted into the Q1'06 plan and will be attended to then; also need to include tasks on UKLight and outside access to the continually updating database of ingested pipeline products. 7/ An http post/get SQL service that accepts free-form SQL and streams results sets back to the client Deferred until it becomes clearer what kind of client-side infrastructure will be provided by Astrogrid. NCH noted that the final UKIDSS Science Verification meeting is now scheduled for the 15th December, 10:30am to 4pm at ROE in the IfA lecture theatre. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: The meeting noted the minutes of 17/11; there were no comments Networking: ETWS and JB have discovered that certain of the new floating-point keys written in CASU pipeline processed image FITS files have lower case "e"s in the exponents. This is contrary to the FITS standard, is reported as an error by fverify, and causes the latest versions of the ingest codes to hiccup because they are now very strict about this kind of thing. Already transfered data will be fixed up at the archive end, but a request has been sent to CASU to fix this for future data. Initial reaction to this hopefully polite request was, er..., less than well received; JPE noted that this is also presumably an issue for raw FITS files transfered to ESO. JMS and JPE additionally volunteered to contact CASU regarding this interface issue... JB reported on the continuing saga of the corrupted USNO-B. It seems that LEDAS have corrupted files as well and that the source of some of the corruption may even go back to the original hard disk supplied by Flagstaff. Given all the uncertainties, NCH recommended that JB and ETWS concentrate on GLIMPSE since UKIDSS GPS have specifically requested this whereas we are unaware of any science usage of USNO-B from our archive. PMW asked at what point do we cut off requests for stitching in external catalogues; NCH recommended a cut-off at number of records being less than 10^7 or so as being sensible. WSA Operations: JB reported that backup tape cataloguing of all LTO-2 tapes was now complete so we have an inventory of all backed-up data from previous backup activities of all instances of Veritas Backup Exec (important for housekeeping and ease of restoring). Hardware: NCH reported that Eclipse would be coming in next week to install the latest UPS for khufu and the Astrogrid deployment servers; when the exact date is known, NCH asked JB to take over the logistics. On the subject of UPSes, PMW noted that he had relayed the WFAU requirements for C1 to premises for inclusion in their projections for the room for the next few years. Allowance for 3 KVA/yr has been made for archive hardware. PMW also noted that individual UPSeses may become redundant at some point in the future, since a single UPS system for C1 was being considered for mission critical servers by IT Support and premises. PMW has put a topic on all of this on the WFAU TWiki. RGM passed on an EUCS questionaire to NCH and PMW concerning provision of central computing resources over the next few years. Given the potential for SAN storage solution for VISTA pixels, projected requirements would be estimated and stitched into the WFAU response started by RGM. NCH and RGM noted that a combination of off-site mass pixel storage in externally managed hardware along with internal catalogue DBMS server provision may be a good working split in the future. ACTION: NCH and PMW to meet next week to insert WFAU requirements in the EUCS questionaire. RSC noted that: "Regarding hardware... Eckhard noticed that some changes I'd made to the C++ code cause CU3/4 to run incorrectly on djoser. They do run as expected on khufu, where of course the changes were tested. We've decided the simplest solution is to stop running CUs 3, 4 & 7 on djoser until it's upgraded. Maybe this provides us with an incentive to upgrade djoser sooner?" Software: MAR reported: "Carried on working on integrating non-Survey programmes with the user-interface (3 out of 4 forms now working). Also tweaks to the existing interface and documentation." ETWS reported: "I've worked on the non-survey programme ingestion tools, did a bit of restructuring of our disk access policy, and worked with Johann on CU3 and the USNOB code." RSC reported: "I've mostly been working on various improvements to the C++ code, and backend C++ development for the new CU16 script." NJC reported working on the following: "CU13 - Testing. Eliminated some bugs but still trying to sort out the different background values. CU14 - Emailed Jim Lewis - no reply Galaxy photometry. Finished writing Petrosian magnitude code. Tested it on standard profiles, removed all major bugs. Started testing the rest of the code, eliminated many bugs." NCH notes that he has debugged a CU7 error that was preventing the rerun functionality from operating correctly, and has also extensively refactored CU19 in the light of non-survey issues and proprietary periods. Much testing has been done on these two CUs. Survey Data Release: The SV 1+2 release continues to be used extensively: 866 logged queries originating outside ROE. Non-Survey Data Release: NCH remarked on the steady progress that has been made towards non-survey data release, and suggested the same 3 team members concerned with this meet again to do another iteration on the procedures. ACTION: NCH, MAR & ETWS to meet this afternoon to review the state of readiness for release of non-survey DB products. Astrogrid deployment: Nothing new this week. Miscellaneous: Finally, JPE noted some VISTA updates: focal-plane array cool-downs and camera assembly are progressing well. JMS also noted that the telescope is about to be disassembled in Texas in readiness for shipping to Chile.