From nch@roe.ac.uk Mon Jan 31 09:13:00 2011 Date: Fri, 28 Jan 2011 16:58:26 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive project meeting mins, 28/01/11 Minutes of WFAU VDFS Science Archive meeting: January 28th 2011 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, ETWS, MAR, NJC, RPB Apologies: JPE, AL, RGM, KTN, MSH DoNM: Friday February 11th 2011 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH to contact CDS concerning copies of a few bulk catalogue datasets requested to be joined in the VSA Discharged; emails received just after meeting and DENIS copying will start imminently. NJC noted that the VMCers had specified some OGLE cats for inclusion as well. ACTION: RSC (with input from MAR and NJC) to prototype a generalised catalogue outgester for ESO-SAF deliverables. Discharged ACTION: NCH to bug RGM/KTN about the status of the hardware (incl. LTO4) bids put into the last IfA round. Discharged; not sure what's going on, but RPB noted that he may be able to weave some magic and get the LTO4 library system talking to the linux mass storage servers so a standalone LTO4 may not be needed afterall. Actions partly discharged but continuing: ----------------------------------------- ACTION: NCH to ask RPB to look into the perennial functions permissions problem. Discussed during this meeting; DRs 6 & 7 done while DRs4 and before pending; RPB will check on the necessary T-SQL scripts/users and then communicate with RSC to insert into CU19 (ACTION RSC) ACTION: RPB to check that VHS, VIKING and VIDEO have required neighbours specified for UKIDSS-LAS and VVV for UKIDSS-GPS, VIDEOxUKIDSSUDS Continues; it was noted during the meeting that the specifications for the general external catalogue crossmatching for VMC would need updating once the latest datasets are incorporated into the archive (see above). ACTION: RSC to copy DR3 constituent reprocessed UDS frames into WFCAMOPENTIME; MAR to fettle UI and document as necessary. Frames are now copied in and UI fettled to make them available to the world; just need a few choice words on the web pages to make it clear how to get what from where. Actions carried forward from 14/01/11 meeting --------------------------------------------- ACTION: ETWS/RPB to update WSA/VSA schemas for new keyword NICOMB Continues; also watch out for TOTEXP ... suggestion was to auto check a couple of the latest examples of both WFCAM and VIRCam MEFs. Decision made to wait for next wholesale reprocessing (version 1.1? see below). Specific points and new actions: -------------------------------- Project management: Further negotiations with ESO this week, but no major progress to report ... WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: Correction/clarification to last weeks minutes from MJI: it turns out all the current products we have are v1.0 including tiles, while "As for v1.1 it only (sic!) involves: re-running the classifier on all catalogues; re-running the photometry on all catalogues; generating monthly detector zero-point updates (and illum corrs); then doing a bunch of FITS header updates; and finally running the grouting code on the tile catalogues i.e. not wholesale reprocessing." Apologises for any confusion caused by poor choice of words. Networking: MAR has organised an extraordinary meeting for this afternoon to review the state of readiness for transfers into ESO-SAF given the prototype catalogue outgest tool developments (3pm in the WFAU Technology Centre, i.e. MAR/MSH/RSC's office). Otherwise, all is fine on the networking front. WSA/VSA Operations: ETWS noted -- Written code to re-assign the data for DDT programme 284.C-5034(A) that has a wrong ESO PROG ID in the relevant tables. -- Finished the installation of third party software on thoth. The mxODBC license has been moved from djoser to thoth, so djoser is not anymore able to connect to the databases. -- Updated the external catalogues' schemas and code when problems appeared during ingest by RPB. -- The WFCAM November data has been transfered and ingested. RPB noted that ingests for VVV P85 are almost complete in it's new separate DB. Hardware and Systems: RPB noted that khufu is back to normal after a short period of ill-health; also that all associated RAID arrays are OK. However, a cluster node suffered a degraded RAID array and needed a disk swap earlier in the week. NCH asked what monitoring was in place, and it was agreed that automated monitoring should be put in place for all RAID arrays rather than relying on picking up on degraded arrays at login (infrequent) or via the audible alarms (inaudible over the aircon racket). ACTION: RPB to look into automated RAID monitoring across all servers. ETWS has fully installed (new) thoth, and this now needs a few SW tests. RSC volunteered to do this, noting that the same has already been done on userkaf with new PyFITS 2.4 Software: RSC noted: "I've completed the implementation of parallel deep tile product creation in CU13, allowing us to produce 4 tile products (each consisting of a filtered/unfiltered tile pair) at a time on userkaf, and 2 at a time on shepseskaf. However, catalogue production for these tiles remains in series due to a tight coupling with database queries. Further work was required to add deep tile support in the provenance table and detection quality flagging. Once NJC is satisfied with the results of the software tests I'll create a new release_20 branch of the software to allow operations to commence the VISTA deep tile curation. A prototype script now exists for performing the database outgests to FITS files and further work is required to ensure it meets the ESO requirements for their archive. The latest systems software on the new curation servers, userkaf & thoth, has all been verified now for operations use including the upgrade to PyFITS 2.4, which contains the official fix (read workaround) to the memory leak problem that I reported last year." As well as rolling out the new tiling/grouting regime, NJC has been working on tile/pawprint curation and has also realised that out total exposure calculations need to be revised for inclusion of DIT/NDITs in VISTA. Survey Data Release: MAR noted that VMC QC is up to date; RPB noted that VIKING/VMC curation with the new deep tile functionality should commence early next week. Non-survey Data Release: RPB/MAR noted new releases of wserv2 and an 05A (!) Japanese PI non-survey WFCAM projects have been made this week; RSC noted that all 09B non-survey data will go world-public next week. ETWS has been dealing with the small number of VISTA non-surveys, aka "DDT" projects. Astrogrid deployment & Data Analysis services: No news this week. Miscellaneous: That's all folks. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. ============================================================= -- Scanned by iCritical.