From nch@roe.ac.uk Mon Jan 26 18:23:56 2009 Date: Fri, 23 Jan 2009 16:00:35 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 23/01/09 Minutes of WFAU VDFS Science Archive meeting: 23rd January 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, RPB, NJC, RSC, PMW, MSH, LGR, RGM Apologies: JPE, JMS, AL, MAR (haste ye back!) DoNM: 10am, Friday January 30th 2009 in the Vista Hut Actions discharged this week: ----------------------------- ACTION: NCH & PMW to meet Monday 12pm for VDFS planning Discharged; see Project management below ACTION: NCH to ring MJI pronto to wish him a Happy New Year Discharged; see WSA operations below ACTION: MSH, ETWS, RPB, PMW (+NCH?) to meet on Wed at 2pm to discuss the plan of attack for server relocation. Discharged; see Hardware below Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; some worries about truncated descriptions owing to long ESO hierarchical FITS keywords, and difficulties in finding online documentation... mind you, some books are lies frae end to end... ACTION: NCH to assign U/08B/14 H2/K frames to the GPS in the WSA. Continues; done for V1 08B but will need to be redone after V2 ingest. Actions carried forward from 16/01/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: The team discussed the plan for Q1'09 and with small adjustments ratified as sensible/doable. Highest priority is being given to outstanding items for VSA operations. Nae man can tether time or tide. PMW has been indefatigable in his dealings with the research council in pushing forward the continuing saga of the grant renewal, but still no signs of official announcements... If naebody care for us, we'll care for naebody. WFCAM & VISTA updates: Nothing new this week. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: ETWS/MSH noted that an alert has gone around concerning JANET/UKERNA network upgrades in March; it is not know if this will affect UKLight but MSH volunteered to have a closer look and in any case, the operations side will keep an eye on things. WSA Operations: The main news this week is that following last week's identification by MAR of photometric inconsistencies in the DR5 LAS merged source list, it seems that the wee sleekit cowrin' timorous beasties have gotten into the machinery: a rather nasty bug has been identified in the standard nightly pipeline catalogue processing. This means that all single passband catalogues observed from March 12th 08A onwards need to be rederived, and their photometry recalculated. The main results for users are that DR5 release will be delayed (by 6 weeks or so) and that any use of flat-file products (both survey and non-survey) from the affected dates will be subject to revision. Notes have been logged on the WSA website and emails distributed appropriately. In addition to the above wholesale reprocessing, NCH and MJI discussed the other odds and sods of reprocessing that CASU have been requested to do: this is being done in temporary work space rather than the working directories, and is well underway and awaiting feedback from users in some cases. It was agreed that CASU will let WFAU know as and when these data are ready for retransfer. Regarding the other large-scale fix to deblended flux assignments and algorithm, this is on hold until all the odds and sods have been tidied up and the fix and software update can be done on an otherwise clean and stable dataset. Otherwise, NCH noted that a small bug in the source merging append mode (new to DR5) has been identified by MAR & NCH, corrected by RSC, with reruns done by RPB and again checked by NCH and all now appears to be well. This one is down to NCH's bad SQL from a while back ... ah well, a man's a man for a' that. ETWS noted: -- Finished the browser page parser to handle new non-surveys and created new web pages. -- Worked on the VSA frame schema. -- Estimated the time it would take to re-upload the reprocessed 08A (~7d) and 08B (~5d) data. Flat file ingest can be run in parallel, so shouldn't contribute to processing times. CU4 may need ~1 week for 08A and 08B. On this the higher CU's processing times need to be added (~1 week for 08A and 08B). ETWS and RPB suggested that it might be worth considering including 08B in DR5 now given the delays; apart from QC1 things are pretty slick on the curation front. ACTION: NCH to contact SJW to see if he's interested in 08B going into DR5... RPB noted: "NonSurvey releases: All shallow nonsurveys up to the end of 07B have now been released. Next step is to try releasing the deep surveys. There's a possibility that deep surveys may show up some problems, but we'll try it and see what breaks. Re-reseaming: GCS and LAS have both been source merged and reseamed again with the latest version of CU7. NCH reports the data is looking good. The soon-to-be-legendary Server Room Move of 2009 is scheduled for the weekend of the 31st. Volunteers welcome!" The issue of what to do with the ingested catalogue photometry from the V1 processing in 08A/B was raised by ETWS, and it was generally agreed that we may as well get rid of them. NCH suggested we think on it for a few days just in case... the best laid schemes o' mice and men gang aft a-gley. Hardware: As noted above, the big move into the new server room starts next week, and archive services will be disrupted over the last weekend of January. ACTION: RPB to put a note on the WSA downtime pages and get MAR to bounce the message to wsa-announce. Software: RSC reported: "Made updates to CU7 and CU19 to improve support for the append mode for source merging following the first test of this mode in a couple of years. Also, further minor improvements to the non-survey curation process. Resumed work on VSA design issues but haven't made much progress due to the above interruptions and work on contributions for the ADASS proceedings. Next week, I'll be focusing on testing our software on Khafre's 64-bit system with the latest Python and mxODBC." Survey Data Release: ... on hold until 08A/B data are fixed. Now a' is done that men can do, and a' is done in vain ... Non-survey Data Release: Shallow surveys up to and including 07B have been worked on and databases prepared without any problems using the new automated curation regime. MAR has been making small adjustments to the web interface and releases to PIs are now taking place. Astrogrid deployment & Data Analysis services: LGR reported that his list-driven tool now includes "luptitude" scale magnitudes with extensive online documentation. Final tweaks to be done over the next day or so, and NCH noted we should get the users who requested it to give it a go and feedback any comments. NCH also requested that all codes/docs etc. been worked on by LGR be placed under version control in the SVN. Miscellaneous: RGM noted that he will be assembling LSST data management schema info over the next week or so and organising a conflab over how to review and input comments/ideas to their development team. Perhaps it may turn out a sang, perhaps turn out a sermon. Finally, Burns Night on Sunday - lang may yer lum reek. PS. Tried to work this one in somewhere as well: "Gars auld claes look amaist as weel's the new" but I haven't got the faintest idea what it means... ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================