From nch@roe.ac.uk Fri Jul 9 20:27:24 2010 Date: Fri, 9 Jul 2010 17:33:08 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Jim Emerson , Keith Noddle , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren , Tom Kerr Subject: WFAU VDFS Science Archive weekly project meet mins, 9/7/10 Minutes of WFAU VDFS Science Archive meeting: July 9th 2010 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: ETWS, RPB, NJC, MAR, MSH, NCH, RSC Apologies: JPE, KTN, AL, RGM DoNM: Friday July 16th in the Vista Hut Actions discharged this week: ----------------------------- ACTION: ETWS to try some test ingests of VIRCam tiles. Discharged ACTION: ETWS to enhance the early CUs to cope better with CASU's process versioning Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RSC & NJC to document new operational procedures on the TWiki Continues (nearly done - NJC needs to add some new UDS/SExtractor things) ACTION on NJC to test SExtractor on shepseskaf and finish off CU13 upgrades Continues; ETWS noted this should be tested on menkaure also. ACTION: ETWS (with help from NJC where necessary) to implement aper mag saturation correction in CU4/exnumeric ETWS and NCH got a little confused over the details in MJI's f77 snippet, so NJC is being consulted. ACTION: ETWS to check that distortion correction is switched OFF for VIRCam tiles in TAN projection in CU4 Discharged but needs testing when the v1.0 tiles arrive. Actions carried forward from 02/07/10 meeting --------------------------------------------- ACTION: TBD to draft and MOU on behalf of PSPIs for ESO-SAF deliveries Continues; reply received from F.Comeron and discussions on-going amongst PSPIs and ESO concerning the exact requirements. It was noted from the PSPI teleconf (see below) that the likely deadline for submission of P85 products is March 2011. ACTION NJC to suggest workshop at next PSPI meeting. ACTION: ETWS to enhance the schema browser parser to include a distinct "VSA QC" entry that excludes objects not accessible to any, but includes views (e.g. vmcMultiframe etc.) and details of the partitioned detection table schema as employed in the ingest DB. Continues; needs a hack of the existing DB-driven approach for the special case of the VSA ingest DB. ACTION: NJC to check that distortion correction is switched OFF for VIRCam tile in TAN projection in CU13 Specific points and new actions: -------------------------------- Project management: NJC and MAR reported back from the PSPIs teleconf earlier this week; specific fall-out is discussed below but the general feeling seems to be one of contentment tempered by constructive criticism. WFCAM & VISTA updates: No updates this week. Comments and issues arising from CASU minutes: No new minutes this week. Networking: WFCAM 10A May transfers running away in the background; VIRCam v1.0 transfers await the green light from CASU. WSA/VSA Operations: RPB reported: "Non-survey cygOB7 finally released, though the decision was made to rename it W/SERV/1 to start a (hopefully) consistant naming scheme for such surveys. Other 08A/09A non surveys were tried, but got clobbered by some of the changes made to CU7 to help W/SERV/1. NJC is in the process of making everything play nicely together. Transferred WFCAM data for May 2010 is currently being processed and ingested. Should be finished early next week. VSA_PS_Views.sql script updated to include new views of individual surveys for QC access, so that the schema browser parser can automatically find the correct info." ETWS noted - Updated the browser pages parsing code for the new WSERV non-surveys. - Rewritten the monitor pages code to work with non-integer version numbers. - Started rewriting the browser pages parsing code to create VSA QC pages. Hardware and Systems: MSH noted that menkaure is a little unwell at the moment, showing lots of memory errors. Otherwise all is well on the hardware front. Software: NJC attended a VISTA PSPI telecon along with MAR discussing the politics of transferring data to ESO, adding corrections and uncorrected aperture magnitudes to the detection tables, amongst other issues (see more below). NJC also worked on adding in a new type of deprecation code to the software, made progress on speeding up matching of epochs to sources and updating the project string to deep stacks to allow WFAU to create new projects from multiple smaller projects. RSC reported: "Lots of PyFITS testing and fruitful discussions with the STSci developers about performance degradations in the 2.x series of PyFITS releases. Future releases should now perform as well as the 1.x series for large numbers of compressed FITS files. Documentation for the curation of joined cross-programme non-surveys should now be complete in the NonSurveyCuration TWiki page." The team discussed the fall-out from the PSPIs teleconf earlier in the week, and the possible SW mods required: o renaming aperMag1-7 to aperMagCorr (or somesuch) in *Detection The beef here is that the VSA (by inheritence from the WSA) general catalogue schema for detections denotes *corrected* aperture mags 1-7 as aperMag1-7, and this may be confusing for users in that the attribute name does not indicate that aperture corrections are applied to them by default. Logically, users might expect to see aperMag1-7 as the uncorrected ones, with additional attributes, say aperMagCorr1-7, as the corrected values. Queue operation "stable door"... The VDFS archive team have discussed the various pros and cons of the proposal (from VHS PSPI RGMcM). Pro1: It clarifies any confusion, primarily for users interested in galaxy photometry we believe. Pro2: If anything is to be done, now is the time to do it, before first release of any static catalogue DBs from the VSA. Con1: it is a "breaking change" for users: existing SQL may not work at all and will need to be changed. Con2: it will create confusion for users if aperMag1-7 change from being corrected to being uncorrected - existing SQL may still work, but will return unexpected results. Possible mitigation: remove aperMag1-7 attribute names altogether and substitute with (for example) aperMagCorr1-7 and aperMagNoCorr1-7 (but Con1 still holds). Con3: what to do for aperMag8-13, which are not aperture corrected - rename (Con1)? Leave alone (inconsistency with smaller aperture attribute names)? Con4: what to propagate into source? Both corrected and uncorrected mags/colours? Rename atributes there too (Cons 1 & 2)? Con5: it is a "breaking change" for curation application codes at WFAU (despite best efforts to make everything schema-driven and immune to such requirements-creep) and hence will divert resources from other, potentially more important, required changes. Generally, it should be noted that the naming scheme must be common to all surveys, so any changes applied for one survey have to apply across all, hence all PSPIs must be consulted and to agree on the required modifications. The risk is in satisfying some (smaller?) subset of users, we may generate a large amount of consternation amongst some (larger?) subset. NCH suggested that one possible compromise might be to materialise uncorrected aperture mags in the detection schema (aperMagNoCorr1-7?) where required (VHS, and possibly VIKING and VIDEO as well) and otherwise leave everything alone. At least this way there are no breaking changes, and the fact that a clearly uncorrected apermag attribute name appears next to each existing apermag attribute helps to reinforce (by implication at least) that the latter are corrected... Anyway, it was felt that the PSPIs should be consulted further before making the final decision ACTION: NCH to send a potted version of the apermag conflab to all PSPIs to solicit opinions and try to reach agreement. o new detection attributes for low-level photometric corrections depending on source position/magnitude This was another request from RGMcM, mainly from QC/sanity checking usage we believe. ACTION: ETWS to add in to the VSA template schema new individual deltaMag attributes for illumination, distortion and saturation corrections. o flagging of objects near to very bright stars VIKING PSPI WJS asked if this could be done, noting that he's had a student working on possible algorithms. RSC noted that this is easy given a well-defined algorithm, and can be included as new ppErrBit(s). ACTION: NJC to contact WJS concerning the latter's bright star proximity studies in VIRCam images (cc RSC) MAR noted that the UI row limit is now 10x bigger following recent changes ACTION: MAR to update the data access documentation to reflect larger download limits. NJC and NCH noted that the wholesale list-driven co-located photometry curation process (CU9) and associated *Remeasurement schemas should all be reviewed/revised in the light of recent dicsussions and optimised for use by the galaxy photometry fraternity. Survey Data Release: DR8 preparations are on hold, awaiting the eyeball QC deprecations for the LAS. NCH noted that it looks like the latest UDS deep stacks are nearly ready in Nottingham, and since they incorporate data up to and including 09B, it makes sense to include this with the LAS/GCS/DXS DR8 release (in late July or early August). ETWS noted that the reprocessed J from 05A through to 09A has not been ingested yet, but can be done soon and relatively quickly. ACTION: ETWS to CU3/4/ingest reprocessed UDS J data 05A-09A in preparation for inclusion of UDS in DR8. Non-survey Data Release: The first combined non-survey programme incorporating several existing programmes was released earlier this week. RPB also reported that some remaining 09A non-surveys have been dealt with this week. Astrogrid deployment & Data Analysis services: Nothing to report this week. Miscellaneous: NCH's suggestion (see apermag discussion above) of renaming certain attributes to crAperMags) was rejected on the grounds of decency and good taste. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================