From nch@roe.ac.uk Fri Apr 24 21:32:01 2009 Date: Fri, 24 Apr 2009 15:34:23 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Rob Blake , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Peredur Williams , Bob Mann , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 24/04/09 Minutes of WFAU VDFS Science Archive meeting: 24th April 2009 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, NJC, RPB, MSH, PMW, RSC, RGM, BCW Apologies: JPE, JMS, AL, MAR DoNM: 10am, Friday May 1st 2009 in the Vista Hut Actions discharged this week: ----------------------------- None this week. Actions partly discharged but continuing: ----------------------------------------- ACTION: ETWS to parse that latest keyword info into the VSA metadata schema files. Continues; ETWS noted that communications with CASU indicate that the VIRCam data FITS keys are yet to be finalised, but that the procedure for parsing a standard set into an SQL schema is set up and can be tested further in the meantime. Actions carried forward from 17/04/09 meeting: ---------------------------------------------- None this week Specific points and new actions: -------------------------------- Project management: The team examined and ratified the Q2'09 plan as sensible and doable. Now that The Budget is out, we eagerly anticipate a sudden flood of Rolling Grant monies from STFC. Ha ha ha ha ha ha and so forth. WFCAM & VISTA updates: JPE responded concerning the timing of VISTA Science Verification, (see WFAU minutes of 13/03/09 meeting) noting that it will take place after the formal hand-over of the telescope to ESO. JPE has also given us advanced notice of an RAS Specialist Discussion meeting on Science with VISTA, scheduled for 12th Nov 2010. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 16/04/09. A few issues were discussed: i) new version of the catalogue software: we're glad to accept a new version at any time; we note that we (i.e. VDFS) need to be a little careful over the version with respect to that used to process data for a given release in order that it is clear what changes have occured and for which data. ii) Regarding class=-9 for saturated, what WFAU do is simply to set ppErrBits bit 16 (="close to saturated") so this can be used to exclude images *possibly* affected by saturation by predicating selections on ppErrBits < 65536 etc. For further details, see http://surveys.roe.ac.uk/wsa/ppErrBits.html#Saturated_source_image The threshold criterion of peak height + sky > 40,000 ADU is somewhat arbitrary, hence the significance of the quality bit in the hierarchy, and the description as "possibly saturated". iii) New sky algorithmic keyword: fine, just let us know what it as and we'll propagate into the WSA DBs. iv) VISTA IOT developments: we trust that CASU and/or JPE will keep us informed of any developments that may impact things at the archive end so no need for us to see minutes from the telecons. Finally, and most importantly, regarding the deblending extended flux attribute fix: we really need more info on this asap. We need to apply the fix to the flat FITS file catalogues held at our end (so please send the code) and we need algorithmic details so that we can write the relevant correction procedure for the data already ingested into DBs. ACTION: NJC to contact CASU regarding the deblending fix. Networking: Transfers from CASU await release of 09A data. WSA Operations: See below under Hardware. Hardware: RPB reported: "New problems with Ahmose. The H: drive went missing, with hilarious consequences. Sometime after the problem developed, the power supply for the affected jbod failed. Replacing the power supply seems to have fixed the problems with the drives as well, so fingers crossed. The full backup that ran last weekend overlapped with the jbod problems so the quality of that backup is in question. As a consequence, a new full backup will be run this weekend. While Ahmose was being fixed, I took the opportunity to switch off hyperthreading to try and head off other problems at the pass. US Naval Observatory have sent two external hard drives for taking a copy of SSA data. The smaller 160Gb drive refuses to run at high speed for some reason. If the 2Tb drive behaves similarly, we may need to consider ordering an eSATA interface card." Software: RSC reported: "I've returned to working on the software design issues for VSA curation, having completed work on the new 64-bit systems software install with the release of the new release_10 software branch after helping to debug the problems with CU6 for DR5. Since then I've also updated the post-CU3 script to allow it be run once after any arbitrary batch of CU3 ingests in preparation for CU4 ingests. The current aim of the VSA related work is to remove unnecessary duplication in WSA-specific code to provide a single place where changes can be made as necessary to run the code for the both the VSA & WSA curation once we know exactly how the data from VISTA differs from that of WFCAM. To this end I've been upgrading all of the old code to make use of the DbSession and ProgrammeTable APIs, the former will provide the means for the code to switch between VSA & WSA operations and the latter provides a single point of reference to handle the various ways of labelling programmes." NJC reported: "I have been working on the problem of the difference in photometry for objects observed in two pointings (i.e. observed on different parts of the WFCAM focal plane). I had noticed that the lightcurves of some of these objects had two groups of magnitudes separated by up to 0.05 mag, much greater than the expected difference in zeropoint. I have investigated all of the different overlaps in the DXS (DR5) and have concluded that most have an offset that is not significantly different from zero (i.e. agreement at the millimag level), but a few have quite large offsets. These are ext 3 - ext 3 along the y-axis, ext 4 - ext 4 along the x axis, ext 3 - ext 4. In all these cases the offset is >~0.02 mag and has some variation with position getting to ~0.05 mag in places." (NB: ext 3 = detector 2; ext 4 = detector 3) NJC noted that he's preparing a paper on synoptic survey photometry in the WSA, and that much more detail will appear in that paper. ETWS reported that third-party software installations are now all up-to-date on all 8 linux servers. All 32bit machines seems to work fine; some issues with the 64bit installation and unixODBC, but ETWS noted after the meeting: "khafre and menkaure are fixed now and should work (but menkaure's installation is still ongoing, will finish later this evening). I've gone one step back (what I thought I had done already) with unixODBC. Since there are no problems with unixODBC-2.2.14 on 32-bit machines I would like to leave those as they are if nobody has strong objections against it. On 64-bit we are for now running unixODBC-2.2.12." Survey Data The team enquired if NCH had looked into the DR4-DR5 frameset "anomalies" reported by SJW last week - he hasn't (yet). ACTION: NCH to look into possible DR4-DR5 frameset anomalies to ensure no frame association bugs in CU7. Non-survey Data Release: NCH thanked RSC for organising some housekeeping info for keeping track of the non-survey release DB status - RSC noted that he has transferred his notes on the status of non-survey releases to the TWiki so that everyone can keep track of our progress and contribute to keeping this list up-to-date. This shouldn't really be needed as the automation process _should_ allow us to just release a batch of non-surveys a semester at a time and then we forget about them until a re-release of all non-surveys is required due to recalibration (or whatever). However, due to a combination of unexpected software and data bugs the status of the non-survey releases is very messy and complicated. Hopefully this list will help clear up any confusion in the meantime whilst we sort out these issues: http://apache.roe.ac.uk/twiki/bin/view/WFAU/NonSurveyReleaseStatus Astrogrid deployment & Data Analysis services: A user reported that the on-demand list-driven photometry tool is bust, so good news that someone's actually using it. MAR and ETWS investigated, and it turned out to be caused by an upgrade to the Python version on the web server. MSH noted that he'll be attending the AG meeting in the Lake District next week. Miscellaneous: Nothing else this week. ============================================================= Nigel Hambly Tel: +44-131-668-8234 Institute for Astronomy Fax: +44-131-668-8416 School of Physics and Astronomy University of Edinburgh Email: nch@roe.ac.uk Royal Observatory Blackford Hill Edinburgh EH9 3HJ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. =============================================================