From nch@roe.ac.uk Tue May 22 08:41:32 2007 Date: Fri, 18 May 2007 17:34:36 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 18/5/07 Minutes of WFAU VDFS Science Archive meeting: 18th May 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, ETWS, JB, LGR, NJC, MSH Apologies: JDT, BCW, JPE, AL, PMW, RGM, JMS DONM: 10am, Friday 25th May 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: Technical focus group to meet after next week's project meeting to sort out the general recalibration problem. Discharged; NJC has kicked off the schema mods and RSC created a new TWiki topic and Trac software enhancement tickets/notes etc. NCH needs to review before giving the final go-ahead. ACTION: ETWS to progress his "ingest status" browsable pages as quickly as possible with a view to semi-automation of eyeball checks on survey data. Discharged; the team discussed the new summary web pages and suggested an extra column along the lines of "ready for eyeball checks" or somesuch so that UKIDSS Survey Heads can see when a new chunk is ready for eyeball QC; otherwise thanks to ETWS for sorting this out. Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. Actions carried forward from 11/05/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES; remaining problems with ZY NDR data from early 06B should be sorted quite soon now, so the schedule should become easier to tie down. ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES ACTION: NCH to remember to ensure the GPS schema is modified as per instructions from the GPS survey head, and Cookbook examples included to recover attributes when needed. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH reported back from the Board, where a presentation was given that focussed on ingest/flat-file release timescales as well as a general update. The Board seemed generally happy, and were reassured that recent time-lags in flat-file access becoming available were being treated seriously and that the situation is improving rapidly now. NCH reported that the case for development resources in the current rolling grant bid has been drafted. The UK VISTA User Requirements Document is undergoing a revision at the same time. NCH asked that RSC, MAR and NJC comment on the draft case. WFCAM & VISTA updates: As regards VISTA ESO Public Surveys, JPE, MJI and NCH are all getting bombarded with revised Survey Management Plans and RIX responses; hopefully the terabyte RAID arrays can cope with all the email generated. Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB reported that UKLight was fixed earlier in the week, the problem was a piece of UKERNA hardware that was fixed as a priority as it broke the ESLEA collection of projects as well as ATLAS and our own project. NCH noted that MJI had told him about the recent JAC-CASU network tests, which can sustain a most respectable 4 Mbyte/s using multithreaded ftp. JB noted that we would be happy to try out testing FTP again as it should be faster over the UKLight connection due to the lack of firewall rules that it would have to negotiate to get through. NCH has received a bit more info from MJI concerning astrometric recalibrations, and has forwarded this on to the appropriate people. WSA Operations: JB reported: "Having looked at the recent logs for data build and ingest it appears that we're having a very clean run of data for 06B so far - no files have failed on ingest, impressive!" Hardware: JB noted: "Ahmose was dropping connections again but the problem identified and steps taken to avoid the problems in future, this slowed down ingest for a few hours but overall we should be okay." ETWS noted that software upgardes to khufu, khafre and djedefre will take place early next week (see below). Software: MAR reported more refactoring of UI code including investigating if it's easy/worthiwhile converting CGI scripts from PERL to Python and work on batch queue system code and schema. RSC reported: "Helped thrash out the new strategy for approaching recalibrations and created a TWiki page, RelcalibrationStrategy, to capture this discussion, and several actions on the trac bug report system tickets: 57-60 (but also 55, 56, 61 are relevant). Enhanced the DB API to allow table constraints to be easily and specifically dropped and recreated in a schema driven way (trac ticket: 51), and to ensure table copies keep constraints. This has allowed me clean up a few bits of stray SQL that shouldn't be scattered around the scripts (trac ticket: 28), and also to improve the functionality of the script to synchronise test databases (SyncTestDb.py). Also cleaned-up some schema scripts and updated the WSA_Constraints.sql file to include the calibration table constraints. Investigated why CU8 has stopped working following the recent software upgrade - turns out to be a bug in new version of PyFITS that was installed - ETWS has remedied this by replacing PyFITS with the cutting edge version from SVN that contains the relevant bug fix (track ticket: 54). This has also given me the opportunity to return to investigating a solution to the potential data corruption bug if a keyboard interrupt occurs during a pyfits.flush() operation (trac ticket: 4)." ETWS reported: "Ran CU3 data ingest in parallel and ingested 20 days (64000 files) in ~7h. This means a speed up by a factor of 3 without any fine tuning done yet. Created progress web pages for CUs 1 to 4. Finished software update on djoser with upgrading pyFITS. Reminder: The other servers will be upraded on Monday/Tuesday, 21/22 May." NJC reported: "This week I have worked on the testSchema for recalibration and other issues relating to recalibration. I have been updating CU6 to measure the variability statistics and implementing the best aperture approach as discussed in paper by J.M.Irwin et al. 2007. I have been carefully testing each step and have reduced the RMS to about 7mmag. More testing is necessary as there are still a few bugs to remove." Finally, JB noted: "Have been following and, hopefully constructively, contributing to the recalibration discussions from the operator and database manager point of view. (I'm looking forward to the new streamlined system - even if it will take a while! :)" Survey Data Release: NJC has been looking into extended source photometry, and noticed a few features of the Kron and Petrosian mags that should perhaps be explained a bit more on the archive pages. Non-survey Data Release: JB noted that the two remaining Non-Surveys to be released from the recent batch that were held back by a bug are on their way. Astrogrid deployment: Nothing new to report this week Miscellaneous: Nothing else this week.