From nch@roe.ac.uk Mon Aug 20 11:51:14 2007 Date: Fri, 17 Aug 2007 15:03:15 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly project meeting minutes, 17/08/07 Minutes of WFAU VDFS Science Archive meeting: 17th August 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, PMW, ETWS, MAR, NJC, JB, LGR, MSH Apologies: JDT, BCW, JPE, RGM, AL, JMS DONM: 10am, Friday 24th August 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: MAR to switch off proprietary status of UKIDSS EDR in the WSA interface on August 10th. Discharged ACTION: PMW to speak to JB concerning CASU-WFAU network investigation. Discharged Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. ACTION: RGM to look through requirements docs to see if any users have mentioned GALEX. Continues; RGM supplied GALEX details (included in last week's minutes) Actions carried forward from 03/08/07 meeting: ---------------------------------------------- - none this week. Specific points and new actions: -------------------------------- Project management: The team noted that an ESO Workshop on UKIDSS has been announced for the 17th-19th December: http://www.ukidss.org/esoworkshop Attendance from WFAU will be at least NJC, NCH and ETWS, and possibly JB. Note that travel should be booked asap as it's so close to xmas. NCH noted that he has sent comments to JAC director Gary Davis concerning the draft MoU for data flow system ops for WFCAM; also had a face-to-face meeting with Gary earlier this week when he was over for Scuba-2 things. WFCAM & VISTA updates: Apparently earthquakes and hurricanes have interupted WFCAM ops this last week - we trust all at JAC and WFCAM/UKIRT are safe and well. NCH noted that JAC survey support scientist Luca Rizzi is planning a visit to the UK in late October/early November, and that it will be a good opportunity for the team to show him what goes on at this end of the data flow system. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of the meeting of 8/8/07. There were a few comments: regarding network transfer tests, we thank PSB for his patience and work in benchmarking the network upstream from WFAU - see below; as regards the DR3 release schedule, we're not sure exactly who suggested that "A release date some 6 weeks after this [i.e. early 06B] data is released to ROE" is a reasonable timescale; in any case, NCH notes that he has consistently maintained that releases will happen around 8 weeks after the last data to be included has arrived at WFAU (NB: not just released). Maybe it's a misunderstanding from something NCH said at the last VDMT. No biggy. A couple of other clarifications: in order to expedite DR3, we will not be applying low-level 2d recalibration enhancements (e.g. the varying pixel scale across the FoV photometric tweaks) nor will we be incorporating any newly reprocessed pre-06B data (e.g. the re-crosstalk corrected GPS frames) because, especially in the case of the former, the worry is that attempting to do so will cause yet further delays. In the case of the recal issue, this has been agreed with UKIDSS Survey Head SJW; in the case of the GPS, needless to say the relevant Survey Head is p***ed off (oh well, you can't please all the people all of the time). In any case, WFAU are aware of these issues and will be progressing them post-DR3. Networking: After tenacious benchmarking efforts by PSB, it is apparent that there is a bottleneck at the WFAU end that is resulting in sluggish network transfer performance. JB suspects it is the rats nest of NFS crossmounts on all the WSA servers that severely impacts local file transfer speeds. Work is underway to sort this out. JB also noted that ftp will be used in place of scp to alleviate CPU stress at both ends. JB noted: "A test machine (Seaforth) was setup by HME and connected to UKLight to allow UKLight and CASU to ROE network switch tests to be conducted. These conclusively proved that the current transfer speed problems exist between the ROE network switch and the disk being written to. Investigations and tests continue to try and isolate the current bottleneck but, despite progress, these remain inconclusive at present." WSA Operations: JB reported: "The setup of SQL on the Windows machines has been recorded on the twiki with notes explaining variations from the norm. Some broken files have been reported to CASU as well and the data from 20061229, 20061230 and 20070213 being scrubbed in preparation for fixed versions being transfered. Provenance was ingested for all the data between 20061220 (2nd part of 06B) and the end of 07A. Quality Flagging has been run for the GPS currently ingested for 06B and 07A. CU1 transfers have started for the data between 20061201 and 20061219." Some concern was expressed during the meeting as regards database backups. MAR also noted that DR2 needs a backup after NCH's seaming fixes. ACTION: JB to backup the WSA ingest DB this weekend. MAR noted that DR3 QC first pass should finish next week (final pass and application of the deprecations will be made when the 06B data is completely ingested). Hardware: JB reported: "One of the RAIDs on khufu had a glitch with one of it's disks, it is being monitored with a view to replacement if there are any more problems. The final details in transfering disk03 to osorkon were successfully implemented and it seems to work well, this work continues with one of djoser's RAID disks being transfered to osorkon, this will take time but will improve local system architecture. Snerferu is being used as a testbed for a new local system architecture which should hopefully reduce load on the PAN." JB sketched up a summary of the rearrangement of the WSA local area network for comments from the team - all agreed it seems sensible. NCH raised the subject of catalogue server disk space in light of requirements for DR3 ACTION: JB to check out the available disk space on ahmose/amenhotep to ensure there's enough space for DR3. RSC asked about the possibility of DB tests with new SQL2005 catalogue server hatshepsut. JB and MSH noted that this is now on the network. NCH suggested a stress-test by loading some (or all?!) of the SSA as a first step. ACTION: MSH to liaise with JNTD on transfering the SSA ingest files from cosaxp6 to hatshepsut and to do some load tests. Software: NJC reported: "Tested mosaicing and extraction software. Some bugs in the python script due to updates of 3rd party software and development work. These have been fixed. The latest release of the CASU extractor gives the expected results. I have mainly been working on testing the galaxy photometry. This is ongoing." RSC reported: "I've started to work on the implementation of the cross-talk flagging for the UDS `deepmosaic` frames (trac ticket:33). The main hurdle is to be able to convert from pixel (X,Y) co-ordinates in the detector plane to sky (RA, Dec) co-ordinates in the celestial sphere. For this I need to create a Python wrapper to a WCS C library, either Starlink's AST or Mark Calabretta's WCSLib (see trac ticket:90). WCSLib would be preferred as it has a cleaner interface for utilising WCS values extracted from the database, whereas AST is more useful for dealing with FITS files in the way it is used in CU4. However we've had problems dealing with zenith projections in WCSLib in the past. This PyWCS wrapper will be accessed through a new Astrometry module for general astrometric calculations in Python that also includes the Slalib functions converted to NumPy Python. Work on this has been somewhat delayed due to time spent tracking down a mystery bug in the development branch that only affects CU19. Because of this I've now stopped working on the development branch (trac ticket:68) and am focusing on the quality-bit flag enhancements. Also there's been some overhead on SVN branch maintenance this past week, ensuring bug-fixes are applied to all branches." ETWS reported: "Fixed problems with automatic default ArchiveCurationHistory entries showing the processed date. Fixed CU1 loging problem due to changes in StringIO from Python 2.4 to 2.5. Included statistics for calibration and non-survey catalogue data in the monitoring pages' creator. More tests on the Python Remote Objects implementation." NCH suggested that ETWS had a quick trawl through the Python software to check for any other knock-on effects of the StringIO change. ACTION: ETWS to check WSA Python code for StringIO change. NCH noted that the GPS DR2 reseam has finally finished after various crashes and interruptions. These fixes need to be copied into the batch catalogue server version of the DB, and the code enhancements checked into the SVN trunk version of CU7. ACTION: NCH and RSC to merge in latest speed enhancements for CU7 into SVN. Survey Data Release: No update possible on T0, but the 06B data is on it's way in. T0+ What: 3.0 CU1 (transfer - assuming sustained 3MB/s) 1.5 CU2 (jpegs) 1.0 CU3/4 (ingest and provenance updating) 1.0 CU8 (photometric zeropoint recalibration) 2.0 QC1 (much work to be done in parallel with previous tho') 0.0 CU5 (diff images for the GPS - after QC, but in parallel with the following and it's very quick anyway) 1.0 Quality bit flagging 3.0 CU7 (source merging from scratch again, unfortunately) 2.0 Final CUs. Watch this space for further updates... Non-survey Data Release: MAR noted that 5 more non-surveys have been set up for flat-file access this week; checks will be made and PIs informed if any are affected by the recent retransfer/reingest of dark-reprocessed nights from 06B. NCH noted a conflab earlier in the week with local WFCAM PI non-survey user Annette Ferguson which raised some interesting general quality issues for non-survey users. MAR and NCH have discussed adding in automated broad-brush QC for deprecations that require no decision making (e.g. interupted MSBs/incomplete groups etc) and that these should be incorporated in the standard quality flagging procedure to make, for example, any non-survey merged source catalogue products easier to use. Astrogrid deployment: Nothing of note this week. Miscellaneous: Finally, NCH noted that there is a workshop/seminar on Digital Preservation on 21st Sept in the University: http://www.ucisa.ac.uk/groups/cisg/misgevents/DCPSeminar that might be of interested to the operations side.