From nch@roe.ac.uk Sat Mar 18 11:25:06 2006 Date: Fri, 17 Mar 2006 14:44:23 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 17th March 2006 Minutes of WFCAM Science Archive meeting: 17th March 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, PMW, JB, NJC, AL, MAR, RSC, RGM Apologies: JPE, JDT, JMS, MSH DONM: 10am, Friday 24th March 2006, plate library; PMW in the chair Actions discharged: ------------------- ACTION: RGM to look into the possibility of external funds for an SDSS catalogue server. Discharged; AL and RGM looking into this possibility. ACTION: NCH review aperture mag documentation in the LAS schema. Discharged; ETWS has propagated the changes into a newly implemented template schema system - see Software below ACTION: NCH to ask MAR to put the enhanced browser online on his return from hols. Discharged; MAR will do this once the recently updated browser documentation has been reparsed. Actions partly discharged but continuing: ----------------------------------------- None this week Actions carried forward from 10/03/06 meeting: ---------------------------------------------- ACTION: MAR to update the IfA WFAU web pages. - CONTINUES ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted that the Vista public surveys deadline has now passed, and that there are a dozen or so proposals in that explicitly mention use of VDFS processing and archiving. WFCAM update: CASU minutes imply (see below) that there was a problem with one WFCAM detector towards the end of the just-completed semester; we at WFAU would like to know some more details just in case it impacts anything at the archive end. Comments and issues arising from CASU fortnightly minutes: The team noted the minutes of 6/3/06 AM. A number of issues were discussed: The question of the data transfer rate between CASU and WFAU is raising some concerns at the WFAU end because sometimes we do not seem to be able to sustain the rate necessary to be able to keep up with WFCAM data, let alone Vista. JB and PSB have been investigating possible causes of this; the team recognises that CASU have to transfer raw data to ESO which probably impacts the bandwidth available to WFAU. However, the bottom line is that CASU should acknowledge that bandwidth is sometimes an issue at the moment, and will undoubtedly be an issue for Vista. Note also that power outages (to be fair, at both ends) and whole-sale reprocessing, cause delays in data transfer and require that we need plenty of spare capacity. The team noted also the hiccups caused in the pipeline by absence of one or other detectors' data; as far as we are aware there should be no problems at the archive end in handling data with one (or more) missing detector data sets. NJC noted the item concerning dynamic range in the standard processing philosophy. Again, this may not be an issue for CASU in the nightly pipeline (although NCH noted that it is marginal since some nightly J stacks in the DXS are aproaching a noise level of just a few ADU) but it most certainly is an issue for WFAU's running the toolkit codes to create yet deeper stacks... this one is going to return to haunt everyone prior to the DR1 release. Networking: JB reported: "UKLight is progressing, I have regular contact with Peter Bunclark and things are moving ahead smoothly enough, transfer speeds looking like becoming more of an issue if we are to be able to handle the reprocessing of data (ie needing to redownload it)." WSA Operations: JB reported: "CU1 has been running most of the week and I am now in the final stages of cleaning up the broken files caused by power-cuts and the like. We have transfered up to January 13th (inclusive). CU3 and CU4 are undergoing cleanup operations due to problem files but are in progress, they have ingested up to the beginning of November." Hardware: No news this week. Software: NJC reported: "Completed CU13 - mosaics are now being correctly updated without running out of memory. Started making changes to Schema and CU7 (with RSC) to do extinction correction. Started doing comparison to UKIRT standards." RSC reported: "Mostly I've been bug fixing - solved the problem of spurious false detections of NaN values on data ingest. Fixed the issue of skysubscale values not honouring the specified default value. Modified the Python database interface so that SQL code strings containing apostrophes are handled gracefully. I have also been generally improving some of the Python and C++ code designs to make development easier in the future." ETWS reported: "Compartmentalised the survey schemas to be updated via a template schema. Applied the changes made to the lasSchema." Survey Data Release: On the question of survey releases, NCH outlined the current working assumption that the next data release will happen on or around July 1st (the so-called Data Release 1) and that this would consist of the current processed version of 05B data plus reprocessed 05A data (reprocessed to bring it into line with 05B that is). NCH noted that immediately after this meeting, AL, RGM, MAR and NCH would have a top-level brainstorm to discuss the feasibility of more frequent, less highly polished releases given the experience of running the various procedures in anger for a year or so, but at this stage no interim releases prior to DR1 are envisaged since agreement on exactly what form this should take and how it should be done remains elusive (not to mention finding the time to actually do it...) Non-Survey Data Release: JB noted: "Three new NonSurveys have been added to the database (U/05A/37, U/05B/15 and U/05B/18), with U/05A/37 and potentially another being released soon. BestDR2 has almost been finished being split and should finish next week some time. BestDR3 still pending on this." ACTION: JB to release newly registered 05A data sets. Astrogrid deployment: The planned visit from Guy Rixon is looking like happening towards the end of March or possibly the beginning of April. AL noted that around the same time, we should have a serious discussion on exaclty how we would like the presentation of UKIDSS data to look on the AG work bench, and exactly what functionality should be included. Miscellaneous: Several folks have noted the announcement of the Astronomical Image Processing workshop to be held in Dubrovnik Sept 4-9. NJC expressed tentative interest in attending this. Finally, NCH reminded everyone that he would be away for the next three weeks, and the entire team assured him that they all have plenty to do ...