From nch@roe.ac.uk Mon Dec 12 17:01:26 2005 Date: Fri, 9 Dec 2005 12:32:18 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 9th December 2005 Minutes of WFCAM Science Archive meeting: 9th December 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, ETWS, NJC, JB, MAR Apologies: JPE, JDT, MCH, RGM, AL, JMS, PMW, RSC DONM: 10am, Friday 16th December 2005, plate library Actions discharged: ------------------- ACTION: JB to look at the GLIMPSE info and begin SQL implementation with help from ETWS and NCH. Discharged; SQL script in the CVS. ETWS is developing the ingest file codes; NCH suggested he asked RSC for help as necessary ACTION: MAR to put a link to the CASU survey progress pages from the WSA. Discharged; MAR noted that a link from the top level page appeared to be broken. ACTION: NCH and PMW to remember to factor the following things in to to the Q1'06 plan of work: 4/ Quality control: flagging of residual cross-talk images, 128 pix either side of v. bright / saturated objects in single-passband catalogues 5/ Application of 2d photometric correction maps of systematic calibration errors (of order a few %) measured from all 2MASS standard residuals, averaged as a function of position on frames 6/ New data model required for poor-seeing time programme (multi-epoch photometry in search of planetary transists) also need to include tasks on UKLight and outside access to the continually updating database of ingested pipeline products. Discharged. ACTION: PMW & NCH to have a planning meeting, Mon 5th PM. Discharged ACTION: ETWS to email MR to ask if anything out of the ordinary happened at the CASU end wrt 17th April data Discharged ACTION: JB & ETWS to examine CU1 housekeeping and logs to see if there is a bug that resulted in missing catalogues for the 17th April Discharged ACTION: MAR to put a note on the WSA website noting that the catalogue data for the 17th April is missing in the current release Discharged ACTION: NCH to email SJW a warning to be circulated amongst the UKIDSS Science Verifiers that April 17th catalogues are missing. Discharged ACTION: NJC to contact DXS/UDS to update them on WSA stacking/ mosaicing and ask about their required optimisation of release stacks/mosaics. Discharged; attitude of the DXS/UDS is that they will do things themselves. This issue will undoubtedly crop up again, but in the meantime we will carry on implementing an archive-end stacking facility for default database-driven stacked/mosaiced products, including catalogues derived from them, in the absence of any guidance from the outside. Actions partly discharged but continuing: ----------------------------------------- ACTION: JDT to set up a Datascope Launcher for the WSA on the AG workbench pages. Continues. Actions carried forward from 02/12/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted the meetings next week: UKIDSS SV all day on the 15th; VDMT on the 16th at 10:30am (hence next week's project meeting will be short). PMW has already assembled the reports for the VDMT. WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 9/12 AM. Networking: With reference to the interface snarl-up that resulted in the absence of catalogue data for the 17th April, ETWS and JB reported investigating the codes at the WFAU end, and can find no problems. It seems we finished copying the data up around the 19th October, and wrote SUCCESSFULLY_COPIED files on the CASU server around 25th October (after manually checking all 49 directories and ensuring the data was all copied). However we note that the OK_TO_COPY file in the April 17th directory has a timestamp of 15th November, implying that there was activity in that directory, unknown to WFAU, up to 2 days before our last release point - clearly it would have been impossible for us to include the data in the release, even if we had known about any fixes that had to be applied for the 17th April data. All very mysterious... Incidentally, we note that the "Checked" date for the 17th April on the CASU processing status pages is 7th Oct, regardless of the 15th Nov timestamp on the OK_TO_COPY file. This is all rather confusing... In order to disabuse us of our confusion, perhaps, for the time being, it would be good if we had a weekly email update, from MR to JB & ETWS, that noted the latest availability of newly processed (or reprocessed) data, since we don't seem to be able to rely on an automated procedure at this stage. Finally, NCH and ETWS have been in touch with folks in the States regarding SDSS DR3, and it seems we need to get this from the Chicago distribution hub rather than pestering the JHU folks for it. ETWS has made contact with UChicago for copying DR3 (DR4 is not available for copying yet). WSA Operations: JB reported that he has been finishing off the 05A ingest; and operations are proceeding well. Hardware: JB noted that all is quite on the hardware front. Software: RSC reported: "I've completed the new Cu16 script. Alas, due to various database issues I haven't been able to fully test it yet. Further development to the C++ source matching code is also required. In between times, I've been helping Eckhard with C++ code related development issues." MAR reported: "All interface functionality for non-Survey data has been implemented (except for MultiGetImage). Updated code deployed and being tested. Schema browser & glossary updated to include non-Survey docs. CrossID interface released having updated to work with non-Survey and current schema. Usual updating of web pages." ETWS reported: "Finished the compressed image creation. In total 236431 files have been processed. Also worked on the non survey ingestion programmes. Started to include a routine in CU4 to check for missing FITS keywords which cause problems during the ingestion procedure." NJC has noted the absence of DQC attributes for many images. This has been traced to an ingest feature that only picks up DQC from catalogues for certain parameters (so if there is no catalogue for an image, some DQC is not being ingested). NCH suggested that he and ETWS review the metadata schema and CU3 proceedure again to fix this one asap. ACTION: ETWS and NCH to sort out image DQC schema and ingest NJC (and SJW) have been looking at the pipeline files delivered, and reconciling those with the processed products and image descriptor attributes. As fars a we can tell, pipeline and archive are functioning correctly in terms of grouping, processing and ingesting the metadata to keep track of where an image comes from; however there appear to be some examples where the image keys are inconsistent (e.g. reduction recipes not consistent with jitter and microstep keys). NJC expressed the concern that this may be an issue for JAC to look into, and volunteered to send some examples. ACTION: NJC to send some examples of apparently inconsistent image attribute sets to JAC. NCH noted that, in general, it will be difficult for us to reconcile what we receive from the data flow system with the original MSB inputs since we don't have full access to the OMP database; CASU are mirroring this (we believe) - maybe we could access their mirror, or perhaps we should have our own. Survey Data Release: No news this week Non-Survey Data Release: NCH asked about the status. ACTION: JB and MAR to close out non-survey release of available DBs asap. Astrogrid deployment: MAR noted that he will attend AG-DAG in Leicester next Thursday, and has been asked to give a short presentation on what we want from AG. NCH suggest he reviewed the existing requirements doc (the SRAD) to pull out a list of all requirements where we have explicitly cited the need for AG infrastructure, or where we are struggling because of lack of technical know-how or resources. Miscellaneous: Nothing further this week.