From nch@roe.ac.uk Fri Aug 19 13:23:10 2005 Date: Fri, 19 Aug 2005 12:31:22 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA Weekly project meeting, 19th August 2005 Minutes of WFCAM Science Archive meeting: 19th August 2005 ------------------------------------------------------------- ------------------------------------------------------------- Present: NCH, PMW, ETWS, MAR, RSC, NJC Apologies: MCH, JPE, JDT, RGM, JMS, AL DONM: 10am, Friday 26th August 2005, plate library Actions discharged: ------------------- ACTION: PMW to inform catering that a mighty host (up to 30 folks) will descend on the canteen at 12:30pm next Thursday. Discharged. ACTION: NCH to book a digital projector for the day (Thurs 18th) Discharged. ACTION: NCH to notify site staff of the influx of hungry science verifiers for lunch at 12:30pm next Thurs. Discharged ACTION: NCH to contact MJI to ask about error-bit flags and parent/child deblending info, in the CASU standard catalogue parameters. Discharged; response from MJI (without even having to ask!): "I've just finished putting the final touches to the standard catalogue parameter set. The next reprocessed version of the catalogues will contain all the fluxes and flux error estimates plus the error estimates for the other standard catalogue parameter set. In other words the catalogues will have columns 1-62 filled in. This will happen over the next 4 weeks. Column 55 now has an error flag set which is currently set to the number of bad pixels in the "rcore" radius of an image. Column 58 is sort of redundant but left in anyway, since ALL images are always deblended and only the unique deblended images are recorded. [We could possibly use this column to later record some other aspect of image processing problems eg. hit edges, contains saturated pixels......] When we add in the PSF processing these catalogues will be further updated to include the columns relating to stellar PSF photometry and astrometry ie. 63-70, this will happen during the Autumn for the 05A data. 71-80 still TBD (to be developed) so not sure yet when." ACTION: RSC to submit a poster abstract for ADASS, on the WSA. Discharged. Actions partly discharged but continuing: ----------------------------------------- ACTION: PMW to initiate removal of old hardware from C1 by communicating with all concerned parties (JNTD, HMG, RL, ...) Building works will finish with concrete screeding of the floor outside C1 on 20/21st Aug; major equipment movements to be instigated once the goo is dry. - continues. ACTION: NCH liase with JNTD to sort out Windows accounts, file share access and an increased remote connection limit on W2K3 DB servers. Continues; helpdesk in to IT Support; NCH has done everything possibe within his (admin group) priviledge. ACTION: MAR & RSC to assemble an advertisement-feature poster concerning the WSA for ADASS. Continues; MAR has dug out the old poster. Actions carried forward from 12/08/05 meeting: ---------------------------------------------- ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. Continues ACTION: JDT and MAR to continue sorting out AG access option functionality for saving WSA results to "MySpace". Continues; RGM noted that Astrogrid installations on new hardware will commence next week, and then implementation and testing of this interface option can take place. Specific points and new actions: -------------------------------- Project management: NCH noted the outcome of yesterday's UKIDSS science verification meeting in Edinburgh. Dates of next meetings are: 20th Sept (calibration meeting; NJC to attend) and 21st Sept (second SV meeting; NCH & NJC to attend). Overall, the impression given by the community at the meeting seemed to be quite positive. A list of archive issues/wants was compiled by NCH, and the team discussed extensively the requirements along with implementation issues and an action plan in response to UKIDSS feedback has been produced (see below under Software). WFCAM update: See the stop press pages linked from the TWiki for latest news. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 19/08/2005. Networking: NCH notes (after the meeting) that Peter Clarke (network guru and UKLight chap from NeSC) came to speak with AL, who brought him around to speak with NCH & ETWS earlier this week. It seems that the UKLight project is in a deployment testing phase for specific projects requiring high bandwidth, and the impression given was that a dedicated Gbit/s link through SRIF connections may be obtainable as part of connectivity tests etc. Peter will be asking NeSC bods (including our old colleague Clive Davenhall) to make contact to discuss requirements at a more technical level over the next few weeks. Hardware: All hardware still functioning normally. New storage unit "khufu" has been assembled at Eclipse and will be delivered shortly; there is some uncertainty over the status/location of the new rackmount cabinet though. Software: The major topic discussed this week was the work required as a result of feedback from yesterday's UKIDSS SV meeting. The wish list, as prioritised at this meeting, is as follows: High priority: 1) Fix ingest bug on aperture corrections [ETWS, NCH] 2) Update archive schema (e.g. merge source tables; include QC info and other missing metadata delivered through the pipeline) [NCH, ETWS] More parameter description in the schema browser [ETWS, NCH] "All params" and "Default subset" buttons for attribute selection in the menu-driven catalogue search [MAR, ETWS] 3) Interface upload facility for cross-ids & thumbnails [MAR] 4) Post-ingest QC [awaits implementation of 1)] Medium priority: 5) Check merge classification consistency: e.g. p(class) and class assignment in merge tables (e.g. p(Star)=0.9999 and class != -1 ?!); Make multi-epoch frame choice more intelligent when forming frame sets; DXS & UDS cataloguing & current "best" stacks and mosaics [NJC] 6) "Show me SQL" option on region query form [MAR] 7) Batch job query queue with increased row limits [MAR] Lower priority (for the time being): - More views for filtering spurious objects (e.g. moving objects) [awaits any response to SJW asking around interested parties] - Flag cross-talk images [awaits pipeline refinements in removal at the pix processing stage] - Login/auto-logout issues [awaits further tests; not entirely sure what the problems/issues are] - Client-side Query Tool [this is in the existing plan for later versions (4/5?)] - Better survey progress/monitoring, in particular for the narrow/deep surveys (DXS/UDS) [awaits liaison with relevant survey heads] Ongoing work: - Update the release history with known bugs/issues and link in the SV TWiki topics. Data Release: ETWS reported that SV2 catalogue data have now been ingested following a small burp caused by one catalogue file from one night (22 April). However, the impression gained from yesterday's UKIDSS SV meet is that as many bugs and high priority enhancements as possible should be addressed before the next release (i.e. we should not rush out the SV2 release without fixing these). Given that wholesale catalogue re-ingestion and source merging is required along with major schema revision, these will be addressed as a higher priority than releasing more SV data. SSA: Nothing new this week Astrogrid deployment: Nothing new this week. Miscellaneous: MAR again raised the thorny issue of non-survey data release, and noted that several PIs are still knocking on the door. ETWS suggested we test some procedures with existing data, and the team agreed to not make any releases until the major issues noted above have been addressed.