From nch@roe.ac.uk Mon Oct 2 11:57:40 2006 Date: Thu, 28 Sep 2006 15:36:45 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS Science Archive weekly meeting minutes, 22 Sept 2006 Minutes of WFAU VDFS Science Archive meeting: 22nd September 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, MAR, JB, ETWS, MSH Apologies: JPE, AL, JMS, PMW, RGM, JDT, NJC DONM: 10am, Friday 29th September 2006 in the Plate Library Actions discharged this week: ----------------------------- ACTION: MAR to add angular distance to Moon in archive listing to aid 05A v3 analysis. Discharged ACTION: ETWS to email SJW when all of 05A April images and JPEGs are in for UKIDSS heads to peruse. Discharged ACTION: MAR to tidy up unused/unusable procs/funcs in WSA stored procedures and functions script, and fix up the ingest DB. Discharged; see Software below ACTION: PMW to include EVA in VDF-WFA-VSA-003 Discharged (with some misgivings, apparently...) ACTION: ETWS to check through the documentation for formatting and consistency of figures etc. Discharged; ETWS has circulated suggestions for nice html formatting via use of a much more up-to-date latex2html available on archive server khufu. Actions partly discharged but continuing: ----------------------------------------- The following from last week partly done but continue: m) rearrange external catalogues onto batch catalogue server - ACTION: JB p) Release calibration and transit survey data as non-survey databases - ACTION: MAR, NCH & JB Continues; some final issues related to the size of the (temporary) neighbour tables. NCH: VDF-WFA-VSA-002 Science Requirements Analysis PMW: " " " 003 Management and Planning NCH: " " " 004 Interface Control JB: " " " 006 Hardware/OS/DBMS Design NJC: " " " 007 Database Design MAR: " " " 008 User Interface RSC: " " " 009 Software Architecture Design, incl. local data flow RGM: " " " 010 VO Enablement ACTION: ALL to draft their respective documents (see above) by the 1st September. Continues; doc release on schedule for the 30th September Actions carried forward from 22/09/06 meeting: ---------------------------------------------- The following from last week continue: d) Debug CU7 for deprecated frame sets and deletions on rerun - ACTION: NCH i) Enhance seaming to use quality bit information - ACTION: RSC (on hold until after h) k) Ensure ingest code can cope with 06A missing data quirks and any associated new attributes - ACTION: ETWS o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH - all are on hold until after the documentation for the review has been prepared. ACTION: RGM to email Ken Rice (IfA contact concerning IBM P690s) about potential WFAU interest in high-end freebies from IBM. - CONTINUES ACTION: NCH to circulate a release schedule for DR2 to all concerned. - CONTINUES ACTION: JB to make a full WSA backup around mid-September. - CONTINUES ACTION: NCH to include discussion of VISTA PSP requests for enhanced VDFS archive functionality in VDF-WFA-VSA-002 - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH noted that continual disruption to the location of meetings held in the Vista Hut is p***ing him off sufficiently for NCH to unilaterally declare that all future meetings will revert to the Plate Library. ACTION: NCH to remember to ask PMW to revoke the rolling Vista Hut meeting room booking for these meetings. The team discussed who should attend the UK VDFS FDR review in Cambridge in October. After some deliberations, it was decided to delay this decision a week or so. WFCAM update: Nothing new to report this week; RSC & NCH will be observing during December. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 22nd Sept AM. Networking: It was noted (albeit with some skepticism!) that UKLight is "ready to go at the Cambridge end" so WFAU apologises for the delay at the Edinburgh end, which is due to circumstances beyond our control (i.e. EUCS). JB noted that he and ETWS intend to attend a network meeting at NeSC concerning "UKLight and beyond" (or something like that). WSA Operations: NCH emphasised the need for a robust disaster recovery plan (i.e. a good backup policy), when it transpired that no full backup has been done yet. Some discussion ensued concerning the use of regular incremental backups in combination with occasional full backups. JB expressed concern about the feasibility; NCH suggested that we should seriously consider at least doing some tests. JB reported: "System backups continue as normal and servers have been operating as normal. CU1 is running again for some data at the start of 06A, CU2 is running (thanks Eckhard), CU3 and CU4 have now finished for April of 05A version 3 and some of the broken files have been dealt with. CU16 has also been run for calDetection though this produced about 150GB of data in the WSA and created a table containing over 6 billion rows... steps are being taken by Ross and Nick to reduce this, my thanks!" Hardware: MSH raised the concern over the number of available IP addresses on the SRIF network, now that AG server watten has moved over. ACTION: MSH to liaise with IT Support over obtaining another allocation of IP addresses for SRIF networked machines. Software: MAR reported: "Altered archive listing access to return angular distance from moon when querying the WSA. Cleaned up function SQL script so that browser can display what working functions are available to the user. Also updated interface scripts to make the necessary permission changes. Revised user interface doc based on comments and added figures etc, essentailly finished." ETWS reported: "Created 2XMMP catalogue, added unit vector coordiates and HTMIDs, and updated/wrote the necessary SQL scripts for ingestion. Fixed some problems in the function parsing part of the web page parser. Compressed image creation is nearly finished for 05AV3 data. Installed newer versions of latex2html and xpdf scripts on our machines." NJC reported: "Continued with the Database Design Document. I have corrected parts and rewritten the section on the Synoptic ERM. Read through other documents and supplied a few comments. Written much of the recalibration code and tested it. With this I found errors in the DXS deep stacks due to duplication of header keywords by the CASU code cir_imcore. I have added a function into CU14 to remove duplicate keywords. Started CU6, and updated the calibration and pts schemas. Thought about useful variability statistics and made suggestions." NCH noted that all of the new data modelling work was being incorporated into the review documentation for all to have a look at. JB reported: "Documentation is going along well with the first half of the document down to the odd typo and most sections now having been well checked. A number of software bugs have been ironed out from the merge although more changes have been made and need testing. CU1, CU2, CU3 and CU4 now all definitely work in real work conditions." RSC is working so hard on so many things that he now keeps a work log on the TWiki at http://apache.roe.ac.uk/twiki/bin/view/Misc/RossWorkLog where the main areas of work done were in the development branch merge and associated bug fixes, overall software architecture design implementation and documentation, and modifications to the TWiki pages themselves. Survey Data Release: No news this week. Non-survey Data Release: Nothing new this week. Astrogrid deployment: Nothing new this week. Miscellaneous: SJW has forwarded a suggestion to the team concerning automated IAU naming being done in the archive. Since the naming is based on co-ordinates strings to arbitrarily high precision, some thought needs to go into this to make it future-proof against astrometric recalibration, but in principle the team felt it was a good idea. ACTION: NCH to try to put something into the DB design document concerning this. JB noted the Digital Curation Seminar at NeSC announced recently, and that he intend to go along.