From nch@roe.ac.uk Fri Jul 2 14:10:18 2004 Date: Fri, 2 Jul 2004 12:16:43 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Harvey MacGillivray , Ian Bond , Mike Read , Nigel Hambly , Bob Mann Cc: CCs for WSA weekly meeting minutes distribution -- Clive Davenhall , Andrew Lawrence , Andy Adamson , Peter Shillan , John Taylor , Jim Emerson , Martin Hill , Mike Irwin , Peredur Williams Subject: WFAU WSA weekly meeting minutes, 2nd July 2004 Minutes of WFCAM Science Archive meeting: 2nd July 2004 ------------------------------------------------------------ ------------------------------------------------------------ Present: NCH, MAR, PMW, RGM, ETWS, JDT, MCH, RGM Apologies: JPE, HMG, GPS, JMS, AL DONM: 10am, Friday 9th July, plate library. Actions discharged: ------------------- ACTION: ETWS to send details of the RAS vacancy web pages to PMW and PMW to place the advertisements there Discharged ACTION: RGM to put an AstroGrid news item on the AG TWiki concerning the new WSA positions Discharged ACTION: PMW to ask Eleanor/Jason to put the adverts on the site-wide web site "what's new" pages, and forward adverts to UKIDSS consortium. Discharged ACTION: NCH to arrange a meeting between Matt/Ian/Jim of Eclipse and us (NCH, RGM, PMW, JNTD, and if poss AL) to try to push the catalogue server situation forward to a satisfactory conclusion. Discharged: meeting arranged for Mon 5th pm; see Hardware below. ACTION: NCH to consult independent expert Andy Knox for more advice concerning the large SCSI disk arrays. Discharged: see Hardware below. ACTION: PMW to order another 3 boxes (5 tapes each) of Ultrium LTO-2s Discharged. ACTION: RGM to chase up MCH and JDT concerning the AG deployment. Discharged: see below. Actions partly discharged but continuing: ----------------------------------------- ACTION: ALL to review risk register and think of any new internal or external risks that should be documented in the risk register. Defered until top level plan is explicitly defined. PMW tabled the VISTA risk register as an example. ACTION: JPE to design and set up centralised VDFS web pages. Progressing. (there is now a link to it from our WSA Twiki). - continuing; thanks for progressing these. Actions carried forward from 23/06/04 meeting: ----------------------------------------------- None this week. Specific points and new actions: -------------------------------- Project management: NCH asked PMW about the funding situation, which continues to be uncertain owing to delays in getting agreement and confirmation of grants between PPARC and the VEGA management. NCH expressed the worry that it would be a disaster if failure to sort this out quickly resulted in loss of more team member(s) from WFAU because they feel compelled to apply elsewhere for jobs when they have no written guarantee of employment here from April 1st 2005 onwards... PMW raised the issue of the Q3 plan, and suggested that he and NCH meet next week to finalise this in advance of the July 15th VDMT face-to-face being held here in Edinburgh. NCH thanked the relevant people for discharging the requests for greater dissemination of the job adverts. The current tally of applicants stands at 36, despite the AAS jobs register only just appearing. Comments and issues arising from CASU fortnightly minutes: No new minutes as of today. Networking: NCH reported that Alex Szalay and Jan Vandenberg had called yesterday, asking for an SQL DB backup of the USNOB to save them from loading it. NCH is creating the necessary files (after fixing up the v. small no. of rows corrupted by a LEDAS disk problem) and will be communicating with Jan as to transfering them via the network. If the network is too slow for a viable copy, then the fall back is to write a disk or tape and for RGM to take this over to the USA on Saturday week. NCH suggested he keep ETWS and RGM in the loop from here onwards in case any issues arise over optimising the transfer. NCH noted that the amount of data needing to be transfered is just over 200 GB. Hardware: NCH reported arranging a meeting with Eclipse on Monday PM; and agenda has been circulated for discussion. In the meantime, NCH has been in touch with Andy Knox (IBM) who has emailed a reply, the subtext of which is "what else do you expect with cheap hardware?". NCH noted that ahmose is now behaving normally after it's little tantrum; the SSA is back to normal (see below); an experiment has been done with amenhotep whereby the JBOD disks have been removed which halves the length of the SCSI chains - so far, the system has not fallen over. Whether this indicates SCSI cabling problems or interconnect problems in the JBOD chassis is presently unclear. Eclipse are still trying to source shorter SCSI cables. These issues will all be discussed more at Monday's meeting. NCH noted that as part of his soak-testing of the reconfigured amenhotep, he has benchmarked the trawl-rate at 170 MB/s (cf. ahmose 300 MB/s). This is the price to be paid for using hardware RAID it seems... none of this is unexpected. Software: MAR reported: Mainly still working on image/pixel extraction. Created a test WSA DB on my laptop's SQL Server, contains the tables CurrentAstrometry, Multiframe & MultiframeDetector. All 3 of these tables need to be jointly queried to retrieve the relevant WCS, file paths etc. Developed a servlet that utilizes a connection pool, queries the laptop's DB, trawls the resultset, finds the best file to use (based on user suppled RA & Dec & the DB extracted WCS info), extracts a FITS subimage coverts the FITS.gz to jpeg and returns links to the files. The FITS extraction and jpeg creation on done using non-Java code (Java runtime excutions), should be revisited and some point but cfitsio is too useful (no Java version). The DB tables/query should be optimised to quicky find all possible files and minmise the number that have to be checked via the WCS info. Might require indexing of some new parameters in the CurrentAstrometry table eg min and maxDec. NCH challenged MAR to make the changes to the image table schema that he feels are necessary, and then ETWS to add in methods in the ingest code to cope with these new attributes. ACTION: MAR to update the image astrometry tables in the WSA ACTION: ETWS to add in required ingest functionality for MAR's changes and check to see if the ingest code can cope with this schema evolution. ETWS reported: I've started to install mxODBC on ericht for tests with the wsa on toshnt71. FreeTDS and unixODBC are now running. mxODBC needed a bit more tweaking and in the end Python needed to be recompiled with Unicode enabled explicitly. Now I'm getting a long list of nonsupported types but the test program finishes cleanly. Post-meeting, ETWS reported very good news on the Python DB interfacing front: The test program I ran yesterday was suspected to give the mentioned output. Running a real query in Python works fine! Now I will directly start an installation on djoser. Seems that it were really the FreeTDS and unixODBC bugfixes. And the way Python is compiled. NCH reported little coding progress owing to attendance of SPIE; working with JNTD & Eclipse on amenhotep and sorting out the corrupted SSA. A small amount of new code has been checked into the CVS; yet another bug in PyMSSQL has been identified and fixed; some niggly problems with the WSA server file shares have prevented extensive testing of existing code (an IT-support helpdesk ticket has been reopened again to address this one). SSA: NCH noted that the SSA large tables are now fixed up, backed up (a second time) and will be ready to be reactivated at lunch time. If the same corruption reoccurs at some point in the future, the downtime should be limited to the one day period required to restore the full backup (provided there are no other unforeseen problems like site-wide power cuts etc). Astrogrid deployment: MCH has been liasing with HME over installation of a separate PC to run tomcat on the WSA network. NCH noted that HME appears to be attaching a new PC (ex-Wiglaf) for the Astrogrid deployment. In the meantime, NCH asked ETWS to provide JDT and MCH with user accounts for the WFAU CVS repository so that they can check out the archive schemas and create the necessary registry XML documents for the deployment. ACTION: ETWS to set up CVS logins for MCH and JDT Miscellaneous: PMW pointed out the recent announcement of an LSST (Large Synoptic Survey Telescope) workshop meeting in Washington in September 20-22, suggesting a WFAU "presence" maybe a good thing. RGM and NCH countered this with the view that if UK representatives of UKIDSS/VDFS/AstroGrid etc were going anyway, and if there was nothing to be gained from it other than describing what we're doing, then there was little benefit from attending. NCH suggested the group thinks more about the cost/benefits of attending this and other similar meetings.