From nch@roe.ac.uk Sun Mar 5 21:56:50 2006 Date: Fri, 3 Mar 2006 13:47:28 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly meeting minutes, 3rd March 2006 Minutes of WFCAM Science Archive meeting: 3rd March 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, PMW, RGM, JB, NJC, MSH Apologies: JPE, JMS, AL, JDT, MAR, RSC DONM: 10am, Friday 10th March 2006, plate library Actions discharged: ------------------- ACTION: PMW to start the ball rolling on getting quotes for a new storage server with an expanded spec over khufu/djedefre. Discharged; quotes being circulated amongst interested parties ACTION: RGM to seek out MSDNAA SQL 2005 CDs. Discharged; beta version only has been distributed so far; online downloads have limited functionality so we wait for the official distribution of the full release via the next quarterly MSDNAA CD set. NCH noted that this was not a particularly high priority at the moment anyway. Actions partly discharged but continuing: ----------------------------------------- ACTION: RGM to look into the possibility of external funds for an SDSS catalogue server. Continues; AL has been emailed. Actions carried forward from 24/02/06 meeting: ---------------------------------------------- ACTION: MAR to update the IfA WFAU web pages. - CONTINUES ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: ETWS to go thru the CUs one by one and note any issues as regards performance/scalability. - CONTINUES ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH welcomed Mark Holliman (MSH) who has joined the IfA VOTech team as IT support, and who will attend WFAU WSA weekly meetings to get familiar with goings-on concerning archiving. MSH expressed an interest in the archive systems, and NCH suggested he talked to JB to find out what equipment is being used, and the set-ups thereof. WFCAM update: No news this week. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 3/3/06 AM. Networking: NCH noted a couple of points of interest regarding SDSS DRs: the sizes noted in last week minutes were a little off the mark in the following cases: DR3 is 1.5 TB while DR4 is likely around 2.8TB. Bob Nichol from Portsmouth visited the IfA this week, and apparently they have been copying DR4 from Chicago, so this may be an alternative for WFAU to obtain a copy if this proves necessary. As regards DR3, JB noted that a certain amount of DB shuffling is needed to put it online asap. JB and ETWS reported back from the UKLight meeting this week at NeSC. Useful discussions with George Howatt indicate that things should progress with reasonable rapidity on this; JB is keeping on top of the hardware and networking issues involved. NCH noted that this was all good stuff, given necessity for us to support our colleagues in the School as regards e-Science innovations etc. by becoming UKLight adopters. Besides, it will be most useful for Vista data transfers. ETWS volunteered to put a TWiki note up concerning the outcome of the meeting. ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. For the record, JB reported: Wednesday ETWS and I went to the NeSC UKLight meeting in Edinburgh and got a better overview of how UKLight works and how it is progressing, Saleem Bhatti (St. Andrews' University Computer Science department) in particular had been doing protocol testing over UKLight and would be someone who may be able to help us make the most of our connection if we've the time to play with it a bit (needless to say this will require good co-ordination with CASU - as will the whole setting up of a UKLight connection). Following the NeSC UKLight meeting I contacted George Howat with regard to contacting his man down in Cambridge and pointing him in the direction of Peter Bunclark. He also offered to get some estimates on hardware costs for us (which I took him up on). I CC'd JNTD to keep local IT support in the loop. WSA Operations: JB reported: The System backups have been done as normal except that the UserInterface databases on both Ahmose and Amenhotep are also now backed up along with the system backups for Djedefre. CU1, CU3 and CU4 have been running most of the week and after getting terrible transfer rates (1.8MB/s) for a few days we have gotten the TCP windows on Djedefre changed, comparison tests show that this makes the transfer speeds more reliable and can approximately double them... the last run last night clocked in at 13.5MB/s (as opposed to a (less reliable) best of 6.1MB/s with the old tcp window settings). JB and ETWS noted that space is again becoming tight on the storage servers. NCH suggested a spring clean: ACTION: ALL to remove as much personal scratch files from the mass storage servers ACTION: JB, NCH & ETWS to scrub old V1 processing of 05A data, and remember to update the flat-file links in and released DBs that point to the same. NCH expressed a little nervousness about ingest DB (i.e. WSA on ahmose) backups. Some discussion ensued; JB suggested continuing on an ad-hoc basis, detaching the DB every few weeks to make a tape backup overnight. All agreed that this is fine as long as we remember to do it. ACTION: JB to make an overnight WSA tape backup early next week Hardware: PMW noted that the new catalogue server "thutmose" has been ordered. Software: NCH asked about the enhanced browser, since a user had enquired about the mismatch between the one available browser and the multifarious DBs that are online. ETWS pointed out that he had completed the changes sometime ago; there was uncertainty as to why this was not online. ACTION: NCH to ask MAR to put the enhanced browser online on his return from hols. NCH further noted some user confusion over the business of aperture corrections. The documentation is clearly inadequate, hence NCH will make some suggested changes, then these can be propagated into all the other static schema files where necessary. ACTION: NCH review aperture mag documentation in the LAS schema. ETWS reported: "Finished installation of software on djoser, now all machines run the same instances of gcc, Python, etc. Visited the UKLight workshop, there will be some notes on the TWiki soon. Updated the sql schemas and CUs where necessary." NJC reported: " I have done some tests on the extinction code and have determined the bottle-neck, although I still need to sort this out. I have updated the FILTER table schema and Initiate Archive schema with A/E(B-V) values, although I haven't committed these yet. I have finished revising CU13 for the stacking and tested this thoroughly. I have added options to include/exclude previous deep stacks and science verification data. I have been trying to pin down a memory problem with the mosaicing part of CU13 code, but have been unable to so far. I will try to finish this soon and finish CU14." RSC has been absent this week due to illness (the curse of the pharoahs strikes again...) Survey Data Release: NCH reported that 05B QC1 continues, in collaboration with SJW. Non-Survey Data Release: JB noted further 05B non-survey data set-ups and ingest. Astrogrid deployment: Nothing new this week. Miscellaneous: Nothing else this week.