From nch@roe.ac.uk Fri Mar 10 15:19:24 2006 Date: Fri, 10 Mar 2006 12:12:48 +0000 (GMT) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Mark Holliman , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , John Taylor , Jim Emerson , Malcolm Stewart , Martin Hill , Mike Irwin , Peredur Williams , Stephen Warren Subject: WFAU WSA weekly project meeting minutes 10th March 2006 Minutes of WFCAM Science Archive meeting: 10th March 2006 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, ETWS, PMW, JB, NJC, MSH, AL, JMS Apologies: JPE, JDT, MAR, RSC, RGM DONM: 10am, Friday 17th March 2006, plate library Actions discharged: ------------------- ACTION: ETWS to go thru the CUs one by one and note any issues as regards performance/scalability. Discharged; ETWS noted that the only big performance issue is with CU1 (transfer) - see Networking below. ACTION: ALL to remove as much personal scratch files from the mass storage servers Discharged ACTION: JB, NCH & ETWS to scrub old V1 processing of 05A data, and remember to update the flat-file links in and released DBs that point to the same. Discharged ACTION: JB to make an overnight WSA tape backup early next week Discharged Actions partly discharged but continuing: ----------------------------------------- ACTION: RGM to look into the possibility of external funds for an SDSS catalogue server. Continues; AL has been emailed. ACTION: NCH review aperture mag documentation in the LAS schema. Continues; ETWS will now propagate these changes into the other static schema files. Actions carried forward from 3/03/06 meeting: ---------------------------------------------- ACTION: MAR to update the IfA WFAU web pages. - CONTINUES ACTION: JB to review Hardware/OS/DBMS design doc to note areas that need updating and itemise new sections required - CONTINUES ACTION: NCH to investigate the usefulness of a NAS solution for medium term archive mass storage. - CONTINUES ACTION: ETWS and JB to put a TWiki note on the intranet detailing the outcome of the UKLight meeting at NeSC. - CONTINUES ACTION: NCH to ask MAR to put the enhanced browser online on his return from hols. - CONTINUES Specific points and new actions: -------------------------------- Project management: PMW noted that WFAU RG has been (finally) announced, and this covers staff resources and hardware for VDFS operations until 2010. NCH noted that Vista Public Survey proposals are being generated at a furious rate, with much interest in using VDFS, not only in the UK, but in Europe (and even Chile). JMS gave the team an update concerning dates and the schedule from the latest Vista project plan. A VDMT will be scheduled for some time after March 15th (the deadline for Public Survey proposals). WFCAM update: No news this week. Comments and issues arising from CASU fortnightly minutes: No new minutes as of 10/3/06 AM. Networking: ETWS and JB both noted the drop-off in network transfer speeds from CASU (see WSA Operations below). NCH suggested that we push ahead with investigations, and also in parallel push forward on the UKLight deployment. JB noted: "Progress is being made on looking at the possibility of UKLight, I have made contact with George Howat (EUCS/EaStMAN) and have some costings options for the ROE to EaStMAN end of the connection, I have also been in touch with JNTD (ROE IT Support) who reckons it shouldnt take much time at all from a hardware perspective - maybe an hours work if things don't go smoothly. I have also been in contact with Peter Bunclark (CASU) though I need to follow up my initial email and his questions about it with him as well as having heard from David Salmon (UKLight) who sent details of the form to officially get the UKLight bandwidth and (via George Howat) from Chris Cheney (Cambridge University Computing Service) with some details of the network setup and possible connection plans at that end... I believe Chris has also been in contact with Peter Bunclark. WSA Operations: JB reported: "This week Ahmose and Amenhotep were reboot on Monday which fixed the tape drive problem with Ahmose. I used the down time as an opportunity to make a fresh backup of UKIDSSR1 after the recent changes. System backups went as normal and a backup of the Ingest WSA happened overnight on Wednesday (this took about 10 hours for 800GB of data including verification). CU1 is running as normal with connection speeds of 13MB/s over last weekend though most of this week (and at the present) this has averaged 3.5MB/s instead... this may well become a problem for WFCAM as we need to get about 5MB/s as standard (VISTA scalability is non-existant if this continues). CU3 is running smoothly now that the corrupted files caused by the crash of apm7.ast.cam.ac.uk have been redownloaded. CU4 seems to be doing okay though is a little behind CU3. A number of attempts have been made to break the SDSS BestDR2 database into multiple files... these continue as success continues to elude. BestDR3 transfer to Amenhotep is waiting on a successful resolution to this." Hardware: No news this week. Software: NJC reported: " I have been struggling with the memory problems associated with large mosaics. I spent several days removing memory leaks and made good progress on calculating the means quickly. However, changing image data was more problematic since writing out to the file will read the whole image into memory even if you only update a strip. In PyFITS I do not know how to get around this problem. However, I then tried using pyraf, the PYTHON wrapper for IRAF. This worked beautifully, used almost no memory and does a 0.9G file in less than a minute. I should have thought about this a week ago since IRAF was written back in the dark ages when computers had next to nothing memory-wise. Anyway, I have submitted a helpdesk ticket to get IRAF installed on djoser, and then Eckhard will install pyraf and I finish off CU13." NCH asked NJC to have a look at the calibration information currently stored in the WSA to help out in the on-going investigations of the various measurement datasets of the UKIRT primary Faint Standards. Survey Data Release: NCH reported that 05B QC has been on hold this week owing to SJW being unavailable to help; QC will resume next week, the plan being to finish Aug/Sep for the wide-area shallow surveys in order to feedback any issues to UKIDSS and upstream in the VDFS. Non-Survey Data Release: NCH noted that a local user (Annette Ferguson) has two WFCAM projects spread over 05A and 05B and is now ready to look at the data. Astrogrid deployment: JDT has emailed to say that Guy Rixon is available to visit the Unit to demonstrate the latest security features of DSA/PAL. NCH is away the week after next for 2.5 weeks, so MSH noted that Astrogrid deployment server "heilan" is to be reconfigured on Tuesday next week, this may (?) affect DSA/PAL access to WFAU archives. Miscellaneous: NCH noted the VO Special Session at the IAU in Prague, 17-18 and 21-22 August. AL and RGM will be attending anyway, but NCH suggested that we should have further representation at this and asked for volunteer(s). The web site is http://www.astronomy2006.com/special-sessions.php#sps3