From nch@roe.ac.uk Sun Jun 3 09:38:55 2007 Date: Fri, 1 Jun 2007 16:47:38 +0100 (BST) From: Nigel Hambly To: WFCAM Science Archive Team -- Eckhard Sutorius , Johann Bryant , Mike Read , Nigel Hambly , Nicholas Cross , Bob Mann , Ross Collins Cc: CCs for WSA weekly meeting minutes distribution -- Andrew Lawrence , Andy Adamson , Brian Walshe , John Taylor , Jim Emerson , Malcolm Stewart , Lorenzo Rimoldini , Mike Irwin , Mark Holliman , Peredur Williams , Stephen Warren Subject: WFAU VDFS science archive weekly project meeting minutes, 1 Jun 07 Minutes of WFAU VDFS Science Archive meeting: 1st June 2007 ------------------------------------------------------------------------- ------------------------------------------------------------------------- Present: NCH, RSC, JB, LGR, NJC, MSH, PMW Apologies: JDT, BCW, JPE, AL, RGM, JMS, ETWS, MAR DONM: 10am, Friday 8th June 2007 in the Plate Library Actions discharged this week: ----------------------------- ACTION: NCH to remember to ensure the GPS schema is modified as per instructions from the GPS survey head, and Cookbook examples included to recover attributes when needed. Discharged (thanks to NJC!) Actions partly discharged but continuing: ----------------------------------------- The following from last time partly done but continue: o) Add in a default row for every detector appearing in every detection table (for schema consistency when querying merged sources and individual detections) - ACTION: RSC & NCH Continues; will get sorted (honestly!) as part of the general ingest DB schema revamp to facilitate rapid generalised photometric/astrometric recalibration. Actions carried forward from 18/05/07 meeting: ---------------------------------------------- The following from last time continue: ACTION: NCH to discuss and review AstroWISE interfacing with MAR and JDT. ACTION: NCH to start some hard-nosed negotiations with the UKIDSS PI and CSS concerning the contents and timing of DR3. - CONTINUES; remaining problems with ZY NDR data from early 06B should be sorted quite soon now, so the schedule should become easier to tie down. ACTION: AL to check on the exact wording of the survey release policy with reference to world release of proprietary survey data. - CONTINUES Specific points and new actions: -------------------------------- Project management: NCH reported writing the WFAU Rolling Grant cases for VDFS development and operations; thanks to all who have read these and contributed. PMW noted that it is time once again to fill in progress charts for the quarter and prepare the usual materials for next Friday's VDMT. WFCAM & VISTA updates: VISTA M2 has arrived: JPE notes: "http://www.vista.ac.uk/mirror_images.html shows a picture (which you can click on to enlarge) of the secondary mirror approaching VISTA. The picture also gives a nice impression of the position of VISTA relative to the VLTs." Comments and issues arising from CASU fortnightly minutes: No new minutes this week. Networking: JB noted that UKLight is definitely back online and though there is an upgrade due next Wednesday evening this shouldn't affect us. WSA Operations: JB reported: "Operations continue as usual, ETWS handed the new system over to me last week and since then I have been running CU1, CU2 and CU3 on the data as it comes in; CU1 is currently about one week of data away from completing download of all the data CASU have currently flagged as ready to copy; CU2 is approximately three weeks behind CU1; CU3 is about a week and a half behind CU1. CU4 is on hold until after the Recalibration work has been completed. Backups and Ingest continue as usual. Ahmose needed a reboot two weeks ago today due to running out of network connections (old problem)." NCH noted that WSA transfer/ingest monitoring (analogous to the CASU processing status pages) are now available: http://surveys.roe.ac.uk/wsa/monitoring/html/WSAMonitor_2006B.html These automatically update daily. Thanks to MAR and ETWS in particular for sorting this out. Hardware: JB noted: "Thutmose had a failure that looks like a backplane problem (though it seems to have been triggered by a failed disk), work is ongoing to fix this. There was also a powercut yesterday that missed the servers (due to the server room UPS) but still managed to hit CU1 (upstream network got hit) and CU3 (because I hadn't properly nohup'ed the terminal it was running in - doh!) Both were up and running again within the hour (once I was fairly sure the power wasn't going to go again and with CU3 properly nohup'ed!) I spent some time dealing with the issue of partition sizes, to summerise it appears that: with our 2.4 kernel on 32-bit machines we are limited to 5TB partitions; with a 2.6 kernel on these same machines we would be limited to 16TB disk partitions due to the filesystems; with 2.6 kernel on 64-bit machines (since 2.4 on a 64-bit machine is unlikely at this stage) we would be limited to 8EB (ExaBytes) but for the Raid controllers (usually more than 256TB) and the file system (16TB)." Software: MAR has implemented the monitoring pages in the user interface, and has produced a rather beautiful mosaic of the UKIDSS GPS: "The following URL points to a mosaic of 2000-3000 detectors in the GPS centred around b=0, l=70 or so aligned with the galactic coord system. Resolution is 2 arcsec per pixel. http://surveys.roe.ac.uk/data/mar/gps/gps.html Generally I was pleasently suprised by the uniformity and colours, though a couple of tiles look strange over on the right. There are 1 billion pixels in the image though a lot are black :)" As regards publicity images, the team felt that we need to assemble more publicity material perhaps as part of the WFAU web page for the IfA (e.g. we should include all of NJC's recent publicity images as well). ACTION: NCH to discuss publicity image gallery with MAR. Note that RSC has been maintaining a proto-gallery at http://apache.roe.ac.uk/twiki/bin/view/WFAU/WfcamImages RSC reported: "Solved the problem of dodgy default values for compFile in MultiframeDetector with a combination of a bug fix to the schema and update to exmeta to put in the correct default now that CU2 updates this value. (see trac ticket 63) Updated exmeta to correctly handle a change to the default value of the TAUDATE FITS keyword. (see trac ticket 62) Tested bug fix in the latest PyFITS to solve the problem of keyboard interrupts causing file corruption whilst pyfits.flush() is active. No data corruption occurs anymore so we're free to perform keyboard interrupts when running CU8 now. (see trac ticket 4) Put the finishing touches to the DB API that will be useful for the new recalibration code as well as else where. Mostly centred around the ability to easily drop and recreate any database object including constraints whilst being schema driven. (see trac ticket 64) Removed obsolete methods in the code base prior to the Python 2.5 upgrade, and software development branch to work with the new recalibration induced schema changes. Merged Python ProgCuSession class into the CuSession class, as distinction not very useful anymore. (see trac ticket 26)" NJC reported: "I have edited template schemas for all the surveys, to include changes for our recalibration and for the slimmed down gpsSource. I have also added in the calibration and wts campaign survey, although I am still waiting to hear back from David Pinfield and Simon Hodgkin about how they would like me to proceed - combining the U/06A/52 and U/CMP/2 data together or keeping them separate. I have also been working on updating code that contains PYFITS and numarray to PYFITS and numpy, but have run into several problems. I will try and solve these shortly." The team discussed the old/new transit survey options, and NCH expressed the preference that two separate programmes should be kept and if the PI requires it, old data can be assigned to the new programme manually. JB reported fixing a couple of minor things in the DataDaemon while learning how it worked, also getting aquainted with the progress calender setup. NCH reported an intensive proof-of-concepts study of the partitioned detection table schema design for recalibration purposes, and noted that although there were one or two implementation wrinkles (caused mainly by quirks in MS SQL Server) there were no show stoppers and the new schema design should work well. An extensively annotated SQL script has been uploaded to the TWiki recalibration topic as an aid to implementation. Survey Data Release: Nothing to report this week. Non-survey Data Release: JB noted that the two remaining non-surveys to be released from the recent batch that were held back by a bug have now been released. Astrogrid deployment: Nothing new to report this week Miscellaneous: Finally NCH noted that there is an "eDIKT" technical workshop on June 13th pm down in the Rutherford Building in KB. Part of this concerns data management and experiences of various groups within the University - could be interesting. Relevant information: http://www.edikt.org. DiGS: http://forge.nesc.ac.uk/projects/qcdgrid/ http://www.edikt.org/edikt2/DataGridForCellBiology ECDF: http://www.is.ed.ac.uk/ECDF Daniel Rutherford Building: http://www.ed.ac.uk/maps/kings-buildings/daniel-rutherford-building/