Latest News
First 'All-hands' Meeting
On October 18th, the first All-hands Meeting for the NSF RII award took place in Baton Rouge. As members of the RII Advisory Board, Ron Hutchins, Jordan Konisky, and Vince McCoy were present. Two of the LONI Institute fellows attended the meeting as well. You can download presentations of Drs. Allen, Cortez, Gaver, Khonsari, and Seidel. Here are pictures and a few more details on the meeting. [Other News]
Description of Work Package 1
The efficient management, retrieval, and mining of distributed, heterogeneous data is a daunting problem, critical to many applications, including those with experimental, observational, and simulation components. The explosion of data from these projects provides a unique opportunity for multi-institutional, large-scale, collaborative knowledge discovery. The complexity and heterogeneity of distributed resources including file systems, compute servers, and networks require a solid underlying layer of information and scheduling services. This workpackage will support and develop enabling applications for data and scheduling services.
Infrastructure Deployment
The software to be built in work package 1 will rely on a large amount of cyberinfrastructure software that must first be deployed on LONI. Some of this is commercial or open source software that exists on many other grid systems, and some of it is being developed by the Cybertools collaborators in other projects. Examples include: a global file system spanning all LONI institutions (e.g. GPFS/Lustre), core Grid software (e.g., Globus Toolkit, Condor, GSI, etc), and extended Grid software (e.g. GRMS, SAGA, SPRUCE, Stork, HARC, etc).
Data Archival and Retrieval Services.
CyberTools will develop a LONI-wide distributed data archive for all LONI projects including simulation, experimental, image, and observation. We will also develop services to automatically archive data directly from simulations and experiments, and to efficiently and reliably retrieve the archived data. PetaShare will provide management of low-level data handling issues, enabling scientists to focus on their research problems. Key technologies to be developed include data-aware storage systems and schedulers to transparently manage data resources and scheduling data tasks for the scientist. The Stork data scheduler, providing user level abstraction for the underlying data transfer protocol, will be further developed to allow on-demand queuing, scheduling, and optimization of data placement jobs.
Scheduling Services.
Enhanced scheduling services will be developed to support automated, on-demand simulations needed, e.g., for hurricane/surge forecasting, or for co-allocated or coupled jobs, e.g. for simultaneous CFD/MD simulations or coupled simulation-visualization processes. We will further develop the HARC Co-allocator, which can already allocate reservations on compute resources and optical networks. SPRUCE will be used to ensure the timely execution of high-priority work. A generic task farm manager will be developed, providing a framework for task farming ensembles of scientific applications through both simple interactive interfaces, and in a fully-automated responsive mode. Higher-level resource discovery and brokering services based on these basic scheduling services are already being planned, as part of the LONI Task Farm Manager Architecture; these services are needed to automate resource selection enabling application scenarios important for CyberTools. We will build on existing tools such as GRMS, the IBM metascheduler, and Platform's Community Scheduler Framework.
High Availability.
Uninterrupted availability must be ensured across this distributed HPC environment for mission-critical applications and services. LA Tech's High Availability Open Source Cluster Application Resource (HA-OSCAR) will be extended to address reliability, availability, and serviceability issues by automatically handling software and hardware faults. HA-OSCAR adopts component redundancy to eliminate single point failures. We will incorporate novel mechanisms such as a self-healing, automatic failover, as well as failure detection and recovery to increase system availability. We will extend the monitoring core and failure recovery mechanisms to provide service request queue migration, a sophisticated rule-based recovery routine, and integration with other important grid services such as Globus, OGSA-DAI, and Gridsib for 7/24 operation.
Metadata Extraction and Indexing.
The astounding volume and complexity of the scientific data generated by these and other LONI projects require not only effective storage and retrieval mechanisms, but metadata descriptions and data mining algorithms. LSU and LA Tech researchers will work with colleagues in Manchester and AEI (Potsdam), active in the use of ontologies to develop metadata services so that archived simulation, experimental, and observational data can be searched, discovered, and retrieved for analysis and comparison. We will further develop novel data mining-based semantic discovery approaches that exploit isomorphisms discovered among information feature vectors of the data. The metadata descriptors will be cross-validated with domain knowledge prior to indexing. Isomorphic relations will be discovered among metadata descriptors across multiple classes of data to enable integrated hypothesis discovery and refinement. Novel multidimensional spatial indexes will be employed for fast retrieval of data using joins on metadata features. We will also develop an autonomous multi-class classification system for domain-specific data mining applications with enhanced specificity and sensitivity for both direct and information-based queries on these databases.
[ Back to top ]