QUESTIONS – Uintah Answers 1. What are the target applications and what is the expected sophistication of the target users (naive, knowledgeable, expert)? The Uintah Computational Framework consists of a set of parallel software components and libraries that facilitate the parallel solution of Partial Differential Equations (PDEs) on Structured AMR grids. Uintah presently contains four main simulation components: 1.) the multi-material ICE code for both low and high-speed compressible flows; 2.) the multi-material, particle-based code MPM for structural mechanics; 3.) the combined fluid-structure interaction (FSI) algorithm MPM-ICE and 4.) the ARCHES turbulent reacting CFD component that was designed for simulating turbulent reacting flows with participating media radiation. Uintah is applicable to a wide range of engineering domains that require fluid-structure interactions and highly deformable models. The expected sophistication of our target users would be domain scientists that are knowledgeable to expert in their respective field, but would be naïve WRT to parallelism. One of the primary strengths of Uintah is that application designers can develop large-scale parallel simulations with little understanding of the underlying parallelism. ------------------------ 2.  List the set of features/concepts of your programming model and divide them into two sets: What is the minimum set of things a new user need's to learn to become productive?  What are the more advanced features a user can potentially take advantage of? Set 1 (a) Understand Simulation Component Class Description and its interface. Each Uintah component can be described as a C++ class that is derived from two other classes: UintahParallelComponent and a SimulationInterface. The new derived class must provide the following virtual methods: problemSetup, scheduleInitialize, scheduleComputeStableTimestep, and scheduleTimeAdvance. 316 page User Guide (b) A new user would also need to be able to think of their problem in terms of task-decomposition, and subsequently need to understand the task interface (c) Uintah data storage concepts – DataWarehouse interface (get-put model) (d) Lastly, a new user would need to understand the input file format. Uintah uses XML like input files to specify the various parameters required by simulation components. These Uintah Problem Specification (.ups) files are validated based on a well documented specification 45 page Developer Guide with numerous examples (e) Uintah provides over 100 fully working example components Set 2 (a) Subschedulers – essentially some iteration within a task (a subcycle in the graph) (b) Infrastructure supported reduction variables (c) Particle System for representing solids in a fluid-structure interaction problem (d) The ability to switch on specialized schedulers, loadbalancers, regridders for specialized functionality. (e) OncePerProcess tasks – e.g. hypre or petsc solve. Uintah offers interfaces for both. (f) GPU task support ------------------------ 3.  How do you decide whether a new application is a good fit?  What metrics do you use to evaluate whether an application is implemented well in your model? New components that solve PDEs on structured grids are ideal candidates for Uintah. Can the application (algorithm) be decomposed into tasks and is the algorithm suitable for a collection of patches on a structured grid. The task granularity is decided by the component developer and typically follows closely with the natural description of a step in an algorithm. ------------------------ 4. What is the plan for interoperability with MPI, OpenMP, Kokkos, etc.?  If you could add requirements to MPI, what would those be? Automated MPI message generation. We have incorporated Kokkos into our data storage representation, and use Kokkos parallel data loop constructs for looping over the elements of our patches. We are investigating adding an OpenMP based scheduler for Kokkos "launch" functionality. In conjunction with the Kokkos based OpenMP back-end, Uintah will be able to interoperate with OpenMP based third party libraries such as Hypre in a manner that insulates the application developer. We would like to see support for message tags in MPI collectives. ------------------------ 5. What is the plan for performance portability? Kokkos ------------------------ 6. What is the plan for fault tolerance? Assuming recovery beyond a simple checkpoint-restart, the multi-threaded Uintah task schedulers naturally handle core failure, and should the effect on performance be severe enough, the loadbalancer (fed by a performance feedback loop) would migrate patches to other nodes to compensate. Node failure becomes more difficult. This hasn’t been handled yet, formally. However, this could be realized by having the infrastructure reconstructing the MPI communicator without the bad node, and going through the regridding process Uintah provides to redistribute patches. WRT a resilience model, work is underway using AMR-based duplication. ------------------------ 7.  What static analyses and transformations could you do?  What do you do today? We do not perform any static analyses or transformations currently. ------------------------ 8.  Questions about task graphs:        * When is the task graph generated (compile-time, load-time, run-time)? At runtime, Uintah performs task dependency analysis and compiles the task graph (compilation phase).        * How do you manage task graph generation vs. task graph execution? Two phases – task graph compilation phase (1) and execution phase (2). Our SimulationController object passes the compiled task graph to the scheduler for execution at the appropriate time.        * What is the value of non-ready tasks in the DAG? The value of non-ready tasks is that as soon as external dependences are met (via MPI), they are ready to execute, providing more options for immediate work.        * Do you exploit the repetitiveness of iterative applications that repeatedly execute the same task graph? Absolutely – reuse the same task graph for each iteration. No overhead, already compiled. Only need to recompile if the grid changes, e.g. grid refinement, AMR. ------------------------ 9.  Questions about tasks:        * How is task granularity managed? By the application code itself. No automated coalescing of tasks within the infrastructure, though MPI messages from related tasks may be combined. Applications may be able to further decompose or combine tasks through experimentation.        * What is the life-cycle of a task? As the Uintah scheduler begins the task execution phase, tasks go through a small series of ready queues. Internally ready tasks (halo requirements met by patches residing on the same node) go immediately into the run queue. Tasks requiring MPI for halo information have an “external dependency count” which is decremented as MPI messages are received. Once this counts goes to 0, these tasks go into the run queue. The run queue is based on priority, and what we’ve used most recently is to prioritize based on the highest number of outgoing messages. We’ve found this to generate the most available work in the shortest time. ------------------------ 10.  What is the relationship between task and data parallelism --- can one be invoked from the other arbitrarily or are there restrictions? For Uintah, a task can become data parallel. No actual nested parallelism within a task beyond this. ------------------------ 11.  Where exactly is concurrency (meaning the ability to have races and deadlocks) exposed to the programmer, if at all?  Data races may present within task code, via inadvertently exposing a shared data structure. We’ve typically worked with applications developers to provide solutions for this via locks or thread-local versions of the contended resource. Using Uintah’s threaded scheduler is the only time applications need to be aware of concurrency (to a limited extent). Deadlocks are usually the result of changes in infrastructure code and in some cases exposed by applications, though regression testing will often find these issues. These situations are rare, but unfortunately part of the landscape. ------------------------ 12.  Legion: What are the plans for people to be able to write mappers more productively? N/A