Web Application for Monitoring

Large-Scale Virtualized Computing Resources

Robert Poenaru

DevOps @ Harman International

Research Assistant @ IFIN-HH

Table of Contents

• Research at IFIN-HH
• Aim & Motivation
• App Development Workflow
• Results
• Conclusions & Future Plans

Short Bio

• PhD student at the Doctoral School of Physics (UB)
  • Topic: #nuclearStructure, #nuclearDeformation, #triaxialNuclei
  • Thesis: A Systematic Description of the Wobbling Motion in Odd-Mass Nuclei Within a Semi-Classical Formalism
• Research Assistant at IFIN-HH, member of DFT and DFCTI
  • ​Topics: #ELK, #webApps, #webServers
• DevOps engineer at Harman:
  • Topics: #ELK, #CI/CD, #SCA

Research @ IFIN-HH

• Scientist at IFIN-HH tackle complex problems on a daily basis

• There are five main research departments, which cover broad areas of physics:
  • Theoretical Physics
  • Nuclear Physics
  • Computational Physics
  • Solid-State Physics
  • ...

Research @ IFIN-HH

• Finding solutions to the research problems implies numerical computation
  • Specific algorithms (i.e., FORTRAN, C++, Python)
  • Complex simulations
  • Machine Learning tools 
• All these require large-scale computing resources that can execute the code simply, reliable, and fast.

fast

reliable

simple

Research @ IFIN-HH

•  Large-scale computing infrastructures present several issues (or “challenges”) for the scientists:
  • deployment (security, software…)
  • maintenance
  • scalability
  • ...
• Will create an overhead for the scientists, if they need to handle them

slow the progress towards actual

research tasks

Motivation

Proposed solution

• Provide to all the research departments a cloud infrastructure (i.e., Cloudifin) that can be used to execute simulations on the available compute resources

Aim

1. A user (or more) require performing multiple calculations.
2. Depending on the complexity of the jobs, the administration team will deploy virtualized resources and will grant access to the user(s).
3. Furthermore, the user can run the jobs on the machines
4. Monitoring possible via the web app at a later time, while the machines are working.

The workflow

Solution

• The computing resources are deployed by the administration team on the bare metal using OpenStack
• At least one controller node and one compute required for deploying virtualized compute instances (agents)

Basic OpenStack architecture

Solution

• VMs are created by the admins whenever user requests compute resources
• OpenStack CLI allows for creating and deployed virtual machines:
  • templates (i.e., [8 cores, 16 GB RAM, 0.5 TB]…)
  • standard tiers
  • custom OS flavors (Ubuntu, Rocky Linux)
  • pre-installed scientific tools (e.g., Root, Python ML tools, SIESTA)
• VMs are managed by a controller 

• Create a Python service that runs on the web and the user can access it in order to see all available virtualized machines
• Flask is used to deploy the web app (currently running on a test machine within the department)
  • socketIO used for processing requests and responses
  • HTML+CSS+JavaScript used for designing the UI
  • mqtt - protocol used for running commands on the controller nodes and the agents (VMs)

Designing the web app

• Available resources must be displayed as a topological view
• Resources are retrieved from a database 
• Database is generated while loading the webpage
• Communication is done through the Flask server

Designing the web app

mqtt

Database

Data is managed within the Python web app through sqlite

Landing page

• Dropdown list with all the types of nodes available to the user
• Example is for a user with admin rights 

Topological view 

• Type of node
• Node name 
• Status

ℹ️ each box is a compute node

• Using python's psutil module, a user can check system statistics for a running machine
• Data is represented graphically using matplotlib
• Current implementation:  only for one specific node

🛠Work in progress

• Histogram view with all the available nodes deployed for the user
• Allows for a quick overview based on the type of node 
• Useful for admins

Full topology view

Summary

• Came up with a simple solution for a user interface that:
  •  allows monitoring of computing resources deployed via OpenStack
  • Check system statistics for all the available compute nodes (current implementation only allows one node)
  • See nodes on a histogram view, categorized on the type of node
• Deployed the service as a web app, for ease-of-access

Future outlook

• Extend the system statistics to show information about all the nodes instead of only one
• Make the database with the computing resources refresh dynamically instead of page reload
  • same for system statistics
• Implement the management of containerized apps (i.e., Docker)
  • similar topological view with the available images/containers

Thank you for your attention!!!