by Jacob Beasley, Director of DevOps
We recently deployed Moodle in Azure Cloud using Kubernetes. Our client deploys a nationwide network of field service technicians who support a variety of technical products. They use Moodle to internally certify their technicians (some employees and some contractors) on best practices around maintaining, repairing and installing various internet of things devices as well as alarm and point of sale devices.
In this article, we give you an understanding of the challenges we faced as well as the technologies we chose to deploy and scale Moodle for them.
Moodle is a learning platform designed to provide educators, administrators and learners with a single robust, secure and integrated system to create personalised learning environments. It is free open source software built on a traditional LAMP (Linux, Apache, MySQL, PHP) stack, with PostgreSQL as an enterprise option.
The Challenges
Our client wanted something simple and easy to scale as their workloads grew. We will cover the challenges specific to our client as well as the challenges in general of deploying and scaling Moodle.
Maintaining one or many instances of Moodle comes with some technical challenges, especially when running on bare metal or virtualized infrastructure. Some of the most common challenges include:
Scalability and Availability
Many Moodle instances experience great variations in load depending on the time of day or year due to the nature of what it is used for in schools and corporations. Our client would experience a major influx when doing major rollouts across the country in new regions. The ability to scale to meet demand, and scale back down to save on costs is a challenge, and one that costs many organizations a lot of money and/or downtime. Also, in this day and age, students want to be able to login and study or take tests at any time, so it is important the system is highly available and fault tolerant.
Storage and Retention
Many Moodle instances have a great deal of powerpoints, videos, file uploads (homework) and other course materials available for students to use as they use the learning management system. Files need to be securely stored, retained, and scaled as the number of participating students grows.
Extensibility
One of the main strengths of the Moodle open source ecosystem is the plethora of plugins, content, and integrations available. Implementing them is an ongoing process, similar to a software development process. Both require ongoing deployments of new code and configuration which can cause downtime or complicated rollbacks.
Many environments have plugins and themes that have been uploaded as part of the setup by the site administrator. These need to be retained between deployments and possibly updated or installed on demand through the Moodle user interface. At times, custom plugins may need to be written and deployed for a Moodle environment.
Compliance
Many Moodle environments are controlled by regulatory or contractual obligations. Securing traffic from browser to database is essential in meeting many organizations’ obligations.
Internationalization can also add another layer of complexity to compliance. User data is protected differently in different regions of the world, GDRP for example. All content potentially needs to be localized in different languages, or more importantly sensitive, protected user data must remain in its region of origin.
Integrations
Moodle instances many times have several custom integrations with 3rd party vendors and other organizations. Safely deploying, securing, and maintaining 3rd party integrations can be very time consuming, and also add additional risk of expensive downtime.
These integrations often come in the form of APIs or Scheduled Jobs that synchronize data with other software systems.
Migration Strategy
While a brief period of downtime is tolerable, we do not want an extended period of downtime and we want assurances that the new hardware is capable of supporting the load that we had supported previously. Also, terabytes of file storage may need to be migrated, so this will have to be taken into account.
Chosen Cloud Native Technologies
While you could install the software on a virtual machine from a prebuilt Moodle image, that is really not ideal because it is highly available. We chose the following technologies in order to enable a highly available deployment on Microsoft Azure’s Cloud.
Kubernetes
Kubernetes is an open source system for automating deployments, scaling, and managing applications. Kubernetes uses declarative configuration to easily automate scaling of secure, compliant software. It also provides a platform for other software vendors to extend and build on top of. This is very powerful because technologies like Istio, as we will learn, are able to extend Kubernetes to layer security and compliance on top of Kubernetes.
We chose to deploy our instance of Moodle on Kubernetes because it made it easy for us to deploy the backend database, version and manage changes, manage and schedule integration jobs, as well as our scale and deploy our application in a scalable, compliant fashion.
Also, Microsoft Azure Kubernetes Service (AKS) makes it easy to enable automatic scaling to reduce our costs as well as spin up a Kubernetes cluster in minutes manually or automatically using Terraform. It is worth noting that Google Cloud and Amazon Web Services both offer similar products.
Istio
Istio automatically secures your services through managed authentication, authorization, and encryption of communication between. Utilizing Istio we are able to secure all communication between Moodle, its datastores, and 3rd party integrations. It also provides us with insights and dashboards for Grafana to manage and trigger alerting.
Azure File Storage
Microsoft offers a redundant, scalable, fault tolerant storage solution that can be easily configured and mounted as a local directory inside of your virtual machine or Kubernetes cluster. This will be used to store videos, file uploads, and whatever else we need to store securely to support Moodle. It supports geo-redundancy as well as onsite and offsite backups out-of-box with just a click of a button. It is worth noting that Google Cloud and Amazon Web Services both offer similar products.
Azure Pipelines in Devops
Azure Pipelines is a cloud service that you can use to automatically build and test your code project and make it available to other users. It works with just about any language or project type. Azure Pipelines combines continuous integration (CI) and continuous delivery (CD) to constantly and consistently test and build your code and ship it to any target.
Plan for Addressing Each Challenge
Scalability and Availability
Many instances of Moodle are subject to very elastic loads, meaning the load on the system can greatly vary depending on the time of day or year. We were able to utilize a combination of node autoscaling and Kubernetes HorizontalPodAutoscalers to automatically scale up our compute resources while sustaining high load, and automatically scale back down when the load decreased. This provided a perfect balance between system uptime, resilience, and overall hosting costs.
Storage and Retention
Azure Kubernetes Service (AKS) provides Kubernetes StorageClasses out of the box, making it very quick and easy for us to deploy PersistantVolumeClaims that are backed by Azure File Storage. The Volume mounts directly into Moodle’s file system transparently, allowing to use existing Moodle configuration to seamlessly migrate and manage Moodle’s various persistent storage needs.
We chose to run MySQL on Kubernetes with its data written to Azure Disk Storage (similar to Azure File Storage, but faster). For more information about this, see our article on running databases in Kubernetes.
Extensibility
Utilizing automated Azure Pipelines we were able to quickly and reliably deploy new versions of Moodle, its plugins, content, or 3rd party integrations. Kubernetes allows for quickly standing up new environments, so we can test our changes in lower environments without live traffic before promoting to our Production environments.
In order to persist plugins, we store and mount then from a Azure Disk Storage. In this particular case, we did not write any of our own Moodle Plugins, but if that had been the case, we might include them without Moodle deployment instead of managing them in Azure Disk Storage. It would ultimately depend on how often they were changing and what the administrators of the Moodle sites preferred.
Compliance
Many Moodle environments require protection of personally identifiable information (PII), or are held to higher compliance standards such as SOC2 and HIPAA. Using Istio we were able to ensure all traffic is encrypted end to end with mutual TLS for authentication and authorization. Istio works by injecting a sidecar proxy into the running Moodle pod, which allows us to terminate TLS and gain valuable metrics for monitoring and alerting with no code or configuration changes to Moodle itself.
Internationalization of many Moodle instances require regionalized data stores, retention/disaster recovery policies, and localization of user facing content. Most importantly personally identifiable information (PII) must only be stored in the region of origin. Despite the number of clusters and/or cloud providers within the same region all datastores will be localized with regional specific compliance and network policy to ensure 100% segregation and full localized compliance.
Integrations
Kubernetes lends itself well to the microservices architectural patterns. For 3rd party integrations we are able to quickly and reliably stand up microservices that encompass the 3rd party integration without having to make code changes to Moodle using Moodle’s APIs. All communications between integration services and Moodle are secured by Istio’s mutual TLS.
Many 3rd party integrations consist of integration jobs running on a schedule. We are able to utilize Kubernetes CronJobs to reliably schedule integration jobs to run on a schedule, while having visibility into failures that we can monitor with Prometheus and route alerts with Alert Manager.
Rollout Strategy
We will rollout in several phases to reduce downtime:
- Setup Environment
- Setup MySQL in our Kubernetes Cluster
- Setup Moodle in our Kubernetes Cluster
- Test and Validate Moodle works properly
- Prepare for Migration
- Migrate files from previous file storage to Azure File Storage
- Migrate MySQL Database from previous MySQL database
- Test and validate everything is working properly
- Lower DNS TTL to 5 minutes so that it is not cached for long
- Generate SSL certs using DNS entry and Certificate Manager
- Schedule Migration
- Scheduled a brief maintenance window
- Put previous version in maintenance mode
- Perform Migration
- Re-migrate SQL database as well as new files added since step-4
- Test and validate it is working locally with hosts entry.
- Update DNS to point on new cluster
- Test and validate. Wait 10 minutes for DNS to propagate globally.
- Declare victory! We have now migrated successfully to the Cloud with just a brief maintenance window.
Note: We could have shifted all traffic to the Istio load balancers earlier on in order to avoid any delays with DNS propagation, but we chose not to do so in this case for the sake of time.
Monitoring
We chose to use Prometheus to scrape metrics from Moodle. Grafana is used to visualize the performance metrics and Alert Manager monitors and alerts us via phone calls, text messages or emails to any issues if they arise.
Conclusion
Moodle has been a great learning management system for our client and, with the help of Azure and Kubernetes, we have been able to successfully deliver a scalable, secure and highly available courseware solution for training their rapidly expanding nationwide network of field service technicians.
The nature of Moodle’s workload makes it an excellent fit for running in the cloud. We were able to successfully migrate Moodle to Kubernetes with fairly minimal downtime without an enormous amount of effort, and best of all, we can be assured it can handle load as it can be automatically scaled at will.
Are you ready to enhance your DevOps capabilities?
If so, reach out to our team! We deliver a full range of DevOps services and solutions that help propel transformative change for our customers. Our range of capabilities includes:
- Software Development
- Site Reliability Engineering
- Build and Support Services
- Secure Software and Systems
- Respond and Support Services
- Compliance and Security Capabilities
If there’s software that needs to be built, secured, or improved, our team can help. Feel free to reach out to us at connectwithu[email protected] or by filling out our brief Contact Us inquiry form if you’re interested in learning more about our DevOps services. We’re confident we can help you build reliable, secure, and powerful software that can truly move the needle for your organization and your customers.