• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer
Codemotion Magazine

Codemotion Magazine

We code the future. Together

  • Discover
    • Live
    • Tech Communities
    • Hackathons
    • Coding Challenges
    • For Kids
  • Watch
    • Talks
    • Playlists
    • Edu Paths
  • Magazine
    • Backend
    • Frontend
    • AI/ML
    • DevOps
    • Dev Life
    • Soft Skills
    • Infographics
  • Talent
    • Discover Talent
    • Jobs
  • Partners
  • For Companies
  • IT
  • ES
Home » DevOps » Cloud » A journey riding in the cloud(s)
Cloud

A journey riding in the cloud(s)

Codemotion and Facebook organized the Tech Leadership Training boot camp, heres a personal reportage from one of our attendees.

February 21, 2019 by Sergio Monteleone

The cloud computing panorama is a huge and messy collection of technologies, services and providers. Building elastic distributed systems is fun but orientating such a vast environment can be quite challenging.

Talking about some do’s and don’ts in cloud computing, Suman Kumari and Wamika Singh from ThoughtWorks shared their adventure in migrating a monolithic application to the cloud at Codemotion Milan 2018.

ThoughtWorks was awarded by an Italian manufacturing company the task of re-engineering a multi-factory global scale application using an elastic infrastructure, delivering a proof of concept as soon as possible while keeping operational and maintenance costs low. The main objective of the project was to achieve better insight on the production process by collecting, streaming and aggregating data produced at each plant.

Another project requirement was to design a cloud agnostic application, to allow the integration of different cloud providers if needed.
Kumari and Singh described their journey by talking about four main topics: the overall system infrastructure, the data streaming architecture, the data retrieval system they used and the DevOps procedures they adopted.
 

Application Infrastructure

To run and deploy their services, ThoughtWorks went with the well-known and industry acclaimed approach of containerisation. Containers are portable, safe and cost-effective so they quickly became the de-facto standard for cloud applications. In particular, ThoughtWorks decided to host their containers on a Kubernetes cluster to benefit from features such as auto scaling, automated roll-outs and roll-backs, autodiscovery, etc.
The infrastructure was created with Terraform on AWS and the Kubernetes cluster was provisioned using Kops.
A few YAML files later, the cluster was up-and-running.

Data Streaming

Several services were evaluated to implement the data streaming infrastructure. In particular, they evaluated SQS and KINESIS from Amazon, before deciding to go with Apache Kafka. Using a custom deployed streaming platform rather than an hosted one allowed keeping the operational costs low without sacrificing performance.
Kafka was deployed on the Kubernetes cluster with confluent using the official Docker images.
 

 

Querying Service

For the querying service, ThoughtWorks went with Amazon Athena. Athena has a variety of built-in importers, supporting CSV, Parquet, JSON and others. It is based on the Presto engine and does not require extra ETL steps to run, as data is stored directly on S3 buckets. As with many other serverless services, it has a low infrastructure cost as the client pays only for the queries he/she runs.
Athena is interrogated by the application, written in Python, using the PyAthena interface library.

Continuous Deployment

Adopting a continuous integration and deployment model is almost mandatory to maintain cloud applications, as they allow an effective improvement of the development team productivity.
ThoughtWorks evaluated two on-premise solutions to implement CI/CD pipelines for their application, comparing Travis and CircleCI. The latter was ultimately chosen for its better starting cost for enterprises.
 


 

Learnings

Although Athena was initially chosen for the development of a proof of concept, it showed its limits when used as a frequently accessed service. Athena doesn’t handle high concurrent loads. Since it is designed as a non-ETL service, it doesn’t cache data and this may be inefficient in certain applications. ThoughtWorks ultimately decided to drop Athena and use RDS instead, developing a custom interface to RDS for Kafka and performing some pre-processing before dumping the raw data to RDS. As the reader may expect, moving from many small Parquet files to a relational database allowed a great performance improvement.
Once again, serverless services are great and powerful, but choosing the one that fits a specific use case is a matter of both experience and good testing.

facebooktwitterlinkedinreddit
Share on:facebooktwitterlinkedinreddit

Tagged as:Codemotion Milan

SELECT for Cities contest: solve challenges and create value for citizens
Previous Post
Azure Durable Functions and Serverless Orchestration
Next Post

Related articles

  • What Is CloudOps and How to Implement It in Your Organization?
  • Serverless Computing: The Advantages and Disadvantages
  • Migrating to the Cloud With Kubernetes – A Step-by-Step Guide
  • The Science of Cloud Cost Optimization
  • Intelligent Document Processing With Serverless Cloud and Machine Learning
  • K8s Readiness Probe Failed Error
  • 5 Tips for a Successful GitOps Implementation
  • The Benefits of Cloud Computing for Customer Service Teams
  • Automating AWS Releases Without Cloud Expertise
  • Get to Know Cloud Run: Easy Deployment From the Cloud

Primary Sidebar

Free Whitepaper: The Ultimate Azure Cybersecurity Handbook.

Codemotion Talent · Remote Jobs

Flutter Developer

3Bee
Full remote · Android · Flutter · Dart

Python Back-end Developer

h-trips.com
Full remote · Django · Pandas · PostgreSQL · Python

AWS Cloud Architect

Kirey Group
Full remote · Amazon-Web-Services · Ansible · Hibernate · Kubernetes · Linux

AWS SysOps Administrator

S2E | Solutions2Enterprises
Full remote · Amazon-Web-Services · Terraform · Linux · Windows · SQL · Docker · Kubernetes

Latest Articles

scalable vue.js application

Best Practices for Building a Scalable Vue.js Application

Frontend

microservices digital transformation. From monolith to microservices concept.

Why You Need Application Mapping for Your Microservices Applications

Microservices

cross-platform development, frameworks

Ionic Framework: Why It’s Still Relevant

Mobile Developer

Linux: The Open Source Revolution and Its Impact on the Lives of Developers

Dev Life

Footer

  • Magazine
  • Events
  • Community
  • Learning
  • Kids
  • How to use our platform
  • Contact us
  • Become a Contributor
  • About Codemotion Magazine
  • How to run a meetup
  • Tools for virtual conferences

Follow us

  • Facebook
  • Twitter
  • LinkedIn
  • Instagram
  • YouTube
  • RSS

© Copyright Codemotion srl Via Marsala, 29/H, 00185 Roma P.IVA 12392791005 | Privacy policy | Terms and conditions

Follow us

  • Facebook
  • Twitter
  • LinkedIn
  • Instagram
  • RSS