Tag: Data

Data Mesh Architecture in Cloud-Based Data Warehouses

Posted on by

Data is the new black gold in business. In this post, we explore how shifts in technology, organization processes, and people are critical to achieving the vision for a data-driven company that deploys data mesh architecture in cloud-based warehouses like Snowflake and Azure Synapse.

The true value of data comes from the insights gained from data that is often siloed and spans across structured, semi-structured, and unstructured storage formats in terabytes and petabytes. Data mining helps companies to gather reliable information, make informed decisions, improve churn rate and increase revenue.

Every company could benefit from a data-first strategy, but without effective data architecture in place, companies fail to achieve data-first status.

For example, a company’s Sales & Marketing team needs data to optimize cross-sell and up-sell channels, while its product teams want cross-domain data exchange for analytics purposes. The entire organization wishes there was a better way to source and manage the data for its needs like real-time streaming and near-real-time analytics. To address the data needs of the various teams, the company needs a paradigm shift to fast adoption of Data Mesh Architecture, which should be scalable & elastic.

Data Mesh architecture is a shift both in technology as well as in organization, processes, and people.

Before we dive into Data Mesh Architecture, let’s understand its 4 core principles:

  1. Domain-oriented decentralized data ownership and architecture
  2. Data as a product
  3. Self-serve data infrastructure as a platform
  4. Federated computational governance

Big data is about Volume, Velocity, Variety & Veracity. The first principle of Data mesh is founded on decentralization and distribution of responsibility to the SME\Domain Experts who own the big data framework.  

This diagram articulates the 4 core principles of Data Mesh and the distribution of responsibility at a high level.

Azure: Each team is responsible for its own domain, and data is decentralized and shared with other domains for data exchange and data as a product.
Snowflake: Each team is responsible for its own domain, and data is decentralized and shared with other domains for data exchange and data as a product.

Each Domain data is decentralized in its own data warehouse cloud. This model applies to all data warehouse clouds, such as Snowflake, Azure Synapse, and AWS Redshift.  

A cloud data warehouse is built on top of a multi-cloud infrastructure like AWS, Azure, and Google Cloud Platform (GCP), which allows compute and storage to scale independently. These data warehouse products are fully managed and provide a single platform for data warehousing, data lakes, data science team and to provide data sharing for external consumers.

As shown below, data storage is backed by cloud storage from AWS S3, Azure Blob, and Google, which makes Snowflake highly scalable and reliable. Snowflake is unique in its architecture and data sharing capabilities. Like Synapse, Snowflake is elastic and can scale up or down as the need arises.

From legacy monolithic data architecture to more scalable & elastic data modeling, organizations can connect decentralized enriched and curated data to make an informed decision across departments. With Data Mesh implementation on Snowflake, Azure Synapse, AWS Redshift, etc., organizations can strike the right balance between allowing domain owners to easily define and apply their own fine-grained policies and having centrally managed governance processes.

Additional resources:

Cloud Migration and Cloud Services

Posted on by
By Luca Junghans

A look inside these cloud capabilities

By joining forces, Valence and MajorKey offer an even greater set of cloud services for businesses that want to power their digital transformation with cloud technologies. 

MajorKey works with clients to migrate business applications to the cloud, and Valence builds services on the cloud. This is one reason these businesses are such a powerful combined force. 

The cloud refers to software and services that run on a (usually) regionally located server owned by the cloud service provider, instead of on an on-premise server owned by a customer. Cloud servers are in data centers all over the world. By using cloud computing, companies don’t have to manage physical servers or run software applications on their own machines. 

It’s big business. In fact, one of our partners, AWS contributed 14.5% of revenue to Amazon’s overall business in 2021, which would have operated at a $1.8 billion loss in Q4 without it – and AWS revenue was up nearly 39% compared to 2020. 

There are many ways to use and understand the business impact of cloud technology. We are breaking down the distinction between cloud services and cloud migration for you here!

Contact Us

Cloud Migration and Cloud Services 

Simply put, cloud migration is what happens when a company moves some or all of its software onto cloud servers.

In other words, cloud migration is moving your software to a managed server operated by the cloud provider; and cloud services are technology solutions built on top of those managed servers. There’s a whole range of capabilities bridging the two. 

Let’s take a closer look.  

Cloud services range in how much they abstract away from the customer.  A good example is Amazon Cognito, which is a user management cloud service. Amazon Cognito has implementations of basic user functions such as login, logout, sessions, and security, so a customer doesn’t have to worry about a deeper technical implementation of these features and can focus on managing users.  

Cloud services are so flexible that there are seemingly infinite ways to deploy them for a business. Cloud services are the infrastructure, platforms, and software hosted by cloud providers, and there are three common solutions:   

  1. Infrastructure as a service: The renting out of virtual machines and space to customers, while providing a way to remotely manage the resource. When a company migrates to the cloud, they are using this service. 
  2. Platforms: Providers like AWS and Azure build specialized software on top of their own cloud hardware and offer the software to customers as a service. These are specialty services and can provide patterns for things such as Data Analysis, Compute, IoT, APIs, Security, Identity, and Containerization. We wrote about Digital Twins in a previous post, which referenced Digital Twin platforms offered by AWS and Azure.  
  3. Software as a service (SaaS): Software can be built on top of the platforms offered by the cloud providers. Software developers can also partner with other third parties to provide fully built instances of software that typically come with subscription rates, customer support, and personal configurations of the software. Examples of this include Atlassian Jira and Confluence, Dropbox, Salesforce, and G suite. 

These services can be transformative for businesses in general, but it’s not always easy to know the best way for your business to use them. The added benefits to this migration range per case, and here are four examples: 

  • Scalability: Cloud services often offer on demand scaling options that can satisfy unexpected or planned growth. Depending on your product, this can be a lot easier than upgrading on-premise hardware, but not always cheaper. 
  • Cost: Although we expect the costs to be passed to the consumer in some way, the logistics of maintenance and upgrades to the cloud systems is handled by the provider. In many cases this can translate to a huge amount of money saved for the customers. 
  • Performance: Performance-enhancing services like CDNs and regional hosting, when understood and configured properly, can have tangible and positive performance impacts. 
  • Local Management: Being on the cloud means access to the digital portals to manage the services (most times). This creates a lower bar of entry for employees to manage and observe the resources. 

Many businesses start their digital transformation journey by migrating infrastructure or applications from on-premises servers to the cloud. Notably, cloud migration can also refer to a situation where a business needs to bring the cloud resources they manage into an on-premises environment. It can also describe a situation where a business moves its data resources from one cloud provider to another.  

Cloud migration to use cloud services is a process that presents many upsides, and is worth investigating!  The process will add additional complexities – specifically, security and governance will generally be instituted upfront as a base for the rest of the migration. We design and engineer performant, scalable, and maintainable applications that save businesses money, fill in knowledge gaps, and provide users with a positive experience.  

Here are two examples of cloud services that we’ve built for clients:  

  • Building cloud applications with AWS lambda: We have bridged the gap between multiple third-party APIs and created new databases that consolidate data and deliver it to a web application. Cloud services remove the need for our clients to interact with these multiple services, which saves them time and money. At the same time, we used AWS Cognito to help our customer manage roles and users in a secure and trusted way. This removed the need for our engineers to write our own user management software, a cumbersome task. 
  • Data pipelines:  We identify problems in our customers’ current database providers and migrate data to a more performant and better structured database in cloud-to-cloud migrations.  

We will continue to build and migrate while we investigate the future of the cloud. What are the new services and platforms? Who can benefit the most from them? How can we do it right? We will be prepared for the cloud migration and services needed from the real world to the metaverse, and beyond.  

Contact Us

Additional Resources: