Gourav Shivhare Tech

ELT Process Extraction, Load and Transform (ELT) is the technique of extracting raw data from the source and storing it in data warehouse of the target server and preparing it for endstream users. ELT comprises of 3 different operations performed on the data – Extract Extracting data is the technique of identifying data from one or more sources. The sources may be databases, files, ERP, CRM or any other useful source of data. Load Loading is the process of storing the extracted raw data in data warehouse or data lakes. Transform Data transformation is the process in which the raw data source is transformed to the target format required for analysis. Data from the sources are extracted and stored in the data warehouse. The entire data is not transformed but only the required transformation is done when necessary. Raw data can be retrieved from the warehouse anytime when required. The data transformed as required is then sent forward for analysis. When you use ELT, you move the entire da...

MuleSoft's Anypoint Runtime Fabric (RTF) has many features that help with deployment and management of Mule applications: Deployment: RTF can deploy applications to any environment, including on-premises, in the cloud, or in a hybrid setup. It can also automatically deploy Mule runtimes into containers. Isolation: RTF can isolate applications by running a separate Mule runtime server for each application. Scaling: RTF can scale applications across multiple replicas. Fail-over: RTF can automatically fail over applications. Monitoring and logging: RTF has built-in monitoring and logging capabilities to help teams troubleshoot issues and gain insights into application performance. Containerization: RTF supports containerization, which allows applications to be packaged with their dependencies and run consistently across different environments. Integration: RTF can integrate with services like SaveMyLeads to automate data flow between applications. Management: RTF can be managed with A...

Service Mesh A service mesh is an architectural pattern for microservices deployments . It’s primary goal is to make service-to-service communications secure, fast, and reliable. In a service mesh architecture, microservices within a given deployment or cluster interact with each other through sidecar proxy’s. The security and communication rules behind these interactions are directed through a control plane. The developer can configure and add policies at the control plane level, and abstract the governance considerations behind microservices, regardless of the technology used to build. Popular Service Mesh frameworks, such as Istio, have emerged to help organizations implement these architectural patterns.  A service mesh is a dedicated infrastructure layer that controls service-to-service communication within a distributed application. This method enables separate parts of an application to communicate with each other. Service meshes appear commonly in concert with cloud-native ...

What is Jenkins? Jenkins is an easy-to-use open-source java-based CI/CD tool. Jenkins has huge community support and an ocean of plugins that can integrate with many open-source and enterprise tools to make your life so easy. The following diagram shows the overall architecture of Jenkins and the connectivity workflow. Following are the key components in Jenkins Jenkins Master Node Jenkins Agent Nodes/Clouds Jenkins Web Interface Jenkins Server (Formerly Master) Jenkins’s server or master node holds all key configurations. Jenkins master server is like a control server that orchestrates all the workflow defined in the pipelines. For example, scheduling a job, monitoring the jobs, etc. Let’s have a look at the key Jenkins master components. Jenkins Jobs A job is a collection of steps that you can use to build your source code, test your code, run a shell script, run an Ansible role in a remote host or execute a terraform play, etc. We normally call it a Jenkins pipeline . Jenkins JOB :...

RabbitMQ , known for flexibility and support for multiple messaging protocols, is ideal for complex routing, message prioritization, and reliable delivery in microservices architectures. However, it may not handle high-throughput scenarios as efficiently as Kafka.   On the other hand, Apache Kafka , a distributed streaming platform, excels in high-throughput and fault-tolerant messaging, making it perfect for real-time data processing and analytics. Kafka typically outperforms RabbitMQ and ActiveMQ in high-volume scenarios due to its distributed nature. ActiveMQ , a Java-based message broker supporting the Java Message Service API, is favored in traditional enterprise messaging scenarios requiring robust security and transactional messaging. While offering good performance and enterprise features, ActiveMQ may not match Kafka's throughput capabilities in streaming scenarios. Each platform has its strengths and best-fit scenarios. Understanding their architectures and performance c...

To achieve optimal performance from your Mule applications, you must evaluate both the applications themselves and the environment in which they run. Although Mule 4 is designed to tune itself, your applications might exhibit performance issues due to their initial construction or dependencies. Similarly, for on-premises installations, you might need to tune the environment itself so that your Mule applications can take full advantage of it. Because many variables influence it, tuning the performance of your application requires some trial and error. You can simplify performance tuning by using documented best practices and testing your applications in ideal test environments. The following recommendations come from the Development and Services Engineering teams and benchmarking efforts by MuleSoft Performance Engineering. Optimizing the performance of your Mule apps requires the following actions: Applying tuning recommendations at the application level        ...

Understanding Integration Design Patterns: Integration design patterns serve as reusable templates for solving common integration problems encountered in software development. They encapsulate best practices and proven solutions, empowering developers to architect complex systems with confidence. These patterns abstract away the complexities of integration, promoting modularity, flexibility, and interoperability across components. Most Common Integration Design Patterns: Point-to-Point Integration: Point-to-Point Integration involves establishing direct connections between individual components. While simple to implement, this pattern can lead to tight coupling and scalability issues as the number of connections grows. Visualizing this pattern, imagine a network of interconnected nodes, each communicating directly with specific endpoints. Publish-Subscribe (Pub/Sub) Integration: Pub/Sub Integration decouples producers of data (publishers) from consumers (subscribers) through a central ...

Saga design pattern: Implement each business transaction that spans multiple services as a saga. A saga is a sequence of local transactions. Each local transaction updates the database and publishes a message or event to trigger the next local transaction in the saga. If a local transaction fails because it violates a business rule then the saga executes a series of compensating transactions that undo the changes that were made by the preceding local transactions. There are two ways of coordination sagas:  Choreography - each local transaction publishes domain events that trigger local transactions in other services Orchestration - an orchestrator (object) tells the participants what local transactions to execute Choreography based SAGA: An e-commerce application that uses this approach would create an order using a choreography-based saga that consists of the following steps:The Order Service receives the POST /orders request and creates an Order in a PENDING state It then emit...

REST API (also called a RESTful API or RESTful web API) is an application programming interface (API) that conforms to the design principles of the representational state transfer (REST) architectural style. REST APIs provide a flexible, lightweight way to integrate applications and to connect components in microservices architectures. REST stands for Representational State Transfer, a term coined by Roy Fielding in 2000. It is an architecture style for designing loosely coupled applications over the network, that is often used in the development of web services. REST does not enforce any rule regarding how it should be implemented at the lower level, it just put high-level design guidelines and leaves us to think of our own implementation. Let’s start with standard design-specific stuff to clear what ‘Roy Fielding’ wants us to build. Then we will discuss my thoughts, which will be more towards finer points while you design your RESTful APIs. Architectural Constraints: REST defines 6 ...