API (Application Programming Interface) security is critical for protecting sensitive data and maintaining the integrity of systems and applications. APIs are used to connect different systems and applications, and as a result, they can provide a gateway for attackers to gain access to sensitive information.

API security involves implementing measures to protect the integrity and confidentiality of data that is transmitted through APIs. This includes securing the API endpoint, authenticating and authorizing API requests, and encrypting data in transit.

API security is important for several reasons:

1. Protection of sensitive data: APIs are often used to access and transmit sensitive information, such as financial data, personal information, and confidential business information. Without proper security measures in place, this data can be vulnerable to theft, alteration, or unauthorized access.

2. Compliance: Many industries have strict regulations and compliance requirements, such as HIPAA and PCI-DSS, that require organizations to implement robust security measures to protect sensitive data.

3. Maintaining the integrity of systems and applications: Unauthorized access to APIs can allow attackers to modify or disrupt the functionality of systems and applications. This can result in service disruption, data loss, and reputational damage.

4. Preventing identity theft: APIs are also used to authenticate users and authorize access to resources. Insecure APIs can allow attackers to steal the identities of legitimate users and gain unauthorized access to sensitive information.

To ensure API security, organizations should implement a comprehensive security strategy that includes:

• Strong authentication and authorization mechanisms

• Encryption of data in transit

• Regularly testing and monitoring APIs for vulnerabilities

• Use of API gateways, firewalls, and web application firewalls

• Regularly updating and patching systems and applications

• Regularly train employees on API security best practices and how to identify and report suspicious activity.

In conclusion, API security is a critical component of an organization's overall security strategy. It is important to implement robust security measures to protect sensitive information and maintain the integrity of systems and applications. By staying informed and taking proactive measures, organizations can reduce the risk of falling victim to API-based attacks.

[Karthikeyan S](https://www.linkedin.com/in/herbie36/)

Community and Social Footprints :

- [GitHub](https://github.com/cloudnloud)

- [Twitter](https://twitter.com/cloudnloud)

- [YouTube Cloud DevOps Free Trainings](https://www.youtube.com/c/CloudnLoud)

- [Linkedin Page](https://www.linkedin.com/company/cloudnloud/)

- [Linkedin Group](https://www.linkedin.com/groups/9124202/)

- [Discord Channel](https://discord.com/invite/vbjRQGVhuF)

- [Dev](https://dev.to/cloudnloud)

I was browsing through meetup.com and was intrigued by one event from AWS user group Pune. It is AWS Reinvent 2022 -Recap on 21st Jan. Having missed the actual event in December 2022, I felt this is an opportunity to get a sneak peek into what else AWS has to introduce to us. I immediately RSVPd and here is how it went!!

EPAM Systems Pune hosted the event with 200+ participants and organized it very well. I could meet and share a conversation with Sr. Solutions architect Mayur Bhagia and AWS community builders and professionals from the city.

Dheeraj Chaudhary kickstarted the session with the AWS application composer which is still in preview. this service is a visual builder and makes it easier to design a serverless application architecture with options like dragging, grouping and connecting various AWS services. It is like a Canvas where you can drag and drop, group and configure various AWS services.

Then came a question why did AWS introduce application composer when AWS cloud formation is already in place?

AWS cloud formation is difficult and complex for beginners to use. Hence AWS introduced the service to simplify IaaS. An attempt towards ‘No-Code’ IaaS.

Amazon Code Whisperer(in preview):

This is gonna be a developer's best friend. This service recommends inbuilt functions and gives suggestions based on a developer's style of coding for which AWS uses AI ML in the backend. It supports Java, Python,JavaScript,C# and Typescript. This service is now available in VScode,Cloud9,Jetbrains. One needs to install the AWS toolkit and integrate with AWS IAM identity centre and log in to be able to use this feature

Then, Mayur Bhagia introduced us to many other services from AWS.

Amazon ECS Service Connect(GA) can simplify the Service Discovery, Connectivity and traffic observability for ECS. This service is in General Availability now and can be used in Prod applications

Amazon RDS Blue/Green Deployments (GA): This feature lets us make DB-related changes like upgrades, modifying parameters or schema in the staging environments without impacting the Production and then cut over the staging environment into Production with minimal downtime(usually less than 60s).

Mayur elaborately discussed AWS Elastic Disaster Recovery Automated Failback(GA), AWS Cloudwatch Internet monitor(Preview), Amazon Cloudwatch Logs Data Protection(GA), S3 Multi-Region Access points Failover Controls(GA), Amazon Route 53 ARC-Zonal Shift(Preview).

He had explained in detail

• The difference between Blue/green deployment and Canary deployment and

• How an on-prem application/ data is migrated onto the cloud.

• Difference between snapshot and AMI

• Difference between Cloudwatch and Cloud Trail

• HAR- HTTP ARchive format,

• RUM : Real User Monitoring (Lol..I knew only one RUM.. good to know another meaning to it :P)

• Synthetics monitoring

Somesh Srivastava, spoke on networking related topics like Amazon VPC Lattice(Preview) and AWS Verified Access( Preview)

AWS VPC lattice is not a new service altogether. It is a construct within VPC servie. This is integrated with IAM. He had given a demo on how to make use this service

Somesh also explained how cross-zone loadbalancing works, different use cases and addressed queries from the group.

As the session came to an end, it is quiz time to check how much we have grasped from this event and guess what! I stood second and was given some nice cute goodies...Lol!!!

All in all, it was a great event where you get to interact with professionals, students and IT aspirants. I got some key advice from seniors on my next steps and I advised some freshers and entry-level professionals on their career paths. Sometimes you counsel others, and sometimes you get counseled!! Good either ways!

The sequel to this event, Part 2 would be conducted on 4th Feb. I will keep you all posted about that as well. So follow me on my linkedin . Happy learning until then!! Bub-byee!!

Cybersecurity is a defensive method to defend devices and services from cyber-attacks/cyber criminals and ensure the data integrity/confidentiality/availability is maintained. Today the complete World is having data stored across the globe (Cloud/Data centers) and with the emerging trend of cyber-attacks around the world every organization must maintain data security and privacy through a strong defensive security model.

📌 How Cyber security helps?

• Security to safeguard the computing environment

• Ensuring strong controls and security is implemented across the environment

• Protection of computer resources and data from theft/damage/attacks from external or unauthorized sources

• Protecting the network from digital attacks that aims to destroy the system and its information

📌 How Does Cyber Security Work? The Challenges of Cyber Security

Cyber security works in a series of subdomains within Security – like Application Security, Cloud Security /Identity Management and Data Security/Mobile Security/Network Security/Disaster Recovery and Business Continuity Planning. Each of these is segregated to maintain the security measures within its limit, and ensure strong and robust security is implemented to make it the attackers impossible to initiate the attack.

📌 Types of Cyber Threats (Listed some but not limited to)

• Malware attack

• Trojan

• Botnets

• Adware

• SQL injection

• Phishing

• Man in The Middle (MITM) attack

• Denial Of Service (DOS) attack

• Ransomware

• Crypto jacking

• Social engineering

📌Different Domains in Cyber security?

• Access control systems & methodologies – dealing with the protection of systems and it's resources from unauthorized access. Ex: MFA, SSO etc

• Telecommunication and Network Security – dealing with network communications, protocols, services, and threats/vulnerabilities associated with those services.

• Security Management practices – dealing and managing the systems against system failures, cyber-attacks, natural disasters, and other interruptions to the services.

• Security Architecture and Engineering -Policies and Procedures to facilitate Security controls and involves in policy planning for every type of security issues.

• Law, Investigation and Ethics -Legal issues associated with security of the system, such as dealing with cyber-attacks is dealt in this domain

• Application and system development security - Security during the application development and database security models and implementation of security during the testing and development phase including code security etc.

• Cryptography – helps to understand how, when, and what encryptions to be used and covers various types of encryption and logic behind the same.

• Computer operations security – day-to-day security operations on computers. Preventing malware attacks/incidents reported.

• Physical security – deals with Physical access to the computer resources servers, workstations etc.

📌How Does Cyber Security Work? The Challenges of Cyber Security

Cyber security works in a series of subdomains within Security – like Application Security, Cloud Security /Identity Management Data Security/Mobile Security/Network Security/Disaster Recovery and Business Continuity Planning. Each of these is segregated to maintain the security measures within its limit and ensure robust security implemented to make attackers impossible to initiate the attack.

Happy learning folks 🙂

Community and Social Footprints :

• Kannammal G

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

As companies increasingly adopt machine learning (ML) for their business applications, they are looking for ways to improve governance of their ML projects with simplified access control and enhanced visibility across the ML lifecycle. A common challenge in that effort is managing the right set of user permissions across different groups and ML activities. For example, a data scientist in your team that builds and trains models usually requires different permissions than an MLOps engineer that manages ML pipelines. Another challenge is improving visibility over ML projects. For example, model information, such as intended use, out-of-scope use cases, risk rating, and evaluation results, is often captured and shared via emails or documents. In addition, there is often no simple mechanism to monitor and report on your deployed model behavior.

That’s why I’m excited to announce a new set of ML governance tools for Amazon SageMaker.

As an ML system or platform administrator, you can now use Amazon SageMaker Role Manager to define custom permissions for SageMaker users in minutes, so you can onboard users faster. As an ML practitioner, business owner, or model risk and compliance officer, you can now use Amazon SageMaker Model Cards to document model information from conception to deployment and Amazon SageMaker Model Dashboard to monitor all your deployed models through a unified dashboard.

Let’s dive deeper into each tool, and I’ll show you how to get started.

Amazon SageMaker Role Manager SageMaker Role Manager lets you define custom permissions for SageMaker users in minutes. It comes with a set of predefined policy templates for different personas and ML activities. Personas represent the different types of users that need permissions to perform ML activities in SageMaker, such as data scientists or MLOps engineers. ML activities are a set of permissions to accomplish a common ML task, such as running SageMaker Studio applications or managing experiments, models, or pipelines. You can also define additional personas, add ML activities, and your managed policies to match your specific needs. Once you have selected the persona type and the set of ML activities, SageMaker Role Manager automatically creates the required AWS Identity and Access Management (IAM) role and policies that you can assign to SageMaker users.

A Primer on SageMaker and IAM Roles A role is an IAM identity that has permissions to perform actions with AWS services. Besides user roles that are assumed by a user via federation from an Identity Provider (IdP) or the AWS Console, Amazon SageMaker requires service roles (also known as execution roles) to perform actions on behalf of the user. SageMaker Role Manager helps you create these service roles:

• SageMaker Compute Role – Gives SageMaker compute resources the ability to perform tasks such as training and inference, typically used via PassRole. You can select the SageMaker Compute Role persona in SageMaker Role Manager to create this role. Depending on the ML activities you select in your SageMaker service roles, you will need to create this compute role first.

• SageMaker Service Role – Some AWS services, including SageMaker, require a service role to perform actions on your behalf. You can select the Data Scientist, MLOps, or Custom persona in SageMaker Role Manager to start creating service roles with custom permissions for your ML practitioners.

Now, let me show you how this works in practice.

There are two ways to get to SageMaker Role Manager, either through Getting started in the SageMaker console or when you select Add user in the SageMaker Studio Domain control panel.

I start in the SageMaker console. Under Configure role, select Create a role. This opens a workflow that guides you through all required steps.

Let’s assume I want to create a SageMaker service role with a specific set of permissions for my team of data scientists. In Step 1, I select the predefined policy template for the Data Scientist persona.

I can also define the network and encryption settings in this step by selecting Amazon Virtual Private Cloud (Amazon VPC) subnets, security groups, and encryption keys.

In Step 2, I select what ML activities data scientists in my team need to perform.

Some of the selected ML activities might require you to specify the Amazon Resource Name (ARN) of the SageMaker Compute Role so SageMaker compute resources have the ability to perform the tasks.

In Step 3, you can attach additional IAM policies and add tags to the role if needed. Tags help you identify and organize your AWS resources. You can use tags to add attributes such as project name, cost center, or location information to a role. After a final review of the settings in Step 4, select Submit, and the role is created.

In just a few minutes, I set up a SageMaker service role, and I’m now ready to onboard data scientists in SageMaker with custom permissions in place.

Amazon SageMaker Model Cards
SageMaker Model Cards helps you streamline model documentation throughout the ML lifecycle by creating a single source of truth for model information. For models trained on SageMaker, SageMaker Model Cards discovers and autopopulates details such as training jobs, training datasets, model artifacts, and inference environment. You can also record model details such as the model’s intended use, risk rating, and evaluation results. For compliance documentation and model evidence reporting, you can export your model cards to a PDF file and easily share them with your customers or regulators.

To start creating SageMaker Model Cards, go to the SageMaker console, select Governance in the left navigation menu, and select Model cards.

Select Create model card to document your model information.

Amazon SageMaker Model Dashboard SageMaker Model Dashboard lets you monitor all your models in one place. With this bird’s-eye view, you can now see which models are used in production, view model cards, visualize model lineage, track resources, and monitor model behavior through an integration with SageMaker Model Monitor and SageMaker Clarify. The dashboard automatically alerts you when models are not being monitored or deviate from expected behavior. You can also drill deeper into individual models to troubleshoot issues.

To access SageMaker Model Dashboard, go to the SageMaker console, select Governance in the left navigation menu, and select Model dashboard.

Note: The risk rating shown above is for illustrative purposes only and may vary based on input provided by you.

Community and Social Footprints :

• Sampath Kumar Basa

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

What is PKI?

• Set of processes/rules/procedures/policies which create/manage/store/revoke the digital signature and manage public key encryption.

• It uses two sets of keys - Public key - shared with anyone you connect and Private Key -shared only to a known person

• It is the most common method of encrypting the data between web server and browsers

Components of PKI

Certificate: A digital document, signed by a CA, and used to prove the owner of a public key, within a PKI. The certificate has several attributes, such as usage of the key, Client authentication, Server authentication or Digital signature and the public key. The certificate also contains the subject name which is information identifying the owner. This could be, for example, a DNS name or IP address.

Certificate Authority (CA) - is an authority in a network that issues and manages security credentials and public keys for message encryption. a CA checks with a registration authority (RA) to verify information provided by the requestor of a digital certificate. If the RA verifies the requestor's information, the CA can then issue a certificate. CA maintains a directory of digital certificates for the reference of those receiving them. It manages the certificate life cycle, including certificate directory maintenance and certificate revocation list maintenance and publication.

Registration Authority (RA)- A person or organization responsible for the identification and authentication of an applicant for a digital certificate. An RA does not issue or sign certificates. RA verifies information supplied by the subject requesting a certificate. A registration authority (RA) is an entity that is trusted by the certificate authority (CA) to register or vouch for the identity of users to a CA and is a component of PKI.

Certificate Revocation List (CRL): Checks the continued validity of the certificates for which the CA has responsibility. The CRL details Digital Certificates that are no longer valid because they were revoked by the CA.

Certificate Repository: A location where all certificates are stored as well as their public keys, validity details, revocation lists, and root certificates. These locations are accessible through LDAP, FTP or web servers.

Certification Practice Statement (CPS) - is a PKI element that provides detailed descriptions for dealing with a compromised private key.

Validation Authority: A VA allows an entity to check that a certificate has not been revoked. The VA role is often carried out by an online facility hosted by an organization that operates the PKI. A validation authority will often use OCSP or CRL to advertise revoked certificates.

Benefits

Confidentiality (only authorized person can read an encrypted message) - Authenticity (Senders sign/encrypt the message so it ensures the recipient that message is not altered during transit) - Non-repudiation (senders can't deny the message/contents sent)

Community and Social Footprints :

• Kannammal G

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

In this blog, We are going to discuss the following services

• AWS CodeStar

• AWS Fault Injection Simulator

• AWS X-Ray

AWS CodeStar

AWS CodeStar is a fully managed service that makes it easy to develop, build, and deploy applications on AWS. It provides a unified user interface to easily manage the full application development life cycle, including source control, build and test, and deployment.

CodeStar provides a variety of project templates for popular languages and frameworks, such as Java, Python, Ruby, and more, that include preconfigured build and deployment settings for AWS services such as Amazon EC2, AWS Elastic Beanstalk, and AWS Lambda.

CodeStar also provides an integrated development environment (IDE) called CodeStar WorkSpaces, which is a cloud-based IDE that makes it easy to write, run, and debug code.

CodeStar integrates with other AWS services such as CodeCommit, CodeBuild, CodeDeploy, and CodePipeline, which enables you to create a complete end-to-end continuous integration and continuous delivery (CI/CD) pipeline.

Additionally, CodeStar provides access to a variety of tools for monitoring and troubleshooting your application, such as AWS CloudWatch, AWS X-Ray, and AWS CloudTrail.

CodeStar is designed to be highly scalable and can handle multiple projects, developers and teams. It also supports role-based access control (RBAC) to give developers access to the resources they need, while keeping your applications secure.

AWS CLI

aws codestar create-project --name sample-project --project-template-id aws-java-maven --region us-west-2

aws codestar list-projects

AWS Fault Injection Simulator

AWS Fault Injection Simulator (FIS) is a fully managed service that allows you to simulate various types of failures in your applications, such as network failures, latency, and throttling. This helps you test and validate the resiliency of your applications, without affecting production environments or incurring additional costs.

AWS FIS enables you to create a set of failure scenarios that you can apply to your applications, and then run those scenarios in a controlled environment. You can use the service to test various failure scenarios in your applications, such as network failures, service failures, and resource failures.

AWS FIS supports the ability to simulate failures across different layers of your application, including the network, application, and infrastructure layers. This allows you to test the resiliency of your applications across different components, and identify potential issues before they affect production.

AWS FIS supports a variety of configurations, including custom failure scenarios, targeted failure scenarios, and scheduled failure scenarios. This allows you to simulate different types of failures, such as network latency and packet loss, and to schedule those failures at specific times.

AWS FIS integrates with AWS CloudWatch and CloudTrail, allowing you to monitor and troubleshoot the results of your failure scenarios. This helps you identify any issues and make necessary adjustments to improve the resiliency of your applications.

AWS CLI

aws fis create-failure-injection --type network --failure-rate 50 --target-type ec2 --target-id i-1234567890abcdef0

aws fis list-failure-injections

AWS X-Ray

AWS X-Ray is a fully-managed service that allows you to analyze and debug production, distributed applications, such as microservices or serverless applications. It allows you to trace requests and data flows through your application, and identify performance bottlenecks and errors.

AWS X-Ray provides an interactive console that allows you to view and analyze traces, create service maps, and view performance metrics. It also provides a SDK that can be integrated with a variety of languages, such as Java, .NET, Node.js, and more, to instrument your application and automatically generate traces.

AWS X-Ray also provides a variety of features to help you analyze and troubleshoot issues in your applications, such as:

• Distributed Tracing: Allows you to trace requests and data flows through your application, and identify performance bottlenecks and errors.

• Service Maps: Allows you to view a map of all the services in your application, and how they interact with each other.

• Anomaly Detection: Allows you to identify and troubleshoot performance issues in your application.

• Error reporting: Allows you to view and analyze errors and exceptions in your application.

• Search and filter: Allows you to search and filter traces and service maps.

AWS X-Ray integrates with other AWS services, such as AWS Lambda, Amazon Elastic Container Service (ECS), and Amazon Elastic Container Service for Kubernetes (EKS), to provide deeper visibility into your applications.

AWS X-Ray is designed to be highly scalable and can handle high volumes of requests and data. It also supports role-based access control (RBAC) to give developers access to the resources they need, while keeping your applications secure.

Terraform code :

example of Terraform code that enables X-Ray for an Elastic Beanstalk application

resource "aws_elastic_beanstalk_environment" "x-ray" {
  name = "example-environment"
  application = "example-application"
  solution_stack_name = "64bit Amazon Linux 2 v3.3.3 running Multi-container Docker 2.3.5"
  xray_tracing = true
}

AWS CLI

aws xray get-sampling-rules --sampling-rule-name example-sampling-rule

That's it guys. We completed our AWS developer tools series. Let's celebrate.

Keep Learning Keep Growing !!!

Community and Social Footprints :

• Veerasolaiyappan

• GitHub

• Twitter

• YouTube Cloud DevOps Free Training

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

List of Developer Tools

• Amazon Corretto

• AWS Cloud9

• AWS CloudShell

• AWS CodeArtifact

• AWS CodeBuild

• AWS CodeCommit

• AWS CodeDeploy

• AWS CodePipeline

• AWS CodeStar

• AWS Fault Injection Simulator

• AWS X-Ray

Let's discuss each service in detail

Amazon Corretto

Amazon Corretto is a free, open-source, and production-ready distribution of the Open Java Development Kit (OpenJDK). It is developed and maintained by Amazon Web Services (AWS) and is designed to provide a consistent and reliable runtime environment for Java applications.

One of the key features of Amazon Corretto is that it provides long-term support for both Java 8 and 11 versions, with security patches and updates provided at no additional cost. This allows developers to run their applications on a stable and supported version of Java for extended periods of time.

Amazon Corretto also includes performance enhancements, such as the ability to use the Aarch64 (Arm64) architecture and support for the latest version of the Linux kernel, which can improve the performance of Java applications running on AWS.

Another important feature of Amazon Corretto is that it is fully compatible with the Java SE standard and has passed the Java SE TCK. This means that Java applications written to run on the standard OpenJDK will run unchanged on Amazon Corretto.

Amazon Corretto is also easy to use, developers can simply download the Amazon Corretto distribution and use it as a drop-in replacement for their current JDK.

Terraform code:

resource "aws_instance" "correto" {
  ami           = "ami-0ff8a91507f77f867"
  instance_type = "t2.micro"

  user_data = <<-EOF
    #!/bin/bash
    yum install -y java-1.8.0-amazon-corretto-devel
    EOF
}

AWS CLI code

aws ssm send-command --document-name "AWS-RunShellScript" --instance-ids "i-1234567890abcdef0" --parameters commands="yum install -y java-1.8.0-amazon-corretto-devel"

This command uses the AWS Systems Manager (SSM) to run a shell script on the specified EC2 instance (i-1234567890abcdef0) that installs the Amazon Corretto JDK

AWS Cloud9

AWS Cloud9 is a cloud-based integrated development environment (IDE) that makes it easy to write, run, and debug code. It provides a web-based development environment that can be accessed from anywhere and supports a wide range of languages and frameworks, including JavaScript, Python, Ruby, and C++.

One of the key features of Cloud9 is its built-in collaboration capabilities. Developers can share their development environments with others and work together in real-time, regardless of their location. This makes it easy for teams to collaborate and work together on the same codebase.

Cloud9 also includes a wide range of development tools and features, such as code completion, debugging, and version control integration. This makes it easy for developers to write, test, and debug their code all within the same environment.

Cloud9 also integrates with other AWS services, such as AWS Lambda and Elastic Beanstalk, making it easy to develop, test, and deploy applications in the cloud. This allows developers to easily build and deploy their applications without having to leave the Cloud9 environment.

Terraform code

resource "aws_cloud9_environment_ec2" "example" {
  name = "example-environment"
  instance_type = "t2.micro"
}

AWS CLI

aws cloud9 create-environment-ec2 --name "example-environment" --instance-type "t2.micro"

AWS CloudShell

AWS CloudShell is a browser-based shell that allows developers to easily interact with AWS services and resources. It is a fully managed service that provides a temporary, secure environment with the necessary tools and permissions to interact with AWS resources.

One of the key features of AWS CloudShell is its built-in access to the AWS Management Console, AWS CLI, and other development tools. This allows developers to quickly and easily perform common actions, such as creating and managing resources, without having to install or configure any tools on their local machine.

AWS CloudShell also includes pre-configured environments for popular programming languages and frameworks, such as Python, Node.js, and .NET Core. This allows developers to quickly set up a development environment and start coding, without having to spend time installing dependencies or configuring the environment.

AWS CloudShell also integrates with other AWS services, such as AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy, making it easy to automate the development process. This allows developers to easily build, test, and deploy their applications on AWS, without having to leave the CloudShell environment.

AWS CodeArtifact

AWS CodeArtifact is a fully managed artifact repository service for storing, publishing, and sharing software packages. It makes it easy for teams to manage, share, and use software packages across their organization.

One of the key features of AWS CodeArtifact is its support for multiple package formats, such as npm, Maven, and PyPI, and it can store multiple versions of a package. This allows teams to use the package manager and format that they are already familiar with and also to use the same package across different projects and applications.

AWS CodeArtifact also provides fine-grained access control, allowing teams to specify who can access and consume packages, and also to configure permissions at the package, repository, and domain level.

AWS CodeArtifact also integrates with other AWS services such as AWS CodeBuild, AWS CodeDeploy, and AWS CodePipeline, making it easy to automate the software release process. This allows teams to easily build, test, and deploy their applications using the packages stored in CodeArtifact

Terraform Code

resource "aws_codeartifact_repository" "example" {
  domain = "cloudnloud.com"
  repository = "cloudnloud-repo"
}

AWS CLI

aws codeartifact create-repository --domain cloudnloud.com --repository cloudnloud-repo

AWS CodeBuild

AWS CodeBuild is a fully managed build service that compiles source code, runs tests, and produces software packages that are ready to deploy. It supports multiple programming languages and build environments, and can be easily integrated with other AWS services such as CodeCommit, CodeDeploy, and CodePipeline. CodeBuild scales automatically to meet the needs of your builds, and it can be used to build and test code in a continuous integration and continuous delivery (CI/CD) pipeline. Additionally, CodeBuild provides a variety of features to help you optimize your build process, such as caching, environment variables, and build logs.

Terraform Code

resource "aws_codebuild_project" "example" {
  name            = "example-project"
  source {
    type     = "GITHUB"
    location = "https://github.com/user/repo.git"
  }
  artifacts {
    type = "S3"
    location = "example-bucket"
  }
  environment {
    compute_type = "BUILD_GENERAL1_SMALL"
    image = "aws/codebuild/standard:2.0"
    type = "LINUX_CONTAINER"
  }
  service_role = aws_iam_role.codebuild_role.arn
}

AWS CLI

aws codebuild create-project --name example-project --source https://github.com/user/repo.git --artifacts-type S3 --artifacts-location example-bucket --environment-type LINUX_CONTAINER --environment-image aws/codebuild/standard:2.0 --service-role arn:aws:iam::<ACCOUNT_ID>:role/example-codebuild-role

AWS CodeCommit

AWS CodeCommit is a fully-managed, source control service that hosts private Git repositories. It is a native Git service that makes it easy for developers to store, manage, and track code changes. CodeCommit is integrated with other AWS services such as CodeBuild, CodeDeploy, and CodePipeline, which enables you to create a complete end-to-end continuous integration and continuous delivery (CI/CD) pipeline.

CodeCommit supports standard Git functionality and provides additional features such as Git-based authentication, repository access control, and an API for programmatic access. It also integrates with IAM, allowing you to control access to your repositories at a granular level.

One of the advantages of CodeCommit is that it allows you to store your code in a centralized and secure location that is backed by AWS's highly available and scalable infrastructure. Also, it is cost-effective, as you only pay for what you use and there are no upfront costs or long-term commitments

Terraform Code

resource "aws_codecommit_repository" "example" {
  repository_name = "cloudnloud-repo"
}

AWS CLI

aws codecommit create-repository --repository-name cloudnloud-repo

AWS CodeDeploy

AWS CodeDeploy is a fully managed deployment service that automates software deployments to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and on-premises servers. It helps you rapidly release new features, handle network and application outages, and roll back when necessary, with minimal downtime.

CodeDeploy supports both in-place and blue/green deployment options. In-place deployments replace the existing instances with new ones, while blue/green deployments create a parallel environment and then switch traffic to the new instances. This allows for a more controlled and predictable deployment process, and it makes it easy to roll back to a previous version if necessary.

CodeDeploy can be integrated with other AWS services such as CodeBuild, CodeCommit, and CodePipeline, which enables you to create a complete end-to-end continuous integration and continuous delivery (CI/CD) pipeline. It also supports integration with third-party tools such as Jenkins, GitHub, and Bitbucket.

CodeDeploy supports multiple platforms including Windows and Linux, making it easy to deploy applications written in any language, including .NET, Java, Ruby, and more. It also provides built-in support for deploying applications to Amazon EC2 instances, AWS Lambda functions, and on-premises servers.

AWS CodeDeploy provides detailed tracking and visibility into the deployment process, and it can be integrated with AWS CloudWatch, AWS Config, and AWS CloudTrail for monitoring and auditing purposes.

AWS CodePipeline

AWS CodePipeline is a fully managed continuous delivery service that helps you automate your release pipelines for fast and reliable application and infrastructure updates. It enables you to rapidly and reliably deliver features and updates, while minimizing the chance of failed deployments.

CodePipeline is a highly customizable service that allows you to integrate with a variety of tools and services such as AWS CodeCommit, GitHub, CodeBuild, CodeDeploy, Jenkins, and more. You can use it to create a pipeline that builds, tests, and deploys your code every time there is a change in your source repository.

CodePipeline allows you to easily visualise your pipeline stages, actions and the status of each stage. It also provides detailed tracking and visibility into the pipeline process, and it can be integrated with AWS CloudWatch, AWS Config, and AWS CloudTrail for monitoring and auditing purposes.

CodePipeline also supports manual approvals, which allows you to add a human approval step to your pipeline, for example, you can use it to add a manual approval step for your production deployments.

AWS CodePipeline is designed to be highly scalable and can handle multiple pipelines, thousands of actions and many parallel actions. It also integrates with other AWS services, such as AWS CodeStar, AWS CloudFormation, and AWS Elastic Beanstalk, making it easy to build and deploy applications on AWS.

Terraform Code

resource "aws_codepipeline" "codepipeline" {
  name     = "example-pipeline"
  role_arn = aws_iam_role.codepipeline_role.arn

  artifact_store {
    location = "code-bucket"
    type     = "S3"
  }

  stage {
    name = "Source"

    action {
      name            = "Source"
      category        = "Source"
      owner           = "ThirdParty"
      provider        = "GitHub"
      version         = "1"
      output_artifacts = ["example"]

      configuration = {
        Repo = "example-repo"
        Branch = "main"
        OAuthToken = "example-token"
      }
    }
  }
}

resource "aws_iam_role" "codepipeline_role" {
  name = "example-codepipeline-role"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "codepipeline.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

AWS CLI

aws codepipeline create-pipeline --name example-pipeline --role-arn arn:aws:iam::<ACCOUNT_ID>:role/example-codepipeline-role --artifact-store type=S3,location=example-bucket --stage-name Source --action-name Source --action-type Source --action-configuration Repo=example-repo,Branch=main,OAuthToken=example-token,Owner=ThirdParty,Provider=GitHub

Okay. That's it. Part 1 is completed. We will discuss the remaining services in Developer Tools - Part 2. Stay tuned

Keep Learning Keep Growing !!!

Community and Social Footprints :

• Veerasolaiyappan

• GitHub

• Twitter

• YouTube Cloud DevOps Free Training

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

Do you have any idea about AWS Managed blockchain services and why you should consider them for your business?

Don't have any idea. No issues. Let's discuss on further,

Blockchain technology has the potential to revolutionize many industries, but setting up and managing a blockchain network can be a complex and time-consuming process. That's where Amazon Web Services (AWS) Managed Blockchain comes in.

AWS Managed Blockchain is a fully managed service that makes it easy to create and manage scalable blockchain networks using popular open-source frameworks like Ethereum and Hyperledger Fabric. With AWS Managed Blockchain, you can set up and deploy a blockchain network in a matter of minutes and easily scale it as your needs change.

One of the key benefits of AWS Managed Blockchain is that it takes care of the underlying infrastructure, including the network nodes, storage, and networking. This means you can focus on building your applications and not worry about the maintenance and management of the network.

AWS Managed Blockchain also offers built-in security measures to protect your network, including encryption of data in transit and at rest, secure access controls, and monitoring and logging of network activity.

In addition to its ease of use and security features, AWS Managed Blockchain can be easily integrated with other AWS services, such as Amazon S3 and Amazon Kinesis, to build and deploy decentralized applications. This makes it a powerful and flexible solution for businesses of all sizes.

Another advantage of AWS Managed Blockchain is its flexible pricing model. As a fully managed service, you only pay for the resources you use, such as the number of nodes and the amount of storage you need. This makes it an affordable and scalable solution that can grow with your business.

Features:

1. Quick setup

2. Scalability

3. Network management

4. Security

5. Integration with other AWS services

6. Flexible pricing

I hope, Now You understand AWS-managed blockchain services benefits

Anyways we should know about the basics of Blockchain. Without knowing the basics and fundamentals, we can't go deeper into any technologies.

So, Let's learn some basics of Blockchain

What is the blockchain

A blockchain is a decentralized, distributed database that consists of a series of interconnected blocks of data. It is a digital ledger of transactions that are secured and validated by a network of computers, rather than a central authority.

There are several different types of blockchain networks, including public and private blockchains. Public blockchains, such as Bitcoin, are open to anyone and are secured by a network of miners who compete to validate transactions and create new blocks. Private blockchains, on the other hand, are restricted to a specific group of users and are often used by organizations to manage internal processes and data.

Blockchain components

There are several key components of a blockchain:

1. Blocks: As mentioned earlier, a block is a unit of data that contains a list of transactions and a unique hash code that links it to the previous block.

2. Nodes: A node is a computer or device that participates in the blockchain network and helps to validate and process transactions.

3. Miners: Miners are nodes that perform the computationally intensive task of creating new blocks and adding them to the blockchain. They are typically rewarded with cryptocurrency for their efforts.

4. Consensus algorithm: A consensus algorithm is a set of rules that allows the nodes in a blockchain network to agree on the state of the blockchain. This ensures the integrity and security of the blockchain.

5. Smart contracts: A smart contract is a self-executing contract with the terms of the agreement between buyer and seller being directly written into lines of code. Smart contracts are used to automate complex processes and can be implemented on a blockchain platform.

6. Cryptocurrency: Cryptocurrency is a digital or virtual currency that uses cryptography for secure financial transactions. It is often used as a form of payment or reward for participating in a blockchain network.

That's it all about basics. Now let's get into some hands-on part

How to create Hyperledger Fabric private blockchain

Do you know what is Hyperledger Fabric?

Hyperledger Fabric is a permissioned blockchain platform that is well-suited for enterprise use cases. It allows businesses to build and deploy decentralized applications, or "smart contracts," in a private and secure environment. Hyperledger Fabric uses a modular architecture that allows businesses to choose the components they need and easily customize their blockchain network to meet their specific requirements.

you need a AWS account. Go the Console panel and search AWS Managed blockchain services

Select create private networks option

Select Hyperledger fabric option, version, network edition

Add name, description, Voting policy (default) and tags

Now your chain is created successfully.

You can add an admin member and invite other members

Here, you can create a new vpc endpoint to interact with your blockchain from your application

that's it guys. We created our private blockchain using AWS. Using this VPC endpoint we can interact with blockchain for deploying our smart contract and query details from blockchain

Hope you understood.

Okay. Let's proceed further to create a public blockchain node

How to create Ethereum Public Blockchain?

what is Ethereum public blockchain?

Ethereum is a decentralized blockchain platform that establishes a peer-to-peer network that securely executes and verifies application code, called smart contracts. Smart contracts allow participants to transact with each other without a trusted central authority. Transaction records are immutable, verifiable, and securely distributed across the network, giving participants full ownership and visibility into transaction data. Transactions are sent from and received by user-created Ethereum accounts. A sender must sign transactions and spend Ether, Ethereum's native cryptocurrency, as a cost of processing transactions on the network.

Select join public network option

and select Ethereum Testnet Rinkeby network for testing purposes. Please go with Ethereum mainnet for production

That's it guys. We created our public blockchain node also. We can deploy our smart contrat and interact with chain using Node ID.

Hope you get some insightful knowledge from this blog.

Keep Learning Keep Growing !!!

Community and Social Footprints :

• Veerasolaiyappan

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

👉RSA (Rivest-Shamir-Adleman) is an algorithm used for secure data transmission. It is an asymmetric encryption that is widely used for secure data transmission.
👉In RSA, each person has a pair of keys: a public key and a private key.
👉The public key can be shared with anyone, while the private key must be kept secret.
👉When a message is sent using RSA, the sender encrypts the message using the recipient's public key.
👉The recipient then decrypts the message using their private key.
👉This ensures that only the intended recipient can read the message, as only they have the private key needed to decrypt it.

📌A simple demonstration of how the RSA algorithm works

🌲Suppose Swetha wants to send a secure message to Vijay.
🌲Vijay generates a pair of keys using the RSA algorithm and sends his public key to Swetha.
🌲Swetha uses Vijay's public key to encrypt her message and sends the encrypted message (ciphertext) to Vijay.
🌲Vijay then uses his private key to decrypt the ciphertext and read the original message (plaintext).

📌Key generation:

🌲Vijay selects two prime numbers, p, and q, and calculates n = p *q and in this example, p = 5 and q = 11, so n = 5 *11 = 55.
🌲Vijay calculates a value called the totient, denoted as φ(n), which is the number of positive integers less than n that are relatively prime to n. In this case, φ(55) = (p - 1) (q - 1) = 4 *10 = 40.
🌲The factors of 40 are 2*2*2*5.
🌲Vijay selects a public key, e, that is relatively prime to φ(n) and none of the factors of 2 and 5. In this example, let's say e =7.
🌲Vijay calculates his private key, d, such that e d ≡ 1 (mod φ(n)) or we can derive d= (1+ x φ(n))/e while x can be 0,1,2, 3, etc.
After some calculations using Excel, d is calculated as d = (1+4*40)/7 = 23.

🌲We now have n=55, e=7, d=23
🌲Vijay's public key is the pair (e, n), and his private key is the pair (d, n). Vijay sends his public key to Swetha.

📌Encryption:

🌲Swetha wants to send the message "HELLO" to Vijay.
🌲She converts the message using a predetermined scheme (e.g., A=1, B=2, till Z=26.).
🌲Swetha calculates the ciphertext, c, using the formula c ≡ m^e (mod n), where m is the plain text.
🌲In this case, c is calculated as c = 8^7 (mod 55) 5^7 (mod 55) 12^7 (mod 55) 12^7 (mod 55) 15^7 (mod 55).
🌲c= 2 25 23 23 5. Swetha sends the ciphertext to Vijay.

📌Decryption:

🌲Vijay receives Swetha's ciphertext, 2 25 23 23 5.
🌲Vijay calculates the plaintext, m, using the formula m = c^d (mod n).
🌲In this case, m = 2^23 (mod 55) 25^23 (mod 55)23^23 (mod 55)23^23 (mod 55)5^23 (mod 55) = 8 5 12 12 15.
🌲Vijay converts the number back to the message "HELLO" using the predetermined scheme.
👉The link for the calculation is a calculator.

🎯Below are the screenshots of the Python code used to create the RSA public, private, encryption, and decryption algorithms.

👉Please refer to the link to generate RSA keys using python RSA Keygen

🎯The Public and Private key Generation

🎯Encryption Algorithm

🎯Decryption Algorithm

🌲Strengths
👉When compared to symmetric encryption, there is no need to exchange the keys ahead of time.
👉"Non-repudiation" is achieved because data cannot be changed during communication.
👉It's a one-way function, so knowing one prime key won't get you the other primes.

🌲Weakness
👉The difficulty of generating keys.
👉The RSA algorithm is relatively slow when compared to symmetric algorithms.

• Community and Social Footprints:

  • Swetha Mudunuri

  • GitHub

  • Twitter

  • YouTube Cloud DevOps Free Trainings

  • Linkedin Page

  • Linkedin Group

  • Discord Channel

  • Dev

1. Create a new container using the below command

sudo docker run -it ubuntu /bin/bash

• The "docker run" command provides all launching capabilities for docker to create a container. We use docker run to create new containers.

• -i: open STDIN from the container

• -t: tells docker to assign a pseudo-tty to the container

• -it : provides an interactive shell

• ubuntu: ubuntu is an image and also called as "stock image" or "base image". This image will be downloaded from Docker hub when we run 'docker run' command first time

• /bin/bash: 'shell program' that will be installed in the terminal

2. Inspect the new container

Let's believe that it's a separate machine by using the below commands

1. hostname

2. cat /etc/hosts

3. hostname -i

4. ps -ef

3. SSH setup for containers

Setup SSH in the containers so that they can communicate with each other

1. Create two containers having IP addresses - 172.17.0.2, 172.17.0.3

2. Try to connect to 172.17.0.3 from 172.17.0.2 using the below command. You will get an error.

   $ ssh demo@172.17.0.3 (you won't be able to connect by default)

3. Install ssh server

   $ apt-get update

   $ apt-get install openssh-server

4. Start the server

   $ service ssh start (status/stop/restart)

5. Create a user and set up a password

   $ useradd -m -d /home/demo -s /bin/bash demo

   $ passwd demo

6. Connect to the container using ssh from 172.17.0.2 or any other machine.

   $ ssh demo@172.17.0.3

7. Enable root user over ssh

   Add the below line under "# Authentication:" in "/etc/ssh/sshd_config" PermitRootLogin yes

4. Shutdown a container

"exit" to stop the container

5. Log in to a stopped container

$docker start

$docker attach

6. List all containers(stopped and running)

$ docker container ls -a

$ docker ps -a

7. List given no. of containers

$ docker ps -a -n1

8. List running containers only

$ docker container ls

$ docker ps

9. List Stopped containers only

$ docker container ls -f status=exited (Where Status can be exited/running)

The "docker container ls" command output shows:

- Image name from which container is created

- ID - the container can be identified using short UUID, longer UUID Or name.

- Status of the container (Up / Exited)

- Name of the container

10. Show the last container which you have created (stopped/running)

$ docker container ls -l

11. Naming the container

$ docker run --name demo -it ubuntu /bin/bash

Note: Two containers can't have the same name.

12. Rename a container

$ docker rename container_name_1 container_name_2

13. Deleting a container by giving its name or ID

$ docker rm ID/name

14. Delete all (running/stopped) containers at once

$ docker rm -f $(docker container ls -a -q)

$ docker rm -f $(docker ps -a -q)

15. Delete running containers only

$ docker rm -f $(docker container ls -q)

$ docker rm -f $(docker ps -q)

16. List stopped containers only

$ docker container ls -a -f status=exited

17. Starting a stopped container

$ docker start container_name

18. Attaching to a running container

$ docker attach container_name (OR)

$ docker attach b1b1c8dc1939

19. Run a Linux command remotely in a container Or Get an independent terminal from a container remotely (from the Host)

$ docker exec -it tomcat-server ps -ef

20. Stopping a container from the 'host machine'

$ docker stop container_name(Gracefully stop the container)

$ docker kill container_name(Forcibly stop the container)

21. Inspecting the container's processes from the host machine

$ docker top container_name

22. Show the last 4 containers (stopped/running)

$ docker ps -n4

23. Create a container in a background mode ( without terminal access )

$ docker run -it -d ubuntu /bin/bash

24. Find More About The Container

The 'docker inspect' command interrogates the container and returns complete information about it.

Ex: image name, IP, Memory details, hostname ..etc

Examples:

$ docker inspect container_name

$ docker inspect -f '{{.Config.Hostname}}' container_name

$ docker inspect -f '{{.NetworkSettings.Networks.bridge.IPAddress}}' container_name

Note: use --format (OR) -f

25. List all container names

$ docker inspect --format "{{.Name}}" $(docker ps -a -q) | tr -d '/'

2. STATS:

1. Display usage statistics of a container

$ docker stats --no-stream container_name

$ docker stats --no-stream --all

$ docker stats --no-stream --format {{.MemUsage}} container_name

$ docker stats --no-stream --format {{.CPUPerc}} container_name

2. Allocating memory for a container (below command allocates 1 GB RAM)

$ docker run -it --name container_name -m 1g ubuntu /bin/bash

$ docker run -it --name container_name -m 1024m ubuntu /bin/bash

3. Updating memory of an existing container

$ docker update -m 2024m container_name

4. CPU Allocation

$ docker run -it --cpus="2" --name container_name ubuntu /bin/bash

$ docker update --cpus="2" conatiner_name

Community and Social Footprints :

• Rajiv Ravi

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

Following are the three ways to run the Prometheus

1. Script.

2. Systemd service.

3. Docker container.

   This blog will cover only two methods run as a script and systemd service.

Environment used:

RAM=2GB

CPU=2cores

Disk=20GB free

OS= RHEL9

1. Run as a script

Prometheus supports multiple Operating Systems so, we can download the Prometheus files as per the OS and architecture.

1. Download the Prometheus using wget command.

   Download Prometheus

   https://github.com/prometheus/prometheus/releases/download/v2.40.5/prometheus-2.40.5.linux-amd64.tar.gz

2. Extract the TAR file.

    tar -xvf prometheus-2.40.5.linux-amd64.tar.gz
   

3. Run Prometheus

./prometheus

1. Access the Prometheus UI

   http://IPaddress or hostname:9090/

   example: http://192.168.0.102:9090/

2. Run as a systemd service.

1. Download the Prometheus using wget command.

   https://github.com/prometheus/prometheus/releases/download/v2.40.5/prometheus-2.40.5.linux-amd64.tar.gz

2. Extract the TAR file.

    tar -xvf prometheus-2.40.5.linux-amd64.tar.gz
   

3. Create user, directories and change ownership.

    useradd --no-create-home -s /bin/false prometheus
    mkdir /etc/prometheus       ## configuration file
    mkdir /var/lib/prometheus   ## libraries 
    chown prometheus:prometheus /etc/prometheus
    chown prometheus:prometheus /var/lib/prometheus
   

4. copy extracted files to /var/lib/prometheus

    cp -r prometheus-2.40.5.linux-amd64/* /var/lib/prometheus/
   

5. Change ownership.

    chown -R prometheus:prometheus /var/lib/prometheus
   

6. Move config file to /etc/prometheus

    mv /var/lib/prometheus/prometheus.yml /etc/prometheus/
   

7. Check configuration file

    grep -v '#' /etc/prometheus/prometheus.yml
   

8. Create a symbolic link for Prometheus at /usr/bin directory to make it globally executable from any path.

     cp -s /var/lib/prometheus/prometheus /usr/bin
     cp -s /var/lib/prometheus/promtool /usr/bin
   

9. Create Systemd Service Unit:-

    vim /usr/lib/systemd/system/prometheus.service
   
    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
   
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    ExecStart=/usr/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/var/lib/prometheus/consoles \
    --web.console.libraries=/var/lib/prometheus/console_libraries
   
    [Install]
    WantedBy=multi-user.target
   

10. Configure the firewall if you have enabled it.

    firewall-cmd --permanent --add-port=9090/tcp
    firewall-cmd --reload
    

11. Enable and start service.

    systemctl enable --now prometheus.service
    

12. Access the prometheus UI

    http://IPaddress or hostname:9090/

    example: http://192.168.0.102:9090/

Follow the each and every steps carefully to successfull installation.

Prerequisites

To complete this walkthrough, you need:

• An IAM account or IAM Identity Center to sign in to Studio. For information, see Onboard to Amazon SageMaker Domain.

• Permission to use SageMaker-provided project templates. For information, see SageMaker Studio Permissions Required to Use Projects.

• Basic familiarity with the Studio user interface. For information, see Amazon SageMaker Studio UI Overview.

Topics

• Step 1: Create the Project

• Step 2: Clone the Code Repository

• Step 3: Make a Change in the Code

• Step 4: Approve the Model

• (Optional) Step 5: Deploy the Model Version to Production

• Step 6: Clean Up Resources

Step 1: Create the Project

In this step, you create a SageMaker MLOps project by using a SageMaker-provided project template to build, train, and deploy models.

To create the SageMaker MLOps project

1. Sign in to Studio. For more information, see Onboard to Amazon SageMaker Domain.

2. In the Studio sidebar, choose the Home icon ( ).

3. Select Deployments from the menu, and then select Projects.

4. Choose Create project.

   The Create project tab appears.

5. If not selected already, choose SageMaker templates, then choose MLOps template for model building, training, and deployment.

6. For Project details, enter a name and description for your project.

When the project appears in the Projects list with a Status of Create completed, move on to the next step.

Step 2: Clone the Code Repository

After you create the project, two CodeCommit repositories are created in the project. One of the repositories contains code to build and train a model, and one contains code to deploy the model. In this step, you clone the repository to your local SageMaker project that contains the code to build and train the model to the local Studio environment so that you can work with the code.

To clone the code repository

1. In the Studio sidebar, choose the Home icon ( ).

2. Select Deployments from the menu, and then select Projects.

3. Select the project you created in the previous step to open the project tab for your project.

4. In the project tab, choose Repositories, and in the Local path column for the repository that ends with modelbuild, choose clone repo....

5. In the dialog box that appears, accept the defaults and choose Clone repository.

   

   When clone of the repository is complete, the local path appears in the Local path column. Choose the path to open the local folder that contains the repository code in Studio.

Step 3: Make a Change in the Code

Now make a change to the pipeline code that builds the model and check in the change to initiate a new pipeline run. The pipeline run registers a new model version.

To make a code change

1. In Studio, choose the file browser icon ( ), and navigate to the pipelines/abalone folder. Double-click pipeline.py to open the code file.

2. In the pipeline.py file, find the line that sets the training instance type.

   training_instance_type = ParameterString(
           name="TrainingInstanceType", default_value="ml.m5.xlarge"
   

   Change ml.m5.xlarge to ml.m5.large, then type Ctrl+S to save the change.

3. Choose the Git icon ( ). Stage, commit, and push the change in pipeline.py. Also, enter a summary in the Summary field and an optional description in the Description field.

   

After pushing your code change, the MLOps system initiates a run of the pipeline that creates a new model version. In the next step, you approve the new model version to deploy it to production.

Step 4: Approve the Model

Now you approve the new model version that was created in the previous step to initiate a deployment of the model version to a SageMaker endpoint.

To approve the model version

1. In the Studio sidebar, choose the Home icon ( ).

2. Select Deployments from the menu, and then select Projects.

3. Select the name of the project you created in the first step to open the project tab for your project.

4. In the project tab, choose Model groups, then double-click the name of the model group that appears.

   The model group tab appears.

5. In the model group tab, double-click Version 1. The Version 1 tab opens. Choose Update status.

6. In the model Update model version status dialog box, in the Status dropdown list, select Approve, then choose Update status.

   Approving the model version causes the MLOps system to deploy the model to staging. To view the endpoint, choose the Endpoints tab on the project tab.

(Optional) Step 5: Deploy the Model Version to Production

Now you can deploy the model version to the production environment.

Note

To complete this step, you need to be an administrator in your Studio domain. If you are not an administrator, skip this step.

To deploy the model version to the production environment

1. Log in to the CodePipeline console at https://console.aws.amazon.com/codepipeline/

2. Choose Pipelines, then choose the pipeline with the name sagemaker-projectname-projectid-modeldeploy, where projectname is the name of your project, and projectid is the ID of your project.

3. In the DeployStaging stage, choose Review.

4. In the Review dialog box, choose Approve.

   Approving the DeployStaging stage causes the MLOps system to deploy the model to production. To view the endpoint, choose the Endpoints tab on the project tab in Studio.

Step 6: Clean Up Resources

To stop incurring charges, clean up the resources that were created in this walkthrough. To do this, complete the following steps.

Note

To delete the AWS CloudFormation stack and the Amazon S3 bucket, you need to be an administrator in Studio. If you are not an administrator, ask your administrator to complete those steps.

1. In the Studio sidebar, choose the Home icon ( ).

2. Select Deployments from the menu, and then select Projects.

3. Select the target project from the dropdown list. If you don’t see your project, type the project name and apply the filter to find your project.

4. You can delete a Studio project in one of the following ways:

   1. You can delete the project from the projects list.

      Right-click the target project and choose Delete from the dropdown list.

   2. You can delete a project from the Project details section.

      1. When you've found your project, double-click it to view its details in the main panel.

      2. Choose Delete from the Actions menu.

5. Confirm your choice by choosing Delete from the Delete Project window.

   This deletes the AWS Service Catalog provisioned product that the project created. This includes the CodeCommit, CodePipeline, and CodeBuild resources created for the project.

6. Delete the AWS CloudFormation stacks that the project created. There are two stacks, one for staging and one for production. The names of the stacks are sagemaker-projectname-project-id-deploy-staging and sagemaker-projectname-project-id-deploy-prod, where projectname is the name of your project, and project-id is the ID of your project.

   For information about how to delete a AWS CloudFormation stack, see Deleting a stack on the AWS CloudFormation console in the AWS CloudFormation User Guide.

7. Delete the Amazon S3 bucket that the project created. The name of the bucket is sagemaker-project-project-id, where project-id is the ID of your project.

Conclusion

I hope this blog has been helpful. I'll see you in the next blog.

Community and Social Footprints :

• Sampath Kumar Basa

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

In this episode, I would like to define the problem statements & Agile overview. The next episode will focus on solution architecture and extensive story building methodologies to achieve the desired outcome.

This one in particular, is my favorite episode as it sets up the tone to the entire series.

Before I commence, I would like to thank the product owners, business analysts, Agile coaches, and scrum masters with whom I have worked throughout my career and the amount of knowledge I have gathered with numerous personals with similar interests, during coffee catch-ups and external meet-ups over a period of time.

From now on, the key focus is more on problem statements and solutions. To keep it more interactive, I would request the readers to kindly comment and ask questions wherever they see fit.

Overview on Agile

Before getting into more details, I would like to give a brief introduction to some of the agile jargons

The hierarchy of Agile works like this: Epic -> Features -> User Stories/Backlog Items -> Tasks

An Epic is a big chunk of requirement that business wants to accomplish.

A Feature is a subdivision of an Epic which will define the MVP business wants to achieve. MVP is nothing but a minimum or most viable product that can be shipped or can be used by Business.

An User story or a Backlog Item will be smaller pieces of a feature breakdown, which will help shape up or complete a feature.

A real world example can be:

Epic - Business wants to build a car.

1. Feature#1 – Build the Interior Components

   • Story#1 – Design the car seat layout

   • Story#2 – Design the car seat belt mechanism

   • Story#3 – Design the color of the car seat

   • Story#4 – Design the floor mat

2. Feature#2 – Build the exterior Components

   • Story#1 – Design the car locking mechanism

   • Story#2 – Design the car’s wiper system

   • Story#3 – Design the color of the car

   • Story#4 – Design the roof antenna system

3. Feature#3 – Build the Electrical system/components

4. Feature#4 – Build the Engine Components

Features can be worked in parallel and integrated. Better control, flexibility, superior quality, continuous improvement, efficiency, and high team morale are some of the key benefits of working in an Agile driven project.

Now that we’ve got ourselves familiarized with the Agile Jargons, it is time we see some examples from risk, compliance & KYC perspective.

Real time use case

In the below section(s), I have defined a few Epics, from a business point of view, with the assumption that you will be having discussions with your respective AML and CTF, Marketing and Campaign and KYC teams.

• Anti-Money Laundering and Counter-Terrorism Financing

As the Transaction due diligence manager,

I want to track transactions real time which meets the following two criteria:

a. Cash withdrawal using credit card

b. Money transfer to ultra-high- or high-risk countries as per FATF (Financial Action Task Force)

So that we can mitigate Fraud, AML and CFT activities.

• Marketing and Campaign

a. As the marketing and campaigns manager,

I want to identify the customers who were on-boarded in the last 3 months, who can be potential candidates and are open for discussions about new product and services through phone calls or emails,

So that these customers will benefit from new product offers and recommendations, thus improving the overall customer experience.

b. As the marketing and campaign manager,

I want to list customers who closed their accounts in the last 3 months

So that we can reach out to seek their feedback on the reason for closure

• Know Your Customer

As the KYC Country lead,

I want to find out how many (summary) and detailed customers missing residential, mailing, and digital (mobile and email id) addresses which are critical for any customer due diligence process,

So that a DQ remediation plan can be worked out by an appropriate team to help improve the DQ and thus the E2E process itself.

We now have a very good understanding of the source system (from Episode 2), and problem statements are well defined with the current episode.

The original plan was to have the problem statement and solution architecture covered in a single episode.

However, I realized it would become cumbersome to cover them both in one go.

Hope you enjoyed this episode and got a better understanding of how Agile works by going through some of the real time use cases.

In the next episode we will focus more on solution design architecture and write the corresponding user stories to fulfil the required business requirements.

In case you have missed out on my previous episodes refer to https://lnkd.in/giq5YPku

Community and Social Footprints :

• Srinath Babu Kunka Suburam Dwaraganath

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

By default your application running in the pods are not available for outside world in order to make your application available to outside services are being used which routes the traffic to container into the pod. Services is a mechanism which exposes you pod on a network. There are three service which are used :

• NodePort

• ClusterIP

• LoadBalancer

NodePort

In this type of service you are allowing the external traffic by opening TCP port of your worker node and via kube-proxy which available in all node will proxy requests from the TCP port to the pod on this node. Behind it will create the ClusterIP which will route the traffic from this ClusterIP to the port. Internally load balancing will performed to divert traffic to different pods in the node.

NodePort Type:

• Will be tied up with you host eg: EC2.

• Depends on host, if host is not available then this service won't work.

• it will provide access to the pod only to same worker node.

• NodePort will allocate the port the port range 30000-32767

NodePort Manifest File:

apiVersion: v1
kind: Service
metadata:
  name: mysvc
spec:
  ports:
  - port: 80
    nodePort: 30123
  selector:
    app: myapp
  type: NodePort

Now you can simply apply the above YAML file to create service. To check the service type :

kubectl get svc mysvc

ClusterIP

This is the default type of service when you create any service and if yo didn't specify any type ClusterIP will be the default allocated but this will be available internally to the pod to communicate with each other. Application can internally communicate within cluster without any access from outside world.

It will use IP's from the IP-pool and will be accessible via a DNS-name in the cluster’s scope.

ClusterIP Manifest file:

apiVersion: v1
kind: Service
metadata:
  name: mysvc
spec:
  ports:
  - port: 80
  selector:
    app: myapp
  type: ClusterIP

Here type is optional even if you won't specify it will create ClusterIP default.

Yo can apply the above yaml file and Check:

kubectl get svc mysvc

LoadBalancer

This is the most common used service but this can used only with the managed Kubernetes like GKS, AKS, EKS. In case of AWS it will create a classic load balancer in which it will route the traffic to the EC2 node and via Nodeport service to all pods. There’s no automatic filtering or routing. Traffic to the external IP and port will be sent straight to your service. This means that they’re suitable for all traffic types.

LoadBalancer Type:

• Will provide external access to pod.

• provides the load balancing to the nodes.

LoadBalncer Manifest File:

apiVersion: v1
kind: Service
metadata:
  name: mysvc
spec:
  ports:
  - port: 80
  selector:
    app: myapp
  type: LoadBalancer

If you are not using any managed Kubernetes and creating service with loadbalancer then it will simply create the service and won't show any error but it can't be used.

By applying the above file you will get external IP form the cloud provider IP pool which you can access. To check the service created :

kubectl get svc mysvc

FAQs

• What is the difference between NodePort and LoadBalncer ?

• Does ClusterIP have LoadBalnce ?

The ClusterIP provides a load-balanced IP address. One or more pods that match a label selector can forward traffic to the IP address. The ClusterIP service must define one or more ports to listen on with target ports to forward TCP/UDP traffic to containers.

• Does LoadBalancer use NodePort?

The service can then be accessed through the IP address provided by the Cloud Service load balancer, which will route the request to a NodePort and from there forwarded to a ClusterIp. So, LoadBalancer builds upon NodePort and ClusterIp.

Conclusion

ClusterIPs, NodePorts, LoadBalncers route the external traffic to your pod in the cluster. Each one has its own different use-cases. They enable the network access to your services make them publicly accessible.

Community and Social Footprints :

• Shubh Dadhich

• GitHub

• Twitter

• YouTube Cloud DevOps Free Trainings

• Linkedin Page

• Linkedin Group

• Discord Channel

• Dev

The Azure Storage platform is Microsoft's cloud storage solution for modern data storage scenarios. Azure Storage offers highly available, scalable and secure storage for a variety of data objects in the cloud. Azure Storage data objects are accessible from anywhere in the world over HTTP or HTTPS via a REST API. Azure Storage Explorer provide user-interface tools for interacting with Azure Storage.

Azure Storage data services:

• Azure Blobs: scalable object store for text and binary data.
• Azure Files: Managed file shares for cloud or on-premises deployments.
• Azure Queues: A messaging store for reliable messaging between application components.
• Azure Tables: A NoSQL store for schema less storage of structured data.
• Azure Disks: Block-level storage volumes for Azure VMs. Each service is accessed through a storage account.

Storage Types Available in Azure:

Blob storage:

Azure Blob storage is Microsoft object storage solution for the cloud. Blob storage is optimized for storing unstructured data, such as text or binary data. Blob storage is ideal for:

• Serving images or documents directly to a browser.
• Storing files for distributed access.
• Streaming video and audio.
• Storing data for backup and restore, disaster recovery, and archiving.

Blob Types:

• Block blobs: store text and binary data. Block blobs are made up of blocks of data that can be managed individually. Block blobs can store up to about 190.7 TiB.
• Append blobs: are made up of blocks like block blobs, but are optimized for append operations. Append blobs are ideal for scenarios such as logging data from virtual machines.
• Page blobs store random access files up to 8 TiB in size. Page blobs store virtual hard drive (VHD) files and serve as disks for Azure virtual machines.

Objects in Blob storage can be accessed from anywhere using HTTP or HTTPS.

Azure Files:

Azure Files used for network file shares that can be accessed by using the standard Server Message Block (SMB) protocol. That means that multiple VMs can share the same files with both read and write access. One thing that distinguishes Azure Files from files on a corporate file share is that you can access the files from anywhere using a URL that points to the file and includes a shared access signature (SAS) token. You can generate SAS tokens; they allow specific access to a private asset for a specific amount of time.

File shares can be used for many common scenarios:

• Many on-premises applications use file shares. This feature makes it easier to migrate those applications that share data to Azure. If you mount the file share to the same drive letter that the on-premises application uses, the part of your application that accesses the file share should work with minimal, if any, changes.
• Configuration files can be stored on a file share and accessed from multiple VMs. Tools and utilities used by multiple developers in a group can be stored on a file share, ensuring that everybody can find them, and that they use the same version.
• Logs, metrics, and crash dumps are just three examples of data that can be written to a file share and processed or analyzed later.

Queue storage:

Azure Queue Storage is a service for storing large numbers of messages. You access messages from anywhere in the world via authenticated calls using HTTP or HTTPS. A queue message can be up to 64 KB in size. A queue may contain millions of messages, up to the total capacity limit of a storage account. Queues are commonly used to create a backlog of work to process asynchronously.

Azure Table:

Azure Table storage stores large amounts of structured data. The service is a NoSQL datastore which accepts authenticated calls from inside and outside the Azure cloud. Azure tables are ideal for storing structured, non-relational data. Common uses of Table storage include:

• Storing TBs of structured data capable of serving web scale applications
• Storing datasets that don't require complex joins, foreign keys, or stored procedures and can be denormalized for fast access
• Quickly querying data using a clustered index
• Accessing data using the OData protocol and LINQ queries You can use
• Table storage to store and query structured, non-relational data, and your tables will scale as demand increases.

Disk storage:

An Azure managed disk is a virtual hard disk (VHD). You can think of it like a physical disk in an on-premises server but, virtualized. Azure-managed disks are stored as page blobs, which are a random IO storage object in Azure. We call a managed disk 'managed' because it is an abstraction over page blobs, blob containers, and Azure storage accounts. With managed disks, all you have to do is provision the disk, and Azure takes care of the rest.

Redundancy:

To ensure that your data is durable, Azure Storage stores multiple copies of your data. When you set up your storage account, you select a redundancy option.

Transfer data to and from Azure Storage.

You have several options for moving data into or out of Azure Storage. Which option you choose depends on the size of your dataset and your network bandwidth.

Migration:

There is several options for migrating data into or out of Azure Storage. Which option you choose depends on the size of your dataset and your network bandwidth. Data transfer can be offline or over the network connection. Choose your solution depending on your:

• Data size - Size of the data intended for transfer,
• Transfer frequency - One-time or periodic data ingestion, and
• Network – Bandwidth available for data transfer in your environment.
• There are several native and 3rd party Tools available to migrate the data. Below are the Azure Native Migration Tools.
• Data Box, Azure Storage Explorer, Data Box Gateway and AZ-Copy.

Community and Social Footprints :

• Sathish Jeyabalan
• GitHub
• Twitter
• YouTube Cloud DevOps Free Trainings
• Linkedin Page
• Linkedin Group
• Discord Channel
• Dev

• Prometheus is an open-source monitoring tool used for event monitoring and alerting.
• Prometheus is written in go language and it is licensed under Apache 2.0 license.
• it is used for infrastructure and customized services monitoring.
• it allows you to analyze how your applications and infrastructure are performing from the metric.
• It uses a multi-dimensional data model with time series data identified by metric name and key values pairs.
• It uses very simple query language PromQL
• To monitor the customized services, you can add instrumentation to your code, via Prometheus client libraries like Go, python, java, Scala.
• This is full-fledged monitoring system with its own alert manager.
• It can handle the millions of metrics ingestions per second.

What are the alternative monitoring tools?

Below are the alternative monitoring tools for Prometheus

1.Graphite

2.influxdb

3.opentsdb

4.nagios

5.sensu

So, question comes why we should go for Prometheus?

It has more feature than any other tools

1. Provides the flexible query language
2. Push gateway for collecting metrics from short lived batch jobs
3. Wide range of exporters are available.
4. 3rd party tools provide endpoint to tools like Graphana.

What can we monitor with Prometheus:

1.Service Metrics

2.Host Metrics

3.Uptime/Status of the website

There are 5 Stages monitoring of Prometheus:

1. Data collection
2. Data storage
3. Alerting
4. Visualization
5. Analytics and monitoring.

What are the components of Prometheus?

1.Monitoring- it is the systematic process of collecting and encoding the activities taking place into target project.

2.Alert/Alerting- Alert is the outcome of an alerting rule in Prometheus that I actively firing. Alert are sent from Prometheus to the alert manager.

3.Alert manager- Prometheus servers generate the alert if thing go out the rules it sends to alert manager The job of the alert manager is grouping that alert and apply filter on them and send it via email, pagerduty, slack.

4.Target- Target is an object whose metrics are to be monitored. Target can be anything windows, Linux, or own application.

5.Instance- A endpoint you can scrape is called an instance.

Ex. 3.4.9.5:8790, 8.4.5.9:5678

Instance denoted after the colon, means 8790 and 5678 are instance.

6.Job:- Job is a collection of targets /instance.

7.Sample:- Sample is a single value at a point in the time series.

Architecture of Prometheus

Prometheus server

Prometheus servers collect the multi-dimensional data in time series and then analyze and aggregates the collected data, this process of collecting metrics is called scraping. It pulls the metrics automatically from the targets. Hence user don’t need to push the metrics for analysis. The only we need to do expose metrics such that Prometheus can access them. For that we must create the HTTP endpoint with metrics which returns the complete metrics.

1. Retrival: - it helps to retrieve the data from the application endpoint.

2. TSDB: - Stores the metrics data so that alter it can be retrieved and analyzed.

3. HTTP Server: - HTTP server is responsible to push the collected data to the dashboard.

Pull Metrics

Prometheus pull the data from targets via pull method and store on the TSDB.

Pushgateway

Prometheus works on pull base model but some of the component metric cannot be pulled. Short lived job cannot be scarped through pulling metric. For monitoring these job Prometheus uses push gateway. once the push gateway fetches the metrics from Prometheus use pull method for scraping. Once push gateway push the data Prometheus server the get the data via pull method.

Service Discovery

Discovers the targets.

Alert manager

Prometheus servers push the alerts to alert manger, alert manger apply the filter on it send it via email, slack.

HTTP Server

HTTP server fetch the metrics from TSDB and push the collected metrics to the dashboard.

Kubernetes is a widely used container orchestration tool. It has its own pros and cons but having a solid understanding of how it works is going to help you a lot to stay with technology trends in the cloud native space. Spend some time getting yourself familiarized and you would not regret it.

Prerequisites:

1. Virtual Machine 1 (named as MASTER) with 2cpu and 4 GB RAM Ubuntu Machine
2. Virtual Machine 2 (named as WorkerNode1) with 2cpu and 4GB RAM Ubuntu Machine
3. Virtual Machine 3 (named as WorkerNode2) with 2cpu and 4GB RAM Ubuntu Machine

Once we have the 3 Machines set up, the next step is to install the requisite packages and software on the Machines so they are ready to function as a Kubernetes Cluster.

COMMANDS TO BE RUN ON ALL 3 MACHINES UNLESS STATED OTEHRWISE

INSTALL DOCKER ENGINE ON ALL NODES:

apt-get update && apt-get install -y apt-transport-https curl

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -

cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
> deb https://apt.kubernetes.io/ kubernetes-xenial main
> EOF

apt-get update

apt-get install -y kubelet kubeadm kubectl

(TO BE RUN ONLY ON THE WORKER NODES):

apt-mark hold kubelet kubeadm kubectl

INSTALL DOCKER PACKAGES ON ALL THE NODES

As a root user execute the below commands on master and worker nodes

sudo apt-get remove docker docker-engine docker.io containerd runc

sudo apt-get update

sudo apt-get install     ca-certificates     curl     gnupg     lsb-release

sudo mkdir -p /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin

mkdir -p /etc/systemd/system/docker.service.d

systemctl daemon-reload
systemctl restart docker
systemctl enable docker

docker -v

rm /etc/containerd/config.toml

systemctl restart containerd

INITIALIZATION OF KUBE MASTER: TO BE RUN ONLY ON THE MASTER NODE

As a root user on the master node, execute the below commands

kubeadm init --apiserver-advertise-address $(hostname -i) --pod-network-cidr=192.168.0.0/16

Once we have run the above command, it produces output that contains the information about the kubernetes cluster and the information has to be kept safe and secure.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export kubever=$(kubectl version | base64 | tr -d '\n')

(Kube Proxy Addon Installation):

kubectl apply -f https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml

kubectl get nodes

When we run the kubectl get nodes on the Master Machine, the output just displays the master machine.

To add the worker nodes on to the kubernetes cluster, run the output from the kubeadm init command on the worker nodes as shown below

Now when we run the “kubectl get nodes” on the master node, we will be able to see the master and workernode1 listed as nodes to the kubernetes cluster. The same has to be repeated on worker node 2 to join to the cluster.

So why would should we try setting this up manually when there are lot of managed kubernetes instance provided by cloud providers. Setting up the kubernetes cluster help us understand the various components that are installed while we run the kubeadm init command and how a node is joined as worker to the master. This helps in creating a Mind Map that retains in the memory for a longer time than while reading.

If you are someone who would like to fork a github repo and try out the commands, please refer to

https://github.com/cloudnloud/Kubernetes_Admin_Training/blob/main/class3-k8s-installation/installation.md

If video format makes it impressive, please refer to

https://www.youtube.com/watch?v=md2BtnJYtt8&list=PLh_VNk4-EHTMhIR-NIgI4tCEHdO9U-A8F&index=3

Happy Learning!!!

Please feel free to post any queries/clarifications.

Community and Social Footprints :

• Vijayalakshmi Bakthavachalam
• GitHub
• Twitter
• YouTube Cloud DevOps Free Trainings
• Linkedin Page
• Linkedin Group
• Discord Channel
• Dev

Deposits Accounts – Surplus from individual customers as savings and pay interest.

Loan / Credit Accounts – Banks offer loans or credit to individual customers and earn interest.

Cash management – Banks offers variety of services to manage and transact their money. For instance – ATM, Cards, UPI, Online Transfers etc.

Note – Retail banking is vast, and in this blog series am trying to narrow down and build a use case so that I can explain in detail the architecture, design principles, data model, data engineering, governance and visualization concepts E2E

Retail Banking in Data perspective

Accounts Details

Customer Details

Transaction Details

Assume Mr. and Mrs. Dhoni walks to nearby bank, below are step by step process:

Bank verifies the customers, request for identity documents, does background checks and generates CIF (Customer Information File). Understand the customer needs and recommends products as per needs of customer, for instance,

As per above table,

• Mr and Mrs Dhoni have opened a personal saving account for day-to-day transactional activities.
• Mr and Mrs Dhoni have opened a joint term deposit account, to lock an amount of money for an agreed length of time (the ‘term’) to get a guaranteed rate of interest for the term selected. From data perspective, we understand that Customer Account is many to many relationship

Mr. Dhoni now wants to transfer money to his friend Mr. Sachin hence adds a payee. Payee is related with Mr Dhoni.

Mr Dhoni performs the following tasks,

1. Deposits cash into his newly created account “A1”
2. Performs a payment from his account to newly added payee.
3. Payee, can be to account details (or) mobile (or) email-id etc.

Payment table captures from customer, from account, payee and amount transferred details.

Both cash deposit and account to account transfer are two separate transactions, entries are made in Transaction table.

While going through data model, understand, considering below points :

• Please keep in mind retail banking is a vast subject. Have tried my best to make data model easy to understand and almost look complete from data engineer perspective.

• Customer can originate through mobile app, customer care (telephonic), branch, web site, or others (broker) etc.

• Data is stored in database and data modelling plays a critical part in data management, governance, and intelligence. Have defined normalized, simple, well defined, and organized data model.

• Relationship between the tables. Well defined primary and foreign keys.

• Audit fields for monitoring activities and compliance.

• For CUSTOMER, limiting my scope to only “INDIVIDUALS”. Other possible CUSTOMER_PARTY_TYPE “ORGANIZATION”, “VISITOR” etc. and the data model can expand by addition of new customer party.

• For ACCOUNT, limiting my scope to only “SAVINGS”, “PERSONAL LOAN”, “CREDIT ACCOUNT” (Credit card account) and “MORTGAGE” (Home Loan).

• Finally, the Transaction and Product are other entities.

I understand the content is theoretical but please spend sometime and understand. In upcoming episodes when we start tasks technically, good understanding of source system and requirements will make it easier and the process enjoyable.

Upcoming episode, we will play architect role build data pipeline and understand, how the data flows from source data model (OLTP) to data lake and then into data warehouse (OLAP). Also, will define real time use cases from perspective of KYC, marketing and campaign and Anti-money Laundering teams which is common to most of banks.

In case you have missed out on my previous episodes refer to https://lnkd.in/gk5DuJbT

Community and Social Footprints :

• Srinath Babu Kunka Suburam Dwaraganath
• GitHub
• Twitter
• YouTube Cloud DevOps Free Trainings
• Linkedin Page
• Linkedin Group
• Discord Channel
• Dev

is easy-to-use object storage with a simple web service interface that you can use to store and retrieve any amount of data from anywhere on the web. Amazon S3 also allows you to pay only for the storage you actually use.

Advantage of Amazon S3

Create Buckets. Store data in Buckets. Download data. Permissions. Standard interfaces.

Creating 2 Buckets

Services => Storage => S3 => Create Bucket Bucket Name => Bucket1 & Bucket2 (Bucket Name should be unique globally).

regions => Mumbai & Singapore.

Next => Next => uncheck block public access => Next=>Create bucket.

Uploading Object into Bucket

click on the test bucket => Upload => Add files => select any image =>Next => Manage public permissions => Grant public read access=>Next =>Next=> Upload

Note: As bucket is public, and object is also public, anyone in the world can access the content. Click on the Object=> Get Object URL
Using Object URL, anyone can access.

Features of S3

1. Versioning
2. Static website hosting
3. storage classes/ tiers
4. Cross region replication (CRR )
5. Transfer Acceleration
6. encryption
7. Metadata and Tags
8. ACL & Bucket policies
9. Life cycle management

1. Versioning is a means of keeping multiple variants of an object in the same bucket.

Create new bucket Bucket Name=> ( bucket1 ) => Region - Mumbi =>Next=>Next =>un check Block all public access => Next => create bucket.

enable versioning

Click on the bucket =>Properties tab( Observation: By default version is disabled ) =>Edit =>Enable => Save Changes.

upload one object

Upload the file from Desktop=>Next => Grant public access => Next => Next =>Upload

Advantage of versioning

- recover deleted object.

Delete the object => (Select the check box=> Actions=> Delete
Recover the object => Enable list version We can see the object and its delete marker. select the delete marker check box=> Actions=>Delete=> Delete =>Disable list version.

Note: When we delete, object is not deleted. It is marked as deleted. So, If you remove the delete marker, We will get the object.

- We can maintain different versions of the file.

Upload the same file again. Get the object URL, and check from browser, we get the latest file. Even if you delete the file, we can recover both the versions. Select the object => actions => delete => delete

select show button We can see both the versions of the file.

2. Static website hsoting

Bucket name - bucket1 =>Next =>Next => uncheck block all public access =>Next=>Create bucket.

Select the bucket=>Properties => Static website hosting=>Edit => Enable => Host a static website => index document - index.html error document - error.html Save

Upload index.html and error.html=> Next=> Next=> Next => Upload

Now, go to the properties of the bucket => Static website hosting => get URL of the website ( endpoint )

Note: Individual files should have public access.

What is the use of error.html ?

Incase of any reason,if index.html is not accessible then error page will be displayed.

Lets make the index.html page as private. select index.html=>ACL =>Edit => public access => read => uncheck => Save Changes

Now, refresh the URL, we get error.html page!!!

• Delete the files => Delete the bucket.

3. storage classes/ tiers

Amazon S3 offers a range of storage classes that you can choose from based on the data access, resiliency, and cost requirements of your workloads.

S3 Storage Classes can be configured at the object level, and a single bucket can contain objects stored across S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA.

4. Cross region replication (CRR )

Lets say, we have two buckets ( 1st bucket in Mumbi & 2nd bucket in Sidney)

We we upload the object in Mumbi, the object should also be available in Sidney).

As we are replicating an object in another region, it is called cross region replication.

vice-versa will not happen.

If we delete object in Mumbi, it will not be deleted in Sidney. If we edit object in Mumbi, it will not de edited in Sidney.

Lets create bucket bucket name - Mumbi-bucket Region - Mumbi=>Next => Next => uncheck block all public access => Next => create bucket.

Lets create 2nd bucket in Sidney Lets create bucket bucket name - Sidney-bucket Region - Sidney Next =>Next => uncheck block all public access =>Next=>create bucket.

Enable cross region replication in Mumbi bucket

Select Mumbi bucket => Management => Replication Rules =>Create Replication Rule => Enable Bucket versioning => Replication Rule Name - CRR1

Destination bucket =>Sydney bucket => Enable versioning=>IAM Role=>

( TO establish connection between two regions, we need role )

IAM Role - Create new role => Save.

Now, lets upload object in Mumbi bucket, it will be replicated in Sydni bucket!!!!

5. Transfer Acceleration

When we enable transfer acceleration, data will be transferred to edge location and then from edge location data will be transferred to bucket. ( Look at the image )

Select Mumbi bucket => Properties=> Transfer acceleration =>Edit=> Enabled => Save => Changes.

6. encryption

There are two types of encryption

• AES - 256 ( Advanced Encryption standard ) - Single encryption
• AWS - KMS ( Key management service ) - Double encryption ( More secured )

Select the required encryption.

Select the bucket => Properties => Default Encryption => Edit => Enable

7. Metadata and Tags

Metadata => To provide more information about the object in key-value pairs. Keys are predefined. eg: Content-type, Content-language etc

Tags => To provide more information about the object in key-value pairs. Keys and values we need to provide.

Select the object => Properties, we can see the metadata and tags.

8. Access Control List & Bucket policies

Select the bucket => Permissions tab => ACL Edit=>Add grantee Enter canonical ID => Save Changes

Note: ACL we can apply at bucket level and object level Select the Object and provide the access by entering canonical ID

Note: Bucket policy, we can apply only to bucket.

Select the bucket => Permission , We can see bucket policy. Bucket Policy are written in JSON Code.

Bucket policy should be defined in JSON code. Its the job of AWS administrator.

+++++++++++ Select any object => Permissions tab Observe: We do not have bucket policy. As bucket policy, we need to apply at bucket level only.

9. Life cycle management

Lets create a new bucket

Select the bucket => Management tab => Create lifecycle rule

Rule name - Myrule This rule applies to all objects => I Acknowledge Transit current version of objects between storage classes

Standard 1A =>30 Days
Add transition

One Zone-IA => 60 Days

Create Rule

From now, any object uploaded in the bucket will follow the rule for transition.