ClickOps and AWS CloudTrail
Remember the last time your colleague modified a resource directly in the AWS console, which led to a production issue?
Instances like these highlight why amending cloud resources directly through the AWS console, commonly called “ClickOps,” is not the best practice.
If you haven’t encountered such an issue yet, that’s great! But remember, ClickOps can be likened to riding a motorcycle—you either have had an accident or will have one in the future.
In this blog we will review AWS CloudTrail and how to fight ClickOps with best Practices.
📚 Not sure what ClickOps is? Here a full guide about it and how to face it
The Pitfalls of Direct AWS Console Access: ClickOps
Working directly from the AWS console can often bypass organizational policies and security controls, leading to significant risks.
Operations performed directly through the console may not undergo the same review and validation processes as those executed through Infrastructure-as-Code, potentially leading to misconfigurations and security breaches.
This approach also compromises your audit and compliance readiness. To maintain a well-managed Software Development Lifecycle (SDLC), it is crucial to avoid allowing engineers to change resources via ClickOps.
You may still use direct console access with read-only permissions, but avoid it for creating, updating, or deleting files.
Promoting Best Practices: The Shift to GitOps
To mitigate these risks, organizations are increasingly adopting a GitOps methodology. GitOps defines and modifies infrastructure through Git repositories, treating them as the source of truth. It enforces change reviews, version control, and full auditability.
By restricting AWS console permissions—ideally, to read-only access—organizations can ensure that all modifications undergo the proper review process, thereby maintaining security and compliance.
Think about how software deployment has evolved over the past 20 years. Would you let an R&D team run software updates directly from a local machine today? Probably not.
The same principles should apply to your infrastructure delivery as you have established processes and automation for software deployment.
Don’t misunderstand; removing AWS credentials from multiple engineers across an organization is not trivial. However, this shift is critical, especially when managing cloud infrastructure at scale across multiple accounts and even multiple cloud providers.
Detecting Unauthorized Operations, ClickOps: Leveraging AWS CloudTrail Logs
Now, let’s discuss how to track down ClickOps activity—identifying who’s using the AWS console and what changes they’re making directly through it. The critical tool for this task is AWS CloudTrail.
AWS CloudTrail is a powerful tool for governance, compliance, and operational and risk auditing of your AWS account. It captures every API call to AWS APIs, including those from the management console, SDKs, command line tools, and other AWS management interfaces.
Parsing AWS CloudTrail data isn’t straightforward. You need to sift through JSON files to pull out key details—who made the change, what resource they modified, and more.
Example of how a AWS CloudTrail event JSON might look:
6 Steps to leveraging AWS CloudTrail logs safeguard from ClickOps
We created a 6-step guide shows you how to use CloudTrail logs to detect manual console activity and help you to tighten control, and drive IaC adoption – without disrupting your workflows.
Step1: Parse and Analyze Events:
Start by analyzing AWS CloudTrail events to extract insights. Each AWS service may report slightly different fields to AWS CloudTrail , so normalizing this data for later querying is essential.
- You’ll need to extract:
- User Identity: Identify the IAM user/role involved in the operations. This might include handling edge cases like extracting session names with the user’s email for SSO roles rather than the role itself.
Usually, you can use the userIdentity field for that. Still, there are all kinds of edge cases where this field might be empty, and then you’ll need a particular behavior for specific CloudTrail-logged events. - Action Taken: Identify the action the user performed.
- Resource Amended: Identify the resource(s) involved. In cases where multiple resources are affected by a single action, establish a logic to determine the primary resource.
- Operation Source: Determine where the action originated—was it from the AWS console, an infra-as-code tool, or an SDK call? This can be identified using fields like sessionCredentialFromConsole and analyzing the userAgent.
- User Identity: Identify the IAM user/role involved in the operations. This might include handling edge cases like extracting session names with the user’s email for SSO roles rather than the role itself.
Step 2: Group Events by Resource
Group the events by resources to identify which are frequently modified directly from the console. These are prime candidates for management via Terraform as part of your Infrastructure as Code (IaC) strategy.
Step 3: Group Events by User:
This helps identify which teams or individuals need more training on GitOps and Terraform practices.
Step 4: Set Up a Dashboard
Create a dashboard to filter and query events by user, resource, action, and time range. This tool will aid in tracking and investigating production issues.
Step 5: Set Up Alerts
With the data organized, alerts for specific resources or operation thresholds are set up. For instance, alert if your Production RDS is manually altered from the console or if there are over 20 ClickOps operations within 24 hours. Connect these alerts to your alerting system or a Slack/Teams channel. Ensure there is a straightforward procedure for handling these alerts to effectively reduce ClickOps activity in your account.
Step 6: Generate Monthly Reports
Use the collected data to generate monthly reports to share with your team or management about the success of your GitOps strategy and identify areas needing additional training on infra-as-code and GitOps practices. These reports can also support the case for removing console permissions, mainly when an account is fully managed by infra-as-code.
Reaching the Oasis: Removing Direct AWS Console Access
After implementing the previous steps, you now have:
- An easy-to-use dashboard to visualize ClickOps activity.
- Alerts for any abnormal ClickOps activities.
- Monthly reports showcasing general trends of your GitOps versus ClickOps migration.
This setup signals the right time to start removing console access permissions. Carefully carry out this process to avoid disrupting your teams’ daily operations—it’s not a one-day task.
When your dashboard and reports confirm that specific areas of your cloud environment no longer rely on the AWS console for management, begin removing “write” permissions from console users. Still, keep a “break-glass” user reserved strictly for emergencies, so you can handle critical situations effectively.
Conclusion: Fighting ClickOps
Moving away from direct AWS console operations and adopting a GitOps methodology enhances your security posture and aligns with best practices for cloud governance. By leveraging tools like AWS CloudTrail and implementing strict access controls, organizations can detect unauthorized operations and ensure their cloud environments are secure, compliant, and optimized for operational excellence.
About ControlMonkey
ControlMonkey is the most comprehensive Terraform Automation Platform, providing users with a 360 solution to manage the cloud at scale with Terraform.
You get a single control plane with a complete cloud inventory and alerts on ClickOps activity. It also offers Terraform code generation for your existing cloud environments, as well as drift detection and remediation.
With ControlMonkey, you can standardize your infrastructure delivery at scale with out-of-the-box GitOps Terraform CI/CD, incorporating cost, security, and compliance policies, plus a self-service catalog of pre-defined, compliant infrastructure blueprints for other teams in the organization to spin up infrastructure, enabling agility without sacrificing control.
ControlMonkey verifies configurations and confirms only intended resources run in your cloud.
Book a 1:1 consultation session with our Terraform Experts to learn more about our Terraform Automation platform.
