About SRGTechnical articleRecruitment Information
About SRGTechnical articleRecruitment Information
HOME/Technical article/Design and reflections on multi-tenancy in the Ameba Platform

Design and reflections on multi-tenancy in the Ameba Platform

Ishikawa Kumo, Service Reliability Group (SRG), Media Management Division@ishikawa_kumo)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article summarizes the design and review of the multi-tenancy support implemented on the Ameba Platform from 2023 to 2024.

Background and Challenges

The Ameba Platform is a platform centered around the infrastructure (primarily EKS) of Ameba Blog and its related services. Launched around 2020 with the aim of unifying development and deployment flows and simplifying the technology stack, the project's central goal was to integrate many services into a more efficient and manageable infrastructure.
While the core components were successfully migrated in the initial stages of the project, EKS and other security challenges prevented the migration of authentication services and other high-security services to the Ameba Platform. These services had to continue operating on the existing infrastructure or in a separately prepared EKS environment.

Reasons why the migration could not be completed

  1. Istio Authorization Policy
  1. The challenge of integrating authentication and authorization systemsIdeally, we wanted to fully integrate Kubernetes RBAC, AWS IAM, and all developer and monitoring tools using our internal authentication and authorization infrastructure, but we didn't have the resources to do so during the initial stages of platform development in 2020-2021.
  1. Compatibility issues between Istio and EKS
      2. Security Groups for Pods（SGP）

Timing for redesign

In 2023, approximately three years after the platformization began, we had more human resources and a wider range of technical options, presenting an opportunity to fundamentally redesign the multi-tenant architecture and restart the migration process. First, the stability of VPC CNI and SGP operations was proven, enabling the use of SGP. Furthermore, VPC CNI began native support for NetworkPolicy in 2022, minimizing vendor dependency. In 2023, shortly after joining CyberAgent, I participated in this project, handling authentication integration and the multi-tenant design on AWS and EKS.

Clarification of absolute requirements

In redesigning the project, the following requirements were established as absolute, non-negotiable conditions:
  1. Complete communication isolation according to security level
      2. Pod to Pod, Pod to AWS resources
  1. Centralized authentication of all communications through a common authentication platform
      2. AWS、EKS、Datadog、ArgoCD、Github Teams
  1. Strict access control for AWS and Kubernetes resources
      2. Utilizing RBAC and ABAC

Design policy

Furthermore, the following principles were used as design guidelines.
  • Minimize reliance on specific vendor products.
  • We aim to achieve this using Kubernetes and AWS default features as much as possible.
  • Ensuring simplicity and maintainability of the authentication and authorization process.

From a security isolation perspective

Security level

Services on AWS are independent of each other, yet they maintain an equal relationship. To achieve security isolation, it is crucial to leverage IAM's ABAC protocol to identify the security level of each service and control communication based on that level. Resource tags are one of the best ways to identify the security level of a service.
On the Ameba platform, considering the characteristics of microservices, we classified services as follows:
  • Protected ServicesServices with high security requirements and strict management
  • Non-Protected ServiceServices with relatively low security requirements
In communication control, we established the following principles:
  1. Services with a high level of security (Protected)
      • Inbound (receiving) communications are strictly restricted.
      • Outbound (sending) communication is relatively free.
  1. Services with low security levels (Non-Protected)
      • Restrictions are relatively lax for both inbound and outbound travel.

Exception handling

In traditional multi-tenant models, it's common to impose strict restrictions on each tenant, limiting access to only their own resources. However, in actual operation, more complex requirements exist. For example, a team managing authentication services needs to deal with services with different security levels on a daily basis.
Even highly secure services sometimes require exceptions, such as exposing certain APIs, to allow inbound communication. When allowing such exceptions, we have adjusted the security level for some of the affected communications.
The following communication restrictions have been set on the Ameba platform.
  • Non-Protected services cannot access Protected services.
  • Protected services can access non-protected services.
  • Protected services that expose specific endpoints will have the relevant parts demoted to non-protected services.

Approach and details

Our specific implementation approach for multi-tenant design focused on the following four key areas:

1. Integrated Strategy for Certification and Authorization

Unification of authentication infrastructure
To achieve centralized management of authentication and authorization, the Ameba Platform employs the following integrated approach.
  • AWS, Datadog, and GitHub Authentication: Leveraging Internal SAML Infrastructure
  • Node SSH access: Utilizes the company's LDAP infrastructure.
  • ArgoCD: Using OAuth2 and OIDC together with GitHub Teams
Due to functional limitations of our internal authentication infrastructure, integrating authentication was one of the most complex aspects of this project. While some authentication methods could ideally be centralized using OIDC, the lack of OIDC functionality in our internal infrastructure necessitated the adoption of a variety of methods.
In the case of ArgoCD in particular, due to SAML security concerns regarding dex, we were unable to directly integrate with our internal authentication infrastructure, so we implemented OIDC integration via GitHub Teams. Since GitHub Teams already has SAML integration, a user inventory was not required.
Please refer to previous articles for information regarding issues with ArgoCD SAML integration.
ABAC: Role
developer、admin
<product>-<tenant>-<role>
ameba-A-developer
ameba-A-secure
We have a system in place that allows for the addition of attributes corresponding to other roles, depending on actual operational needs.
ABAC: Policy
To achieve advanced access control, we implemented a Resource Tag-based attribute management system.
  • ameba.jp/protected=true
  • ameba.jp/sensitive=true
  • ameba.jp/exposed=true
StringNotEquals
  1. To distinguish between Admins and Developers, we use NotActions.
  1. StringNotEquals Condition
  1. StringEquals Condition
While ABAC allows you to control most AWS services, some services cannot be controlled by Resource Tags. In such cases, you will need to handle them individually using the corresponding Conditions.
For example, the following services and APIs
Detailed information can be found in the Service Authorization documentation.
EKS RBAC
developer/admin
ClusterRoleBinding
RoleBinding

2. Implementation of network security

Pod ↔ Pod-to-Pod communication
ConfigurationValues
PodSelector
When applying a tagging strategy similar to IAM's ABAC, please note the following:
  • ameba.jp/protected=true
  • When exposing Pods within a Protected namespace to the outside world, an Expose Tag is required.
Hierarchical Namespace
namespaceSelector
ameba.jp/exposed: "true"
Furthermore, by using the inheritance feature of Hierarchical Namespaces, it becomes unnecessary to create each Child Namespace as described above.
Pod ↔ AWS resource communication
SecurityGroupsForPod(SGP)
Using SGP involves two steps.
  1. Change the following settings in vpccni
    1. We will use SGP's CustomResource.
      SGP has several risks.
      1. There is a limit to the number of Pods to which this can be applied.
          2. Branch ENI
          Trunk ENI
          Branch ENI
      1. Pod startup speed will be slower.
          2. Branch ENI
      1. Potential conflict with other network vendors
          2. Although the issue has now been resolved, there were (unconfirmed) conflicts with Istio in the past. AWS support recommends IAM authentication, so SGP should only be considered as a last resort.
      If you are interested in learning more about SGP, please refer to our previous articles.

      3. Protecting shared resources, volumes, and backups

      Shared resources
      For shared resources such as ECR and S3, access control using ResourceTag is implemented within the Shared account, as these resources are centrally managed. For services where ResourceTag control is difficult (e.g., S3), a prefix identification method for resource names is used in conjunction with ResourceTag.
      Geolayer
      All EBS volume operations can be controlled by Resource Tags. However, there is one exception when using EKS.
      Kubernetes PersistentVolume (PV)
      backup
      All AWS Backup APIs, such as create/copy, are subject to Resource Tag control.

      4. Security of Monitoring and APM

      When integrating with monitoring tools, particularly Datadog, we attempted to connect with the authentication infrastructure, but we encountered challenges in controlling access for APM.
      While limitations on APM do exist, we found that fine-grained access control is difficult. Dividing access at the level shown in the diagram below leaves us with only two options: either completely block access from everyone without specific permissions, or allow everyone to see it.
      Therefore, we adopted an approach that masks sensitive data before it enters the APM (Application Processing Manager).

      Looking back

      Approximately six months have passed since we began supporting the entire Ameba Platform environment, but due to a lack of human resources, the migration has not yet begun. In this review,Previous trial operationsI will summarize what I felt and what I thought after looking at examples from other companies.

      Fine-grained processing is not supported.

      The two separate roles, developer and admin, do not allow for granular permission settings for users who only have access to certain services and resources. Whenever a request for additional permissions is made,Such a powerful roleI find myself wondering, "Is it okay to hand it over?"
      Because there's a one-to-one relationship between IAM roles and roles in our internal authentication infrastructure, it's practically difficult to increase the number of roles as needed. I'm still struggling to figure out what to do. If anyone has any good ideas, please let me know.

      The actual operational details for each tenant are far more complex.

      <product>-<tenant>-<role>
      For example, some tenants use authentication infrastructure roles for multiple purposes, and they manage their members and integrations independently. Because these roles cannot be integrated into the Ameba Platform, it seems that the multi-tenant environment will be incomplete.
      Furthermore, because our authentication infrastructure has a limit on the number of references between roles, it is currently unclear to what extent we can integrate member management into the Ameba Platform.

      Network Policy is quite useful.

      Cloudflare Tunnel
      Cloudflare Tunnel

      In conclusion

      I wrote this while recalling the entire process of implementing multi-tenancy support at Ameba. Looking back, my memory is hazy in many places, and I was saved many times by the detailed documentation I wrote at the time.
      This article is based on just one example from within CyberAgent, but I hope it will be helpful as a reference.
      SRG is looking for new team members. If you are interested, please contact us here.