HOME/Technical article/Translated article - Discussion on "Is an infrastructure department really necessary?"
Translated article - Discussion on "Is an infrastructure department really necessary?"
Ishikawa Kumo, Service Reliability Group (SRG), Media Management Division@ishikawa_kumo)is.
#SRGThe Service Reliability Group primarily provides comprehensive support for the infrastructure surrounding our media services, focusing on improving existing services, launching new ones, and contributing to open-source software (OSS).
This article is a translation of two articles that were popular within the Chinese infrastructure engineers community.
While reading a Chinese IT blog, I came across an article arguing that "infrastructure departments are unnecessary," and another article refuting it. Although some of the arguments lacked strong evidence, as someone who works in an infrastructure department, I felt I gained insights into the trends and challenges of the Chinese IT industry, as well as some revelations for infrastructure engineers, so I decided to translate it.
Issue raised:Is an infrastructure department really necessary?
Author: Machi
He has held PM positions in DB engine development and Cloud Computing product development at Tencent and Huawei, and currently serves as a Senior Infrastructure Engineer at PayPal Sweden.
Over the past 20 years, Chinese internet companies have achieved great success. Infrastructure departments (or technology platform departments, operations and development departments, architecture platform departments) have made significant contributions as the backbone of these internet companies' technological foundations. However, with technological advancements, the outdated experience of these departments is becoming increasingly irrelevant in the cloud era, and they are generally unable to address new challenges. These two factors combine to raise the question: Do infrastructure departments still have value?
The value of the infrastructure division lies in providing technology to the business divisions.
The Infrastructure Division was established with the aim of consolidating the company's infrastructure and architecture talent and providing technical support to business divisions.
To achieve this objective, the Infrastructure Division must meet the following two conditions:
Having excellent skills
To lead business units and facilitate the utilization of superior technologies.
Unfortunately, over the past few years, the infrastructure sector has been scoring increasingly lower in these two areas. Below, I would like to list some common problems in the infrastructure sector and invite readers to discuss them.
Problem 1: Lack of core technology
In annual presentations by infrastructure departments, it's common to emphasize how they've built a platform that provides business units with general-purpose functions such as big data management, operations, observability, and enhanced security.
Some infrastructure department technical experts mock business unit developers as "CRUD programmers" and take pride in their own accumulated technical knowledge.
However, the truth is that these infrastructure department masters are actually nothing more than craftsmen who assemble open-source software (OSS). Their main job is to find some OSS on GitHub, perform some benchmark tests, and deploy the selected technology to their data center. In this process, the diligent ones simply tweak the parameters of the community version and deploy it as is. The clever ones heavily modify it so that others can't take it over. The even cleverer ones imitate the selected OSS so that others don't want to take it over. I call these people "parameter tuning craftsmen," "heavily modified craftsmen," and "imitation craftsmen," respectively.
Parameter Tuning Specialist
Parameter tuning specialists are often underestimated, but in reality, they are among the most reliable professionals. At the very least, even if a specialist leaves, employers can find someone to take over. Let's take a look at the documentation for Tencent Cloud's CLB.
Layer 7 load balancing is primarilyIt is implemented based on this. STGW is a 7-tier load balancing service developed by Tencent based on Nginx, which supports a large number of simultaneous connections. It handles the traffic for many of Tencent's 7-tier businesses (Tencent News, Tencent Games, WeChat, etc.).
This CLB is based on Nginx, so what exactly was "developed independently"? Was it the development of the installation script?
The Master of Extreme Modifications
AliSQL
💡
Translator's Note: AliSQL is a database engine based on MySQL 5.6, developed by Alibaba Cloud. Its last release was in May 2018. Currently, the AliSQL GitHub repository is not maintained, and community pull requests are completely ignored. Therefore, there are many critical comments within the GitHub Issues section.https://github.com/alibaba/AliSQL/issues/112Alibaba Cloud's open-source projects are popular among Chinese IT engineers.Open Source for KPIsThey are sometimes ridiculed with this comment.
Imitation craftsman
For example, Tencent Cloud's CDN NWS.
All Tecent Cloud CDN edge nodes areWe employ servers to provide our customers with optimal service performance. NWS is a high-performance HTTP server developed in-house by Tencent, and it is said to offer a clear improvement in both functionality and performance compared to common servers such as Nginx.
In reality, NWS is nothing more than a copy of Nginx. The performance improvements were nonsense, simply because the Nginx parameters used for comparison were not optimized.
💡
Translator's note: The information supporting the benchmark results for NWS and Nginx has not yet been found.
Problem 2: Unable to deliver business value
For the past few years, the field of big data has been best known for parameter tuning, modification, and imitation. Initially, everyone was using Hadoop, but the infrastructure department compared Impala, Presto, Spark Streaming, and Storm to select a technology, claiming to the CTO that they had a complete understanding of big data technology. They then began building a big data platform. After securing a budget and working on it for a year, all they ended up with was a web UI, which they named the "Integrated Big Data Management Platform." In cases with more personnel, they even created a user management platform, but often it didn't even support SSO.
Several years later, the Hadoop ecosystem was abandoned in Europe and America, and these same craftsmen turned to Clickhouse as their last resort. The same method is repeated once again.
For the past decade, at least several hundred infrastructure (or big data platform) departments across China have been doing this. When these craftsmen get together, they engage in childish competitions like, "You need 600 machines, but I can do it with 300," or "I can process 5TB of data in an hour, but you take 70 minutes—that's 14% faster." It's like two middle schoolers competing to see who can pee the furthest, saying, "You can only pee two meters, but I can pee three meters."
In reality, what business units are concerned with isn't the number of machines they're using. What they're truly interested in is whether the big data platform can generate business value. For example, consider these challenges:
Can that big data platform automatically incorporate new data sources?
Can it be easily integrated with BI tools?
Is it possible to implement granular access control and access auditing?
Can we accurately charge each business unit?
Can you comply with personal information protection regulations?
However, due to the operational culture of the infrastructure department, they pay little attention to these issues. Day by day, they focus on tinkering with machine performance within their comfort zone. Sometimes they even submit bizarre reports like, "Three engineers spent six months saving the company 50,000 yen, equivalent to half a machine's cost."
Athena
Problem 3: Lagging technological concept
Over the past decade or so, the software engineering industry has seen a slow but steady series of technological innovations, making today's software industry significantly different from what it was 15 years ago. However, because these innovations haven't had the dramatic impact of something like the iPhone or ChatGPT, many in the field haven't felt the progress. This includes decision-makers and engineers in the architecture and platform divisions. This lag in understanding has led to a lag in technology, causing them to lose their technological advantage over business units, and in some cases, even be overtaken.
Sticking to solved scalability problems
W11
To be honest, 20 years ago, LAMP configurations were common, and many teams were struggling with the C10K problem. During that time, BAT (Baidu, Alibaba, Tencent) companies were exploring large-scale service engineering practices, and there was value in that supporting the companies' business growth.
However, 20 years have passed since then, and scalability is now a solved problem. The basic approach remains largely the same.
Using microservices, each service has its own independent resources.
Keep the web server stateless and enable automatic scaling out/in.
Using load balancing to process external requests
Avoid manual deployment of binaries by using containers.
Instead of using a file system that is difficult to expand, use object storage that is separate from the computing nodes.
Ensure that each Pod is replaceable.
Use an auto-scaling distributed database and add caches as needed.
Use message queues to bridge the processing power gap between different systems.
Infrastructure as Code
Build a CI/CD pipeline that includes code, deployment files, and secrets to ensure code quality.
Provide a standard observability dashboard for each microservice.
If necessary, partition users at the business layer.
Basically, these best practices can handle most loads. And there are tools for each step, so any engineer can implement them within their team.
In contrast, when infrastructure experts talk about high parallelism, they either get bogged down in the details of threads and processes, or use vague terms like "active-active configuration." If you ask even slightly more detailed questions, they get angry and ask things like, "Have you ever handled Double Eleven?", "Do you know how many machines I manage?", or "Do you have a billion users?". Frankly, this kind of talk is basically like a cult performing a ritual.
Sticking to ClickOps
LY.COM
While this type of ClickOps might have been acceptable 15 years ago, the industry has now already shifted to the concept of IaC. Engineers should use IaC code to communicate directly with resource providers and eliminate operational middlemen.
💡
Translator's note:
LY.COM is a Chinese travel booking website.
LY.COM's Platform Engineer has refuted the above statement.
The process described above has already been significantly improved, and now the HBase cluster is automatically created by the Kubernetes Operator after workflow approval.
LY.COM: In fact, the backend of LY.COM's HBase cluster service request process integrates the request workflow configuration parameters with HBase cluster configuration templates and implements a process to automatically launch cluster instances using Kubernetes. This automated process works very smoothly, and situations where "an operations engineer receives the ticket and runs a formulaic script to create the cluster" no longer exist.
Problem 4: Intentional obstruction of DevOps
In addition to technological lag, infrastructure departments sometimes intentionally hinder team efficiency for the sake of team interests (budget and survival). For example, some departments exclusively manage all infrastructure resources under various pretexts, requiring developers to submit requests every time they need an object storage bucket. Even if a developer needs a database, a request is required, and they must wait for DBA approval even to update a single SQL statement.
In CI/CD pipelines led by infrastructure departments, often only the CI portion functions, while the CD portion is intentionally isolated. This is done "to ensure the stability of the production environment," but in reality, they fear that opening up CD will make their department obsolete, as business unit development teams will realize they can do efficient, high-quality work without intermediate procedures.
Suspicion towards the vendor
Even within the same company, the relationship between business units and infrastructure units is essentially like that between a user and a vendor. Naturally, vendors tend to exclude other vendors. Even if a vendor offers excellent observability tools externally, the infrastructure unit might choose not to use them, instead trying to build their own cumbersome ELK. Even if a reliable RDS solution exists externally, the infrastructure unit might try to build their own MySQL.
Before Lark and DingTalk became popular, at least 200 of China's Top 500 companies had teams dedicated to developing internal IM tools. Much talent was wasted reinventing the wheel with inferior tools. These IM tools, supposedly tailored to the specific needs of each company, are often difficult to use, have weak security, and frequently lead to entire teams leaving the company.
💡
Translator's note:
Lark and DingTalk are enterprise-grade instant messaging/chat tools from China.
Furthermore, when game companies develop their own HR systems, food delivery companies develop their own code management tools, or telecommunications equipment companies develop their own travel management systems, it's as if they're just using their employers' money to expand the influence of their own departments because they have nothing better to do.
Distortion in relationships with business divisions
Many infrastructure departments claim their value lies in their in-house understanding of business needs. This sounds plausible at first glance.
However, in reality, for many internet companies, technology isn't a core competence; general software technologies suffice, and there aren't many specific needs. Conversely, if a business unit has specific needs not generally supported in many industries, there may be problems with software selection.
A typical example is using MySQL to store time-series data with Zabbix. Many requirements arise for MySQL, but these requirements shouldn't be met because MySQL wasn't designed for processing time-series data in the first place.
Another typical example is the common need for fixed IP addresses for Kubernetes in China. In reality, businesses requiring fixed IP addresses shouldn't use Kubernetes; if containers are absolutely necessary, they should be deployed directly on virtual machines. However, because infrastructure departments catered to this need, the demand spread throughout the country and even affected the Cilium project. The Cilium project failed to address this demand, and the issue has been left unresolved for three years.https://github.com/cilium/cilium/issues/17026
💡
Translator's note: Immediately after the publication of this Chinese article, we received responses from Chinese committers.
The operations team's conservative thinking
Many companies' infrastructure departments are derived from operations teams and place a strong emphasis on reliability. While this is a strength, it also fosters a conservative culture. A common saying among veteran operations staff is, "Don't touch what's working."
If you look at the technology stacks of many advanced IT companies, you'll see a lot of outdated software in use, such as MySQL 5.7, CentOS, Python 2.X, and GCC 4.9, which are no longer maintained. These programs pose significant risks in terms of compatibility and security because patches are not provided even when security vulnerabilities are discovered. While infrastructure departments should ideally be responsible for updating such older software, they often prioritize availability above all else, making them the most reluctant to upgrade technology.
Photo quoted in the original text
Counterargument: The infrastructure sector will not die, it will simply disappear.
Author: Leo Li
He previously served as Chief Architect at DiDi and CTO at MeiQia, and is currently Co-Founder and CEO of ClapDB.com.
Translator's note: ClapDB is a startup company that offers serverless database SaaS for data analytics.
Yesterday, Mr. Ma from PayPal wrote an article titled "Is an Infrastructure Department Really Necessary?" which questioned the very existence of infrastructure departments. As someone who has been involved in infrastructure for many years, I would like to respond to the issues raised in this article from a relatively objective standpoint. And with this writing, I would like to express my respect to the veterans of infrastructure.
What is the value of the infrastructure division?
As Mr. Ma mentioned, the infrastructure department exists to provide technology to the business departments, and for that purpose, the following two prerequisites are necessary:
Having excellent technical skills
To lead business units and facilitate the utilization of superior technologies.
The definition of "excellence" in technology varies depending on the situation.
However, there isn't a single standard for "excellence." What constitutes excellent technology? Technology always involves trade-offs. For example, even if a technology offers top performance, if its functionality is too simple or the cost is too high, it's meaningless for many companies.
The criteria for excellence differ in each business scenario. Since companies don't compete on the same playing field, the infrastructure departments that support a company's business shouldn't have unified goals.
For example, let's talk about the goal of pursuing the best possible parallel processing capabilities. Salesforce, a globally renowned company in the SaaS industry, has long gone against the trend known as "de-IOE" (eliminating IBM, Oracle, and EMC storage) by using Oracle as its underlying storage. Salesforce's parallel processing capabilities are also low, but that doesn't mean their technology is inferior. It simply means that their technology is appropriate for their specific domain and doesn't meet the requirements of other domains. A cat doesn't need to compete with a penguin; penguins are better swimmers.
Leading the business division?
The infrastructure department exists as a cross-functional department, playing a role in reducing redundant and flawed development across departments. In fact, during the internet and mobile internet eras, internet companies prioritized "growth" as their most important metric, hiring a large number of inexperienced programmers and engaging in trial and error in new business areas. If a cross-functional department does not provide "guidance" and "standardization," the business could grow in a far too "barbaric" manner.
Does the infrastructure division lack core technologies?
Is he really a "parameter tuning expert," a "modification expert," or a "replica expert"?
In Mr. Ma's article, he discusses whether the infrastructure department has core technologies and categorizes engineers into "parameter tuning specialists," "extreme modification specialists," and "imitation specialists."
I completely agree with this point.But so what?Many Chinese internet companies model themselves after American companies and conduct business in much the same way as their American counterparts. Therefore, there is absolutely no problem with using almost the same technologies as their American counterparts, as it is the most reliable and low-risk technology selection.CTC(The original meaning of ) is to cross a river by stepping on steps (like America).If you can copy the business model, then there shouldn't be any problems with the technology stack either.
💡
Translator's Note:
Copy To China: This refers to Chinese companies imitating the business models of successful foreign companies. The degree of imitation varies, ranging from simply providing services that directly compete to mimicking the appearance of UIs, trademarks, or names with similar pronunciations. While this practice is often criticized in countries outside of China, including Japan, it is widely recognized as a successful model in China, from private companies to the government.
Furthermore, major companies in Silicon Valley, such as Twitter (now X), Facebook (now Meta), and Airbnb, have fundamentally grown based on open-source technologies. And when problems arise during that process, they are solved through "parameter tuning" and "extreme modifications." However, "imitation" is a sign of the ailments of large corporations.
"Imitation" is a symbol of the ailments of large corporations.
As companies grow, the "measures" for individual promotion and evaluation change, shifting from "contribution" to "contribution x difficulty." However, as companies grow and businesses stabilize, most contributions become less noticeable. As a result, employees try to artificially increase the difficulty of their contributions to gain better evaluation and promotion opportunities.
The phenomenon of "imitation" stems from the fact that as companies grow, team goals diverge from the overall company goals. Even if companies introduce KPIs and OKRs, this problem cannot be prevented. In reality, "imitation" is often justified for some plausible reason, and problems that can be solved through imitation are often found. Whether it is imitation or transcendence is up to the individual, but without this gray area, innovation would not occur. You cannot understand the core of anything until you try to imitate it at least once. The devil is in the details, but you can't know which details are important and which are the devil until you actually get your hands dirty.
In short, the ailments of a large corporation are not necessarily a bad thing; rather, the resources and profits of a large corporation provide a certain amount of room for innovation. However, while investment inherently involves risk, it's also true that you gain nothing if you don't invest.
What's wrong with modifying open-source software (OSS)?
It is known that OSS is developed by developers such as:
Large corporations open-source software for their own commercial purposes (such as Android, Chrome, Kubernetes, and VS Code) in order to gain market share and beat competitors.
Companies manage open-source software (OSS) and acquire paying customers through open-sourcing (e.g., Elasticsearch, Nginx, MySQL, Cassandra).
Software provided by the Open Source Foundation (GNU, Linux, etc.)
Generally, type 1 companies will follow their own roadmap and have little interest in niche needs. Type 2 companies will intentionally weaken enterprise-level features (otherwise, how would they sell a commercial version?). Type 3 companies will have a "welcome to your PR" stance and lack the service-oriented feel of commercial software.
Internet companies worldwide encourage their employees to modify (even heavily modify) open-source software. Facebook has taken control of HBase internally, and Google has submitted cgroup patches upstream of Linux. Thus, modifying open-source software is not a crime; the problem lies in the technical immaturity.
💡
Translator's note: The original expression encompasses a wide range of nuances, including not only technical immaturity but also organizational structure, culture, and perceptions.
How does the infrastructure division deliver business value?
Infrastructure departments generally function as cross-functional departments and often lack business revenues that allow them to calculate their own ROI. However, infrastructure departments demonstrate their value through more general metrics such as availability, scalability, and business iteration speed. A good cat is a mouse catcher, even if the cat isn't wearing the latest nail polish. In internet companies, technology serves the business, and the vast majority of companies are not FAANG. It's unfair and unrealistic to ask a security guard at a regular company to invent Star Wars equipment.
Regarding the delay in technological philosophy
Copy from Google
Why are we so fixated on scalability?
The responsibility of the infrastructure department lies in availability and scalability. Anything else would make it appear as if they are neglecting their core duties.
Is the infrastructure sector hindering technological progress?
Regardless of the team, proving their value within an organization is an essential task, especially for cross-functional departments. An infrastructure department can only demonstrate its value when problems arise and it is held accountable. Therefore, it's natural that the overall development foundation becomes conservative, as this aligns with the department's role and raison d'être. Infrastructure teams often play a role in regulating the demands and actions of business teams, and their efforts can frequently seem unrewarded. Nevertheless, they still provide essential constraints within the company.
Organizational inertia
This is an inertia seen in all organizations throughout history; most organizations do not progress or innovate from within, but are driven by external forces. For example, even when the Soviet Red Army faced German tanks, they still placed their hopes on Cossack cavalry. The infrastructure team is no different from other teams; their mission is to support the company's business. If they are fulfilling that mission, there is little motivation for self-reform. However, this is not the fault of the Cossack cavalry (old ideas).
The barrier to entry for non-tech companies is not technology.
Many internet companies in the Chinese market are not technology-driven and do not require a continuously evolving infrastructure department. Rather, if the business is not experiencing rapid growth, it is more practical to keep costs down.
Why does the infrastructure department exclude external vendors?
Infrastructure departments are often internal technology providers, and it's natural for them to exclude external competitors. After all, external vendors take almost the same approach as infrastructure departments. They also build upon open-source systems, heavily modifying them, tuning parameters, and even creating imitations. In such a situation, it's difficult for infrastructure departments to abandon their own tools and buy others'. Both users and vendors have their own problems.
A destiny to vanish
Old soldiers never die; they just fade away.
New productivity will inevitably create new production relationships.
The infrastructure team itself,"They try to achieve growth by using excessive resources."This is a microcosm of internet companies from a bygone era.
However, every era has an end, and the infrastructure sector, like other excess resources in periods of growth, will be gradually reduced. Internet companies will shift from high-investment to rational, high-ROI-oriented investments. They will move from a desire for autonomy and control to achieving control at a more reasonable cost.
💡
Translator's note:"Trying to achieve growth by using excessive resources."This is an example of a growth strategy often seen in Chinese internet companies. Smartphones (especiallyIn the mobile internet era, which began with the spread of the internet, many companies excessively invested resources and aimed to lock in users and rapidly expand market share by cramming as many functions as possible into a single app. This also aimed to prevent user churn and discourage migration to other services.
For example, even single-function apps like weather apps often become all-in-one apps in China, adding features such as news, short videos, food delivery, and payment functions. This approach, which prioritizes scale over efficiency, is called "barbaric growth" and reflects the culture and structure of Chinese internet companies. (This phenomenon is also seen in Japan recently.)
The infrastructure team will move in one of two directions:
Embrace the cloud and reduce human resource costs.
Accept the market, offer your products to the market, and participate in the competition.
By re-evaluating and reallocating the role of infrastructure from an ROI perspective, unnecessary elements will be eliminated from the team. Products of the times naturally evolve in accordance with the flow of the times.
In conclusion
These two articles discuss the significance and future prospects of the "infrastructure department," which supports a company's technological foundation, from different perspectives. The reality is that the role of the infrastructure department within a company is changing with technological advancements. As Mr. Ma points out, clinging to past practices risks being left behind. On the other hand, as Mr. Li states, the infrastructure department still possesses technical value and plays a crucial role in stably supporting the business.
Going forward, infrastructure departments and infrastructure engineers will face challenges such as re-evaluating their roles, adapting to the evolution of cloud technology and changing business needs, and finding ways to balance efficiency and value delivery.
By the way, I was surprised by the "Copy To China" concept in the Chinese IT industry and its goal of competing with Silicon Valley. I would like to continue translating good Chinese articles if I come across any.
SRG is looking for new team members.
If you are interested, please contact us here.
