From Make to Soong: The Evolution of Android Builds

Prior to Android 7.0 (Nougat), the build process was primarily orchestrated by Android.mk files, which leveraged GNU Make. While Make was a staple for many years, its limitations became apparent as Android grew. Makefiles can become intricate and difficult to maintain, especially with conditional logic and complex dependencies. Incremental builds, crucial for developer productivity, were often unreliable, frequently necessitating a full rebuild to ensure correctness.

Soong was introduced to address these challenges, aiming to provide a faster, more declarative, and maintainable build experience. It doesn't entirely replace all existing tools but rather integrates with them to create a streamlined workflow. At its core, Soong leverages the Kati GNU Make clone tool (for legacy Android.mk files during the transition) and the highly efficient Ninja build system as its backend.

The Heart of Soong: Android.bp Files

The most visible change with Soong is the shift from Android.mk files to Android.bp files. These Android.bp files are declarative descriptions of "modules" to be built, similar in syntax and semantics to Bazel BUILD files. This declarative nature is a key advantage, making build configurations more readable and less prone to errors compared to the imperative style of Makefiles.

Here's what defines an Android.bp file:

• Modules: The basic unit of building in Soong is a module. Each module starts with a module type (e.g., cc_binary, java_library, android_app) followed by a set of properties in a name: value format. Every module must have a unique name property.

• Declarative Syntax: Android.bp files focus on what needs to be built and its dependencies, rather than how it should be built. The build logic itself is implemented in Go within Soong's internal components.

• No Explicit If-Statements (mostly): Unlike Android.mk files that allowed complex conditional logic using ifeq, Android.bp files primarily rely on pre-defined distinctions like processor architecture or debug/release builds. Custom case distinctions are handled by the Go-based Soong modules themselves, promoting a cleaner and more structured build graph.

• Globs and Variables: Android.bp supports glob patterns for file lists (e.g., src/**/*.java) and allows for top-level variable assignments, enhancing flexibility and reducing redundancy.

• Comments: Both C-style /* */ and C++ style // comments are supported for better readability.

How Soong Works Under the Hood

The Soong build system operates in a multi-stage process:

• Parsing Android.bp Files: Soong parses all Android.bp files across the AOSP tree. This involves reading the declarative module definitions and their properties.

• Generating Ninja Build Files: Based on the parsed Android.bp files and the internal build logic (written in Go), Soong generates a highly optimized build.ninja file. Ninja is designed for speed and efficiency, particularly for incremental builds, by creating a precise dependency graph and executing build commands in parallel.

• Kati for Legacy Android.mk: During the transition period, Kati acts as an intermediary, processing Android.mk files and generating additional .ninja files that integrate with the overall Ninja build graph. This allows for a gradual migration without requiring a complete rewrite of the entire AOSP at once.

• Ninja Execution: Finally, the Ninja build system takes the generated .ninja files and executes the actual build commands, compiling source code, linking libraries, and creating the final Android images and artifacts.

• soong_ui as the Orchestrator: The soong_ui script acts as the primary driver for the entire build process, orchestrating the different stages and managing auxiliary tools.

Advantages of the Soong Build System

The shift to Soong brings several significant advantages to Android platform development:

• Improved Build Performance: By leveraging Ninja, Soong significantly speeds up build times, especially for incremental builds. This is crucial for developers who frequently make small changes and need quick feedback.

• Enhanced Reliability of Incremental Builds: Soong's precise dependency tracking and Ninja's efficient execution minimize the need for full rebuilds, making incremental builds more reliable and consistent.

• Declarative and Readable Configuration: Android.bp files offer a cleaner, more understandable syntax compared to complex Makefiles, reducing the learning curve for new developers and improving maintainability.

• Stronger Type Checking: Soong's Go-based implementation allows for stronger type checking of variables and properties, catching errors earlier in the development cycle.

• Better Scalability: The modular and declarative nature of Soong, combined with Ninja's parallelism, makes the build system more scalable to the ever-growing size and complexity of Android.

• Pathway to Bazel: The Android.bp file format is intentionally similar to Bazel's BUILD files, indicating a potential future direction for the Android build system to fully embrace Bazel, a highly scalable and reliable build system used by Google.

Challenges and the Road Ahead

While Soong offers substantial improvements, the transition from Make has not been without its challenges. The migration of a massive codebase from one build system to another is a complex undertaking. Developers have had to adapt to the new Android.bp syntax and understand the underlying principles of Soong.

Furthermore, dependencies between legacy Android.mk modules and new Android.bp modules can sometimes be intricate. While Android.mk modules can depend on Android.bp modules, the reverse is not always straightforward.

Looking ahead, Google is actively exploring the integration of Bazel as the ultimate build system for AOSP. Soong serves as a crucial stepping stone in this evolution, providing a modern foundation that is more compatible with a Bazel-like philosophy.

Conclusion

The Android Soong build system represents a pivotal advancement in Android platform development. By moving away from the limitations of Make and embracing a more declarative, performant, and scalable approach with Android.bp files and Ninja, Soong has significantly improved the efficiency and reliability of building the world's most popular mobile operating system. As Android continues to grow and evolve, Soong lays the groundwork for even more sophisticated build capabilities, ultimately contributing to a faster and more seamless development experience for the entire Android ecosystem.

It is not just fast but blazing fast.

This tool made me work much faster and spent less time staring at the screen.

Checkout the post on X

Then I found XDA developers – a forum dedicated to share and collaborate aftermarket operating system development based on AOSP. I quickly learnt that it was the base of all the Android powered smart devices.

After that, I came across the Linux kernel and how Android is based on it. On visiting the AOSP's website, I was mesmerized with the size and complexity of the code base and how the entirety of it is built. I also discovered something called as the Team Win Recovery Project, TWRP for short.

After getting my second smartphone, the Redmi note 4(x), I experimented further and I got addicted. Now this was bad as it took away my free time meddling with my phone. But it didn't stop me and went ahead with my next smartphone: Samsung Galaxy A30.

I wanted to get a headstart on the development so I voided it's warranty and unlocked the bootloader. Samsung is a different league altogether as it they also use to their own chipset (Exynos) and security measures (Knox) which was a hindrance. I nevertheless managed to port LineageOS. I felt great and then I joined a public Telegram group chat dedicated to this device and exchange knowledge. I eventually got fed up of this distraction and swapped my phone with my Moms’ – a Lenovo mobile. She was happy to get an upgrade in performance.

That Lenovo mobile was actually more irritating as it was older than the Galaxy A30 and had a mediatek chipset. I started to understand why Qualcomm Snapdragon is way better. It unfortunately broke and I used an even older Sony Xperia as having a phone became a necessity.

Time for another device – this time a Realme phone. I created and maintained a forum for this device. I also officially maintained the TWRP recovery for this device. The chat still exists till date with 700+ odd members. I handed over the responsibility to someone else to focus on college.

All of this might have taken away my precious time on something I do for free but I'm proud of my achievements.

That's what matters – to think about the good stuff and not crib about what one couldn't do.

I am, at the time of writing, a student pursuing Masters degree in Computer Science, and an origami artist. This art was the actual reason for getting interested with the field of Mathematics and over the course of time.

I have published a research paper in the IEEE journal, an amazing feat to achieve when I was just 20 years old.