/ README.md
README.md
1 # Programming Language Benchmark Suite 2 3 A comprehensive, cross-language performance benchmarking framework comparing execution speed across **16+ programming languages** with multiple compilers, assemblers, and computational workloads. 4 5 [](https://github.com/killerdevildog/Programming_Benchmark) 6 [](https://codeberg.org/killerdevildog/Programming_Benchmark) 7 [](https://app.radicle.xyz/nodes/seed.radicle.xyz/rad:z4RRJ6vMp6Ysa1WSRW7opPzQSDTfX) 8 [](https://opensource.org/licenses/MIT) 9 10 ## 🎯 Project Overview 11 12 This project provides **apples-to-apples performance comparisons** of compiled executables across diverse programming languages, compilers, and optimization strategies. Unlike interpreter-based benchmarks, all implementations are compiled to native executables, providing true performance insights into language efficiency and compiler optimization capabilities. 13 14 ### Why This Matters 15 16 - **Real-world Performance**: Measures actual executable performance, not interpreter overhead 17 - **Compiler Comparison**: Tests the same language with different compilers (GCC vs Clang) 18 - **Assembler Variants**: Compares NASM, YASM, and GAS (GNU Assembler) for assembly code 19 - **Fair Evaluation**: Identical algorithms across all languages ensure fair comparisons 20 - **Reproducible Results**: Automated benchmarking with hyperfine ensures statistical validity 21 22 ## ✨ Key Features 23 24 - **48 Hello World Implementations**: Minimal overhead testing across languages 25 - **32 Computational Benchmarks**: Prime number calculation (first 1000 primes) 26 - **Multi-Compiler Support**: GCC and Clang for compatible languages 27 - **Assembly Variants**: NASM (Intel syntax), YASM, and GAS (AT&T syntax) 28 - **Embedded Language Support**: R via embedded API, Python via Cython 29 - **Automated Benchmark Tool**: Custom wrapper around [hyperfine](https://github.com/sharkdp/hyperfine) 30 - **Multiple Output Formats**: Markdown tables and JSON for further analysis 31 32 ## 📊 Supported Languages 33 34 ### System Programming Languages 35 - **C** - Both GCC and Clang 36 - **C++** - 6 different implementations (iostreams, printf variants) 37 - **Rust** - Using rustc 38 - **Zig** - Latest stable 39 - **Go** - Using gccgo and standard go compiler 40 - **D** - Using gdc/ldc 41 - **Nim** - Compiled via C backend 42 43 ### Assembly Languages 44 - **NASM** (Intel syntax) 45 - **YASM** (NASM-compatible) 46 - **GAS** (GNU Assembler with AT&T syntax) 47 48 ### High-Level/Managed Languages 49 - **Ada** - Using GNAT 50 - **Haskell** - GHC compiler 51 - **C#** - Mono runtime 52 - **Pascal** - Free Pascal Compiler 53 54 ### Legacy Languages 55 - **Fortran 90** 56 - **COBOL** - GnuCOBOL 57 58 ### Scripting Languages (Compiled) 59 - **Python** - Via Cython to native executable 60 - **R** - Embedded R API with C wrapper 61 62 ## 🏗️ Project Structure 63 64 ``` 65 Programming_Benchmark/ 66 ├── test_src/ # Source code for all languages 67 │ ├── c/ 68 │ │ ├── hello_world/001/hello.c 69 │ │ └── first1000primes/001/primes.c 70 │ ├── asm/ 71 │ │ ├── hello_world/ 72 │ │ │ ├── 001/hello.asm (NASM) 73 │ │ │ ├── 001/hello_yasm.asm (YASM) 74 │ │ │ └── 001/hello_gas.s (GAS) 75 │ ├── r/ 76 │ │ ├── hello_world/001/ 77 │ │ │ ├── hello.R 78 │ │ │ └── hello_wrapper.c 79 │ └── [other languages]/ 80 ├── bin/ 81 │ ├── tests/ # Compiled executables 82 │ │ ├── gcc/ 83 │ │ │ ├── c/hello_world/001/hello 84 │ │ │ ├── asm_nasm/hello_world/001/hello 85 │ │ │ └── [...] 86 │ │ └── clang/ 87 │ ├── benchmark # Benchmark automation tool 88 │ └── hyperfine # Hyperfine binary 89 ├── CMakeLists.txt # Build configuration 90 ├── results_hello_world.md # Benchmark results (Markdown) 91 ├── results_first1000primes.md 92 └── README.md 93 ``` 94 95 ## 🚀 Getting Started 96 97 ### Prerequisites 98 99 Install the required compilers and tools for the languages you want to benchmark: 100 101 ```bash 102 # Essential build tools 103 sudo apt install cmake build-essential 104 105 # Language compilers 106 sudo apt install gcc clang gfortran gnat gccgo gdc mono-mcs ghc fpc cobol 107 108 # Assembly tools 109 sudo apt install nasm yasm 110 111 # Modern languages (install via respective package managers) 112 # Rust: curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh 113 # Zig: https://ziglang.org/download/ 114 # Nim: https://nim-lang.org/install.html 115 116 # Python (Cython) 117 pip install cython 118 119 # R development files 120 sudo apt install r-base r-base-dev 121 122 # Optional: Hyperfine (for benchmarking) 123 # Will be built automatically if not present 124 cargo install hyperfine 125 ``` 126 127 ### Building the Project 128 129 ```bash 130 # Clone the repository 131 git clone https://github.com/killerdevildog/Programming_Benchmark.git 132 cd Programming_Benchmark 133 134 # Create build directory 135 mkdir -p build && cd build 136 137 # Configure with CMake 138 cmake .. 139 140 # Build all implementations 141 cmake --build . 142 143 # Or build with multiple cores 144 cmake --build . -j$(nproc) 145 ``` 146 147 The build process will: 148 1. Detect available compilers 149 2. Build all language implementations 150 3. Create executables in `bin/tests/` 151 4. Build the benchmark tool 152 153 ## 🔬 Running Benchmarks 154 155 ### Quick Start 156 157 ```bash 158 # Run hello_world benchmark (500 runs) 159 ./bin/benchmark hello_world --runs 500 160 161 # Run first1000primes benchmark (100 runs) 162 ./bin/benchmark first1000primes --runs 100 163 ``` 164 165 ### Benchmark Series 166 167 Two benchmark series are available: 168 169 1. **hello_world**: Minimal I/O overhead testing 170 - Tests: Basic executable startup + output 171 - Implementations: 48 variants 172 - Typical runtime: 73µs (ASM) to 178ms (R) 173 174 2. **first1000primes**: Computational workload 175 - Tests: Prime number calculation (find 1000th prime = 7919) 176 - Algorithm: Trial division with optimizations 177 - Implementations: 32 variants 178 - Typical runtime: 216µs (ASM) to 213ms (R) 179 180 ### Understanding Results 181 182 Benchmark results are saved in two formats: 183 184 - **Markdown** (`results_*.md`): Human-readable tables 185 - **JSON** (`results_*.json`): Machine-parseable data 186 187 Example output: 188 ``` 189 Summary 190 gcc_asm_001 ran 191 1.03 ± 0.41 times faster than gcc_zig_001 192 2.00 ± 0.59 times faster than gcc_c_001 193 988.00 ± 273.66 times faster than gcc_r_001 194 ``` 195 196 ## 📈 Performance Insights 197 198 ### Hello World Results (500 runs) 199 200 | Category | Fastest | Slowest | Range | 201 |----------|---------|---------|-------| 202 | **Assembly** | 73µs (GCC/Zig) | 146µs (Clang ASM) | ~2x | 203 | **System Languages** | 317µs (C) | 898µs (Ada) | ~3x | 204 | **Managed** | 1.5ms (D) | 12ms (C#) | ~8x | 205 | **Legacy** | 1.7ms (COBOL) | 11.3ms (Haskell) | ~7x | 206 | **Embedded** | 16ms (Python/Cython) | 178ms (R) | ~11x | 207 208 **Key Takeaway**: R's embedded initialization overhead is ~2400x slower than raw assembly, but provides full statistical computing capabilities. 209 210 ### First 1000 Primes Results (100 runs) 211 212 | Category | Fastest | Slowest | Notes | 213 |----------|---------|---------|-------| 214 | **Low-level** | 216µs (ASM) | 243µs (Zig) | Highly optimized | 215 | **System** | 432µs (C) | 974µs (Ada) | GCC optimizations effective | 216 | **Functional** | 11.2ms (Haskell) | N/A | Lazy evaluation overhead | 217 | **Legacy** | 23.8ms (COBOL) | N/A | Surprising competitiveness | 218 | **Embedded** | 17.5ms (Python) | 213ms (R) | Cython provides good performance | 219 220 **Key Insight**: Algorithm matters more than language for compute-heavy tasks. Well-optimized C/Zig/ASM are within 2x of each other. 221 222 ## 🔧 Technical Highlights 223 224 ### Assembly Variants 225 226 Three assemblers tested with identical functionality: 227 - **NASM**: Intel syntax (`mov rax, 1`) 228 - **YASM**: NASM-compatible, different optimizer 229 - **GAS**: AT&T syntax (`mov $1, %rax`), uses `#` for comments 230 231 Both GCC and Clang linkers tested with each assembler. 232 233 ### R Language Integration 234 235 R uses embedded API approach: 236 ```c 237 #include <R.h> 238 #include <Rinternals.h> 239 #include <Rembedded.h> 240 241 Rf_initEmbeddedR(4, r_argv); // Initialize R runtime 242 R_ParseVector(code_str, -1, &status, R_NilValue); // Parse R code 243 Rf_endEmbeddedR(0); // Cleanup 244 ``` 245 246 This demonstrates the overhead of embedding interpreted languages in compiled executables. 247 248 ### Python via Cython 249 250 Python code compiled to C and then to native executable: 251 ```bash 252 cython --embed -o hello.c hello.py 253 gcc hello.c $(python3-config --cflags --ldflags) -o hello 254 ``` 255 256 Significantly faster than interpreted Python, but slower than pure C. 257 258 ## 🧪 Adding New Benchmarks 259 260 ### Adding a New Language 261 262 1. **Create source directory**: 263 ```bash 264 mkdir -p test_src/newlang/hello_world/001 265 mkdir -p test_src/newlang/first1000primes/001 266 ``` 267 268 2. **Implement the benchmarks**: 269 - `hello_world`: Print "Hello, World!\n" to stdout 270 - `first1000primes`: Calculate and print the 1000th prime (7919) 271 272 3. **Update CMakeLists.txt**: 273 ```cmake 274 find_program(NEWLANG_COMPILER newlangc) 275 if(NEWLANG_COMPILER) 276 # Add build rules 277 endif() 278 ``` 279 280 4. **Rebuild**: `cmake --build build` 281 282 ### Adding a New Benchmark Series 283 284 1. Create source files in `test_src/[lang]/newseries/001/` 285 2. Update CMakeLists.txt with build rules 286 3. Executables will be auto-discovered by `./bin/benchmark newseries` 287 288 ## 🤝 Contributing 289 290 Contributions welcome! Areas of interest: 291 292 - **New Languages**: Julia, OCaml, Erlang, Elixir, Kotlin Native, Swift 293 - **New Benchmarks**: Sorting algorithms, matrix operations, string processing 294 - **Optimizations**: Language-specific optimizations while maintaining algorithm parity 295 - **Documentation**: Detailed analysis of compiler optimizations 296 297 ## 📜 License 298 299 MIT License - see LICENSE file for details. 300 301 ## 🙏 Acknowledgments 302 303 - [Hyperfine](https://github.com/sharkdp/hyperfine) - Excellent benchmarking tool 304 - The compiler teams behind GCC, Clang, and all language-specific compilers 305 - Open source language communities 306 307 ## 📚 References 308 309 - [Computer Language Benchmarks Game](https://benchmarksgame-team.pages.debian.net/benchmarksgame/) 310 - [Programming Language Comparison](https://programming-language-benchmarks.vercel.app/) 311 - [Compiler Explorer](https://godbolt.org/) - Analyze compiler output 312 313 --- 314 315 **Note**: Benchmark results are system-dependent. Your mileage may vary based on CPU architecture, OS, compiler versions, and system load. All results in this README are from a Linux system with 16 cores.