CS_learn/hello-algo
1
0
Fork 0
mirror of https://github.com/krahets/hello-algo.git synced 2026-07-07 03:26:39 +08:00
Code Issues Packages Projects Releases Wiki Activity

deploy

Browse Source
This commit is contained in:
krahets
2025-12-31 19:38:45 +08:00
parent fdc032e300
commit f31cbc3f9f
474 changed files with 130810 additions and 103691 deletions
Download Patch File Download Diff File Expand all files Collapse all files

290
en/chapter_data_structure/character_encoding/index.html
View File

@@ -37,7 +37,7 @@
<title>3.4 Character encoding * - Hello Algo</title>
<title>3.4 Character Encoding * - Hello Algo</title>
@@ -58,8 +58,8 @@
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,300i,400,400i,700,700i%7CRoboto+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Roboto";--md-code-font:"Roboto Mono"}</style>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Lato:300,300i,400,400i,700,700i%7CJetBrains+Mono:400,400i,700,700i&display=fallback">
<style>:root{--md-text-font:"Lato";--md-code-font:"JetBrains Mono"}</style>
@@ -154,7 +154,7 @@
<div class="md-header__topic" data-md-component="header-topic">
<span class="md-ellipsis">
3.4 Character encoding *
3.4 Character Encoding *
</span>
</div>
@@ -371,7 +371,7 @@
<span class="md-ellipsis">
Before starting
Before Starting
@@ -388,7 +388,7 @@
<span class="md-nav__icon md-icon"></span>
Before starting
Before Starting
</label>
@@ -487,7 +487,7 @@
<span class="md-ellipsis">
0.1 About this book
0.1 About This Book
@@ -515,7 +515,7 @@
<span class="md-ellipsis">
0.2 How to read
0.2 How to Use This Book
@@ -604,7 +604,7 @@
<span class="md-ellipsis">
Chapter 1. Encounter with algorithms
Chapter 1. Encounter With Algorithms
@@ -626,7 +626,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 1. Encounter with algorithms
Chapter 1. Encounter With Algorithms
</label>
@@ -648,7 +648,7 @@
<span class="md-ellipsis">
1.1 Algorithms are everywhere
1.1 Algorithms Are Everywhere
@@ -676,7 +676,7 @@
<span class="md-ellipsis">
1.2 What is an algorithm
1.2 What Is an Algorithm
@@ -769,7 +769,7 @@
<span class="md-ellipsis">
Chapter 2. Complexity analysis
Chapter 2. Complexity Analysis
@@ -791,7 +791,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 2. Complexity analysis
Chapter 2. Complexity Analysis
</label>
@@ -813,7 +813,7 @@
<span class="md-ellipsis">
2.1 Algorithm efficiency assessment
2.1 Algorithm Efficiency Evaluation
@@ -841,7 +841,7 @@
<span class="md-ellipsis">
2.2 Iteration and recursion
2.2 Iteration and Recursion
@@ -869,7 +869,7 @@
<span class="md-ellipsis">
2.3 Time complexity
2.3 Time Complexity
@@ -897,7 +897,7 @@
<span class="md-ellipsis">
2.4 Space complexity
2.4 Space Complexity
@@ -992,7 +992,7 @@
<span class="md-ellipsis">
Chapter 3. Data structures
Chapter 3. Data Structures
@@ -1014,7 +1014,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 3. Data structures
Chapter 3. Data Structures
</label>
@@ -1036,7 +1036,7 @@
<span class="md-ellipsis">
3.1 Classification of data structures
3.1 Classification of Data Structures
@@ -1064,7 +1064,7 @@
<span class="md-ellipsis">
3.2 Basic data types
3.2 Basic Data Types
@@ -1092,7 +1092,7 @@
<span class="md-ellipsis">
3.3 Number encoding *
3.3 Number Encoding *
@@ -1129,7 +1129,7 @@
<span class="md-ellipsis">
3.4 Character encoding *
3.4 Character Encoding *
@@ -1147,7 +1147,7 @@
<span class="md-ellipsis">
3.4 Character encoding *
3.4 Character Encoding *
@@ -1176,7 +1176,7 @@
<a href="#341-ascii-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.1 &nbsp; ASCII character set
3.4.1 &nbsp; Ascii Character Set
</span>
</a>
@@ -1187,7 +1187,7 @@
<a href="#342-gbk-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.2 &nbsp; GBK character set
3.4.2 &nbsp; Gbk Character Set
</span>
</a>
@@ -1198,7 +1198,7 @@
<a href="#343-unicode-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.3 &nbsp; Unicode character set
3.4.3 &nbsp; Unicode Character Set
</span>
</a>
@@ -1209,7 +1209,7 @@
<a href="#344-utf-8-encoding" class="md-nav__link">
<span class="md-ellipsis">
3.4.4 &nbsp; UTF-8 encoding
3.4.4 &nbsp; Utf-8 Encoding
</span>
</a>
@@ -1220,7 +1220,7 @@
<a href="#345-character-encoding-in-programming-languages" class="md-nav__link">
<span class="md-ellipsis">
3.4.5 &nbsp; Character encoding in programming languages
3.4.5 &nbsp; Character Encoding in Programming Languages
</span>
</a>
@@ -1315,7 +1315,7 @@
<span class="md-ellipsis">
Chapter 4. Array and linked list
Chapter 4. Array and Linked List
@@ -1337,7 +1337,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 4. Array and linked list
Chapter 4. Array and Linked List
</label>
@@ -1387,7 +1387,7 @@
<span class="md-ellipsis">
4.2 Linked list
4.2 Linked List
@@ -1443,7 +1443,7 @@
<span class="md-ellipsis">
4.4 Memory and cache *
4.4 Memory and Cache *
@@ -1534,7 +1534,7 @@
<span class="md-ellipsis">
Chapter 5. Stack and queue
Chapter 5. Stack and Queue
@@ -1556,7 +1556,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 5. Stack and queue
Chapter 5. Stack and Queue
</label>
@@ -1634,7 +1634,7 @@
<span class="md-ellipsis">
5.3 Double-ended queue
5.3 Double-Ended Queue
@@ -1725,7 +1725,7 @@
<span class="md-ellipsis">
Chapter 6. Hash table
Chapter 6. Hashing
@@ -1747,7 +1747,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 6. Hash table
Chapter 6. Hashing
</label>
@@ -1769,7 +1769,7 @@
<span class="md-ellipsis">
6.1 Hash table
6.1 Hash Table
@@ -1797,7 +1797,7 @@
<span class="md-ellipsis">
6.2 Hash collision
6.2 Hash Collision
@@ -1825,7 +1825,7 @@
<span class="md-ellipsis">
6.3 Hash algorithm
6.3 Hash Algorithm
@@ -1964,7 +1964,7 @@
<span class="md-ellipsis">
7.1 Binary tree
7.1 Binary Tree
@@ -1992,7 +1992,7 @@
<span class="md-ellipsis">
7.2 Binary tree traversal
7.2 Binary Tree Traversal
@@ -2020,7 +2020,7 @@
<span class="md-ellipsis">
7.3 Array Representation of tree
7.3 Array Representation of Tree
@@ -2048,7 +2048,7 @@
<span class="md-ellipsis">
7.4 Binary Search tree
7.4 Binary Search Tree
@@ -2076,7 +2076,7 @@
<span class="md-ellipsis">
7.5 AVL tree *
7.5 AVL Tree *
@@ -2239,7 +2239,7 @@
<span class="md-ellipsis">
8.2 Building a heap
8.2 Building a Heap
@@ -2267,7 +2267,7 @@
<span class="md-ellipsis">
8.3 Top-k problem
8.3 Top-K Problem
@@ -2430,7 +2430,7 @@
<span class="md-ellipsis">
9.2 Basic graph operations
9.2 Basic Operations on Graphs
@@ -2458,7 +2458,7 @@
<span class="md-ellipsis">
9.3 Graph traversal
9.3 Graph Traversal
@@ -2597,7 +2597,7 @@
<span class="md-ellipsis">
10.1 Binary search
10.1 Binary Search
@@ -2625,7 +2625,7 @@
<span class="md-ellipsis">
10.2 Binary search insertion
10.2 Binary Search Insertion
@@ -2653,7 +2653,7 @@
<span class="md-ellipsis">
10.3 Binary search boundaries
10.3 Binary Search Edge Cases
@@ -2681,7 +2681,7 @@
<span class="md-ellipsis">
10.4 Hashing optimization strategies
10.4 Hash Optimization Strategy
@@ -2709,7 +2709,7 @@
<span class="md-ellipsis">
10.5 Search algorithms revisited
10.5 Search Algorithms Revisited
@@ -2858,7 +2858,7 @@
<span class="md-ellipsis">
11.1 Sorting algorithms
11.1 Sorting Algorithms
@@ -2886,7 +2886,7 @@
<span class="md-ellipsis">
11.2 Selection sort
11.2 Selection Sort
@@ -2914,7 +2914,7 @@
<span class="md-ellipsis">
11.3 Bubble sort
11.3 Bubble Sort
@@ -2942,7 +2942,7 @@
<span class="md-ellipsis">
11.4 Insertion sort
11.4 Insertion Sort
@@ -2970,7 +2970,7 @@
<span class="md-ellipsis">
11.5 Quick sort
11.5 Quick Sort
@@ -2998,7 +2998,7 @@
<span class="md-ellipsis">
11.6 Merge sort
11.6 Merge Sort
@@ -3026,7 +3026,7 @@
<span class="md-ellipsis">
11.7 Heap sort
11.7 Heap Sort
@@ -3054,7 +3054,7 @@
<span class="md-ellipsis">
11.8 Bucket sort
11.8 Bucket Sort
@@ -3082,7 +3082,7 @@
<span class="md-ellipsis">
11.9 Counting sort
11.9 Counting Sort
@@ -3110,7 +3110,7 @@
<span class="md-ellipsis">
11.10 Radix sort
11.10 Radix Sort
@@ -3203,7 +3203,7 @@
<span class="md-ellipsis">
Chapter 12. Divide and conquer
Chapter 12. Divide and Conquer
@@ -3225,7 +3225,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 12. Divide and conquer
Chapter 12. Divide and Conquer
</label>
@@ -3247,7 +3247,7 @@
<span class="md-ellipsis">
12.1 Divide and conquer algorithms
12.1 Divide and Conquer Algorithms
@@ -3275,7 +3275,7 @@
<span class="md-ellipsis">
12.2 Divide and conquer search strategy
12.2 Divide and Conquer Search Strategy
@@ -3303,7 +3303,7 @@
<span class="md-ellipsis">
12.3 Building binary tree problem
12.3 Building a Binary Tree Problem
@@ -3331,7 +3331,7 @@
<span class="md-ellipsis">
12.4 Tower of Hanoi Problem
12.4 Hanoi Tower Problem
@@ -3468,7 +3468,7 @@
<span class="md-ellipsis">
13.1 Backtracking algorithms
13.1 Backtracking Algorithm
@@ -3496,7 +3496,7 @@
<span class="md-ellipsis">
13.2 Permutation problem
13.2 Permutations Problem
@@ -3524,7 +3524,7 @@
<span class="md-ellipsis">
13.3 Subset sum problem
13.3 Subset-Sum Problem
@@ -3552,7 +3552,7 @@
<span class="md-ellipsis">
13.4 n queens problem
13.4 N-Queens Problem
@@ -3649,7 +3649,7 @@
<span class="md-ellipsis">
Chapter 14. Dynamic programming
Chapter 14. Dynamic Programming
@@ -3671,7 +3671,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 14. Dynamic programming
Chapter 14. Dynamic Programming
</label>
@@ -3693,7 +3693,7 @@
<span class="md-ellipsis">
14.1 Introduction to dynamic programming
14.1 Introduction to Dynamic Programming
@@ -3721,7 +3721,7 @@
<span class="md-ellipsis">
14.2 Characteristics of DP problems
14.2 Characteristics of Dynamic Programming Problems
@@ -3749,7 +3749,7 @@
<span class="md-ellipsis">
14.3 DP problem-solving approach
14.3 Dynamic Programming Problem-Solving Approach
@@ -3777,7 +3777,7 @@
<span class="md-ellipsis">
14.4 0-1 Knapsack problem
14.4 0-1 Knapsack Problem
@@ -3805,7 +3805,7 @@
<span class="md-ellipsis">
14.5 Unbounded knapsack problem
14.5 Unbounded Knapsack Problem
@@ -3833,7 +3833,7 @@
<span class="md-ellipsis">
14.6 Edit distance problem
14.6 Edit Distance Problem
@@ -3970,7 +3970,7 @@
<span class="md-ellipsis">
15.1 Greedy algorithms
15.1 Greedy Algorithm
@@ -3998,7 +3998,7 @@
<span class="md-ellipsis">
15.2 Fractional knapsack problem
15.2 Fractional Knapsack Problem
@@ -4026,7 +4026,7 @@
<span class="md-ellipsis">
15.3 Maximum capacity problem
15.3 Maximum Capacity Problem
@@ -4054,7 +4054,7 @@
<span class="md-ellipsis">
15.4 Maximum product cutting problem
15.4 Maximum Product Cutting Problem
@@ -4187,7 +4187,7 @@
<span class="md-ellipsis">
16.1 Installation
16.1 Programming Environment Installation
@@ -4215,7 +4215,7 @@
<span class="md-ellipsis">
16.2 Contributing
16.2 Contributing Together
@@ -4243,7 +4243,7 @@
<span class="md-ellipsis">
16.3 Terminology
16.3 Terminology Table
@@ -4360,7 +4360,7 @@
<a href="#341-ascii-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.1 &nbsp; ASCII character set
3.4.1 &nbsp; Ascii Character Set
</span>
</a>
@@ -4371,7 +4371,7 @@
<a href="#342-gbk-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.2 &nbsp; GBK character set
3.4.2 &nbsp; Gbk Character Set
</span>
</a>
@@ -4382,7 +4382,7 @@
<a href="#343-unicode-character-set" class="md-nav__link">
<span class="md-ellipsis">
3.4.3 &nbsp; Unicode character set
3.4.3 &nbsp; Unicode Character Set
</span>
</a>
@@ -4393,7 +4393,7 @@
<a href="#344-utf-8-encoding" class="md-nav__link">
<span class="md-ellipsis">
3.4.4 &nbsp; UTF-8 encoding
3.4.4 &nbsp; Utf-8 Encoding
</span>
</a>
@@ -4404,7 +4404,7 @@
<a href="#345-character-encoding-in-programming-languages" class="md-nav__link">
<span class="md-ellipsis">
3.4.5 &nbsp; Character encoding in programming languages
3.4.5 &nbsp; Character Encoding in Programming Languages
</span>
</a>
@@ -4448,70 +4448,70 @@
<!-- Page content -->
<h1 id="34-character-encoding">3.4 &nbsp; Character encoding *<a class="headerlink" href="#34-character-encoding" title="Permanent link">&para;</a></h1>
<p>In the computer system, all data is stored in binary form, and <code>char</code> is no exception. To represent characters, we need to develop a "character set" that defines a one-to-one mapping between each character and binary numbers. With the character set, computers can convert binary numbers to characters by looking up the table.</p>
<h2 id="341-ascii-character-set">3.4.1 &nbsp; ASCII character set<a class="headerlink" href="#341-ascii-character-set" title="Permanent link">&para;</a></h2>
<p>The <u>ASCII code</u> is one of the earliest character sets, officially known as the American Standard Code for Information Interchange. It uses 7 binary digits (the lower 7 bits of a byte) to represent a character, allowing for a maximum of 128 different characters. As shown in Figure 3-6, ASCII includes uppercase and lowercase English letters, numbers 0 ~ 9, various punctuation marks, and certain control characters (such as newline and tab).</p>
<h1 id="34-character-encoding">3.4 &nbsp; Character Encoding *<a class="headerlink" href="#34-character-encoding" title="Permanent link">&para;</a></h1>
<p>In computers, all data is stored in binary form, and character <code>char</code> is no exception. To represent characters, we need to establish a "character set" that defines a one-to-one correspondence between each character and binary numbers. With a character set, computers can convert binary numbers to characters by looking up the table.</p>
<h2 id="341-ascii-character-set">3.4.1 &nbsp; Ascii Character Set<a class="headerlink" href="#341-ascii-character-set" title="Permanent link">&para;</a></h2>
<p><u>ASCII code</u> is the earliest character set, with the full name American Standard Code for Information Interchange. It uses 7 binary bits (the lower 7 bits of one byte) to represent a character, and can represent a maximum of 128 different characters. As shown in Figure 3-6, ASCII code includes uppercase and lowercase English letters, numbers 0 ~ 9, some punctuation marks, and some control characters (such as newline and tab).</p>
<p><a class="glightbox" href="../character_encoding.assets/ascii_table.png" data-type="image" data-width="100%" data-height="auto" data-desc-position="bottom"><img alt="ASCII code" class="animation-figure" src="../character_encoding.assets/ascii_table.png" /></a></p>
<p align="center"> Figure 3-6 &nbsp; ASCII code </p>
<p>However, <strong>ASCII can only represent English characters</strong>. With the globalization of computers, a character set called <u>EASCII</u> was developed to represent more languages. It expands from the 7-bit structure of ASCII to 8 bits, enabling the representation of 256 characters.</p>
<p>Globally, various region-specific EASCII character sets have been introduced. The first 128 characters of these sets are consistent with the ASCII, while the remaining 128 characters are defined differently to accommodate the requirements of different languages.</p>
<h2 id="342-gbk-character-set">3.4.2 &nbsp; GBK character set<a class="headerlink" href="#342-gbk-character-set" title="Permanent link">&para;</a></h2>
<p>Later, it was found that <strong>EASCII still could not meet the character requirements of many languages</strong>. For instance, there are nearly a hundred thousand Chinese characters, with several thousand used regularly. In 1980, the Standardization Administration of China released the <u>GB2312</u> character set, which included 6763 Chinese characters, essentially fulfilling the computer processing needs for the Chinese language.</p>
<p>However, GB2312 could not handle some rare and traditional characters. The <u>GBK</u> character set expands GB2312 and includes 21886 Chinese characters. In the GBK encoding scheme, ASCII characters are represented with one byte, while Chinese characters use two bytes.</p>
<h2 id="343-unicode-character-set">3.4.3 &nbsp; Unicode character set<a class="headerlink" href="#343-unicode-character-set" title="Permanent link">&para;</a></h2>
<p>With the rapid evolution of computer technology and a plethora of character sets and encoding standards, numerous problems arose. On the one hand, these character sets generally only defined characters for specific languages and could not function properly in multilingual environments. On the other hand, the existence of multiple character set standards for the same language caused garbled text when information was exchanged between computers using different encoding standards.</p>
<p>Researchers of that era thought: <strong>What if a comprehensive character set encompassing all global languages and symbols was developed? Wouldn't this resolve the issues associated with cross-linguistic environments and garbled text?</strong> Inspired by this idea, the extensive character set, Unicode, was born.</p>
<p><u>Unicode</u> is referred to as "统一码" (Unified Code) in Chinese, theoretically capable of accommodating over a million characters. It aims to incorporate characters from all over the world into a single set, providing a universal character set for processing and displaying various languages and reducing the issues of garbled text due to different encoding standards.</p>
<p>Since its release in 1991, Unicode has continually expanded to include new languages and characters. As of September 2022, Unicode contains 149,186 characters, including characters, symbols, and even emojis from various languages. In the vast Unicode character set, commonly used characters occupy 2 bytes, while some rare characters may occupy 3 or even 4 bytes.</p>
<p>Unicode is a universal character set that assigns a number (called a "code point") to each character, <strong>but it does not specify how these character code points should be stored in a computer system</strong>. One might ask: How does a system interpret Unicode code points of varying lengths within a text? For example, given a 2-byte code, how does the system determine if it represents a single 2-byte character or two 1-byte characters?</p>
<p><strong>A straightforward solution to this problem is to store all characters as equal-length encodings</strong>. As shown in Figure 3-7, each character in "Hello" occupies 1 byte, while each character in "算法" (algorithm) occupies 2 bytes. We could encode all characters in "Hello 算法" as 2 bytes by padding the higher bits with zeros. This method would enable the system to interpret a character every 2 bytes, recovering the content of the phrase.</p>
<p>However, <strong>ASCII code can only represent English</strong>. With the globalization of computers, a character set called <u>EASCII</u> that can represent more languages emerged. It expands from the 7-bit basis of ASCII to 8 bits, and can represent 256 different characters.</p>
<p>Worldwide, a batch of EASCII character sets suitable for different regions have appeared successively. The first 128 characters of these character sets are unified as ASCII code, and the last 128 characters are defined differently to adapt to the needs of different languages.</p>
<h2 id="342-gbk-character-set">3.4.2 &nbsp; Gbk Character Set<a class="headerlink" href="#342-gbk-character-set" title="Permanent link">&para;</a></h2>
<p>Later, people found that <strong>EASCII code still cannot meet the character quantity requirements of many languages</strong>. For example, there are nearly one hundred thousand Chinese characters, and several thousand are used daily. In 1980, the China National Standardization Administration released the <u>GB2312</u> character set, which included 6,763 Chinese characters, basically meeting the needs for computer processing of Chinese characters.</p>
<p>However, GB2312 cannot handle some rare characters and traditional Chinese characters. The <u>GBK</u> character set is an extension based on GB2312, which includes a total of 21,886 Chinese characters. In the GBK encoding scheme, ASCII characters are represented using one byte, and Chinese characters are represented using two bytes.</p>
<h2 id="343-unicode-character-set">3.4.3 &nbsp; Unicode Character Set<a class="headerlink" href="#343-unicode-character-set" title="Permanent link">&para;</a></h2>
<p>With the vigorous development of computer technology, character sets and encoding standards flourished, which brought many problems. On the one hand, these character sets generally only define characters for specific languages and cannot work normally in multilingual environments. On the other hand, multiple character set standards exist for the same language, and if two computers use different encoding standards, garbled characters will appear during information transmission.</p>
<p>Researchers of that era thought: <strong>If a sufficiently complete character set is released that includes all languages and symbols in the world, wouldn't it be possible to solve cross-language environment and garbled character problems</strong>? Driven by this idea, a large and comprehensive character set, Unicode, was born.</p>
<p><u>Unicode</u> is called "统一码" (Unified Code) in Chinese and can theoretically accommodate over one million characters. It is committed to including characters from around the world into a unified character set, providing a universal character set to handle and display various language texts, reducing garbled character problems caused by different encoding standards.</p>
<p>Since its release in 1991, Unicode has continuously expanded to include new languages and characters. As of September 2022, Unicode has included 149,186 characters, including characters, symbols, and even emojis from various languages. In the vast Unicode character set, commonly used characters occupy 2 bytes, and some rare characters occupy 3 bytes or even 4 bytes.</p>
<p>Unicode is a universal character set that essentially assigns a number (called a "code point") to each character, <strong>but it does not specify how to store these character code points in computers</strong>. We can't help but ask: when Unicode code points of multiple lengths appear simultaneously in a text, how does the system parse the characters? For example, given an encoding with a length of 2 bytes, how does the system determine whether it is one 2-byte character or two 1-byte characters?</p>
<p>For the above problem, <strong>a straightforward solution is to store all characters as equal-length encodings</strong>. As shown in Figure 3-7, each character in "Hello" occupies 1 byte, and each character in "算法" (algorithm) occupies 2 bytes. We can encode all characters in "Hello 算法" as 2 bytes in length by padding the high bits with 0. In this way, the system can parse one character every 2 bytes and restore the content of this phrase.</p>
<p><a class="glightbox" href="../character_encoding.assets/unicode_hello_algo.png" data-type="image" data-width="100%" data-height="auto" data-desc-position="bottom"><img alt="Unicode encoding example" class="animation-figure" src="../character_encoding.assets/unicode_hello_algo.png" /></a></p>
<p align="center"> Figure 3-7 &nbsp; Unicode encoding example </p>
<p>However, as ASCII has shown us, encoding English only requires 1 byte. Using the above approach would double the space occupied by English text compared to ASCII encoding, which is a waste of memory space. Therefore, a more efficient Unicode encoding method is needed.</p>
<h2 id="344-utf-8-encoding">3.4.4 &nbsp; UTF-8 encoding<a class="headerlink" href="#344-utf-8-encoding" title="Permanent link">&para;</a></h2>
<p>Currently, UTF-8 has become the most widely used Unicode encoding method internationally. <strong>It is a variable-length encoding</strong>, using 1 to 4 bytes to represent a character, depending on the complexity of the character. ASCII characters need only 1 byte, Latin and Greek letters require 2 bytes, commonly used Chinese characters need 3 bytes, and some other rare characters need 4 bytes.</p>
<p>The encoding rules for UTF-8 are not complex and can be divided into two cases:</p>
<p>However, ASCII code has already proven to us that encoding English only requires 1 byte. If the above scheme is adopted, the size of English text will be twice that under ASCII encoding, which is very wasteful of memory space. Therefore, we need a more efficient Unicode encoding method.</p>
<h2 id="344-utf-8-encoding">3.4.4 &nbsp; Utf-8 Encoding<a class="headerlink" href="#344-utf-8-encoding" title="Permanent link">&para;</a></h2>
<p>Currently, UTF-8 has become the most widely used Unicode encoding method internationally. <strong>It is a variable-length encoding</strong> that uses 1 to 4 bytes to represent a character, depending on the complexity of the character. ASCII characters only require 1 byte, Latin and Greek letters require 2 bytes, commonly used Chinese characters require 3 bytes, and some other rare characters require 4 bytes.</p>
<p>The encoding rules of UTF-8 are not complicated and can be divided into the following two cases.</p>
<ul>
<li>For 1-byte characters, set the highest bit to <span class="arithmatex">\(0\)</span>, and the remaining 7 bits to the Unicode code point. Notably, ASCII characters occupy the first 128 code points in the Unicode set. This means that <strong>UTF-8 encoding is backward compatible with ASCII</strong>. This implies that UTF-8 can be used to parse ancient ASCII text.</li>
<li>For characters of length <span class="arithmatex">\(n\)</span> bytes (where <span class="arithmatex">\(n &gt; 1\)</span>), set the highest <span class="arithmatex">\(n\)</span> bits of the first byte to <span class="arithmatex">\(1\)</span>, and the <span class="arithmatex">\((n + 1)^{\text{th}}\)</span> bit to <span class="arithmatex">\(0\)</span>; starting from the second byte, set the highest 2 bits of each byte to <span class="arithmatex">\(10\)</span>; the rest of the bits are used to fill the Unicode code point.</li>
<li>For 1-byte characters, set the highest bit to <span class="arithmatex">\(0\)</span>, and set the remaining 7 bits to the Unicode code point. It is worth noting that ASCII characters occupy the first 128 code points in the Unicode character set. That is to say, <strong>UTF-8 encoding is backward compatible with ASCII code</strong>. This means we can use UTF-8 to parse very old ASCII code text.</li>
<li>For characters with a length of <span class="arithmatex">\(n\)</span> bytes (where <span class="arithmatex">\(n &gt; 1\)</span>), set the highest <span class="arithmatex">\(n\)</span> bits of the first byte to <span class="arithmatex">\(1\)</span>, and set the <span class="arithmatex">\((n + 1)\)</span>-th bit to <span class="arithmatex">\(0\)</span>; starting from the second byte, set the highest 2 bits of each byte to <span class="arithmatex">\(10\)</span>; use all remaining bits to fill in the Unicode code point of the character.</li>
</ul>
<p>Figure 3-8 shows the UTF-8 encoding for "Hello算法". It can be observed that since the highest <span class="arithmatex">\(n\)</span> bits are set to <span class="arithmatex">\(1\)</span>, the system can determine the length of the character as <span class="arithmatex">\(n\)</span> by counting the number of highest bits set to <span class="arithmatex">\(1\)</span>.</p>
<p>But why set the highest 2 bits of the remaining bytes to <span class="arithmatex">\(10\)</span>? Actually, this <span class="arithmatex">\(10\)</span> serves as a kind of checksum. If the system starts parsing text from an incorrect byte, the <span class="arithmatex">\(10\)</span> at the beginning of the byte can help the system quickly detect anomalies.</p>
<p>The reason for using <span class="arithmatex">\(10\)</span> as a checksum is that, under UTF-8 encoding rules, it's impossible for the highest two bits of a character to be <span class="arithmatex">\(10\)</span>. This can be proven by contradiction: If the highest two bits of a character are <span class="arithmatex">\(10\)</span>, it indicates that the character's length is <span class="arithmatex">\(1\)</span>, corresponding to ASCII. However, the highest bit of an ASCII character should be <span class="arithmatex">\(0\)</span>, which contradicts the assumption.</p>
<p>Figure 3-8 shows the UTF-8 encoding corresponding to "Hello算法". It can be observed that since the highest <span class="arithmatex">\(n\)</span> bits are all set to <span class="arithmatex">\(1\)</span>, the system can parse the length of the character as <span class="arithmatex">\(n\)</span> by reading the number of highest bits that are <span class="arithmatex">\(1\)</span>.</p>
<p>But why set the highest 2 bits of all other bytes to <span class="arithmatex">\(10\)</span>? In fact, this <span class="arithmatex">\(10\)</span> can serve as a check symbol. Assuming the system starts parsing text from an incorrect byte, the <span class="arithmatex">\(10\)</span> at the beginning of the byte can help the system quickly determine an anomaly.</p>
<p>The reason for using <span class="arithmatex">\(10\)</span> as a check symbol is that under UTF-8 encoding rules, it is impossible for a character's highest two bits to be <span class="arithmatex">\(10\)</span>. This conclusion can be proven by contradiction: assuming the highest two bits of a character are <span class="arithmatex">\(10\)</span>, it means the length of the character is <span class="arithmatex">\(1\)</span>, corresponding to ASCII code. However, the highest bit of ASCII code should be <span class="arithmatex">\(0\)</span>, which contradicts the assumption.</p>
<p><a class="glightbox" href="../character_encoding.assets/utf-8_hello_algo.png" data-type="image" data-width="100%" data-height="auto" data-desc-position="bottom"><img alt="UTF-8 encoding example" class="animation-figure" src="../character_encoding.assets/utf-8_hello_algo.png" /></a></p>
<p align="center"> Figure 3-8 &nbsp; UTF-8 encoding example </p>
<p>Apart from UTF-8, other common encoding methods include:</p>
<p>In addition to UTF-8, common encoding methods also include the following two.</p>
<ul>
<li><strong>UTF-16 encoding</strong>: Uses 2 or 4 bytes to represent a character. All ASCII characters and commonly used non-English characters are represented with 2 bytes; a few characters require 4 bytes. For 2-byte characters, the UTF-16 encoding equals the Unicode code point.</li>
<li><strong>UTF-32 encoding</strong>: Every character uses 4 bytes. This means UTF-32 occupies more space than UTF-8 and UTF-16, especially for texts with a high proportion of ASCII characters.</li>
<li><strong>UTF-16 encoding</strong>: Uses 2 or 4 bytes to represent a character. All ASCII characters and commonly used non-English characters are represented with 2 bytes; a few characters need to use 4 bytes. For 2-byte characters, UTF-16 encoding is equal to the Unicode code point.</li>
<li><strong>UTF-32 encoding</strong>: Every character uses 4 bytes. This means that UTF-32 takes up more space than UTF-8 and UTF-16, especially for text with a high proportion of ASCII characters.</li>
</ul>
<p>From the perspective of storage space, using UTF-8 to represent English characters is very efficient because it only requires 1 byte; using UTF-16 to encode some non-English characters (such as Chinese) can be more efficient because it only requires 2 bytes, while UTF-8 might need 3 bytes.</p>
<p>From a compatibility perspective, UTF-8 is the most versatile, with many tools and libraries supporting UTF-8 as a priority.</p>
<h2 id="345-character-encoding-in-programming-languages">3.4.5 &nbsp; Character encoding in programming languages<a class="headerlink" href="#345-character-encoding-in-programming-languages" title="Permanent link">&para;</a></h2>
<p>Historically, many programming languages utilized fixed-length encodings such as UTF-16 or UTF-32 for processing strings during program execution. This allows strings to be handled as arrays, offering several advantages:</p>
<p>From the perspective of storage space occupation, using UTF-8 to represent English characters is very efficient because it only requires 1 byte; using UTF-16 encoding for some non-English characters (such as Chinese) will be more efficient because it only requires 2 bytes, while UTF-8 may require 3 bytes.</p>
<p>From a compatibility perspective, UTF-8 has the best universality, and many tools and libraries support UTF-8 first.</p>
<h2 id="345-character-encoding-in-programming-languages">3.4.5 &nbsp; Character Encoding in Programming Languages<a class="headerlink" href="#345-character-encoding-in-programming-languages" title="Permanent link">&para;</a></h2>
<p>For most past programming languages, strings during program execution use fixed-length encodings such as UTF-16 or UTF-32. Under fixed-length encoding, we can treat strings as arrays for processing, and this approach has the following advantages.</p>
<ul>
<li><strong>Random access</strong>: Strings encoded in UTF-16 can be accessed randomly with ease. For UTF-8, which is a variable-length encoding, locating the <span class="arithmatex">\(i^{th}\)</span> character requires traversing the string from the start to the <span class="arithmatex">\(i^{th}\)</span> position, taking <span class="arithmatex">\(O(n)\)</span> time.</li>
<li><strong>Character counting</strong>: Similar to random access, counting the number of characters in a UTF-16 encoded string is an <span class="arithmatex">\(O(1)\)</span> operation. However, counting characters in a UTF-8 encoded string requires traversing the entire string.</li>
<li><strong>String operations</strong>: Many string operations like splitting, concatenating, inserting, and deleting are easier on UTF-16 encoded strings. These operations generally require additional computation on UTF-8 encoded strings to ensure the validity of the UTF-8 encoding.</li>
<li><strong>Random access</strong>: UTF-16 encoded strings can be easily accessed randomly. UTF-8 is a variable-length encoding. To find the <span class="arithmatex">\(i\)</span>-th character, we need to traverse from the beginning of the string to the <span class="arithmatex">\(i\)</span>-th character, which requires <span class="arithmatex">\(O(n)\)</span> time.</li>
<li><strong>Character counting</strong>: Similar to random access, calculating the length of a UTF-16 encoded string is also an <span class="arithmatex">\(O(1)\)</span> operation. However, calculating the length of a UTF-8 encoded string requires traversing the entire string.</li>
<li><strong>String operations</strong>: Many string operations (such as splitting, joining, inserting, deleting, etc.) on UTF-16 encoded strings are easier to perform. Performing these operations on UTF-8 encoded strings usually requires additional calculations to ensure that invalid UTF-8 encoding is not generated.</li>
</ul>
<p>The design of character encoding schemes in programming languages is an interesting topic involving various factors:</p>
<p>In fact, the design of character encoding schemes for programming languages is a very interesting topic involving many factors.</p>
<ul>
<li>Java’s <code>String</code> type uses UTF-16 encoding, with each character occupying 2 bytes. This was based on the initial belief that 16 bits were sufficient to represent all possible characters and proven incorrect later. As the Unicode standard expanded beyond 16 bits, characters in Java may now be represented by a pair of 16-bit values, known as “surrogate pairs.”</li>
<li>JavaScript and TypeScript use UTF-16 encoding for similar reasons as Java. When JavaScript was first introduced by Netscape in 1995, Unicode was still in its early stages, and 16-bit encoding was sufficient to represent all Unicode characters.</li>
<li>C# uses UTF-16 encoding, largely because the .NET platform, designed by Microsoft, and many Microsoft technologies, including the Windows operating system, extensively use UTF-16 encoding.</li>
<li>Java's <code>String</code> type uses UTF-16 encoding, with each character occupying 2 bytes. This is because at the beginning of Java language design, people believed that 16 bits were sufficient to represent all possible characters. However, this was an incorrect judgment. Later, the Unicode specification expanded beyond 16 bits, so characters in Java may now be represented by a pair of 16-bit values (called "surrogate pairs").</li>
<li>The strings of JavaScript and TypeScript use UTF-16 encoding for reasons similar to Java. When Netscape first introduced the JavaScript language in 1995, Unicode was still in its early stages of development, and at that time, using 16-bit encoding was sufficient to represent all Unicode characters.</li>
<li>C# uses UTF-16 encoding mainly because the .NET platform was designed by Microsoft, and many of Microsoft's technologies (including the Windows operating system) extensively use UTF-16 encoding.</li>
</ul>
<p>Due to the underestimation of character counts, these languages had to use "surrogate pairs" to represent Unicode characters exceeding 16 bits. This approach has its drawbacks: strings containing surrogate pairs may have characters occupying 2 or 4 bytes, losing the advantage of fixed-length encoding. Additionally, handling surrogate pairs adds complexity and debugging difficulty to programming.</p>
<p>Addressing these challenges, some languages have adopted alternative encoding strategies:</p>
<p>Due to the underestimation of character quantities by the above programming languages, they had to adopt the "surrogate pair" method to represent Unicode characters with lengths exceeding 16 bits. This is a reluctant compromise. On the one hand, in strings containing surrogate pairs, one character may occupy 2 bytes or 4 bytes, thus losing the advantage of fixed-length encoding. On the other hand, handling surrogate pairs requires additional code, which increases the complexity and difficulty of debugging in programming.</p>
<p>For the above reasons, some programming languages have proposed different encoding schemes.</p>
<ul>
<li>Python’s <code>str</code> type uses Unicode encoding with a flexible representation where the storage length of characters depends on the largest Unicode code point in the string. If all characters are ASCII, each character occupies 1 byte, 2 bytes for characters within the Basic Multilingual Plane (BMP), and 4 bytes for characters beyond the BMP.</li>
<li>Go’s <code>string</code> type internally uses UTF-8 encoding. Go also provides the <code>rune</code> type for representing individual Unicode code points.</li>
<li>Rust’s <code>str</code> and <code>String</code> types use UTF-8 encoding internally. Rust also offers the <code>char</code> type for individual Unicode code points.</li>
<li>Python's <code>str</code> uses Unicode encoding and adopts a flexible string representation where the stored character length depends on the largest Unicode code point in the string. If all characters in the string are ASCII characters, each character occupies 1 byte; if there are characters exceeding the ASCII range but all within the Basic Multilingual Plane (BMP), each character occupies 2 bytes; if there are characters exceeding the BMP, each character occupies 4 bytes.</li>
<li>Go language's <code>string</code> type uses UTF-8 encoding internally. Go language also provides the <code>rune</code> type, which is used to represent a single Unicode code point.</li>
<li>Rust language's <code>str</code> and <code>String</code> types use UTF-8 encoding internally. Rust also provides the <code>char</code> type for representing a single Unicode code point.</li>
</ul>
<p>It’s important to note that the above discussion pertains to how strings are stored in programming languages, <strong>which is different from how strings are stored in files or transmitted over networks</strong>. For file storage or network transmission, strings are usually encoded in UTF-8 format for optimal compatibility and space efficiency.</p>
<p>It should be noted that the above discussion is about how strings are stored in programming languages, <strong>which is different from how strings are stored in files or transmitted over networks</strong>. In file storage or network transmission, we usually encode strings into UTF-8 format to achieve optimal compatibility and space efficiency.</p>
<!-- Source file information -->
@@ -4534,7 +4534,7 @@ aria-label="Footer"
<a
href="../number_encoding/"
class="md-footer__link md-footer__link--prev"
aria-label="Previous: 3.3 Number encoding *"
aria-label="Previous: 3.3 Number Encoding *"
rel="prev"
>
<div class="md-footer__button md-icon">
@@ -4546,7 +4546,7 @@ aria-label="Footer"
Previous
</span>
<div class="md-ellipsis">
3.3 Number encoding *
3.3 Number Encoding *
</div>
</div>
</a>
@@ -4659,7 +4659,7 @@ aria-label="Footer"
<nav class="md-footer__inner md-grid" aria-label="Footer" >
<a href="../number_encoding/" class="md-footer__link md-footer__link--prev" aria-label="Previous: 3.3 Number encoding *">
<a href="../number_encoding/" class="md-footer__link md-footer__link--prev" aria-label="Previous: 3.3 Number Encoding *">
<div class="md-footer__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11z"/></svg>
@@ -4669,7 +4669,7 @@ aria-label="Footer"
Previous
</span>
<div class="md-ellipsis">
3.3 Number encoding *
3.3 Number Encoding *
</div>
</div>
</a>