Merge branch 'family/ambz2'

docs/TODO.md

@@ -43,7 +43,6 @@ Explicit is better than implicit.
 
 ### Other
 
-- refactor `SerialClass` to have a shared header `Serial.h` in the common core (and `SerialData`, just like WiFi). Move ADR to common core
 - implement Wire on BK, refactor the API and class
 - watchdog API
 - `Preferences` library

docs/boards_tuya_all.json

@@ -441,6 +441,34 @@
 			"datasheet_id": "Kc5x1p35fs5zf"
 		}
 	},
+	"t2": {
+		"t2-u": {
+			"mcu": "bk7231n",
+			"flash": 2097152,
+			"ram": 262144,
+			"pins_total": 21,
+			"connectivity": [
+				"wifi",
+				"ble"
+			],
+			"datasheet_name": "T2-U-module-datasheet",
+			"datasheet_id": "Kce1tncb80ldq"
+		}
+	},
+	"t1": {
+		"t1-2s": {
+			"mcu": "t1a",
+			"flash": 1048576,
+			"ram": 294912,
+			"pins_total": 11,
+			"connectivity": [
+				"wifi",
+				"ble"
+			],
+			"datasheet_name": "T1-2S-module-datasheet",
+			"datasheet_id": "Kcl2d5xjy1rly"
+		}
+	},
 	"axy": {
 		"axy2s": {
 			"mcu": "ecr6600",
@@ -493,7 +521,7 @@
 	},
 	"wt": {
 		"wt3": {
-			"mcu": "t2",
+			"mcu": "bk7231n",
 			"flash": 2097152,
 			"ram": 262144,
 			"pins_total": 16,

docs/contrib/lt-api.md

@@ -0,0 +1,13 @@
+# API functions guide
+
+The [LibreTiny C API](../dev/lt-api.md) functions are split between three types: common, weak and family.
+
+- Common functions are implemented in the base, common core and are the same between all families.
+- Weak functions are provided in the common core, but can (and sometimes should) be overridden by family cores. They sometimes provide usable default implementations (which *can* be overriden to provide e.g. a better way to do something), otherwise they're empty (e.g. if a family doesn't support such a feature).
+- Family functions are not provided in the common core and have to be implemented in the family core.
+
+A quick outline of all available functions and their types:
+
+{%
+	include-markdown "lt-api-functions.md"
+%}

docs/contrib/porting.md

@@ -0,0 +1,101 @@
+# Porting new families
+
+This document briefly outlines what needs to be done, in order to port a new chip family to LibreTiny.
+
+## Base framework + builders
+
+The base framework is the core part, that provides little functionality and a small HAL (over some things like OTA or sys control). It also includes a builder script for the vendor SDK.
+
+Here's what has to be done to make that work:
+
+1. Find vendor SDK - should be self-explanatory. We can't work without a working SDK (yet).
+2. Test vendor SDK - compile a sample program "as it was meant to be done".
+
+	- Most SDKs provide some example programs (like Hello World, WiFi scanning, etc.) that can usually be compiled by running a single "make" command.
+	- Sometimes you need to configure your environment in a weird and complicated way. For me, using Cygwin on Windows was usually enough, though.
+	- You need to flash this to the chip as well. The SDK usually bundles some flashing tools.
+	- This step is crucial to understand the vendor build system, and to have working binaries to compare out results against.
+
+3. "Clean up" vendor SDK.
+
+	- SDKs usually bundle entire compiler toolchains, which can take up hundreds of megabytes. We want to keep the downloaded PlatformIO packages as small as possible.
+	- On existing families, GitHub Workflows produce the packages by removing some files and adding `package.json` to them. See [framework-beken-bdk/.github/workflows/platformio-package.yml](https://github.com/libretiny-eu/framework-beken-bdk/blob/actions/.github/workflows/platformio-package.yml) for an example.
+
+4. Write base family and board definitions.
+
+	- `families.json` needs to have the new family added to it.
+	- `platform.json` needs to know the vendor SDK repository.
+	- Add any boards and base JSONs to the `boards/` directory. It's easiest to start with generic boards.
+	- Use `boardgen ltci` to generate variant sources (.c and .h).
+
+5. Add base core code.
+
+	- `lt_defs.h`, `lt_family.h` and `lt_api.c` files need to be created, and initialized with (even empty) functions and definitions.
+	- The list of family functions can be found [here](lt-api.md).
+	- Make the SDK call `lt_main()` as the entrypoint. If needed, use fixups.
+
+6. Write a binary manipulation tool.
+
+	- While this step could be optional, as these tools are provided in the SDK, they're usually platform-specific (i.e. Windows-only) and use proprietary executables, with no source code nor documentation. This is unacceptable for LibreTiny, as we need to support multiple architectures & platforms (Windows, Linux, Raspberry Pi, etc.). Naturally, doing that in Python seems to be the best choice.
+	- All binary tools are currently in [ltchiptool/soc/.../binary.py](https://github.com/libretiny-eu/ltchiptool/blob/master/ltchiptool/soc/bk72xx/binary.py). The `elf2bin()` function is what takes an .ELF file, and generates a set of binaries that can be flashed to the chip.
+	- It's best to test if the generation is correct, by taking an .ELF compiled by vendor SDK, running it through ltchiptool and checking if the resulting binaries are identical.
+	- Ghidra/IDA Pro is your friend here; you can decompile the SDK tools.
+
+7. Write a flashing tool.
+
+	- mostly the same as above. Refer to the existing tools for examples. It's useful to make the flasher class "standalone", i.e. a class that is then wrapped by ltchiptool, like in [`realtek-ambz2`](https://github.com/libretiny-eu/ltchiptool/blob/master/ltchiptool/soc/ambz2/util/ambz2tool.py).
+
+8. Write builder scripts.
+
+	- `builder/family/xxx.py` files are builders, which contain all SDK sources and include paths. Write the script, based on the existing families, and any Makefiles or other scripts from the SDK.
+	- Make sure not to make a mess in the `CCFLAGS`/`CPPDEFINES`, and only include what's needed there. Some flags are project-wide (family-independent) in `builder/frameworks/base.py`.
+	- Use a **pure PlatformIO** project - **not ESPHome!**. Pass one of the generic boards you created before, and `framework = base` in `platformio.ini`. Generally, try to get the thing to compile.
+	- Use a simple Hello World program - C, not C++. Only add `main()` function with a `printf()` and a `while(1)` loop.
+	- I've noticed that using `nano.specs` instead of `nosys.specs` produces smaller binaries.
+
+9. When you get it to link successfully, build a UF2 file.
+
+	- UF2 packages are for flashing and for OTA.
+	- Add `UF2OTA` to the env, to provide binaries that will go to the UF2. Some understanding of the chip's partition and flash layout will be needed.
+
+10. Flash it, test if it works!
+
+	- It probably won't. You may need to remove `__libc_init_array()` from `cores/common/base/lt_api.c` so that it doesn't crash. Most SDKs don't support C++ properly.
+
+## Making it *actually* work
+
+1. Write `flashdb` and `printf` ports.
+
+	- The ports are in `cores/.../base/port/`. It's a simple flash access layer, and a character printing function. Not a lot of work, but it needs to be done first.
+
+2. Add fixups so that string & memory stdlib functions are not from SDK.
+
+	- Refer to [stdlib.md](stdlib.md) to find functions that need to be wrapped.
+	- SDK should not define them, you have to figure out a way to remove them from headers. Fixups can mess with includes and trick the SDK into using our own functions.
+
+3. Clean up FreeRTOS.
+
+	- FreeRTOS' headers usually include some SDK headers, which pull in a lot of macros and typedefs, which usually break lots of non-SDK code, which doesn't expect these macros.
+	- [library-freertos](https://github.com/libretiny-eu/library-freertos) repo contains some FreeRTOS versions, adapted for SDKs. Basically, copy a clean (straight from FreeRTOS github) version to the repo, commit it. Then copy the version from SDK and compare the differences.
+	- Try to make it look as "stock" as possible. Discard any formatting differences (and backports).
+	- Annotate any parts that can't be removed with `#ifdef FREERTOS_PORT_REALTEK_AMB1`.
+	- Put the FreeRTOS vendor-specific port in [library-freertos-port](https://github.com/libretiny-eu/library-freertos-port).
+	- Remove all FreeRTOS sources from builder scripts. Replace with:
+
+	```py
+	env.Replace(FREERTOS_PORT=env["FAMILY_NAME"], FREERTOS_PORT_DEFINE="REALTEK_AMB1")
+	queue.AddExternalLibrary("freertos")
+	queue.AddExternalLibrary("freertos-port")
+	```
+
+4. Do the same with lwIP - later.
+
+5. Write LibreTiny C APIs - in `lt_api.c`.
+
+6. At this point, your Hello World code should work fine.
+
+## Porting Arduino Core - C++ support
+
+1. Add main.cpp and write wiring_*.c ports. GPIOs and stuff should work even without proper C++ support.
+
+2. Port Serial library first. This should already show whether C++ works fine or if it doesn't. For example, calling `Serial.println()` refers to the virtual function `Print::write`, which will probably crash the chip if C++ is not being linked properly.

docs/dev/config.md

@@ -70,7 +70,7 @@ build_flags =
 
 #### Per-module logging & debugging
 
-The following options enable library-specific logging output. They are effective **for all loglevels** - i.e. disabling `LT_DEBUG_WIFI` will disable WiFi debug messages, as well as errors.
+The following options enable library-specific logging output. They are effective **for all loglevels** - i.e. disabling `LT_DEBUG_WIFI` will disable WiFi debug messages, warnings, as well as errors.
 
 To see debug messages from i.e. OTA, loglevel must also be changed.
 
@@ -119,24 +119,43 @@ Options for controlling default UART log output.
 - `LT_MICROS_HIGH_RES` (1) - count runtime microseconds using a high-resolution timer (if possible); disable if your application doesn't need `micros()`
 - `LT_AUTO_DOWNLOAD_REBOOT` (1) - automatically reboot into "download mode" after detecting a flashing protocol command; [read more](../flashing/tools/adr.md)
 
-### Family feature config
+### Family configuration
 
 !!! bug "Warning"
 	These options are not meant for end-users. They're provided here as a reference for developers. **Do not set these options manually**.
 
 These options are selectively set by all families, as part of the build process. They are used for enabling LT core API parts, if the family has support for it. Files named `lt_defs.h`, containing these options, are read by the PlatformIO builders (note: they're never included by C code).
 
-The `LT_ARD_*` options are only used with Arduino frameworks.
+Checking for option value should be done with `#if` (not with `#ifdef`!) - if it's not defined, it will evaluate to 0. Otherwise, it will use the defined value, either 0 or 1.
 
-The meaning of most flags is as follows:
-
-- `LT_HAS_FREERTOS` - FreeRTOS supported and used
-- `LT_HAS_LWIP` - LwIP in SDK (any version)
-- `LT_HAS_LWIP2` - LwIP v2.0.0 or newer
-- `LT_HAS_MBEDTLS` - mbedTLS in SDK
-- `LT_HAS_PRINTF` - printf library implemented
-- `LT_ARD_HAS_SERIAL` - Serial class implemented, `Serial.h` available
-- `LT_ARD_HAS_SOFTSERIAL` - SoftwareSerial library implemented, `SoftwareSerial.h` available
-- `LT_ARD_HAS_WIFI` - WiFi library implemented, `WiFiData.h` available
-- `LT_HEAP_FUNC` - function name used to get available heap size (for `LT_HEAP_I()`)
-- `LT_REALLOC_FUNC` - function name used for `realloc()` call
+- family-/chip-specific hardware peripherals
+	- `LT_HW_WIFI` - WiFi supported on the chip
+	- `LT_HW_BT` - Bluetooth Classic supported on the chip
+	- `LT_HW_BLE` - Bluetooth Low Energy supported on the chip
+	- `LT_HW_WATCHDOG` - watchdog available
+	- `LT_HW_DEEP_SLEEP` - deep sleep possible
+- board-specific peripherals (note: defined in `lt_pins.h`, depending on available pin numbers)
+	- `LT_HW_UART#` - UART number # available on the board
+	- `LT_HW_I2C#` - I²C number # available on the board
+	- `LT_HW_SPI#` - SPI number # available on the board
+- family software options (SDK features, LT implementation status)
+	- `LT_HAS_FREERTOS` - FreeRTOS supported and used
+	- `LT_HAS_LWIP` - LwIP in SDK (any version)
+	- `LT_HAS_LWIP2` - LwIP v2.0.0 or newer
+	- `LT_HAS_MBEDTLS` - mbedTLS in SDK
+	- `LT_HAS_PRINTF` - printf library implemented
+	- `LT_HAS_FLASH` - FAL flash port implemented
+	- `LT_HAS_OTA` - OTA implemented in base framework
+- Arduino Core implementation status (only available and used along with Arduino framework)
+	- `LT_ARD_HAS_SERIAL` - Serial class implemented
+	- `LT_ARD_HAS_SOFTSERIAL` - SoftwareSerial library implemented
+	- `LT_ARD_HAS_WIFI` - WiFi library implemented
+	- `LT_ARD_HAS_WIRE` - Wire (I²C) library implemented
+	- `LT_ARD_HAS_SPI` - SPI library implemented
+	- `LT_ARD_MD5_POLARSSL` - use PolarSSL for MD5 library
+	- `LT_ARD_MD5_MBEDTLS` - use mbedTLS for MD5 library
+	- `LT_ARD_MD5_HOSTAPD` - use hostapd for MD5 library
+- misc options
+	- `LT_HEAP_FUNC` - function name used to get available heap size (for `LT_HEAP_I()`)
+	- `LT_REALLOC_FUNC` - function name used for `realloc()` call
+	- `LT_REMALLOC` - use `malloc()` and `memcpy()` in `realloc()` call

docs/flashing/tools/adr.md

@@ -16,3 +16,7 @@ The code listens on UART1 for a link-check command (`01 E0 FC 01 00`). The baudr
 ## Realtek AmebaZ
 
 This only works when using [ltchiptool](ltchiptool.md) for flashing. Upon starting UART communication, the tool sends `55 AA 22 E0 D6 FC` (0x55AA followed by the `realtek-ambz` family ID). After detecting that pattern, the chip proceeds to reboot into UART download mode (using [`lt_reboot_download_mode()`](../../../ltapi/lt__device_8h.md))
+
+## Realtek AmebaZ2
+
+The code listens on UART2 for a `ping\n` command, that is sent by [ltchiptool](ltchiptool.md) (and possibly by the vendor flasher, too). The device is then rebooted to download mode.

docs/resources/documents.md

@@ -17,4 +17,4 @@ UM0201 | [Ameba Common BT Application User Manual EN](https://raw.githubusercont
   | Found elsewhere
 AN0400 | [Ameba-D Application Note_v3_watermark](https://files.seeedstudio.com/products/102110419/Basic%20documents/AN0400%20Ameba-D%20Application%20Note_v3_watermark.pdf)
 AN0500 | [Realtek Ameba-ZII application note](https://www.e-paper-display.com/99IOT/00015797-AN0500-Realtek-Ameba-ZII-application-note.en_233850.pdf)
-  | [Realtek Ameba-ZII datasheet v0.8](https://www.e-paper-display.com/Ameba-Z_II_DataSheet_v0r8_RTL8720Cx_20190424%29.pdf)
+  | [Realtek Ameba-ZII datasheet v0.8](https://web.archive.org/web/20230523175304if_/https://cetest02.cn-bj.ufileos.com/100001_2110255103/RTL872xZ2%20IC%20Datasheet.pdf)

docs/script.js

@@ -0,0 +1,6 @@
+document$.subscribe(function () {
+	var tables = document.querySelectorAll("article table:not([class])")
+	tables.forEach(function (table) {
+		new Tablesort(table)
+	})
+})

docs/scripts/build_json.py

@@ -1,7 +1,7 @@
 import json
 
 from ltchiptool import Board
-from update_docs import board_obj_sort
+from write_boards import board_obj_sort
 
 boards = map(Board, Board.get_list())
 boards = list(sorted(boards, key=board_obj_sort))

docs/scripts/write_apis.py

@@ -0,0 +1,129 @@
+# Copyright (c) Kuba Szczodrzyński 2023-06-22.
+
+import re
+from glob import glob
+from os.path import dirname, join
+
+import colorama
+from colorama import Fore, Style
+from markdown import Markdown
+
+if __name__ == "__main__":
+    colorama.init()
+
+    api_path = join(dirname(__file__), "..", "..", "cores/common/base/api/lt_*.*")
+    out_path = join(dirname(__file__), "..", "contrib")
+
+    declaration = ""
+    implementation = ""
+
+    for file in glob(api_path):
+        with open(file, "r") as f:
+            data = f.read()
+        if file.endswith(".h"):
+            declaration += data
+        elif file.endswith(".c"):
+            implementation += data
+
+    block_comment_regex = r"\/\*[\d\D]+?\*\/"
+    line_comment_regex = r"\/\/.+?$"
+    macro_regex = r"#(?:[^\n\\]|\\\n)+$"
+    line_regex = r"\n+"
+    declaration = re.sub(block_comment_regex, "", declaration)
+    declaration = re.sub(line_comment_regex, "", declaration, flags=re.MULTILINE)
+    declaration = re.sub(macro_regex, "", declaration, flags=re.MULTILINE)
+    declaration = re.sub(line_regex, "\n", declaration)
+    implementation = re.sub(block_comment_regex, "", implementation)
+    implementation = re.sub(line_comment_regex, "", implementation, flags=re.MULTILINE)
+    implementation = re.sub(macro_regex, "", implementation, flags=re.MULTILINE)
+    implementation = re.sub(line_regex, "\n", implementation)
+
+    declaration_regex = r"^([^\s][\d\w\s]+ \*?)([\S]+?)\(.*?\);$"
+    implementation_regex = r"^(__attribute__\(\(weak\)\) )?([\w\d* ]+?)([\w\d]+)\(.+?{"
+
+    function_types = {}
+    decl_functions = set()
+    impl_functions = set()
+    weak_functions = set()
+
+    for match in re.finditer(
+        pattern=declaration_regex,
+        string=declaration,
+        flags=re.DOTALL | re.MULTILINE,
+    ):
+        function_type = match[1].strip()
+        function_name = match[2].strip()
+
+        if function_types.get(function_name, function_type) != function_type:
+            print(
+                Fore.YELLOW
+                + "WARNING: Wrong return type: "
+                + f"'{function_types[function_name]} {function_name}'"
+                + f"vs '{function_type} {function_name}'"
+                + Style.RESET_ALL
+            )
+
+        function_types[function_name] = function_type
+        decl_functions.add(function_name)
+
+    for match in re.finditer(
+        pattern=implementation_regex,
+        string=implementation,
+        flags=re.DOTALL | re.MULTILINE,
+    ):
+        is_weak = match[1]
+        function_type = match[2].strip()
+        function_name = match[3].strip()
+        function_types[function_name] = function_type
+
+        if function_types.get(function_name, function_type) != function_type:
+            print(
+                Fore.YELLOW
+                + "WARNING: Wrong return type: "
+                + f"'{function_types[function_name]} {function_name}'"
+                + f"vs '{function_type} {function_name}'"
+                + Style.RESET_ALL
+            )
+
+        function_types[function_name] = function_type
+        if is_weak:
+            weak_functions.add(function_name)
+        else:
+            impl_functions.add(function_name)
+
+for function in list(impl_functions):
+    if "static" in function_types[function]:
+        impl_functions.remove(function)
+
+common_functions = impl_functions.union(weak_functions)
+family_functions = decl_functions - common_functions
+undecl_functions = common_functions - decl_functions
+if undecl_functions:
+    print(Fore.RED + "ERROR: Undeclared functions: " + ", ".join(undecl_functions))
+    exit(1)
+
+md = Markdown(out_path, "lt-api-functions")
+header = [
+    "Type",
+    "Function",
+    "Common",
+    "Weak",
+    "Family",
+]
+rows = []
+
+for function in (
+    sorted(family_functions) + sorted(weak_functions) + sorted(impl_functions)
+):
+    rows.append(
+        [
+            f"`{function_types[function]}`",
+            f"{function}()",
+            "✔️" if function in impl_functions else "",
+            "✔️" if function in weak_functions else "",
+            "✔️" if function not in common_functions else "",
+        ]
+    )
+
+md.add_table(header, *rows)
+md.write()