[release] v1.4.0

* [platform] Install ltchiptool in separate virtual environment

* [platform] Fix f-string syntax, set LibreTiny path in ltchiptool

* [platform] Fix venv site-packages path

* [platform] Fix installing pip without ensurepip

* [platform] Install binary dependencies only

.github/workflows/lint.yml

@@ -1,35 +0,0 @@
-name: Lint check
-
-on: [push, pull_request]
-
-jobs:
-  lint-clang-format:
-    name: Lint with clang-format
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-      - name: Check code with clang-format
-        uses: jidicula/clang-format-action@v4.5.0
-        with:
-          clang-format-version: "14"
-  lint-black:
-    name: Lint with black
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-      - name: Set up Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: "3.9"
-      - name: Install test dependencies
-        uses: BSFishy/pip-action@v1
-        with:
-          packages: |
-            black
-            isort
-      - name: Check code with black
-        run: black --check .
-      - name: Check code with isort
-        run: isort --profile black . --check-only

.github/workflows/platformio-publish.yml

@@ -1,46 +0,0 @@
-name: PlatformIO Publish
-
-on:
-  push:
-    tags:
-      - v*.*.*
-
-jobs:
-  publish:
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v2
-      - name: Cache pip
-        uses: actions/cache@v2
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-
-      - name: Cache PlatformIO
-        uses: actions/cache@v2
-        with:
-          path: ~/.platformio
-          key: ${{ runner.os }}-${{ hashFiles('**/lockfiles') }}
-      - name: Set up Python
-        uses: actions/setup-python@v2
-      - name: Install PlatformIO
-        run: |
-          python -m pip install --upgrade pip
-          pip install --upgrade platformio
-      - name: Publish PlatformIO package
-        run: pio package publish --non-interactive
-        env:
-          CI: true
-          PLATFORMIO_AUTH_TOKEN: ${{ secrets.PLATFORMIO_AUTH_TOKEN }}
-      - name: Get latest version
-        id: get_version
-        run: echo ::set-output name=VERSION::${GITHUB_REF/refs\/tags\//}
-      - name: Release on GitHub
-        uses: softprops/action-gh-release@v1
-        with:
-          name: ${{ steps.get_version.outputs.VERSION }}
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/push-dev.yml

@@ -0,0 +1,12 @@
+name: Push (dev), Pull Request
+on:
+  push:
+    branches: ["**"]
+  pull_request:
+jobs:
+  lint-clang:
+    name: Run Clang lint
+    uses: kuba2k2/kuba2k2/.github/workflows/lint-clang.yml@master
+  lint-python:
+    name: Run Python lint
+    uses: kuba2k2/kuba2k2/.github/workflows/lint-python.yml@master

.github/workflows/push-master.yml

@@ -1,10 +1,8 @@
-name: Deploy docs on GitHub Pages
-
+name: Push (master)
 on:
   push:
-    branches:
-      - master
-
+    branches: ["master"]
+  workflow_dispatch:
 jobs:
   docs:
     name: Deploy docs

.github/workflows/release.yml

@@ -0,0 +1,22 @@
+name: Release
+on:
+  push:
+    tags: ["v*.*.*"]
+jobs:
+  lint-clang:
+    name: Run Clang lint
+    uses: kuba2k2/kuba2k2/.github/workflows/lint-clang.yml@master
+  publish-pio-platform:
+    name: Publish PlatformIO platform
+    needs:
+      - lint-clang
+    uses: kuba2k2/kuba2k2/.github/workflows/publish-pio-platform.yml@master
+    secrets:
+      PLATFORMIO_AUTH_TOKEN: ${{ secrets.PLATFORMIO_AUTH_TOKEN }}
+  gh-release:
+    name: Publish GitHub release
+    needs:
+      - publish-pio-platform
+    uses: kuba2k2/kuba2k2/.github/workflows/gh-release.yml@master
+    permissions:
+      contents: write

README.md

@@ -1,9 +1,11 @@
 # LibreTiny
 
+<small>(formerly LibreTuya)</small>
+
 <div align="center" markdown>
 
-[![GitHub Workflow Status](https://img.shields.io/github/workflow/status/kuba2k2/libretiny/Deploy%20docs%20on%20GitHub%20Pages?label=docs&logo=markdown)](https://kuba2k2.github.io/libretiny/)
-![GitHub last commit](https://img.shields.io/github/last-commit/kuba2k2/libretiny?logo=github)
+[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/libretiny-eu/libretiny/push-master.yml?label=docs&logo=markdown)](https://docs.libretiny.eu/)
+![GitHub last commit](https://img.shields.io/github/last-commit/libretiny-eu/libretiny?logo=github)
 
 [![Code style: clang-format](https://img.shields.io/badge/code%20style-clang--format-purple.svg)](.clang-format)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
@@ -16,13 +18,7 @@
 
 </div>
 
-## LibreTuya is now LibreTiny! 🎉
-
-We have [renamed the project](https://github.com/kuba2k2/libretiny/issues/92) to LibreTiny, also marking the very first v1.0.0 release, along with a huge structure refactor. While some care has been taken to ensure that things don't break, you may still need to update some references in your code to use the new name.
-
----
-
-PlatformIO development platform for IoT modules manufactured by Tuya Inc.
+PlatformIO development platform for BK7231 and RTL8710 IoT chips.
 
 The main goal of this project is to provide a usable build environment for IoT developers. While also providing vendor SDKs as PlatformIO cores,
 the project focuses on developing working Arduino-compatible cores for supported families. The cores are inspired by Espressif's official core for ESP32,
@@ -32,62 +28,11 @@ which should make it easier to port/run existing ESP apps on less-common, unsupp
 
 **Note:** this project is work-in-progress.
 
-## Usage
+<div align="center" markdown>
 
-1. [Install PlatformIO](https://platformio.org/platformio-ide)
-2. `platformio platform install -f https://github.com/kuba2k2/libretiny`
-3. Create a project, build it and upload!
-4. See the [docs](https://docs.libretiny.eu/) for any questions/problems.
+## [⭐ Getting started ⭐](https://docs.libretiny.eu/docs/getting-started/)
 
-<!--
-## Arduino Core support status
-
-Note: this list will probably change with each functionality update.
-
-&nbsp;              | `realtek-ambz` | `beken-72xx`
---------------------|----------------|-------------
-Core functions      | ✔️             | ✔️
-GPIO/PWM/IRQ        | ✔️/✔️/✔️       | ✔️/✔️/✔️
-Analog input (ADC)  | ✔️             | ✔️
-Serial              | ✔️             | ✔️
-Serial (extra)      | 0, 1, 2        | 1, 2
-Flash I/O           | ✔️             | ✔️
-**CORE LIBRARIES**  |                |
-SoftwareSerial      | ✔️             | ❌
-SPI                 | ❌              | ❌
-Wire                | ❗              | ❌
-**OTHER LIBRARIES** |                |
-Wi-Fi STA/AP/Mixed  | ✔️             | ✔️
-Wi-Fi Events        | ✔️             | ✔️
-TCP Client (SSL)    | ✔️ (✔️)        | ✔️ (❗)
-TCP Server          | ✔️             | ✔️
-IPv6                | ❌              | ❌
-HTTP Client (SSL)   | ✔️ (✔️)        | ❓
-HTTP Server         | ✔️             | ✔️
-NVS / Preferences   | ✔️              | ✔️
-SPIFFS              | ❌              | ❌
-BLE                 | -              | ❌
-NTP                 | ✔️              | ✔️
-OTA                 | ✔️             | ✔️
-MDNS                | ✔️             | ✔️
-MQTT                | ✅              | ❌
-SD                  | ❌              | ❌
-
-Symbols:
-
-- ✔️ working
-- ✅ tested, external library
-- ❓ untested
-- ❗ broken
-- ❌ not implemented (yet?)
-- \- not applicable
-
-Names:
-
-- Core functions - stuff like delay(), millis(), yield(), etc.
-- **CORE LIBRARIES** - included normally in all Arduino cores
-- **OTHER LIBRARIES** - included in ESP32 core or downloadable
--->
+</div>
 
 ## License
 

SUMMARY.md

@@ -1,14 +1,15 @@
 * [Home](README.md)
-* [](SUMMARY.md)
 * [😊 Getting started](docs/getting-started/README.md)
 	* [➡️ Info on accessing GPIOs](docs/getting-started/gpio.md)
+* [](SUMMARY.md)
 * [📺 Cloudcutter & ESPHome video guide](https://www.youtube.com/watch?v=sSj8f-HCHQ0)
 * [💡 ESPHome setup guide](docs/projects/esphome.md)
 * [🛖 ESPHome Hassio Add-On](https://github.com/libretiny-eu/esphome-hass-addon/pkgs/container/libretiny-esphome-hassio)
 * [](SUMMARY.md)
 * [📲 Flashing/dumping guide](docs/flashing/)
 * [🔌 How to flash/enter download mode?](docs/platform/)
-* [💻 Supported chips](docs/status/supported.md)
+* [](SUMMARY.md)
+* [💻 Chips, boards, features](docs/status/supported.md)
 	* [All boards](boards/)
 * [](SUMMARY.md)
 * 🍪 Chip family docs & info

boards/generic-rtl8710bx-4mb-980k.json

@@ -18,7 +18,7 @@
 		"extra": [
 			"## Information",
 			"This is a generic board definition for RTL8710BX with 4 MiB of flash. It has a bigger application partition size (980 KiB). The used bootloader is also different from the standard Tuya one.",
-			"It can be found in [Ezviz T31 smart plug](https://www.ezviz.com/product/T31/2021) - bare chip soldered onto the manufacturer-made PCB. The plug is not Tuya/SmartLife-compatible and has a 25Q32CSIG flash chip. Refer to [libretiny#23](https://github.com/kuba2k2/libretiny/issues/23) for photos and more information.",
+			"It can be found in [Ezviz T31 smart plug](https://www.ezviz.com/product/T31/2021) - bare chip soldered onto the manufacturer-made PCB. The plug is not Tuya/SmartLife-compatible and has a 25Q32CSIG flash chip. Refer to [libretiny#23](https://github.com/libretiny-eu/libretiny/issues/23) for photos and more information.",
 			"Note that stock firmware seems to use smaller app images (0x80000 / 512 KiB). After 0x180000 some product-test data and device logs can be found. Because the OTA2 offset is 0x100000, the board definition was configured to use all available space."
 		]
 	},

builder/main.py

@@ -17,6 +17,13 @@ env: Environment = DefaultEnvironment()
 platform: PlatformBase = env.PioPlatform()
 board: PlatformBoardConfig = env.BoardConfig()
 
+python_deps = {
+    "ltchiptool": ">=4.5.1,<5.0",
+}
+env.SConscript("python-venv.py", exports="env")
+env.ConfigurePythonVenv()
+env.InstallPythonDependencies(python_deps)
+
 # Utilities
 env.SConscript("utils/config.py", exports="env")
 env.SConscript("utils/cores.py", exports="env")
@@ -24,7 +31,7 @@ env.SConscript("utils/env.py", exports="env")
 env.SConscript("utils/flash.py", exports="env")
 env.SConscript("utils/libs-external.py", exports="env")
 env.SConscript("utils/libs-queue.py", exports="env")
-env.SConscript("utils/ltchiptool.py", exports="env")
+env.SConscript("utils/ltchiptool-util.py", exports="env")
 
 # Firmware name
 if env.get("PROGNAME", "program") == "program":

builder/python-venv.py

@@ -0,0 +1,122 @@
+# Copyright (c) Kuba Szczodrzyński 2023-09-07.
+
+import json
+import site
+import subprocess
+import sys
+from pathlib import Path
+
+import semantic_version
+from platformio.compat import IS_WINDOWS
+from platformio.package.version import pepver_to_semver
+from platformio.platform.base import PlatformBase
+from SCons.Script import DefaultEnvironment, Environment
+
+env: Environment = DefaultEnvironment()
+platform: PlatformBase = env.PioPlatform()
+
+# code borrowed and modified from espressif32/builder/frameworks/espidf.py
+
+
+def env_configure_python_venv(env: Environment):
+    venv_path = Path(env.subst("${PROJECT_CORE_DIR}"), "penv", ".libretiny")
+
+    pip_path = venv_path.joinpath(
+        "Scripts" if IS_WINDOWS else "bin",
+        "pip" + (".exe" if IS_WINDOWS else ""),
+    )
+    python_path = venv_path.joinpath(
+        "Scripts" if IS_WINDOWS else "bin",
+        "python" + (".exe" if IS_WINDOWS else ""),
+    )
+    site_path = venv_path.joinpath(
+        "Lib" if IS_WINDOWS else "lib",
+        "." if IS_WINDOWS else f"python{sys.version_info[0]}.{sys.version_info[1]}",
+        "site-packages",
+    )
+
+    if not pip_path.is_file():
+        # Use the built-in PlatformIO Python to create a standalone virtual env
+        result = env.Execute(
+            env.VerboseAction(
+                f'"$PYTHONEXE" -m venv --clear "{venv_path.absolute()}"',
+                "LibreTiny: Creating a virtual environment for Python dependencies",
+            )
+        )
+        if not python_path.is_file():
+            # Creating the venv failed
+            raise RuntimeError(
+                f"Failed to create virtual environment. Error code {result}"
+            )
+        if not pip_path.is_file():
+            # Creating the venv succeeded but pip didn't get installed
+            # (i.e. Debian/Ubuntu without ensurepip)
+            print(
+                "LibreTiny: Failed to install pip, running get-pip.py", file=sys.stderr
+            )
+            import requests
+
+            with requests.get("https://bootstrap.pypa.io/get-pip.py") as r:
+                p = subprocess.Popen(
+                    args=str(python_path.absolute()),
+                    stdin=subprocess.PIPE,
+                )
+                p.communicate(r.content)
+                p.wait()
+
+    assert (
+        pip_path.is_file()
+    ), f"Error: Missing the pip binary in virtual environment `{pip_path.absolute()}`"
+    assert (
+        python_path.is_file()
+    ), f"Error: Missing Python executable file `{python_path.absolute()}`"
+    assert (
+        site_path.is_dir()
+    ), f"Error: Missing site-packages directory `{site_path.absolute()}`"
+
+    env.Replace(LTPYTHONEXE=python_path.absolute(), LTPYTHONENV=venv_path.absolute())
+    site.addsitedir(str(site_path.absolute()))
+
+
+def env_install_python_dependencies(env: Environment, dependencies: dict):
+    try:
+        pip_output = subprocess.check_output(
+            [
+                env.subst("${LTPYTHONEXE}"),
+                "-m",
+                "pip",
+                "list",
+                "--format=json",
+                "--disable-pip-version-check",
+            ]
+        )
+        pip_data = json.loads(pip_output)
+        packages = {p["name"]: pepver_to_semver(p["version"]) for p in pip_data}
+    except:
+        print(
+            "LibreTiny: Warning! Couldn't extract the list of installed Python packages"
+        )
+        packages = {}
+
+    to_install = []
+    for name, spec in dependencies.items():
+        install_spec = f'"{name}{dependencies[name]}"'
+        if name not in packages:
+            to_install.append(install_spec)
+        elif spec:
+            version_spec = semantic_version.Spec(spec)
+            if not version_spec.match(packages[name]):
+                to_install.append(install_spec)
+
+    if to_install:
+        env.Execute(
+            env.VerboseAction(
+                '"${LTPYTHONEXE}" -m pip install --prefer-binary -U '
+                + " ".join(to_install),
+                "LibreTiny: Installing Python dependencies",
+            )
+        )
+
+
+env.AddMethod(env_configure_python_venv, "ConfigurePythonVenv")
+env.AddMethod(env_install_python_dependencies, "InstallPythonDependencies")

builder/utils/env.py

@@ -8,6 +8,7 @@ from subprocess import PIPE, Popen
 from typing import Dict
 
 from ltchiptool import Family, get_version
+from ltchiptool.util.lvm import LVM
 from ltchiptool.util.misc import sizeof
 from platformio.platform.base import PlatformBase
 from platformio.platform.board import PlatformBoardConfig
@@ -77,7 +78,7 @@ def env_configure(
         # ltchiptool config:
         # -r    output raw log messages
         # -i 1  indent log messages
-        LTCHIPTOOL='"${PYTHONEXE}" -m ltchiptool -r -i 1',
+        LTCHIPTOOL='"${LTPYTHONEXE}" -m ltchiptool -r -i 1 -L "${LT_DIR}"',
         # Fix for link2bin to get tmpfile name in argv
         LINKCOM="${LINK} ${LINKARGS}",
         LINKARGS="${TEMPFILE('-o $TARGET $LINKFLAGS $__RPATH $SOURCES $_LIBDIRFLAGS $_LIBFLAGS', '$LINKCOMSTR')}",
@@ -87,6 +88,8 @@ def env_configure(
     )
     # Store family parameters as environment variables
     env.Replace(**dict(family))
+    # Set platform directory in ltchiptool (for use in this process only)
+    LVM.add_path(platform.get_dir())
     return family
 
 

cores/beken-72xx/arduino/libraries/Serial/Serial.cpp

@@ -47,6 +47,10 @@ void SerialClass::configure(unsigned long baudrate, uint16_t config) {
 		.flow_control = FLOW_CTRL_DISABLED,
 	};
 
+	if (port == 1)
+		uart1_init();
+	else if (port == 2)
+		uart2_init();
 	uart_hw_set_change(port, &cfg);
 	uart_rx_callback_set(port, callback, &BUF);
 

cores/beken-72xx/base/api/lt_init.c

@@ -6,4 +6,9 @@
 void lt_init_family() {
 	// set default UART output port
 	uart_print_port = LT_UART_DEFAULT_PORT - 1;
+	// initialize the UART (needed e.g. after deep sleep)
+	if (uart_print_port == 1)
+		uart1_init();
+	else if (uart_print_port == 2)
+		uart2_init();
 }

cores/beken-72xx/base/api/lt_sleep.c

@@ -9,9 +9,9 @@ void lt_deep_sleep_config_gpio(uint32_t gpio_index_map, bool on_high) {
 	deep_sleep_param.wake_up_way |= PS_DEEP_WAKEUP_GPIO;
 	deep_sleep_param.gpio_index_map |= gpio_index_map;
 	if (on_high) {
-		deep_sleep_param.gpio_edge_map &= (~gpio_index_map);
-	} else {
 		deep_sleep_param.gpio_edge_map |= gpio_index_map;
+	} else {
+		deep_sleep_param.gpio_edge_map &= (~gpio_index_map);
 	}
 }
 

cores/common/arduino/libraries/common/MD5/MD5PolarSSLImpl.h

@@ -1,19 +0,0 @@
-/* Copyright (c) Kuba Szczodrzyński 2022-06-03. */
-
-#pragma once
-
-#ifdef __cplusplus
-extern "C" {
-#endif
-
-typedef struct {
-	unsigned long total[2];	  /*!< number of bytes processed  */
-	unsigned long state[4];	  /*!< intermediate digest state  */
-	unsigned char buffer[64]; /*!< data block being processed */
-} md5_context;
-
-#define LT_MD5_CTX_T md5_context
-
-#ifdef __cplusplus
-} // extern "C"
-#endif

cores/common/arduino/libraries/common/Update/Update.cpp

@@ -61,6 +61,10 @@ bool UpdateClass::begin(
 	lt_ota_begin(this->ctx, size);
 	this->ctx->callback		  = reinterpret_cast<void (*)(void *)>(progressHandler);
 	this->ctx->callback_param = this;
+
+	this->md5Ctx = static_cast<LT_MD5_CTX_T *>(malloc(sizeof(LT_MD5_CTX_T)));
+	MD5Init(this->md5Ctx);
+
 	return true;
 }
 
@@ -79,6 +83,10 @@ bool UpdateClass::end(bool evenIfRemaining) {
 		// abort if not finished
 		this->errArd = UPDATE_ERROR_ABORT;
 
+	if (!this->md5Digest)
+		this->md5Digest = static_cast<uint8_t *>(malloc(16));
+	MD5Final(this->md5Digest, this->md5Ctx);
+
 	this->cleanup(/* clearError= */ evenIfRemaining);
 	return !this->hasError();
 }
@@ -97,6 +105,10 @@ void UpdateClass::cleanup(bool clearError) {
 		// activating firmware failed
 		this->errArd = UPDATE_ERROR_ACTIVATE;
 		this->errUf2 = UF2_ERR_OK;
+	} else if (this->md5Digest && this->md5Expected && memcmp(this->md5Digest, this->md5Expected, 16) != 0) {
+		// MD5 doesn't match
+		this->errArd = UPDATE_ERROR_MD5;
+		this->errUf2 = UF2_ERR_OK;
 	} else if (clearError) {
 		// successful finish and activation, clear error codes
 		this->clearError();
@@ -116,6 +128,12 @@ void UpdateClass::cleanup(bool clearError) {
 
 	free(this->ctx);
 	this->ctx = nullptr;
+	free(this->md5Ctx);
+	this->md5Ctx = nullptr;
+	free(this->md5Digest);
+	this->md5Digest = nullptr;
+	free(this->md5Expected);
+	this->md5Expected = nullptr;
 }
 
 /**
@@ -132,6 +150,7 @@ size_t UpdateClass::write(const uint8_t *data, size_t len) {
 		return 0;
 
 	size_t written = lt_ota_write(ctx, data, len);
+	MD5Update(this->md5Ctx, data, len);
 	if (written != len)
 		this->cleanup(/* clearError= */ false);
 	return written;
@@ -171,6 +190,8 @@ size_t UpdateClass::writeStream(Stream &data) {
 		// read data to fit in the remaining buffer space
 		auto bufSize = this->ctx->buf_pos - this->ctx->buf;
 		auto read	 = data.readBytes(this->ctx->buf_pos, UF2_BLOCK_SIZE - bufSize);
+		// update MD5
+		MD5Update(this->md5Ctx, this->ctx->buf_pos, read);
 		// increment buffer writing head
 		this->ctx->buf_pos += read;
 		// process the block if complete

cores/common/arduino/libraries/common/Update/Update.h

@@ -1,6 +1,7 @@
 #pragma once
 
 #include <Arduino.h>
+#include <MD5.h>
 #include <functional>
 #include <uf2ota/uf2ota.h>
 
@@ -56,6 +57,9 @@ class UpdateClass {
 	UpdateClass &onProgress(THandlerFunction_Progress handler);
 	static bool canRollBack();
 	static bool rollBack();
+	bool setMD5(const char *md5);
+	String md5String();
+	void md5(uint8_t *result);
 	uint16_t getErrorCode() const;
 	bool hasError() const;
 	void clearError();
@@ -71,6 +75,9 @@ class UpdateClass {
 	uf2_err_t errUf2{UF2_ERR_OK};
 	UpdateError errArd{UPDATE_ERROR_OK};
 	THandlerFunction_Progress callback{nullptr};
+	LT_MD5_CTX_T *md5Ctx{nullptr};
+	uint8_t *md5Digest{nullptr};
+	uint8_t *md5Expected{nullptr};
 
   public:
 	/**

cores/common/arduino/libraries/common/Update/UpdateUtil.cpp

@@ -71,6 +71,42 @@ bool UpdateClass::rollBack() {
 	return lt_ota_switch(/* revert= */ false);
 }
 
+/**
+ * @brief Set the expected MD5 of the firmware (hexadecimal string).
+ */
+bool UpdateClass::setMD5(const char *md5) {
+	if (strlen(md5) != 32)
+		return false;
+	if (!this->md5Expected)
+		this->md5Expected = static_cast<uint8_t *>(malloc(16));
+	if (!this->md5Expected)
+		return false;
+	lt_xtob(md5, 32, this->md5Expected);
+	return true;
+}
+
+/**
+ * @brief Return a hexadecimal string of calculated firmware MD5 sum.
+ */
+String UpdateClass::md5String() {
+	if (!this->md5Digest)
+		return "";
+	char out[32 + 1];
+	lt_btox(this->md5Digest, 16, out);
+	return String(out);
+}
+
+/**
+ * @brief Get calculated MD5 digest of the firmware.
+ */
+void UpdateClass::md5(uint8_t *result) {
+	if (!this->md5Digest) {
+		memset(result, '\0', 16);
+		return;
+	}
+	memcpy(result, this->md5Digest, 16);
+}
+
 /**
  * @brief Get combined error code of the update.
  */

cores/common/base/api/lt_sleep.h

@@ -21,7 +21,7 @@ void lt_deep_sleep_unset_gpio(uint32_t gpio_index_map);
 
 /**
  * @brief Set a sleep timer to wake up the device
- * @param sleep_duration the time in seconds to sleep
+ * @param sleep_duration the time in milliseconds to sleep
  */
 void lt_deep_sleep_config_timer(uint32_t sleep_duration);
 

cores/common/base/api/lt_utils.c

@@ -39,3 +39,36 @@ void hexdump(const uint8_t *buf, size_t len, uint32_t offset, uint8_t width) {
 		pos += lineWidth;
 	}
 }
+
+char *lt_btox(const uint8_t *src, int len, char *dest) {
+	// https://stackoverflow.com/a/53966346
+	const char hex[] = "0123456789abcdef";
+	len *= 2;
+	dest[len] = '\0';
+	while (--len >= 0)
+		dest[len] = hex[(src[len >> 1] >> ((1 - (len & 1)) << 2)) & 0xF];
+	return dest;
+}
+
+uint8_t *lt_xtob(const char *src, int len, uint8_t *dest) {
+	// https://gist.github.com/vi/dd3b5569af8a26b97c8e20ae06e804cb
+
+	// mapping of ASCII characters to hex values
+	// (16-byte swapped to reduce XOR 0x10 operation)
+	const uint8_t mapping[] = {
+		0x00, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x00, // @ABCDEFG
+		0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // HIJKLMNO
+		0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, // 01234567
+		0x08, 0x09, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // 89:;<=>?
+	};
+
+	int j = 0;
+	uint8_t idx0;
+	uint8_t idx1;
+	for (int i = 0; i < len; i += 2) {
+		idx0	  = ((uint8_t)src[i + 0] & 0x1F);
+		idx1	  = ((uint8_t)src[i + 1] & 0x1F);
+		dest[j++] = (mapping[idx0] << 4) | (mapping[idx1] << 0);
+	}
+	return dest;
+}

cores/common/base/api/lt_utils.h

@@ -45,3 +45,23 @@ void hexdump(
 	uint8_t width
 #endif
 );
+
+/**
+ * @brief Convert a byte array to hexadecimal string.
+ *
+ * @param src source byte array
+ * @param len source length (bytes)
+ * @param dest destination string
+ * @return destination string
+ */
+char *lt_btox(const uint8_t *src, int len, char *dest);
+
+/**
+ * @brief Convert a hexadecimal string to byte array.
+ *
+ * @param src source string
+ * @param len source length (chars)
+ * @param dest destination byte array
+ * @return destination byte array
+ */
+uint8_t *lt_xtob(const char *src, int len, uint8_t *dest);

docs/getting-started/README.md

@@ -3,7 +3,7 @@
 Using LibreTiny is simple, just like every other PlatformIO development platform.
 
 1. [Install PlatformIO](https://platformio.org/platformio-ide)
-2. `platformio platform install -f https://github.com/kuba2k2/libretiny`
+2. `platformio platform install -f https://github.com/libretiny-eu/libretiny`
 
 !!! tip
 	See the [Cloudcutter video guide](https://www.youtube.com/watch?v=sSj8f-HCHQ0) for a complete tutorial on flashing with [Cloudcutter](https://github.com/tuya-cloudcutter/tuya-cloudcutter) and installing [LibreTiny-ESPHome](../projects/esphome.md). **Includes Home Assistant Add-On setup.**

docs/projects/esphome.md

@@ -56,7 +56,7 @@ If your board isn't listed, use one of the **Generic** boards, depending on the
 	Assuming you have PlatformIO, git and Python installed:
 
 	1. Open a terminal/cmd.exe, create `esphome` directory and `cd` into it.
-	2. `git clone https://github.com/kuba2k2/libretiny-esphome`
+	2. `git clone https://github.com/libretiny-eu/libretiny-esphome`
 	3. `cd` into the newly created `libretiny-esphome` directory.
 	4. Check if it works by typing `python -m esphome`
 

docs/requirements.txt

@@ -4,4 +4,5 @@ mkdocs-literate-nav==0.5.0
 mkdocs-section-index
 mkdocs-include-markdown-plugin
 mkdocs-git-revision-date-localized-plugin
--e git+https://github.com/kuba2k2/mkdoxy#egg=mkdoxy
+-e git+https://github.com/libretiny-eu/mkdoxy#egg=mkdoxy
+mkdocs-redirects

docs/status/arduino.md

@@ -1,7 +0,0 @@
-# Implementation status
-
-{%
-	include-markdown "../../README.md"
-   start="\n## Arduino Core support status\n"
-   end="\n## License\n"
-%}

docs/status/supported.md

@@ -26,12 +26,49 @@ A list of chip families currently supported by this project.
 !!! note
 	The term *family* was chosen over *platform*, in order to reduce possible confusion between LibreTiny supported "platforms" and PlatformIO's "platform", as an entire package. *Family* is also more compatible with the UF2 term.
 
-The following list corresponds to UF2 OTA format family names, and is also [available as JSON](../../families.json). The IDs are also present in [lt_types.h](../../ltapi/lt__types_8h.md).
+The following list corresponds to UF2 OTA format family names, and is also [available as JSON](../../families.json). The IDs are also present in [lt_types.h](../../ltapi/lt__types_8h.md). You can view the family list by using `ltchiptool list families`.
 
 {%
 	include-markdown "./supported_families.md"
 %}
 
+## Feature support
+
+If you notice a feature that you've tested, which works (or not) and doesn't match this table, feel free to submit an issue on GitHub.
+
+                    | `BK7231T` | `BK7231N` | `RTL8710B` | `RTL8720C` | `BK7231Q`
+-------------------------|-----------|-----------|------------|------------|----------
+Stability                | 5/5       | 5/5       | 4/5        | 2/5        | 1/5
+LibreTiny Core           | ✔️        | ✔️        | ✔️         | ✔️         | ✔️
+Wiring Core              | ✔️        | ✔️        | ✔️         | ✔️         | ✔️
+**PERIPHERALS** (Core)   |           |           |            |            |
+UART I/O                 | ✔️        | ✔️        | ✔️         | ✔️         | ✔️
+Flash I/O                | ✔️        | ✔️        | ✔️         | ❓          | ❓
+Deep sleep               | ❓         | ✔️        | ❌          | ❌          | ❓
+Watchdog timer           | ✔️        | ✔️        | ✔️         | ❓          | ❓
+**PERIPHERALS** (Wiring) |           |           |            |            |
+Digital I/O              | ✔️        | ✔️        | ✔️         | ❓          | ❓
+PWM                      | ✔️        | ✔️        | ✔️         | ❓          | ❓
+Interrupts               | ✔️        | ✔️        | ✔️         | ❓          | ❓
+Analog input (ADC)       | ✔️        | ✔️        | ✔️         | ❓          | ❓
+`Wire` (I²C)             | ❌         | ❌         | ❗          | ❌          | ❌
+`SPI`                    | ❌         | ❌         | ❌          | ❌          | ❌
+`Serial`                 | ✔️        | ✔️        | ✔️         | ✔️         | ❓
+`SoftwareSerial`         | ❌         | ❌         | ✔️         | ❌          | ❌
+**NETWORKING**           |           |           |            |            |
+Wi-Fi STA/AP/Mixed       | ✔️        | ✔️        | ✔️         | ❓          | ❌
+Wi-Fi Events             | ✔️        | ✔️        | ✔️         | ❓          | ❌
+OTA updates              | ✔️        | ✔️        | ✔️         | ❌          | ❌
+MDNS                     | ✔️        | ✔️        | ✔️         | ❓          | ❓
+
+Symbols:
+
+- ✔️ working
+- ❓ untested
+- ❗ broken
+- ❌ not implemented (yet?)
+- \- not applicable
+
 ## Unsupported boards
 
 ### Tuya Inc.

mkdocs.yml

@@ -2,7 +2,7 @@ site_name: LibreTiny
 docs_dir: .
 
 site_url: https://docs.libretiny.eu/
-repo_url: https://github.com/kuba2k2/libretiny
+repo_url: https://github.com/libretiny-eu/libretiny
 
 theme:
   name: material
@@ -32,6 +32,18 @@ plugins:
       save-api: .
   - literate-nav:
       nav_file: SUMMARY.md
+  - redirects:
+      redirect_maps:
+        "link/cloudcutter-video.md": 'https://www.youtube.com/watch?v=sSj8f-HCHQ0'
+        "link/cloudcutter-digiblur.md": 'https://digiblur.com/2023/08/19/updated-tuya-cloudcutter-with-esphome-bk7231-how-to-guide/'
+        "link/boards.md": "docs/status/supported.md"
+        "link/gpio-access.md": "docs/getting-started/gpio.md"
+        "link/config.md": "docs/dev/config.md"
+        "link/config-debug.md": "docs/dev/config.md#per-module-logging-debugging"
+        "link/config-serial.md": "docs/dev/config.md#serial-output"
+        "link/flashing-beken-72xx.md": "docs/platform/beken-72xx/README.md#flashing"
+        "link/flashing-realtek-ambz.md": "docs/platform/realtek-ambz/README.md#flashing"
+        "link/kickstart.md": "https://github.com/libretiny-eu/esphome-kickstart/releases/latest"
   - section-index
   - include-markdown
   - search

platform.json

@@ -4,9 +4,9 @@
   "description": "PlatformIO development platform for IoT modules",
   "repository": {
     "type": "git",
-    "url": "https://github.com/kuba2k2/platformio-libretiny"
+    "url": "https://github.com/libretiny-eu/libretiny.git"
   },
-  "version": "1.2.1",
+  "version": "1.4.0",
   "frameworks": {
     "base": {
       "title": "Base Framework (SDK only)",

platform.py

@@ -1,12 +1,12 @@
 # Copyright (c) Kuba Szczodrzyński 2022-04-20.
 
-import importlib
 import json
 import os
 import platform
+import site
 import sys
 from os.path import dirname
-from subprocess import Popen
+from pathlib import Path
 from typing import Dict, List
 
 import click
@@ -15,73 +15,8 @@ from platformio.debug.exception import DebugInvalidOptionsError
 from platformio.package.meta import PackageItem
 from platformio.platform.base import PlatformBase
 from platformio.platform.board import PlatformBoardConfig
-from semantic_version import SimpleSpec, Version
 
-LTCHIPTOOL_VERSION = "^4.2.3"
-
-
-# Install & import tools
-def check_ltchiptool(install: bool):
-    if install:
-        # update ltchiptool to a supported version
-        print("Installing/updating ltchiptool")
-        p = Popen(
-            [
-                sys.executable,
-                "-m",
-                "pip",
-                "install",
-                "-U",
-                "--force-reinstall",
-                f"ltchiptool >= {LTCHIPTOOL_VERSION[1:]}, < 5.0",
-            ],
-        )
-        p.wait()
-
-        # unload all modules from the old version
-        for name, module in list(sorted(sys.modules.items())):
-            if not name.startswith("ltchiptool"):
-                continue
-            del sys.modules[name]
-            del module
-
-    # try to import it
-    ltchiptool = importlib.import_module("ltchiptool")
-
-    # check if the version is known
-    version = Version.coerce(ltchiptool.get_version()).truncate()
-    if version in SimpleSpec(LTCHIPTOOL_VERSION):
-        return
-    if not install:
-        raise ImportError(f"Version incompatible: {version}")
-
-
-def try_check_ltchiptool():
-    install_modes = [False, True]
-    exception = None
-    for install in install_modes:
-        try:
-            check_ltchiptool(install)
-            return
-        except (ImportError, AttributeError) as ex:
-            exception = ex
-    print(
-        "!!! Installing ltchiptool failed, or version outdated. "
-        "Please install ltchiptool manually using pip. "
-        f"Cannot continue. {type(exception).name}: {exception}"
-    )
-    raise exception
-
-
-try_check_ltchiptool()
-import ltchiptool
-
-# Remove current dir so it doesn't conflict with PIO
-if dirname(__file__) in sys.path:
-    sys.path.remove(dirname(__file__))
-
-# Let ltchiptool know about LT's location
-ltchiptool.lt_set_path(dirname(__file__))
+site.addsitedir(Path(__file__).absolute().parent.joinpath("tools"))
 
 
 def get_os_specifiers():
@@ -119,6 +54,12 @@ class LibretinyPlatform(PlatformBase):
         super().__init__(manifest_path)
         self.custom_opts = {}
         self.versions = {}
+        self.verbose = (
+            "-v" in sys.argv
+            or "--verbose" in sys.argv
+            or "PIOVERBOSE=1" in sys.argv
+            or os.environ.get("PIOVERBOSE", "0") == "1"
+        )
 
     def print(self, *args, **kwargs):
         if not self.verbose:
@@ -137,11 +78,8 @@ class LibretinyPlatform(PlatformBase):
         return spec
 
     def configure_default_packages(self, options: dict, targets: List[str]):
-        from ltchiptool.util.dict import RecursiveDict
+        from libretiny import RecursiveDict
 
-        self.verbose = (
-            "-v" in sys.argv or "--verbose" in sys.argv or "PIOVERBOSE=1" in sys.argv
-        )
         self.print(f"configure_default_packages(targets={targets})")
 
         pioframework = options.get("pioframework") or ["base"]
@@ -298,19 +236,19 @@ class LibretinyPlatform(PlatformBase):
         return result
 
     def update_board(self, board: PlatformBoardConfig):
+        from libretiny import Board, Family, merge_dicts
+
         if "_base" in board:
-            board._manifest = ltchiptool.Board.get_data(board._manifest)
+            board._manifest = Board.get_data(board._manifest)
             board._manifest.pop("_base")
 
         if self.custom("board"):
-            from ltchiptool.util.dict import merge_dicts
-
             with open(self.custom("board"), "r") as f:
                 custom_board = json.load(f)
             board._manifest = merge_dicts(board._manifest, custom_board)
 
         family = board.get("build.family")
-        family = ltchiptool.Family.get(short_name=family)
+        family = Family.get(short_name=family)
         # add "frameworks" key with the default "base"
         board.manifest["frameworks"] = ["base"]
         # add "arduino" framework if supported

tools/libretiny/__init__.py

@@ -0,0 +1,14 @@
+# Copyright (c) Kuba Szczodrzyński 2023-09-07.
+
+from .board import Board
+from .dict import RecursiveDict, merge_dicts
+from .family import Family
+
+# TODO refactor and remove all this from here
+
+__all__ = [
+    "Board",
+    "Family",
+    "RecursiveDict",
+    "merge_dicts",
+]

tools/libretiny/board.py

@@ -0,0 +1,34 @@
+# Copyright (c) Kuba Szczodrzyński 2022-07-29.
+
+from typing import Union
+
+from genericpath import isfile
+
+from .dict import merge_dicts
+from .fileio import readjson
+from .lvm import lvm_load_json
+
+
+class Board:
+    @staticmethod
+    def get_data(board: Union[str, dict]) -> dict:
+        if not isinstance(board, dict):
+            if isfile(board):
+                board = readjson(board)
+                if not board:
+                    raise FileNotFoundError(f"Board not found: {board}")
+            else:
+                source = board
+                board = lvm_load_json(f"boards/{board}.json")
+                board["source"] = source
+        if "_base" in board:
+            base = board["_base"]
+            if not isinstance(base, list):
+                base = [base]
+            result = {}
+            for base_name in base:
+                board_base = lvm_load_json(f"boards/_base/{base_name}.json")
+                merge_dicts(result, board_base)
+            merge_dicts(result, board)
+            board = result
+        return board

tools/libretiny/dict.py

@@ -0,0 +1,65 @@
+# Copyright (c) Kuba Szczodrzyński 2022-07-29.
+
+from .obj import get, has, pop, set_
+
+
+class RecursiveDict(dict):
+    def __init__(self, data: dict = None):
+        if data:
+            data = {
+                key: RecursiveDict(value) if isinstance(value, dict) else value
+                for key, value in data.items()
+            }
+            super().__init__(data)
+        else:
+            super().__init__()
+
+    def __getitem__(self, key):
+        if "." not in key:
+            return super().get(key, None)
+        return get(self, key)
+
+    def __setitem__(self, key, value):
+        if "." not in key:
+            return super().__setitem__(key, value)
+        set_(self, key, value, newtype=RecursiveDict)
+
+    def __delitem__(self, key):
+        if "." not in key:
+            return super().pop(key, None)
+        return pop(self, key)
+
+    def __contains__(self, key) -> bool:
+        if "." not in key:
+            return super().__contains__(key)
+        return has(self, key)
+
+    def get(self, key, default=None):
+        if "." not in key:
+            return super().get(key, default)
+        return get(self, key) or default
+
+    def pop(self, key, default=None):
+        if "." not in key:
+            return super().pop(key, default)
+        return pop(self, key, default)
+
+
+def merge_dicts(d1, d2):
+    # force RecursiveDict instances to be treated as regular dicts
+    d1_type = dict if isinstance(d1, RecursiveDict) else type(d1)
+    d2_type = dict if isinstance(d2, RecursiveDict) else type(d2)
+    if d1 is not None and d1_type != d2_type:
+        raise TypeError(f"d1 and d2 are of different types: {type(d1)} vs {type(d2)}")
+    if isinstance(d2, list):
+        if d1 is None:
+            d1 = []
+        d1.extend(merge_dicts(None, item) for item in d2)
+    elif isinstance(d2, dict):
+        if d1 is None:
+            d1 = {}
+        for key in d2:
+            d1[key] = merge_dicts(d1.get(key, None), d2[key])
+    else:
+        d1 = d2
+    return d1

tools/libretiny/family.py

@@ -0,0 +1,97 @@
+# Copyright (c) Kuba Szczodrzyński 2022-06-02.
+
+from dataclasses import dataclass, field
+from typing import List, Optional, Union
+
+from .lvm import lvm_load_json, lvm_path
+
+LT_FAMILIES: List["Family"] = []
+
+
+@dataclass
+class Family:
+    name: str
+    parent: Union["Family", None]
+    code: str
+    description: str
+    id: Optional[int] = None
+    short_name: Optional[str] = None
+    package: Optional[str] = None
+    mcus: List[str] = field(default_factory=lambda: [])
+    children: List["Family"] = field(default_factory=lambda: [])
+
+    # noinspection PyTypeChecker
+    def __post_init__(self):
+        if self.id:
+            self.id = int(self.id, 16)
+        self.mcus = set(self.mcus)
+
+    @classmethod
+    def get_all(cls) -> List["Family"]:
+        global LT_FAMILIES
+        if LT_FAMILIES:
+            return LT_FAMILIES
+        families = lvm_load_json("families.json")
+        LT_FAMILIES = [
+            cls(name=k, **v) for k, v in families.items() if isinstance(v, dict)
+        ]
+        # attach parents and children to all families
+        for family in LT_FAMILIES:
+            if family.parent is None:
+                continue
+            try:
+                parent = next(f for f in LT_FAMILIES if f.name == family.parent)
+            except StopIteration:
+                raise ValueError(
+                    f"Family parent '{family.parent}' of '{family.name}' doesn't exist"
+                )
+            family.parent = parent
+            parent.children.append(family)
+        return LT_FAMILIES
+
+    @classmethod
+    def get(
+        cls,
+        any: str = None,
+        id: Union[str, int] = None,
+        short_name: str = None,
+        name: str = None,
+        code: str = None,
+        description: str = None,
+    ) -> "Family":
+        if any:
+            id = any
+            short_name = any
+            name = any
+            code = any
+            description = any
+        if id and isinstance(id, str) and id.startswith("0x"):
+            id = int(id, 16)
+        for family in cls.get_all():
+            if id and family.id == id:
+                return family
+            if short_name and family.short_name == short_name.upper():
+                return family
+            if name and family.name == name.lower():
+                return family
+            if code and family.code == code.lower():
+                return family
+            if description and family.description == description:
+                return family
+        if any:
+            raise ValueError(f"Family not found - {any}")
+        items = [hex(id) if id else None, short_name, name, code, description]
+        text = ", ".join(filter(None, items))
+        raise ValueError(f"Family not found - {text}")
+
+    @property
+    def has_arduino_core(self) -> bool:
+        if lvm_path().joinpath("cores", self.name, "arduino").is_dir():
+            return True
+        if self.parent:
+            return self.parent.has_arduino_core
+        return False
+
+    @property
+    def target_package(self) -> Optional[str]:
+        return self.package or self.parent and self.parent.target_package

tools/libretiny/fileio.py

@@ -0,0 +1,17 @@
+# Copyright (c) Kuba Szczodrzyński 2022-06-10.
+
+import json
+from json import JSONDecodeError
+from os.path import isfile
+from typing import Optional, Union
+
+
+def readjson(file: str) -> Optional[Union[dict, list]]:
+    """Read a JSON file into a dict or list."""
+    if not isfile(file):
+        return None
+    with open(file, "r", encoding="utf-8") as f:
+        try:
+            return json.load(f)
+        except JSONDecodeError:
+            return None

tools/libretiny/lvm.py

@@ -0,0 +1,19 @@
+#  Copyright (c) Kuba Szczodrzyński 2023-3-18.
+
+import json
+from pathlib import Path
+from typing import Dict, Union
+
+json_cache: Dict[str, Union[list, dict]] = {}
+libretiny_path = Path(__file__).parents[2]
+
+
+def lvm_load_json(path: str) -> Union[list, dict]:
+    if path not in json_cache:
+        with libretiny_path.joinpath(path).open("rb") as f:
+            json_cache[path] = json.load(f)
+    return json_cache[path]
+
+
+def lvm_path() -> Path:
+    return libretiny_path

tools/libretiny/obj.py

@@ -0,0 +1,62 @@
+# Copyright (c) Kuba Szczodrzyński 2022-06-02.
+
+from enum import Enum
+from typing import Type
+
+# The following helpers force using base dict class' methods.
+# Because RecursiveDict uses these helpers, this prevents it
+# from running into endless nested loops.
+
+
+def get(data: dict, path: str):
+    if not isinstance(data, dict) or not path:
+        return None
+    if dict.__contains__(data, path):
+        return dict.get(data, path, None)
+    key, _, path = path.partition(".")
+    return get(dict.get(data, key, None), path)
+
+
+def pop(data: dict, path: str, default=None):
+    if not isinstance(data, dict) or not path:
+        return default
+    if dict.__contains__(data, path):
+        return dict.pop(data, path, default)
+    key, _, path = path.partition(".")
+    return pop(dict.get(data, key, None), path, default)
+
+
+def has(data: dict, path: str) -> bool:
+    if not isinstance(data, dict) or not path:
+        return False
+    if dict.__contains__(data, path):
+        return True
+    key, _, path = path.partition(".")
+    return has(dict.get(data, key, None), path)
+
+
+def set_(data: dict, path: str, value, newtype=dict):
+    if not isinstance(data, dict) or not path:
+        return
+    # can't use __contains__ here, as we're setting,
+    # so it's obvious 'data' doesn't have the item
+    if "." not in path:
+        dict.__setitem__(data, path, value)
+    else:
+        key, _, path = path.partition(".")
+        # allow creation of parent objects
+        if key in data:
+            sub_data = dict.__getitem__(data, key)
+        else:
+            sub_data = newtype()
+            dict.__setitem__(data, key, sub_data)
+        set_(sub_data, path, value)
+
+
+def str2enum(cls: Type[Enum], key: str):
+    if not key:
+        return None
+    try:
+        return next(e for e in cls if e.name.lower() == key.lower())
+    except StopIteration:
+        return None