From 64fb2323c33e774e05069ae2ce7674954f71ca84 Mon Sep 17 00:00:00 2001 From: Thorsten von Eicken Date: Wed, 12 Oct 2016 20:51:50 -0700 Subject: [PATCH] Update docs and split up the main readme (#200) --- BUILDING.md | 112 ++++++++++ FLASHING.md | 192 ++++++++++++++++ Makefile | 1 + README.adoc | 545 --------------------------------------------- README.md | 163 ++++++++++++++ RESTMQTT.md | 14 ++ TROUBLESHOOTING.md | 36 +++ UC-FLASHING.md | 177 +++++++++++++++ WIFI-CONFIG.md | 87 ++++++++ 9 files changed, 782 insertions(+), 545 deletions(-) create mode 100644 BUILDING.md create mode 100644 FLASHING.md delete mode 100644 README.adoc create mode 100644 README.md create mode 100644 RESTMQTT.md create mode 100644 TROUBLESHOOTING.md create mode 100644 UC-FLASHING.md create mode 100644 WIFI-CONFIG.md diff --git a/BUILDING.md b/BUILDING.md new file mode 100644 index 0000000..dea1875 --- /dev/null +++ b/BUILDING.md @@ -0,0 +1,112 @@ +Building esp-link +================= + +Before you build esp-link, consider that you can download ready-made firmware images! +Just head over to the [release section](https://github.com/jeelabs/esp-link/releases) +and download the tgz archive. + +If you decide to build your own, there are a number of options: +- On linux x86 download the ready-built toolchain and patched SDK like the automated build does + and compile the firmware +- On linux use a docker image with the toolchain and the SDK to compile the firmware +- On linux download and build the toolchain, download and patch the SDK, then compile the firmware +- On windows use a docker image with the toolchain and the SDK to compile the firmware +- On windows install mingw, python, java, and a slew of other tools and then build the + firmware + +Once you have built the firmware you will want to flash it to your esp8266 module. +Assuming you already have esp-link running you can either go back to the initial flashing +via the serial port or you can use the over-the-air (i.e. Wifi) update method, which is faster +and more reliable (unless you have a non-booting version of esp-link). +The OTA flashing is described at the end of this page, +the serial flashing is described in [FLASHING.md](FLASHING.md). + +### Automated builds + +For every commit on github an automated build is made. This means that every branch, including +master, and every pull request always has an up-to-date build. These builds are made by Travis +using the instructions in `.travis.yml`, which basically consist of cloning the esp-link repo, +downloading the compiler toolchain, downloading the Espressif SDK, and running `make`. +If you're looking for how to build esp-link the travis instructions will always give you +accurate pointers to what to download. + +### Docker (linux or windows) + +The [esp-link docker image](https://hub.docker.com/r/jeelabs/esp-link/) contains all the +tools to build esp-link as well as the appropriate Espressif SDK. *It does not contain the +esp-link source code!*. You use the docker image just to build the firmware, you don't have +to do your editing in there. The steps are: +- clone the esp-link github repo +- checkout the branch or tag you want (for example the tag `v2.2.3` for that release) +- cd into the esp-link top directory +- run `make` in docker while mounting your esp-link directory into the container: + - linux: `docker run -v $PWD:/esp-link jeelabs/esp-link:latest` + - windows: `docker run -v c:\somepath\esp-link:/esp-link jeelabs/esp-link:latest`, + where `somepath` is the path to where you cloned esp-link, you probably end up with + something like `-v c:\Users\tve\source\esp-link:/esp-link` +- if you are not building esp-link `master` then read the release notes to see which version of + the Espressif SDK you need and use that as tag for the container image, such as + `jeelabs/esp-link:SDK2.0.0.p1`; you can see the list of available SDKs on + [dockerhub](https://hub.docker.com/r/jeelabs/esp-link/tags/)o + +Sample steps to build esp-link v2.2.3 on a Win7 Pro x64 (these use the docker terminal, there +are multiple way to skin the proverbial cat...): +1) Install Docker Toolbox ( http://www.docker.com/products/docker-toolbox ) +2) Install Git Desktop ( https://desktop.github.com/ ) +3) Clone esp-link from Github master to local repository ( https://github.com/jeelabs/esp-link ) +4) Open Docker Quickstart Terminal +5) cd to local esp-link git repository ( C:\Users\xxxxx\Documents\GitHub\esp-link ) +6) Run "docker run -v $PWD:/esp-link jeelabs/esp-link" command in Docker Quickstart Terminal window + +Note: there has been one report of messed-up timestamps on windows, the symptom is that `make` +complains about file modification times being in the future. This may be due to the different +way Windows and Linux handle time zones and daylight savings time. PLease report if you +encounter this or know a solution. + +### Linux + +The firmware has been built using the https://github.com/pfalcon/esp-open-sdk[esp-open-sdk] +on a Linux system. Create an esp8266 directory, install the esp-open-sdk into a sub-directory +using the *non-standalone* install (i.e., there should not be an sdk directory in the esp-open-sdk +dir when done installing, *if you use the standalone install you will get compilation errors* +with std types, such as `uint32_t`). + +Download the Espressif "NONOS" SDK (use the version mentioned in the release notes) from their +http://bbs.espressif.com/viewforum.php?f=5[download forum] and also expand it into a +sub-directory. Often there are patches to apply, in that case you need to download the patches +from the same source and apply them. + +You can simplify your life (and avoid the hour-long build time for esp-open-sdk) if you are +on an x86 box by downloading the packaged and built esp-open-sdk and the fully patches SDKfrom the +links used in the `.travis.yaml`. + +Clone the esp-link repository into a third sub-directory and check out the tag you would like, +such as `git checkout v2.2.3`. +This way the relative paths in the Makefile will work. +If you choose a different directory structure look at the top of the Makefile for the +appropriate environment variables to define. +Do not use the source tarballs from the release page on github, +these will give you trouble compiling because the Makefile uses git to determine the esp-link +version being built. + +In order to OTA-update the esp8266 you should `export ESP_HOSTNAME=...` with the hostname or +IP address of your module. + +Now, build the code: `make` in the top-level of esp-link. If you want to se the commands being +issued, use `VERBOSE=1 make`. + +A few notes from others (I can't fully verify these): + +- You may need to install `zlib1g-dev` and `python-serial` +- Make sure you have the correct version of the SDK +- Make sure the paths at the beginning of the makefile are correct +- Make sure `esp-open-sdk/xtensa-lx106-elf/bin` is in the PATH set in the Makefile + +### Windows + +Please consider installing docker and using the docker image to save yourself grief getting all +the tools installed and working. +If you do want to compile "natively" on Windows it certainly is possible. + +It is possible to build esp-link on Windows, but it requires a +[gaggle of software to be installed](WINDOWS.md) diff --git a/FLASHING.md b/FLASHING.md new file mode 100644 index 0000000..18d5abf --- /dev/null +++ b/FLASHING.md @@ -0,0 +1,192 @@ +Flashing esp-link +================= + +### Hardware configuration + +This firmware is designed for any esp8266 module. +The recommended connections for an esp-01 module are: + +- URXD: connect to TX of microcontroller +- UTXD: connect to RX of microcontroller +- GPIO0: connect to RESET of microcontroller +- GPIO2: optionally connect green LED to 3.3V (indicates wifi status) + +The recommended connections for an esp-12 module are: + +- URXD: connect to TX of microcontroller +- UTXD: connect to RX of microcontroller +- GPIO12: connect to RESET of microcontroller +- GPIO13: connect to ISP of LPC/ARM microcontroller or to GPIO0 of esp8266 being programmed + (not used with Arduino/AVR) +- GPIO0: optionally connect green "conn" LED to 3.3V (indicates wifi status) +- GPIO2: optionally connect yellow "ser" LED to 3.3V (indicates serial activity) + +If your application has problems with the boot message that is output at ~74600 baud by the ROM +at boot time you can connect an esp-12 module as follows and choose the "swap_uart" pin assignment +in the esp-link web interface: + +- GPIO13: connect to TX of microcontroller +- GPIO15: connect to RX of microcontroller +- GPIO1/UTXD: connect to RESET of microcontroller +- GPIO3/URXD: connect to ISP of LPC/ARM microcontroller or to GPIO0 of esp8266 being programmed + (not used with Arduino/AVR) +- GPIO0: optionally connect green "conn" LED to 3.3V (indicates wifi status) +- GPIO2: optionally connect yellow "ser" LED to 3.3V (indicates serial activity) + +If you are using an FTDI connector, GPIO12 goes to DTR and GPIO13 goes to CTS (or vice-versa, I've +seen both used, sigh). + +The GPIO pin assignments can be changed dynamically in the web UI and are saved in flash. + +### Initial serial flashing + +Download the latest [release](https://github.com/jeelabs/esp-link/releases) or use the +`user1.bin` and `user2.bin` files that are produced by the build process. +You will need to flash the bootloader, the firmware, blank wifi settings, and init data. +Detailed instructions are provided in the release notes. + +_Important_: the firmware adapts to the size of the flash chip using information +stored in the boot sector (address 0). This is the standard way that the esp8266 SDK detects +the flash size. What this means is that you need to set this properly when you flash the bootloader. +If you use esptool.py you can do it using the -ff and -fs options. See the end of this page for +instructions on installing esptool.py. + +The short version for the serial flashing is: +- flash `boot_v1.X.bin` from the official SDK or from the release tgz to `0x00000` +- flash `blank.bin` from the official SDK or from the tgz to `0x3FE000` +- flash `esp_init_data_default.bin` from the official SDK or from the tgz to `0x3FC000` +- flash `user1.bin` to `0x01000` +- be sure to use the commandline flags when flashing the bootloader to set the correct flash size +- some of the addresses vary with flash chip size + +After the initial flashing if you want to update the firmware it is recommended to use the +over-the-air update described further down. If you want to update serially you only need to +reflash `user1.bin`. + +### 32Mbit / 4Mbyte module +On Linux using esptool.py this turns into the following for a 32mbit=4MByte flash chip, +such as an esp-12 module typically has (_substitute the appropriate release number and bootloader +version number_): +``` +curl -L https://github.com/jeelabs/esp-link/releases/download/v2.2.3/esp-link-v2.2.3.tgz | \ + tar xzf - +cd esp-link-v2.2.3 +esptool.py --port /dev/ttyUSB0 --baud 230400 write_flash -fs 32m -ff 80m \ + 0x00000 boot_v1.5.bin 0x1000 user1.bin \ + 0x3FC000 esp_init_data_default.bin 0x3FE000 blank.bin +``` +I use a high baud rate as shown above because I'm impatient, but that's not required. + +### 4Mbit / 512Kbyte module +``` +curl -L https://github.com/jeelabs/esp-link/releases/download/v2.2.3/esp-link-v2.2.3.tgz | \ + tar xzf - +cd esp-link-v2.2.3 +esptool.py --port /dev/ttyUSB0 --baud 460800 write_flash -fs 4m -ff 40m \ + 0x00000 boot_v1.5.bin 0x1000 user1.bin \ + 0x7C000 esp_init_data_default.bin 0x7E000 blank.bin +``` +The `-fs 4m -ff40m` options say 4Mbits and 40Mhz as opposed to 32Mbits at 80Mhz for the 4MByte +flash modules. Note the different address for esp_init_data_default.bin and blank.bin +(the SDK stores its wifi settings near the end of flash, so it changes with flash size). + +For __8Mbit / 1MByte__ modules the addresses are 0xFC000 and 0xFE000. + +## Updating the firmware over-the-air + +This firmware supports over-the-air (OTA) flashing for modules with 1MByte or more flash, +so you do not have to deal with serial flashing again after the initial one! +The recommended way to flash is to use `make wiflash` +if you are also building the firmware and `./wiflash` if you are downloading firmware binaries. + +The resulting commandlines are: +``` +ESP_HOSTNAME=192.168.1.5 make wiflash +``` +or assuming mDNS is working: +``` +ESP_HOSTNAME=esp-link.local make wiflash +``` +or using wiflash.sh: +``` +./wiflash.sh user1.bin user2.bin +``` + +The flashing, restart, and re-associating with your wireless network takes about 15 seconds +and is fully automatic. The first 1MB of flash are divided into two 512KB partitions allowing for new +code to be uploaded into one partition while running from the other. This is the official +OTA upgrade method supported by the SDK, except that the firmware is POSTed to the module +using curl as opposed to having the module download it from a cloud server. On a module with +512KB flash there is only space for one partition and thus no way to do an OTA update. + +If you need to clear the wifi settings you need to reflash the `blank.bin` +using the serial method. + +The flash configuration and the OTA upgrade process is described in more detail +in [FLASH.md](FLASH.md). + +## Installing esptool.py on Linux + +On Linux use [esptool.py](https://github.com/themadinventor/esptool) to flash the esp8266. +If you're a little python challenged then the following install instructions might help: + - Install ez_setup with the following two commands (I believe this will do something + reasonable if you already have it): + + wget https://bootstrap.pypa.io/ez_setup.py + python ez_setup.py + + - Install esptool.py: + + git clone https://github.com/themadinventor/esptool.git + cd esptool + python setup.py install + cd .. + esptool.py -h + +## Installing esptool.py on Windows + +Esptool is a pythin pgm that works just fine on windows. These instructions assume that git and +python are available from the commandline. + +Start a command line, clone esptool, and run `python setup.py install` in esptool's +directory (this step needs to be done only once): +``` +> git clone https://github.com/themadinventor/esptool.git +Cloning into 'esptool'... +remote: Counting objects: 268, done. +emote: Total 268 (delta 0), reused 0 (delta 0), pack-reused 268 +Receiving objects: 100% (268/268), 99.66 KiB | 0 bytes/s, done. +Resolving deltas: 100% (142/142), done. +Checking connectivity... done. + +> cd esptool + +> python setup.py install +running install +... +... +... +Finished processing dependencies for esptool==0.1.0 +``` + +Download and unzip the latest esp-link release package, and start a commandline +in that directory. The command to run is pretty much the same as for linux. +Adjust the path to esptool and the COM port if you don't have the ESP on COM12. 460800 +baud worked just fine for me, writing at ~260kbit/s instead of ~80kbit/s. +``` +>python "../esptool/esptool.py" --port COM12 --baud 115200 write_flash \ + --flash_freq 80m --flash_mode qio --flash_size 32m \ + 0x0000 boot_v1.6.bin 0x1000 user1.bin \ + 0x3FC000 esp_init_data_default.bin 0x3FE000 blank.bin +Connecting... +Erasing flash... +Wrote 3072 bytes at 0x00000000 in 0.3 seconds (79.8 kbit/s)... +Erasing flash... +Wrote 438272 bytes at 0x00001000 in 43.4 seconds (80.7 kbit/s)... +Erasing flash... +Wrote 1024 bytes at 0x003fc000 in 0.1 seconds (83.6 kbit/s)... +Erasing flash... +Wrote 4096 bytes at 0x003fe000 in 0.4 seconds (83.4 kbit/s)... + +Leaving... +``` diff --git a/Makefile b/Makefile index 973209b..4172785 100644 --- a/Makefile +++ b/Makefile @@ -61,6 +61,7 @@ SDK_VERS ?= esp_iot_sdk_v2.0.0.p1 # Try to find the firmware manually extracted, e.g. after downloading from Espressif's BBS, # http://bbs.espressif.com/viewforum.php?f=46 +# USING THE SDK BUNDLED WITH ESP-OPEN-SDK WILL NOT WORK!!! SDK_BASE ?= $(wildcard ../$(SDK_VERS)) # If the firmware isn't there, see whether it got downloaded as part of esp-open-sdk diff --git a/README.adoc b/README.adoc deleted file mode 100644 index 9ed6ceb..0000000 --- a/README.adoc +++ /dev/null @@ -1,545 +0,0 @@ -ESP-LINK: Wifi-Serial Bridge w/REST&MQTT -======================================== -Thorsten von Eicken -:toc: -:toc-title!: -:toc-placement!: - -This firmware connects an attached micro-controller to the internet using a ESP8266 Wifi module. -It implements a number of features: - -[options="compact"] -- transparent bridge between Wifi and serial, useful for debugging or inputting into a uC -- flash-programming attached Arduino/AVR microcontrollers, esp8266 modules, as well as - LPC800-series and other ARM microcontrollers via Wifi -- built-in stk500v1 programmer for AVR uC's: program using HTTP upload of hex file -- outbound REST HTTP requests from the attached micro-controller to the internet -- MQTT client pub/sub from the attached micro-controller to the internet - -The firmware includes a tiny HTTP server based on -http://www.esp8266.com/viewforum.php?f=34[esphttpd] -with a simple web interface, many thanks to Jeroen Domburg for making it available! -The REST and MQTT functionality are loosely based on https://github.com/tuanpmt/espduino -but significantly rewritten and no longer protocol compatible, thanks to tuanpmt for the -inspiration! - -Many thanks to https://github.com/brunnels for contributions in particular around the espduino -functionality. Thank you also to https://github.com/susisstrolch, https://github.com/bc547, -and https://github.com/katast for additional contributions! - -[float] -Table of Contents ------------------ - -toc::[] - -Releases & Downloads --------------------- - -- https://github.com/jeelabs/esp-link/releases/tag/v2.2.3[V2.2.3] is the most recent release. - It has a built-in stk500v1 programmer (for AVRs), work on all modules, and supports mDNS, - sNTP, and syslog. It is built using the Espressif SDK 1.5.4. -- https://github.com/jeelabs/esp-link/releases/tag/v2.1.7[V2.1.7] is the previous release. -- See https://github.com/jeelabs/esp-link/releases[all releases]. - -For quick support and questions chat at -image:https://badges.gitter.im/Join%20Chat.svg[link="https://gitter.im/jeelabs/esp-link"] - -Intro ------ - -### Esp-link goals - -The goal of the esp-link project is to create an advanced Wifi co-processor. Esp-link assumes that -there is a "main processor" (also referred to as "attached uController") and that esp-link's role -is to facilitate communication over Wifi. Where esp-link is a bit unusual is that it's not really -just a Wifi interface or a slave co-processor. In some sense it's the master, because the main -processor can be reset, controlled and reprogrammed through esp-link. The three main areas of -functionality in esp-link are: - -- reprogramming and debugging the attached uC -- letting the attached uC make outbound communication and offloading the protocol processing -- forwarding inbound communication and offloading the protocol processing (this part is the -least developed) - -The goal of the project is also to remain focused on the above mission. In particular, esp-link -is not a platform for stand-alone applications and it does not support connecting sensors or -actuators directly to it. A few users have taken esp-link as a starting point for doing these -things and that's great, but there's also value in keeping the mainline esp-link project -focused on a clear mission. - -### Esp-link uses - -The simplest use of esp-link is as a transparent serial to wifi bridge. You can flash an attached -uC over wifi and you can watch the uC's serial debug output by connecting to port 23 or looking -at the uC Console web page. - -The next level is to use the outbound connectivity of esp-link in the uC code. For example, the -uC can use REST requests to services like thingspeak.com to send sensor values that then get -stored and plotted by the external service. -The uC can also use REST requests to retrieve simple configuration -information or push other forms of notifications. (MQTT functionality is forthcoming.) - -An additional option is to add code to esp-link to customize it and put all the communication -code into esp-link and only keep simple sensor/actuator control in the attached uC. In this -mode the attached uC sends custom commands to esp-link with sensor/acturator info and -registers a set of callbacks with esp-link that control sensors/actuators. This way, custom -commands in esp-link can receive MQTT messages, make simple callbacks into the uC to get sensor -values or change actuators, and then respond back with MQTT. The way this is architected is that -the attached uC registers callbacks at start-up such that the code in the esp doesn't need to -know which exact sensors/actuators the attached uC has, it learns that through the initial -callback registration. - -### Eye Candy - -These screen shots show the Home page, the Wifi configuration page, the console for the -attached microcontroller, and the pin assignments card: - -image:https://cloud.githubusercontent.com/assets/39480/8261425/6ca395a6-167f-11e5-8e92-77150371135a.png[width="45%"] -image:https://cloud.githubusercontent.com/assets/39480/8261427/6caf7326-167f-11e5-8085-bc8b20159b2b.png[width="45%"] -image:https://cloud.githubusercontent.com/assets/39480/8261426/6ca7f75e-167f-11e5-827d-9a1c582ad05d.png[width="45%"] -image:https://cloud.githubusercontent.com/assets/39480/8261658/11e6c64a-1681-11e5-82d0-ea5ec90a6ddb.png[width="45%"] - -Getting Started ---------------- - -### Hardware configuration - -This firmware is designed for any esp8266 module. -The recommended connections for an esp-01 module are: - -- URXD: connect to TX of microcontroller -- UTXD: connect to RX of microcontroller -- GPIO0: connect to RESET of microcontroller -- GPIO2: optionally connect green LED to 3.3V (indicates wifi status) - -The recommended connections for an esp-12 module are: - -- URXD: connect to TX of microcontroller -- UTXD: connect to RX of microcontroller -- GPIO12: connect to RESET of microcontroller -- GPIO13: connect to ISP of LPC/ARM microcontroller or to GPIO0 of esp8266 being programmed - (not used with Arduino/AVR) -- GPIO0: optionally connect green "conn" LED to 3.3V (indicates wifi status) -- GPIO2: optionally connect yellow "ser" LED to 3.3V (indicates serial activity) - -If your application has problems with the boot message that is output at ~74600 baud by the ROM -at boot time you can connect an esp-12 module as follows and choose the "swap_uart" pin assignment -in the esp-link web interface: - -- GPIO13: connect to TX of microcontroller -- GPIO15: connect to RX of microcontroller -- GPIO1/UTXD: connect to RESET of microcontroller -- GPIO3/URXD: connect to ISP of LPC/ARM microcontroller or to GPIO0 of esp8266 being programmed - (not used with Arduino/AVR) -- GPIO0: optionally connect green "conn" LED to 3.3V (indicates wifi status) -- GPIO2: optionally connect yellow "ser" LED to 3.3V (indicates serial activity) - -If you are using an FTDI connector, GPIO12 goes to DTR and GPIO13 goes to CTS (or vice-versa, I've -seen both used, sigh). - -The GPIO pin assignments can be changed dynamically in the web UI and are saved in flash. - -### Initial flashing - -If you want to simply flash a pre-built firmware binary, you can download the latest -https://github.com/jeelabs/esp-link/releases[release] and use your favorite -ESP8266 flashing tool to flash the bootloader, the firmware, and blank settings. -Detailed instructions are provided in the release notes. - -_Important_: the firmware adapts automatically to the size of the flash chip using information -stored in the boot sector (address 0). This is the standard way that the esp8266 SDK detects -the flash size. What this means is that you need to set this properly when you flash the bootloader. -If you use esptool.py you can do it using the -ff and -fs options. - -### Wifi configuration overview - -For proper operation the end state that esp-link needs to arrive at is to have it -join your pre-existing wifi network as a pure station. -However, in order to get there esp-link will start out as an access point and you'll have -to join its network to configure it. The short version is: - - 1. esp-link creates a wifi access point with an SSID of the form `ESP_012ABC` (some modules - use a different SSID form, such as `ai-thinker-012ABC`) - 2. you join your laptop or phone to esp-link's network as a station and you configure - esp-link wifi with your network info by pointing your browser at `http://192.168.4.1/` - 3. you set a hostname for esp-link on the "home" page, or leave the default ("esp-link") - 4. esp-link starts to connect to your network while continuing to also be an access point - ("AP+STA"), the esp-link may show up with a `${hostname}.local` hostname - (depends on your DHCP/DNS config) - 4. esp-link succeeds in connecting and shuts down its own access point after 15 seconds, - you reconnect your laptop/phone to your normal network and access esp-link via its hostname - or IP address - -#### Notes on using AP (access point) mode - -Esp-link does not support STA+AP mode, however it does support STA mode and AP mode. What happens -is that STA+AP mode is used at boot and when making STA changes to allow for recovery: the AP -mode stays on for a while so you can connect to it and fix the STA mode. Once STA has connected, -esp-link switches to STA-only mode. There is no setting to stay in STA+AP mode. So... if you want -to use AP ensure you set esp-link to AP-only mode. If you want STA+AP mode you're gonna have to -modify the source for yourself. (This stuff is painful to test and rather tricky, so don't expect -the way it works to change.) - -### LED indicators - -Assuming appropriate hardware attached to GPIO pins, the green "conn" LED will show the wifi -status as follows: - -- Very short flash once a second: not connected to a network and running as AP+STA, i.e. - trying to connect to the configured network -- Very short flash once every two seconds: not connected to a network and running as AP-only -- Even on/off at 1HZ: connected to the configured network but no IP address (waiting on DHCP) -- Steady on with very short off every 3 seconds: connected to the configured network with an - IP address (esp-link shuts down its AP after 60 seconds) - -The yellow "ser" LED will blink briefly every time serial data is sent or received by the esp-link. - -### Troubleshooting - -- verify that you have sufficient power, borderline power can cause the esp module to seemingly - function until it tries to transmit and the power rail collapses -- if you just cannot flash your esp8266 module (some people call it the zombie mode) make sure you - have gpio0 and gpio15 pulled to gnd with a 1K resistor, gpio2 tied to 3.3V with 1K resistor, and - RX/TX connected without anything in series. If you need to level shift the signal going into the - esp8266's RX use a 1K resistor. Use 115200 baud in the flasher. - (For a permanent set-up I would use higher resistor values but - when nothing seems to work these are the ones I try.) -- if the flashing succeeded, check the "conn" LED to see which mode esp-link is in (see LED info above) -- reset or power-cycle the esp-link to force it to become an access-point if it can't - connect to your network within 15-20 seconds -- if the LED says that esp-link is on your network but you can't get to it, make sure your - laptop is on the same network (and no longer on the esp's network) -- if you do not know the esp-link's IP address on your network, try `esp-link.local`, try to find - the lease in your DHCP server; if all fails, you may have to turn off your access point (or walk - far enough away) and reset/power-cycle esp-link, it will then fail to connect and start its - own AP after 15-20 seconds - -Configuration details ---------------------- - -### Wifi - -After you have serially flashed the module it will create a wifi access point (AP) with an -SSID of the form `ESP_012ABC` where 012ABC is a piece of the module's MAC address. -Using a laptop, phone, or tablet connect to this SSID and then open a browser pointed at -http://192.168.4.1/, you should then see the esp-link web site. - -Now configure the wifi. The desired configuration is for the esp-link to be a -station on your local wifi network so you can communicate with it from all your computers. - -To make this happen, navigate to the wifi page and you should see the esp-link scan -for available networks. You should then see a list of detected networks on the web page and you -can select yours. -Enter a password if your network is secure (highly recommended...) and hit the connect button. - -You should now see that the esp-link has connected to your network and it should show you -its IP address. _Write it down_. You will then have to switch your laptop, phone, or tablet -back to your network and then you can connect to the esp-link's IP address or, depending on your -network's DHCP/DNS config you may be able to go to http://esp-link.local - -At this point the esp-link will have switched to STA mode and be just a station on your -wifi network. These settings are stored in flash and thereby remembered through resets and -power cycles. They are also remembered when you flash new firmware. Only flashing `blank.bin` -via the serial port as indicated above will reset the wifi settings. - -There is a fail-safe, which is that after a reset or a configuration change, if the esp-link -cannot connect to your network it will revert back to AP+STA mode after 15 seconds and thus -both present its `ESP_012ABC`-style network and continue trying to reconnect to the requested network. -You can then connect to the esp-link's AP and reconfigure the station part. - -One open issue (#28) is that esp-link cannot always display the IP address it is getting to the browser -used to configure the ssid/password info. The problem is that the initial STA+AP mode may use -channel 1 and you configure it to connect to an AP on channel 6. This requires the ESP8266's AP -to also switch to channel 6 disconnecting you in the meantime. - -### Hostname, description, DHCP, mDNS - -You can set a hostname on the "home" page, this should be just the hostname and not a domain -name, i.e., something like "test-module-1" and not "test-module-1.mydomain.com". -This has a number of effects: - -- you will see the first 12 chars of the hostname in the menu bar (top left of the page) so - if you have multiple modules you can distinguish them visually -- esp-link will use the hostname in its DHCP request, which allows you to identify the module's - MAC and IP addresses in your DHCP server (typ. your wifi router). In addition, some DHCP - servers will inject these names into the local DNS cache so you can use URLs like - `hostname.local`. -- someday, esp-link will inject the hostname into mDNS (multicast DNS, bonjour, etc...) so - URLs of the form `hostname.local` work for everyone (as of v2.1.beta5 mDNS is disabled due - to reliability issues with it) - -You can also enter a description of up to 128 characters on the home page (bottom right). This -allows you to leave a memo for yourself, such as "installed in basement to control the heating -system". This descritpion is not used anywhere else. - -Building the firmware ---------------------- - -### Linux - -The firmware has been built using the https://github.com/pfalcon/esp-open-sdk[esp-open-sdk] -on a Linux system. Create an esp8266 directory, install the esp-open-sdk into a sub-directory -using the *non-standalone* install (i.e., there should not be an sdk directory in the esp-open-sdk -dir when done installing, if you use the standalone install you will get compilation errors -with std types, such as `uint32_t`). - -Download the Espressif SDK (use the version mentioned in the release notes) from their -http://bbs.espressif.com/viewforum.php?f=5[download forum] and also expand it into a -sub-directory. - -Clone the esp-link repository into a third sub-directory and check out the tag you would like, -such as `git checkout v2.1.7`. -This way the relative paths in the Makefile will work. -If you choose a different directory structure look at the Makefile for the appropriate environment -variables to define. -Do not use the source tarballs from the release page on github, -these will give you trouble compiling because the Makefile uses git to determine the esp-link -version being built. - -In order to OTA-update the esp8266 you should `export ESP_HOSTNAME=...` with the hostname or -IP address of your module. - -Now, build the code: `make` in the top-level of esp-link. If you want to se the commands being -issued, use `VERBOSE=1 make`. - -A few notes from others (I can't fully verify these): - -- You may need to install `zlib1g-dev` and `python-serial` -- Make sure you have the correct version of the esp_iot_sdk -- Make sure the paths at the beginning of the makefile are correct -- Make sure `esp-open-sdk/xtensa-lx106-elf/bin` is in the PATH set in the Makefile - -### Windows - -It is possible to build esp-link on Windows, but it requires a gaggle of software to be installed: - -- Install the unofficial sdk, mingw, SourceTree (gui git client), python 2.7, git cli, Java -- Use SourceTree to checkout under C:\espressif or wherever you installed the unofficial sdk, - (see this thread for the unofficial sdk http://www.esp8266.com/viewtopic.php?t=820) -- Create a symbolic link under c:/espressif for the git bin directory under program files and - the java bin directory under program files. -- ... - -### Updating the firmware over-the-air - -This firmware supports over-the-air (OTA) flashing, so you do not have to deal with serial -flashing again after the initial one! The recommended way to flash is to use `make wiflash` -if you are also building the firmware. -If you are downloading firmware binaries use `./wiflash`. -`make wiflash` assumes that you set `ESP_HOSTNAME` to the hostname or IP address of your esp-link. -You can easily do that using something like `ESP_HOSTNAME=192.168.1.5 make wiflash`. - -The flashing, restart, and re-associating with your wireless network takes about 15 seconds -and is fully automatic. The first 1MB of flash are divided into two 512KB partitions allowing for new -code to be uploaded into one partition while running from the other. This is the official -OTA upgrade method supported by the SDK, except that the firmware is POSTed to the module -using curl as opposed to having the module download it from a cloud server. On a module with -512KB flash there is only space for one partition and thus no way to do an OTA update. - -If you are downloading the binary versions of the firmware (links forthcoming) you need to have -both `user1.bin` and `user2.bin` handy and run `wiflash.sh user1.bin user2.bin`. -This will query the esp-link for which file it needs, upload the file, and then reconnect to -ensure all is well. - -Note that when you flash the firmware the wifi settings are all preserved so the esp-link should -reconnect to your network within a few seconds and the whole flashing process should take 15-30 -from beginning to end. If you need to clear the wifi settings you need to reflash the `blank.bin` -using the serial port. - -The flash configuration and the OTA upgrade process is described in more detail in [FLASH.md](FLASH.md) - -Serial bridge and connections to Arduino, AVR, ARM, LPC microcontrollers ------------------------------------------------------------------------- - -In order to connect through the esp-link to a microcontroller use port 23. For example, -on linux you can use `nc esp-hostname 23` or `telnet esp-hostname 23`. - -Note that multiple connections to port 23 and 2323 can be made simultaneously. Esp-link will -intermix characters received on all these connections onto the serial TX and it will -broadcast incoming characters from the serial RX to all connections. Use with caution! - -### Flashing an attached AVR/Arduino - -There are three options for reprogramming an attached AVR/Arduino microcontroller: - -- Use avrdude and point it at port 23 of esp-link. Esp-link automatically detects the programming - sequence and issues a reset to the AVR. -- Use avrdude and point it at port 2323 of esp-link. This is the same as port 23 except that the - autodectection is not used and the reset happens because port 2323 is used -- Use curl or a similar tool to HTTP POST the firmware to esp-link. This uses the built-in - programmer, which only works for AVRs/Arduinos with the optiboot bootloader (which is std). - -To reprogram an Arduino / AVR microcontroller by pointing avrdude at port 23 or 2323 you -specify a serial port of the form `net:esp-link:23` in avrdude's -P option, where -`esp-link` is either the hostname of your esp-link or its IP address). -This is instead of specifying a serial port of the form /dev/ttyUSB0. -Esp-link detects that avrdude starts its connection with a flash synchronization sequence -and sends a reset to the AVR microcontroller so it can switch into flash programming mode. - -To reprogram using the HTTP POST method you need to first issue a POST to put optiboot into -programming mode: POST to `http://esp-link/pgm/sync`, this starts the process. Then check that -synchronization with optiboot has been achieved by issuing a GET to the same URL -(`http://esp-link/pgm/sync`). Repeat until you have sync (takes <500ms normally). Finally -issue a POST request to `http://esp-link/pgm/upload` with your hex file as POST data (raw, -not url-encoded or multipart-mime. Please look into the avrflash script for the curl command-line -details or use that script directly (`./avrflash esp-link.local my_sketch.hex`). -_Important_: after the initial sync request that resets the AVR you have 10 seconds to get to the -upload post or esp-link will time-out. So if you're manually entering curl commands have them -prepared so you can copy&paste! - -Beware of the baud rate, which you can set on the uC Console page. Sometimes you may be using -115200 baud in sketches but the bootloader may use 57600 baud. When you use port 23 or 2323 you -need to set the baud rate correctly. If you use the built-in programmer (HTTP POST method) then -esp-link will try the configured baud rate and also 9600, 57600, and 115200 baud, so it should -work even if you have the wrong baud rate configured... - -When to use which method? If port 23 works then go with that. If you have trouble getting sync -or it craps out in the middle too often then try the built-in programmer with the HTTP POST. -If your AVR doesn't use optiboot then use port 2323 since esp-link may not recognize the programming -sequence and not issue a reset if you use port 23. - -If you are having trouble with the built-in programmer and see something like this: - --------------------- -# ./avrflash 192.168.3.104 blink.hex -Error checking sync: FAILED to SYNC: abandoned after timeout, got: -:\xF/\x00\xCj\xCz\xCJ\xCZ\xC\xAÜ\xC\xAä\xC\xAÜ\xC\xAä\xC\xBì\xC\xBô\xC\xBì\xC\xBô\xC\xAÜ\xC\xAä\xC --------------------- - -the most likely cause is a baud rate mismatch and/or a bad connection from the esp8266 to the -AVRs reset line. -The baud rate used by esp-link is set on the uC Console web page and, as mentioned above, it will -automatically try 9600, 57600, and 115200 as well. -The above garbage characters are most likely due to optiboot timing out and starting the sketch -and then the sketch sending data at a different baud rate than configured into esp-link. -Note that sketches don't necessarily use the same baud rate as optiboot, so you may have the -correct baud rate configured but reset isn't functioning, or reset may be functioning but the -baud rate may be incorrect. - -The output of a successful flash using the built-in programmer looks like this: - --------------------- -Success. 3098 bytes at 57600 baud in 0.8s, 3674B/s 63% efficient --------------------- - -This says that the sketch comprises 3098 bytes of flash, was written in 0.8 seconds -(excludes the initial sync time) at 57600 baud, -and the 3098 bytes were flashed at a rate of 3674 bytes per second. -The efficiency measure is the ratio of the actual rate to the serial baud rate, -thus 3674/5760 = 0.63 (there are 10 baud per character). -The efficiency is not 100% because there is protocol overhead (such as sync, record type, and -length characters) -and there is dead time waiting for an ack or preparing the next record to be sent. - -### Details of built-in AVR flash algorithm - -The built-in flashing algorithm differs a bit from what avrdude does. The programming protocol -states that STK_GET_SYNC+CRC_EOP (0x30 0x20) should be sent to synchronize, but that works poorly -because the AVR's UART only buffers one character. This means that if STK_GET_SYNC+CRC_EOP is -sent twice there is a high chance that only the last character (CRC_EOP) is actually -received. If that is followed by another STK_GET_SYNC+CRC_EOP sequence then optiboot receives -CRC_EOP+STK_GET_SYNC+CRC_EOP which causes it to abort and run the old sketch. Ending up in that -situation is quite likely because optiboot initializes the UART as one of the first things, but -then goes off an flashes an LED for ~300ms during which it doesn't empty the UART. - -Looking at the optiboot code, the good news is that CRC_EOP+CRC_EOP can be used to get an initial -response without the overrun danger of the normal sync sequence and this is what esp-link does. -The programming sequence runs as follows: - -- esp-link sends a brief reset pulse (1ms) -- esp-link sends CRC_EOP+CRC_EOP ~50ms later -- esp-link sends CRC_EOP+CRC_EOP every ~70-80ms -- eventually optiboot responds with STK_INSYNC+STK_OK (0x14;0x10) -- esp-link sends one CRC_EOP to sort out the even/odd issue -- either optiboot responds with STK_INSYNC+STK_OK or nothing happens for 70-80ms, in which case - esp-link sends another CRC_EOP -- esp-link sends STK_GET_SYNC+CRC_EOP and optiboot responds with STK_INSYNC+STK_OK and we're in - sync now -- esp-link sends the next command (starts with 'u') and programming starts... - -If no sync is achieved, esp-link changes baud rate and the whole thing starts over with a reset -pulse about 600ms, esp-link gives up after about 5 seconds and reports an error. - -### Flashing an attached ARM processor - -You can reprogram NXP's LPC800-series and many other ARM processors as well by pointing your -programmer similarly at the esp-link's port 23. For example, if you are using -https://github.com/jeelabs/embello/tree/master/tools/uploader a command line like -`uploader -t -s -w esp-link:23 build/firmware.bin` does the trick. -The way it works is that the uploader uses telnet protocol escape sequences in order to -make esp-link issue the appropriate "ISP" and reset sequence to the microcontroller to start the -flash programming. If you use a different ARM programming tool it will work as well as long as -it starts the connection with the `?\r\n` synchronization sequence. - -### Flashing an attached esp8266 - -Yes, you can use esp-link running on one esp8266 module to flash another esp8266 module, -however it is rather tricky! The problem is not electric, it is wifi interference. -The basic idea is to use some method to direct the esp8266 flash program to port 2323 of -esp-link. Using port 2323 with the appropriate wiring will cause the esp8266's reset and -gpio0 pins to be toggled such that the chip enters the flash programming mode. - -One option for connecting the programmer with esp-link is to use my version of esptool.py -at http://github.com/tve/esptool, which supports specifying a URL instead of a port. Thus -instead of specifying something like `--port /dev/ttyUSB0` or `--port COM1` you specify -`--port socket://esp-link.local:2323`. Important: the baud rate specified on the esptool.py -command-line is irrelevant as the baud rate used by esp-link will be the one set in the -uC console page. Fortunately the esp8266 bootloader does auto-baud detection. (Setting the -baud rate to 115200 is recommended.) - -Another option is to use a serial-to-tcp port forwarding driver and point that to port 2323 -of esp-link. On windows users have reported success with -http://www.hw-group.com/products/hw_vsp/hw_vsp2_en.html[HW Virtual Serial Port] - -Now to the interference problem: once the attached esp8266 is reset it -starts outputting its 26Mhz clock on gpio0, which needs to be attached to -the esp8266 running esp-link (since it needs to drive gpio0 low during -the reset to enter flash mode). This 26Mhz signal on gpio0 causes a -significant amount of radio interference with the result that the esp8266 -running esp-link has trouble receiving Wifi packets. You can observe this -by running a ping to esp-link in another window: as soon as the target -esp8266 is reset, the pings become very slow or stop altogetehr. As soon -as you remove power to the attached esp8266 the pings resume beautifully. - -To try and get the interference under control, try some of the following: -add a series 100ohm resistor and 100pf capacitor to ground as close to -the gpio0 pin as possible (basically a low pass filter); and/or pass -the cable connecting the two esp8266's through a ferrite bead. - -### Debug log - -The esp-link web UI can display the esp-link debug log (os_printf statements in the code). This -is handy but sometimes not sufficient. Esp-link also prints the debug info to the UART where -it is sometimes more convenient and sometimes less... For this reason three UART debug log -modes are supported that can be set in the web UI (and the mode is saved in flash): - -- auto: the UART log starts enabled at boot using uart0 and disables itself when esp-link - associates with an AP. It re-enables itself if the association is lost. -- off: the UART log is always off -- on0: the UART log is always on using uart0 -- on1: the UART log is always on using uart1 (gpio2 pin) - -Note that even if the UART log is always off the ROM prints to uart0 whenever the -esp8266 comes out of reset. This cannot be disabled. - -Outbound HTTP REST requests and MQTT client -------------------------------------------- - -The V2 versions of esp-link use the SLIP protocol over the serial link to support simple outbound -HTTP REST requests as well as an MQTT client. The SLIP protocol consists of commands with -binary arguments sent from the -attached microcontroller to the esp8266, which then performs the command and responds back. -The responses back use a callback address in the attached microcontroller code, i.e., the -command sent by the uC contains a callback address and the response from the esp8266 starts -with that callback address. This enables asynchronous communication where esp-link can notify the -uC when requests complete or when other actions happen, such as wifi connectivity status changes. - -You can find REST and MQTT libraries as well as demo sketches in the -https://github.com/jeelabs/el-client[el-client] repository. - -Contact -------- - -If you find problems with esp-link, please create a github issue. If you have a question, please -use the gitter chat link at the top of this page. diff --git a/README.md b/README.md new file mode 100644 index 0000000..00efe40 --- /dev/null +++ b/README.md @@ -0,0 +1,163 @@ +ESP-LINK: Wifi-Serial Bridge w/REST&MQTT +======================================== + + + +The esp-link firmware connects a micro-controller to the internet using an ESP8266 Wifi module. +It implements a number of features: + +- transparent bridge between Wifi and serial, useful for debugging or inputting into a uC +- flash-programming attached Arduino/AVR microcontrollers and + LPC800-series and other ARM microcontrollers via Wifi +- built-in stk500v1 programmer for AVR uC's: program using HTTP upload of hex file +- outbound REST HTTP requests from the attached micro-controller to the internet +- MQTT client pub/sub from the attached micro-controller to the internet +- serve custom web pages containing data that is dynamically pulled from the attached uC and + that contain buttons and fields that are transmitted to the attached uC (feature not + fully ready yet) + +The firmware includes a tiny HTTP server based on +[esphttpd](http://www.esp8266.com/viewforum.php?f=34) +with a simple web interface, many thanks to Jeroen Domburg for making it available! +The REST and MQTT functionality are loosely based on [espduino](https://github.com/tuanpmt/espduino) +but significantly rewritten and no longer protocol compatible, thanks to tuanpmt for the +inspiration! + +The following people contributed significant functionality to esp-link: +[brunnels](https://github.com/brunnels) (espduino integration), +[cskarai](https://github.com/cskarai) (custom dynamic web pages), +[beegee-tokyo](https://github.com/beegee-tokyo) (lots of code documentation), +[susisstrolch](https://github.com/susisstrolch) (syslog feature), +[bc547](https://github.com/bc547) and [katast](https://github.com/katast) (misc contributions). +Esp-link is the work of many contributors! + +Note that http://github.com/jeelabs/esp-link is the original esp-link software which has +notably been forked by arduino.org as [Esp-Link](https://github.com/arduino-org/Esp-Link) and shipped +with the initial Arduino Uno Wifi. The JeeLabs esp-link has evolved significantly since the +fork and added cool new features as well as bug fixes. + +### Quick links + +In this document: [goals](#esp-link-goals), [uses](#esp-link-uses), [eye candy](#eye-candy), +[getting-started](#getting-started), [serial-bridge](#serial-bridge), [contact](#contact). + +Separate documents: +- [hardware configuration](FLASHING.md), [serial flashing](FLASHING.md#initial-serial-flashing) +- [wifi configuration](WIFI-CONFIG.md) +- [troubleshooting](TROUBLESHOOTING.md), [LED indicators](TROUBLESHOOTING.md#led-indicators) +- [flashing an attached uC](UC-FLASHING.md) +- [MQTT and outbound REST requests](RESTMQTT.md) +- [service web pages](WEB-SERVER.md) +- [building esp-link](BUILDING.md), [over-the-air flashing](BUILDING.md#updating-the-firmware-over-the-air) +- [flash layout](FLASH.md) + +For quick support and questions chat at +[![Chat at https://gitter.im/jeelabs/esp-link](https://badges.gitter.im/esp-link.svg)](https://gitter.im/jeelabs/esp-link) +or (a little slower) open a github issue. + +Releases & Downloads +-------------------- +Esp-link uses semantic versioning. The main change between versions 1.x and 2.x was the +addition of MQTT and outbound REST requests from the attached uC. The main change between 2.x +and 3.x will be the addition of custom web pages (this is not ready yet). + +- The master branch is currently unstable as we integrate a number of new features to get + to version 3.0. Please use v2.2.3 unless you want to hack up the latest code! + This being said, the older functionality seems to work fine on master, YMMV... +- [V2.2.3](https://github.com/jeelabs/esp-link/releases/tag/v2.2.3) is the most recent release. + It has a built-in stk500v1 programmer (for AVRs), work on all modules, and supports mDNS, + sNTP, and syslog. It is built using the Espressif SDK 1.5.4. +- [V2.1.7](https://github.com/jeelabs/esp-link/releases/tag/v2.1.7) is the previous release. +- See [all releases](https://github.com/jeelabs/esp-link/releases). + +## Esp-link goals + +The goal of the esp-link project is to create an advanced Wifi co-processor. Esp-link assumes that +there is a "main processor" (also referred to as "attached uController") and that esp-link's role +is to facilitate communication over Wifi. This means that esp-link does not just connect TCP/UDP +sockets through to the attached uC, rather it implements mostly higher-level functionality to +offload the attached uC, which often has much less flash and memory than esp-link. + +Where esp-link is a bit unusual is that it's not really +just a Wifi interface or a slave co-processor. In some sense it's the master, because the main +processor can be reset, controlled and reprogrammed through esp-link. The three main areas of +functionality in esp-link are: + +- reprogramming and debugging the attached uC +- letting the attached uC make outbound communication and offloading the protocol processing +- forwarding inbound communication and offloading the protocol processing + +The goal of the project is also to remain focused on the above mission. In particular, esp-link +is not a platform for stand-alone applications and it does not support connecting sensors or +actuators directly to it. A few users have taken esp-link as a starting point for doing these +things and that's great, but there's also value in keeping the mainline esp-link project +focused on a clear mission. + +## Esp-link uses + +The simplest use of esp-link is as a transparent serial to wifi bridge. You can flash an attached +uC over wifi and you can watch the uC's serial debug output by connecting to port 23 or looking +at the uC Console web page. + +The next level is to use the outbound connectivity of esp-link in the uC code. For example, the +uC can use REST requests to services like thingspeak.com to send sensor values that then get +stored and plotted by the external service. +The uC can also use REST requests to retrieve simple configuration +information or push other forms of notifications. (MQTT functionality is forthcoming.) + +An additional option is to add code to esp-link to customize it and put all the communication +code into esp-link and only keep simple sensor/actuator control in the attached uC. In this +mode the attached uC sends custom commands to esp-link with sensor/acturator info and +registers a set of callbacks with esp-link that control sensors/actuators. This way, custom +commands in esp-link can receive MQTT messages, make simple callbacks into the uC to get sensor +values or change actuators, and then respond back with MQTT. The way this is architected is that +the attached uC registers callbacks at start-up such that the code in the esp doesn't need to +know which exact sensors/actuators the attached uC has, it learns that through the initial +callback registration. + +## Eye Candy + +These screen shots show the Home page, the Wifi configuration page, the console for the +attached microcontroller, and the pin assignments card: + + + + + + + + +Getting Started +--------------- + +To get started you need to: + 1. prepare your esp8266 module for serial flashing + 2. download the latest esp-link release image (you can build your own later) + 3. flash the firmware + 4. configure the Wifi in esp-link for your network + +You can then attach a uC and upload a sketch: + 1. attach a uC (e.g. arduino) to your esp8266 module + 2. connect via the serial port to see a pre-loaded sketch running + 3. upload a fresh version of the sketch + +From there, more advanced steps are: +- write a sketch that uses MQTT to communicate, or that makes outbound REST requests +- create some web pages and write a sketch that populates data in them or reacts to buttons + and forms +- make changes or enhancements to esp-link and build your own firmware + +### Serial bridge + +In order to connect through the esp-link to a microcontroller use port 23. For example, +on linux you can use `nc esp-hostname 23` or `telnet esp-hostname 23`. + +Note that multiple connections to port 23 and 2323 can be made simultaneously. Esp-link will +intermix characters received on all these connections onto the serial TX and it will +broadcast incoming characters from the serial RX to all connections. Use with caution! + +Contact +------- + +If you find problems with esp-link, please create a github issue. If you have a question, please +use the gitter chat link at the top of this page. diff --git a/RESTMQTT.md b/RESTMQTT.md new file mode 100644 index 0000000..1383db8 --- /dev/null +++ b/RESTMQTT.md @@ -0,0 +1,14 @@ +Esp-link: Outbound HTTP REST requests and MQTT client +------------------------------------------- + +The V2 versions of esp-link use the SLIP protocol over the serial link to support simple outbound +HTTP REST requests as well as an MQTT client. The SLIP protocol consists of commands with +binary arguments sent from the +attached microcontroller to the esp8266, which then performs the command and responds back. +The responses back use a callback address in the attached microcontroller code, i.e., the +command sent by the uC contains a callback address and the response from the esp8266 starts +with that callback address. This enables asynchronous communication where esp-link can notify the +uC when requests complete or when other actions happen, such as wifi connectivity status changes. + +You can find REST and MQTT libraries as well as demo sketches in the +[el-client](https://github.com/jeelabs/el-client) repository. diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md new file mode 100644 index 0000000..be18e7d --- /dev/null +++ b/TROUBLESHOOTING.md @@ -0,0 +1,36 @@ +Esp-Link troubleshooting +======================== + +### Troubleshooting + +- verify that you have sufficient power, borderline power can cause the esp module to seemingly + function until it tries to transmit and the power rail collapses +- if you just cannot flash your esp8266 module (some people call it the zombie mode) make sure you + have gpio0 and gpio15 pulled to gnd with a 1K resistor, gpio2 tied to 3.3V with 1K resistor, and + RX/TX connected without anything in series. If you need to level shift the signal going into the + esp8266's RX use a 1K resistor. Use 115200 baud in the flasher. + (For a permanent set-up I would use higher resistor values but + when nothing seems to work these are the ones I try.) +- if the flashing succeeded, check the "conn" LED to see which mode esp-link is in (see LED info above) +- reset or power-cycle the esp-link to force it to become an access-point if it can't + connect to your network within 15-20 seconds +- if the LED says that esp-link is on your network but you can't get to it, make sure your + laptop is on the same network (and no longer on the esp's network) +- if you do not know the esp-link's IP address on your network, try `esp-link.local`, try to find + the lease in your DHCP server; if all fails, you may have to turn off your access point (or walk + far enough away) and reset/power-cycle esp-link, it will then fail to connect and start its + own AP after 15-20 seconds + +### LED indicators + +Assuming appropriate hardware attached to GPIO pins, the green "conn" LED will show the wifi +status as follows: + +- Very short flash once a second: not connected to a network and running as AP+STA, i.e. + trying to connect to the configured network +- Very short flash once every two seconds: not connected to a network and running as AP-only +- Even on/off at 1HZ: connected to the configured network but no IP address (waiting on DHCP) +- Steady on with very short off every 3 seconds: connected to the configured network with an + IP address (esp-link shuts down its AP after 60 seconds) + +The yellow "ser" LED will blink briefly every time serial data is sent or received by the esp-link. diff --git a/UC-FLASHING.md b/UC-FLASHING.md new file mode 100644 index 0000000..fd2ba63 --- /dev/null +++ b/UC-FLASHING.md @@ -0,0 +1,177 @@ +Flashing an attached Microcontroller +==================================== + +In order to connect through the esp-link to a microcontroller use port 23. For example, +on linux you can use `nc esp-hostname 23` or `telnet esp-hostname 23`. + +Note that multiple connections to port 23 and 2323 can be made simultaneously. Esp-link will +intermix characters received on all these connections onto the serial TX and it will +broadcast incoming characters from the serial RX to all connections. Use with caution! + +### Flashing an attached AVR/Arduino + +There are three options for reprogramming an attached AVR/Arduino microcontroller: + +- Use avrdude and point it at port 23 of esp-link. Esp-link automatically detects the programming + sequence and issues a reset to the AVR. +- Use avrdude and point it at port 2323 of esp-link. This is the same as port 23 except that the + autodectection is not used and the reset happens because port 2323 is used +- Use curl or a similar tool to HTTP POST the firmware to esp-link. This uses the built-in + programmer, which only works for AVRs/Arduinos with the optiboot bootloader (which is std). + +To reprogram an Arduino / AVR microcontroller by pointing avrdude at port 23 or 2323 you +specify a serial port of the form `net:esp-link:23` in avrdude's -P option, where +`esp-link` is either the hostname of your esp-link or its IP address). +This is instead of specifying a serial port of the form /dev/ttyUSB0. +Esp-link detects that avrdude starts its connection with a flash synchronization sequence +and sends a reset to the AVR microcontroller so it can switch into flash programming mode. + +To reprogram using the HTTP POST method you need to first issue a POST to put optiboot into +programming mode: POST to `http://esp-link/pgm/sync`, this starts the process. Then check that +synchronization with optiboot has been achieved by issuing a GET to the same URL +(`http://esp-link/pgm/sync`). Repeat until you have sync (takes <500ms normally). Finally +issue a POST request to `http://esp-link/pgm/upload` with your hex file as POST data (raw, +not url-encoded or multipart-mime. Please look into the avrflash script for the curl command-line +details or use that script directly (`./avrflash esp-link.local my_sketch.hex`). +_Important_: after the initial sync request that resets the AVR you have 10 seconds to get to the +upload post or esp-link will time-out. So if you're manually entering curl commands have them +prepared so you can copy&paste! + +Beware of the baud rate, which you can set on the uC Console page. Sometimes you may be using +115200 baud in sketches but the bootloader may use 57600 baud. When you use port 23 or 2323 you +need to set the baud rate correctly. If you use the built-in programmer (HTTP POST method) then +esp-link will try the configured baud rate and also 9600, 57600, and 115200 baud, so it should +work even if you have the wrong baud rate configured... + +When to use which method? If port 23 works then go with that. If you have trouble getting sync +or it craps out in the middle too often then try the built-in programmer with the HTTP POST. +If your AVR doesn't use optiboot then use port 2323 since esp-link may not recognize the programming +sequence and not issue a reset if you use port 23. + +If you are having trouble with the built-in programmer and see something like this: + +``` +# ./avrflash 192.168.3.104 blink.hex +Error checking sync: FAILED to SYNC: abandoned after timeout, got: +:\xF/\x00\xCj\xCz\xCJ\xCZ\xC\xAÜ\xC\xAä\xC\xAÜ\xC\xAä\xC\xBì\xC\xBô\xC\xBì\xC\xBô\xC\xAÜ\xC\xAä\xC +``` + +the most likely cause is a baud rate mismatch and/or a bad connection from the esp8266 to the +AVRs reset line. +The baud rate used by esp-link is set on the uC Console web page and, as mentioned above, it will +automatically try 9600, 57600, and 115200 as well. +The above garbage characters are most likely due to optiboot timing out and starting the sketch +and then the sketch sending data at a different baud rate than configured into esp-link. +Note that sketches don't necessarily use the same baud rate as optiboot, so you may have the +correct baud rate configured but reset isn't functioning, or reset may be functioning but the +baud rate may be incorrect. + +The output of a successful flash using the built-in programmer looks like this: + +``` +Success. 3098 bytes at 57600 baud in 0.8s, 3674B/s 63% efficient +``` + +This says that the sketch comprises 3098 bytes of flash, was written in 0.8 seconds +(excludes the initial sync time) at 57600 baud, +and the 3098 bytes were flashed at a rate of 3674 bytes per second. +The efficiency measure is the ratio of the actual rate to the serial baud rate, +thus 3674/5760 = 0.63 (there are 10 baud per character). +The efficiency is not 100% because there is protocol overhead (such as sync, record type, and +length characters) +and there is dead time waiting for an ack or preparing the next record to be sent. + +### Details of built-in AVR flash algorithm + +The built-in flashing algorithm differs a bit from what avrdude does. The programming protocol +states that STK_GET_SYNC+CRC_EOP (0x30 0x20) should be sent to synchronize, but that works poorly +because the AVR's UART only buffers one character. This means that if STK_GET_SYNC+CRC_EOP is +sent twice there is a high chance that only the last character (CRC_EOP) is actually +received. If that is followed by another STK_GET_SYNC+CRC_EOP sequence then optiboot receives +CRC_EOP+STK_GET_SYNC+CRC_EOP which causes it to abort and run the old sketch. Ending up in that +situation is quite likely because optiboot initializes the UART as one of the first things, but +then goes off an flashes an LED for ~300ms during which it doesn't empty the UART. + +Looking at the optiboot code, the good news is that CRC_EOP+CRC_EOP can be used to get an initial +response without the overrun danger of the normal sync sequence and this is what esp-link does. +The programming sequence runs as follows: + +- esp-link sends a brief reset pulse (1ms) +- esp-link sends CRC_EOP+CRC_EOP ~50ms later +- esp-link sends CRC_EOP+CRC_EOP every ~70-80ms +- eventually optiboot responds with STK_INSYNC+STK_OK (0x14;0x10) +- esp-link sends one CRC_EOP to sort out the even/odd issue +- either optiboot responds with STK_INSYNC+STK_OK or nothing happens for 70-80ms, in which case + esp-link sends another CRC_EOP +- esp-link sends STK_GET_SYNC+CRC_EOP and optiboot responds with STK_INSYNC+STK_OK and we're in + sync now +- esp-link sends the next command (starts with 'u') and programming starts... + +If no sync is achieved, esp-link changes baud rate and the whole thing starts over with a reset +pulse about 600ms, esp-link gives up after about 5 seconds and reports an error. + +### Flashing an attached ARM processor + +You can reprogram NXP's LPC800-series and many other ARM processors as well by pointing your +programmer similarly at the esp-link's port 23. For example, if you are using +https://github.com/jeelabs/embello/tree/master/tools/uploader a command line like +`uploader -t -s -w esp-link:23 build/firmware.bin` does the trick. +The way it works is that the uploader uses telnet protocol escape sequences in order to +make esp-link issue the appropriate "ISP" and reset sequence to the microcontroller to start the +flash programming. If you use a different ARM programming tool it will work as well as long as +it starts the connection with the `?\r\n` synchronization sequence. + +### Flashing an attached esp8266 + +__Flashing another esp8266 module is possible in theory but real-world attempts haveso far been +rather unsuccessful due to Wifi interference. This section is left here in case someone else +wants to dig in and find a solution.__ + +You can use esp-link running on one esp8266 module to flash another esp8266 module, +however it is rather tricky! The problem is not electric, it is wifi interference. +The basic idea is to use some method to direct the esp8266 flash program to port 2323 of +esp-link. Using port 2323 with the appropriate wiring will cause the esp8266's reset and +gpio0 pins to be toggled such that the chip enters the flash programming mode. + +One option for connecting the programmer with esp-link is to use my version of esptool.py +at http://github.com/tve/esptool, which supports specifying a URL instead of a port. Thus +instead of specifying something like `--port /dev/ttyUSB0` or `--port COM1` you specify +`--port socket://esp-link.local:2323`. Important: the baud rate specified on the esptool.py +command-line is irrelevant as the baud rate used by esp-link will be the one set in the +uC console page. Fortunately the esp8266 bootloader does auto-baud detection. (Setting the +baud rate to 115200 is recommended.) + +Another option is to use a serial-to-tcp port forwarding driver and point that to port 2323 +of esp-link. On windows users have reported success with +http://www.hw-group.com/products/hw_vsp/hw_vsp2_en.html[HW Virtual Serial Port] + +Now to the interference problem: once the attached esp8266 is reset it +starts outputting its 26Mhz clock on gpio0, which needs to be attached to +the esp8266 running esp-link (since it needs to drive gpio0 low during +the reset to enter flash mode). This 26Mhz signal on gpio0 causes a +significant amount of radio interference with the result that the esp8266 +running esp-link has trouble receiving Wifi packets. You can observe this +by running a ping to esp-link in another window: as soon as the target +esp8266 is reset, the pings become very slow or stop altogetehr. As soon +as you remove power to the attached esp8266 the pings resume beautifully. + +To try and get the interference under control, try some of the following: +add a series 100ohm resistor and 100pf capacitor to ground as close to +the gpio0 pin as possible (basically a low pass filter); and/or pass +the cable connecting the two esp8266's through a ferrite bead. + +### Debug log + +The esp-link web UI can display the esp-link debug log (os_printf statements in the code). This +is handy but sometimes not sufficient. Esp-link also prints the debug info to the UART where +it is sometimes more convenient and sometimes less... For this reason three UART debug log +modes are supported that can be set in the web UI (and the mode is saved in flash): + +- auto: the UART log starts enabled at boot using uart0 and disables itself when esp-link + associates with an AP. It re-enables itself if the association is lost. +- off: the UART log is always off +- on0: the UART log is always on using uart0 +- on1: the UART log is always on using uart1 (gpio2 pin) + +Note that even if the UART log is always off the ROM prints to uart0 whenever the +esp8266 comes out of reset. This cannot be disabled. diff --git a/WIFI-CONFIG.md b/WIFI-CONFIG.md new file mode 100644 index 0000000..02e13c3 --- /dev/null +++ b/WIFI-CONFIG.md @@ -0,0 +1,87 @@ +Esp-link Wifi configuration +=========================== + +For proper operation the end state that esp-link needs to arrive at is to have it +join your pre-existing wifi network as a pure station. +However, in order to get there esp-link will start out as an access point and you'll have +to join its network to configure it. The short version is: + + 1. esp-link creates a wifi access point with an SSID of the form `ESP_012ABC` (some modules + use a different SSID form, such as `ai-thinker-012ABC`) + 2. you join your laptop or phone to esp-link's network as a station and you configure + esp-link wifi with your network info by pointing your browser at `http://192.168.4.1/` + 3. you set a hostname for esp-link on the "home" page, or leave the default ("esp-link") + 4. esp-link starts to connect to your network while continuing to also be an access point + ("AP+STA"), the esp-link may show up with a `${hostname}.local` hostname + (depends on your DHCP/DNS config) + 4. esp-link succeeds in connecting and shuts down its own access point after 15 seconds, + you reconnect your laptop/phone to your normal network and access esp-link via its hostname + or IP address + +### Notes on using AP (access point) mode + +Esp-link does not support STA+AP mode, however it does support STA mode and AP mode. What happens +is that STA+AP mode is used at boot and when making STA changes to allow for recovery: the AP +mode stays on for a while so you can connect to it and fix the STA mode. Once STA has connected, +esp-link switches to STA-only mode. There is no setting to stay in STA+AP mode. So... if you want +to use AP ensure you set esp-link to AP-only mode. If you want STA+AP mode you're gonna have to +modify the source for yourself. (This stuff is painful to test and rather tricky, so don't expect +the way it works to change.) + +Configuration details +--------------------- + +### Wifi + +After you have serially flashed the module it will create a wifi access point (AP) with an +SSID of the form `ESP_012ABC` where 012ABC is a piece of the module's MAC address. +Using a laptop, phone, or tablet connect to this SSID and then open a browser pointed at +http://192.168.4.1/, you should then see the esp-link web site. + +Now configure the wifi. The desired configuration is for the esp-link to be a +station on your local wifi network so you can communicate with it from all your computers. + +To make this happen, navigate to the wifi page and you should see the esp-link scan +for available networks. You should then see a list of detected networks on the web page and you +can select yours. +Enter a password if your network is secure (highly recommended...) and hit the connect button. + +You should now see that the esp-link has connected to your network and it should show you +its IP address. _Write it down_. You will then have to switch your laptop, phone, or tablet +back to your network and then you can connect to the esp-link's IP address or, depending on your +network's DHCP/DNS config you may be able to go to http://esp-link.local + +At this point the esp-link will have switched to STA mode and be just a station on your +wifi network. These settings are stored in flash and thereby remembered through resets and +power cycles. They are also remembered when you flash new firmware. Only flashing `blank.bin` +via the serial port as indicated above will reset the wifi settings. + +There is a fail-safe, which is that after a reset or a configuration change, if the esp-link +cannot connect to your network it will revert back to AP+STA mode after 15 seconds and thus +both present its `ESP_012ABC`-style network and continue trying to reconnect to the requested network. +You can then connect to the esp-link's AP and reconfigure the station part. + +One open issue (#28) is that esp-link cannot always display the IP address it is getting to the browser +used to configure the ssid/password info. The problem is that the initial STA+AP mode may use +channel 1 and you configure it to connect to an AP on channel 6. This requires the ESP8266's AP +to also switch to channel 6 disconnecting you in the meantime. + +### Hostname, description, DHCP, mDNS + +You can set a hostname on the "home" page, this should be just the hostname and not a domain +name, i.e., something like "test-module-1" and not "test-module-1.mydomain.com". +This has a number of effects: + +- you will see the first 12 chars of the hostname in the menu bar (top left of the page) so + if you have multiple modules you can distinguish them visually +- esp-link will use the hostname in its DHCP request, which allows you to identify the module's + MAC and IP addresses in your DHCP server (typ. your wifi router). In addition, some DHCP + servers will inject these names into the local DNS cache so you can use URLs like + `hostname.local`. +- someday, esp-link will inject the hostname into mDNS (multicast DNS, bonjour, etc...) so + URLs of the form `hostname.local` work for everyone (as of v2.1.beta5 mDNS is disabled due + to reliability issues with it) + +You can also enter a description of up to 128 characters on the home page (bottom right). This +allows you to leave a memo for yourself, such as "installed in basement to control the heating +system". This descritpion is not used anywhere else.